Face image replaying method and device, electronic equipment and storage medium
By introducing a priori information and optical flow estimation completion strategy of the three-dimensional face model, the problem of insufficient robustness of face image repetition in the prior art for head pose changes is solved, and the repetition effect of higher quality and accuracy is achieved .
Patent Information
- Application Number
- CN202411179431.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-08
AI Technical Summary
The existing facial image replay method is difficult to generate accurate replay results when the head postures between the source image and the driving image are large, and it shows weak robustness.
The prior information of the three-dimensional face model is adopted, and the calculation-completion strategy of optical flow estimation and occlusion mask is calculated and completed, combined with the encoding and decoding generation method, the rendering image guides the generation process to improve the quality and robustness of the reenacted image.
It effectively improves the robustness of facial image replay to head pose changes, improves the accuracy of optical flow field and occlusion mask, and generates replay image quality and identity information retention.
Smart Images

Figure CN118918266A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a facial image replay method, device, electronic equipment and storage medium. Background Art
[0002] Face images are a special and important type of images that are widely used in people's daily lives. The face image reenactment task aims to transfer the facial expression and head posture of the driving image to the source image while keeping the identity information of the source image unchanged, thereby generating a new face image. This task has rich application prospects in the fields of media, entertainment, and virtual reality.
[0003] Early solutions mainly came from the field of computer graphics, and most of them relied on 3D modeling and image rendering of the face. In recent years, with the development of artificial intelligence, some deep learning-based methods have significantly improved the quality of generated images through techniques such as image conversion and image deformation. However, when there is a large difference in head posture between the source image and the driving image, existing methods still have difficulty in generating accurate reenactment results and show weak robustness to changes in head posture. Summary of the invention
[0004] The present invention provides a facial image replay method, device, electronic device and storage medium, aiming to solve one of the technical problems in the related art at least to a certain extent.
[0005] On the one hand, an embodiment of the present invention provides a method for replaying a face image, comprising the following steps:
[0006] Obtain a source image and a driving image; construct a source face model and a driving face model based on the source image and the driving image, and then combine and construct a target face model;
[0007] Obtaining first optical flow estimation parameters of the facial region according to the source face model and the target face model; rendering the target face model into a two-dimensional facial image; the first optical flow estimation parameters include a first optical flow field and a first occlusion mask;
[0008] Based on the expression and posture parameters of the source image and the driving image, optical flow estimation parameters are completed for the non-facial area to obtain second optical flow estimation parameters; the second optical flow estimation parameters include a second optical flow field and a second occlusion mask;
[0009] Based on the source image, the two-dimensional facial image and the second optical flow estimation parameters are combined for encoding and decoding to generate a reconstructed image.
[0010] Optionally, constructing a source face model and a driving face model based on the source image and the driving image, and then combining and constructing a target face model, includes the following steps:
[0011] A first parameter of a source image is extracted using a pre-trained three-dimensional reconstruction network; a second parameter of a driving image is extracted using the three-dimensional reconstruction network; the first parameter and the second parameter represent parameters of a three-dimensional deformable model, and the first parameter and the second parameter both include identity parameters and expression posture parameters;
[0012] Based on the first parameter, a source face model is constructed in combination with a basic model of the three-dimensional deformable model; based on the second parameter, a driving face model is constructed in combination with the basic model;
[0013] The identity parameter of the first parameter and the expression and posture parameter of the second parameter are combined to construct a target face model.
[0014] Optionally, obtaining a first optical flow estimation parameter of a facial region according to the source face model and the target face model includes the following steps:
[0015] The first optical flow field of the facial area is obtained according to the mesh vertices of the source face model and the target face model;
[0016] Obtaining a first occlusion mask of the facial region according to mesh triangles of the source face model and the target face model;
[0017] Based on the difference value of the first optical flow field and the first occlusion mask between the source face model and the target face model, the target face model is rendered into a two-dimensional facial image.
[0018] Optionally, completing the optical flow estimation parameters of the non-facial area based on the expression posture parameters of the source image and the driving image to obtain the second optical flow estimation parameters includes the following steps:
[0019] Based on the expression and posture parameters of the source image and the driving image, the motion of the non-facial area is predicted using a pre-trained completion network; the motion includes a third optical flow field, an optical flow field weight coefficient, a third occlusion mask, and an occlusion mask weight coefficient;
[0020] Performing a first weighted fusion on the third optical flow field and the first optical flow field based on the optical flow field weight coefficient to obtain a second optical flow field;
[0021] A second weighted fusion is performed on the third occlusion mask and the first occlusion mask based on the occlusion mask weight coefficient to obtain a second occlusion mask.
[0022] Optionally, based on the source image, encoding and decoding the two-dimensional facial image and the second optical flow estimation parameter to generate a reconstructed image includes the following steps:
[0023] Extracting initial image features of the source image through an encoder;
[0024] Transform the initial image features based on the second optical flow estimation parameters to obtain second image features; the transformation process includes deformation and masking;
[0025] The two-dimensional facial image and the second image features are fused and decoded by the decoder to obtain a reconstructed image.
[0026] On the other hand, an embodiment of the present invention further provides a method for replaying a face image, comprising the following steps:
[0027] Obtain video datasets; configure target models based on 3D reconstruction networks, completion networks, and editing networks; the editing network includes encoders and decoders;
[0028] Obtain two different frames of images from the same video in the video dataset as the first image and the second image;
[0029] Based on the first image and the second image, the second image is used as a driving target to replay the face image using the target model to obtain a target image;
[0030] Constructing a loss function according to the target image and the second image, and adjusting parameters of the target model based on the loss function;
[0031] The target model after parameter adjustment is used to execute the facial image replay method as described in the first aspect above.
[0032] Optionally, constructing a loss function according to the target image and the second image comprises the following steps:
[0033] Based on the target image and the second image, the content loss and style loss are constructed using the pre-trained perceptual network.
[0034] Based on content loss and style loss, the loss function is constructed in combination with preset hyperparameters.
[0035] On the other hand, an embodiment of the present invention provides a facial image replay device, comprising:
[0036] The first module is used to obtain a source image and a driving image; construct a source face model and a driving face model based on the source image and the driving image, and then combine them to obtain a target face model;
[0037] The second module is used to obtain first optical flow estimation parameters of the facial area according to the source face model and the target face model; render the target face model into a two-dimensional facial image; the first optical flow estimation parameters include a first optical flow field and a first occlusion mask;
[0038] The third module is used to complete the optical flow estimation parameters of the non-facial area based on the expression posture parameters of the source image and the driving image to obtain the second optical flow estimation parameters; the second optical flow estimation parameters include the second optical flow field and the second occlusion mask;
[0039] The fourth module is used to generate a re-enacted image based on the source image, in combination with the two-dimensional facial image and the second optical flow estimation parameters for encoding and decoding.
[0040] On the other hand, an embodiment of the present invention further provides a facial image replay device, comprising:
[0041] The fifth module is used to obtain a video data set; configure a target model based on a 3D reconstruction network, a completion network, and an editing network; the editing network includes an encoder and a decoder;
[0042] A sixth module is used to obtain two different frames of images from the same video segment in the video data set as a first image and a second image;
[0043] A seventh module is used for reproducing the face image by using the target model based on the first image and the second image, taking the second image as a driving target, to obtain a target image;
[0044] An eighth module is used to construct a loss function according to the target image and the second image, and adjust parameters of the target model based on the loss function;
[0045] The target model after parameter adjustment is used to execute the facial image replay method of the first aspect mentioned above.
[0046] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used to store programs; the processor executes the program to implement the above-mentioned face image replay method.
[0047] On the other hand, an embodiment of the present invention provides a computer storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned facial image replay method.
[0048] The embodiment of the present invention obtains a source image and a driving image; constructs a source face model and a driving face model based on the source image and the driving image, and then combines and constructs a target face model; obtains the first optical flow estimation parameter of the facial area according to the source face model and the target face model; renders the target face model into a two-dimensional facial image; the first optical flow estimation parameter includes a first optical flow field and a first occlusion mask; the optical flow estimation parameter of the non-facial area is supplemented based on the expression posture parameters of the source image and the driving image to obtain a second optical flow estimation parameter; the second optical flow estimation parameter includes a second optical flow field and a second occlusion mask; based on the source image, the two-dimensional facial image and the second optical flow estimation parameter are combined for encoding and decoding to generate a reenactment image. The embodiment of the present invention makes full use of the prior information of the three-dimensional face model to improve the robustness of the reenactment of the face image to the change of the head posture. The present invention introduces the prior information of the three-dimensional face model from multiple angles of face modeling, optical flow estimation and image generation, adopts a supplementation strategy, improves the accuracy of the optical flow field and the occlusion mask; uses the rendered image to guide the generation process of the reenactment image, and improves the quality of the reenactment image. The embodiments of the present invention can accurately reproduce facial images. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.
[0050] Figure 1 A schematic diagram of an implementation environment for performing face image replay provided by an embodiment of the present invention;
[0051] Figure 2 A schematic flow chart of a facial image replay method provided by an embodiment of the present invention;
[0052] Figure 3 A schematic diagram of the expansion process of constructing a target face model provided by an embodiment of the present invention;
[0053] Figure 4 A schematic diagram of the architecture of a completion network provided by an embodiment of the present invention;
[0054] Figure 5 A schematic diagram of the expanded flow of encoding and decoding to generate a replay image provided by an embodiment of the present invention;
[0055] Figure 6 A schematic diagram of the architecture of an editing network provided by an embodiment of the present invention;
[0056] Figure 7 A schematic diagram of the process architecture principle of a specific application of face image replay provided by an embodiment of the present invention;
[0057] Figure 8A schematic diagram of an example of the effect of comparing experimental results provided by an embodiment of the present invention;
[0058] Fig. 9 A schematic structural diagram of a facial image replay device provided by an embodiment of the present invention;
[0059] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0061] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0063] It is understandable that the face image replay method provided in the embodiment of the present invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this.
[0064] To facilitate the understanding of the technical solution of the present invention, the technical feature terms that may appear in the embodiments of the present invention are first explained:
[0065] 3DMM (3D Morphable Model): a parameterized three-dimensional face model.
[0066] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.
[0067] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.
[0068] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.
[0069] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention.
[0070] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a facial image replay method, which is described below using the facial image replay method applied in the terminal 102 as an example. It can be understood that the facial image replay method can also be applied in the server 101.
[0071] On the one hand, referring to Figure 2 , Figure 2 The flowchart of the face image replay method applied to the terminal provided by the embodiment of the present invention, the execution subject of the face image replay method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method comprises the following steps:
[0072] S100, obtaining a source image and a driving image; constructing a source face model and a driving face model based on the source image and the driving image, and then combining and constructing a target face model;
[0073] Among them, the source face model, the driving face model and the target face model are all three-dimensional face models;
[0074] It should be noted that in some embodiments, Figure 3 As shown, a source face model and a driving face model are constructed based on the source image and the driving image, and then a target face model is constructed by combining them. The following steps may be included: S101, extracting a first parameter of the source image by using a pre-trained 3D reconstruction network; extracting a second parameter of the driving image by using the 3D reconstruction network; the first parameter and the second parameter represent the parameters of the 3D deformable model, and the first parameter and the second parameter both include identity parameters and expression and posture parameters; S102, based on the first parameter, the source face model is constructed in combination with the basic model of the 3D deformable model; based on the second parameter, the driving face model is constructed in combination with the basic model; S103, the identity parameter of the first parameter and the expression and posture parameter of the second parameter are combined to construct a target face model.
[0075] For example, in some specific implementations, a three-dimensional face model is first constructed, which can be specifically implemented as follows:
[0076] 1. Use the pre-trained 3D reconstruction network to extract the 3DMM parameters of the source image and the driving image respectively;
[0077] 2. Construct the source face model and the driving face model based on the extracted parameters and the basic model of 3DMM;
[0078] 3. Combine the identity parameters of the source image and the expression and posture parameters of the driving image to construct the target face model.
[0079] Specifically, the 3D reconstruction network can adopt any pre-trained 3D reconstruction network that can extract 3DMM parameters from a single image.
[0080] S200, obtaining a first optical flow estimation parameter of a facial region according to the source face model and the target face model; rendering the target face model into a two-dimensional facial image;
[0081] Wherein, the first optical flow estimation parameters include a first optical flow field and a first occlusion mask;
[0082] It should be noted that, in some embodiments, obtaining the first optical flow estimation parameter of the facial area based on the source face model and the target face model may include the following steps: S201, obtaining the first optical flow field of the facial area based on the mesh vertices of the source face model and the target face model; S202, obtaining the first occlusion mask of the facial area based on the mesh triangles of the source face model and the target face model.
[0083] For example, in some specific implementations, the facial image is rendered by calculating the optical flow field and the occlusion mask of the facial region, which can be specifically implemented as follows:
[0084] 1. Calculate the optical flow field of the facial area based on the mesh vertices of the source face model and the target face model;
[0085] 2. Obtain the occlusion mask of the facial area based on the mesh triangles of the source face model and the target face model;
[0086] 3. Render the target face model into a two-dimensional face image. The two-dimensional face image is obtained by directly rendering the target face model onto a two-dimensional plane (which can be achieved through an image processing software plug-in).
[0087] Specifically, optical flow field and occlusion mask are key components in optical flow estimation, and they play an important role in unsupervised optical flow techniques. Unsupervised optical flow techniques mainly focus on how to estimate the optical flow field through unsupervised learning while dealing with occlusion issues. The optical flow field represents the motion offset of each pixel from the reference image to the target image, while the occlusion mask is used to deal with occlusion issues in the image, ensuring that the occluded area can be correctly handled when calculating the optical flow.
[0088] Among them, mesh vertices and mesh triangles can be directly acquired through the established three-dimensional face model. Calculating the optical flow field from mesh vertices is a mesh-based method, in which the optical flow field is defined only on the vertices of the mesh, rather than on each pixel. This method can be regarded as a downsampled dense optical flow, in which the motion vectors are only defined on the vertices of the mesh, and the mesh itself can be regarded as a spatially smooth sparse motion field. This method is suitable for application scenarios that need to reduce computational complexity and storage requirements while still providing relatively high-quality motion estimation. In the calculation of mesh triangles, the generation of occlusion masks involves the judgment of triangle visibility, which is usually optimized by culling invisible triangles to optimize rendering performance. In graphics rendering, the calculation of occlusion masks is an important step, which involves determining which triangles need to be rendered and which can be safely culled, thereby improving rendering efficiency and performance. Specifically for the calculation of occlusion masks for mesh triangles, this usually involves the following steps or methods: 1. Visibility test: First, each triangle needs to be tested for visibility to determine whether it is visible from the current viewpoint. This includes checking the position and orientation of the triangles, as well as their relationship with the observer. 2. Eliminate invisible triangles: For those triangles that are determined to be invisible, they can be directly eliminated to avoid processing during the rendering process, thereby saving computing resources. 3. Use an additional array to save filtered occluders: In addition to directly culling invisible triangles, an additional array can be used to save filtered occluder information. Although this method will result in unnecessary data copy consumption, it may be more efficient in some cases, especially when most objects can be filtered out by occlusion culling. 4. Determination of self-occlusion and valid parts: In addition, it is necessary to consider the case of self-occlusion, that is, an occluder not only hides the occluded object, but also may hide part of itself. In this case, only rasterizing the valid occluded part can further improve efficiency. 5. Use normal cone and half angle: In a more specific implementation, geometric information such as normal cone and half angle can be used to define a cone, and the visibility of the triangular face can be determined by this cone. For example, if the observer faces the cone, then all faces in the cone are facing forward and can be rendered; vice versa.
[0089] S300, completing optical flow estimation parameters of the non-facial area based on the expression and posture parameters of the source image and the driving image to obtain second optical flow estimation parameters;
[0090] Wherein, the second optical flow estimation parameters include a second optical flow field and a second occlusion mask;
[0091] It should be noted that, in some embodiments, step S300 may include the following steps: S301, based on the expression and posture parameters of the source image and the driving image, using a pre-trained completion network to predict the movement of the non-facial area; the movement includes a third optical flow field, an optical flow field weight coefficient, a third occlusion mask and an occlusion mask weight coefficient; S302, based on the optical flow field weight coefficient, performing a first weighted fusion of the third optical flow field and the first optical flow field to obtain a second optical flow field; S303, based on the occlusion mask weight coefficient, performing a second weighted fusion of the third occlusion mask and the first occlusion mask to obtain a second occlusion mask.
[0092] Among them, the expression posture parameters of the driving image can be extracted based on the pre-trained 3D reconstruction network.
[0093] For example, in some specific implementations, the optical flow field and the occlusion mask are complemented, which can be implemented as follows:
[0094] 1. Expression and posture parameters v of the driving image / driving face model d Under the guidance of From the source image I s Estimate the motion of non-face areas and predict the optical flow field F p , occlusion mask O p And the weight coefficient M of the two f and M o :
[0095]
[0096] Among them, Figure 4 As shown, Figure 4 To complete the architecture diagram of the network, the parameter injection method is AdaIN (Adaptive Instance Normalization), and the completion network can adopt a conventional encoder-decoder structure.
[0097] 2. According to the weight coefficient M f The calculation result of the optical flow field F c And the prediction result F p Fusion is performed to obtain the complete optical flow field F:
[0098] F=M f ⊙F p +(1-M f )⊙F c ;
[0099] 3. According to the weight coefficient M o The calculation result of the occlusion mask O c And the prediction result O pFusion is performed to obtain the complete occlusion mask O:
[0100] O=M o ⊙O p +(1-M o )⊙O c ;
[0101] S400 , based on the source image, encoding and decoding is performed in combination with the two-dimensional facial image and the second optical flow estimation parameter to generate a reconstructed image.
[0102] It should be noted that in some embodiments, Figure 5 As shown, step S400 may include the following steps: S401, extracting initial image features of the source image through an encoder; S402, transforming the initial image features based on second optical flow estimation parameters to obtain second image features; the transformation processing includes deformation and masking; S403, fusing and decoding the two-dimensional facial image and the second image features through a decoder to obtain a replayed image.
[0103] For example, in some specific implementations, generating a replay image may be implemented as follows:
[0104] 1. By editing the encoder part of the network Extract source image I s Features of X:
[0105]
[0106] Among them, Figure 6 As shown, Figure 6 This is a schematic diagram of the editing network architecture. The injection method of the rendered image is SPADE (Spatially-adaptive Normalization). The editing network can adopt a conventional encoder-decoder structure.
[0107] 2. According to the completed optical flow field F and occlusion mask O, the image feature X is deformed and masked to obtain the transformed image feature X ′ :
[0108] X ′ =O⊙W(X,F);
[0109] 3. By editing the decoder part of the network For the transformed image feature X ′ and the rendered facial image I r Fusion is performed and decoding is performed to obtain the replayed image I g :
[0110]
[0111] Among them, ⊙ represents the mask operation, which can be understood as the mask operation on the image. All the previous ⊙ have the same meaning. This operation is the element-by-element multiplication of two tensors, but the mask is often single-channel, for example, length × width × channel is 256 × 256 × 1, and the image or image feature is often multi-channel, such as RGB image 256 × 256 × 3, then the two tensors have the same spatial size and different number of channels. In this case, the operation can also be performed, as long as the mask is broadcast to the same number of channels as the image / image feature, that is, the mask is copied 3 times in the channel dimension to become 256 × 256 × 3, and then element-by-element multiplication can be performed; W() represents the warp operation, that is, the elements of the image features are sampled according to the optical flow to obtain the deformed image features.
[0112] On the other hand, an embodiment of the present invention further provides a method for replaying a face image, which may include the following steps:
[0113] T100, obtain video data set; configure target model based on 3D reconstruction network, completion network and editing network; editing network includes encoder and decoder;
[0114] T200, obtaining two different frames of images from the same video segment in the video dataset as the first image and the second image;
[0115] T300, based on the first image and the second image, using the second image as a driving target and using a target model to replay the face image to obtain a target image;
[0116] T400, constructing a loss function according to the target image and the second image, and adjusting parameters of the target model based on the loss function;
[0117] The target model after parameter adjustment is used to execute the facial image replay method as described in the first aspect above.
[0118] In some embodiments, constructing a loss function based on the target image and the second image may include the following steps: based on the target image and the second image, using a pre-trained perceptual network to construct content loss and style loss; based on the content loss and style loss, combining preset hyperparameters to construct a loss function.
[0119] For example, in some specific implementations, model training and reasoning can be implemented as follows:
[0120] 1. The entire process is trained end-to-end using a publicly licensed speaking video dataset. In each iteration, two frames of the same video are selected as the source image (i.e., the first image) and the driving image (i.e., the second image). Since the source image and the driving image are from the same video of the same person, the optimization goal is to make the reenactment image as close to the driving image as possible. Loss function Includes content loss based on pre-trained perceptual networks and style loss And using the hyperparameter λ s Perform a weighted sum:
[0121]
[0122] 2. Use the trained model to reenact face images. In the inference stage, the identities of the people in the source image and the driving image can be different.
[0123] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.
[0124] First of all, it should be noted that the existing face image reconstructing methods are not robust enough to the changes in head posture. The purpose of the present invention is to make full use of the prior information of the three-dimensional face model to improve the robustness of face image reconstructing to the changes in head posture. Figure 7 As shown (where Figure 7 The source image and driving image in the figure are from a publicly authorized speaker video dataset, and the replayed image is an effect example of an embodiment of the present invention. The process of the technical solution proposed by the present invention is specifically implemented as follows:
[0125] The first step is to build a 3D face model:
[0126] 1. Use the pre-trained 3D reconstruction network to extract the 3DMM parameters of the source image and the driving image respectively;
[0127] 2. Construct the source face model and the driving face model based on the extracted parameters and the basic model of 3DMM;
[0128] 3. Combine the identity parameters of the source image and the expression and posture parameters of the driving image to construct the target face model.
[0129] The second step is to calculate the optical flow field and occlusion mask of the facial area and render the facial image:
[0130] 1. Calculate the optical flow field of the facial area based on the mesh vertices of the source face model and the target face model;
[0131] 2. Obtain the occlusion mask of the facial area based on the mesh triangles of the source face model and the target face model;
[0132] 3. Render the target face model into a two-dimensional facial image.
[0133] The third step is to complete the optical flow field and occlusion mask:
[0134] 1. Expression and posture parameters v of the driving image / driving face model d Under the guidance of From the source image I s Estimate the motion of non-face areas and predict the optical flow field F p , occlusion mask O p And the weight coefficient M of the two f and M o :
[0135]
[0136] 2. According to the weight coefficient M f The calculation result of the optical flow field F c And the prediction result F p Fusion is performed to obtain the complete optical flow field F:
[0137] F=M f ⊙F p +91-M f )⊙F c ;
[0138] 3. According to the weight coefficient M o The calculation result of the occlusion mask O c And the prediction result O p Fusion is performed to obtain the complete occlusion mask O:
[0139] O=M o ⊙O p +(1-M o )⊙O c ;
[0140] The fourth step is to generate the replay image:
[0141] 1. By editing the encoder part of the network Extract source image I s Features of X:
[0142]
[0143] 2. According to the completed optical flow field F and occlusion mask O, the image feature X is deformed and masked to obtain the transformed image feature X ′ :
[0144] X ′ =O⊙W(X,F);
[0145] 3. By editing the decoder part of the network For the transformed image feature X ′ and the rendered facial image I r Fusion is performed and decoding is performed to obtain the replayed image I g :
[0146]
[0147] Step 5: Model training and reasoning:
[0148] 1. The entire process is trained end-to-end using a publicly licensed speaking video dataset. In each iteration, two frames of the same video are selected as the source image and the driving image. Since the source image and the driving image are from the same video of the same person, the optimization goal is to make the reenactment image as close to the driving image as possible. The loss function Includes content loss based on pre-trained perceptual networks and style loss And using the hyperparameter λ s Perform a weighted sum:
[0149]
[0150] 2. Use the trained model to reenact face images. In the inference stage, the identities of the people in the source image and the driving image can be different.
[0151] Among them, in some specific implementation methods, the specific implementation process of the technical solution of the present invention can be as follows:
[0152] 1. Given two face images, one of them is used as the source image and the other as the driving image. The trained reenactment model is input to obtain the reenactment image, and the facial expression and head posture transfer between the two face images is realized;
[0153] 2. Given a single face image, use it as the source image. First, use the pre-trained 3D reconstruction network to extract 3DMM parameters and construct a source face model. Then, construct the target face model by modifying the expression and posture parameters of the source image. Next, implement the second, third, and fourth steps of the technical solution (the expression and posture parameters used in the third step are the modified parameters) to obtain the replayed image, thereby realizing the facial expression and head posture editing of a single face image.
[0154] Specifically, Figure 8As shown, it is a schematic diagram of the evaluation of the effect comparison between the technical solution of the present invention and other image replay solutions (such as X2Face, FOMM, PIRenderer, DaGAN), wherein the CSIM (comparative similarity index, characterizing the cosine similarity between the source face and the generated face) indicator in the left line chart measures the degree of identity information retention, and the higher the better; the FID (Fréchet Inception Distance, a metric used to evaluate the effect of the generative model, which evaluates the quality of the generative model by calculating the distance between the real image distribution and the generated image distribution) indicator in the right line chart measures the image authenticity, and the lower the better.
[0155] In summary, the present invention introduces prior information of the 3D face model from multiple perspectives, including motion estimation, occlusion modeling, and image generation, and adopts a computation-completion motion estimation and occlusion modeling strategy to improve the accuracy of the optical flow field and occlusion mask; and uses the rendered image to guide the generation process of the reconstructed image to improve the quality of the reconstructed image. Compared with the prior art, the present invention has at least the following beneficial effects:
[0156] 1. A computation-completion motion estimation and occlusion modeling strategy is proposed to improve the accuracy of optical flow field and occlusion mask;
[0157] 2. Injecting the rendered image into the editing network guides the generation process of the reenacted image and improves the quality of the generated image;
[0158] 3. Introducing prior information of 3D face model from multiple perspectives, such as motion estimation, occlusion modeling and image generation, can effectively retain the identity information of the source image and improve the robustness of face image reenactment to head posture changes (see the attached figure for experimental results comparison). Figure 8 ).
[0159] On the other hand, Fig. 9 As shown, the embodiment of the present invention provides a facial image replay device 900, comprising:
[0160] The first module 901 is used to obtain a source image and a driving image; construct a source face model and a driving face model based on the source image and the driving image, and then combine and construct a target face model;
[0161] The second module 902 is used to obtain first optical flow estimation parameters of the facial area according to the source face model and the target face model; render the target face model into a two-dimensional facial image; the first optical flow estimation parameters include a first optical flow field and a first occlusion mask;
[0162] The third module 903 is used to complete the optical flow estimation parameters of the non-face area based on the expression posture parameters of the source image and the driving image to obtain second optical flow estimation parameters; the second optical flow estimation parameters include a second optical flow field and a second occlusion mask;
[0163] The fourth module 904 is used to generate a reconstructed image by encoding and decoding based on the source image in combination with the two-dimensional facial image and the second optical flow estimation parameter.
[0164] On the other hand, an embodiment of the present invention further provides a facial image replay device, which may include:
[0165] The fifth module is used to obtain a video data set; configure a target model based on a 3D reconstruction network, a completion network, and an editing network; the editing network includes an encoder and a decoder;
[0166] A sixth module is used to obtain two different frames of images from the same video segment in the video data set as a first image and a second image;
[0167] A seventh module is used for reproducing the face image by using the target model based on the first image and the second image, taking the second image as a driving target, to obtain a target image;
[0168] An eighth module is used to construct a loss function according to the target image and the second image, and adjust parameters of the target model based on the loss function;
[0169] The target model after parameter adjustment is used to execute the facial image replay method of the first aspect mentioned above.
[0170] On the other hand, an embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above sensitive information method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0171] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0172] like Fig.10 As shown, Fig.10 A specific example of the hardware structure of an electronic device 1000 according to an embodiment is shown. The electronic device 1000 includes:
[0173] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.
[0174] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the network node population optimization method of the embodiment of the present invention;
[0175] Input / output interface 1003, used to implement information input and output;
[0176] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);
[0177] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );
[0178] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .
[0179] The electronic device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0180] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0181] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.
[0182] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read to Only Memory, CD to ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0183] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0184] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.
[0185] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0186] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0187] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD to ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.
[0188] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0189] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0190] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0191] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution device, apparatus or device (such as a computer-based device, a device including a processor, or other device that can fetch instructions from an instruction execution device, apparatus or device and execute the instructions), or in conjunction with such instruction execution device, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution device, apparatus or device, or in conjunction with such instruction execution device, apparatus or device.
[0192] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0193] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0194] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0195] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0196] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.
Claims
1. A method for reenacting a face image, characterized in that: The following steps are involved: Acquire a source image and a driving image; construct a source face model and a driving face model based on the source image and the driving image, and then combine and construct a target face model; Obtaining first optical flow estimation parameters of a facial region according to the source face model and the target face model; rendering the target face model into a two-dimensional facial image; the first optical flow estimation parameters include a first optical flow field and a first occlusion mask; Based on the expression and posture parameters of the source image and the drive image, optical flow estimation parameters are completed for the non-facial area to obtain second optical flow estimation parameters; the second optical flow estimation parameters include a second optical flow field and a second occlusion mask; Based on the source image, the two-dimensional facial image and the second optical flow estimation parameters are combined to perform encoding and decoding to generate a reconstructed image.
2. The facial image replay method according to claim 1, characterized in that: The method of constructing a source face model and a driving face model based on the source image and the driving image, and then combining and constructing a target face model, comprises the following steps: A first parameter of the source image is extracted using a pre-trained three-dimensional reconstruction network; a second parameter of the driving image is extracted using the three-dimensional reconstruction network; the first parameter and the second parameter represent three-dimensional deformable model parameters, and the first parameter and the second parameter both include identity parameters and the expression posture parameters; Based on the first parameter, the source face model is constructed in combination with a basic model of a three-dimensional deformable model; based on the second parameter, the driving face model is constructed in combination with the basic model; The identity parameter of the first parameter and the expression posture parameter of the second parameter are combined to construct the target face model.
3. The facial image replay method according to claim 1, characterized in that: The step of obtaining a first optical flow estimation parameter of a facial region according to the source face model and the target face model comprises the following steps: Obtaining the first optical flow field of the facial area according to mesh vertices of the source face model and the target face model; The first occlusion mask of the facial area is obtained according to mesh triangle processing of the source face model and the target face model.
4. The facial image replay method according to claim 1, characterized in that: The method of completing the optical flow estimation parameters of the non-facial area based on the expression posture parameters of the source image and the drive image to obtain the second optical flow estimation parameters includes the following steps: Based on the expression and posture parameters of the source image and the drive image, the motion of the non-facial area is predicted using a pre-trained completion network; the motion includes a third optical flow field, an optical flow field weight coefficient, a third occlusion mask, and an occlusion mask weight coefficient; Performing a first weighted fusion on the third optical flow field and the first optical flow field based on the optical flow field weight coefficient to obtain the second optical flow field; The third occlusion mask and the first occlusion mask are subjected to a second weighted fusion based on the occlusion mask weight coefficient to obtain the second occlusion mask.
5. The facial image replay method according to claim 1, characterized in that: The method of encoding and decoding the source image in combination with the two-dimensional facial image and the second optical flow estimation parameter to generate a reconstructed image comprises the following steps: Extracting initial image features of the source image through an encoder; Performing transformation processing on the initial image feature based on the second optical flow estimation parameter to obtain a second image feature; the transformation processing includes deformation and masking; The two-dimensional facial image and the second image feature are fused and decoded by a decoder to obtain the replayed image.
6. A method for reenacting a face image, characterized in that: The following steps are involved: Acquire a video data set; configure a target model based on a 3D reconstruction network, a completion network, and an editing network; the editing network includes an encoder and a decoder; Acquire two different frames of images from the same video segment in the video data set as a first image and a second image; Based on the first image and the second image, using the second image as a driving target and using the target model to replay a face image to obtain a target image; Constructing a loss function according to the target image and the second image, and adjusting parameters of the target model based on the loss function; Wherein, the target model after the parameter adjustment is used to execute the facial image replay method as described in any one of claims 1 to 5.
7. The facial image replay method according to claim 6, characterized in that: The step of constructing a loss function according to the target image and the second image comprises the following steps: Based on the target image and the second image, constructing a content loss and a style loss using a pre-trained perceptual network; The loss function is constructed based on the content loss and the style loss in combination with preset hyperparameters.
8. A facial image replay device, characterized in that: include: The first module is used to obtain a source image and a driving image; Based on the source image and the driving image, a source face model and a driving face model are correspondingly constructed, and then combined to obtain a target face model; The second module is used to obtain first optical flow estimation parameters of a facial area according to the source face model and the target face model; render the target face model into a two-dimensional facial image; the first optical flow estimation parameters include a first optical flow field and a first occlusion mask; A third module is used to complete the optical flow estimation parameters of the non-facial area based on the expression posture parameters of the source image and the drive image to obtain second optical flow estimation parameters; the second optical flow estimation parameters include a second optical flow field and a second occlusion mask; The fourth module is used to generate a reconstructed image by encoding and decoding based on the source image, the two-dimensional facial image and the second optical flow estimation parameters.
9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5 or 6 to 7.
10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 5 or 6 to 7 when executed by the processor.
