Digital face image generation method and device, video generation method and device and storage medium

Abstract

The invention relates to the technical field of three-dimensional face images, in particular to a digital face image generation method and device, a video generation method and device and a storage medium. According to the method, source three-dimensional mesh grid points and target three-dimensional mesh grid points are generated through cross-identity three-dimensional mesh grid point generation processing, the source three-dimensional mesh grid points are constructed by parameters of a source image, the target three-dimensional mesh grid points fuse identity features and facial details of the source image, and dense three-dimensional facial geometric information is obtained by combining expression and posture information of a driving image. In order to further improve the effect of speaker video generation, two three-dimensional mesh point sensing mechanisms are adopted, the first mechanism is a key point detection mechanism guided by three-dimensional mesh points, and geometric representation of the three-dimensional mesh points and appearance features of an input image are combined to predict more accurate face key points. And the second method is based on a multi-layer cross-modal attention mechanism, the completeness of a face structure can be kept, and fine face movement is captured, so that the fine granularity of face detail generation is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional face images, and in particular to a digital face image generation method, a video generation method, a device and a storage medium. Background Art

[0002] In recent years, with the growing demand for highly realistic and controllable facial animation, significant progress has been made in the field of digital talking video generation. Talking video generation aims to generate talking videos by driving the target face from an arbitrary portrait video. Currently, depth-aware generative adversarial networks have achieved state-of-the-art synthesis results in the field of talking video generation. However, due to the lack of sufficient real data, accurate depth estimation becomes extremely difficult, and this method cannot effectively decompose identity information from other 3D information, resulting in inefficient utilization of 3D geometric information during the generation process. Summary of the invention

[0003] The main purpose of the present invention is to provide a digital face image generation method, a video generation method, a device and a storage medium, aiming to generate a digital face image required for a high-quality speaking video.

[0004] To achieve the above object, the present invention discloses a method for generating a digital face image in a first aspect, comprising the following steps:

[0005] Obtain source image and driving image;

[0006] Performing cross-identity three-dimensional mesh point generation processing on the source image and the driving image respectively to obtain source three-dimensional mesh points and target three-dimensional mesh points;

[0007] The source 3D mesh grid points are spliced ​​with the source image to perform key point detection to obtain the sparse key points of the source image; the target 3D mesh grid points are spliced ​​with the driving image to perform key point detection to obtain the sparse key points of the driving image;

[0008] Generate motion field and occlusion map according to sparse key points of source image and sparse key points of driving image;

[0009] Based on the multi-layer cross-modal attention mechanism, the source image, source 3D mesh points, target 3D mesh points, motion field and occlusion map are processed to integrate the facial collection information into each layer of the generation process to obtain a digital face synthetic image.

[0010] Optionally, in the first aspect of the present invention, performing cross-identity three-dimensional mesh point generation processing on the source image and the drive image respectively comprises the following steps:

[0011] Extract identity features, expression features and posture features of source images and driving images;

[0012] Extract the detail features of the source image;

[0013] Generate three-dimensional mesh points of the source image according to the identity features of the source image, the expression features of the source image and the posture features of the source image;

[0014] Generate three-dimensional mesh points of the driving image according to the identity features of the source image, the expression features of the driving image and the posture features of the driving image;

[0015] Generate UV displacement map related to expression based on detail features;

[0016] The UV displacement map is fused with the source image three-dimensional mesh points and the drive image three-dimensional mesh points respectively to obtain source three-dimensional mesh points and target three-dimensional mesh points.

[0017] Optionally, in the first aspect of the present invention, processing a source image, a source three-dimensional mesh grid point, a target three-dimensional mesh grid point, a motion field, and an occlusion map based on a multi-layer cross-modal attention mechanism comprises the following steps:

[0018] Extracting features of the source image, source 3D mesh points and target 3D mesh points respectively to obtain appearance feature information, source 3D mesh point feature information and target 3D mesh point feature information;

[0019] The deepest appearance feature information is distorted by the motion field and fused with the occlusion map to obtain the distorted feature information;

[0020] The distorted feature information and the deepest target 3D mesh point feature information are input into the attention module to obtain the deepest fine feature information;

[0021] Starting from the deepest layer, the fine feature information obtained in each layer is fused with the attention results of the previous layer and up-sampled to finally generate a digital face composite image. The attention results of the previous layer are the output of the attention module after the feature information of the source 3D mesh points of the previous layer and the appearance feature information of the previous layer are input into the attention module.

[0022] The second aspect of the present invention discloses a method for generating a digital face video, comprising the following steps:

[0023] Extract each frame of image as a driving image according to the video timing sequence;

[0024] Each driving image executes the steps of the digital face image generation method according to the first aspect of the present invention in a time sequence to generate a corresponding digital face composite image;

[0025] The digital face composite images are merged in the order of generation to generate a speaking video.

[0026] The third aspect of the present invention discloses a digital face image generating device, comprising:

[0027] An acquisition module, used for acquiring a source image and a driving image;

[0028] A cross-identity 3D mesh grid point generation module is used to perform cross-identity 3D mesh grid point generation processing on the source image and the driving image respectively to obtain source 3D mesh grid points and target 3D mesh grid points;

[0029] A key point detection module, used for performing key point detection after splicing the source three-dimensional mesh grid points with the source image to obtain the sparse key points of the source image; and for performing key point detection after splicing the target three-dimensional mesh grid points with the driving image to obtain the sparse key points of the driving image;

[0030] Dense motion module, used to generate motion fields and occlusion maps based on sparse key points of source images and sparse key points of driving images;

[0031] A multi-layer cross-modal attention module is used to process source images, source 3D mesh points, target 3D mesh points, motion fields and occlusion maps based on a multi-layer cross-modal attention mechanism to integrate facial set information into each layer of the generation process to obtain a digital face synthetic image.

[0032] Optionally, in the third aspect of the present invention, the cross-identity three-dimensional mesh grid point generation module includes:

[0033] A first extraction unit, used for extracting identity features, expression features and posture features of the source image and the driving image;

[0034] A second extraction unit, used for extracting detail features of the source image;

[0035] FLAME model, used to generate three-dimensional mesh points of the source image according to the identity features of the source image, the expression features of the source image and the posture features of the source image;

[0036] The FLAME model is also used to generate the three-dimensional mesh points of the driving image based on the identity features of the source image, the expression features of the driving image, and the posture features of the driving image;

[0037] A UV displacement map generating unit, used for generating a UV displacement map related to the expression according to the detail features;

[0038] The fusion unit is used to fuse the UV displacement map with the source image three-dimensional mesh grid points and the drive image three-dimensional mesh grid points respectively to obtain source three-dimensional mesh grid points and target three-dimensional mesh grid points.

[0039] Optionally, in the third aspect of the present invention, the multi-layer cross-modal attention module comprises the following steps:

[0040] A feature extraction unit is used to extract features from a source image, source three-dimensional mesh points, and target three-dimensional mesh points, respectively, to obtain appearance feature information, source three-dimensional mesh point feature information, and target three-dimensional mesh point feature information;

[0041] A warping unit, used to warp the deepest appearance feature information through the motion field and fuse it with the occlusion map to obtain the warped feature information;

[0042] The attention unit is used to generate the deepest fine feature information based on the distorted feature information and the deepest target 3D mesh network;

[0043] The attention unit is also used to fuse the fine feature information obtained in each layer with the attention results of the previous layer, starting from the deepest layer, and perform upsampling operations to finally generate a digital face composite image, where the attention results of the previous layer are the output after the feature information of the grid points of the previous layer source three-dimensional mesh and the appearance feature information of the previous layer are input into the attention module.

[0044] The fourth aspect of the present invention discloses a digital face video generation device, comprising:

[0045] A driving image extraction module is used to extract each frame of image as a driving image according to the video time sequence;

[0046] A calling module, used to input each driving image into the digital face image generating device disclosed in the third aspect of the present invention in a time sequence to generate a corresponding digital face composite image;

[0047] The merging module is used to merge the digital face synthesis images according to the generation order to generate the speaking person video.

[0048] The fifth aspect of the present invention discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the first aspect or the second aspect of the present invention is implemented.

[0049] A sixth aspect of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in any one of the first aspect or the second aspect of the present invention is implemented.

[0050] The technical solution provided by the present invention may include the following beneficial effects:

[0051] The digital face image generation method disclosed in the present invention first generates source three-dimensional mesh points and target three-dimensional mesh points through cross-identity three-dimensional mesh point generation processing. The source three-dimensional mesh points are constructed by the parameters of the source image, and the target three-dimensional mesh points fuse the identity features and facial details of the source image, and combine the expression and posture information of the driving image to obtain dense three-dimensional facial geometric information. In order to further improve the effect of speaker video generation, two three-dimensional mesh point perception mechanisms are adopted. The first is a key point detection mechanism guided by three-dimensional mesh points, which combines the geometric representation of three-dimensional mesh points with the appearance features of the input image to predict more accurate facial key points. The second is based on a multi-layer cross-modal attention mechanism, which can maintain the integrity of facial structure and capture subtle facial movements, thereby enhancing the fine-grained generation of facial details. By introducing more accurate three-dimensional mesh points with rich geometric information, the method proposed in this study is superior to the existing technology in terms of generation effect. A large number of experimental results show that the method provided by the present invention can generate digital face images required for high-quality speaker videos, and surpasses the current state-of-the-art technology in both qualitative and quantitative evaluations. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0053] Figure 1 A network architecture diagram of the digital face image generation method of the present invention;

[0054] Figure 2 A module diagram for generating cross-identity three-dimensional mesh points of the present invention;

[0055] Figure 3 This is a multi-layer cross-modal attention module diagram of the present invention;

[0056] Figure 4 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0059] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text is to include three parallel solutions. Taking "A and / or B as an example", it includes solution A, or solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0061] In recent years, with the growing demand for highly realistic and controllable facial animation, significant progress has been made in the field of digital talking video generation. Talking video generation aims to generate talking videos by driving the target face from an arbitrary portrait video. Currently, depth-aware generative adversarial networks have achieved state-of-the-art synthesis results in the field of talking video generation. However, due to the lack of sufficient real data, accurate depth estimation becomes extremely difficult, and this method cannot effectively decompose identity information from other 3D information, resulting in inefficient utilization of 3D geometric information during the generation process.

[0062] To this end, the following Figures 1 to 3 , describing a method for generating a digital face image according to a first embodiment of the present invention, comprising the following steps:

[0063] Step S1: Acquire a source image and a drive image;

[0064] Step S2: performing cross-identity three-dimensional mesh point generation processing on the source image and the driving image respectively to obtain source three-dimensional mesh points and target three-dimensional mesh points;

[0065] Step S3: After the source 3D mesh grid points are spliced ​​with the source image, key point detection is performed to obtain the sparse key points of the source image; the target 3D mesh grid points are spliced ​​with the driving image to perform key point detection to obtain the sparse key points of the driving image;

[0066] Step S4: generating a motion field and an occlusion map according to the sparse key points of the source image and the sparse key points of the driving image;

[0067] Step S5: Process the source image, source 3D mesh points, target 3D mesh points, motion field and occlusion map based on a multi-layer cross-modal attention mechanism to integrate facial set information into each layer of the generation process to obtain a digital face composite image.

[0068] The digital face image generation method disclosed in the present invention first generates source three-dimensional mesh points and target three-dimensional mesh points through cross-identity three-dimensional mesh point generation processing. The source three-dimensional mesh points are constructed by the parameters of the source image, and the target three-dimensional mesh points fuse the identity features and facial details of the source image, and combine the expression and posture information of the driving image to obtain dense three-dimensional facial geometry information. In order to further improve the effect of speaker video generation, two three-dimensional mesh point perception mechanisms are adopted. The first is a key point detection mechanism guided by three-dimensional mesh points, which combines the geometric representation of three-dimensional mesh points with the appearance features of the input image to predict more accurate facial key points. The second is based on a multi-layer cross-modal attention mechanism, which can maintain the integrity of the facial structure and capture subtle facial movements, thereby enhancing the fine-grained generation of facial details. By introducing more accurate three-dimensional mesh points with rich geometric information, the method proposed in this study is superior to the existing technology in terms of generation effect. A large number of experimental results show that the method provided by the present invention can generate digital face images required for high-quality speaking videos, and surpasses the current most advanced technology in qualitative evaluation, such as visual effect, and quantitative evaluation, such as PSNR peak signal-to-noise ratio, SSIM structural similarity and other evaluation indicators.

[0069] Among them, Figure 2 As shown, the cross-identity three-dimensional mesh point generation process for the source image and the driving image includes the following steps:

[0070] Step S21: extracting identity features, expression features and posture features of the source image and the driving image;

[0071] Step S22: extracting detail features of the source image;

[0072] Step S23: generating three-dimensional mesh points of the source image according to the identity features of the source image, the expression features of the source image and the posture features of the source image;

[0073] Step S24: generating three-dimensional mesh points of the driving image according to the identity features of the source image, the expression features of the driving image and the posture features of the driving image;

[0074] Step S25: generating a UV displacement map related to the expression according to the detail features;

[0075] Step S26: fusing the UV displacement map with the source image three-dimensional mesh points and the drive image three-dimensional mesh points respectively to obtain source three-dimensional mesh points and target three-dimensional mesh points.

[0076] Specifically, the cross-identity three-dimensional mesh point generation process can be implemented by inputting the source image and the driving image into a cross-identity three-dimensional mesh point generation module.

[0077] More specifically, the cross-identity 3D mesh point generation module constructs source 3D mesh points and target 3D mesh points from the input image for use in the 3D mesh point perception mechanism. The workflow of the cross-identity 3D mesh point generation module is as follows: Figure 2 As shown:

[0078] After inputting the image I, the rough encoder of EMOCA is used to generate the FLAME geometric parameters ψ, β and θ, namely E coarse (I)→(ψ,β,θ), where E coarse represents the rough encoder, I represents the input image, ψ is the expression feature, β is the identity feature and θ is the facial pose feature.

[0079] The geometric parameters ψ, β and θ corresponding to the image I are input into the FLAME model, and the FLAME model generates a rough three-dimensional mesh with vertices V and triangles F; that is, M f (ψ,β,θ)→M′, where, M f represents the FLAME model, (ψ, β, θ) represents the input geometric parameters ψ, β and θ, and M′ is a rough three-dimensional mesh point with vertex V and triangle F.

[0080] In order to add details to the coarse 3D mesh points, the DECA detail encoder is used to extract the subject-specific detail parameters δ, namely E d (I)→δ,E d represents the detail encoder, (I) represents the input image, and δ is the detail parameter. DECA’s detail encoder generates a UV displacement map related to the expression according to the detail parameter, i.e., Dd (δ,ψ,θ jaw )→D, where D d is the detail decoder, (δ, ψ, θjaw) are the input parameters, δ is the detail parameter, ψ is the expression feature parameter, θjaw is the jaw posture parameter, and D is the UV displacement map.

[0081] Finally, the UV displacement map is combined with the rough 3D mesh grid points to generate detailed 3D mesh grid points, namely F c (D,M′)→M, where F C is the fusion function, D is the UV displacement map, M′ is the rough 3D mesh point, and M is the detailed 3D mesh point.

[0082] Then in this embodiment, when the input image is the source image I s and drive image I d When , we get the source image I s The identity features, expression features and posture features and the driving image I d identity features, expression features and posture features.

[0083] The source image I s The identity features, expression features and posture features of the source image are input into the FLAME model, and the FLAME model generates a three-dimensional mesh grid point M with vertices V and triangles F. s ′; the source image I s Identity features, driving image I d The facial expression and posture features are input into the FLAME model, and the FLAME model generates a three-dimensional mesh point M of the driving image with vertices V and triangles F. t ′;

[0084] After the source image is input into DECA’s detail encoder to extract the subject-specific detail parameters, the detail decoder of DECA generates a UV displacement map related to the expression. Finally, the UV displacement map is respectively compared with the source image 3D mesh points M s ′ and the driving image 3D mesh grid point M t ′ Combined with the source three-dimensional mesh point M s and the target 3D mesh point M t In this way, the dense three-dimensional mesh points required for the target face can be obtained.

[0085] After obtaining the source 3D mesh points M s and the target 3D mesh point M t After that, they will be associated with their corresponding source image I s and drive image Id spliced ​​and input into the key point detection module to more accurately predict a set of sparse key points of the face, expressed as and where K is the number of detected facial keypoints, and the subscripts s and d denote the source image and the driving image, respectively.

[0086] like Figure 3 As shown, processing the source image, source 3D mesh points, target 3D mesh points, motion field and occlusion map based on the multi-layer cross-modal attention mechanism includes the following steps:

[0087] Step S51: extracting features of the source image, source 3D mesh points and target 3D mesh points respectively to obtain appearance feature information, source 3D mesh point feature information and target 3D mesh point feature information;

[0088] Step S52: distorting the deepest appearance feature information through the motion field and fusing it with the occlusion map to obtain distorted feature information;

[0089] Step S53: input the distorted feature information and the deepest target three-dimensional mesh point feature information into the attention module to obtain the deepest fine feature information;

[0090] Step S54: Starting from the deepest layer, the fine feature information obtained in each layer is respectively fused with the attention result of the previous layer and an upsampling operation is performed to finally generate a digital face composite image, wherein the attention result of the previous layer is the output after the feature information of the source three-dimensional mesh points of the previous layer and the appearance feature information of the previous layer are input into the attention module. The facial set information is integrated into each layer of the generation process to obtain a digital face composite image.

[0091] In the specific generation process, Figure 3 As shown, we use an encoder-decoder architecture to generate images. First, we use three encoders to generate N layers of encoded appearance feature information. Source 3D mesh grid point feature information and target 3D mesh point feature information Then the appearance feature information and source 3D mesh point feature information A cross-modal attention mechanism is used to incorporate facial geometry information into the generation process.

[0092] It is worth noting that in the deepest layer of the encoder-decoder, the motion field generated by the dense motion generation module To distort the appearance feature information and compare it with the occlusion map Fusion to identify the feature information area that needs to be filled, so as to obtain the distorted feature information F w .

[0093] After that, we use the distorted feature information F w and target 3D mesh point feature information A cross-modal attention mechanism is performed between the two to capture tiny facial movements and obtain the deepest and finest feature information. We then use this fine-grained feature information Combined with the attention result of the previous layer and up-sampled, the synthetic image I is finally generated. g .

[0094] The second aspect of the present invention discloses a method for generating a digital face video, comprising:

[0095] Extract each frame of image as a driving image according to the video timing sequence;

[0096] Each driving image executes the steps of the digital face image generation method in a time sequence to generate a corresponding digital face composite image;

[0097] The digital face composite images are merged in the order of generation to generate a speaking video.

[0098] In this way, a high-quality speaking video is generated by merging the digital face synthetic images in the order in which they are generated.

[0099] The third aspect of the present invention discloses a digital face image generation device, comprising the following steps:

[0100] An acquisition module, used for acquiring a source image and a driving image;

[0101] A cross-identity 3D mesh grid point generation module is used to perform cross-identity 3D mesh grid point generation processing on the source image and the driving image respectively to obtain source 3D mesh grid points and target 3D mesh grid points;

[0102] A key point detection module, used for performing key point detection after splicing the source three-dimensional mesh grid points with the source image to obtain the sparse key points of the source image; and for performing key point detection after splicing the target three-dimensional mesh grid points with the driving image to obtain the sparse key points of the driving image;

[0103] Dense motion module, used to generate motion fields and occlusion maps based on sparse key points of source images and sparse key points of driving images;

[0104] A multi-layer cross-modal attention module is used to process source images, source 3D mesh points, target 3D mesh points, motion fields and occlusion maps based on a multi-layer cross-modal attention mechanism to integrate facial set information into each layer of the generation process to obtain a digital face synthetic image.

[0105] The digital face image generation method disclosed in the present invention first generates source three-dimensional mesh points and target three-dimensional mesh points through cross-identity three-dimensional mesh point generation processing. The source three-dimensional mesh points are constructed by the parameters of the source image, and the target three-dimensional mesh points fuse the identity features and facial details of the source image, and combine the expression and posture information of the driving image to obtain dense three-dimensional facial geometric information. In order to further improve the effect of speaker video generation, two three-dimensional mesh point perception mechanisms are adopted. The first is a key point detection mechanism guided by three-dimensional mesh points, which combines the geometric representation of three-dimensional mesh points with the appearance features of the input image to predict more accurate facial key points. The second is based on a multi-layer cross-modal attention mechanism, which can maintain the integrity of facial structure and capture subtle facial movements, thereby enhancing the fine-grained generation of facial details. By introducing more accurate three-dimensional mesh points with rich geometric information, the method proposed in this study is superior to the existing technology in terms of generation effect. A large number of experimental results show that the method provided by the present invention can generate digital face images required for high-quality speaker videos, and surpasses the current state-of-the-art technology in both qualitative and quantitative evaluations.

[0106] Specifically, the cross-identity three-dimensional mesh grid point generation module includes:

[0107] A first extraction unit, used for extracting identity features, expression features and posture features of the source image and the driving image;

[0108] The second extraction unit is used to extract detail features of the source image; the second extraction unit may be a detail encoder of DECA.

[0109] FLAME model, used to generate three-dimensional mesh points of the source image according to the identity features of the source image, the expression features of the source image and the posture features of the source image;

[0110] The FLAME model is also used to generate the three-dimensional mesh points of the driving image based on the identity features of the source image, the expression features of the driving image, and the posture features of the driving image;

[0111] The UV displacement map generation unit is used to generate a UV displacement map related to the expression according to the detail features; the UV displacement map generation unit can be a detail decoder of DECA.

[0112] The fusion unit is used to fuse the UV displacement map with the source image three-dimensional mesh grid points and the drive image three-dimensional mesh grid points respectively to obtain source three-dimensional mesh grid points and target three-dimensional mesh grid points.

[0113] In this embodiment, when the input image is a source image I s and drive image I d When , the first extraction unit will obtain the source image I s The identity features, expression features and posture features and the driving image I d identity features, expression features and posture features.

[0114] The source image I s The identity features, expression features and posture features of the source image are input into the FLAME model, and the FLAME model generates a three-dimensional mesh grid point M with vertices V and triangles F. s ′; the source image I s Identity features, driving image I d The facial expression and posture features are input into the FLAME model, and the FLAME model generates a three-dimensional mesh point M of the driving image with vertices V and triangles F. t ′;

[0115] After the source image is input into the second extraction unit to extract the detail parameters specific to the subject, the UV displacement map generation unit generates a UV displacement map related to the expression. Finally, the UV displacement map is respectively compared with the grid points M of the three-dimensional mesh of the source image. s ′ and the driving image 3D mesh grid point M t ′ Combined with the source three-dimensional mesh point M s and the target 3D mesh point M t In this way, the dense three-dimensional mesh points required for the target face can be obtained.

[0116] After obtaining the source 3D mesh points M s and the target 3D mesh point M t After that, they will be associated with their corresponding source image I s and drive image I d spliced ​​and input into the key point detection module to more accurately predict a set of sparse key points of the face, expressed as and where K is the number of detected facial keypoints, and the subscripts s and d denote the source image and the driving image, respectively.

[0117] Specifically, the multi-layer cross-modal attention module includes the following steps:

[0118] A feature extraction unit is used to extract features from a source image, source three-dimensional mesh points, and target three-dimensional mesh points, respectively, to obtain appearance feature information, source three-dimensional mesh point feature information, and target three-dimensional mesh point feature information;

[0119] A warping unit, used to warp the deepest appearance feature information through the motion field and fuse it with the occlusion map to obtain the warped feature information;

[0120] The attention unit is used to generate the deepest fine feature information based on the distorted feature information and the deepest target 3D mesh network;

[0121] The attention unit is also used to fuse the fine feature information obtained in each layer with the attention results of the previous layer, starting from the deepest layer, and perform upsampling operations to finally generate a digital face composite image, where the attention results of the previous layer are the output after the feature information of the grid points of the previous layer source three-dimensional mesh and the appearance feature information of the previous layer are input into the attention module.

[0122] In the specific generation process, Figure 3 As shown, we use an encoder-decoder architecture to generate images. First, we use three encoders to generate N layers of encoded appearance feature information. Source 3D mesh grid point feature information and target 3D mesh point feature information Then the appearance feature information and source 3D mesh point feature information A cross-modal attention mechanism is used to incorporate facial geometry information into the generation process.

[0123] It is worth noting that in the deepest layer of the encoder-decoder, the motion field generated by the dense motion generation module To distort the appearance feature information and compare it with the occlusion map Fusion to identify the feature information area that needs to be filled, so as to obtain the distorted feature information F w .

[0124] After that, we use the distorted feature information F w and target 3D mesh point feature information A cross-modal attention mechanism is performed between the two to capture tiny facial movements and obtain the deepest and finest feature information. We then use this fine-grained feature information Combined with the attention result of the previous layer and up-sampled, the synthetic image I is finally generated. g .

[0125] The fourth aspect of the present invention discloses a digital face video generation device, comprising:

[0126] A driving image extraction module is used to extract each frame of image as a driving image according to the video time sequence;

[0127] A calling module, used to input each driving image into the digital face image generating device in time sequence to generate a corresponding digital face composite image;

[0128] The merging module is used to merge the digital face synthesis images according to the generation order to generate the speaking person video.

[0129] like Figure 4 As shown, the fifth aspect of the present invention discloses an electronic device 400, including: a processor 401 and a memory 402. Among them, the processor 401 and the memory 402 are connected, such as through a bus 403. Further, the electronic device 400 may also include a transceiver 404. It should be noted that in actual applications, the transceiver 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application. Among them, the processor 401 is applied to the embodiments of the present application to implement the acquisition module, cross-identity three-dimensional mesh grid point generation module, key point detection module, dense motion module, multi-layer cross-modal attention module, drive image extraction module, call module and merging module of the third and fourth aspects of the present invention. The processor 401 can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present application. The processor 401 can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The bus 403 may include a path to transmit information between the above components. The bus 403 may be a PCI bus or an EISA bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4Only one bold line is used to represent it, but it does not mean that there is only one bus or one type of bus. The memory 402 can be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, or an EEPROM, CD-ROM or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 402 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 402 to implement the digital face image generation method and the digital face video generation method provided by the present invention.

[0130] The sixth aspect of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in any one of the first and second aspects of the present invention is implemented.

[0131] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for generating a digital face image, characterized in that: The steps include: Obtain source image and driving image; Performing cross-identity three-dimensional mesh point generation processing on the source image and the driving image respectively to obtain source three-dimensional mesh points and target three-dimensional mesh points; The source 3D mesh points are spliced ​​with the source image to perform key point detection to obtain the sparse key points of the source image; The target 3D mesh points are spliced ​​with the driving image to perform key point detection to obtain the sparse key points of the driving image; Generate motion field and occlusion map according to sparse key points of source image and sparse key points of driving image; Based on the multi-layer cross-modal attention mechanism, the source image, source 3D mesh points, target 3D mesh points, motion field and occlusion map are processed to integrate the facial collection information into each layer of the generation process to obtain a digital face synthetic image.

2. The method for generating a digital face image according to claim 1, characterized in that: The cross-identity three-dimensional mesh point generation process for the source image and the driving image includes the following steps: Extract identity features, expression features and posture features of source images and driving images; Extract the detail features of the source image; Generate three-dimensional mesh points of the source image according to the identity features of the source image, the expression features of the source image and the posture features of the source image; Generate three-dimensional mesh points of the driving image according to the identity features of the source image, the expression features of the driving image and the posture features of the driving image; Generate UV displacement map related to expression based on detail features; The UV displacement map is fused with the source image three-dimensional mesh points and the drive image three-dimensional mesh points respectively to obtain source three-dimensional mesh points and target three-dimensional mesh points.

3. The method for generating a digital face image according to claim 1, wherein: Processing source images, source 3D mesh points, target 3D mesh points, motion fields, and occlusion maps based on a multi-layer cross-modal attention mechanism includes the following steps: Extracting features of the source image, source 3D mesh points and target 3D mesh points respectively to obtain appearance feature information, source 3D mesh point feature information and target 3D mesh point feature information; The deepest appearance feature information is distorted by the motion field and fused with the occlusion map to obtain the distorted feature information; The distorted feature information and the deepest target 3D mesh point feature information are input into the attention module to obtain the deepest fine feature information; Starting from the deepest layer, the fine feature information obtained in each layer is fused with the attention results of the previous layer and up-sampled to finally generate a digital face composite image. The attention results of the previous layer are the output of the attention module after the feature information of the source 3D mesh points of the previous layer and the appearance feature information of the previous layer are input into the attention module.

4. A method for generating digital face video, characterized in that: The steps include: Extract each frame of image as a driving image according to the video timing sequence; Each driving image executes the steps of the digital face image generation method according to any one of claims 1 to 3 in a time sequence to generate a corresponding digital face composite image; The digital face composite images are merged in the order of generation to generate a speaking video.

5. A digital face image generating device, characterized in that: include: An acquisition module, used for acquiring a source image and a driving image; A cross-identity 3D mesh grid point generation module is used to perform cross-identity 3D mesh grid point generation processing on the source image and the driving image respectively to obtain source 3D mesh grid points and target 3D mesh grid points; A key point detection module is used to perform key point detection after splicing the source 3D mesh points with the source image to obtain sparse key points of the source image; and for splicing the target three-dimensional mesh points with the driving image to perform key point detection to obtain sparse key points of the driving image; Dense motion module, used to generate motion fields and occlusion maps based on sparse key points of source images and sparse key points of driving images; A multi-layer cross-modal attention module is used to process source images, source 3D mesh points, target 3D mesh points, motion fields and occlusion maps based on a multi-layer cross-modal attention mechanism to integrate facial set information into each layer of the generation process to obtain a digital face synthetic image.

6. The digital face image generation device according to claim 5, characterized in that: The cross-identity 3D mesh point generation module includes: A first extraction unit, used for extracting identity features, expression features and posture features of the source image and the driving image; A second extraction unit, used for extracting detail features of the source image; FLAME model, used to generate three-dimensional mesh points of the source image according to the identity features of the source image, the expression features of the source image and the posture features of the source image; The FLAME model is also used to generate the three-dimensional mesh points of the driving image based on the identity features of the source image, the expression features of the driving image, and the posture features of the driving image; A UV displacement map generating unit, used for generating a UV displacement map related to the expression according to the detail features; The fusion unit is used to fuse the UV displacement map with the source image three-dimensional mesh grid points and the drive image three-dimensional mesh grid points respectively to obtain source three-dimensional mesh grid points and target three-dimensional mesh grid points.

7. The digital face image generation device according to claim 6, characterized in that: The multi-layer cross-modal attention module includes the following steps: A feature extraction unit is used to extract features from a source image, source three-dimensional mesh points, and target three-dimensional mesh points, respectively, to obtain appearance feature information, source three-dimensional mesh point feature information, and target three-dimensional mesh point feature information; A warping unit, used to warp the deepest appearance feature information through the motion field and fuse it with the occlusion map to obtain the warped feature information; The attention unit is used to generate the deepest fine feature information based on the distorted feature information and the deepest target 3D mesh network; The attention unit is also used to fuse the fine feature information obtained in each layer with the attention results of the previous layer, starting from the deepest layer, and perform upsampling operations to finally generate a digital face composite image, where the attention results of the previous layer are the output after the feature information of the grid points of the previous layer source three-dimensional mesh and the appearance feature information of the previous layer are input into the attention module.

8. A digital face video generation device, characterized in that: include: A driving image extraction module is used to extract each frame of image as a driving image according to the video time sequence; A calling module, used for inputting each driving image into the digital face image generating device according to any one of claims 5 to 7 in time sequence to generate a corresponding digital face composite image; The merging module is used to merge the digital face synthesis images according to the generation order to generate the speaking person video.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images