Audio-driven face reconstruction methods and related equipment

Abstract

This application relates to the field of computer technology and discloses an audio-driven face reconstruction method and related equipment, comprising: acquiring target audio; obtaining a target facial contour of the target object under the driving force of the target audio based on the target audio and an input facial image of the target object, wherein the target facial contour includes a contour related to lip shape; determining optical flow information based on the target facial contour of the target object, a reference facial image of the target object, and a reference facial contour of the reference facial image, wherein the optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object; and performing reconstruction processing on the input facial image based on the reference facial image of the target object and the optical flow information to obtain a reconstructed facial image. This solution can realize audio-driven image reconstruction, so that the lip shape presented in the obtained reconstructed facial image is consistent with the lip shape of the target object pronouncing the target audio.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an audio-driven face reconstruction method and related equipment. Background Technology

[0002] In audio-driven scenarios, it's necessary to use audio to drive the lip movements of characters in existing videos, ensuring that the resulting driven video displays the lip movements of the characters as they pronounce the audio. However, related technologies suffer from issues with unsatisfactory lip-syncing effects after audio-driven processing. Summary of the Invention

[0003] In view of the above problems, this application proposes an audio-driven face reconstruction method and related device to improve the above problems.

[0004] According to one aspect of the embodiments of this application, an audio-driven face reconstruction method is provided, comprising: acquiring target audio; obtaining a target face contour of the target object under the action of the target audio, the target face contour including a lip-related contour, based on the target audio and an input face image of the target object; determining optical flow information based on the target face contour of the target object, a reference face image of the target object, and a reference face contour of the reference face image, the optical flow information being used to characterize the offset information between the reconstructed face image and the reference face image of the target object; and performing reconstruction processing on the input face image based on the reference face image of the target object and the optical flow information to obtain the reconstructed face image.

[0005] According to one aspect of the embodiments of this application, an audio-driven face reconstruction apparatus is provided, comprising: an acquisition module for acquiring target audio; a first driving module for obtaining a target facial contour of the target object under the driving action of the target audio, the target facial contour including a lip-related contour, based on the target audio and an input facial image of the target object; an optical flow information determination module for determining optical flow information based on the target facial contour of the target object, a reference facial image of the target object, and a reference facial contour of the reference facial image, the optical flow information being used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object; and a reconstruction module for performing reconstruction processing on the input facial image based on the reference facial image of the target object and the optical flow information to obtain the reconstructed facial image.

[0006] In some embodiments, the optical flow information determination module includes: a selection unit, configured to select i reference video frames as reference face images from the speech video frame sequence of the target object; i is an integer greater than 1; a first acquisition unit, configured to acquire the reference face contour of the i reference video frames; and an optical flow information determination unit, configured to determine optical flow information by an optical flow network based on the target face contour of the target object, the i reference video frames, and the reference face contour of the i reference video frames.

[0007] In some embodiments, selecting i reference video frames as reference facial images from the speech video frame sequence of the target object includes at least one of the following two selection methods: selecting at least one speech video frame with the most tooth pixels from the speech video frame sequence of the target object; or, when the input facial image comes from the speech video frame sequence, selecting at least one speech video frame from the video frame window containing the input facial image in the speech video frame sequence.

[0008] In some embodiments, the optical flow information determination unit is configured to: when the input face image comes from the speech video frame sequence, acquire the facial contour of at least one adjacent face image adjacent to the input face image; input the target face contour, the facial contour of the at least one adjacent face image, i frames of the reference video frame, and the reference face contour of i frames of the reference video frame into the optical flow network to obtain optical flow information.

[0009] In some embodiments, the reconstruction module includes: an occlusion unit for occluding a lip-related region of the input face image to obtain a lip-occluded image; and a face reconstruction unit for performing face reconstruction by a face reconstruction network based on a reference face image of the target object, the optical flow information, and the lip-occluded image to obtain the reconstructed face image.

[0010] In some embodiments, the face reconstruction network includes a first image encoder, a second image encoder, and an image decoder; the face reconstruction unit includes: a first encoding unit, used by the first image encoder to encode a reference face image of the target object to obtain reference visual features; an optical flow mapping unit, used to perform optical flow mapping based on the optical flow information and the reference visual features to obtain optical flow fusion features; a fusion encoding unit, used by the second image encoder to perform fusion encoding based on the image features of the lip-occlusion image and the optical flow fusion features to obtain target visual features; and a decoding unit, used by the image decoder to decode the target visual features to obtain the reconstructed face image.

[0011] In some embodiments, the optical flow information includes the optical flow map of the i-th reference video frame and the weight of the optical flow map; the reference visual feature includes the k-th layer sub-reference visual feature output by the first coding layer of the k-th layer in the first image encoder for each of the reference video frames; the optical flow fusion feature includes the k-th layer sub-optical flow fusion feature; 1 ≤ k ≤ M; M is the number of the first coding layers in the first image encoder; an optical flow mapping unit, configured to: perform optical flow mapping on the k-th layer sub-reference visual feature of the w-th reference video frame and the optical flow map of the w-th reference video frame to obtain the k-th layer optical flow mapping feature corresponding to the w-th reference video frame; 1 ≤ w ≤ i; w is a positive integer; according to the weights of the optical flow maps of the respective reference video frames, perform weighted processing on the k-th layer optical flow mapping features respectively corresponding to the i reference video frames to obtain the k-th layer sub-optical flow fusion feature.

[0012] In some embodiments, the second image encoder includes M layers of second coding layers; a fusion coding unit, configured to: splice the k-th layer sub-optical flow fusion feature and the k-th layer coding feature to obtain the k-th layer splicing feature; the k-th layer coding feature is output by the k-th layer second coding layer in the second image encoder; 1 ≤ k ≤ M; when k = 1, the first layer coding feature is the image feature of the mouth occlusion image; if k < M, the k + 1-th layer second coding layer in the second image encoder performs fusion coding on the k-th layer splicing feature to obtain the k + 1-th layer coding feature; increment k by 1 and return to execute the step of splicing the k-th layer sub-optical flow fusion feature and the k-th layer coding feature to obtain the k-th layer splicing feature; if k = M, use the M-th layer splicing feature as the target visual feature.

[0013] In some embodiments, the audio-driven face reconstruction device further includes: a first acquisition module for acquiring multiple training samples; wherein each training sample includes a first sample speech image and a second sample speech image of the same reference object; a second acquisition module for acquiring sample facial contours; the sample facial contours are obtained by performing lip-sync keypoint driving on the sample facial contours corresponding to the first sample speech image based on the sample audio corresponding to the second sample speech image, or are the facial contours corresponding to the second sample speech image; a first optical flow estimation module for performing optical flow estimation by the optical flow network based on the sample facial contours, the sample reference facial image of the reference object, and the facial contours of the sample reference facial image to obtain sample optical flow information; a first face reconstruction module for performing face reconstruction on the first sample speech image by the face reconstruction network based on the sample reference facial image and the sample optical flow information to obtain a sample reconstructed face image; a reconstruction loss calculation module for calculating a reconstruction loss based on the sample reconstructed face image and the second sample speech image; and an adjustment module for adjusting at least the parameters of the optical flow network and the face reconstruction network based on the reconstruction loss until the training termination condition is met.

[0014] In some embodiments, the reconstruction loss calculation module is configured to: determine a first loss based on the pixel difference between the sample reconstructed face image and the second sample speech image; determine a second loss based on the depth features of the sample reconstructed face image and the depth features of the second sample speech image; determine a third loss based on the discrimination result of the discriminator network on the second sample speech image; and weight at least one of the second loss and the third loss with the first loss to obtain the reconstruction loss.

[0015] In some embodiments, the first driving module includes: a facial pose key point determination unit, configured to acquire facial pose key points of an input facial image of the target object, wherein the facial pose key points are facial key points unrelated to the lip shape; a lip shape key point determination unit, configured to acquire lip shape key points of the target object related to the lip shape; and a target facial contour determination unit, configured to obtain the target facial contour of the target object based on the target audio, the lip shape key points, and the facial pose key points.

[0016] In some embodiments, the lip shape key point determination unit is configured to: select multiple speech images of the target object with different lip shapes; and extract lip shape key points related to the lip shapes based on the multiple speech images with different lip shapes.

[0017] In some embodiments, the input facial image is a video of the target object speaking in a first language; the target audio is audio in a second language; the first language is different from the second language.

[0018] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0019] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer-readable instructions thereon, which, when executed by a processor, implement the method described above.

[0020] According to one aspect of the embodiments of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the method described above.

[0021] In this application, the target facial image of the target object is first obtained by using the target audio and the target facial image of the target object. Since the target facial contour includes a lip-syncing-related contour, the lip shape presented in this contour is equivalent to the lip shape presented when the target object pronounces the target audio. Then, using the target facial contour, a reference facial image of the target object, and a reference facial contour of the reference facial image, the offset information (i.e., optical flow information) between the reconstructed facial image to be generated and the reference facial image of the target object is estimated in reverse. Using the reference facial image and the optical flow information, the input facial image is reconstructed to obtain the reconstructed facial image. In this application, using the target facial contour as a constraint in the reconstruction process ensures that the lip shape presented in the obtained reconstructed facial image is consistent with the lip shape presented in the target facial contour, i.e., the lip shape presented when the target object pronounces the target audio. Therefore, the problem of poor lip shape after audio driving in related technologies can be effectively solved, ensuring the quality of the reconstructed facial image. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0023] Figure 1A A schematic diagram illustrates an application scenario where the technical solutions of the embodiments of this application can be applied.

[0024] Figure 1B This is a schematic diagram illustrating a speech video frame sequence driven by a target audio according to an embodiment of this application.

[0025] Figure 2 This is a flowchart illustrating an audio-driven face reconstruction method according to an embodiment of this application.

[0026] Figure 3A This is a flowchart illustrating an audio-driven face reconstruction method according to another embodiment of this application.

[0027] Figure 3B This is a flowchart illustrating step 230 according to an embodiment of this application.

[0028] Figure 4 This is a schematic diagram illustrating lip-shape keypoint driving according to an embodiment of this application.

[0029] Figure 5 This is a flowchart illustrating step 240 according to another embodiment of this application.

[0030] Figure 6 This is a flowchart illustrating optical flow estimation according to an embodiment of this application.

[0031] Figure 7 This is a flowchart illustrating face reconstruction of the t-th frame of a speech video frame according to an embodiment of this application.

[0032] Figure 8 This is a flowchart illustrating step 740 according to an embodiment of this application.

[0033] Figure 9 This is a schematic diagram illustrating facial reconstruction using a facial reconstruction network according to an embodiment of this application.

[0034] Figure 10 This is a flowchart illustrating the training of an optical flow network and a face reconstruction network according to an embodiment of this application.

[0035] Figure 11 This is a block diagram of an audio-driven face reconstruction apparatus according to an embodiment of this application.

[0036] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0037] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0039] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] In this document, "multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. In the following description, references to "some embodiments or some embodiment methods" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0041] Figure 1A The diagram illustrates an application scenario where the technical solutions of the embodiments of this application can be applied. For example... Figure 1A As shown, this application scenario includes a terminal 110 and a server 120, which are connected via wired or wireless network communication.

[0042] Terminal 110 can upload the target audio and the input facial image of the target object to server 120. Server 120 can, according to the scheme of this application, drive the input facial image of the target object through the target audio to obtain the reconstructed facial image of the target object when it speaks according to the target audio.

[0043] Specifically, server 120 can first obtain the target facial contour of the target object under the driving force of the target audio based on the target audio and the input facial image of the target object. The target facial contour includes the contour related to the mouth shape. Then, based on the target facial contour of the target object, the reference facial image of the target object, and the reference facial contour of the reference facial image, optical flow information is determined. The optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object. Subsequently, the input facial image is reconstructed by combining the reference facial image of the target object and the optical flow information to obtain the reconstructed facial image.

[0044] In some embodiments, the input facial image of the target object can be a speaking video of the target object, and the speaking video frame sequence of the target object can be determined based on the speaking video. In this case, the server 120 can proceed according to... Figure 1B The process shown involves driving the speech video frames in the speech video frame sequence through the target audio, such as... Figure 1B As shown, after selecting multiple key video frames reflecting the lip-sync style of the target object from the narrative video frame sequence, lip-sync key points can be driven on the facial key point maps corresponding to each speech video frame in the speech video frame sequence based on the audio features of the target audio and the facial key point maps (also known as facial contours) corresponding to the multiple key video frames, resulting in a driven facial key point map sequence. Subsequently, optical flow estimation is performed based on each driven facial key point map in the driven facial key point map sequence, the reference video frame selected in the speech video frame sequence, and the facial key point map of the reference video frame, resulting in an optical flow information sequence. Finally, facial reconstruction is performed on the speech video frame sequence based on the reference video frame and the optical flow information sequence to obtain the target speech video frame sequence simulating the target object speaking according to the target audio.

[0045] Terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart TV, in-vehicle terminal, virtual interactive device, smart wearable device, etc. Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0046] Furthermore, the method of this application is not limited to being executed by server 120, but can also be executed by terminal 110, or implemented by interaction between server 120 and terminal 110.

[0047] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0048] Figure 2 This is a flowchart illustrating an audio-driven face reconstruction method according to an embodiment of this application. The method can be executed by an electronic device with processing capabilities, such as a server or terminal, and is not specifically limited thereto. (Refer to...) Figure 2 As shown, the method includes at least steps 201 to 204, which are described in detail below:

[0049] Step 201: Obtain the target audio.

[0050] Step 202: Based on the target audio and the input face image of the target object, obtain the target face contour of the target object driven by the target audio. The target face contour includes the contour related to the mouth shape.

[0051] The target object can be a real person, a virtual character (such as a digital human), or a virtual role; no specific limitations are imposed here. The input facial image of the target object represents the facial region of the target object.

[0052] Target audio refers to the audio used to drive the input facial image of the target object. This target audio can be the audio of the target object itself, the audio of other objects, or audio obtained by converting text content into speech. The duration of the target audio can be the duration of a single vocal unit.

[0053] In some embodiments, the input facial image is a video of the target object speaking in a first language; the target audio is audio in a second language; the first language and the second language are different.

[0054] In some embodiments, audio features of the target audio can be extracted, and facial key points can be extracted from the input face image to obtain the facial contour of the input face image, which includes multiple facial key points extracted from the input face image. Then, using the audio features of the target audio and the facial contour of the input face image, lip-sync key point-driven lip-sync key point extraction is performed to obtain the target face contour of the target object under the influence of the target audio; that is, the lip shape of the target object presented in the target face contour is the lip shape of the target object when it pronounces the target audio.

[0055] Audio feature extraction models can be used to extract features from target audio, obtaining the audio features of the target audio. Audio feature extraction models can be models constructed using one or more neural networks. In some embodiments, the audio feature extraction model can be a whisper model or a wav2vec model. In some embodiments, MFCC (Mel Frequency Cepstral Coefficients) features of the target audio can also be extracted as audio features.

[0056] Facial key points are the key locations or features that represent facial characteristics. Examples of facial key points include features of the eyebrows, eyes, nose, mouth, and facial contours.

[0057] In some embodiments, facial keypoints can be extracted from the input face image to obtain a facial keypoint map corresponding to the input face image, which serves as the facial contour corresponding to the input face image. In other embodiments, facial keypoints can be extracted from the input face image to obtain multiple facial keypoints. Then, related facial keypoints among the multiple facial keypoints are connected by curves to obtain a facial keypoint map that reflects the facial contour, which serves as the facial contour corresponding to the input face image. Related facial keypoints can be multiple facial keypoints located in the same facial feature region. For example, multiple facial keypoints located in the nose region are related multiple facial keypoints, multiple facial keypoints located in the left eye region are related multiple facial keypoints, and multiple facial keypoints located in the lip region are related multiple facial keypoints.

[0058] In some embodiments, facial landmark extraction models can be used to extract facial landmarks from an input facial image. Different facial landmark extraction models may extract different facial landmarks. For example, a facial landmark extraction model that extracts 256 facial landmarks or a facial landmark extraction model that extracts 68 facial landmarks can be used to extract facial landmarks.

[0059] In some embodiments, the facial keypoint extraction model can be a model for extracting the two-dimensional coordinates of facial keypoints or a model for extracting the three-dimensional coordinates of facial keypoints. If the facial keypoint extraction model is a model for extracting the three-dimensional coordinates of facial keypoints, after extracting the three-dimensional coordinates of each facial keypoint through the facial keypoint extraction model, the three-dimensional coordinates of the facial keypoints can be transformed into a two-dimensional plane to obtain the facial keypoint map of the input facial image. Alternatively, after transforming the three-dimensional coordinates of the facial keypoints into a two-dimensional plane, multiple related facial keypoints can be connected to obtain the facial keypoint map of the input facial image.

[0060] In some embodiments, step 202 includes the following steps A1-A3:

[0061] Step A1: Obtain the facial pose key points of the input face image of the target object. The facial pose key points are facial key points that are independent of the lip shape.

[0062] Facial key points can be extracted from the input face image to obtain the corresponding facial key point map. Then, the mouth shape key points in the corresponding facial key point map of the input face image are occluded to obtain the occluded key point map. The facial key points presented in the occluded key point map (i.e. the unoccluded facial key points) are used as the facial pose key points of the input face image.

[0063] Step A2: Obtain the lip shape key points related to the lip shape of the target object.

[0064] Lip key points refer to facial key points related to the shape of the mouth. Examples of lip key points include multiple lip key points located on the lips and cheek key points near the lips.

[0065] In some embodiments, multiple speech images reflecting the lip-syncing style of the target object are determined, and then lip-syncing key points are extracted from each of the multiple speech images to obtain the lip-syncing key points in each speech image.

[0066] The lip-syncing style of a target audience member can be understood as their habitual lip movements during speech. This can be conveyed through changes in lip shape during speech. Therefore, multiple images showing different lip sizes can be used to reflect the target audience member's lip-syncing style. It's understandable that a larger number of images accurately expresses the target audience member's lip-syncing style, but this also increases the subsequent processing workload. Therefore, in specific embodiments, a trade-off between accuracy and processing volume can be struck, determining the number of selected images based on needs.

[0067] In some embodiments, multiple speaking images with different lip sizes can be selected from a sequence of speaking video frames of the target subject (e.g., the speaking video frame sequence from which the input face image is derived, or other speaking video frame sequences). Alternatively, multiple speaking images can be randomly selected from the speaking video frame sequence. Since it is a random selection, the probability of selecting speaking images with exactly the same or very similar lip sizes is low. Therefore, randomly selected multiple speaking images can reflect the speaking lip style of the target subject.

[0068] For example, if we call the pixels representing the lips in a speech image "lip pixels," then the closed pixel region formed by the lip pixels as boundaries in a video frame is the pixel region representing the mouth shape, which is called the mouth shape pixel region. Based on this, we can count the number of pixels in the mouth shape pixel region in each video frame of the speech video frame sequence. The larger the number of pixels in the mouth shape pixel region, the larger the mouth shape. Therefore, based on the number of pixels in the mouth shape pixel region of each video frame, we can select multiple video frames with different numbers of pixels in the mouth shape pixel region from the speech video frame sequence as multiple speech images.

[0069] In some embodiments, step A3 may include: selecting multiple speech images of the target object with different lip shapes; and extracting lip shape key points related to the lip shape based on the multiple speech images with different lip shapes. That is, extracting lip shape key points related to the lip shape from each speech image to obtain a lip shape key point map corresponding to each speech image. This lip shape key point map presents the pixel positions of the lip shape key points in each speech image, correspondingly reflecting the lip shape of the target object presented in each speech image.

[0070] In some embodiments, facial key points can be extracted from each speech image in a manner similar to that described above, resulting in a facial key point map corresponding to each speech image. This facial key point map presents the lip shape key points of the target object in the speech image that are related to the lip shape.

[0071] Step A3: Based on the target audio, lip shape key points, and facial pose key points, obtain the target facial contour of the target object.

[0072] In some embodiments, after determining the lip shape key points of the target object in the facial key point map corresponding to the input face image, the lip shape key points in the facial key point map corresponding to the input face image can be occluded to obtain an occluded key point map. In this way, the facial key points presented in the occluded key point map are the facial pose key points in the input face image.

[0073] The target audio, occlusion keypoint map, and facial keypoint maps (or lip keypoint maps) corresponding to multiple speech images are input into the keypoint reconstruction network. The keypoint reconstruction network reconstructs the lip keypoints of the occluded area in the occlusion keypoint map to obtain the target facial contour of the target object.

[0074] In the target facial contour of the target object, apart from the reconstructed lip-shape keypoints, the positions of other facial keypoints can be the same as the positions of the corresponding facial keypoints in the occluded keypoint map. The lip shape presented by the target facial contour is the lip shape for pronunciation according to the target audio.

[0075] A keypoint reconstruction network is a neural network model used for audio-driven generation of lip-sync keypoints. In some embodiments, the keypoint reconstruction network may include a first encoder network and a first decoder network; for example, the keypoint reconstruction network may be a Transformer network. The first encoder network fuses and encodes the audio features of the target audio, facial keypoint maps (or lip-sync keypoint maps) corresponding to multiple speech images, and occlusion keypoint maps to obtain a fused feature map. Then, the fused feature map is input into the first decoder network, which decodes the fused feature map and outputs the target facial contour of the target object.

[0076] In the above embodiments, multiple speech images with different lip shapes are introduced to correspond to facial key point maps, which assist in driving the lip shape key point of the facial key point map corresponding to the input face image. This is to inject the lip shape style of the target object, because multiple speech images with different lip shapes can reflect the lip shape style of the target object, and the facial key point maps corresponding to multiple speech images with different lip shapes can also reflect the lip shape style of the target object. By injecting the lip shape style of the target object, it is ensured that the obtained target facial contour of the target object not only corresponds to the lip shape of the speech content of the target audio, but also conforms to the lip shape style of the target object.

[0077] Step 203: Determine optical flow information based on the target facial contour of the target object, the reference facial image of the target object, and the reference facial contour of the reference facial image. The optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object.

[0078] It is understandable that the target face contour can be approximated as the face key point map of the face image to be reconstructed. The optical flow between the reconstructed face image and the reference face image of the target object can reflect the optical flow of the face key point map of the reconstructed face image relative to the reference face image of the target object. Therefore, in this application, the optical flow between the reference face contour of the reference face image and the target face contour, as well as the pixels of the reference face image itself, can be used to estimate the optical flow information between the reconstructed face image and the reference face image of the target object in reverse.

[0079] In some embodiments, step 203 includes the following steps B1 to B3:

[0080] Step B1: Select i reference video frames from the target object's speech video frame sequence as reference facial images; i is an integer greater than 1.

[0081] The speech video frame sequence of the target object can be a speech video frame sequence from which the input facial image is derived, or a speech video frame sequence that presents the speech process of the target object. Each video frame in the speech video frame sequence of the target object presents the facial region of the target object.

[0082] Step B2: Obtain the reference facial contour of the i-th reference video frame. Facial keypoint extraction can be performed in a similar manner to that described above to obtain the reference facial contour for each reference video frame.

[0083] In some embodiments, step B2 may be performed by at least one of the following two selection methods: selecting at least one speech video frame with the most tooth pixels in the speech video frame sequence of the target object; or selecting at least one speech video frame in the video frame window of the speech video frame sequence that contains the input face image when the input face image is from the speech video frame sequence.

[0084] A video frame window containing the input face image refers to a window with a certain width centered on the input face image. The width of the window can be represented by the number of video frames it covers. In some embodiments, at least one speech video frame can be randomly selected from the video frame window containing the input face image, or at least one speech video frame can be selected at equal intervals within the video frame window containing the input face image, serving as the i-th reference video frame mentioned above.

[0085] In other embodiments, at least one speech video frame with the most tooth pixels may be selected from the speech video frame sequence from which the face image is derived; and at least one speech video frame may be selected from the video frame window containing the input face image in the speech video frame sequence as i-frame reference video frame (i.e. reference face image).

[0086] Step B3: The optical flow network determines the optical flow information based on the target facial contour of the target object, the i-frame reference video frame, and the reference facial contour of the i-frame reference video frame.

[0087] Optical flow network refers to a neural network model used for optical flow estimation. In some embodiments, the optical flow network may be U-net (a U-shaped network based on convolutional neural networks), FlowNet (Learning Optical Flow with Convolutional Networks, an optical flow estimation network based on convolutional neural networks), a neural network model based on a transformer network, or other neural network models with an encoder-decoder structure, without being specifically limited here.

[0088] The encoder network in the optical flow network is referred to as the second encoding network, and the decoder network in the optical flow network is referred to as the second decoding network. In some embodiments, the target facial contour of the target object, the i-frame reference video frame, and the reference facial contour of the i-frame reference video frame can be input into the second encoding network, which performs fusion encoding to obtain the first fused encoding feature; then, the second decoding network decodes the first fused encoding feature to obtain the optical flow information.

[0089] Optical flow information may include one or more optical flow maps, the number of which is the same as the number of selected reference video frames. An optical flow map reflects the offset information between the reconstructed face image to be generated and a reference face image (reference video frame). In some embodiments, the optical flow information may also include weights corresponding to each optical flow map. The weight of an optical flow map can be understood as its confidence level; that is, the higher the weight, the higher the confidence and accuracy of the optical flow map.

[0090] In some embodiments, step B3 includes: when the input face image comes from a speech video frame sequence, obtaining the facial contour of at least one adjacent face image adjacent to the input face image; inputting the target face contour, the facial contour of at least one adjacent face image, the i-frame reference video frame, and the reference face contour of the i-frame reference video frame into an optical flow network to obtain optical flow information.

[0091] The facial contour of at least one adjacent facial image adjacent to the input facial image is also used to determine optical flow information to ensure the temporal coherence of multiple optical flow maps.

[0092] Step 204: Based on the reference face image and optical flow information of the target object, the input face image is reconstructed to obtain a reconstructed face image.

[0093] In some embodiments, the process may involve reconstructing facial regions from the input face image. The reconstructed face image includes facial pixel regions representing the face of the target object. The lip movements of the target object presented in the reconstructed face image are identical to those of the target object when pronouncing the target audio, and conform to the target object's speaking lip movement style. Furthermore, since the facial regions from the input face image are reconstructed, the background region of the reconstructed face image can be the same as the background region in the input face image.

[0094] In some embodiments, step 204 includes the following steps C1-C2:

[0095] Step C1: Occlude the lip-related region of the input face image to obtain a lip-occluded image.

[0096] Lip-related occlusion refers to obscuring the pixel region in the input face image that represents the mouth shape. Therefore, the occluded image contains the same pixel regions as the input face image, except for the occluded area. It is understood that during lip-occlusion, the occluded pixel region includes at least the pixel region representing the mouth shape, and may also include the surrounding pixel regions.

[0097] In some embodiments, the lower half of the target object's cheek in the input face image can be occluded to obtain a lip-masking image, or the occluded area can be a pixel region that is based on the lower half of the target object's cheek and expanded outwards.

[0098] It's understandable that the occluded pixel areas in the lip-masking image are the pixel areas that need to be reconstructed in the subsequent face reconstruction process. From this perspective, the lip-masking image is used to indicate the pixel areas in the input face image that need to be reconstructed, while other unoccluded pixel areas do not need to be reconstructed in the subsequent process.

[0099] In step C2, the face reconstruction network performs face reconstruction based on the reference face image of the target object, optical flow information, and lip occlusion image to obtain the reconstructed face image.

[0100] In some embodiments, face reconstruction can be performed using a face reconstruction network, which is a neural network model capable of generating images. The face reconstruction network can perform reconstruction as follows: extracting image features from a reference face image of the target object, optical flow information features from optical flow information, and image features from a lip-occlusion image, fusing the three features to obtain a third fused feature, and decoding the third fused feature to obtain a reconstructed face image.

[0101] In some embodiments, the face reconstruction network includes a first image encoder, a second image encoder, and an image decoder; step C2 includes the following ①-④:

[0102] ① The reference facial image of the target object is encoded by the first image encoder to obtain the reference visual features.

[0103] ② Optical flow mapping is performed based on optical flow information and reference visual features to obtain optical flow fusion features.

[0104] Optical flow mapping refers to the transformation processing of an image based on optical flow. In some embodiments, optical flow mapping can be performed on the sub-reference visual features of each reference video frame relative to the sub-reference optical flow information of the reconstructed face image, to obtain the optical flow mapping features corresponding to each reference video frame. Then, the optical flow mapping features corresponding to all reference video frames are weighted to obtain the optical flow fusion features.

[0105] In some embodiments, the optical flow information includes the optical flow map and weights of the i-th reference video frame; the reference visual features include the k-th layer sub-reference visual features output by the k-th layer first coding layer of the first image encoder for each reference video frame; the optical flow fusion features include the k-th layer sub-optical flow fusion features; 1≤k≤M; M is the number of first coding layers in the first image encoder; the step of performing optical flow mapping based on the optical flow information and the reference visual features to obtain the optical flow fusion features includes: performing optical flow mapping between the k-th layer sub-reference visual features of the w-th reference video frame and the optical flow map of the w-th reference video frame to obtain the k-th layer optical flow mapping features corresponding to the w-th reference video frame; 1≤w≤i; w is a positive integer; and weighting the k-th layer optical flow mapping features corresponding to the i-th reference video frames according to the weights of the optical flow maps of each reference video frame to obtain the k-th layer sub-optical flow fusion features.

[0106] For example, for an i-th reference video frame, if the k-th layer of the first encoding layer in the first image encoder is the k-th layer sub-reference visual feature output from the w-th reference frame, then... In the i-th reference video frame, the optical flow graph of the w-th reference video frame relative to the reconstructed face image is f. w The weight of the w-th reference video frame relative to the optical flow graph of the reconstructed face image is α. w The sub-optical flow fusion feature Z of the k-th layer can be determined according to the following formula 1. k :

[0107]

[0108] in, This represents the optical flow mapping feature of the k-th layer corresponding to the w-th reference video frame.

[0109] ③ The second image encoder performs fusion encoding based on the image features of the lip-occlusion image and the optical flow fusion features to obtain the target visual features.

[0110] In some embodiments, the image features of the lip-occluded image and the optical flow fusion features are stitched together and then input into a second image encoder, which performs fusion encoding to obtain the target visual features.

[0111] In some other embodiments, the second image encoder includes M second encoding layers; in this embodiment, the step of the second image encoder performing fusion encoding based on the image features and the optical flow fusion features of the mouth occlusion image to obtain the target visual features includes: splicing the k-th sub-optical flow fusion feature and the k-th encoding feature to obtain the k-th splicing feature; the k-th encoding feature is output by the k-th second encoding layer in the second image encoder; 1 ≤ k ≤ M; when k = 1, the first encoding feature is the image feature of the mouth occlusion image; if k < M, the (k + 1)-th second encoding layer in the second image encoder performs fusion encoding on the k-th splicing feature to obtain the (k + 1)-th encoding feature; add 1 to k in accumulation and return to execute the step of splicing the k-th sub-optical flow fusion feature and the k-th encoding feature to obtain the k-th splicing feature; if k = M, take the M-th splicing feature as the target visual feature.

[0112] That is to say, the k-th sub-optical flow fusion feature is first spliced with the feature output by the k-th second encoding layer in the second image encoder, and then input into the (k + 1)-th second encoding layer in the second image encoder for fusion encoding. In this way, the features output by different first encoding layers in the first image encoder can be fully utilized, rather than only using the features output by the last first encoding layer in the first image encoder, so as to ensure the accuracy of the target visual features, and further ensure the quality of the reconstructed face image obtained by subsequent decoding.

[0113] ④ The image decoder decodes the target visual features to obtain the reconstructed face image.

[0114] First, the target audio and the input face image of the target object are used to obtain the target face contour of the target object under the driving of the target audio. Since the target face contour includes the contour related to the mouth shape, the mouth shape presented in the target face contour is equivalent to the mouth shape when the target object pronounces according to the target audio. Then, the offset information (i.e., optical flow information) between the to-be-generated reconstructed face image and the reference face image of the target object is inversely estimated by using the target face contour of the target object, the reference face image of the target object, and the reference face contour of the reference face image, and the reference face image of the target object and the optical flow information are used to perform reconstruction processing on the input face image to obtain the reconstructed face image. In this application, using the target face contour as the constraint in the reconstruction process can ensure that the mouth shape presented in the obtained reconstructed face image is consistent with the mouth shape presented in the target face contour, that is, the mouth shape when the target object pronounces according to the target audio. Thus, the problem of poor mouth shape after audio driving in the related art can be effectively solved, and the effect of the reconstructed face image can be ensured.

[0115] The i-frame reference video frame selected for the input face image can provide rich oral cavity detail information, such as tooth details, as well as background information. This ensures that the reconstructed face image can accurately present the tooth details of the target object when pronouncing the target audio. Furthermore, the background pixel region in the i-frame reference video frame can provide accurate background information for the reconstruction process, especially when part of the background is also occluded in the lip-sync image, thereby ensuring the quality of the obtained reconstructed video frame.

[0116] The following explanation will focus on the approach of obtaining a simulated target speech video frame sequence based on the target object's speech video frame sequence driven by the first target video, thus simulating the target object speaking according to the first target audio. Figure 3A As shown, the method may include the following steps 210-250:

[0117] Step 210: Obtain the first target audio and the video frame sequence of the target object's speech.

[0118] The target subject's speech video frame sequence shows the target subject's facial region in each frame. The target subject's speech video frame sequence is the video frame sequence to be driven. The target subject's speech video frame sequence can be a sequence of multiple video frames extracted sequentially and at intervals from the target subject's speech video. Alternatively, the target subject's speech video can be used as the speech video frame sequence. The target subject's speech video can be captured facing the target subject during the first target video's speech.

[0119] In some embodiments, the speech video frame sequence is extracted from a speech video of a target object, in which the target object speaks in a first language; the first target audio is the audio of the speech content corresponding to the speech video in a second language; the first language and the second language are different. The speech content corresponding to the speech video refers to the speech content of the target object in the speech video. The first language can be, for example, Mandarin Chinese and the second language can be English; or, for example, the first language can be Mandarin Chinese and the second language can be German, without specific limitations. In this case, it is equivalent to using the first target audio and the speech video frame sequence of the target object to generate a first target video frame sequence that simulates the target object speaking the speech content in the first target audio in the second language, thereby automatically translating the speech video frame sequence of the target object.

[0120] Understandably, assuming the number of speech video frames in the speech video frame sequence is N, the first target audio can be divided into N audio units. The x-th audio unit among the N audio units is used to drive the x-th speech video frame in the speech video frame sequence. Therefore, the N audio units can be regarded as the target audio mentioned above, so as to drive the corresponding speech video frame. Each speech video frame in the speech video frame sequence can be regarded as the input face image to be reconstructed mentioned above. Step 220: Select multiple key video frames in the speech video frame sequence. These multiple key video frames are used to reflect the speaking lip-sync style of the target object.

[0121] Key video frames refer to video frames selected from a sequence of spoken video frames that reflect the lip-syncing style of a target subject. The lip-syncing style of a target subject can also be understood as their habitual mouth movements, which can be reflected by changes in their mouth shapes during speech. Therefore, multiple video frames showing different mouth shapes can be used to reflect the lip-syncing style of a target subject. It is understood that the more key video frames there are, the more accurately the lip-syncing style of the target subject can be expressed, but this also increases the subsequent processing workload. Therefore, in specific embodiments, a trade-off can be struck between accuracy and processing volume, and the number of key video frames selected can be determined according to needs.

[0122] In some embodiments, multiple speech video frames with different lip sizes can be selected from the speech video frame sequence as multiple key video frames. For example, multiple video frames can be randomly selected from the speech video frame sequence as key video frames. Since the selection is random, the probability of selecting video frames with completely identical or very similar lip sizes is small. Therefore, the randomly selected multiple video frames can reflect the speaking lip style of the target subject.

[0123] For example, if we call the pixels representing lips in a video frame "lip pixels," then the closed pixel region formed by the lip pixels as boundaries within the video frame is the pixel region representing the mouth shape, and is called the mouth shape pixel region. Based on this, we can count the number of pixels in the mouth shape pixel region in each video frame of the speech video frame sequence. The larger the number of pixels in the mouth shape pixel region, the larger the mouth shape. Therefore, based on the number of pixels in the mouth shape pixel region of each video frame, we can select multiple video frames with different numbers of pixels in the mouth shape pixel region from the speech video frame sequence as key video frames.

[0124] Step 230: Based on the audio features of the first target audio and the facial key point maps of multiple key video frames, perform lip-shape key point driving on the facial key point maps of each speech video frame in the speech video frame sequence to obtain a driven facial key point map sequence.

[0125] The first target audio audio driving facial keypoint map sequence includes multiple driving facial keypoint maps arranged in sequence. The t-th driving facial keypoint map in the sequence is obtained by driving the lip-shape keypoints of the facial keypoint map corresponding to the t-th video frame in the speech video frame sequence. The number of video frames in the speech video frame sequence is the same as the number of driving facial keypoint maps in the driving facial keypoint map sequence.

[0126] Lip-sync keypoint driving is applied to the facial keypoint maps corresponding to each speech video frame in the speech video frame sequence to simulate the facial keypoint map of the target object when it speaks according to the first target audio. In other words, the driving facial keypoint map sequence is used to present the facial keypoint map of the target object during the process of simulating the target object speaking according to the first target audio.

[0127] The t-th driving facial landmark map corresponding to the t-th speech video frame in the speech video frame sequence presents the facial landmarks of the target object when it pronounces according to the t-th audio segment in the first target audio. The first target audio can be sequentially divided into N audio segments of equal duration according to the number of video frames N in the speech video frame sequence. The t-th audio segment in the first target audio is used to drive the t-th speech video frame in the speech video frame sequence. Furthermore, the other audio segments in the first target audio besides the t-th audio segment can serve as auxiliary information to help determine the lip-sync landmark driving result of the t-th speech video frame in the speech video frame sequence.

[0128] In step 230, facial keypoint maps corresponding to multiple key video frames are introduced to assist in driving the lip-shape keypoints of the facial keypoint maps corresponding to each speech video frame in the speech video frame sequence. This is to inject the target object's speech lip-shape style, because multiple key video frames can reflect the target object's speech lip-shape style, and the facial keypoint maps corresponding to multiple key video frames can also reflect the target object's speech lip-shape style. By injecting the target object's speech lip-shape style, it is ensured that the resulting driven facial keypoint map sequence not only corresponds to the lip-shape of the speech content at the corresponding moment in the first target audio, but also conforms to the target object's speech lip-shape style. In other words, on the one hand, the lip-shape presented by the t-th driven facial keypoint map is the lip-shape when the t-th audio segment is pronounced; on the other hand, the introduction of facial keypoint maps corresponding to multiple key video frames ensures that the lip-shape presented by the generated t-th driven facial keypoint map conforms to the target object's speech lip-shape style.

[0129] In some embodiments, such as Figure 3BAs shown, step 230 includes: Step 310, obtaining the t-th facial keypoint image of the t-th frame in the speech video frame sequence; t is a positive integer ranging from 1 to N; N is the number of speech video frames in the speech video frame sequence. Step 320, performing lip shape keypoint removal processing on the t-th facial keypoint image to obtain the t-th candidate keypoint image.

[0130] The t-th candidate keyframe image refers to the facial keyframe image after removing lip-related keyframes from the t-th facial keyframe image. After removing lip-related keyframes from the t-th facial keyframe image, the facial keyframes unrelated to lip-related features are retained, such as keyframes located in the bridge of the nose area, the eye area, and the forehead area. Lip-related keyframes in the t-th facial keyframe image can also be understood as keyframes representing facial poses other than lip-related features.

[0131] Retain the facial key points in the t-th facial key point image that are not related to the lip shape. This ensures that after the lip shape key point is driven in the subsequent facial key point image, the position of the facial key points that are not related to the lip shape is basically the same as the position in the t-th facial key point image. That is, the facial poses other than the lip shape are basically the same as the facial poses other than the lip shape in the t-th facial key point image.

[0132] It is understandable that the missing facial key points in the t-th candidate key point image compared to the t-th facial key point image are the facial key points that need to be regenerated.

[0133] Step 330: The key point reconstruction network reconstructs the lip shape key points of the t-th candidate key point map based on the audio features of the first target audio and the facial key point maps of multiple key video frames, to obtain the t-th driving facial key point map corresponding to the t-th speech video frame.

[0134] A keypoint reconstruction network is a neural network model used to generate lip-sync keypoints for audio-driven audio. In some embodiments, the keypoint reconstruction network may include a first encoder network and a first decoder network; for example, the keypoint reconstruction network may be a Transformer network. The first encoder network fuses and encodes the audio features of the first target audio, facial keypoint maps of multiple key video frames, and the t-th candidate keypoint map to obtain a fused feature map. Then, the fused feature map is input into the first decoder network, which decodes the fused feature map and outputs the t-th driving facial keypoint map corresponding to the t-th speech video frame.

[0135] In some embodiments, the sub-audio features of the t-th audio segment can be obtained from the audio features of the first target audio. The first encoder network fuses and encodes the sub-audio features of the t-th audio segment, the facial key point maps corresponding to multiple key video frames, and the t-th candidate key point map to obtain a fused feature map. Then, the fused feature map is input into the decoder network, and the first decoder network decodes the fused feature map to output the t-th driving facial key point map corresponding to the t-th speech video frame.

[0136] Similarly, the relevant facial key points in the t-th driving facial key point map can be connected to obtain a driving facial key point map that reflects the facial contour.

[0137] Figure 4 This is a schematic diagram of lip-shape keypoint driving according to an embodiment of this application. The audio features of the first target audio, the facial keypoint maps corresponding to multiple key video frames, and the t-th candidate keypoint map are input into the keypoint reconstruction network. Based on the audio features of the first target audio and the facial keypoint maps corresponding to multiple key video frames, lip-shape keypoints representing the lip shape pronounced according to the t-th audio unit in the first target audio and conforming to the speaking lip-shape style of the target object are generated, i.e., reconstructed lip-shape keypoints. Then, the reconstructed lip-shape keypoints and the facial keypoints in the t-th candidate keypoint map are combined, and the related facial keypoints are connected to output the t-th driving facial keypoint map.

[0138] In some embodiments, instead of removing lip-shape key points from the t-th facial key point map corresponding to the t-th speech video frame, the t-th facial key point map corresponding to the t-th speech video frame, the facial key point maps corresponding to the multiple key video frames, and the audio features of the first target audio (or the sub-audio features of the t-th audio unit in the first target audio) are input into the key point reconstruction network. The key point reconstruction network then reconstructs the lip-shape key points in the t-th facial key point map and outputs the t-th driving facial key point map.

[0139] Step 240: Optical flow estimation is performed based on each driving facial key point map in the driving facial key point map sequence, the reference video frame selected in the speech video frame sequence, and the facial key point map of the reference video frame to obtain the optical flow information sequence.

[0140] The optical flow information sequence includes optical flow information applicable to each reconstructed video frame. A reconstructed video frame refers to the video frame to be generated using the video frames in the first target audio-driven speech video frame sequence, i.e., the video frames in the target speech video frame sequence below. The optical flow information applicable to a single reconstructed video frame describes the pixel movement of the reference video frame relative to the reconstructed video frame, such as the pixel offset and offset direction.

[0141] It is understandable that the t-th driving facial keypoint map in the driving facial keypoint map sequence can be approximated as the facial keypoint map of the t-th reconstructed video frame to be generated. The optical flow of the reference video frame relative to the reconstructed video frame can reflect the facial keypoint map of the reference video frame relative to the facial keypoint map of the t-th reconstructed video frame. Therefore, in this application, the optical flow of the facial keypoint map of the reference video frame relative to the facial keypoint map of the t-th reconstructed video frame, as well as the pixels of the reference video frame itself, can be used to inversely estimate the optical flow information of the reference video frame relative to the t-th reconstructed video frame.

[0142] The reference video frame selected from the speech video frame sequence can be one or more, without specific limitation. It is understood that if there are multiple reference video frames, the optical flow information applicable to the t-th reconstructed video frame includes the sub-optical flow information of each of the multiple reference video frames relative to the t-th reconstructed video frame. The sub-optical flow information of a reference video frame relative to the t-th reconstructed video frame may include the optical flow map of that reference video frame relative to the t-th reconstructed video frame. In other embodiments, the sub-optical flow information of a reference video frame relative to the t-th reconstructed video frame may also include the weights corresponding to the optical flow map.

[0143] In some embodiments, the reference video frames selected for different reconstructed video frames may be the same or different. In some embodiments, one or more video frames may be randomly selected from the speech video frame sequence as reference video frames.

[0144] In some embodiments, such as Figure 5 As shown, step 240 includes: step 510, in the speech video frame sequence, selecting i reference video frames for the reconstructed video frame t; t is a positive integer from 1 to N; N is the number of reconstructed video frames in the target speech video frame sequence; 1≤i≤N, i is a positive integer.

[0145] The value of i can be set as needed; it can be 1 or other values. In specific embodiments, to facilitate the provision of rich optical flow information, i can be an integer greater than 1.

[0146] In some embodiments, the multiple key video frames selected above can be used as the i-th reference video frame selected for the reconstructed video frame. In other embodiments, i-th video frames can be randomly selected from the speech video frame sequence as the i-th reference video frame selected for the reconstructed video frame t.

[0147] In some embodiments, to ensure that each video frame in the generated target speech video frame sequence has richer tooth details, multiple video frames with rich tooth details can be selected from the speech video frame sequence as the i-frame reference video frames selected for reconstructing the video frame of the t-th frame. For example, in the speech video frame sequence, the i-frame speech video frames with the largest number of tooth pixels can be selected as the i-frame reference video frames selected for reconstructing the video frame of the t-th frame. Tooth pixels refer to the pixels representing teeth in the video frame.

[0148] In some other embodiments, where i > 1, step 510 includes: in the speech video frame sequence, selecting m frame speech video frames with the largest number of tooth pixels; 1 ≤ m < i, and m is a positive integer; in the window where the t-th frame speech video frame is located in the speech video frame sequence, selecting i - m frame speech video frames; and using the m frame speech video frames and the i - m frame speech video frames as the i-frame reference video frames selected for reconstructing the video frame of the t-th frame.

[0149] That is to say, among the i-frame reference video frames selected for reconstructing the video frame of the t-th frame, a part is the m frame speech video frames with the largest number of tooth pixels, and the other part is one or more speech video frames near the t-th frame speech video frame in the speech video frame sequence.

[0150] The window where the t-th frame speech video frame is located refers to a window with a certain width centered on the t-th frame speech video frame. Here, the width of the window can be represented by the number of video frames covered in the window. In some embodiments, it can be randomly selecting i - m frame speech video frames in the window where the t-th frame speech video frame is located, or equally spacing and selecting i - m frame speech video frames in the window where the t-th frame speech video frame is located.

[0151] For example, if i = 10 and m = 5, select 5 frame speech video frames with the largest number of tooth pixels from the speech video frame sequence to ensure the richness and clarity of tooth details in the subsequent driven generated video frames. Assume that the window where the t-th frame speech video frame is located ranges from the (t - a / 2)-th frame to the (t + a / 2)-th frame, a is an even number and a is the width of the window, for example, a = 20. Equally spacing and selecting 5 frame video frames from the window ranging from the (t - a / 2)-th frame to the (t + a / 2)-th frame in the speech video frame sequence as the other 5 reference frames. By selecting the video frames near the t-th frame speech video frame, accurate background details are provided in this way.

[0152] Step 520, obtaining the facial key point maps of the i-frame reference video frames.

[0153] Step 530: The optical flow network estimates the optical flow of the i-th reference video frame relative to the t-th reconstructed video frame based on the t-th driving facial keypoint map in the driving facial keypoint map sequence, the i-th reference video frame, and the facial keypoint map of the i-th reference video frame, and obtains the optical flow information applicable to the t-th reconstructed video frame.

[0154] In some embodiments, the t-th driving facial key point map, the i-th reference video frame, and the reference facial key point map of the i-th reference video frame can be input into a second coding network, which performs fusion coding to obtain a first fusion coding feature; then, the second decoding network decodes the first fusion coding feature to obtain optical flow information suitable for the reconstructed video frame t.

[0155] The optical flow information applicable to the reconstructed video frame t may include i optical flow graphs, each representing the optical flow of a reference video frame relative to the reconstructed video frame t. In some embodiments, the optical flow information applicable to the reconstructed video frame t may further include weights of the i optical flow graphs.

[0156] In other embodiments, step 530 includes the following ①-③:

[0157] ① Obtain at least one adjacent driving face keypoint map from the t-th driving face keypoint map sequence.

[0158] Multiple driving facial keypoint maps adjacent to and before the t-th driving facial keypoint map can be selected as adjacent driving facial keypoint maps determined for the t-th driving facial keypoint map. The number of adjacent driving facial keypoint maps selected before the t-th driving facial keypoint map can be the same as or different from the number of adjacent driving facial keypoint maps selected after the t-th driving facial keypoint map.

[0159] For example, if it is required to determine 2b adjacent driving facial keypoint maps for each driving facial keypoint map, the preceding b frames of driving facial keypoint maps and the following b frames of driving facial keypoint maps adjacent to the t-th driving facial keypoint map in the driving facial keypoint map sequence can be selected as the 2b adjacent driving facial keypoint maps determined for the t-th driving facial keypoint map, where b is a positive integer and b is less than N / 2.

[0160] At least one adjacent driving face keypoint map is determined for the t-th driving face keypoint map to help determine the optical flow information corresponding to the t-th driving face keypoint map, so as to ensure the temporal continuity of the optical flow information corresponding to the continuous driving face keypoint maps.

[0161] In some embodiments, when t=1, b consecutive driving facial keypoint maps following the first driving facial keypoint map can be obtained from the driving facial keypoint map sequence as the corresponding adjacent driving facial keypoint maps. When t=N, b consecutive driving facial keypoint maps preceding the Nth driving facial keypoint map can be obtained from the driving facial keypoint map sequence as the corresponding adjacent driving facial keypoint maps.

[0162] ② Input the t-th driving facial landmark map, at least one adjacent driving facial landmark map, i-frame reference video frame, and the facial landmark map of i-frame reference video frame into the optical flow network.

[0163] The second coding network in the optical flow network can fuse and encode the t-th driving facial keypoint map, at least one adjacent driving facial keypoint map, i-frame reference video frame, and the reference facial keypoint map corresponding to i-frame reference video frame respectively to obtain the first fused coding feature; then, the second decoding network decodes the first fused coding feature to obtain the optical flow information corresponding to the t-th driving facial keypoint map.

[0164] ③ Obtain the optical flow information output by the optical flow network that is suitable for reconstructing the video frame at frame t.

[0165] Figure 6 This is a flowchart illustrating optical flow estimation according to an embodiment of this application, such as... Figure 6 As shown, to determine the optical flow information corresponding to the t-th driving facial keypoint map, the input of the optical flow network includes three parts: ① the tb-th to t+b-th driving facial keypoint maps in the driving facial keypoint map sequence (where the b driving facial keypoint maps before the t-th driving facial keypoint map, the b driving facial keypoint maps after the t-th driving facial keypoint map, and the t-th driving facial keypoint map are the 2b adjacent driving facial keypoint maps determined for the t-th driving facial keypoint map); ② 5 reference video frames selected for the reconstructed video frame t-th; ③ facial keypoint maps of the 5 reference video frames.

[0166] An optical flow network can perform optical flow estimation based on the three inputs mentioned above, obtaining optical flow information suitable for reconstructing the video frame at frame t. For example... Figure 6 As shown, the optical flow information of the reconstructed video frame t includes optical flow maps of 5 reference video frames relative to the reconstructed video frame t, and the weights of the 5 optical flow maps.

[0167] It is worth mentioning that, Figure 6 In order to protect portrait rights, Figure 6 The facial areas in the five reference video frames shown are occluded. In practical applications of the scheme of this application, it is not necessary to occlude or blur the facial areas of the reference video frames that need to be input into the optical flow network.

[0168] Step 250: Based on the reference video frames and optical flow information sequence, perform face reconstruction on the speech video frame sequence to obtain the target speech video frame sequence when the simulated target object speaks according to the first target audio.

[0169] The reconstructed video frame t in the target speech video frame sequence is obtained by reconstructing the face from the speech video frame t in the speech video frame sequence. The number of video frames in the target speech video frame sequence is the same as the number of speech video frames in the speech video frame sequence. If the speech video frame t in the narration video frame sequence is considered as the input face image mentioned above, then the reconstructed video frame t in the target speech video frame sequence can be considered as the reconstructed face image mentioned above.

[0170] The reconstructed video frame includes a facial pixel region representing the target object's face. The lip shape of the target object presented in the t-th reconstructed video frame of the target speech video frame sequence is the same as the lip shape of the target object when pronouncing the t-th audio unit in the first target video, and conforms to the target object's speaking lip-shape style. Furthermore, since the facial region in the speech video frame is reconstructed, the background region in the t-th reconstructed video frame of the target speech video frame sequence can be the same as the background region in the t-th speech video frame of the speech video frame sequence.

[0171] In some embodiments, face reconstruction can be performed using a face reconstruction network, which is a neural network model capable of generating images. The face reconstruction network can sequentially reconstruct faces from each video frame in a speech video frame sequence. For example, for the t-th speech video frame in the speech video frame sequence, face reconstruction can be performed as follows: Optical flow information suitable for the t-th reconstructed video frame is obtained from the optical flow information sequence; the face reconstruction network can extract image features from the reference video frame, optical flow information features corresponding to the t-th driving facial keypoint map, and image features from the t-th speech video frame; these three features are then fused to obtain a third fused feature; and the third fused feature is decoded to obtain the t-th reconstructed video frame.

[0172] In other embodiments, it can be done as follows Figure 7 The process involves facial reconstruction of the t-th frame in the speech video frame sequence, such as... Figure 7 As shown, it includes:

[0173] Step 710: Obtain optical flow information applicable to the reconstructed video frame t.

[0174] Step 720: Obtain the i-th reference video frame selected for reconstructing the video frame of frame t.

[0175] Step 730: Lip occlusion is performed on the t-th frame of the speech video to obtain the t-th frame lip occlusion image.

[0176] Lip occlusion refers to obscuring the pixel region in the t-th frame of a spoken video frame that represents the mouth shape. Therefore, the lip-occluded image of the t-th frame, except for the obscured area, contains the same pixel regions as the t-th frame of the spoken video frame. It is understood that during lip occlusion, the obscured pixel region includes at least the pixel region representing the mouth shape, and may also include the surrounding pixel regions.

[0177] In some embodiments, the lower half of the target object's cheek in the t-th frame of the speaking video can be occluded to obtain the t-th frame lip-sync image, or the occluded area can be a pixel region based on the lower half of the target object's cheek and expanded outwards.

[0178] It is understandable that the occluded pixel areas presented in the lip-sync occlusion image of frame t are the pixel areas that need to be reconstructed in the subsequent face reconstruction process. From this perspective, the lip-sync occlusion image of frame t is used to indicate the pixel areas that need to be reconstructed in the speaking video frame of frame t, while other unoccluded pixel areas do not need to be reconstructed in the subsequent process.

[0179] Step 740: The face reconstruction network performs face reconstruction based on the optical flow information applicable to the reconstructed video frame t, the i-frame reference video frame selected for the reconstructed video frame t, and the lip occlusion image of the t-th frame, to obtain the reconstructed video frame t.

[0180] The i-th reference video frame selected for the t-th driving facial keypoint map can provide rich oral cavity detail information, such as tooth details, and background information. This ensures that the reconstructed video frame can accurately represent the tooth details of the target object when pronouncing the corresponding audio unit. Furthermore, the background pixel regions in the i-th reference video frame provide accurate background information for the reconstruction process, especially when part of the background is occluded in the t-th lip-sync image, thus ensuring the quality of the obtained reconstructed video frame. For each speech video frame in the speech video frame sequence, the face reconstruction network can proceed according to... Figure 7 A similar process is used to handle this.

[0181] In this application, lip-sync keypoints are used to drive the facial keypoint maps corresponding to each speech video frame in the target object's speech video frame sequence, based on the audio features of the first target audio and facial keypoint maps of multiple key video frames. This results in a driven facial keypoint map sequence, which presents the facial keypoints of the target object speaking according to the first target audio. Since multiple key video frames can reflect the target object's lip-sync style, it is equivalent to injecting the target object's lip-sync style into the lip-sync keypoint driving process. This ensures that the lip-sync presented in each driven facial keypoint map in the obtained driven facial keypoint map sequence not only matches the first target audio but also conforms to the target object's lip-sync style.

[0182] Therefore, subsequent optical flow estimation based on the facial keypoint map sequence results in an optical flow information sequence that incorporates the target subject's lip-sync style. This ensures that subsequent facial reconstruction of the speech video frame sequence based on the reference video frame and the optical flow information sequence, yielding the target speech video frame sequence, guarantees that the lip-sync of the target subject in the obtained target speech video frame sequence matches the lip-sync of the subject speaking according to the first target audio, and also conforms to the target subject's lip-sync style. The solution presented in this application effectively addresses the problem in related technologies where images generated by audio-driven lip-sync lack the unique lip-sync style of the individual.

[0183] Considering the optical flow of the facial keypoint map of the reference video frame relative to each driving facial keypoint map in the driving facial keypoint map sequence, which reflects the optical flow of the reference video frame relative to the video frame to be generated, optical flow estimation is performed by combining the facial keypoint map of the reference video frame, the driving facial keypoint map sequence, and the reference video frame to obtain an optical flow information sequence. This optical flow information sequence reflects the optical flow of the reference video frame relative to the video frame to be generated. Subsequently, the optical flow information sequence and the reference video frame are used to reconstruct the face of the speech video frame sequence, resulting in a target speech video frame sequence simulating the target object speaking according to the first target audio. Thus, it is possible to back-estimate the optical flow information between different video frames based on the optical flow information between facial keypoints in different video frames, and then combine the estimated optical flow information with the reference video frame as a reference for lip-syncing.

[0184] In related technologies, it is generally required that the mouth shape presented in the pre-driving speech video frame sequence is a closed mouth shape, in order to avoid significant jitter in the timing of the mouth shape after driving if the pre-driving speech video frame sequence shows a large amount of talking.

[0185] In this application, lip-sync keypoints are driven first to obtain a sequence of driven facial keypoint maps. Optical flow estimation and face reconstruction are then performed based on this sequence. Each driven facial keypoint map in the sequence constrains the facial keypoints in the reconstructed video frames. By driving lip-sync keypoints first, this application prioritizes the stability of the lip movements presented in the reconstructed video frames. Furthermore, during the optical flow estimation stage, at least one adjacent driven facial keypoint map is used as auxiliary information to assist in optical flow estimation. This ensures the temporal continuity of optical flow information applicable to consecutive reconstructed video frames, thereby guaranteeing the temporal continuity of facial poses (including lip movements) presented in the reconstructed video frames and preventing jitter. Thus, even if the lip movements in the preceding speech video frame sequence are large-scale speech movements, lip movement jitter will not occur. Therefore, this application's solution does not have requirements regarding the lip movements presented in the preceding speech video frame sequence, making it more universally applicable.

[0186] Furthermore, in this application, the m-frame speech video with the most tooth pixels can be selected as part or all of the reference video frames. The m-frame speech video with the most tooth pixels can provide rich tooth detail information of the target object, which can ensure that the tooth details presented in the generated reconstructed video frame are basically the same as the tooth details of the target object presented in the speech video frame sequence before driving. In this way, while ensuring the preservation of the target object's lip-sync style, the tooth details of the target object are well preserved, ensuring the lip-sync driving quality of the speech video frame sequence.

[0187] Furthermore, in the solution of this application, the reference video frames used for auxiliary driving can be determined according to the requirements. The more reference video frames there are, the more stable the target speech video frame sequence obtained by driving will be, but the corresponding time spent on lip-syncing will also be longer. Conversely, the fewer reference video frames there are, the shorter the time spent on lip-syncing can be.

[0188] In some embodiments, the face reconstruction network includes a first image encoder, a second image encoder, and an image decoder; such as Figure 8 As shown, step 740 includes:

[0189] Step 810: The first image encoder encodes the i-th reference video frame selected as the reconstructed video frame for the t-th frame to obtain reference visual features.

[0190] The reference visual features include the sub-reference visual features extracted by the first image encoder for each reference video frame selected for the reconstruction of the video frame t.

[0191] Since the first image encoder includes multiple image coding layers, for ease of distinction, the image coding layer in the first image encoder is referred to as the first coding layer. In some embodiments, the visual features output by the last layer of the first coding layer in the first image encoder as the reference video frame can be used as the sub-reference visual features of the reference video frame.

[0192] In other embodiments, the visual features output by each first coding layer in the first image encoder for a reference video frame can also be used as sub-reference visual features of that reference video frame. In other words, in this case, the reference visual features include the k-th layer sub-reference visual features output by the k-th first coding layer in the first image encoder for each reference video frame; 1≤k≤M; M is the total number of first coding layers in the first image encoder.

[0193] Step 820: Perform optical flow mapping based on the optical flow information corresponding to the t-th driving facial key point map and the reference visual features to obtain optical flow fusion features.

[0194] Optical flow mapping refers to the transformation of an image based on optical flow. In some embodiments, it can be achieved by using the sub-reference optical flow information of each reference video frame relative to the reconstructed video frame t in the optical flow information applicable to the t-th frame, performing optical flow mapping on the sub-reference visual features of the corresponding reference video frames, obtaining the optical flow mapping features corresponding to each reference video frame, and then weighting the optical flow mapping features corresponding to all reference video frames corresponding to the t-th frame reconstructed video frame to obtain the optical flow fusion features.

[0195] In some embodiments, if the optical flow information corresponding to the t-th driving facial keypoint map includes the sub-reference optical flow information of the i-th reference video frame corresponding to the t-th driving facial keypoint map relative to the t-th driving facial keypoint map; the sub-reference optical flow information includes the optical flow map and the weight of the optical flow map; in this embodiment, step 820 includes: performing optical flow mapping on the k-th layer sub-reference visual features of the w-th reference video frame and the optical flow map of the w-th reference video frame relative to the t-th reconstructed video frame to obtain the k-th layer optical flow mapping features corresponding to the w-th reference video frame; 1≤w≤i; w is a positive integer; according to the weight of the optical flow map of each reference video frame relative to the t-th reconstructed video frame, the k-th layer optical flow mapping features corresponding to the i-th reference video frames are weighted to obtain the k-th layer sub-optical flow fusion features.

[0196] For example, for the i-th reference video frame selected for reconstructing the t-th video frame, if the k-th layer of the first coding layer in the first image encoder is the k-th layer sub-reference visual feature output from the w-th reference frame, then... Among the i-th reference video frames selected for the reconstructed video frame t, the optical flow diagram of the w-th reference video frame relative to the reconstructed video frame t is as follows: The weight of the optical flow map of the w-th frame reference video frame relative to the t-th frame reconstructed video frame is The k-th layer sub-optical flow fusion feature can be determined according to Formula 2 below

[0197] where represents the k-th layer optical flow mapping feature corresponding to the w-th frame reference video frame.

[0198] Step 830: The second image encoder performs fusion coding based on the image features and optical flow fusion features of the t-th frame mouth occlusion image to obtain the target visual features.

[0199] In some embodiments, it may be to splice the image features and optical flow fusion features of the t-th frame mouth occlusion image and then input them into the second image encoder, and the second image encoder performs fusion coding to obtain the target visual features.

[0200] In other embodiments, the second image encoder may include multiple coding layers. The coding layers in the second image encoder are referred to as the second coding layers. The number of the second coding layers in the second image encoder is M, which is the same as the number of the first coding layers in the first image encoder. Step 830 includes: splicing the k-th layer sub-optical flow fusion feature and the k-th layer coding feature to obtain the k-th layer splicing feature; the k-th layer coding feature is output by the k-th layer second coding layer in the second image encoder; 1 ≤ k ≤ M; when k = 1, the first layer coding feature is the image feature of the t-th frame mouth occlusion image; if k < M, the k + 1-th layer second coding layer in the second image encoder performs fusion coding on the k-th layer splicing feature to obtain the k + 1-th layer coding feature; k is incremented by 1 and returns to execute the step of splicing the k-th layer sub-optical flow fusion feature and the k-th layer coding feature to obtain the k-th layer splicing feature; if k = M, the M-th layer splicing feature is used as the target visual feature.

[0201] That is, the k-th layer sub-optical flow fusion feature is first spliced with the feature output by the k-th layer second coding layer in the second image encoder, and then input into the k + 1-th layer second coding layer in the second image encoder for fusion coding. In this way, the features output by different first coding layers in the first image encoder can be fully utilized, rather than just using the features output by the last first coding layer in the first image encoder, to ensure the accuracy of the target visual features, and further ensure the quality of the reconstructed video frame obtained by subsequent decoding.

[0202] Step 840: The image decoder decodes the target visual features to obtain the t-th frame reconstructed video frame.

[0203] Understandably, by processing each video frame in the speech video frame sequence according to the above process, the corresponding reconstructed video frame can be obtained.

[0204] Figure 9 This is a schematic diagram illustrating facial reconstruction using a facial reconstruction network according to an embodiment of this application; and... Figure 6 Correspondingly, in order to determine the reconstructed video frame corresponding to the t-th speech video frame (i.e., the t-th reconstructed video frame), the 5 reference video frames selected for the t-th reconstructed video frame during optical flow estimation will continue to be used for face reconstruction. For example... Figure 9 As shown, five reference video frames selected for the reconstruction of the t-th frame are input into the first image encoder. Each first coding layer in the first image encoder performs feature encoding on each reference video frame, resulting in features output by each first coding layer for each reference video frame. Assuming the feature output by the k-th layer of the first image encoder is the input of the w-th reference frame, and the feature of the k-th layer sub-reference visual feature is...

[0205] Subsequently, using the optical flow maps of the five reference video frames determined above relative to the reconstructed video frame t, and the weights corresponding to the five optical flow maps, the features output by the same first coding layer in the first image encoder for each reference video frame are fused according to Formula 1 above to obtain the sub-optical flow fusion features of each layer. For example, the features output by the first coding layer of the k-th layer in the first image encoder for the five reference video frames are fused to obtain the sub-optical flow fusion features of the k-th layer.

[0206] Subsequently, the lip-sync image of frame t is encoded by the first layer (second coding layer) of the second image encoder to obtain the image features of frame t. Then, the corresponding sub-optical flow fusion features of the same layer are concatenated with the features output from the corresponding layer in the second image encoder. The concatenated features are then input into the next layer (second coding layer) of the second image encoder for fusion encoding. For example, the sub-optical flow fusion features of layer k... The features output from the second coding layer at layer k in the second image encoder are concatenated and then input into the second coding layer at layer k+1 in the second image encoder for fusion encoding.

[0207] After the features output from the last layer of the second encoding layer in the second image encoder are spliced ​​with the sub-optical flow fusion features of the corresponding layer to obtain the target visual features, the image decoder decodes the target visual features to obtain the t-th reconstructed video frame.

[0208] It is worth mentioning that, Figure 9 In order to protect portrait rights, Figure 9The facial regions in the five reference video frames shown, as well as the facial regions in the t-th reconstructed video frame obtained from facial reconstruction, are occluded. In practical applications of the scheme of this application, it is not necessary to occlude or blur the facial regions of the reference video frames.

[0209] In some embodiments, to ensure the quality of the reconstructed video frames, the optical flow network and face reconstruction network mentioned above need to be trained in advance. The training process for the optical flow network and face reconstruction network can be as follows: Figure 10 As shown, it includes:

[0210] Step 1010: Obtain multiple training samples; wherein each training sample includes a first sample speech video and a second sample speech video of the same reference object.

[0211] In both the first and second sample speech videos, the facial area of ​​the reference subject is shown in each frame. Similarly, the reference subject can be a real person, a virtual person, a digital person, etc. The lip movements of the reference subject are different in the same positions shown in the first and second sample speech videos. For example, the first and second sample speech videos could be recorded while the reference subject is speaking different content; the different content could be different content in the same language, or content expressing the same meaning in different languages.

[0212] In some embodiments, the number of video frames in the first sample speech video and the second sample speech video are the same in the same training sample. In some embodiments, if the number of video frames in the first sample speech video and the second sample speech video are different, some video frames in the sample speech video with more video frames can be discarded to ensure that the number of video frames in both is the same.

[0213] Step 1020: Obtain the sample-driven facial key point map sequence; the sample-driven facial key point map sequence is obtained by driving the lip shape key points of the sample facial key point maps corresponding to each video frame in the first sample speech video based on the first sample audio corresponding to the second sample speech video, or it is a sequence formed by the sample facial key point maps corresponding to each video frame in the second sample speech video.

[0214] The first sample audio corresponding to the second sample speech video refers to the audio representing the speech content of the reference object in the second sample speech video, which can be the audio of the reference object synchronously collected during the recording of the second sample speech video.

[0215] In some embodiments, if it is not necessary to train the keypoint reconstruction network in sync with the optical flow network and the face reconstruction network, the sequence of sample face keypoint maps corresponding to each video frame in the second sample speech video can be used as the sample-driven face keypoint map sequence.

[0216] In some embodiments, where it is necessary to train the keypoint reconstruction network synchronously with the optical flow network and the face reconstruction network, the keypoint reconstruction network can perform lip-sync keypoint driving on the sample facial keypoint maps corresponding to each video frame in the first sample speech video based on the first sample audio corresponding to the second sample speech video, thereby obtaining a sample-driven facial keypoint map sequence. Alternatively, in some embodiments, the trained keypoint reconstruction network can perform lip-sync keypoint driving on the sample facial keypoint maps corresponding to each video frame in the first sample speech video based on the first sample audio corresponding to the second sample speech video, thereby obtaining a sample-driven facial keypoint map sequence. This process is basically similar to the implementation process of step 230 above, and will not be described again here.

[0217] Step 1030: The optical flow network performs optical flow estimation based on the sample-driven facial key point map sequence, the sample reference frame selected from the first sample speech video, and the facial key point map of the sample reference frame, to obtain the sample optical flow information sequence.

[0218] In step 1040, the face reconstruction network performs face reconstruction on each video frame in the first sample speech video based on the sample reference frame and the sample optical flow information sequence, thereby obtaining the reconstructed speech video. The implementation processes of steps 1030 and 1040 are similar to those described above and will not be repeated here.

[0219] Step 1050: Calculate the reconstruction loss based on the reconstructed speech video and the second sample speech video.

[0220] In some embodiments, a first loss can be determined based on the pixel differences between relatively aligned video frames in the reconstructed speech video and the second sample speech video, and the first loss can be used as the reconstruction loss.

[0221] The x-th video frame in the reconstructed speech video and the x-th video frame in the second sample speech video are video frames that are aligned in position, where x is a positive integer of any value.

[0222] First, the pixel difference between two relatively aligned video frames in the reconstructed speech video and the second sample speech video can be calculated using a first loss function. Then, the pixel differences between all relatively aligned video frames are summed to obtain the first loss. The first loss function can be an absolute value loss function, a mean squared error loss function, a cosine loss function, a mean absolute value loss function, a Euclidean distance loss function, etc., and is not specifically limited here. The first loss can be expressed as:

[0223] l pix =||x r -x h || 1 ; (Formula 3)

[0224] x r The video of the speech indicating reconstruction, x h This refers to the second sample speech video, l pix This is the first loss.

[0225] In some embodiments, step 1050 includes: determining a first loss based on the pixel differences between video frames that are aligned in position in the reconstructed speech video and the second sample speech video; determining a second loss based on the depth features of each video frame in the reconstructed speech video and the depth features of each video frame in the second sample speech video; determining a third loss based on the discrimination results of the discriminator network for each video frame in the reconstructed speech video and for each video frame in the second sample speech video; and weighting at least one of the second loss and the third loss with the first loss to obtain a reconstruction loss.

[0226] In some embodiments, a deep feature extraction network capable of accurately extracting depth features can be introduced to extract depth features from each video frame in the reconstructed speech video and from each video frame in the second sample speech video. Then, a second loss is calculated based on the depth features of two video frames that are aligned in position in the reconstructed speech video and the second sample speech video.

[0227] In some embodiments, the depth feature loss between two relatively aligned video frames in the reconstructed speech video and the second sample speech video can be calculated according to the second loss function. Then, the depth feature losses corresponding to all relatively aligned video frames in the reconstructed speech video and the second sample speech video are added together to obtain the second loss. The second loss function can be an absolute value loss function, a mean squared error loss function, a cosine loss function, a mean absolute value loss function, an Euclidean distance loss function, etc., and is not specifically limited here.

[0228] In some embodiments, the deep feature extraction network may be a VGG network (Visual Geometry Group Network), but is not limited to this. The second loss can be expressed as:

[0229] l per =||φ(x) r )-φ(x h )|| 1 ; (Formula 4)

[0230] l perThis represents the second loss, where φ is the deep feature extraction network, such as the VGG network, φ(x r ) represents the deep features extracted by the deep feature extraction network for reconstructing video frames in the speech video xr, φ(x h ) indicates that the deep feature extraction network is used for the second sample speech video x h Depth features extracted from video frames.

[0231] In some embodiments, the third loss can be determined according to the following formula:

[0232]

[0233] Where D(x) h ) indicates that the discriminator network evaluates the second sample speech video x. h The discrimination results of each video frame; D(x) r ) indicates that the discriminator network reconstructs the speech video x r The discrimination results of each video frame, l adv This is the third loss. The discriminator network is used to determine whether the input image (video frame) is a real image or a generated image, and the discrimination result indicates whether the corresponding video frame is a real recorded video frame or an automatically generated video frame.

[0234] In some embodiments, the first loss and the second loss can be weighted to obtain the reconstruction loss. In other embodiments, the first loss and the third loss can be weighted to obtain the reconstruction loss. In yet another embodiment, the first loss, the second loss, and the third loss can be weighted to obtain the reconstruction loss. The weighting coefficients in the weighting process can all be integers, and the weighting coefficients for different items can be the same or different. For example, the reconstruction loss can be determined according to the following formula 5:

[0235] l = l pix +l per +l adv ;(Formula 6)

[0236] In the above embodiments, the introduction of a second loss and / or a third loss, instead of focusing solely on the pixel differences between video frames that are aligned in the reconstructed speech video and the second sample speech video, allows the supervised optical flow network and face reconstruction network to learn the high-frequency information of the supervised second sample speech video. This can make the images (video frames) generated by the optical flow network and face reconstruction network after training smoother, and make the high-frequency information of the generated images closer to the high-frequency information of the real images.

[0237] Step 1060: Based on the reconstruction loss, adjust the parameters of at least the optical flow network and the face reconstruction network until the training termination condition is met.

[0238] In some embodiments, if it is not necessary to train the keypoint reconstruction network, optical flow network, and face reconstruction network synchronously, in step 1060, the parameters of the optical flow network and face reconstruction network can be adjusted according to the reconstruction loss and in accordance with the gradient descent method.

[0239] In another embodiment, if it is necessary to train the keypoint reconstruction network, optical flow network, and face reconstruction network synchronously, and the sample-driven face keypoint map sequence is obtained by the keypoint reconstruction network driving the lip shape keypoints of the sample face keypoint maps corresponding to each video frame in the first sample speech video based on the sample audio corresponding to the second sample speech video, the parameters of the optical flow network, face reconstruction network, and keypoint reconstruction network can be adjusted according to the reconstruction loss and in the gradient descent manner.

[0240] Training can end when the number of iterations reaches a threshold or the reconstruction loss converges; in such cases, training is not performed. Following the above process ensures that the trained optical flow network accurately estimates optical flow and guarantees the quality of video frames / images obtained by the face reconstruction network.

[0241] In some embodiments, a training sample may include a first sample speech image and a second sample speech image of the same reference object. In this case, the optical flow network and the face reconstruction network can be trained at least according to the following steps 1)-6):

[0242] 1) Obtain multiple training samples; wherein each training sample includes a first sample speech image and a second sample speech image of the same reference object;

[0243] 2) Obtain the sample facial contour; the sample facial contour is obtained by driving the lip shape key points of the sample facial contour corresponding to the first sample speech image based on the sample audio corresponding to the second sample speech image, or it is the facial contour corresponding to the second sample speech image.

[0244] 3) The optical flow network performs optical flow estimation based on the sample face contour, the sample reference face image of the reference object, and the face contour of the sample reference face image to obtain the sample optical flow information;

[0245] 4) The face reconstruction network reconstructs the face of the first sample speech image based on the sample reference face image and sample optical flow information to obtain the sample reconstructed face image;

[0246] 5) Reconstruct the face image from the sample and the speech image from the second sample, and calculate the reconstruction loss.

[0247] Similarly, a first loss can be determined based on the pixel differences between the reconstructed face image and the second sample speech image; a second loss can be determined based on the depth features of the reconstructed face image and the depth features of the second sample speech image; a third loss can be determined based on the discrimination results of the discriminator network on the second sample speech image; and at least one of the second and third losses can be weighted with the first loss to obtain the reconstruction loss.

[0248] 6) Based on the reconstruction loss, adjust the parameters of at least the optical flow network and the face reconstruction network until the training termination condition is met.

[0249] The implementation details of 1)-6) above are the same as those in the text above. Figure 10 The corresponding implementation methods are similar and will not be described in detail here.

[0250] In some embodiments, the keypoint reconstruction network can be trained according to ①-⑧ as follows:

[0251] ① Obtain multiple third-sample speech videos and corresponding sample speech audio for each third-sample speech video; the third-sample speech videos present the facial region of the first reference object;

[0252] ② Based on the number Y of video frames in the third sample speech video, divide the corresponding sample speech audio into Y sample audio units of equal duration. Thus, the lip movements displayed in the y-th frame of the third sample speech video are based on the lip movements of the y-th sample audio unit in the corresponding sample speech audio. y is less than or equal to Y, and y is a positive integer.

[0253] ③ Obtain key video frames from the third sample speech video. These key video frames are used to reflect the speaking style of the target subject.

[0254] ④ Facial key points were extracted from each video frame of the third sample speech video to obtain facial key point maps of each video frame of the third sample speech video.

[0255] ⑤ The lip shape key point removal process is performed on the facial key point map corresponding to each video frame in the third sample speech video to obtain the sample candidate key point map corresponding to each video frame in the third sample speech video.

[0256] ⑥ For the y-th video frame in the third sample speech video, the keypoint reconstruction network reconstructs the lip shape keypoints of the candidate keypoint map corresponding to the y-th video frame based on the audio features of the y-th sample audio unit in the sample speech audio and the facial keypoint maps corresponding to the key video frames of multiple samples, and obtains the prediction-driven facial keypoint map corresponding to the y-th video frame; y is a positive integer that takes values ​​sequentially from 1 to Y.

[0257] ⑦ Calculate the lip shape key point reconstruction loss based on the predicted driving facial key point map corresponding to the y-th video frame and the facial key point map of the y-th video frame.

[0258] ⑧ Adjust the parameters of the keypoint reconstruction network based on the lip shape keypoint reconstruction loss until the first training termination condition is met.

[0259] Through the training process described above, the keypoint reconstruction network can accurately drive lip-sync keypoints, ensuring that the lip shape presented in the reconstructed facial keypoints is the same as the lip shape corresponding to the audio applied to the drive. Moreover, it guarantees that the lip shape presented by the reconstructed facial keypoints conforms to the speaker's lip-sync style.

[0260] One approach to addressing the lack of distinctive lip-syncing styles in videos is to train a separate neural network model for each subject, learning the subject's lip-syncing style during training. However, this method requires training a separate model for each subject before it can be used to drive lip-syncing in their videos. This approach demands significant time for training models applicable to different subjects, thus lacking universality.

[0261] In this application, the approach of combining keypoint reconstruction network, optical flow network and face reconstruction network to drive lip movements through audio is applicable to different objects and does not require separate training for different objects. Moreover, it can ensure that the lip movements in the video obtained by driving are the same as the lip movements corresponding to the audio, and it can conform to the speaking lip movement style of the object in the video frame before driving, thus well preserving the speaking lip movement style of the object. Even if it is necessary to drive the speaking video of a new object, it is not necessary to retrain.

[0262] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0263] Figure 11 This is a block diagram of an audio-driven face reconstruction apparatus according to an embodiment of this application, as shown below. Figure 11As shown, the audio-driven face reconstruction device includes: an acquisition module 1110 for acquiring target audio; a first driving module 1120 for obtaining the target face contour of the target object under the driving action of the target audio, the target face contour including a contour related to lip shape, based on the target audio and the input face image of the target object; an optical flow information determination module 1130 for determining optical flow information based on the target face contour of the target object, a reference face image of the target object, and a reference face contour of the reference face image, the optical flow information being used to characterize the offset information between the reconstructed face image and the reference face image of the target object; and a reconstruction module 1140 for performing reconstruction processing on the input face image based on the reference face image of the target object and the optical flow information to obtain a reconstructed face image.

[0264] In some embodiments, the optical flow information determination module 1130 includes: a selection unit, configured to select i reference video frames as reference face images from a sequence of speech video frames of a target object; i is an integer greater than 1; a first acquisition unit, configured to acquire the reference face contour of the i reference video frames; and an optical flow information determination unit, configured to determine optical flow information by an optical flow network based on the target face contour of the target object, the i reference video frames, and the reference face contour of the i reference video frames.

[0265] In some embodiments, in the speech video frame sequence of the target object, selecting i reference video frames as reference facial images includes at least one of the following two selection methods: selecting at least one speech video frame with the most tooth pixels in the speech video frame sequence of the target object; or, when the input facial image comes from the speech video frame sequence, selecting at least one speech video frame in the video frame window containing the input facial image in the speech video frame sequence.

[0266] In some embodiments, the optical flow information determination unit is configured to: when the input face image comes from a speech video frame sequence, acquire the facial contour of at least one adjacent face image adjacent to the input face image; input the target face contour, the facial contour of at least one adjacent face image, the i-frame reference video frame, and the reference face contour of the i-frame reference video frame into the optical flow network to obtain optical flow information.

[0267] In some embodiments, the reconstruction module 1140 includes: an occlusion unit for occluding a lip-related region of an input face image to obtain a lip-occluded image; and a face reconstruction unit for performing face reconstruction by a face reconstruction network based on a reference face image of a target object, optical flow information, and the lip-occluded image to obtain a reconstructed face image.

[0268] In some embodiments, the face reconstruction network includes a first image encoder, a second image encoder, and an image decoder: The face reconstruction unit includes: A first encoding unit configured to encode a reference face image of a target object by the first image encoder to obtain a reference visual feature; An optical flow mapping unit configured to perform optical flow mapping based on optical flow information and the reference visual feature to obtain an optical flow fusion feature; A fusion encoding unit configured to perform fusion encoding on the image feature of the mouth occlusion image and the optical flow fusion feature by the second image encoder to obtain a target visual feature; A decoding unit configured to decode the target visual feature by the image decoder to obtain a reconstructed face image.

[0269] In some embodiments, the optical flow information includes an optical flow map of the i-th reference video frame and the weight of the optical flow map; the reference visual feature includes the k-th layer sub-reference visual feature output by the k-th layer first encoding layer in the first image encoder for each reference video frame; the optical flow fusion feature includes the k-th layer sub-optical flow fusion feature; where 1 ≤ k ≤ M; M is the number of first encoding layers in the first image encoder; The optical flow mapping unit is configured to: perform optical flow mapping on the k-th layer sub-reference visual feature of the w-th reference video frame and the optical flow map of the w-th reference video frame to obtain the k-th layer optical flow mapping feature corresponding to the w-th reference video frame; where 1 ≤ w ≤ i; w is a positive integer; perform weighted processing on the k-th layer optical flow mapping features respectively corresponding to the i reference video frames according to the weights of the optical flow maps of the respective reference video frames to obtain the k-th layer sub-optical flow fusion feature.

[0270] In some embodiments, the second image encoder includes M layers of second encoding layers; The fusion encoding unit is configured to: splice the k-th layer sub-optical flow fusion feature and the k-th layer encoding feature to obtain a k-th layer splicing feature; the k-th layer encoding feature is output by the k-th layer second encoding layer in the second image encoder; where 1 ≤ k ≤ M; when k = 1, the first layer encoding feature is the image feature of the mouth occlusion image; if k < M, perform fusion encoding on the k-th layer splicing feature by the (k + 1)-th layer second encoding layer in the second image encoder to obtain the (k + 1)-th layer encoding feature; increment k by 1 and return to execute the step of splicing the k-th layer sub-optical flow fusion feature and the k-th layer encoding feature to obtain the k-th layer splicing feature; if k = M, use the M-th layer splicing feature as the target visual feature.

[0271] In some embodiments, the audio-driven face reconstruction device further includes: a first acquisition module for acquiring multiple training samples; wherein each training sample includes a first sample speech image and a second sample speech image of the same reference object; a second acquisition module for acquiring sample face contours; the sample face contours are obtained by performing lip-sync keypoint driving on the sample face contours corresponding to the first sample speech image based on the sample audio corresponding to the second sample speech image, or are the face contours corresponding to the second sample speech image; a first optical flow estimation module for performing optical flow estimation by an optical flow network based on the sample face contours, a sample reference face image of the reference object, and the face contours of the sample reference face image to obtain sample optical flow information; a first face reconstruction module for performing face reconstruction on the first sample speech image by a face reconstruction network based on the sample reference face image and the sample optical flow information to obtain a sample reconstructed face image; a reconstruction loss calculation module for calculating reconstruction loss based on the sample reconstructed face image and the second sample speech image; and an adjustment module for adjusting the parameters of at least the optical flow network and the face reconstruction network based on the reconstruction loss until the training termination condition is met.

[0272] In some embodiments, the reconstruction loss calculation module is configured to: determine a first loss based on the pixel difference between the sample reconstructed face image and the second sample speech image; determine a second loss based on the depth features of the sample reconstructed face image and the depth features of the second sample speech image; determine a third loss based on the discrimination result of the discriminator network on the second sample speech image; and weight at least one of the second loss and the third loss with the first loss to obtain the reconstruction loss.

[0273] In some embodiments, the first driving module 1120 includes: a facial pose key point determination unit, configured to acquire facial pose key points of an input facial image of a target object, wherein the facial pose key points are facial key points unrelated to lip shape; a lip shape key point determination unit, configured to acquire lip shape key points of the target object related to lip shape; and a target facial contour determination unit, configured to obtain the target facial contour of the target object based on the target audio, lip shape key points, and facial pose key points.

[0274] In some embodiments, the lip shape key point determination unit is configured to: select multiple speech images of a target object with different lip shapes; and extract lip shape key points related to the lip shape based on the multiple speech images with different lip shapes.

[0275] In some embodiments, the input facial image is a video of the target object speaking in a first language; the target audio is audio in a second language; the first language and the second language are different.

[0276] Figure 12A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 12 The computer system 1200 of the illustrated electronic device is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application. This electronic device may be a server, terminal, or other similar device, and can be used to execute the methods provided in this application.

[0277] like Figure 12 As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0278] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0279] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0280] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0281] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0282] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0283] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0284] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal. It can be implemented wholly or partially using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0285] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods of any of the above embodiments.

[0286] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0287] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0288] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0289] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An audio-driven face reconstruction method, characterized in that, include: Obtain the target audio; Based on the target audio and the input facial image of the target object, the target facial contour of the target object is obtained under the driving action of the target audio, and the target facial contour includes a contour related to the mouth shape; Optical flow information is determined based on the target facial contour of the target object, the reference facial image of the target object, and the reference facial contour of the reference facial image. The optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object. Based on the reference facial image of the target object and the optical flow information, the input facial image is reconstructed to obtain the reconstructed facial image.

2. The method according to claim 1, characterized in that, The step of determining optical flow information based on the target facial contour of the target object, a reference facial image of the target object, and a reference facial contour of the reference facial image includes: In the video frame sequence of the target object's speech, i reference video frames are selected as reference facial images; i is an integer greater than 1. Obtain the reference facial contour of the reference video frame i; The optical flow network determines the optical flow information based on the target facial contour of the target object, the i-frame of the reference video frame, and the reference facial contour of the i-frame of the reference video frame.

3. The method according to claim 2, characterized in that, The step of selecting i reference video frames as reference facial images from the speech video frame sequence of the target object includes at least one of the following two selection methods: In the speech video frame sequence of the target object, at least one speech video frame with the highest number of tooth pixels is selected; When the input face image comes from the speech video frame sequence, at least one speech video frame is selected from the video frame window in the speech video frame sequence that contains the input face image.

4. The method according to claim 2, characterized in that, The process of determining optical flow information by the optical flow network based on the target facial contour of the target object, the i-frame of the reference video frame, and the reference facial contour of the i-frame of the reference video frame includes: When the input face image comes from the speech video frame sequence, obtain the facial contour of at least one adjacent face image that is adjacent to the input face image; The target facial contour, the facial contour of at least one adjacent facial image, the i-th frame of the reference video frame, and the reference facial contour of the i-th frame of the reference video frame are input into the optical flow network to obtain optical flow information.

5. The method according to any one of claims 2 to 4, characterized in that, The step of reconstructing the input face image based on the reference face image of the target object and the optical flow information to obtain the reconstructed face image includes: The lip-related region of the input face image is occluded to obtain a lip-occluded image; The face reconstruction network performs face reconstruction based on the reference face image of the target object, the optical flow information, and the lip occlusion image to obtain the reconstructed face image.

6. The method according to claim 5, characterized in that, The face reconstruction network includes a first image encoder, a second image encoder, and an image decoder; The face reconstruction network performs face reconstruction based on the reference face image of the target object, the optical flow information, and the lip occlusion image to obtain the reconstructed face image, including: The reference facial image of the target object is encoded by the first image encoder to obtain reference visual features; Optical flow mapping is performed based on the optical flow information and the reference visual features to obtain optical flow fusion features; The second image encoder performs fusion encoding based on the image features of the lip-occlusion image and the optical flow fusion features to obtain the target visual features; The image decoder decodes the visual features of the target to obtain the reconstructed face image.

7. The method according to claim 6, characterized in that, The optical flow information includes the optical flow map and weights of the i-th reference video frame; the reference visual features include the k-th layer sub-reference visual features output by the k-th layer of the first encoding layer in the first image encoder for each reference video frame; the optical flow fusion features include the k-th layer sub-optical flow fusion features; 1≤k≤M; M is the number of first coding layers in the first image encoder; The step of performing optical flow mapping based on the optical flow information and the reference visual features to obtain optical flow fusion features includes: Optical flow mapping is performed between the k-th layer sub-reference visual features of the w-th reference video frame and the optical flow map of the w-th reference video frame to obtain the k-th layer optical flow mapping features corresponding to the w-th reference video frame; 1≤w≤i; w is a positive integer; Based on the weights of the optical flow maps of each reference video frame, the k-th layer optical flow mapping features corresponding to the i-th frame and the reference video frame are weighted to obtain the k-th layer sub-optical flow fusion features.

8. The method according to claim 7, characterized in that, The second image encoder includes an M-layer second coding layer; The step of fusing and encoding the target visual features by the second image encoder based on the image features of the lip-occlusion image and the optical flow fusion features includes: The k-th layer sub-optical flow fusion feature is concatenated with the k-th layer coding feature to obtain the k-th layer concatenated feature; the k-th layer coding feature is output by the k-th layer second coding layer in the second image encoder; 1≤k≤M; when k=1, the 1-th layer coding feature is the image feature of the lip-occlusion image; If k < M, the k+1th layer of the second encoding layer in the second image encoder is used to fuse and encode the k-th layer splicing features to obtain the k+1th layer encoded features. Increment k by 1 and return to the step of concatenating the k-th layer sub-optical flow fusion feature with the k-th layer coding feature to obtain the k-th layer concatenated feature; If k = M, the Mth layer stitched features are used as the target visual features.

9. The method according to claim 5, characterized in that, Before the face reconstruction network performs face reconstruction based on the reference face image of the target object, the optical flow information, and the lip occlusion image to obtain the reconstructed face image, the method further includes: Multiple training samples are acquired; wherein each training sample includes a first sample speech image and a second sample speech image of the same reference object; Obtain the sample facial contour; the sample facial contour is obtained by driving the lip shape key points of the sample facial contour corresponding to the first sample speech image based on the sample audio corresponding to the second sample speech image, or it is the facial contour corresponding to the second sample speech image. The optical flow network performs optical flow estimation based on the sample face contour, the sample reference face image of the reference object, and the face contour of the sample reference face image to obtain sample optical flow information; The face reconstruction network performs face reconstruction on the first sample speech image based on the sample reference face image and the sample optical flow information to obtain the sample reconstructed face image; Reconstruct the facial image based on the given sample and the speech image based on the second sample, and calculate the reconstruction loss; Based on the reconstruction loss, at least the parameters of the optical flow network and the face reconstruction network are adjusted until the training termination condition is met.

10. The method according to claim 9, characterized in that, The step of reconstructing the facial image based on the sample and the second sample speech image, and calculating the reconstruction loss, includes: The first loss is determined based on the pixel differences between the reconstructed facial image from the sample and the second sample speech image; The second loss is determined by reconstructing the depth features of the facial image from the sample and the depth features of the speech image from the second sample; The third loss is determined based on the discrimination results of the discriminator network on the second sample speech image. The reconstruction loss is obtained by weighting at least one of the second loss and the third loss with the first loss.

11. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the target facial contour of the target object under the driving force of the target audio, based on the target audio and the input facial image of the target object, includes: Obtain the facial pose key points of the input facial image of the target object, wherein the facial pose key points are facial key points that are independent of the mouth shape; Obtain the key points of the mouth shape of the target object related to the mouth shape; Based on the target audio, lip shape key points, and facial pose key points, the target facial contour of the target object is obtained.

12. The method according to claim 11, characterized in that, The process of obtaining the key lip shape points of the target object related to the lip shape includes: Select multiple images of the target object speaking with different lip movements; Based on the multiple images of speech with different mouth shapes, extract the key points of the mouth shape related to the mouth shape.

13. The method according to any one of claims 1 to 4, characterized in that, The input facial image is a video of the target object speaking in a first language; the target audio is audio in a second language; the first language and the second language are different.

14. An audio-driven face reconstruction device, characterized in that, include: The acquisition module is used to acquire the target audio. The first driving module is used to obtain the target facial contour of the target object under the driving action of the target audio based on the target audio and the input facial image of the target object. The target facial contour includes a contour related to the mouth shape. The optical flow information determination module is used to determine optical flow information based on the target facial contour of the target object, the reference facial image of the target object, and the reference facial contour of the reference facial image. The optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object. The reconstruction module is used to reconstruct the input face image based on the reference face image of the target object and the optical flow information to obtain the reconstructed face image.

15. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-13.

16. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, the method as described in any one of claims 1-13 is implemented.

17. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1-13.

Images