Voice-based portrait video generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a voice-based portrait video generation method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a sample portrait image and sample voice data of a target portrait; carrying out portrait feature extraction on the sample portrait image to obtain sample portrait feature information; performing alignment processing on the sample voice data and the sample portrait feature information to obtain voice portrait alignment information; performing parameter optimization on a preset portrait video frame generation model according to the voice portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model; obtaining target voice data, and performing portrait video frame generation through a target portrait video frame generation model, the target voice data and the sample portrait feature information to obtain a target face video frame; and carrying out audio-video splicing on the target voice data and the target face video frame to obtain a target portrait voice video. According to the embodiment of the invention, the audio and picture synchronous digital human video can be accurately generated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and apparatus, device and storage medium for generating a portrait video based on speech. Background Art

[0002] Speech-Driven Portrait Animation is a technology that converts speech input into synchronized facial animation, so that the virtual character or character in the video can naturally change facial expressions and postures according to the speech content. In the existing speech-driven portrait technology, digital human generation based on the Diffusion model is used. However, digital human generation based on the Diffusion model often cannot accurately match the speech and the picture, resulting in the generated video being not natural and realistic enough, and there are problems such as lip shape and speech being out of sync, and incoherent expressions. Therefore, how to accurately generate digital human videos with synchronized sound and picture has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method and apparatus, device and storage medium for generating a portrait video based on speech, aiming to accurately generate a digital human video with synchronized audio and video.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for generating a portrait video based on speech, the method comprising:

[0005] Obtaining sample portrait images and sample voice data of the target person;

[0006] Extracting portrait features from the sample portrait image to obtain sample portrait feature information;

[0007] Aligning the sample speech data with the sample portrait feature information to obtain speech-portrait alignment information; wherein the sample portrait feature information in the speech-portrait alignment information is semantically aligned with the sample speech data;

[0008] Optimizing parameters of a preset portrait video frame generation model according to the voice-portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model;

[0009] Acquire target voice data, and generate a portrait video frame using the target portrait video frame generation model, the target voice data and the sample portrait feature information to obtain a target face video frame;

[0010] The target voice data and the target face video frame are spliced ​​together to obtain the target face voice video.

[0011] In some embodiments, aligning the sample voice data with the sample portrait feature information to obtain voice-portrait alignment information includes:

[0012] Extracting facial posture features from the sample portrait feature information to obtain facial posture information;

[0013] Performing face alignment node extraction on the sample voice data to obtain voice portrait node information; wherein the voice portrait node information is used to indicate the expression and lip shape of the target portrait;

[0014] Node alignment is performed according to the voice portrait node information, the sample portrait feature information and the facial posture information to obtain the voice portrait alignment information.

[0015] In some embodiments, extracting facial gesture features from the sample portrait feature information to obtain facial gesture information includes:

[0016] Extracting angle and posture features from the sample portrait feature information to obtain angle and posture information; wherein the angle and posture information is used to indicate the posture of the target portrait at different angles;

[0017] Extracting facial expression and posture features from the sample portrait feature information to obtain facial expression and posture information; wherein the facial expression and posture information is used to indicate the posture of the target portrait under different expressions;

[0018] The angle posture information and the expression posture information are concatenated to obtain the facial posture information.

[0019] In some embodiments, extracting face alignment nodes from the sample voice data to obtain voice portrait node information includes:

[0020] Performing speech encoding on the sample speech data to obtain sample speech encoding information;

[0021] Extracting facial gesture tendency according to the sample speech coding information to obtain facial gesture tendency information;

[0022] Perform face alignment node extraction on the face posture tendency information to obtain the voice portrait node information.

[0023] In some embodiments, the step of optimizing parameters of a preset portrait video frame generation model according to the voice portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model includes:

[0024] Generate a portrait video frame using a preset portrait video frame generation model and the voice-portrait alignment information to obtain an original portrait video frame;

[0025] The parameters of the preset portrait video frame generation model are optimized through the original portrait video frame and the sample portrait image to obtain the target face image generation model.

[0026] In some embodiments, obtaining a sample portrait image and sample voice data of the target portrait includes:

[0027] Obtaining the original portrait voice video of the target portrait;

[0028] The original portrait voice video is split into audio and video to obtain the original portrait image and sample voice data

[0029] The target portrait of the original portrait image is subjected to portrait angle correction to obtain the sample portrait image.

[0030] In some embodiments, performing portrait angle correction on the target portrait of the original portrait image to obtain the sample portrait image includes:

[0031] Extracting key frames from the original portrait image to obtain a candidate portrait image;

[0032] Calculating the optical flow change of the candidate portrait image to obtain the optical flow change amount;

[0033] The target portrait of the original portrait image is subjected to portrait angle correction according to the optical flow variation to obtain the sample portrait image.

[0034] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a voice-based portrait video generation device, the device comprising:

[0035] A sample acquisition module, used to acquire sample portrait images and sample voice data of a target person;

[0036] A feature extraction module is used for extracting portrait features from the sample portrait image to obtain sample portrait feature information;

[0037] An information alignment module, used for aligning the sample speech data with the sample portrait feature information to obtain speech-portrait alignment information; wherein the sample portrait feature information in the speech-portrait alignment information is semantically aligned with the sample speech data;

[0038] A model optimization module, used to optimize parameters of a preset portrait video frame generation model according to the voice portrait alignment information and the sample portrait image, to obtain a target portrait video frame generation model;

[0039] A portrait generation module is used to obtain target voice data, and generate a portrait video frame through the target portrait video frame generation model, the target voice data and the sample portrait feature information to obtain a target face video frame;

[0040] The audio and video splicing module is used to perform audio and video splicing on the target voice data and the target face video frame to obtain the target portrait voice video.

[0041] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] The present application proposes a method and apparatus, a device and a storage medium for generating a portrait video based on speech, which obtains a sample portrait image and sample speech data of a target portrait, then extracts portrait features from the sample portrait image to obtain sample portrait feature information, and aligns the sample speech data and the sample portrait feature information, thereby corresponding the portrait information and the picture information to obtain speech-portrait alignment information; further, according to the speech-portrait alignment information and the sample portrait image, the parameters of a preset portrait video frame generation model are optimized to obtain a target portrait video frame generation model, and then the target speech data is obtained, and the portrait video frame is generated by the target portrait video frame generation model, the target speech data and the sample portrait feature information to obtain a target face video frame, and finally the target speech data and the target face video frame are spliced ​​for audio and video to obtain a target portrait voice video, so as to achieve accurate generation of a digital human video with synchronized audio and video by corresponding the portrait and the speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a method for generating a portrait video based on speech provided in an embodiment of the present application;

[0045] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0046] Figure 3 yes Figure 2 Flow chart of step S203 in FIG.

[0047] Figure 4 yes Figure 1 Flow chart of step S103 in FIG.

[0048] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0049] Figure 6 yes Figure 4 Flow chart of step S402 in FIG.

[0050] Figure 7 yes Figure 1 Flow chart of step S104 in FIG.

[0051] Figure 8 is a structural schematic diagram of a voice-based portrait video generation device provided in an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of the hardware structure of the device provided in the embodiment of the present application;

[0053] Fig.10 It is a schematic diagram of the overall process of the voice-based portrait video generation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] First, some nouns involved in this application are analyzed:

[0058] Optical Flow Variation: It is the degree of change of the motion vector of the pixel in the optical flow field calculated between video frames. Optical flow is a technique used in computer vision to describe the motion of objects in consecutive images or video frames, and estimates the velocity field in the image by analyzing the motion of pixels. Optical flow variation is mainly used to measure the change in the intensity and direction of the motion of objects in the scene, and to identify significant areas of motion.

[0059] Speech Coding: In the field of speech-driven portrait videos, speech coding refers to compressing speech signals into a low-bitrate digital format while retaining the main features of the speech, so that the encoded speech data can be used to drive the generation of mouth shape, facial expressions and movements in portrait videos.

[0060] Speech-Driven Portrait Animation is a technology that converts speech input into synchronized facial animation, so that the virtual character or character in the video can naturally change facial expressions and postures according to the speech content. In the existing speech-driven portrait technology, digital human generation based on the Diffusion model is used. However, digital human generation based on the Diffusion model often cannot accurately match the speech and the picture, resulting in the generated video being not natural and realistic enough, and there are problems such as lip shape and speech being out of sync, and incoherent expressions. Therefore, how to accurately generate digital human videos with synchronized sound and picture has become an urgent problem to be solved.

[0061] Based on this, the embodiments of the present application provide a method and apparatus, a device and a storage medium for generating a portrait video based on voice, aiming to accurately generate a digital human video with synchronized audio and video.

[0062] The embodiments of the present application provide a method and apparatus for generating a portrait video based on speech, a device and a storage medium, which are specifically described through the following embodiments. First, the method for generating a portrait video based on speech in the embodiments of the present application is described.

[0063] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] The voice-based portrait video generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The voice-based portrait video generation method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a voice-based portrait video generation method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0068] Figure 1 is an optional flowchart of the method for generating a portrait video based on speech provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0069] Step S101, obtaining a sample portrait image and sample voice data of a target portrait;

[0070] Step S102, extracting portrait features from the sample portrait image to obtain sample portrait feature information;

[0071] Step S103, aligning the sample speech data with the sample portrait feature information to obtain speech-portrait alignment information; wherein the sample portrait feature information in the speech-portrait alignment information is semantically aligned with the sample speech data;

[0072] Step S104, optimizing parameters of a preset portrait video frame generation model according to the voice portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model;

[0073] Step S105, acquiring target voice data, and generating a portrait video frame using a target portrait video frame generation model, the target voice data, and sample portrait feature information to obtain a target face video frame;

[0074] Step S106, performing audio and video splicing on the target voice data and the target face video frame to obtain the target face voice video.

[0075] In steps S101 to S106 shown in the embodiment of the present application, a sample portrait image and sample voice data of the target portrait are obtained, and then portrait features are extracted from the sample portrait image to obtain sample portrait feature information, and the sample voice data and the sample portrait feature information are aligned, so as to correspond the portrait information and the picture information to obtain voice-portrait alignment information; further, parameters of a preset portrait video frame generation model are optimized according to the voice-portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model, and then target voice data is obtained, and portrait video frames are generated by using the target portrait video frame generation model, the target voice data and the sample portrait feature information to obtain a target face video frame, and finally the target voice data and the target face video frame are spliced ​​for audio and video to obtain a target portrait voice video, so as to achieve accurate generation of a digital human video with synchronized audio and video by corresponding the portrait and the voice.

[0076] In some embodiments, the preset portrait video frame generation model in step S104 is also used in the portrait feature extraction in step S102 and the alignment process in step S103.

[0077] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S203:

[0078] Step S201, obtaining the original portrait voice video of the target portrait;

[0079] Step S202, performing audio and video separation on the original portrait and voice video to obtain the original portrait image and sample voice data;

[0080] Step S203, performing portrait angle correction on the target portrait of the original portrait image to obtain a sample portrait image.

[0081] Steps S201 to S203 shown in the embodiment of the present application are to obtain the original portrait voice video of the target portrait, then perform audio and video separation on the original portrait voice video to obtain an original portrait image and sample voice data, and finally perform portrait angle correction on the target portrait in the original portrait image to obtain a sample portrait image, thereby separating the video frame and audio in the portrait voice video, and performing angle correction on the portrait in the video frame, thereby providing a data basis for the portrait and audio for the subsequent process.

[0082] In step S201 of some embodiments, the target person portrait refers to a facial image of a specific person to be analyzed, and the original person portrait voice video is a video containing only the target person portrait, wherein the target person portrait is narrating in the video. For example, if the target person portrait is a speaker, the original person portrait voice video refers to the video content in which only the speaker is speaking in the picture.

[0083] In step S202 of some embodiments, audio and video separation uses video processing technology to separate the audio and video content in the video into sample voice data and an original portrait image, where the original portrait image is a static image containing only the target portrait, and the sample voice data is the audio portion of the target portrait.

[0084] See also Figure 3 In some embodiments, step S203 may include but is not limited to steps S301 to S303:

[0085] Step S301, extracting key frames from the original portrait image to obtain a candidate portrait image;

[0086] Step S302, calculating the optical flow change of the candidate portrait image to obtain the optical flow change amount;

[0087] Step S303 , correcting the portrait angle of the target portrait in the original portrait image according to the optical flow variation to obtain a sample portrait image.

[0088] In steps S301 to S303 shown in the embodiment of the present application, a candidate portrait image is obtained by extracting key frames from the original portrait image, and then an optical flow change is calculated on the candidate portrait image to obtain an optical flow change amount. Finally, the portrait angle of the target portrait in the original portrait image is corrected according to the optical flow change amount to obtain a sample portrait image, thereby achieving portrait angle correction for the target portrait.

[0089] In step S301 of some embodiments, key frame extraction refers to selecting a representative or significantly changed frame from the original portrait image, and the candidate portrait image is an image frame obtained from the key frame extraction process for representing the target portrait. In one embodiment, key frame extraction is performed based on Scale-Invariant Feature Transform (SIFT), which has good detection performance for images with rotation properties and can have accurate detection accuracy after the face of the target portrait is rotated.

[0090] In step S302 of some embodiments, the optical flow change calculation is to measure the movement of pixels between video frames to obtain the optical flow change indicating the intensity and direction of the pixel movement between frames. In one embodiment, the Lucas-Kanade optical flow calculation method has good optical flow calculation accuracy for the face rotation of a small range of target portraits, and obtains the optical flow change.

[0091] In step S303 of some embodiments, the sample portrait image is an angle-corrected portrait image obtained after angle correction processing, specifically, the deflection of the head relative to the shoulder, for example, the shoulder is on the horizontal line in the image, the head is tilted 3 degrees to the right relative to the vertical line of the image, and the head is in a vertical position in the image after angle correction. The correction method is to analyze the optical flow variation, calculate the motion vector of each pixel in the target portrait image, and apply the corresponding geometric transformation to adjust the image so that the head angle is consistent with the vertical direction, thereby achieving portrait angle correction.

[0092] In step S102 of some embodiments, portrait feature extraction is to extract distinguishing and representative feature information from the sample portrait image. In one embodiment, ArcFace Network and E4E network are used for portrait extraction. The sample portrait image includes multiple portrait images. Each portrait image is input into the autoencoder to obtain a sample portrait code. Then the sample portrait code is input into ArcFace Network and E4E network, and the loss value is calculated to obtain the sample portrait code corresponding to the minimum loss value as the sample portrait feature information. The loss function is shown in formula (1):

[0093] IDLoss(x)=1- <R(x),R(G style (E4E(x)))>, (1),

[0094] Among them, IDLoss is the loss value, x is the sample portrait code, R is ArcFace Network, G style For the StyleGan network, <R(x),R(G style (E4E(x)))> is R(x),R(G style Cosine similarity between (E4E(x))).

[0095] See also Figure 4 In some embodiments, step S103 may include but is not limited to steps S401 to S403:

[0096] Step S401, extracting facial gesture features from sample portrait feature information to obtain facial gesture information;

[0097] Step S402, extracting face alignment nodes from the sample voice data to obtain voice portrait node information; wherein the voice portrait node information is used to indicate the facial expression and lip shape of the target portrait;

[0098] Step S403, performing node alignment according to the speech-portrait node information, the sample portrait feature information and the facial posture information to obtain speech-portrait alignment information.

[0099] In steps S401 to S403 shown in the embodiment of the present application, facial posture information is obtained by extracting facial posture features from sample portrait feature information, and then facial alignment node extraction is performed on sample voice data to obtain voice portrait node information. Finally, node alignment is performed based on the voice portrait node information, sample portrait feature information and facial posture information to obtain voice portrait alignment information, thereby matching the portrait and the voice.

[0100] See also Figure 5 In some embodiments, step S401 includes but is not limited to steps S501 to S503:

[0101] Step S501, extracting angle and posture features from sample portrait feature information to obtain angle and posture information; wherein the angle and posture information is used to indicate the posture of the target portrait at different angles;

[0102] Step S502, extracting facial expression and posture features from the sample portrait feature information to obtain facial expression and posture information; wherein the facial expression and posture information is used to indicate the posture of the target portrait under different expressions;

[0103] Step S503, concatenating the angle posture information and the expression posture information to obtain the facial posture information.

[0104] Steps S501 to S503 shown in the embodiment of the present application are to extract angle posture features from sample portrait feature information to obtain angle posture information, then extract expression posture features from the sample portrait feature information to obtain expression posture information, and finally splice the angle posture information and the expression posture information to obtain facial posture information, thereby achieving comprehensive feature extraction of the facial posture of the target portrait.

[0105] In step S501 of some embodiments, the angle posture feature extraction is to extract the yaw angle (Yaw) and pitch angle (Pitch) of the target portrait through the StyleFlow network. Yaw refers to the rotation angle of the head around the vertical axis, that is, the angle of turning the head left and right, and Pitch refers to the rotation angle of the head around the horizontal axis, that is, the angle of nodding up and down. The extraction method is to optimize StyleFlow through a loss function, and then extract the angle posture feature of the sample portrait feature information through the optimized StyleFlow network. The loss function is shown in formula (2):

[0106]

[0107] Among them, id_latent is the sample portrait feature information, Yaw and Pitch are the angle posture information, StyleFlow is the StyleFlow network, G is the StyleGan network, mask is the preset mask used to cover the facial features of the target portrait, X is the sample portrait image, and x i is the i-th portrait image generated by StyleGan, that is, G(StyleFlow(id_latent,pitch,yaw), y i is the i-th portrait image corresponding to the sample portrait image, i.e., X×mask, alexnet is a pre-trained deep convolutional neural network, L2 is the mean square error, and LPIPS is a loss function that measures the perceived similarity between two images. In this process, StyleFlow is optimized, and then the optimized StyleFlow network is used to extract the angle and posture features of the sample portrait feature information to obtain the angle and posture feature information.

[0108] In step S502 of some embodiments, the facial expression and posture feature extraction is to extract the features of facial expression, mouth shape, and facial features of the sample portrait image, and the extraction method is as shown in formula (3):

[0109]

[0110] Among them, pupil, eye, mouth and jaw are facial expression information. c Also the expression posture vector, a c Including pupil, eye, mouth and jaw, id c is the sample portrait feature information, mask pulil is the preset pupil mask, which is used to cover the non-pupil area. eye It is a preset eye mask used to cover the eyes. mouth The preset mouth mask is used to cover the non-mouth area. jaw is the preset chin mask, which is used to cover the non-chin area. G, i.e., the StyleGan network, is optimized by formula (3), and the pupil, eye, mouth, and jaw are extracted by the optimized StyleGan network, i.e., a c Expression posture information.

[0111] In step S503 of some embodiments, the angle posture information and the expression posture information are spliced ​​together to obtain facial posture information. In a real-time embodiment, the angle posture information is a vector with a length of 2, and the expression control information is a vector with a length of 35. After splicing, facial posture information with a length of 37 is obtained.

[0112] See also Figure 6 In some embodiments, step S402 includes but is not limited to steps S601 to S603:

[0113] Step S601, performing speech encoding on the sample speech data to obtain sample speech encoding information;

[0114] Step S602, extracting facial gesture tendency according to the sample speech coding information to obtain facial gesture tendency information;

[0115] Step S603: extracting face alignment nodes from the face posture tendency information to obtain voice portrait node information.

[0116] In steps S601 to S603 shown in the embodiment of the present application, sample speech data is speech encoded to obtain sample speech coding information, and then facial posture tendency extraction is performed based on the sample speech coding information to obtain facial posture tendency information, and finally facial alignment node extraction is performed on the facial posture tendency information to obtain speech portrait node information, thereby realizing the extraction of node information for facial control from speech.

[0117] In step S601 of some embodiments, speech coding refers to converting speech signals into digital form while retaining the main features of speech, so that the coded speech data can be used to drive the generation of mouth shape, facial expression and action of the portrait video. In one embodiment, speech coding is implemented by a pre-trained Hubert network, and the sample speech data is input into the Hubert network to obtain sample speech coding information.

[0118] In step S602 of some embodiments, facial posture tendency extraction is performed through a variational encoder, and facial posture tendency information from speech to facial key points is obtained by inputting sample speech coding information into the variational encoder. Facial key points are basic key points on the face that are not related to expressions and postures, and are used to characterize the basic motion trajectory of the human face, that is, the facial posture tendency information is the change in the motion trajectory of the face, for example, changes in eyebrows, changes in mouth shape, and changes in expressions.

[0119] In step S603 of some embodiments, the facial gesture tendency information is used to identify and analyze key points in the facial gesture tendency information, that is, to extract the key points of change from the facial gesture tendency information to obtain the voice portrait node information.

[0120] In step S403 of some embodiments, according to the voice portrait node information, the sample portrait feature information and the facial posture information are aligned through an offsetNet network, and the loss function of training the offsetNet is shown in formula (4):

[0121] Loss=L2(attributes,offsetNet(landmarks))+L2(G(latent,attributes,)G(latent,offsetNet(landmarks))) (4),

[0122] Among them, attributes is the facial posture information, offsetNet is the offsetNet network, landmarks is the voice portrait node information, and latent is the sample portrait feature information.

[0123] See also Figure 7 In some embodiments, step S104 may include but is not limited to steps S701 to S702:

[0124] Step S701, generating a portrait video frame by using a preset portrait video frame generation model and voice-portrait alignment information to obtain an original portrait video frame;

[0125] Step S702, optimizing parameters of a preset portrait video frame generation model using original portrait video frames and sample portrait images to obtain a target face image generation model.

[0126] In steps S701 to S702 shown in the embodiment of the present application, portrait video frames are generated by using a preset portrait video frame generation model and voice-portrait alignment information to obtain original portrait video frames, and then parameters of the preset portrait video frame generation model are optimized using the original portrait video frames and sample portrait images to obtain a target face image generation model, so that the target face image generation model can well perceive the sample portrait image and accurately generate a video frame image of the target portrait.

[0127] In step S701 of some embodiments, video frame generation is image generation, and the speech-portrait alignment information is input into a preset portrait video frame generation model, that is, a StyleGan network. StyleGan is used to generate portrait videos. StyleGan includes S space and W+ space. When the speech-portrait alignment information is input into the StyleGan network, a preliminary video frame is first generated through the W+ space, and then the preliminary video frame is input into the S space for further video frame generation to obtain a higher precision video frame, that is, the original portrait video frame.

[0128] In step S702 of some embodiments, parameter optimization is performed using the original portrait video frame and the sample portrait image, and the optimized loss function is shown in formula (5):

[0129] Loss=L2(G(S Space latent),X)+LPIPS(G(S Space latent),X) (5),

[0130] Among them, G (SSpace latent) is the output of the S space in StyleGan, that is, the original portrait video frame, and X is the sample portrait image. Through formula (5), the S space of StyleGan is optimized, thereby improving the quality of the image generated by StyleGan.

[0131] In step S105 of some embodiments, target speech data is acquired, and the speech data is variationally encoded through Hubert to obtain target speech coding information, and then the target speech coding information is subjected to facial posture tendency extraction to obtain target facial posture tendency information, and then the target facial posture tendency information is subjected to facial alignment node extraction to obtain target speech portrait node information, and then the target speech portrait node information and the sample portrait feature information are aligned to obtain target speech portrait alignment information, and then the target speech portrait alignment information is input into the target portrait video frame generation model to obtain the target face video frame.

[0132] In step S106 of some embodiments, in some embodiments, the target voice data and the target face video frame are spliced ​​together to obtain the target face voice video.

[0133] See also Fig.10In one embodiment, the original portrait voice video of the target portrait is first obtained, and then the original portrait voice video is split into audio and video to obtain the original portrait image and sample voice data, and then the portrait angle of the target portrait of the original portrait image is corrected by using the key frame-based optical flow prediction alignment to correct the deflection of the portrait relative to the shoulder to obtain the sample portrait image. Then each sample portrait image is input into the ArcFace Network, and the sample portrait image is input into the E4E network, and then the E4E network is input into the StyleGan network to obtain the output of the W+ space, and the cosine similarity between the W+ space output and the ArcFace Network is calculated, and the loss value is calculated, and the E4E network output corresponding to the minimum loss value is obtained to obtain id_latent, that is, the sample portrait feature information. After obtaining the sample portrait feature information, the sample portrait feature information is subjected to angle and posture extraction to obtain angle and posture information. Specifically, id_latent, pitch, yaw and a preset mask are input to StyleFlow, and the output of StyleFlow is input to the StyleGan network. StyleGan outputs the first intermediate image, and calculates the matrix dot product between the sample portrait image and the mask to obtain the second intermediate image. The loss value of the first intermediate image and the second intermediate image is calculated, and then the pitch and yaw parameters are optimized based on the loss value. After the optimization conditions are met, pitch and yaw, that is, angle and posture information, are obtained. Then, expression and posture information is obtained based on the sample portrait image and sample portrait features. Finally, the expression and posture information and the angle and posture information are spliced ​​together to obtain facial posture information, that is, Fig.10 The control vector shown. At the same time, the sample speech data is input into the Hubert network, and then the output of Hubert is input into the decoder to obtain the IDlandmark, that is, the speech portrait node information, and then the IDlandmark and facial posture information are aligned through the OffsetNet network to obtain the speech portrait alignment information. Finally, based on the speech portrait alignment information, it is input into the StyleGan network, that is, the preset portrait video frame generation model, and the S space of the StyleGan network is optimized to obtain the target portrait video frame generation model. Finally, based on the target speech data, the sample portrait feature information and the target portrait video frame generation model, the target face video frame is generated, and then the target speech data and the target face video frame are spliced ​​for audio and video to obtain the target portrait speech video.

[0134] See also Figure 8 The embodiment of the present application further provides a device for generating a portrait video based on speech, which can implement the above-mentioned method for generating a portrait video based on speech, and the device includes:

[0135] The sample acquisition module 801 is used to acquire sample portrait images and sample voice data of the target portrait;

[0136] Feature extraction module 802, used for extracting portrait features from sample portrait images to obtain sample portrait feature information;

[0137] The information alignment module 803 is used to align the sample speech data and the sample portrait feature information to obtain speech-portrait alignment information; wherein the sample portrait feature information in the speech-portrait alignment information is semantically aligned with the sample speech data;

[0138] A model optimization module 804 is used to optimize parameters of a preset portrait video frame generation model according to the voice portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model;

[0139] The portrait generation module 805 is used to obtain the target voice data, and generate the portrait video frame through the target portrait video frame generation model, the target voice data and the sample portrait feature information to obtain the target face video frame;

[0140] The audio and video splicing module 806 is used to perform audio and video splicing on the target voice data and the target face video frame to obtain the target face voice video.

[0141] The specific implementation of the voice-based portrait video generation device is basically the same as the specific implementation of the voice-based portrait video generation method described above, and will not be repeated here.

[0142] The embodiment of the present application also provides a device, the device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned voice-based portrait video generation method when executing the computer program. The device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0143] See also Fig. 9 , Fig. 9 The hardware structure of a device of another embodiment is illustrated, and the device includes:

[0144] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0145] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the voice-based portrait video generation method of the embodiment of this application;

[0146] Input / output interface 903, used to implement information input and output;

[0147] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0148] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0149] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0150] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned voice-based portrait video generation method is implemented.

[0151] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The embodiments of the present application provide a method for generating a portrait video based on speech, a device for generating a portrait video based on speech, a device and a storage medium. The method obtains a sample portrait image and sample speech data of a target portrait, then extracts portrait features from the sample portrait image to obtain sample portrait feature information, and aligns the sample speech data and the sample portrait feature information, thereby matching the portrait information with the picture information to obtain speech-portrait alignment information. Furthermore, the parameters of a preset portrait video frame generation model are optimized according to the speech-portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model, and then the target speech data is obtained, and the portrait video frame is generated by using the target portrait video frame generation model, the target speech data and the sample portrait feature information to obtain a target face video frame. Finally, the target speech data and the target face video frame are spliced ​​for audio and video to obtain a target portrait voice video, thereby matching the portrait with the speech to accurately generate a digital human video with synchronized audio and video.

[0153] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0154] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0155] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0157] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0158] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0159] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0162] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0163] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating a portrait voice video, characterized in that: The method comprises: Obtaining sample portrait images and sample voice data of the target person; Extracting portrait features from the sample portrait image to obtain sample portrait feature information; Aligning the sample speech data with the sample portrait feature information to obtain speech-portrait alignment information; wherein the sample portrait feature information in the speech-portrait alignment information is semantically aligned with the sample speech data; Optimizing parameters of a preset portrait video frame generation model according to the voice-portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model; Acquire target voice data, and generate a portrait video frame using the target portrait video frame generation model, the target voice data and the sample portrait feature information to obtain a target face video frame; The target voice data and the target face video frame are spliced ​​together to obtain the target face voice video.

2. The method according to claim 1, characterized in that The aligning the sample voice data with the sample portrait feature information to obtain voice-portrait alignment information includes: Extracting facial posture features from the sample portrait feature information to obtain facial posture information; Performing face alignment node extraction on the sample voice data to obtain voice portrait node information; wherein the voice portrait node information is used to indicate the expression and lip shape of the target portrait; Node alignment is performed according to the voice portrait node information, the sample portrait feature information and the facial posture information to obtain the voice portrait alignment information.

3. The method according to claim 2, characterized in that The step of extracting facial posture features from the sample portrait feature information to obtain facial posture information includes: Extracting angle and posture features from the sample portrait feature information to obtain angle and posture information; wherein the angle and posture information is used to indicate the posture of the target portrait at different angles; Extracting facial expression and posture features from the sample portrait feature information to obtain facial expression and posture information; wherein the facial expression and posture information is used to indicate the posture of the target portrait under different expressions; The angle posture information and the expression posture information are concatenated to obtain the facial posture information.

4. The method according to claim 2, characterized in that: The step of extracting face alignment nodes from the sample voice data to obtain voice portrait node information includes: Performing speech encoding on the sample speech data to obtain sample speech encoding information; Extracting facial gesture tendency according to the sample speech coding information to obtain facial gesture tendency information; Perform face alignment node extraction on the face posture tendency information to obtain the voice portrait node information.

5. The method according to claim 1, characterized in that The step of optimizing parameters of a preset portrait video frame generation model according to the voice portrait alignment information and the sample portrait image to obtain a target portrait video frame generation model includes: Generate a portrait video frame using a preset portrait video frame generation model and the voice-portrait alignment information to obtain an original portrait video frame; The parameters of the preset portrait video frame generation model are optimized through the original portrait video frame and the sample portrait image to obtain the target portrait video frame generation model.

6. The method according to claim 1, characterized in that The step of obtaining a sample portrait image and sample voice data of a target portrait includes: Obtaining the original portrait voice video of the target portrait; Performing audio and video separation on the original portrait voice video to obtain an original portrait image and sample voice data; The target portrait of the original portrait image is subjected to portrait angle correction to obtain the sample portrait image.

7. The method according to claim 6, characterized in that The step of performing portrait angle correction on the target portrait of the original portrait image to obtain the sample portrait image includes: Extracting key frames from the original portrait image to obtain a candidate portrait image; Calculating the optical flow change of the candidate portrait image to obtain the optical flow change amount; The target portrait of the original portrait image is subjected to portrait angle correction according to the optical flow variation to obtain the sample portrait image.

8. A device for generating a portrait video based on speech, characterized in that: The device comprises: A sample acquisition module, used to acquire sample portrait images and sample voice data of a target person; A feature extraction module is used for extracting portrait features from the sample portrait image to obtain sample portrait feature information; An information alignment module, used for aligning the sample speech data with the sample portrait feature information to obtain speech-portrait alignment information; wherein the sample portrait feature information in the speech-portrait alignment information is semantically aligned with the sample speech data; A model optimization module, used to optimize parameters of a preset portrait video frame generation model according to the voice portrait alignment information and the sample portrait image, to obtain a target portrait video frame generation model; A portrait generation module is used to obtain target voice data, and generate a portrait video frame through the target portrait video frame generation model, the target voice data and the sample portrait feature information to obtain a target face video frame; The audio and video splicing module is used to perform audio and video splicing on the target voice data and the target face video frame to obtain the target portrait voice video.

9. A device, characterized in that: The device includes a memory and a processor, the memory stores a computer program, and the processor implements the voice-based portrait video generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a portrait video based on speech according to any one of claims 1 to 7 is implemented.

Images