Digital human video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a digital human video generation method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models, augmented reality and the like, and can be applied to scenes such as digital human. According to the specific implementation scheme, a human skeleton feature point sequence, a human face sequence, a hand sequence and a lip shape sequence of a target person are generated according to a reference video and driving voice; generating a body sequence of the target person according to the human skeleton feature point sequence and the hand sequence; and according to the body sequence, the human face sequence and the lip-shaped sequence, generating a digital human video, the digital human being matched with the target person.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the technical fields of computer vision, deep learning, large models, augmented reality, etc., and can be applied to scenes such as digital humans, and specifically to a method, device, electronic device and storage medium for generating digital human videos. Background Art

[0002] Digital humans have been widely used in many industries, such as entertainment, education, and e-commerce. As industry demand and user experience gradually increase, the demand for digital human videos in various industries is also increasing. Summary of the invention

[0003] The present invention provides a method, device, electronic device and storage medium for generating a digital human video.

[0004] According to a first aspect of the present disclosure, a method for generating a digital human video is provided, comprising: generating a human skeleton feature point sequence, a human face sequence, a hand sequence and a lip shape sequence of a target person according to a reference video and a driving voice; generating a body sequence of the target person according to the human skeleton feature point sequence and the hand sequence; generating a digital human video according to the body sequence, the human face sequence and the lip shape sequence, wherein the digital human is matched with the target person.

[0005] According to a second aspect of the present disclosure, a digital human video generation device is provided, comprising: a sequence acquisition module, used to generate a human skeleton feature point sequence, a human face sequence, a hand sequence and a lip shape sequence of a target person according to a reference video and a driving voice; a body generation module, used to generate a body sequence of the target person according to the human skeleton feature point sequence and the hand sequence; a video generation module, used to generate a digital human video according to the body sequence, the human face sequence and the lip shape sequence, and the digital human matches the target person.

[0006] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0007] at least one processor; and

[0008] a memory communicatively connected to the at least one processor; wherein,

[0009] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0010] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0011] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0012] The digital human video generation method provided by the present disclosure can improve the generation quality of digital human videos and at the same time improve the production efficiency of digital human videos.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a flow chart of a method for generating a digital human video according to an embodiment of the present disclosure;

[0016] Figure 2 is a schematic diagram of the process of digital human skeleton driving according to an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of a process of digital human skeleton alignment according to an embodiment of the present disclosure;

[0018] Figure 4 is a structural schematic diagram of a digital human video generating device according to an embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of a scene for digital human video generation according to an embodiment of the present disclosure;

[0020] Figure 6 It is a structural diagram of an electronic device used to implement the method for generating a digital human video according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0022] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0023] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0024] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0025] Digital human: a character image generated by digital technology based on a video recorded by a real person. After collecting and analyzing the appearance, expression, and movements of the real person, these data are processed using technical means, and lip movements and body movements are arranged so that they can express and display actions according to the preset content. The digital human generated in this way can simulate the performance of a real person to a certain extent and can be used in various scenarios, such as virtual anchors, virtual customer service, etc. Its advantage is that it can retain certain real-life characteristics, making the audience feel more real and intimate.

[0026] In the related technologies, most of them can only edit lip movements or body movements separately, and only a few tools can choreograph and drive lip movements and body movements of digital humans at the same time. The solution of editing lip movements or body movements separately usually directly predicts the two-dimensional (Two Dimensional, 2D) key points or three-dimensional (Three Dimensional, 3D) model coefficients of the face / body through audio; while the solution that can edit both at the same time usually relies on the unified representation of the face and body, and learns the relationship between the audio and this representation through a neural network. Among them, the solution of editing lip movements separately can only edit the lips, and there will be inconsistencies between the body movements in the audio and the original reference video; and the solution of editing body movements separately is usually limited by the representation itself, and the expression ability of local areas that need to be driven accurately, such as the face and hands, is limited. In the prior art, for the solution of editing lip movements and body movements simultaneously, a unified three-dimensional representation of the face and body is usually used, and the relationship between the audio and the representation is a "many-to-many" relationship, which makes it difficult to learn the relationship between the audio and the representation, resulting in stiff and unnatural generated body movements and inaccurate lip movements.

[0027] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a method for generating digital human videos, which adopts a reconstruction model with head and body separation to ensure control accuracy, thereby improving not only the generation quality of digital human videos, but also the production efficiency of digital human videos.

[0028] The present disclosure provides a method for generating a digital human video. Figure 1 1 is a flow chart of a method for generating a digital human video according to an embodiment of the present disclosure. The method for generating a digital human video can be applied to a digital human video generating device. The digital human video generating device is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the server can be a cloud server or an ordinary server. For example, mobile devices include but are not limited to: mobile phones, tablet computers, and vehicle-mounted terminals. In some possible implementations, the method for generating a digital human video can also be implemented by a processor calling computer-readable instructions stored in a memory. For example, Figure 1 As shown, the digital human video generation method includes:

[0029] S101, generating a human skeleton feature point sequence, a human face sequence, a hand sequence, and a lip shape sequence of a target person according to a reference video and a driving voice;

[0030] S102, generating a body sequence of a target person according to a human skeleton feature point sequence and a hand sequence;

[0031] S103, generating a digital human video according to the body sequence, the human face sequence and the lip shape sequence; and matching the digital human with the target person.

[0032] In the disclosed embodiment, the reference video refers to the video material used as the base layer in the video editing process, that is, the original real-person recorded video. This video contains various information about the real person, such as appearance, expression, action, etc., and can be used as a basic template for digital human generation.

[0033] In the embodiments of the present disclosure, the driving voice refers to the voice signal used to drive or control the movements and expressions of the digital human. The driving voice can be used as input, and through a series of technical processing, the digital human can generate matching facial expressions, lip shapes, and body movements according to the content, rhythm, and intonation of the voice.

[0034] In the disclosed embodiment, human skeleton feature points refer to a series of points representing key parts of the human skeleton. Exemplarily, these human skeleton feature points may include parts such as the head, neck, shoulder, elbow, wrist, hip, knee, ankle, etc. The human skeleton feature point sequence refers to a series of pictures including human skeleton feature points generated based on the driving voice and the reference video, and each picture is used as a frame of the human skeleton feature point sequence. Exemplarily, the human body parts displayed by the target person in the video can be determined based on the reference video, and then the movements of each human body part are generated according to the driving voice to obtain human skeleton feature points, and then obtain human skeleton feature points, and then obtain human skeleton feature point sequence. In particular, the number of frames of the human skeleton feature point sequence is the same as the number of frames of the reference video, and further, the time length of the human skeleton feature point sequence is also the same as the time length of the reference video.

[0035] In the disclosed embodiment, a human face refers to a human face rendering image reconstructed in any frame of a reference video. A human face sequence refers to a series of human face rendering images reconstructed in each frame of a reference video. In particular, the number of frames of the human face sequence is the same as the number of frames of the reference video, and further, the time length of the human face sequence is also the same as the time length of the reference video.

[0036] In the disclosed embodiment, the hand refers to a hand rendering image reconstructed in any frame of a reference video. A hand sequence refers to a series of hand rendering images generated based on a driving voice and a reference video, and each hand rendering image serves as a frame of a hand sequence. Exemplarily, the shape of the hand can be generated based on a reference video, and then the hand movement can be generated by a driving voice, thereby obtaining a hand sequence. In particular, the number of frames of the hand sequence is the same as the number of frames of the reference video, and further, the time length of the hand sequence is also the same as the time length of the reference video.

[0037] In the disclosed embodiment, lip shape refers to a lip generation map obtained by rendering. A lip shape sequence refers to a series of lip generation maps generated based on a driving voice and a reference video, and each lip generation map serves as a frame of a lip shape sequence. Exemplarily, the shape of the lips can be generated based on a reference video, and then the lip movements can be generated by a driving voice, thereby obtaining a lip shape sequence. In particular, the number of frames of the lip shape sequence is the same as the number of frames of the reference video, and further, the time length of the lip shape sequence is also the same as the time length of the reference video.

[0038] In the disclosed embodiment, the body of the target person refers to a picture of all body parts of the target person shown in the reference video except the head, which may include the neck, shoulders, elbows, wrists, hips, knees, ankles and other parts. Exemplarily, the body map at the corresponding moment can be generated by combining and aligning the human skeleton feature points at any moment with the hand rendering at the same moment. A body sequence refers to a series of body maps generated based on the human skeleton feature points and the hand rendering, and each body map serves as a frame of the body sequence. In particular, the number of frames of the body sequence is the same as the number of frames of the reference video, and further, the time length of the body sequence is also the same as the time length of the reference video.

[0039] In the disclosed embodiment, the generated digital human video refers to the target person in the reference video being able to make corresponding lip movements and body movements according to the content of the driving voice. The face rendering image at any moment can be combined with the lip generation image at the same moment to generate a lip movement face image at the corresponding moment, and then each frame of the lip movement face image is combined with the body image in the body sequence at the corresponding moment to generate the digital human video. In particular, the number of frames of the digital human video is the same as the number of frames of the reference video, and further, the time length of the digital human video is also the same as the time length of the reference video.

[0040] In the disclosed embodiment, the hand shape and lip shape of the target person in the video can be first determined based on the reference video, and the body parts displayed by the target person can be determined and the face of the target person can be reconstructed to obtain a face rendering. Then, the movements of the hands, lips and other body parts are determined by driving voice. Based on the hand shape and hand movement, a hand sequence is generated; based on the lip shape and lip movement, a lip shape sequence is generated; based on the movement of the body parts, a human skeleton feature point sequence is generated; based on the face rendering, a human face sequence is generated.

[0041] Furthermore, each frame in the human skeleton feature point sequence and the hand sequence can be combined and aligned to generate a body sequence of the target person.

[0042] Finally, the digital human video can be generated by combining and aligning each frame of the face rendering with the hand rendering at the same moment, and then combining and aligning it with the body image in the body sequence at the corresponding moment.

[0043] The technical solution of the disclosed embodiment can capture and reconstruct the hand shape, lip shape and facial features of the target person respectively, so that the digital human video can highly restore the appearance and movements of the real person and provide a realistic visual experience. At the same time, the movements of the hands, lips and other body parts are synchronized by driving voice to ensure that the digital person's lip shape, gestures and body movements are naturally coordinated when speaking, thereby enhancing the realism and interactivity of the video. In addition, digital humans with different features and movements can be generated according to different reference videos and driving voices to meet personalized and diversified needs.

[0044] In some embodiments, a human skeleton feature point sequence, a human face sequence, a hand sequence and a lip shape sequence of a target person are generated according to a reference video and a driving voice, including: fitting a human skeleton sequence of the target person according to the reference video and the driving voice; projecting the human skeleton sequence onto a two-dimensional plane to generate a human skeleton feature point sequence; generating a hand sequence according to the human skeleton sequence; based on a human body reconstruction model, performing three-dimensional reconstruction of the face of the target person in the reference video to obtain a human face sequence; and based on an audio lip movement model, generating a lip shape sequence that matches the driving voice.

[0045] In the disclosed embodiments, the human skeleton refers to the joints and their connection relationships that form the basis of human movement and structure. These joints and connection relationships together define the dynamic posture and shape of the human body in three-dimensional space. Exemplarily, the human skeleton may include: joints and connection relationships. Among them, the human skeleton is composed of a series of key joints, which represent important parts of the human body, such as the head, shoulders, elbows, wrists, hips, knees, etc. The positions and mutual relationships of the joints in three-dimensional space determine the posture and movement of the human body.

[0046] In the disclosed embodiment, the process of fitting the human skeleton sequence of the target person can first use an articulated three-dimensional human model (Skinned Multi-Person Linear Model eXpressive, SMPL-X) to model the target person in the reference video based on the kinematic skeleton and human dynamics to obtain a 3D human skeleton model, and then use a recurrent neural network (Recurrent Neural Network, RNN) to generate 3D skeletal motion data from the audio sequence, and finally use a conditional generative adversarial network (Conditional Generative Adversarial Network, CGAN) to synthesize the 3D human skeleton model and the 3D skeletal motion data and output multiple frames of pictures in time units to produce a human skeleton sequence rich in motion details. The above is only an exemplary description, and is not intended to limit all possible situations of fitting the human skeleton sequence of the target person, but it is not exhaustive here.

[0047] In the disclosed embodiment, the process of generating a sequence of human skeleton feature points can first determine the human body structure and bone ratio of each frame in the human skeleton sequence, calculate the two-dimensional posture diagram of each frame in the human skeleton sequence, and determine the anchor point, and calculate the transformed coordinates of other joint points in the two-dimensional posture diagram of the human body based on the anchor point according to the bone ratio, and project the human skeleton sequence onto a two-dimensional plane based on the coordinates, thereby generating a sequence of human skeleton feature points. The above is only an exemplary description, and is not intended to limit all possible situations for generating a sequence of human skeleton feature points, but it is not exhaustive here.

[0048] In the disclosed embodiment, the process of generating a hand sequence according to a human skeleton sequence can first identify key points from a human skeleton sequence through a motion recognition model; then the key points of the hand can be extracted from the identified key points, wherein the hand key points include positions such as the wrist, palm, and finger joints; then 3D modeling software or a special animation tool can be used to simulate the movement of the hand based on the hand key point data using skeleton animation techniques such as inverse dynamics or forward dynamics; and then the simulated hand skeleton animation is bound to a 3D hand model, wherein the 3D hand model contains the detailed geometry of the hand. Finally, a 3D rendering engine can be used to generate a sequence of renderings of the hand according to the skeleton animation. The above is only an exemplary description and is not intended to limit all possible situations for generating a hand sequence, except that it is not exhaustive here.

[0049] In the disclosed embodiment, based on the human body reconstruction model, the face of the target person in the reference video is three-dimensionally reconstructed to obtain the process of the human face sequence. The face of the target person in the reference video can be first extracted using video segmentation technology as the input for subsequent reconstruction; then, a human body reconstruction model such as the Multi-Level Pixel-Aligned Implicit Function for High-Res01ution 3D Human Digitization (PIFuHD) model is used to reconstruct a high-resolution face and generate face data; then, a human body reconstruction model such as a three-dimensional face morphable model (Morphable Model For The Synthesis Of 3D Faces, 3DMM) can be used to generate a 3D face model based on the face data; finally, a face sequence can be generated based on the generated 3D face model through animation technology. The above is only an exemplary description and is not intended to limit all possible situations for obtaining a human face sequence, but it is not exhaustive here.

[0050] In the disclosed embodiment, based on the audio lip movement model, the process of generating a lip shape sequence matching the driving voice can first extract acoustic features such as phonemes and intonations in the driving audio through methods such as speech recognition models or spectrogram analysis; then, based on the acoustic features, an audio lip movement model such as a recurrent neural network (RNN) or a convolutional neural network (CNN) can be used to predict the lip shape and obtain the lip shape sequence. The above is only an exemplary description and is not intended to limit all possible situations for generating a lip shape sequence, but it is not exhaustive here.

[0051] In this way, by generating a sequence of human skeleton feature points, human face sequence, hand sequence and lip shape sequence that matches the driving voice, the realism of the digital human can be enhanced. At the same time, the use of artificial intelligence algorithms and deep learning technology simplifies the production process of digital humans. Users can quickly start the digital human training process and generate highly realistic digital human images without having a professional technical background. In addition, by combining 3D human posture reconstruction technology, it can effectively solve the positioning errors and posture detail processing problems caused by the lack of depth information and posture diversity, and improve the accuracy of posture reconstruction.

[0052] In some embodiments, a human skeleton sequence of a target person is fitted based on a reference video and a driving voice, including: based on a human body reconstruction model, obtaining a shape information sequence and a position information sequence of the target person from a reference video; based on a voice-driven model, predicting a posture information sequence of the target person that matches the driving voice; and fitting a human skeleton sequence based on the shape information sequence, the position information sequence, and the posture information sequence.

[0053] In the disclosed embodiment, based on the human body reconstruction model, the process of obtaining the shape information sequence and position information sequence of the target person from the reference video can first use the human body reconstruction model based on SMPL-X to restore the 3D human body model of the target person from the reference video; then, the shape parameters and position parameters can be extracted from the 3D human body model; finally, each frame in the reference video is extracted to obtain the shape information sequence and position information sequence of the target person. The above is only an exemplary description and is not intended to limit all possible situations for obtaining the shape information sequence and the position information sequence, but it is not exhaustive here.

[0054] In the disclosed embodiment, based on the voice-driven model, the process of predicting the posture information sequence of the target person that matches the driving voice can first use the voice-driven model to predict the posture of the human body based on the driving voice; then, extract the posture information from any frame of the human body posture; finally, extract each frame of the human body posture to obtain the posture information sequence of the target person. The above is only an exemplary description and is not intended to limit all possible situations for predicting the posture information sequence, but it is not exhaustive here.

[0055] In the disclosed embodiment, the process of fitting the human skeleton sequence according to the shape information sequence, the position information sequence and the posture information sequence can be implemented using a human reconstruction model based on SMPL-X. Exemplarily, the SMPL-X model can be loaded and initialized first; then the shape parameters, position parameters and posture parameters are input, and the update function of the SMPL-X model is called to calculate the vertices and joint positions: finally, the human skeleton is refitted according to the calculated vertices and joint positions. In particular, the fitting results can be iterated, and the posture and body shape parameters can be adjusted until a satisfactory fitting effect is achieved. The above is only an exemplary description, and is not intended to limit all possible situations for fitting the human skeleton sequence, but it is not exhaustive here.

[0056] In this way, by combining the shape and position information of the target person in the reference video and the posture information predicted by the voice-driven model, more realistic and natural movements can be generated, the realism of the virtual person can be enhanced, and specific postures can be added when keywords appear, making human body movements more expressive and variable, avoiding the video content from appearing monotonous and repetitive, and increasing the richness and attractiveness of the video content. In addition, by synchronizing the digital person's movements with the driving voice, it can provide users with a more smooth and natural interactive experience.

[0057] In some embodiments, a body sequence of a target person is generated based on a human skeleton feature point sequence and a hand sequence, including: performing a first alignment based on wrist feature points and elbow feature points to obtain a first affine result; performing a second alignment based on the wrist feature points and the first affine result to obtain a second affine result; and generating a body sequence based on the first affine result, the second affine result, the human skeleton feature point sequence and the hand sequence.

[0058] In the disclosed embodiment, the first alignment is performed according to the wrist feature points and the elbow feature points to obtain the first affine result. First, a deep learning model such as an open source computer vision library (Open Source Computer Vision Library, OpenCV) or a traditional computer vision algorithm can be used to process the human skeleton feature point sequence and the hand sequence, and the wrist feature points and elbow feature points of the human skeleton feature point sequence and the hand sequence are detected respectively; then the wrist feature points and elbow feature points are matched with the wrist and elbow of the target person in the reference video using the feature point detection algorithm; then the affine transformation matrix is ​​calculated using the calculation homography matrix function in OpenCV; finally, the calculated affine transformation matrix is ​​used to transform the human skeleton feature point sequence and the hand sequence into the coordinate system of the reference video through the perspective transformation function, so as to realize the alignment of the elbow feature points and the elbow, and the alignment result is used as the first affine result. The above is only an exemplary explanation, and is not a limitation of all possible situations for obtaining the first affine result, but it is not exhaustive here.

[0059] In the disclosed embodiment, the second alignment is performed based on the wrist feature point and the first affine result to obtain the second affine result. The feature point detection can be performed again based on the first affine result to obtain the wrist feature point; then the feature point detection algorithm is used to correspond the wrist feature point with the wrist of the target person in the reference video; the affine transformation matrix is ​​calculated using the calculation homography matrix function in OpenCV; finally, the calculated affine transformation matrix is ​​used to transform the first affine result into the coordinate system of the reference video through the perspective transformation function to achieve alignment between the wrist feature point and the wrist, and the alignment result is used as the second affine result. The above is only an exemplary explanation and is not intended to limit all possible situations for obtaining the second affine result, but it is not exhaustive here.

[0060] In the disclosed embodiment, the process of generating the body sequence according to the first affine result, the second affine result, the human skeleton feature point sequence and the hand sequence can first use the first affine result and the second affine result to perform two perspective transformations on the target person in the reference video through the perspective transformation function in OpenCV; then map the human skeleton feature point sequence and the hand sequence back to the coordinate system of each frame in the reference video, and construct a complete body posture to obtain the body sequence. The above is only an exemplary description and is not intended to limit all possible situations for generating the body sequence, but it is not exhaustive here.

[0061] In this way, the affine deformation in the image can be simulated and compensated more accurately through two affine alignments, and the algorithm's adaptability to scenes with large affine deformation can be improved, thereby achieving higher-precision image matching and enhancing the robustness of image matching. By combining the human skeleton feature point sequence and the hand sequence, human body movements can be captured and reconstructed more accurately, which can improve the efficiency of the algorithm.

[0062] In some embodiments, a first alignment is performed based on wrist feature points and elbow feature points to obtain a first affine result, including: determining any key frame based on a human skeleton feature point sequence; extracting a reference frame corresponding to the key frame from a reference video based on a key point detection model; extracting wrist feature points and elbow feature points based on the key frame and the reference frame; calculating a first affine transformation matrix based on the wrist feature points and the elbow feature points; and aligning the elbow feature points based on the first affine transformation matrix to obtain a first affine result.

[0063] In the disclosed embodiment, the process of determining any key frame according to the human skeleton feature point sequence can analyze the action change according to the human skeleton feature point sequence, and determine the frame with the largest action change or the start frame and end frame of the action as the key frame. The above is only an exemplary description and is not intended to limit all possible situations for determining the key frame, but it is not exhaustive here. In particular, the increase in the number of key frames can improve the calculation accuracy of the alignment affine matrix, thereby effectively improving the effect of alignment.

[0064] In the disclosed embodiment, based on the key point detection model, the process of extracting the reference frame corresponding to the key frame from the reference video can first use the media pipeline tool or the open source human posture recognition project and other tools to extract the reference skeleton key point sequence from the reference video; then, the deep learning method such as the double-stream convolutional network can be used to extract the motion angle feature and the angle-based derivative feature in combination with the reference skeleton key point sequence, and then the concept of the motion feature matrix is ​​introduced through the perceptual hash algorithm to describe and map the video frame; then, the visual content distance feature between the video frame and the key frame can be calculated, and the reference frame corresponding to the key frame can be extracted from the reference video through the motion feature. The above is only an exemplary description, not a limitation of all possible situations for extracting the reference frame, but it is not exhaustive here. In particular, the increase in the number of reference frames can improve the calculation accuracy of the alignment affine matrix, thereby effectively improving the alignment effect.

[0065] In the disclosed embodiment, according to the key frame and the reference frame, the process of extracting the wrist feature points and the elbow feature points can firstly use the media pipeline tool or the open source human posture recognition project and other tools to extract the key frame skeleton key points and the reference frame skeleton key points from the key frame and the reference frame respectively; finally, by the position of the wrist and elbow in the human skeleton structure, the feature points of the wrist and elbow are identified and extracted from the key frame skeleton key points and the reference frame skeleton key points. In particular, based on the human body structure, the wrist feature points include the left wrist key points and the right wrist key points extracted from the key frame, and the left wrist reference points and the right wrist reference points extracted from the reference frame; similarly, the elbow feature points include the left elbow key points and the right elbow key points extracted from the key frame, and the left elbow reference points and the right elbow reference points extracted from the reference frame. The above is only an exemplary description, and is not intended to limit all possible situations for extracting wrist feature points and elbow feature points, but it is not exhaustive here.

[0066] In the disclosed embodiment, according to the wrist feature points and the elbow feature points, the process of calculating the first affine transformation matrix can first collect the coordinates of the eight points of the left wrist key point, the right wrist key point, the left wrist reference point, the right wrist reference point, the left elbow key point, the right elbow key point, the left elbow reference point and the right elbow reference point from the key frame and the reference frame respectively; then, the left wrist key point is matched with the left wrist reference point, the right wrist key point is matched with the right wrist reference point, the left elbow key point is matched with the left elbow reference point, and the right elbow key point is matched with the right elbow reference point, and the eight points are matched to obtain four groups of correspondences; then, each group of correspondences is taken as a row to construct the first matrix; finally, the coordinates of the eight points are substituted into the first matrix for calculation to solve the first affine transformation matrix. The above is only an exemplary description, and is not a limitation of all possible situations for calculating the first affine transformation matrix, but it is not exhaustive here.

[0067] In the disclosed embodiment, according to the first affine transformation matrix, the elbow feature points are aligned to obtain the first affine result. The first affine transformation matrix can be applied using the perspective transformation function of OpenCV or similar functions of other image processing libraries. The function can transform the image so that the left elbow key point is aligned with the left elbow reference point, and the right elbow key point is aligned with the right elbow reference point; finally, the aligned image is used as the first affine result. The above is only an exemplary explanation and is not intended to limit all possible situations for obtaining the first affine result. It is just that it is not exhaustive here. In particular, considering that the errors in 3D human body reconstruction will be reflected on the 2D skeleton with obvious bias, it is necessary to align the two elbows separately.

[0068] In this way, by aligning feature points, the robustness of image registration is enhanced, making it more applicable to images with different lighting and grayscale attributes. At the same time, compared with the method of searching for matching point pairs in the entire image, the alignment method based on feature points greatly reduces the amount of calculation and avoids the problem of mismatching caused by local extreme values ​​that may be trapped in the search process. In addition, by aligning feature maps of different scales, low-level detail information and high-level semantic information can be combined to better restore image details and improve image clarity and quality. Moreover, the action recognition method based on skeleton points, combined with a high-quality posture detector, can often achieve good recognition results with less data and has strong generalization ability.

[0069] In some embodiments, a second alignment is performed based on the wrist feature points and the first affine result to obtain a second affine result, including: calculating a second affine transformation matrix based on the wrist feature points; and aligning the wrist feature points in the first affine result based on the second affine transformation matrix to obtain a second affine result.

[0070] In the disclosed embodiment, the coordinates of the four points, namely, the left wrist key point, the right wrist key point, the left wrist reference point, and the right wrist reference point, can be first collected from the first affine result and the reference frame; then, the left wrist key point is matched with the left wrist reference point, and the right wrist key point is matched with the right wrist reference point to obtain two sets of correspondences; then, each set of correspondences is taken as a row to construct a second matrix; finally, the coordinates of the four points are substituted into the second matrix for calculation to solve the second affine transformation matrix. The above is only an exemplary explanation and is not intended to limit all possible situations for calculating the second affine transformation matrix, but it is not exhaustive here.

[0071] In the disclosed embodiment, according to the second affine transformation matrix, the wrist feature points in the first affine result are aligned to obtain the second affine result. The second affine transformation matrix can be applied using the perspective transformation function of OpenCV or similar functions of other image processing libraries. The function can transform the image so that the left wrist key points are aligned with the left wrist reference points, and the right wrist key points are aligned with the right wrist reference points; finally, the aligned image is used as the second affine result. The above is only an exemplary explanation and is not intended to limit all possible situations for obtaining the second affine result. It is just that it is not exhaustive here. In particular, considering that the errors in 3D human body reconstruction will be reflected on the 2D skeleton with obvious bias, the two wrists can be aligned separately.

[0072] In this way, through two consecutive affine transformations, the wrist feature points can be aligned more accurately, thereby improving the alignment accuracy of the wrist part in the overall image or video frame. At the same time, the second alignment can fine-tune the first alignment result to enhance the adaptability to changes in wrist feature points in different scenes and conditions. In addition, in video processing, continuous affine transformation can optimize the tracking of feature points, especially in dynamic parts such as the wrist, which helps to maintain the continuity and stability of tracking, making the visual effect of the synthesized image or video more natural and coherent, and reducing the visual disharmony caused by improper alignment.

[0073] In some embodiments, a body sequence is generated based on a first affine result, a second affine result, a human skeleton feature point sequence and a hand sequence, including: generating an affine alignment sequence based on the first affine result and the second affine result; aligning the hand sequence with the affine alignment sequence based on the human skeleton projection sequence and the affine alignment sequence to obtain a body sequence.

[0074] In the disclosed embodiment, the process of generating an affine alignment sequence according to the first affine result and the second affine result can first obtain the first affine transformation matrix and the second affine transformation matrix from the first affine result and the second affine result; then the two affine transformations can be combined into one affine transformation matrix by matrix multiplication; then, the perspective transformation function of OpenCV or similar functions of other image processing libraries can be used to apply the combined affine transformation matrix so that the points in the image are aligned according to the specified affine transformation; finally, the above steps are applied to each frame to generate a series of aligned images, i.e., an affine alignment sequence. The above is only an exemplary description and is not intended to limit all possible situations for generating an affine alignment sequence, but it is not exhaustive here.

[0075] In the disclosed embodiment, according to the human skeleton projection sequence and the affine alignment sequence, the hand sequence is aligned with the affine alignment sequence to obtain the process of the body sequence. First, the human skeleton projection sequence and the affine alignment sequence are used to determine the position of the human hand in any frame in the affine alignment sequence, and the position of the hand sequence is adjusted based on the position, and then the hand sequence and the affine alignment sequence are aligned; finally, the above steps are applied to each frame to generate a series of aligned images, i.e., the body sequence. The above is only an exemplary description and is not intended to limit all possible situations for obtaining the body sequence, but it is not exhaustive here.

[0076] In this way, by combining the first affine result and the second affine result, the human skeleton feature point sequence and the hand sequence can be aligned more accurately, thereby improving the accuracy of skeleton-based action recognition. At the same time, by considering the affine transformation of spatial and temporal factors, it can better handle changes in different perspectives, skeleton sizes, and action amplitudes, and enhance the robustness to these spatial factors.

[0077] In some embodiments, an affine alignment sequence is generated based on the first affine result and the second affine result, including: performing affine transformation alignment on any frame of the human skeleton feature point sequence based on the first affine result and the second affine result; traversing the human skeleton feature point sequence and performing affine transformation alignment on each frame to obtain an affine alignment sequence.

[0078] In the disclosed embodiment, the process of affine transformation alignment of any frame of the human skeleton feature point sequence according to the first affine result and the second affine result can first obtain the first affine transformation matrix and the second affine transformation matrix from the first affine result and the second affine result; then, the first affine transformation matrix can be used to perform the first affine transformation alignment on any frame of the human skeleton feature point sequence; finally, the second affine transformation matrix can be used to perform the second affine transformation alignment on the frame after the first affine transformation alignment. The above is only an exemplary description, and is not intended to limit all possible situations for affine transformation alignment of any frame of the human skeleton feature point sequence, but it is not exhaustive here.

[0079] In the disclosed embodiment, the process of traversing the human skeleton feature point sequence and performing affine transformation alignment on each frame to obtain the affine alignment sequence can first repeat the affine transformation alignment step twice for each frame of the entire human skeleton feature point sequence to ensure that each frame is aligned by two affine transformations; finally, each frame image sequence that has been aligned by affine transformation can be saved or output to form an affine alignment sequence. The above is only an exemplary description and is not intended to limit all possible situations for obtaining the affine alignment sequence, but it is not exhaustive here.

[0080] In this way, the alignment method based on skeleton pose estimation can better distinguish the severely overlapping human instances in the image and improve the accuracy of human instance segmentation. At the same time, by traversing the sequence of human skeleton feature points and performing affine transformation alignment on each frame, an affine alignment sequence is obtained, which enhances the robustness of the alignment process and enables it to adapt to different human poses and movements.

[0081] In some embodiments, according to the human skeleton projection sequence and the affine alignment sequence, the hand sequence is aligned with the affine alignment sequence to obtain a body sequence, including: calculating a third affine transformation matrix according to the human skeleton projection sequence and the affine alignment sequence; according to the third affine transformation matrix, aligning the hand sequence to the corresponding wrist feature points in the affine alignment sequence to obtain a body sequence.

[0082] In the disclosed embodiment, according to the human skeleton projection sequence and the affine alignment sequence, the process of calculating the third affine transformation matrix can first determine the key points from the human skeleton projection sequence and extract the coordinates of the points, wherein these key points correspond to the key points in the affine alignment sequence; then, the calculation affine transformation matrix function of OpenCV can be used to calculate the third calculation affine transformation matrix according to the coordinates of the corresponding points. The above is only an exemplary description and is not intended to limit all possible situations for calculating the third affine transformation matrix, but it is not exhaustive here.

[0083] In the disclosed embodiment, according to the third affine transformation matrix, the hand sequence is aligned to the corresponding wrist feature points in the affine alignment sequence to obtain the body sequence. The corresponding wrist feature points can be first extracted from the hand sequence and the affine alignment sequence; then the perspective transformation function of OpenCV or similar functions of other image processing libraries can be used to apply the calculated third affine transformation matrix so that the wrist feature points in the hand sequence are aligned to the corresponding wrist feature points in the affine alignment sequence; then, the above steps can be repeated for each frame in the hand sequence and the affine alignment sequence to ensure that each frame is aligned by affine transformation; finally, the hand sequence aligned by affine transformation can be combined with the body parts in the affine alignment sequence to form a complete body sequence. The above is only an exemplary description and is not intended to limit all possible situations for obtaining the body sequence, but it is not exhaustive here.

[0084] In this way, through three precise affine transformation alignments, human skeleton information can be fully utilized, especially in small sample action recognition, to further improve matching accuracy. At the same time, by combining the human skeleton projection sequence and the affine alignment sequence, the generalization ability in different scenarios can be improved, so that it can show good results on a variety of action recognition data sets. In addition, by calculating the third affine transformation matrix and applying it to each frame, the hand sequence and body sequence can be quickly aligned, improving the efficiency of feature matching.

[0085] In some embodiments, generating a digital human video according to a body sequence, a human face sequence and a lip shape sequence includes: generating a digital human face sequence according to a lip shape sequence and a human face sequence; generating a digital human video according to a body sequence and a digital human face sequence.

[0086] In the disclosed embodiment, the process of generating a digital human face sequence based on a lip shape sequence and a human face sequence can first adopt a domain adaptation method based on adversarial training to train a post-processing network for human face movements to bridge the domain difference between a large-scale lip reading recognition dataset and a target person video, and ensure that the lip shape movement matches the facial features of the target person; then, the lips in any frame in the lip shape sequence are used to replace the lips in the corresponding frame of the human face sequence; finally, the above steps can be repeated for each frame in the lip sequence and the human face sequence to ensure that each frame is replaced, and a digital human face sequence can be generated. In particular, the photo realism of the synthesized face can be improved by post-processing, especially for the synthesis result of the lower half of the face, it is necessary to set a separate network to optimize the naturalness of the facial synthesis of the lower half of the face. The above is only an exemplary description, and is not intended to limit all possible situations for generating a digital human face sequence, but it is not exhaustive here.

[0087] In the disclosed embodiment, the process of generating a digital human video based on a body sequence and a digital human face sequence can first combine the body sequence and the digital human face sequence to obtain a digital human; finally, the digital human and the background can be rendered and synthesized to generate a final digital human video. The above is only an exemplary description and is not intended to limit all possible situations for generating digital human videos, but it is not exhaustive here.

[0088] In this way, by combining the body sequence, the human face sequence and the lip sequence, a complete full-body motion effect synchronized with the audio can be generated, thereby improving the naturalness and realism of the motion generation.

[0089] In some embodiments, a digital human video is generated based on a body sequence and a digital human face sequence, including: using a three-dimensional rendering to represent a human face sequence; using a three-dimensional rendering to represent a hand sequence in the body sequence; using a two-dimensional rendering to represent a human skeleton feature point sequence in the body sequence; and generating a digital human video based on the rendered body sequence and the digital human face sequence.

[0090] In the disclosed embodiment, the process of representing a human face sequence in the form of a three-dimensional rendering may firstly use the techniques of face generation, rendering and three-dimensional reconstruction in the prior art to construct a three-dimensional face model; then obtain a three-dimensional rendering based on the three-dimensional face model; finally, the three-dimensional rendering of each frame may be synthesized into a sequence to form a complete human face sequence. In particular, other methods in the prior art may also be used to obtain a three-dimensional face model or a three-dimensional rendering. The above is only an exemplary description and is not intended to limit all possible situations for representing a human face sequence, but it is not exhaustive here.

[0091] In the disclosed embodiment, the process of representing the hand sequence in the body sequence in the form of a three-dimensional rendering can firstly use the human hand generation, rendering and three-dimensional reconstruction technology in the prior art to build a three-dimensional human hand model; then obtain a three-dimensional rendering based on the three-dimensional human hand model; finally, the three-dimensional rendering of each frame can be synthesized into a sequence to form a complete hand sequence. In particular, other methods in the prior art can also be used to obtain a three-dimensional human hand model or a three-dimensional rendering. The above is only an exemplary description and is not intended to limit all possible situations for representing a hand sequence, but it is not exhaustive here.

[0092] In the disclosed embodiment, the process of using a two-dimensional rendering to represent a human skeleton feature point sequence in a body sequence can first extract the skeleton points and connecting lines of any frame from the human skeleton feature point sequence; then, computer graphics technology can be used to render the detected skeleton points and connecting lines into a two-dimensional image to form a two-dimensional rendering of the human skeleton; finally, the two-dimensional rendering of each frame can be synthesized into a sequence to form a complete human skeleton feature point sequence. The above is only an exemplary description and is not intended to limit all possible situations for representing a human skeleton feature point sequence, but it is not exhaustive here.

[0093] In the disclosed embodiment, based on the rendered body sequence and digital human face sequence, the process of generating a digital human video can first use the body sequence and digital human face sequence as constraints, and use existing technology to render and synthesize the virtual character and the background to generate the final digital human video. In particular, the generated video can also be post-processed to improve its quality and viewing experience. The above is only an exemplary description and is not intended to limit all possible situations for generating digital human videos, but it is not exhaustive here.

[0094] In this way, through the representation of the rendering, a digital human video with high realism and detail can be generated, which can capture the complex geometric shapes and surface details of the face and hands, making the digital human video more realistic. By using the three-dimensional face dynamic information to guide the generation of face videos, high-fidelity, identity-invariant face videos can be generated, improving the video effect.

[0095] Figure 2 The schematic diagram of the digital human skeleton driving process is shown in FIG. Figure 2 As shown, the process includes three steps, S201, S202 and S203. Among them, S201, extracting reference skeletons based on reference videos; specifically, 201a, obtaining reference videos of a certain length; 201b, extracting reference skeleton sequences of target persons from the reference videos. S202, performing human skeleton alignment operations based on voice drive and reference skeleton sequences. S203, generating skeleton sequences that can be used to edit videos based on the aligned human skeleton sequences.

[0096] Figure 3 A schematic diagram of the process of digital human skeleton alignment is shown, Figure 3 As shown, the process includes four steps S301, S302, S303 and S304.

[0097] Among them, S301, 3D optimization fitting and 2D projection. Specifically, 301a, 3D human body reconstruction is performed on the reference video to obtain shape parameters and position parameters; 301b, posture parameters are obtained by using driving voice; 301c, posture parameters, shape parameters and position parameters are fitted and optimized; 301d, hand rendering and 2D projection are performed on the 3D fitting result to obtain hand rendering image and 2D projection key points.

[0098] Among them, S302, reference frame alignment. Specifically, 302a, find a pair of similar frames in the 2D projection skeleton and the reference video 2D detection skeleton to form a pair of reference frames. 302b, align the 2D projection reference frame and the reference video reference frame at the elbow and wrist to obtain matrix M1; 302c, based on 302b, align the wrist at two points to obtain matrix M2. Through two alignments, the positions of the transformed human key points are more reasonable and will not be offset.

[0099] Among them, S303, video sequence skeleton alignment. Specifically, 303a, align the reference frame of the 2D projected skeleton and the human skeleton detected by the reference video to obtain M1 and M2. 303b, align the elbow and wrist four points of each frame in the video through M1 to ensure that the positions of the elbow and wrist points are reasonable. 303c, align the wrist two points of each frame in the video again through M2 to ensure that the position of the wrist point is more reasonable. 303d, based on the alignment of 303b and 303c, obtain the human skeleton alignment sequence.

[0100] Among them, S304, hand rendering alignment. Specifically, 304a, wrist alignment is performed according to the 2D projected skeleton, the aligned human skeleton and the original hand rendering; 304b, the hand rendering obtained in S301 is aligned to the new wrist point to generate an alignment sequence of the digital human video.

[0101] It should be understood that Figure 2 and Figure 3 The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figure 2 and Figure 3 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0102] The present disclosure provides a digital human video generation device, such as Figure 4 As shown, the device may include: a sequence acquisition module 401, used to generate a human skeleton feature point sequence, a human face sequence, a hand sequence and a lip shape sequence of a target person according to a reference video and a driving voice; a body generation module 402, used to generate a body sequence of the target person according to the human skeleton feature point sequence and the hand sequence; a video generation module 403, used to generate a digital human video according to the body sequence, the human face sequence and the lip shape sequence.

[0103] In some embodiments, the sequence acquisition module 401 includes: a skeleton fitting submodule, which is used to fit the human skeleton sequence of the target person according to the reference video and the driving voice; a skeleton generation submodule, which is used to project the human skeleton sequence onto a two-dimensional plane to generate a human skeleton feature point sequence; a hand generation submodule, which is used to generate a hand sequence according to the human skeleton sequence; a face generation submodule, which is used to perform three-dimensional reconstruction of the face of the target person in the reference video based on the human body reconstruction model to obtain a human face sequence; and a lip shape generation submodule, which is used to generate a lip shape sequence that matches the driving voice based on the audio lip movement model.

[0104] In some embodiments, the skeleton fitting submodule is used to: obtain the shape information sequence and position information sequence of the target person from the reference video based on the human body reconstruction model; predict the posture information sequence of the target person that matches the driving voice based on the voice-driven model; and fit the human skeleton sequence according to the shape information sequence, position information sequence and posture information sequence.

[0105] In some embodiments, the body generation module 402 includes: a first alignment submodule, used to perform a first alignment based on the wrist feature points and the elbow feature points to obtain a first affine result; a second alignment submodule, used to perform a second alignment based on the wrist feature points and the first affine result to obtain a second affine result; a body generation submodule, used to generate a body sequence based on the first affine result, the second affine result, the human skeleton feature point sequence and the hand sequence.

[0106] In some embodiments, the first alignment submodule is used to: determine any key frame based on a sequence of human skeleton feature points; extract a reference frame corresponding to the key frame from a reference video based on a key point detection model; extract wrist feature points and elbow feature points based on the key frame and the reference frame; calculate a first affine transformation matrix based on the wrist feature points and the elbow feature points; align the elbow feature points according to the first affine transformation matrix to obtain a first affine result.

[0107] In some embodiments, the second alignment submodule is used to: calculate a second affine transformation matrix according to the wrist feature points; and align the wrist feature points in the first affine result according to the second affine transformation matrix to obtain a second affine result.

[0108] In some embodiments, the body generation submodule is used to: generate an affine alignment sequence according to the first affine result and the second affine result; align the hand sequence with the affine alignment sequence according to the human skeleton projection sequence and the affine alignment sequence to obtain a body sequence.

[0109] In some embodiments, the body generation submodule is further used to: perform affine transformation alignment on any frame of the human skeleton feature point sequence according to the first affine result and the second affine result; traverse the human skeleton feature point sequence, perform affine transformation alignment on each frame, and obtain an affine alignment sequence.

[0110] In some embodiments, the body generation submodule is further used to: calculate a third affine transformation matrix based on the human skeleton projection sequence and the affine alignment sequence; and align the hand sequence to the corresponding wrist feature points in the affine alignment sequence according to the third affine transformation matrix to obtain a body sequence.

[0111] In some embodiments, the video generation module 403 includes: a face generation submodule for generating a digital human face sequence according to a lip shape sequence and a human face sequence; and a video generation submodule for generating a digital human video according to a body sequence and a digital human face sequence.

[0112] In some embodiments, the video generation submodule is used to: represent a human face sequence in the form of a three-dimensional rendering; represent a hand sequence in a body sequence in the form of a three-dimensional rendering; represent a human skeleton feature point sequence in a body sequence in the form of a two-dimensional rendering; and generate a digital human video based on the rendered body sequence and the digital human face sequence.

[0113] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0114] The digital human video generation device in the disclosed embodiment can capture and reconstruct the hand shape, lip shape and facial features of the target person respectively, so that the digital human video can highly restore the appearance and movements of the real person and provide a realistic visual experience. At the same time, the movements of the hands, lips and other body parts are synchronized by driving voice to ensure that the digital person's mouth shape, gestures and body movements are naturally coordinated when speaking, thereby enhancing the realism and interactivity of the video. In addition, digital humans with different features and movements can be generated according to different reference videos and driving voices to meet personalized and diversified needs.

[0115] The present disclosure provides a schematic diagram of a scene for generating a digital human video. Figure 5 shown.

[0116] As mentioned above, the digital human video generation method provided by the embodiment of the present disclosure is applied to an electronic device. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can be installed on an autonomous driving vehicle or connected to an autonomous driving vehicle.

[0117] Specifically, the electronic device can perform the following operations:

[0118] Generate a human skeleton feature point sequence, a human face sequence, a hand sequence, and a lip shape sequence of the target person according to the reference video and the driving voice;

[0119] Generate the target person's body sequence based on the human skeleton feature point sequence and hand sequence;

[0120] A digital human video is generated based on the body sequence, human face sequence and lip shape sequence; the digital human is matched with the target person.

[0121] It should be understood that Figure 5 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 5 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0122] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0124] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0125] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0126] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0127] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a digital human video generation method. For example, in some embodiments, the digital human video generation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the digital human video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the digital human video generation method in any other appropriate manner (eg, by means of firmware).

[0128] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0133] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a digital human video, comprising: Generate a human skeleton feature point sequence, a human face sequence, a hand sequence, and a lip shape sequence of the target person according to the reference video and the driving voice; Generating a body sequence of the target person according to the human skeleton feature point sequence and the hand sequence; A digital human video is generated according to the body sequence, the human face sequence and the lip shape sequence; and the digital human is matched with the target person.

2. The method according to claim 1, wherein: The method of generating a human skeleton feature point sequence, a human face sequence, a hand sequence and a lip shape sequence of a target person according to a reference video and a driving voice comprises: Fitting a human skeleton sequence of the target person according to the reference video and the driving voice; Projecting the human skeleton sequence onto a two-dimensional plane to generate the human skeleton feature point sequence; Generate the hand sequence according to the human skeleton sequence; Based on the human body reconstruction model, three-dimensionally reconstruct the face of the target person in the reference video to obtain the human face sequence; Based on the audio lip movement model, the lip shape sequence matching the driving speech is generated.

3. The method according to claim 2, wherein: The step of fitting a human skeleton sequence of the target person according to the reference video and the driving voice comprises: Based on the human body reconstruction model, obtaining a shape information sequence and a position information sequence of the target person from the reference video; Based on the voice-driven model, predicting a posture information sequence of the target person that matches the driving voice; The human skeleton sequence is fitted according to the shape information sequence, the position information sequence and the posture information sequence.

4. The method according to claim 1, wherein: The step of generating a body sequence of the target person according to the human skeleton feature point sequence and the hand sequence comprises: Perform the first alignment based on the wrist feature points and the elbow feature points to obtain the first affine result; Performing a second alignment based on the wrist feature point and the first affine result to obtain a second affine result; A body sequence is generated according to the first affine result, the second affine result, the human skeleton feature point sequence and the hand sequence.

5. The method according to claim 4, wherein: The first alignment is performed according to the wrist feature points and the elbow feature points to obtain a first affine result, including: Determine any key frame according to the human skeleton feature point sequence; Extracting a reference frame corresponding to the key frame from the reference video based on a key point detection model; Extracting the wrist feature points and the elbow feature points according to the key frame and the reference frame; Calculating a first affine transformation matrix according to the wrist feature points and the elbow feature points; The elbow feature points are aligned according to the first affine transformation matrix to obtain the first affine result.

6. The method according to claim 4, wherein: The performing a second alignment according to the wrist feature point and the first affine result to obtain a second affine result includes: Calculating a second affine transformation matrix according to the wrist feature points; The wrist feature points in the first affine result are aligned according to the second affine transformation matrix to obtain the second affine result.

7. The method according to claim 4, wherein: The step of generating a body sequence according to the first affine result, the second affine result, the human skeleton feature point sequence and the hand sequence comprises: Generate an affine alignment sequence according to the first affine result and the second affine result; According to the human skeleton projection sequence and the affine alignment sequence, the hand sequence is aligned with the affine alignment sequence to obtain the body sequence.

8. The method according to claim 7, wherein: Generating an affine alignment sequence according to the first affine result and the second affine result includes: According to the first affine result and the second affine result, performing affine transformation alignment on any frame of the human skeleton feature point sequence; The human skeleton feature point sequence is traversed, and each frame is aligned by affine transformation to obtain the affine alignment sequence.

9. The method according to claim 7, wherein: The step of aligning the hand sequence with the affine alignment sequence according to the human skeleton projection sequence and the affine alignment sequence to obtain the body sequence comprises: Calculating a third affine transformation matrix according to the human skeleton projection sequence and the affine alignment sequence; According to the third affine transformation matrix, the hand sequence is aligned to the corresponding wrist feature points in the affine alignment sequence to obtain the body sequence.

10. The method according to claim 1, wherein: The step of generating a digital human video according to the body sequence, the human face sequence and the lip shape sequence comprises: Generate a digital human face sequence according to the lip shape sequence and the human face sequence; A digital human video is generated according to the body sequence and the digital human face sequence.

11. The method according to claim 10, wherein: The step of generating a digital human video according to the body sequence and the digital human face sequence comprises: Representing the human face sequence in the form of a three-dimensional rendering; Representing the hand sequence in the body sequence in the form of a three-dimensional rendering; Representing the human skeleton feature point sequence in the body sequence in the form of a two-dimensional rendering; The digital human video is generated based on the rendered body sequence and the digital human face sequence.

12. A digital human video generation device, comprising: A sequence acquisition module is used to generate a human skeleton feature point sequence, a human face sequence, a hand sequence and a lip shape sequence of a target person according to a reference video and a driving voice; A body generation module, used to generate a body sequence of the target person according to the human skeleton feature point sequence and the hand sequence; The video generation module is used to generate a digital human video according to the body sequence, the human face sequence and the lip shape sequence; the digital human is matched with the target person.

13. The device according to claim 12, wherein: The sequence acquisition module comprises: A skeleton fitting submodule, used for fitting a human skeleton sequence of the target person according to the reference video and the driving voice; A skeleton generation submodule, used for projecting the human skeleton sequence onto a two-dimensional plane to generate the human skeleton feature point sequence; A hand generation submodule, used for generating the hand sequence according to the human skeleton sequence; A face generation submodule, used for performing three-dimensional reconstruction of the face of the target person in the reference video based on a human body reconstruction model to obtain the human face sequence; The lip shape generation submodule is used to generate the lip shape sequence matching the driving speech based on the audio lip movement model.

14. The device according to claim 13, wherein: The skeleton fitting submodule is used to: Based on the human body reconstruction model, obtaining a shape information sequence and a position information sequence of the target person from the reference video; Based on the voice-driven model, predicting a posture information sequence of the target person that matches the driving voice; The human skeleton sequence is fitted according to the shape information sequence, the position information sequence and the posture information sequence.

15. The device according to claim 12, wherein: The body generation module comprises: A first alignment submodule, used for performing a first alignment according to the wrist feature points and the elbow feature points to obtain a first affine result; A second alignment submodule, used for performing a second alignment according to the wrist feature point and the first affine result to obtain a second affine result; The body generation submodule is used to generate a body sequence according to the first affine result, the second affine result, the human skeleton feature point sequence and the hand sequence.

16. The device according to claim 15, wherein: The first alignment submodule is used to: Determine any key frame according to the human skeleton feature point sequence; Extracting a reference frame corresponding to the key frame from the reference video based on a key point detection model; Extracting the wrist feature points and the elbow feature points according to the key frame and the reference frame; Calculating a first affine transformation matrix according to the wrist feature points and the elbow feature points; The elbow feature points are aligned according to the first affine transformation matrix to obtain the first affine result.

17. The device according to claim 15, wherein: The second alignment submodule is used for: Calculating a second affine transformation matrix according to the wrist feature points; The wrist feature points in the first affine result are aligned according to the second affine transformation matrix to obtain the second affine result.

18. The device according to claim 15, wherein: The body generation submodule is used to: Generate an affine alignment sequence according to the first affine result and the second affine result; According to the human skeleton projection sequence and the affine alignment sequence, the hand sequence is aligned with the affine alignment sequence to obtain the body sequence.

19. The device according to claim 18, wherein: The body generation submodule is further used for: According to the first affine result and the second affine result, performing affine transformation alignment on any frame of the human skeleton feature point sequence; The human skeleton feature point sequence is traversed, and each frame is aligned by affine transformation to obtain the affine alignment sequence.

20. The device according to claim 18, wherein The body generation submodule is further used for: Calculating a third affine transformation matrix according to the human skeleton projection sequence and the affine alignment sequence; According to the third affine transformation matrix, the hand sequence is aligned to the corresponding wrist feature points in the affine alignment sequence to obtain the body sequence.

21. The device according to claim 12, wherein: The video generation module comprises: A face generation submodule, used for generating a digital face sequence according to the lip shape sequence and the human face sequence; The video generation submodule is used to generate a digital human video according to the body sequence and the digital human face sequence.

22. The device according to claim 21, wherein The video generation submodule is used for: Representing the human face sequence in the form of a three-dimensional rendering; Representing the hand sequence in the body sequence in the form of a three-dimensional rendering; Representing the human skeleton feature point sequence in the body sequence in the form of a two-dimensional rendering; The digital human video is generated based on the rendered body sequence and the digital human face sequence.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are for causing a computer to perform a method according to any one of claims 1-11.

25. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 11 when executed by a processor.

Images