Digital human driving method, device, equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a digital human driving method, device, equipment and storage medium, the method comprising: generating digital human audio and phoneme sequence information corresponding to a digital human to be driven according to a language to be broadcasted; determining the digital human body language corresponding to the digital human to be driven based on the phoneme sequence information and an action state machine; generating a digital human image sequence according to the digital human body language; generating a digital human video based on the digital human audio and digital human image sequence; and presenting the digital human video through the digital human to be driven. Since the present invention generates a digital human video based on the digital human audio and digital human body language corresponding to the digital human to be driven, and queries the digital human video through the digital human to be driven, the technical problem that driving a digital human through a real-person performance driving method in the prior art requires a large amount of human resources and is relatively costly is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a digital human driving method, device, equipment and storage medium. Background Art

[0002] With the development of the Internet and digital media technology, the application of virtual digital human technology in various industries has begun to emerge. In the application fields of customer service robots, digital employees, virtual anchors, smart assistants, etc., virtual digital humans are playing an increasingly important role as interactive media.

[0003] At present, digital humans are mostly used in live broadcasts or anchor reports, and their driving methods can include real-life performance driving. The real-life performance driving method first requires the completion of digital human image modeling and key point binding, and then through the camera and motion capture equipment, the real-life model's movements and expressions are reproduced in the digital human image. However, this method usually requires the support of image models, original artists, modelers, animators, and motion capture technicians in the modeling and driving process, resulting in a large amount of human resources and high costs for driving digital humans in this way.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a digital human driving method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that driving a digital human through real-person performance driving requires a lot of human resources and is costly.

[0006] To achieve the above-mentioned purpose, the present invention provides a digital human driving method, the digital human driving method comprising:

[0007] Generate digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcasted;

[0008] Determine the digital human body language corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine;

[0009] generating a digital human image sequence according to the digital human body language;

[0010] Generate a digital human video based on the digital human audio and the digital human image sequence;

[0011] The digital human video is presented through the digital human to be driven.

[0012] Optionally, the step of determining the body language of the digital human corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine includes:

[0013] Determine a facial key point sequence based on the phoneme sequence information and a preset digital human viseme mapping table, wherein the facial key point sequence is a sequence composed of facial key feature points of the digital human to be driven;

[0014] Determine a body key point sequence based on the digital human behavior description information and the action state machine, wherein the body key point sequence is a sequence composed of the body key feature points of the digital human to be driven;

[0015] Acquire an overall key point sequence according to the face key point sequence and the body key point sequence;

[0016] The digital human body language corresponding to the digital human to be driven is determined according to the overall key point sequence.

[0017] Optionally, the step of determining a facial key point sequence based on the phoneme sequence information and a preset digital human viseme mapping table includes:

[0018] Determine a facial key point offset sequence based on the phoneme sequence information and a preset digital human viseme mapping table;

[0019] Performing temporal smoothing processing on the facial key point offset sequence to obtain a smoothed facial key point offset sequence;

[0020] A facial key point sequence is determined based on the smoothed facial key point offset sequence and the neutral facial key points of the digital human.

[0021] Optionally, the step of determining a body key point sequence based on the digital human behavior description information and the action state machine includes:

[0022] Determine the body key point offset sequence based on the digital human behavior description information and action state machine;

[0023] Performing temporal smoothing processing on the body key point offset sequence to obtain a smoothed body key point offset sequence;

[0024] A body key point sequence is determined based on the body key point offset sequence and the neutral body key points of the digital human.

[0025] Optionally, the step of presenting the digital human video by the digital human to be driven includes:

[0026] Converting the digital human audio and the digital human video into an audio stream and a video stream respectively through a target stream pusher;

[0027] Synchronize the audio stream and the video stream according to the target timestamp to obtain a synchronized audio and video stream;

[0028] The synchronized audio and video streams are pushed to a target server, and the synchronized audio and video streams are presented through the target server and the digital human to be driven.

[0029] Optionally, before the step of converting the digital human audio and the digital human video into an audio stream and a video stream respectively by the target stream pusher, the method further includes:

[0030] Segmenting the digital human video to determine static video segments in the digital human video;

[0031] Building a target corpus based on the static video clips;

[0032] Upon receiving an inference request, generating a fused digital human video based on the dynamic video clip and the static video clip in the target corpus;

[0033] The step of converting the digital human audio and the digital human video into an audio stream and a video stream respectively by a target stream pusher comprises:

[0034] The digital human audio and the fused digital human video are converted into an audio stream and a video stream respectively through a target stream pusher.

[0035] Optionally, before the step of generating the digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcasted, the step further includes:

[0036] Constructing state node action behaviors corresponding to an action state machine by a human posture estimation method, wherein the action state machine is a model for controlling the actions of a digital human to be driven;

[0037] Constructing the inter-node transition action behavior corresponding to the action state machine;

[0038] Behavior modeling is performed on the digital human to be driven based on the state node action behavior and the inter-node transition action behavior.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also proposes a digital human driving device, the device comprising:

[0040] A speech generation module, used to generate digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcasted;

[0041] A body language determination module, used for determining the body language of the digital human corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine;

[0042] An image sequence generation module, used for generating a digital human image sequence according to the digital human body language;

[0043] A video generation module, used for generating a digital human video based on the digital human audio and the digital human image sequence;

[0044] The video presentation module is used to present the digital human video through the digital human to be driven.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also proposes a digital human driving device, which includes: a memory, a processor, and a digital human driver program stored in the memory and executable on the processor, wherein the digital human driver program is configured to implement the steps of the digital human driving method described above.

[0046] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a digital human driver program is stored, and when the digital human driver program is executed by a processor, the steps of the digital human driving method described above are implemented.

[0047] The present invention discloses the following methods: generating digital human audio and phoneme sequence information corresponding to a digital human to be driven according to a language to be broadcasted; determining the digital human body language corresponding to the digital human to be driven based on the phoneme sequence information and an action state machine; generating a digital human image sequence according to the digital human body language; generating a digital human video based on the digital human audio and digital human image sequence; and presenting the digital human video through the digital human to be driven. Compared with the prior art in which digital human driving is performed by a real-person performance driving method, the modeling and driving process usually requires the support of a large number of technical personnel and is costly. Since the present invention generates a digital human video based on the digital human audio and digital human body language corresponding to the digital human to be driven, and queries the digital human video through the digital human to be driven, the technical problem that driving a digital human by a real-person performance driving method in the prior art requires a large amount of human resources and is costly is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the structure of a digital human driving device in a hardware operating environment involved in an embodiment of the present invention;

[0049] Figure 2 This is a flow chart of the first embodiment of the digital human driving method of the present invention;

[0050] Figure 3 A schematic diagram of the process of generating digital human audio in the first embodiment of the digital human driving method of the present invention;

[0051] Figure 4 A schematic diagram of the process of driving a digital human in the digital human driving method of the present invention;

[0052] Figure 5 The overall framework diagram of the digital human behavior modeling in the digital human driving method of the present invention;

[0053] Figure 6 This is a flow chart of a second embodiment of a digital human driving method according to the present invention;

[0054] Figure 7 This is a schematic diagram of the process of generating the body language of a digital human in the second embodiment of the digital human driving method of the present invention;

[0055] Figure 8 It is a flow chart of the third embodiment of the digital human driving method of the present invention;

[0056] Fig. 9 This is a schematic diagram of the flow of RTMP protocol streaming in the third embodiment of the digital human driving method of the present invention;

[0057] Fig.10 This is a structural diagram of the digital human video streaming presentation in the third embodiment of the digital human driving method of the present invention;

[0058] Fig.11 This is a diagram showing some fields of an inference message in the third embodiment of the digital human driving method of the present invention;

[0059] Fig.12 This is a structural block diagram of the first embodiment of the digital human driving device of the present invention.

[0060] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0062] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a digital human driving device in the hardware operating environment involved in an embodiment of the present invention.

[0063] like Figure 1As shown, the digital human driving device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0064] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the digital human driving device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0065] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a digital human driver.

[0066] exist Figure 1 In the digital human driving device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the digital human driving device of the present invention can be set in the digital human driving device, and the digital human driving device calls the digital human driving program stored in the memory 1005 through the processor 1001, and executes the digital human driving method provided by the embodiment of the present invention.

[0067] The embodiment of the present invention provides a digital human driving method, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the digital human driving method of the present invention.

[0068] In this embodiment, the digital human driving method includes the following steps:

[0069] Step S10: Generate digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcasted.

[0070] It should be noted that the execution subject of the method of this embodiment can be a digital human driving device for real-time driving of a digital human, such as a mobile phone, a tablet computer, a personal computer, etc., or other digital human driving systems that can achieve the same or similar functions and include the digital human driving device. Here, the digital human driving method provided in this embodiment and the following embodiments is specifically described with a digital human driving system (hereinafter referred to as the system). Among them, a digital human refers to a virtual character that exists on a display device and has a digital appearance.

[0071] It should be noted that the above-mentioned language to be broadcasted can be any natural language that needs to be broadcasted by the digital human. In this embodiment, the language type of the language to be broadcasted can include but is not limited to English, Chinese, etc.

[0072] It can be understood that the above-mentioned digital human to be driven can be the digital human that needs to be driven in this solution; accordingly, the above-mentioned digital human audio can be the audio used to drive the digital human to be driven; the phoneme sequence information can be the sequence information composed of the rhythmic boundaries in the language to be broadcast and the initials, finals and tones in the pinyin.

[0073] In this embodiment, refer to Figure 3 , Figure 3 FIG. 1 is a flow chart of generating digital human audio in the first embodiment of the digital human driving method of the present invention. Figure 3 As shown, the system can n Input to a deep learning neural network (such as Figure 3 The speech synthesis network in the example is used to generate the digital human audio (i.e., the target audio Audio) corresponding to the digital human to be driven through the speech synthesis network. dst ) and phoneme sequence information {ph1, ph2, ..., ph n}, wherein the deep learning neural network can be a VITS deep neural network (Very Deep Iterative Strategy).

[0074] Step S20: determining the digital human body language corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine.

[0075] It should be understood that the above action state machine is a commonly used single-chip microcomputer programming concept, which is usually used in control systems, communication protocols, user interfaces, etc. The state machine can abstract the state of the system and the transition between states into a state diagram, thereby simplifying the design and implementation of the program. In practical applications, the action state machine usually includes elements such as state, transition, event, and action.

[0076] It should be noted that the above-mentioned digital human body language refers to the digital human's body language, including: facial expressions, hand gestures, body postures, etc. of the digital human, which are not limited in this embodiment.

[0077] Step S30: generating a digital human image sequence according to the digital human body language.

[0078] It is understandable that the above digital human image sequence can be used as an image sequence representing the body language of the digital human. In this embodiment, the corresponding digital human image sequence can be generated by a trained deep network model.

[0079] Step S40: Generate a digital human video based on the digital human audio and the digital human image sequence.

[0080] It should be noted that the above-mentioned digital human video may be a video showing information such as the digital human's voice, lip shape, expression, and movement.

[0081] Step S50: presenting the digital human video through the digital human to be driven.

[0082] It should be noted that, in this embodiment, the system can use streaming technology to enable the digital human to be driven to present the generated digital human video in real time, so as to achieve driving of the digital human to be driven.

[0083] In a specific implementation, this solution may be provided with a natural language synthesis digital human voice module, a natural language synthesis digital human voice module and an audio-visual linkage digital human video generation module. The functions of each module refer to Figure 4 , Figure 4 FIG. 1 is a schematic diagram of the process of driving a digital human in the digital human driving method of the present invention. Figure 4 As shown, this solution can convert the natural language T to be broadcast n Input to the natural language synthesis digital human voice module, so that the natural language synthesis digital human voice module generates digital human audio dst and phoneme sequence information {ph1, ph2, ..., ph n Then, the natural language synthesis digital human voice module can be input by the phoneme sequence {ph1, ph2, ..., ph n The facial posture model is obtained, and combined with the body posture model obtained by the action state machine, the two are fused to obtain the body language accompanying action T b Finally, the audio-visual digital human video generation module can combine body language with action b And Digital Human Audio dst , generate digital human video Video dst .

[0084] Furthermore, before step S10, the method further includes:

[0085] The state node action behavior corresponding to the action state machine is constructed by a human posture estimation method, and the action state machine is a model for controlling the action of the digital human to be driven; the inter-node transition action behavior corresponding to the action state machine is constructed; and the behavior modeling of the digital human to be driven is performed based on the state node action behavior and the inter-node transition action behavior.

[0086] It should be noted that the above-mentioned human body posture estimation method can be a method of estimating the position and posture of a human body by using computer vision technology.

[0087] It should be understood that the above-mentioned state node behavior can be a node action behavior corresponding to a state element in an action state machine. Correspondingly, the inter-node transition action behavior can be a transition action behavior corresponding to a transfer element between nodes in an action state machine.

[0088] In this embodiment, refer to Figure 5 , Figure 5 FIG. 1 is a general framework diagram of the digital human behavior modeling in the digital human driving method of the present invention. Figure 5 As shown, first the system can take any original video used for digital human behavior modeling src Converted to the corresponding image sequence {I1, I2, ..., I n}, then the system can model the node actions and transition actions of the digital human based on the two-dimensional posture model and the three-dimensional posture model respectively, among which, the node action behavior construction based on the two-dimensional posture can be implemented based on the DWPose model; the node action behavior construction based on the three-dimensional posture model can be implemented using the HybrIK model; the transition action behavior construction based on the two-dimensional posture can be implemented based on the LERP model; the node action behavior construction based on the three-dimensional posture model can be implemented using the SLERP model.

[0089] It should be noted that DWPose is a model based on RTMPose, and uses a two-stage distillation algorithm to make it lightweight and improve its reasoning speed. The RTMPose model is a real-time multi-person key point detection algorithm for human posture estimation. In practical applications, the DWPose model can achieve effective human body skeleton posture prediction through a two-stage distillation method:

[0090] The goal of the first stage of distillation in the DWPose model is to make the student network learn the F t and T i The feature distillation uses a 1×1 convolution layer to map the student features to the teacher network with the same number of channels, and uses the MSE loss to calculate the student's feature Fs and teacher characteristics F t The distance between them, the feature distillation loss can be expressed as:

[0091]

[0092] For logits distillation, DWPose makes improvements based on RTMPose and abandons the calculation of KL divergence using distillation weight masks. The loss function can be expressed as:

[0093]

[0094]

[0095] Where N is the number of samples in a batch, K is the number of key points, and L is the length of the positioning box. n,k is a target weight mask used to distinguish invisible key points, V i is the value of the tag.

[0096] In addition, DWPose designed a weight decay strategy so that the weights of feature loss and logits loss in the loss function decrease over time. The final loss function of the student network is as follows:

[0097]

[0098]

[0099] In the second stage, the trained student model is self-learned to fix the human skeleton and fine-tune the head to adjust the feature distribution so that the head can achieve better performance. The following formula represents the loss function of the second stage and the overall loss function, where the loss function of the second stage only considers the logits loss, and γ is a hyperparameter representing the loss scale.

[0100]

[0101]

[0102] In addition, the node action behavior based on the 3D posture model is constructed using HybrIK. HybrIK is a neural network combined with inverse kinematics, which solves the problem of unrealistic human structure generated in 3D human posture estimation. It can extract the 3D human skeleton from the image and convert it into the SMPL parameterized human posture model for representation.

[0103] Specifically, HybrIK first uses a CNN to predict the preliminary 3D key points P of the human body, SMPL shape parameters β, twist rotation angle Φ and initial posture T, and inputs them into the HybrIK model that combines inverse dynamics and deep neural networks to infer the SMPL posture parameters θ. Finally, the obtained β and θ parameters are input into SMPL through forward dynamics and regression methods to obtain the final 3D key points Q. This method forms a closed loop between 3D key point estimation and 3D mesh estimation.

[0104] It should be noted that inverse kinematics is a mathematical process of calculating the relative rotation R based on the body joint position P, and there is no unique solution. The solution proposed by HybrIK is to decompose the bone rotation into the swinging part R sw and the torsion part R tw , expressed by the following formula, where and Respectively expressed as template vector and target vector:

[0105]

[0106] The swing part of the bone rotation R sw It can be obtained through analytical calculation and is expressed by the following formula: for The skew-symmetric matrix is Represented as a 3×3 feature matrix, and α are given vectors and The rotation axis and rotation angle.

[0107]

[0108]

[0109]

[0110] Among them, the torsional part of the bone rotation R tw Through neural network prediction, R tw The representation method is shown in the formula below, which needs to be learned through a neural network.

[0111] In the calculation of the loss function, the 3D key point estimation loss, the torsion angle estimation loss and the SMPL parameter loss are combined. The 3D key point estimation uses ResNet as the backbone, adopts 3 deconvolution layers, and adopts the l1 loss for estimating the pose position, where P k Represents the GT joint coordinates.

[0112]

[0113] In the estimation of the torsion angle, a two-dimensional vector (C φk , S φk ) to represent φ k To avoid the discontinuity problem, the l2 loss is adopted, where represents the GT torsion angle.

[0114]

[0115] The loss calculation of SMPL's shape parameter β and posture parameter θ is shown in the formula below.

[0116]

[0117]

[0118] In summary, the overall loss function can be expressed as the following formula, where μ1, μ2, and μ3 represent the weights of each loss term:

[0119]

[0120] In addition, the system can also construct the transfer conditions of the action state machine according to the digital human behavior description B and the state transition table STT, and use the action state machine to complete the conversion and transition between multiple animations, thus realizing the behavior modeling of the digital human. In the above way, this solution can establish a mathematical model of the digital human behavior from five aspects: state, transition, event, action and extension, and construct the node action behavior and transition action behavior of the digital human respectively, so as to facilitate the subsequent driving of the digital human to be driven.

[0121] The present embodiment discloses generating digital human audio and phoneme sequence information corresponding to a digital human to be driven according to a language to be broadcasted; determining the digital human body language corresponding to the digital human to be driven based on the phoneme sequence information and an action state machine; generating a digital human image sequence according to the digital human body language; generating a digital human video based on the digital human audio and digital human image sequence; and presenting the digital human video through the digital human to be driven. Compared with the prior art in which digital human driving is performed by a real-person performance driving method, the modeling and driving process usually requires the support of a large number of technical personnel and is costly. Since the present embodiment generates a digital human video based on the digital human audio and digital human body language corresponding to the digital human to be driven, and queries the digital human video through the digital human to be driven, the technical problem that driving a digital human by a real-person performance driving method in the prior art requires a large amount of human resources and is costly is solved.

[0122] refer to Figure 6 , Figure 6 This is a flow chart of the second embodiment of the digital human driving method of the present invention.

[0123] Based on the above first embodiment, in this embodiment, step S20 includes:

[0124] Step S201: determining a facial key point sequence based on the phoneme sequence information and a preset digital human viseme mapping table, wherein the facial key point sequence is a sequence composed of facial key feature points of the digital human to be driven.

[0125] It should be noted that the above-mentioned preset digital human viseme mapping table may be a mapping table storing the correspondence between the pronunciation action in the syllables in the natural language and the visual expression of the face or lips. In this embodiment, the system can obtain the preset digital human viseme mapping table through expression redirection through the template mouth shape and the key points of the neutral face of the specific digital human.

[0126] It should be understood that the above-mentioned facial key point sequence can be a sequence composed of the facial key feature points of the digital human to be driven, wherein the facial key feature points of the digital human to be driven can include: feature points on facial organs such as eyebrows, eyes, mouth, nose, etc., and this embodiment is not limited to this.

[0127] Furthermore, the step S201 includes:

[0128] Step S201a: determining a facial key point offset sequence based on the phoneme sequence information and a preset digital human viseme mapping table.

[0129] It should be noted that the above-mentioned facial key point offset sequence can be a sequence composed of the position offsets between the current facial key points of the digital human to be driven and the target facial key points, wherein the positions of the target facial key points can be the positions of the facial key points of the digital human to be driven when broadcasting the digital human audio.

[0130] Step S201b: performing temporal smoothing processing on the facial key point offset sequence to obtain a smoothed facial key point offset sequence.

[0131] Step S201c: determining a facial key point sequence based on the smoothed facial key point offset sequence and the neutral facial key points of the digital human.

[0132] It should be understood that the above-mentioned digital human neutral face key points may be the facial key points corresponding to commonly used digital human expressions.

[0133] In practical applications, the system can retrieve the preset digital human viseme mapping table through phoneme sequence information, and obtain the facial key point offset sequence according to the blinking rules. Then, the facial key point offset sequence can be time-smoothed to obtain the smoothed facial key point offset sequence, and the smoothed facial key point offset sequence can be integrated with the neutral facial key points of the digital human to obtain the facial key point sequence.

[0134] Step S202: determining a body key point sequence based on the digital human behavior description information and the action state machine, wherein the body key point sequence is a sequence composed of the body key feature points of the digital human to be driven.

[0135] It should be noted that the above-mentioned digital human behavior description information may be information used to describe the corresponding action behaviors of the digital human in different states.

[0136] It can be understood that the above-mentioned body key point sequence is a sequence composed of the key feature points of the body of the digital human to be driven, wherein the key feature points of the body of the digital human to be driven may include: feature points on limbs such as arms, legs, buttocks, etc., and this embodiment does not limit this.

[0137] Furthermore, the step S202 includes:

[0138] Step S202a: Determine the body key point offset sequence based on the digital human behavior description information and the action state machine.

[0139] It should be noted that the above-mentioned body key point offset sequence can be a sequence composed of the position offsets between the current body key points of the digital human to be driven and the target body key points, wherein the positions of the target body key points can be the positions of the body key points of the digital human to be driven when it broadcasts the digital human audio.

[0140] Step S202b: performing temporal smoothing processing on the body key point offset sequence to obtain a smoothed body key point offset sequence.

[0141] Step S202c: determining a body key point sequence based on the body key point offset sequence and the digital human's neutral body key points.

[0142] It should be understood that the above-mentioned digital human neutral body key points may be the key points of the body corresponding to the commonly used digital human actions.

[0143] In practical applications, the system can obtain the body key point offset sequence based on the digital human behavior description information through the action state machine, and then perform time-series smoothing on the body key point offset sequence to obtain the smoothed body key point offset sequence, and integrate the smoothed body key point offset sequence with the digital human's neutral body key points to obtain the body key point sequence.

[0144] Step S203: obtaining an overall key point sequence according to the facial key point sequence and the body key point sequence.

[0145] It should be noted that the above-mentioned overall key point sequence may be a sequence obtained by integrating the face key point sequence and the body key point sequence.

[0146] Step S204: determining the digital human body language corresponding to the digital human to be driven according to the overall key point sequence.

[0147] In the specific implementation, refer to Figure 7 , Figure 7 FIG. 1 is a flow chart of generating the body language of a digital human in the second embodiment of the digital human driving method of the present invention. Figure 7 As shown, first, the system can use the phoneme sequence information {ph1, ph2, ..., ph n} Retrieve the preset digital human viseme mapping table and obtain the facial key point offset sequence ΔL according to the blinking rule f , then the facial key point offset sequence ΔL can be f Perform temporal smoothing to obtain a smoothed facial key point offset sequence, and compare the smoothed facial key point offset sequence and the digital human neutral facial key point L N f Integrate to obtain the facial key point sequence L f At the same time, the system can obtain the body key point offset sequence ΔL based on the digital human behavior description information B through the action state machine b , then the body keypoints can be offset by the sequence ΔL b Perform temporal smoothing to obtain a smoothed body key point offset sequence, and compare the smoothed body key point offset sequence and the digital human neutral body key point L N b Integrate to obtain the body key point sequence L b Finally, the system can generate the facial key point sequence L f and body key point sequence L b The whole key point sequence L is obtained by integration, so as to generate the digital human body language T corresponding to the digital human to be driven based on the whole key point sequence L. b .

[0148] This embodiment discloses determining a facial key point sequence based on phoneme sequence information and a preset digital human viseme mapping table; determining a body key point sequence based on digital human behavior description information and an action state machine; obtaining an overall key point sequence based on the facial key point sequence and the body key point sequence; and determining the digital human body language corresponding to the digital human to be driven based on the overall key point sequence. Since this embodiment can integrate the facial key point sequence and the body key point sequence of the robot to be driven to obtain an overall key point sequence, the digital human body language corresponding to the digital human to be driven can be accurately determined based on the overall key point sequence, thereby improving the accuracy of the digital human driving.

[0149] refer to Figure 8 , Figure 8This is a flow chart of the third embodiment of the digital human driving method of the present invention.

[0150] Based on the above embodiments, in this embodiment, step S50 includes:

[0151] Step S501: converting the digital human audio and the digital human video into an audio stream and a video stream respectively through a target stream pusher.

[0152] It should be noted that the target stream pusher may be a device or software for implementing real-time audio and video stream push. In this embodiment, the target stream pusher may include but is not limited to an RTMP stream pusher, where RTMP (Real-Time Messaging Protocol) is a real-time audio and video streaming media transmission protocol.

[0153] In the specific implementation, refer to Fig. 9 , Fig. 9 This is a flow chart of RTMP protocol streaming in the third embodiment of the digital human driving method of the present invention. Fig. 9 As shown, streaming through the RTMP streaming device may include: a handshake phase, a connection phase, a stream establishment phase and a streaming phase. After streaming using the RTMP streaming device, the digital human audio and digital human video may be converted into corresponding audio streams and video streams.

[0154] Step S502: Synchronize the audio stream and the video stream according to the target timestamp to obtain synchronized audio and video streams.

[0155] It should be noted that the above target timestamp may be a pre-specified timestamp of the generated audio and video stream.

[0156] Step S503: Pushing the synchronized audio and video streams to a target server, and presenting the synchronized audio and video streams through the target server and the digital human to be driven.

[0157] It should be understood that the above-mentioned target server may be a server for processing concurrent requests, such as an Nginx server, and this embodiment does not impose any limitation on this.

[0158] In the specific implementation, refer to Fig.10 , Fig.10 This is a diagram showing the architecture of the digital human video streaming presentation in the third embodiment of the digital human driving method of the present invention. Fig.10As shown in the figure, in the pre-generation phase, the digital human voice and phoneme sequence information generated by the natural language synthesis digital human voice module, the digital human body language generated by the natural language synthesis digital human voice module, and the digital human video generated by the audio-visual linkage digital human video generation module are all stored in the RabbitMQ process pool. In the online inference phase, when no inference request is received, the stream pusher will loop the video of the statically read default posture; when an inference request is received, the stream pusher will obtain the video from the queue of real-time generated images.

[0159] After generating a digital human video containing voice, lip shape, expression and action, for the carrier of the driving text, this solution can design the message format of the request message so that the system can respond through a local or remote request message. At the same time, in this solution, the API of FFmpeg (i.e., an open source multimedia processing framework) can be used to develop and implement an RTMP stream pusher, and an RTMP live streaming server can be built using the Nginx server, so that the video stream obtained by FFmpeg can be pushed to the Nginx server, so that users can watch the digital human video through the streaming tool, realizing the live broadcast of the digital human. In the data stream of this solution, the driving text requires a carrier to access this system. Therefore, this system designs a message format as a carrier of the driving text and a request for online reasoning to generate digital human videos. After the service of this system is started, the message request file can be read locally or the request message can be obtained from the network through a WebSocket connection. Among them, the message is mainly divided into three types: initInference, inference and closeInference. For each request message type, the system defines the format of the message, refer to Fig.11 , Fig.11This is a partial field display diagram of the inference message in the third embodiment of the digital human driving method of the present invention. In practical applications, after receiving the request message, the system can first search the corpus according to the sentenceId field, and parse the text into the actual input text of the inference process. Then the text is sequentially passed through the natural language synthesis digital human voice module, the voice-driven body language generation module, and the audio-visual linkage digital human video generation module to generate a digital human speaking video. The system specifies the digital human inference model through the modelId field. RabbitMQ, as the message queue middleware, is responsible for data transmission between various modules, and uses the process pool to manage various modules in the driving process. After the digital human video is generated, the system can convert the generated audio and video into audio and video streams through the FFmpeg stream pusher, and specify the timestamp of the generated audio and video streams according to the timestamp field, perform audio and video synchronization, and finally forward it to the push stream address corresponding to the rtmpUrl field through the Nginx server, so that subsequent users can watch the digital human video through the pull stream tool, realizing the live broadcast of the digital human.

[0160] Furthermore, in order to optimize the digital human driving process in real time and improve the accuracy and efficiency of digital human driving, before step S501, it also includes: segmenting the digital human video to determine the static video segments in the digital human video; building a target corpus based on the static video segments; and generating a fused digital human video based on the dynamic video segments and the static video segments in the target corpus when receiving an inference request.

[0161] It should be noted that the static video clips mentioned above may be static video clips that do not change in the digital human video; correspondingly, the dynamic video clips mentioned above may be dynamically changing video clips that are generated in real time.

[0162] It should be understood that the above-mentioned target corpus may be a database for storing static video clips.

[0163] It can be understood that the above-mentioned fused digital human video may be a video segment obtained by fusing a dynamic video segment and a static video segment.

[0164] In practical applications, pre-calculation can be combined to achieve real-time optimization of the digital human-driven process. Specifically, since many video clips have similar contents, such as "Good afternoon, Mr. Wang." and "Good afternoon, Mr. Zhang." In these two sentences, except for the two characters "Wang" and "Zhang", the other parts are the same. Therefore, in practical applications, the video can be segmented, the static parts that do not change (i.e., the above-mentioned static video clips) can be pre-calculated offline, and a pre-generated target corpus can be constructed. During real-time rendering, only the dynamic parts in the target corpus (i.e., the above-mentioned dynamic video clips) are rendered and replaced with the pre-generated video. At the same time, since only the lip shape needs to be replaced, the body model can reuse the pre-calculated content, and only the facial NMFC image of the digital human needs to be rendered during real-time rendering.

[0165] The step S501 includes: converting the digital human audio and the fused digital human video into an audio stream and a video stream respectively through a target stream pusher.

[0166] In this embodiment, when no inference request is received, the target streamer will loop the video of the default posture read statically; when an inference request is received, the streamer will obtain the video from the queue of real-time generated images, that is, obtain the dynamic video clip, so that the dynamic video clip and the static video clip in the target corpus can be fused later, and then the digital human audio and the fused digital human video can be converted into audio stream and video stream. Among them, the target streamer can load and push the offline generated part of the video clip in advance, reducing the user's waiting time, thereby improving the driving efficiency of the digital human.

[0167] In addition, this solution can also achieve real-time optimization of the digital human driving process based on the message queue. Specifically, in the online reasoning process, each module can be used as a process, and the message queue can be used to transfer data between processes in the form of Chunks, and the process pool can be used to manage the processes. In this way, the data communication delay between modules can be reduced and the parallelism of the front module and the back module can be improved.

[0168] Among them, the purpose of real-time optimization combined with pre-calculation is to reduce the time consumption of the digital human video generation module with audio-visual linkage, while the purpose of real-time optimization based on message queue is to reduce the time consumption of data IO operations between modules and increase the parallelism between modules.

[0169] This embodiment discloses converting the digital human audio and digital human video into audio stream and video stream respectively through a target stream pusher; synchronizing the audio stream and video stream according to a target timestamp to obtain a synchronized audio and video stream; pushing the synchronized audio and video stream to a target server, and presenting the synchronized audio and video stream through the target server and the digital human to be driven, so that users can watch the digital human video through a stream pulling tool, thus realizing the live broadcast of the digital human.

[0170] In addition, an embodiment of the present invention further provides a storage medium, on which a digital human driver program is stored. When the digital human driver program is executed by a processor, the steps of the digital human driving method described above are implemented.

[0171] Reference Fig.12 , Fig.12 This is a structural block diagram of the first embodiment of the digital human driving device of the present invention.

[0172] like Fig.12 As shown, the digital human driving device proposed in the embodiment of the present invention includes:

[0173] The speech generation module 501 is used to generate the digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcasted;

[0174] A body language determination module 502, used to determine the body language of the digital human corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine;

[0175] An image sequence generating module 503, used for generating a digital human image sequence according to the digital human body language;

[0176] A video generation module 504, configured to generate a digital human video based on the digital human audio and the digital human image sequence;

[0177] The video presenting module 505 is used to present the digital human video through the digital human to be driven.

[0178] Furthermore, the digital human driving device further comprises: a model building module, wherein:

[0179] The model building module is used to construct the state node action behavior corresponding to the action state machine through the human posture estimation method, wherein the action state machine is a model used to control the action of the digital human to be driven; construct the inter-node transition action behavior corresponding to the action state machine; and perform behavior modeling on the digital human to be driven based on the state node action behavior and the inter-node transition action behavior.

[0180] The digital human driving device of the present embodiment discloses generating digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcast; determining the digital human body language corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine; generating a digital human image sequence according to the digital human body language; generating a digital human video based on the digital human audio and digital human image sequence; presenting the digital human video through the digital human to be driven; compared with the prior art of driving digital humans through real-person performance driving, the modeling and driving process usually requires the support of a large number of technical personnel and is costly. Since the present embodiment generates a digital human video based on the digital human audio and digital human body language corresponding to the digital human to be driven, and queries the digital human video through the digital human to be driven, it solves the technical problem that driving digital humans through real-person performance driving in the prior art requires a large amount of human resources and is costly.

[0181] Based on the first embodiment of the digital human driving device of the present invention, a second embodiment of the digital human driving device of the present invention is proposed.

[0182] In this embodiment, the body language determination module 502 is further used to determine a facial key point sequence based on the phoneme sequence information and a preset digital human viseme mapping table, wherein the facial key point sequence is a sequence composed of facial key feature points of the digital human to be driven; determine a body key point sequence based on the digital human behavior description information and an action state machine, wherein the body key point sequence is a sequence composed of body key feature points of the digital human to be driven; obtain an overall key point sequence based on the facial key point sequence and the body key point sequence; and determine the digital human body language corresponding to the digital human to be driven based on the overall key point sequence.

[0183] Furthermore, the body language determination module 502 is also used to determine a facial key point offset sequence based on the phoneme sequence information and a preset digital human viseme mapping table; perform time-series smoothing on the facial key point offset sequence to obtain a smoothed facial key point offset sequence; and determine a facial key point sequence based on the smoothed facial key point offset sequence and the digital human neutral facial key points.

[0184] Furthermore, the body language determination module 502 is also used to determine a body key point offset sequence based on the digital human behavior description information and the action state machine; perform time series smoothing on the body key point offset sequence to obtain a smoothed body key point offset sequence; and determine a body key point sequence based on the body key point offset sequence and the digital human's neutral body key points.

[0185] This embodiment discloses determining a facial key point sequence based on phoneme sequence information and a preset digital human viseme mapping table; determining a body key point sequence based on digital human behavior description information and an action state machine; obtaining an overall key point sequence based on the facial key point sequence and the body key point sequence; and determining the digital human body language corresponding to the digital human to be driven based on the overall key point sequence. Since this embodiment can integrate the facial key point sequence and the body key point sequence of the robot to be driven to obtain an overall key point sequence, the digital human body language corresponding to the digital human to be driven can be accurately determined based on the overall key point sequence, thereby improving the accuracy of the digital human driving.

[0186] Based on the above device embodiments, a third embodiment of the digital human driving device of the present invention is proposed.

[0187] In this embodiment, the video presentation module 504 is also used to convert the digital human audio and the digital human video into audio stream and video stream respectively through the target stream pusher; synchronize the audio stream and the video stream according to the target timestamp to obtain the synchronized audio and video stream; push the synchronized audio and video stream to the target server, and present the synchronized audio and video stream through the target server and the digital human to be driven.

[0188] Furthermore, the video presentation module 504 is also used to segment the digital human video to determine static video segments in the digital human video; build a target corpus based on the static video segments; when receiving an inference request, generate a fused digital human video based on dynamic video segments and the static video segments in the target corpus; and convert the digital human audio and the fused digital human video into an audio stream and a video stream respectively through a target stream pusher.

[0189] This embodiment discloses converting the digital human audio and digital human video into audio stream and video stream respectively through a target stream pusher; synchronizing the audio stream and video stream according to a target timestamp to obtain a synchronized audio and video stream; pushing the synchronized audio and video stream to a target server, and presenting the synchronized audio and video stream through the target server and the digital human to be driven, so that users can watch the digital human video through a stream pulling tool, thus realizing the live broadcast of the digital human.

[0190] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0191] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0193] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A digital human driving method, characterized in that: The digital human driving method comprises: Generate digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcasted; Determine the digital human body language corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine; generating a digital human image sequence according to the digital human body language; Generate a digital human video based on the digital human audio and the digital human image sequence; Presenting the digital human video by using the digital human to be driven; Before the step of generating the digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcasted, the method further includes: Constructing state node action behaviors corresponding to an action state machine by a human posture estimation method, wherein the action state machine is a model for controlling the actions of a digital human to be driven; Constructing the inter-node transition action behavior corresponding to the action state machine; Modeling the state node action behavior and the transition action behavior between nodes based on the two-dimensional posture model and the three-dimensional posture model respectively, so as to perform behavior modeling on the digital human to be driven; Among them, the state node action behavior based on the two-dimensional posture model is implemented based on the DWPose model; the node action behavior based on the three-dimensional posture model is implemented based on the HybrIK model; the transition action behavior based on the two-dimensional posture model is implemented based on the LERP model; the transition action behavior based on the three-dimensional posture model is implemented based on the SLERP model; The DWPose model is used to perform feature distillation and adjust feature distribution; the HybrIK model uses CNN to predict preliminary human 3D key points, SMPL shape parameters, twist rotation angle and initial posture, and obtains SMPL posture parameters based on the human 3D key points, the SMPL shape parameters, the twist rotation angle and the initial posture, and obtains the final 3D key points based on the SMPL shape parameters and the SMPL posture parameters.

2. The digital human driving method according to claim 1, characterized in that: The step of determining the body language of the digital human corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine comprises: Determine a facial key point sequence based on the phoneme sequence information and a preset digital human viseme mapping table, wherein the facial key point sequence is a sequence composed of facial key feature points of the digital human to be driven; Determine a body key point sequence based on the digital human behavior description information and the action state machine, wherein the body key point sequence is a sequence composed of the body key feature points of the digital human to be driven; Acquire an overall key point sequence according to the face key point sequence and the body key point sequence; The digital human body language corresponding to the digital human to be driven is determined according to the overall key point sequence.

3. The digital human driving method according to claim 2, characterized in that: The step of determining a facial key point sequence based on the phoneme sequence information and a preset digital human viseme mapping table comprises: Determine a facial key point offset sequence based on the phoneme sequence information and a preset digital human viseme mapping table; Performing temporal smoothing processing on the facial key point offset sequence to obtain a smoothed facial key point offset sequence; A facial key point sequence is determined based on the smoothed facial key point offset sequence and the neutral facial key points of the digital human.

4. The digital human driving method according to claim 2, characterized in that: The step of determining the body key point sequence based on the digital human behavior description information and the action state machine includes: Determine the body key point offset sequence based on the digital human behavior description information and action state machine; Performing temporal smoothing processing on the body key point offset sequence to obtain a smoothed body key point offset sequence; A body key point sequence is determined based on the body key point offset sequence and the neutral body key points of the digital human.

5. The digital human driving method according to claim 1, characterized in that: The step of presenting the digital human video by the digital human to be driven comprises: Converting the digital human audio and the digital human video into an audio stream and a video stream respectively through a target stream pusher; Synchronize the audio stream and the video stream according to the target timestamp to obtain a synchronized audio and video stream; The synchronized audio and video streams are pushed to a target server, and the synchronized audio and video streams are presented through the target server and the digital human to be driven.

6. The digital human driving method according to claim 5, characterized in that: Before the step of converting the digital human audio and the digital human video into an audio stream and a video stream respectively through a target stream pusher, the method further includes: Segmenting the digital human video to determine static video segments in the digital human video; Building a target corpus based on the static video clips; Upon receiving an inference request, generating a fused digital human video based on the dynamic video clip and the static video clip in the target corpus; The step of converting the digital human audio and the digital human video into an audio stream and a video stream respectively by a target stream pusher comprises: The digital human audio and the fused digital human video are converted into an audio stream and a video stream respectively through a target stream pusher.

7. A digital human driving device, characterized in that: The device comprises: A speech generation module, used to generate digital human audio and phoneme sequence information corresponding to the digital human to be driven according to the language to be broadcasted; A body language determination module, used for determining the body language of the digital human corresponding to the digital human to be driven based on the phoneme sequence information and the action state machine; An image sequence generation module, used for generating a digital human image sequence according to the digital human body language; A video generation module, used for generating a digital human video based on the digital human audio and the digital human image sequence; A video presentation module, used for presenting the digital human video through the digital human to be driven; The speech generation module is also used to construct the state node action behavior corresponding to the action state machine through the human posture estimation method, the action state machine is a model for controlling the action of the digital human to be driven; construct the node transition action behavior corresponding to the action state machine; model the state node action behavior and the node transition action behavior based on the two-dimensional posture model and the three-dimensional posture model respectively, so as to perform behavior modeling on the digital human to be driven; wherein the state node action behavior based on the two-dimensional posture model is implemented based on the DWPose model; the node action behavior based on the three-dimensional posture model is implemented based on the HybrIK model; the transition action behavior based on the two-dimensional posture model is implemented based on the LERP model; the transition action behavior based on the three-dimensional posture model is implemented based on the SLERP model; the DWPose model is used for feature distillation and adjustment of feature distribution; the HybrIK model adopts CNN to predict preliminary three-dimensional key points of the human body, SMPL shape parameters, twist rotation angle and initial posture, and obtains SMPL posture parameters based on the three-dimensional key points of the human body, the SMPL shape parameters, the twist rotation angle and the initial posture, and obtains the final three-dimensional key points based on the SMPL shape parameters and the SMPL posture parameters.

8. A digital human driving device, characterized in that: The device comprises: a memory, a processor, and a digital human driver program stored in the memory and executable on the processor, wherein the digital human driver program is configured to implement the steps of the digital human driving method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a digital human driver program, which, when executed by a processor, implements the steps of the digital human driving method according to any one of claims 1 to 6.

Images