Method, device and equipment for generating digital human gesture and storage medium

Abstract

This invention provides a method, apparatus, device, and storage medium for generating digital human gestures. The method includes: acquiring a target audio file from which digital human gestures are to be generated; determining an action sequence corresponding to the target audio file based on a script generation model; and controlling the synthesis of representative gestures and rhythmic gestures corresponding to the target audio file based on the action sequence and the gesture generation model. The method provided by this invention effectively controls the synthesis of digital human gestures under synchronized speech by using the action sequence corresponding to the target audio file determined by the script generation model. It decouples and models the gestures to obtain a representative gesture generation model and a rhythmic gesture generation model. By combining the representative gestures and rhythmic gestures obtained from the gesture generation models, more natural and richer gestures can be generated, making the digital human gestures more realistic.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for generating digital human gestures. Background Technology

[0002] Digital humans understand and analyze external inputs through a recognition system, generating feedback results in response to driving signals. Based on these decisions, they synthesize corresponding digital human speech and behavioral actions, enabling interactive communication with humans. The effectiveness of digital human action-driven communication is a key factor influencing the degree of anthropomorphism in digital humans. In particular, gestures have a strong auxiliary expressive role, effectively promoting expression as non-verbal information.

[0003] The development of deep learning technology in recent years has also promoted the development of gesture generation technology, employing large-scale datasets and deep neural networks to model the relationships between multiple modalities. Most existing methods for generating digital human gestures use fixed rules to match gestures in a predefined database. These fixed rules require design by professionals and prior knowledge, and for complex speech scenarios, the generated results are not rich enough, lacking realism and naturalness. The implementation threshold is high, and the corresponding results are not ideal.

[0004] Therefore, how to utilize existing large-scale datasets to generate realistic and natural digital human gestures has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] In view of the technical problems existing in the prior art, the present invention provides a method, apparatus, device and storage medium for generating digital human gestures.

[0006] In a first aspect, the present invention provides a method for generating digital human gestures, comprising:

[0007] Obtain the target audio file for the digital human gestures to be generated;

[0008] Based on a script generation model, the action sequence corresponding to the target audio file is determined; the action sequence is used to indicate whether a gesture action exists at any given time.

[0009] Based on the action sequence and gesture generation model, the generated representative gestures and rhythmic gestures are synthesized into digital human gestures corresponding to the target audio file.

[0010] The script generation model is trained based on training samples determined from a first video file containing speech and motion information; the gesture generation model includes a first gesture generation model and a second gesture generation model, wherein the first gesture generation model is used to generate representative gestures and the second gesture generation model is used to generate rhythmic gestures.

[0011] Optionally, the step of controlling the generation of representative gestures and rhythmic gestures based on the action sequence and gesture generation model to synthesize the digital human gestures corresponding to the target audio file includes:

[0012] Based on the first gesture generation model, a representative gesture corresponding to the target audio file is generated;

[0013] Based on the second gesture generation model, rhythmic gestures corresponding to the target audio file are generated;

[0014] Based on the sequence of actions and a preset synthesis rule, the representative gesture and the rhythmic gesture are fused to obtain the digital human gesture corresponding to the target audio file; the preset synthesis rule is used to limit the digital human gesture at any time to be determined based on any one or a combination of the representative gesture and the rhythmic gesture.

[0015] Optionally, the script generation model is trained based on training samples determined from a first video file containing speech and motion information, and the corresponding training method includes:

[0016] The first N elements and the last M elements of the current element in the initial training sample are determined as one element in the first training sample; the initial training sample is a training sample used to characterize whether there is action in the first video file at different times; the first video file includes voice information and action information; M and N are positive integers;

[0017] Based on the first training sample, if the first loss function is convergent, the training of the script generation model is completed; the first loss function is determined based on the first label corresponding to the first training sample and the prediction result output by the script generation model after the first training sample is input into the script generation model.

[0018] Optionally, the initial training samples are training samples used to characterize whether action occurs in the first video file at different times, and the corresponding acquisition method includes:

[0019] Based on the first video file, a first position sequence signal is obtained; the first position sequence signal is used to characterize the positions of all key points of the human hand bones, body bones and facial bones at different times;

[0020] Determine the starting position corresponding to any gesture represented by the first position sequence signal;

[0021] Based on the distance between the target element and the starting position in the first position sequence, and the distance between the next element of the target element in the first position sequence and the starting position, the target element is labeled with a first label and used as the initial training sample; the target element is any element in the first position sequence; the first label is used to characterize whether an action has occurred in the target element.

[0022] Optionally, determining the starting position corresponding to any gesture represented by the first position sequence signal includes:

[0023] Based on the preset simplified gesture key points, the first position sequence signal is simplified, and the position histogram of the simplified gesture key points is statistically obtained.

[0024] According to the preset complete gesture format, the position that appears most frequently in the histogram is determined as the starting position corresponding to any gesture represented by the first position sequence signal.

[0025] Optionally, the training method corresponding to the first gesture generation model includes:

[0026] Based on the first label, extract samples from the initial training samples that represent the occurrence of actions, and use them as the second training samples;

[0027] The second training sample is uniformly sampled to obtain a third training sample of uniform length; each sample in the third training sample includes L uniformly sampled second training samples; where L is a positive integer;

[0028] Based on the third training sample, if the second loss function is convergent, the training of the first gesture generation model is completed. The second loss function is determined based on the labels of the third training sample and the gesture reconstructed by the first gesture generation model.

[0029] Optionally, before uniformly sampling the second training sample to obtain a third training sample of uniform length, the method further includes:

[0030] Determine whether there is any missing key point data in each sample of the second training sample;

[0031] If any training sample in the second training sample has missing key point data, it is repaired by rotating and translating adjacent training samples.

[0032] Secondly, the present invention also provides a device for generating digital human gestures, comprising:

[0033] The acquisition module is used to acquire the target audio file for the digital human gestures to be generated;

[0034] The determination module is used to determine the action sequence corresponding to the target audio file based on the script generation model; the action sequence is used to indicate whether a gesture action exists at any given time.

[0035] The generation module is used to control the generation of representative gestures and rhythmic gestures to synthesize the digital human gestures corresponding to the target audio file based on the action sequence and the gesture generation model.

[0036] The script generation model is trained based on training samples determined from a first video file containing speech and motion information; the gesture generation model includes a first gesture generation model and a second gesture generation model, wherein the first gesture generation model is used to generate representative gestures and the second gesture generation model is used to generate rhythmic gestures.

[0037] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the digital human gesture generation method described in the first aspect above.

[0038] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating digital human gestures as described in the first aspect above.

[0039] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for generating digital human gestures as described in the first aspect above.

[0040] The present invention provides a method, apparatus, device, and storage medium for generating digital human gestures. By generating a sequence of actions corresponding to a target audio file determined by a script generation model, the invention effectively controls the synthesis of digital human gestures under synchronized speech. By decoupling and modeling the gestures, a representative gesture generation model and a rhythmic gesture generation model are obtained. By combining the representative gestures and rhythmic gestures obtained by the gesture generation models, more natural and richer gestures can be generated, making the digital human gestures more realistic. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1This is a flowchart illustrating the method for generating digital human gestures provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the specific implementation process of digital human gesture generation provided in this embodiment of the invention;

[0044] Figure 3 This is a schematic diagram of the structure of the digital human gesture generation device provided in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0047] In this embodiment of the invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Figure 1 This is a flowchart illustrating the method for generating digital human gestures provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0050] Step 101: Obtain the target audio file for the digital human gesture to be generated;

[0051] Specifically, the target audio file used to generate digital human gestures can be a directly acquired audio file, an audio file extracted from a video file, or an audio file converted from other file types. The target audio file can support any audio format, such as MP3, MP5, WAV, Flac, MIDI, RA, APE, AAC, CDA, MOV, etc. The target audio file is a file that stores sound content. There is no limit to the duration of the audio file; audio files of any length are acceptable.

[0052] Step 102: Based on the script generation model, determine the action sequence corresponding to the target audio file; the action sequence is used to indicate whether a gesture action exists at any time; the script generation model is trained based on training samples determined from a first video file containing speech information and action information.

[0053] The script generation model provided in this application processes a first video file containing speech and motion information to obtain corresponding training samples. These training samples primarily extract whether corresponding motion information exists at different times and establish a correlation between speech and motion information. The speech information can be represented by an audio file, which can be extracted from the first video file; the specific audio file format is not limited. This process is then used to train the script generation model. The first video file can be obtained from existing large-scale datasets such as TED, Speech2Gestrure, and Trinity Gesture Dataset.

[0054] According to the script generation model provided in this application, determining whether a gesture exists at any given time in the target audio file can be understood as generating a sequence of actions or an action script. This sequence of actions or the action script indicates whether an action exists at each time. If an action exists at a certain time, the corresponding gesture is added to that time; if no action exists at a certain time, the corresponding gesture is not added. The existence of an action at a given time is determined by the trained script generation model based on the target audio file. The script generation model is trained using representative training samples.

[0055] Step 103: Based on the action sequence and gesture generation model, control the generation of representative gestures and rhythmic gestures to synthesize the digital human gestures corresponding to the target audio file;

[0056] The gesture generation model includes a first gesture generation model and a second gesture generation model. The first gesture generation model is used to generate representative gestures, and the second gesture generation model is used to generate rhythmic gestures.

[0057] The action sequence obtained by the script generation model indicates whether a gesture exists at any given time. The first gesture generation model can generate a representative gesture based on the target audio file. This representative gesture usually has a certain regularity or conforms to a complete format. For example, the action of raising a hand usually includes several consecutive stages of movement of the elbow, wrist, and fingers, obtained by observing different hand joints. The second gesture generation model is mainly used to generate corresponding rhythmic gestures based on the target audio file. Rhythmic gestures usually do not have substantial meaning; they are usually habitual actions, such as slight swaying of the body to music or rhythmic tapping. Thus, the first gesture generation model generates representative gestures corresponding to the target audio file, and each representative gesture also has a temporal characteristic. That is, the generated representative gestures can be represented as a representative gesture sequence. This representative gesture sequence is temporal, indicating whether a representative gesture exists at any given time and what that representative gesture is. Similarly, rhythmic gestures can also be represented as rhythmic gesture sequences. This rhythmic gesture sequence is practical, indicating whether a rhythmic gesture exists at any given time and what that rhythmic gesture is.

[0058] The same target audio file is used to generate an action sequence through the above script generation model, as well as representative gestures and rhythmic gestures through the gesture generation model. The digital human gestures corresponding to the final synthesized target audio file are controlled by inserting representative gestures, rhythmic gestures, or a combination of both when gestures are present in the action sequence, or by not inserting any gestures, or inserting rhythmic gestures, or other methods when there are no gestures in the action sequence.

[0059] The digital human gesture generation method provided by this invention effectively controls the synthesis of digital human gestures under synchronized speech by using the action sequence corresponding to the target audio file determined by the script generation model. The gestures are decoupled and modeled to obtain a representative gesture generation model and a rhythmic gesture generation model. By combining the representative gestures and rhythmic gestures obtained by the gesture generation models, more natural and richer gestures can be generated, making the digital human gestures more realistic.

[0060] Optionally, the step of controlling the generation of representative gestures and rhythmic gestures based on the action sequence and gesture generation model to synthesize the digital human gestures corresponding to the target audio file includes:

[0061] Based on the first gesture generation model, a representative gesture corresponding to the target audio file is generated;

[0062] Based on the second gesture generation model, rhythmic gestures corresponding to the target audio file are generated;

[0063] Based on the sequence of actions and a preset synthesis rule, the representative gesture and the rhythmic gesture are fused to obtain the digital human gesture corresponding to the target audio file; the preset synthesis rule is used to limit the digital human gesture at any time to be determined based on any one or a combination of the representative gesture and the rhythmic gesture.

[0064] Specifically, the target audio file is input into the trained first gesture generation model M. R The representative gesture P was obtained from the middle R The target audio signal is input into the trained second gesture generation model M. P The rhythmic gesture P is obtained from B .

[0065] By controlling the sequence of actions and pre-defined synthesis rules, the digital human's gestures are controlled to be generated from the representative gesture P at the same moment when a gesture action is present. R and rhythmic gestures P B One or a combination of these elements are used for synthesis, and in the absence of a hand gesture, the digital human's gesture is composed of a representative gesture P at the same moment. R Rhythmic gestures P B Alternatively, a fixed preset gesture can be selected from all the gestures.

[0066] By generating the action sequence from the model using the script described above, and by defining different preset synthesis rules, the rules for generating digital human gestures can be customized and edited, effectively controlling the generated digital human gestures.

[0067] Optionally, the script generation model is trained based on training samples determined from a first video file containing speech and motion information, and the corresponding training method includes:

[0068] The first N elements and the last M elements of the current element in the initial training sample are determined as one element in the first training sample; the initial training sample is a training sample used to characterize whether there is action in the first video file at different times; the first video file includes voice information and action information; M and N are positive integers;

[0069] Based on the first training sample, if the first loss function is convergent, the training of the script generation model is completed; the first loss function is determined based on the first label corresponding to the first training sample and the prediction result output by the script generation model after the first training sample is input into the script generation model.

[0070] Specifically, the script generation model described above is trained based on training samples determined from a first video file containing speech and motion information. First, a speaker video dataset is collected as needed. Text signals with time information, audio signals from the speaker video, and key point position sequence signals of the human skeleton from the speaker video are extracted. The extracted text and audio signals are time-aligned, and combined with the aforementioned key point position sequence signals of the human skeleton to determine the dataset for model training. This dataset can be divided into a training set and a test set. This dataset can be understood as the initial training samples.

[0071] The extracted text and audio signals are time-aligned, including:

[0072] As needed, the Montreal-Forced-Aligner (MFA) tool is used to align the text with the text annotations obtained from the speech, extracting text signals with time information. These extracted text signals are then converted into text features T using a Chinese text pre-training model. i, The text features are input into a text semantic coding model to obtain semantic features; the extracted audio signal is then converted into spectral features S through calculation as needed. i The spectral features are input into the audio coding model to obtain prosodic features; the semantic features and the prosodic features are aligned according to time.

[0073] The first training sample for constructing the script-generated model specifically includes: the N frames before and the M frames after the current frame of the human skeleton keypoint position sequence signal in the initial training sample, and the corresponding aligned audio signal, as one element in the first training sample. Here, M and N are positive integers, which can be the same or different, and the specific values ​​can be set according to actual needs.

[0074] A regression model is constructed using a fully convolutional ResNet-101 network structure as the script generation model. The probability of action occurrence is obtained through a sigmoid activation function. The corresponding first loss function is determined based on the first label corresponding to the first training sample and the prediction result output by the script generation model after inputting the first training sample. It can be expressed in the form of Mean Square Error (MSE). Among them, L Script Let n represent the first loss function. j The value representing the actual action that occurred in the current frame, n′ j This represents the value of the action predicted by the model based on the script, m represents the total number of data in each batch during training, and j represents any one of the training samples in a batch.

[0075] The first training sample mentioned above is input into the script generation model M. A Then, using the difference between the predicted action script and the real action script in the training samples, the parameters in the script generation model are adjusted through gradient backpropagation to learn the model. If the first loss function of gradient backpropagation converges, the model training is complete, and the gesture action generation script n is finally obtained. The first loss function used for gradient backpropagation is the absolute distance function between the predicted action script and the real action script.

[0076] The initial training samples are training samples used to characterize whether action occurs in the first video file at different times, and the corresponding acquisition method includes:

[0077] Based on the first video file, a first position sequence signal is obtained; the first position sequence signal is used to characterize the positions of all key points of the human hand bones, body bones and facial bones at different times;

[0078] Determine the starting position corresponding to any gesture represented by the first position sequence signal;

[0079] Based on the distance between the target element and the starting position in the first position sequence, and the distance between the next element of the target element in the first position sequence and the starting position, the target element is labeled with a first label and used as the initial training sample; the target element is any element in the first position sequence; the first label is used to characterize whether an action has occurred in the target element.

[0080] Specifically, the method for obtaining the dataset for training the above model, i.e. the initial training samples, includes: extracting audio signals from the speaker's video based on the first video file containing speech and action information. Specifically, this can be done by extracting the audio file of each first video file and converting the audio file into Mel spectrogram features as the audio signal.

[0081] Extract the sequence signal of the human skeleton key points in the speaker's video, that is, obtain the first position sequence signal. Specifically, the key points of the human skeleton in the first video file can be extracted using the Mediapipe tool, including 42 hand key points, 33 body skeleton key points and 468 facial key points. Each of the above skeleton key points can be represented by three-dimensional position coordinates.

[0082] The aforementioned first position sequence signal can represent any gesture, and the several consecutive actions included in any gesture may conform to the corresponding rules, each with a corresponding start position and end position, or each with a start point and end point. Here, the start position and start point are used to indicate the starting action of a gesture, that is, the gesture begins at that moment, and the end position is used to indicate the ending action of a gesture, that is, the gesture ends at that moment.

[0083] Determine the distance between the target element and the starting position in the first position sequence, and the distance between the next element of the target element and the starting position. Based on whether these two distances satisfy the rule for determining whether any element in the first position sequence has an action, mark each element in the first position sequence with a first mark. The first mark is used to indicate whether any element in the first position sequence has an action.

[0084] By marking any element in the first position sequence as described above to indicate whether an action occurred, a marked first position sequence, i.e., the initial training sample, is obtained. This initial training sample is then used for further processing or filtering in subsequent operations.

[0085] Optionally, determining the starting position corresponding to any gesture represented by the first position sequence signal includes:

[0086] Based on the preset simplified gesture key points, the first position sequence signal is simplified, and the position histogram of the simplified gesture key points is statistically obtained.

[0087] According to the preset complete gesture format, the position that appears most frequently in the histogram is determined as the starting position corresponding to any gesture represented by the first position sequence signal.

[0088] Specifically, the consecutive actions included in any gesture represented by the first position sequence signal may conform to a corresponding pattern, which can be obtained through observation and statistical analysis. In order to reduce the number of analyses and speed up the processing speed, this invention simplifies the first position sequence signal by using preset simplified gesture key points to simplify the pattern that the consecutive actions included in any gesture may conform to. Here, the preset simplified gesture key points may be several main skeletal key points that can be used to simply represent gestures among all the above skeletal key points, such as the skeletal key points of the elbow, wrist and fingers. Among them, the optimal simplified gesture key points include the positions of four skeletal key points: left elbow, right elbow, left wrist and right wrist, which are used as the optimal configuration of the preset simplified gesture key points.

[0089] According to the preset simplified gesture key points, the above first position sequence is simplified, and the position histogram of the first position sequence at the preset simplified gesture key points is calculated.

[0090] The first position sequence represents any gesture, which may include several consecutive actions that conform to a corresponding pattern. Here, the pattern is to describe a complete gesture as several stages, each with its own characteristics. The above stages are used as a preset complete gesture format. The stages of a complete gesture may include five stages: rest position or starting position, preparation, stroke, hold, and retraction. Any complete gesture starts from the rest position or starting position and continues until the retraction stage is completed, returning to the rest position or starting position corresponding to the next complete gesture.

[0091] Based on the above position histogram and the preset complete gesture format corresponding to each complete gesture, find the position that appears most frequently in the position histogram (the position that appears the most times or the position that appears most often) as the above rest position or starting position.

[0092] Optionally, the training method corresponding to the first gesture generation model includes:

[0093] Based on the first label, extract samples from the initial training samples that represent the occurrence of actions, and use them as the second training samples;

[0094] The second training sample is uniformly sampled to obtain a third training sample of uniform length; each sample in the third training sample includes L uniformly sampled second training samples; where L is a positive integer;

[0095] Based on the third training sample, if the second loss function is convergent, the training of the first gesture generation model is completed. The second loss function is determined based on the labels of the third training sample and the gesture reconstructed by the first gesture generation model.

[0096] Specifically, the first gesture generation model is used to generate representative gestures corresponding to audio files, and the corresponding training methods include:

[0097] The first position sequence signal after the above marking is used to extract the elements marked as having an action, and these elements are used as the second training samples. The extracted second training samples are then normalized to obtain a third training sample of uniform length, where the uniform length is L, and L is a positive integer, usually between 50 and 100.

[0098] The third training sample is input into the first gesture generation model to obtain a reconstructed representative gesture. The training of the first gesture generation model is complete when the second loss function, determined by the real gesture corresponding to the third training sample and the gesture reconstructed by the first gesture generation model, converges. Thus, any audio file input into the first gesture generation model can output a reconstructed representative gesture.

[0099] Optionally, before uniformly sampling the second training sample to obtain a third training sample of uniform length, the method further includes:

[0100] Determine whether there is any missing key point data in each sample of the second training sample;

[0101] If any training sample in the second training sample has missing key point data, it is repaired by rotating and translating adjacent training samples.

[0102] Specifically, the samples in the second training sample mentioned above may have lost key point data due to self-occlusion, subtitle occlusion, etc., which will affect the representative gesture modeling. This invention can also repair the key points of the hand skeleton in the representative gesture. Specifically, by using the complete 21 key points of the hand skeleton, the number of key points of the hand in the second training sample is detected to find samples with missing key point data, such as frame F. i Then use the F from the previous frame. i-1 As filler data, it is calculated from adjacent frames (frame F) i and frame F i-1 The relative rotation R i-1→i Horizontal movement T i-1→i Perform gesture filling, i.e., F i =F i-1 *R i-1→i +T i-1→i The relative rotation R here i-1→i Mainly refers to frame F i-1 To frame F i The required rotation angle and translation T i-1→i Mainly refers to frame F i-1 Move to frame F i The difference in position coordinates is used. The repaired samples are then used to train the first gesture generation model, making the representative gestures generated by the model more accurate. Furthermore, clustering algorithms can be used to perform cluster analysis on the generated representative gestures. For example, the K-means algorithm can be used to perform cascaded clustering of the obtained representative gestures, with the cluster center set to 15, resulting in 15 different types of representative gestures. By configuring the priority of selecting different types of representative gestures in the preset synthesis rules, the dimensionality and flexibility of control over the generated digital human gestures are further improved.

[0103] Optionally, the training method corresponding to the second gesture generation model includes:

[0104] The first audio signal and the first position sequence signal are determined as the fourth training sample;

[0105] Based on the fourth training sample, if the third loss function is convergent, the training of the second gesture generation model is completed. The third loss function is determined based on the positions of each key point in the fourth training sample and the positions of each key point predicted by the second gesture generation model.

[0106] Specifically, a one-dimensional convolutional model in the form of U-NET is used for rhythmic motion modeling, that is, a second gesture generation model is constructed, which converts the audio signal into spectral features S through computation. i The prosodic features are obtained after being input into the audio coding model; the prosodic features are then input into the rhythmic gesture generation model M. P In this process, the corresponding predicted rhythmic gestures are obtained.

[0107] A third loss function is determined based on the positions of keypoints in the fourth training sample and the positions of keypoints predicted by the second gesture generation model. Utilizing the difference between the predicted rhythmic gestures and the actual gestures, the parameters of the second gesture generation model are adjusted through gradient backpropagation. Training of the second gesture generation model is complete when the third loss function converges. The third loss function of gradient backpropagation includes: a first part is the absolute distance function between the actual gesture and the predicted rhythmic gesture; a second part is the first-order absolute distance function between the actual gesture and the predicted rhythmic gesture; and a third part is the second-order absolute distance function between the actual gesture and the predicted rhythmic gesture. Specifically, this third loss function L is expressed in higher-order derivative form. Beat ,Right now in, Let represent the actual gesture, the first derivative of the actual gesture, and the second derivative of the actual gesture, respectively. Let represent the predicted rhythmic gesture, the first derivative of the predicted rhythmic gesture, and the second derivative of the predicted rhythmic gesture, respectively.

[0108] Optionally, the method further includes:

[0109] After the script generation model and the gesture generation model are trained, the target loss function is determined based on the first loss function and the third loss function.

[0110] If the target loss function converges, fine-tuning of the script generation model, the first gesture generation model, and the second gesture generation model is completed.

[0111] Specifically, the present invention addresses the above-mentioned script generation model M. A The first gesture generation model M R The second gesture generation model M P After training each separately, all three are trained together for fine-tuning. The loss function L is used during fine-tuning. Total That is, when the action script indicates the existence of a gesture action (n=1), the network uses the third loss function L. Beat and the first loss function L Script As the aforementioned loss function L Total When the action script indicates that there is no gesture action (n=0), the network uses the first loss function L. Script As the aforementioned loss function L Total , can be represented as:

[0112]

[0113] The target audio files are input into the trained script generation model M. R The predicted action script n is input into the trained rhythmic gesture generation model M. P The rhythmic gesture P is obtained from B The input is fed into the trained representative gesture generation model M. R The representative gesture P was obtained from the middle R Finally, through the action script n and the preset synthesis rules, the representative gesture P is synthesized. B Harmony rhythmic gesture P R The resulting digital human gesture G, formed by fusion, can be represented as:

[0114]

[0115] The above-mentioned preset synthesis rules are for illustrative purposes only and are not intended to limit the specific synthesis rules in this invention. The preset synthesis rules can be set and adjusted according to actual needs.

[0116] The aforementioned digital human gesture G utilizes graphics software to achieve the effect of digital human motion-driven operation.

[0117] The digital human gesture generation method provided by this invention effectively controls the synthesis of digital human gestures under synchronized speech by using the action sequence corresponding to the target audio file determined by the script generation model. The gestures are decoupled and modeled to obtain a representative gesture generation model and a rhythmic gesture generation model. By combining the representative gestures and rhythmic gestures obtained by the gesture generation models, more natural and richer gestures can be generated, making the digital human gestures more realistic.

[0118] To test the effectiveness of the model, it was applied to a test set. The time accuracy F of directly using rest gestures to drive the digital human was measured. Precision Diversity F Diversity The fidelity (FID) is shown in the second row of Table 1. The results of the digital human driven by consistently defined gestures under the three evaluation metrics are shown in the third row of Table 1. The results of the digital human driven by random gestures under the three evaluation metrics are shown in the fourth row of Table 1. The results of the digital human driven by gestures generated using a deep learning model under the three evaluation metrics are shown in the fifth row of Table 1. Finally, using a pre-trained model for controllable generation of digital human gestures under synchronized speech to generate the final gestures of the digital human from the audio signal, and then using these gestures to drive the digital human, significant improvements were made in temporal accuracy, gesture diversity, and gesture realism, demonstrating that the generated digital human gestures have a high-quality visual effect.

[0119] Table 1 shows the time accuracy F of digital humans driven by resting gestures, continuous gestures, random gestures, gestures generated by deep learning models, and gestures generated after invention processing. Precision Diversity F Diversity Comparison results of FID authenticity:

[0120] Table 1

[0121]

[0122] To more clearly illustrate the method for generating digital human gestures provided in the embodiments of the present invention, specific examples are given below.

[0123] Figure 2 This is a schematic diagram illustrating the specific implementation process of digital human gesture generation provided in an embodiment of the present invention, as shown below. Figure 2 As shown, it includes:

[0124] Step S1: Collect and prepare training data;

[0125] Professional Chinese news commentary videos from the Internet were collected as the first video file, and the multimodal data represented by the first video file was labeled through appropriate processing as the initial training samples.

[0126] Specifically, the corresponding audio file is obtained from the first video file, and the audio is converted into Mel-spectrum features as the audio signal. Then, text signals with time information are extracted from the first video file. These text signals are converted into text features using a Chinese text pre-training model, which employs a Long Short-Term Memory (LSTM) network structure. Specifically, extracting the text signals with time information from the first video file involves aligning the text and audio signals using a speech forced aligner (MFA) tool to obtain text signals with time information, i.e., text annotations. These text annotations are then processed through the Chinese pre-training model to obtain text features, which are then used as the text signal. Furthermore, the Mediapipe tool is used to extract the sequence signals of key points on the human skeleton.

[0127] The audio signal, text signal, and human skeleton key point position sequence signal were used as the initial training samples.

[0128] Step S2: Sample acquisition for script-generated models (action script extraction);

[0129] The first step is to describe a complete gesture as starting from a resting position (starting position), going through several stages: resting position (starting position), preparation, emphasis, holding, and withdrawal, before returning to another new resting position (starting position). The second step is to simplify the gesture using the positions of four skeletal points on the human body: the left elbow, right elbow, left wrist, and right wrist. A histogram of the positions appearing at these four skeletal points is then compiled, and the most frequently occurring position is identified as the resting position gesture P. * The third step is to determine the current frame simplified gesture and P of the human skeleton key point position sequence signal. * The distance between them is represented by D. tr The distance between the simplified gesture in the current frame and the next frame of the human skeleton key point position sequence signal is calculated as D. tp Frames containing hand gestures are selected based on a preset threshold. The specific process is as follows:

[0130]

[0131] Where n is the value of the action script. When the flag value is 1, it means that an action has occurred at this position; when the flag value is 0, it means that no action has occurred at this position.

[0132] The first 24 and last 25 frames of the audio signal in the initial training samples are used as the audio signal of each frame of data during training, and are used as the training samples of the script to generate the model, i.e., the first training samples.

[0133] Step S3: Extract representative gestures;

[0134] Gesture fragments with action script 1 were extracted from the initial training samples. Each fragment contains the complete stage of the gesture. Due to self-occlusion, subtitle occlusion, and other reasons, the hand skeleton keypoint sequence is missing, which affects the representative gesture modeling. By detecting the number of 42 hand skeleton keypoints, the first frame F with missing hand was identified. i Then use the F from the previous frame. i-1 As the data to be de-padded, the relative rotation R between adjacent frames is calculated. i-1→i Horizontal movement T i-1→i Perform gesture filling, i.e., F i =F i-1 *R i-1→i +T i-1→i The restored representative gesture sequences were uniformly sampled to ensure that each representative gesture contained 50 frames. The restored representative gestures were clustered, and the video key action recognition pre-trained model was used to extract the video features of the representative gestures. The extracted features were then clustered according to the K-means algorithm, with the cluster center set to 15. Finally, 15 different types of representative gestures were obtained, which served as the third training samples for the first gesture generation model.

[0135] Step S4: Extraction of rhythmic gesture samples;

[0136] The extracted audio signal is converted into spectral features S through calculation. i The spectral features are input into the audio coding model to obtain prosodic features, which are used as rhythmic gesture samples.

[0137] Step S5: Generate the model using the script;

[0138] The script generation model training process includes: constructing a script generation model using a fully convolutional ResNet-101 network structure; inputting the first training sample from step S2 above into the constructed script generation model; outputting the predicted action script; determining the first loss function based on the actual action and the predicted action; and completing the training of the script generation model when the first loss function converges.

[0139] Step S6: First gesture generation model;

[0140] The training process of the first gesture generation model includes: constructing a representative gesture model based on a variational autoencoder. This variational autoencoder (including an encoder and a decoder) can generate data similar to the original distribution by constructing a hidden vector layer. The third training sample in step S2 is then processed according to... The input is fed into the representative gesture generation model mentioned above; where, Let (49,2) represent any segment in the third training sample. In (49,2), 49 represents 42 hand skeletal keypoints and 7 arm and shoulder skeletal keypoints, and 2 indicates that the corresponding position coordinates are two-dimensional. A Gaussian distribution N(μ,δ) is learned through the encoder in the representative gesture generation model, approximating a standard normal distribution; the corresponding gesture is then reconstructed by the decoding network. The loss function used during network training is the reconstruction loss L. Cons and divergence loss function (Kullback-Leibler Divergence, KL) KL , can be represented as:

[0141]

[0142] Among them, L Cons Let represent the reconstruction loss function, m represent the total sequence length corresponding to the gesture, and i represent any element in the gesture sequence. This represents the representative gesture corresponding to training sample i; L represents the i-th representative gesture predicted or reconstructed by the first gesture generation model. KL Let denot be the divergence loss function, δ represent the variance of the probability distribution corresponding to the first gesture generation model, and μ represent the mean of the probability distribution corresponding to the first gesture generation model.

[0143] In the aforementioned reconstruction loss L Cons and KL loss function L KL If the determined second loss function converges, the training of the first gesture generation model is completed.

[0144] Step S7: Second gesture generation model;

[0145] The training process of the first gesture generation model includes: modeling the second gesture generation model using a one-dimensional convolutional model in the form of U-NET; and determining the third loss function L using the difference between the predicted gesture and the real gesture. Beat The parameters of the rhythmic gesture generation model are adjusted through gradient backpropagation, thereby enabling learning. The third loss function used for gradient backpropagation includes: a first part is the absolute distance function between the real and predicted gestures; a second part is the first-order absolute distance function between the real and predicted gestures; and a third part is the second-order absolute distance function between the real and predicted gestures. This third loss function L... Beat It is expressed in the form of higher-order derivatives, specifically as follows:

[0146] in For real gesture information, For predicting rhythmic gestures.

[0147] Step S8, ActionScript;

[0148] Based on the action script obtained in step S2, the script generation model in step S5 is trained, and the trained script generation model outputs the action script corresponding to the audio file.

[0149] Step S9, representative gesture;

[0150] Based on the representative gestures obtained in step S3, the first gesture generation model in step S6 is trained, and the trained first gesture generation model outputs the representative gestures corresponding to the audio file.

[0151] Step S10, rhythmic gestures;

[0152] Based on the representative gestures obtained in step S4, the second gesture generation model in step S7 is trained, and the trained second gesture generation model outputs the representative gestures corresponding to the audio file.

[0153] Step S11, Digital Human Gesture;

[0154] The action script obtained in step S8 and the preset synthesis rules are used to fuse the representative gesture obtained in step S9 and the rhythmic gesture obtained in step S10 to obtain the final digital human gesture. The preset synthesis rules are used to specify that when a gesture action exists, the digital human gesture is synthesized from one or a combination of the representative gesture and the rhythmic gesture at the same moment; and when no gesture action exists, the digital human gesture is determined by the representative gesture, the rhythmic gesture, or other fixed preset gesture at the same moment. Here, the fixed preset gesture can be one selected from all gestures.

[0155] Step S12: Digital human motion-driven display;

[0156] The digital human gestures obtained from step S11 are used to achieve a display effect driven by digital human motion using graphics software.

[0157] This invention proposes a method for controllable digital human gesture generation. By training a script generation model, an action script can be predicted from audio and text, and gestures can be generated in a controlled manner based on the predicted action script. Gestures are decoupled into representative gestures and rhythmic gestures, and corresponding models are trained for each to model the gestures. The representative gesture generation model generates professional representative gestures by sampling from a normal distribution space after training a variational autoencoder. The rhythmic gesture model enriches the gesture display, making it more vivid. The generation of digital human gestures is controlled by the action scripts obtained from the script generation model, using both representative and rhythmic gestures. This allows for more flexible control of the gesture synthesis process, resulting in more natural and richer digital human gestures.

[0158] Figure 3 This is a schematic diagram of the structure of the digital human gesture generation device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the device includes:

[0159] The acquisition module 301 is used to acquire the target audio file of the digital human gesture to be generated;

[0160] The determination module 302 is used to determine the action occurrence sequence corresponding to the target audio file based on the script generation model; the action occurrence sequence is used to indicate whether a gesture action exists at any given time.

[0161] The generation module 303 is used to control the generation of representative gestures and rhythmic gestures to synthesize them into digital human gestures corresponding to the target audio file based on the action sequence and the gesture generation model.

[0162] The gesture generation model includes a first gesture generation model and a second gesture generation model. The first gesture generation model is used to generate representative gestures, and the second gesture generation model is used to generate rhythmic gestures.

[0163] Optionally, in the process of synthesizing representative gestures and rhythmic gestures generated based on the action sequence and gesture generation model into digital human gestures corresponding to the target audio file, the generation module 303 is specifically used for:

[0164] Based on the first gesture generation model, a representative gesture corresponding to the target audio file is generated;

[0165] Based on the second gesture generation model, rhythmic gestures corresponding to the target audio file are generated;

[0166] Based on the sequence of actions and a preset synthesis rule, the representative gesture and the rhythmic gesture are fused to obtain the digital human gesture corresponding to the target audio file; the preset synthesis rule is used to limit the digital human gesture at any time to be determined based on any one or a combination of the representative gesture and the rhythmic gesture.

[0167] The digital human gesture generation apparatus provided in this embodiment of the invention can execute the technical solution of the digital human gesture generation method in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the digital human gesture generation method. Please refer to the implementation principle and beneficial effects of the digital human gesture generation method. It will not be repeated here.

[0168] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a method for generating digital human gestures, the method including:

[0169] Obtain the target audio file for the digital human gestures to be generated;

[0170] Based on a script generation model, the action sequence corresponding to the target audio file is determined; the action sequence is used to indicate whether a gesture action exists at any given time.

[0171] Based on the action sequence and gesture generation model, the generated representative gestures and rhythmic gestures are synthesized into digital human gestures corresponding to the target audio file.

[0172] The script generation model is trained based on training samples determined from a first video file containing speech and motion information; the gesture generation model includes a first gesture generation model and a second gesture generation model, wherein the first gesture generation model is used to generate representative gestures and the second gesture generation model is used to generate rhythmic gestures.

[0173] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the digital human gesture generation method provided by the above methods, the method comprising:

[0175] Obtain the target audio file for the digital human gestures to be generated;

[0176] Based on a script generation model, the action sequence corresponding to the target audio file is determined; the action sequence is used to indicate whether a gesture action exists at any given time.

[0177] Based on the action sequence and gesture generation model, the generated representative gestures and rhythmic gestures are synthesized into digital human gestures corresponding to the target audio file.

[0178] The script generation model is trained based on training samples determined from a first video file containing speech and motion information; the gesture generation model includes a first gesture generation model and a second gesture generation model, wherein the first gesture generation model is used to generate representative gestures and the second gesture generation model is used to generate rhythmic gestures.

[0179] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for generating digital human gestures provided by the methods described above, the method comprising:

[0180] Obtain the target audio file for the digital human gestures to be generated;

[0181] Based on a script generation model, the action sequence corresponding to the target audio file is determined; the action sequence is used to indicate whether a gesture action exists at any given time.

[0182] Based on the action sequence and gesture generation model, the generated representative gestures and rhythmic gestures are synthesized into digital human gestures corresponding to the target audio file.

[0183] The script generation model is trained based on training samples determined from a first video file containing speech and motion information; the gesture generation model includes a first gesture generation model and a second gesture generation model, wherein the first gesture generation model is used to generate representative gestures and the second gesture generation model is used to generate rhythmic gestures.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0185] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating digital human gestures, characterized in that, include: Obtain the target audio file for the digital human gestures to be generated; Based on the script generation model, the action sequence corresponding to the target audio file is determined; The sequence of actions is used to indicate whether a gesture action exists at any given time; Based on the action sequence and gesture generation model, the generated representative gestures and rhythmic gestures are synthesized into digital human gestures corresponding to the target audio file. The script generation model is trained based on training samples determined from a first video file containing speech and motion information. The gesture generation model includes a first gesture generation model and a second gesture generation model. The first gesture generation model is used to generate representative gestures, and the second gesture generation model is used to generate rhythmic gestures. The step of synthesizing representative and rhythmic gestures generated based on the action sequence and gesture generation model into digital human gestures corresponding to the target audio file includes: Based on the first gesture generation model, a representative gesture corresponding to the target audio file is generated; Based on the second gesture generation model, rhythmic gestures corresponding to the target audio file are generated; Based on the sequence of actions and a preset synthesis rule, the representative gesture and the rhythmic gesture are fused to obtain the digital human gesture corresponding to the target audio file; the preset synthesis rule is used to limit the digital human gesture at any time to be determined based on any one or a combination of the representative gesture and the rhythmic gesture.

2. The method for generating digital human gestures according to claim 1, characterized in that, The script generation model is trained based on training samples determined from a first video file containing speech and motion information. The corresponding training methods include: The first N elements and the last M elements of the current element in the initial training sample are determined as one element in the first training sample; the initial training sample is a training sample used to characterize whether there is action in the first video file at different times; the first video file includes voice information and action information; M and N are positive integers; Based on the first training sample, if the first loss function is convergent, the training of the script generation model is completed; the first loss function is determined based on the first label corresponding to the first training sample and the prediction result output by the script generation model after the first training sample is input into the script generation model.

3. The method for generating digital human gestures according to claim 2, characterized in that, The initial training samples are training samples used to characterize whether action occurs in the first video file at different times, and the corresponding acquisition method includes: Based on the first video file, a first position sequence signal is obtained; the first position sequence signal is used to characterize the positions of all key points of the human hand bones, body bones and facial bones at different times; Determine the starting position corresponding to any gesture represented by the first position sequence signal; Based on the distance between the target element and the starting position in the first position sequence, and the distance between the next element of the target element in the first position sequence and the starting position, the target element is labeled with a first label and used as the initial training sample; the target element is any element in the first position sequence; the first label is used to characterize whether an action has occurred in the target element.

4. The method for generating digital human gestures according to claim 3, characterized in that, Determining the starting position corresponding to any gesture represented by the first position sequence signal includes: Based on the preset simplified gesture key points, the first position sequence signal is simplified, and the position histogram of the simplified gesture key points is statistically obtained. According to the preset complete gesture format, the position that appears most frequently in the histogram is determined as the starting position corresponding to any gesture represented by the first position sequence signal.

5. The method for generating digital human gestures according to claim 3, characterized in that, The training methods corresponding to the first gesture generation model include: Based on the first label, extract samples from the initial training samples that represent the occurrence of actions, and use them as the second training samples; The second training sample is uniformly sampled to obtain a third training sample of uniform length; each sample in the third training sample includes L uniformly sampled second training samples; where L is a positive integer; Based on the third training sample, if the second loss function is convergent, the training of the first gesture generation model is completed. The second loss function is determined based on the labels of the third training sample and the gesture reconstructed by the first gesture generation model.

6. The method for generating digital human gestures according to claim 5, characterized in that, Before uniformly sampling the second training sample to obtain a third training sample of uniform length, the method further includes: Determine whether there is any missing key point data in each sample of the second training sample; If any training sample in the second training sample has missing key point data, it is repaired by rotating and translating adjacent training samples.

7. A device for generating digital human gestures, characterized in that, include: The acquisition module is used to acquire the target audio file for the digital human gestures to be generated; The determination module is used to determine the action sequence corresponding to the target audio file based on the script generation model; the action sequence is used to indicate whether a gesture action exists at any given time. The generation module is used to control the generation of representative gestures and rhythmic gestures to synthesize the digital human gestures corresponding to the target audio file based on the action sequence and the gesture generation model. The script generation model is trained based on training samples determined from a first video file containing speech and motion information. The gesture generation model includes a first gesture generation model and a second gesture generation model. The first gesture generation model is used to generate representative gestures, and the second gesture generation model is used to generate rhythmic gestures. The generation module is specifically used for: Based on the first gesture generation model, a representative gesture corresponding to the target audio file is generated; Based on the second gesture generation model, rhythmic gestures corresponding to the target audio file are generated; Based on the action sequence and preset synthesis rules, the representative gestures and the rhythmic gestures are fused to obtain the digital human gestures corresponding to the target audio file; The preset synthesis rules are used to limit the digital human gestures at any time to be determined based on any one or a combination of the representative gestures and the rhythmic gestures.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for generating digital human gestures as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for generating digital human gestures as described in any one of claims 1 to 6.

Images