Limb movement driving method and device of digital human and storage medium

Abstract

The invention provides a body movement driving method and device of a digital human and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring to-be-played voice and a text corresponding to the to-be-played voice; dividing the text into multiple segments of target texts according to the punctuations, and dividing the voice to be played into target voice corresponding to each segment of target text; for a target text in the multiple segments of target texts, generating an action frame sequence corresponding to the target voice and the target text according to the target text and the target voice corresponding to the target text; and in the process of playing the multiple segments of target voices, playing synchronization is carried out on the target voices played before and after and the corresponding action frame sequences, and the digital person is driven in real time. According to the method, the technical effect of improving the language and action coherence of the digital human is achieved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, device and storage medium for driving the limb movements of a digital human. Background Technology

[0002] With the development of metaverse technology, digital humans have entered the public eye. A digital human is a technological product that uses computer digital modeling and graphics processing techniques, combined with graphics rendering methods, to construct a virtual human image. Through the integration of high-precision technologies such as motion capture and facial expression simulation, it can reproduce the expressions and demeanor of a real person in real time.

[0003] In related technologies, digital humans are typically driven by outputting speech and body movements at a fixed frame rate. In some scenarios, there is a problem of speech and movement being out of sync. Summary of the Invention

[0004] This application provides a method, device, and storage medium for driving the limb movements of a digital human, in order to solve the technical problems of poor continuity and asynchrony of language and movements in existing digital humans.

[0005] In a first aspect, this application provides a method for driving the limb movements of a digital human, the method comprising:

[0006] Get the audio to be played and its corresponding text;

[0007] The text is divided into multiple target text segments based on punctuation marks, and the audio to be played is segmented into target audio corresponding to each target text segment.

[0008] For target text in multiple target text segments, generate action frame sequences for the corresponding target speech and target text based on the target text and the target speech corresponding to the target text;

[0009] During the playback of multiple target audio segments, the playback of the target audio segments played before and after and the corresponding action frame sequences are synchronized to drive the digital human in real time.

[0010] Optionally, the target speech and corresponding action frame sequences played before and after are synchronized to drive the digital human in real time, including:

[0011] Each segment of target speech corresponds to an action frame sequence including a start action frame sequence, multiple intermediate action frame sequences, and an end action frame sequence.

[0012] Determine whether the ending action frame sequence in the action frame sequence corresponding to the first target text is a static action; if so, determine the percentage information of the ending action frame sequence in the action frame sequence, and perform frame interpolation processing on the ending action frame sequence according to the percentage information.

[0013] If not, continue to determine whether there is a static action in the action frame sequence corresponding to other target texts after the first target text. Until a static action is found, the first target text and the action frame sequence before the static action appear are spliced ​​together. When the action frame sequence plays to the static action, the motion data of the initial action is added to the static action until the corresponding audio is played.

[0014] Optionally, based on the target text and its corresponding target speech, an action frame sequence is generated for the target speech and target text, including:

[0015] Based on the chronological order, the target text and the target speech corresponding to the target text are input into the limb motion generation model to generate limb motions, resulting in the corresponding input target speech and target text motion frame sequence. The motion frames contained in the motion frame sequence represent the rotation angle of each limb joint.

[0016] Optionally, the limb motion generation model includes an arm motion generator and a finger motion generator. The target text and its corresponding target speech are input into the limb motion generation model to generate limb motions, resulting in a sequence of motion frames corresponding to the input target speech and target text, including:

[0017] Input the target text and the target speech corresponding to the target text into the arm motion generator to obtain the corresponding arm motion frame sequence of the input target speech and target text;

[0018] Input the target text, the target speech corresponding to the target text, and the arm sequence frame sequence into the finger action generator to obtain the corresponding finger action frame sequence of the input target speech and target text.

[0019] The action frame sequences corresponding to the target speech and target text are determined based on the arm frame sequence and the finger frame sequence.

[0020] Alternatively, the limb motion generation model is trained in the following way:

[0021] Obtain training samples and the corresponding actual action frame sequences; whereby the training sample information includes training speech and training text corresponding to the training language;

[0022] The training samples and actual action frame sequences are input into the adversarial generative network. In the adversarial generative network, the training samples and actual action frame sequences are processed by sliding windows based on a set starting offset. The resulting window data is recorded in the replay buffer.

[0023] The first window data corresponding to the training samples is sampled from the replay buffer, and the predicted action frame sequence corresponding to the first window data is obtained through the action generator. The predicted action frame sequence is recorded in the replay buffer and the memory unit. The memory unit can store the training samples and the corresponding predicted action frame sequence, as well as the target text and the corresponding action frame sequence.

[0024] The second window data corresponding to the actual action frame sequence is sampled from the replay buffer, and the reconstructed action frame sequence corresponding to the second window data is obtained through the action feature network.

[0025] The training loss value is determined based on the predicted action frame sequence, the actual action frame sequence, and the reconstructed action frame sequence.

[0026] Adjust the model parameters of the action generator and / or action feature network based on the training loss value;

[0027] Once training is complete, the trained motion generator will be designated as the limb motion generation model.

[0028] Alternatively, the training samples and actual action frame sequences are obtained in the following way:

[0029] Collect action sets and audio files from recorded motion data, and determine multiple basic actions and static actions based on the action sets;

[0030] Determine the training text, determine the training speech based on the training text, and align the training text and the corresponding training speech according to the CTC caption alignment algorithm;

[0031] The training action frame sequence corresponding to the training sample is synthesized based on the training text, training speech, multiple basic actions, and static actions.

[0032] A second aspect of this application provides a digital human limb movement driving device, comprising:

[0033] The acquisition module is used to acquire the audio to be played and the corresponding text.

[0034] The first processing module is used to divide the text into multiple target text segments according to punctuation marks, and to segment the audio to be played into the target audio corresponding to each target text segment.

[0035] The second processing module is used to generate action frame sequences of target speech and target text based on the target text and the target speech corresponding to the target text in multiple target text segments.

[0036] The third processing module is used to synchronize the playback of the target audio and the corresponding action frame sequence played before and after the playback of multiple target audio segments, thereby driving the digital human in real time.

[0037] Optionally, the third processing module is also used for:

[0038] Each segment of target speech corresponds to an action frame sequence including a start action frame sequence, multiple intermediate action frame sequences, and an end action frame sequence.

[0039] Determine whether the ending action frame sequence in the action frame sequence corresponding to the first target text is a static action; if so, determine the percentage information of the ending action frame sequence in the action frame sequence, and perform frame interpolation processing on the ending action frame sequence according to the percentage information.

[0040] If not, continue to determine whether there is a static action in the action frame sequence corresponding to other target texts after the first target text. Until a static action is found, the first target text and the action frame sequence before the static action appear are spliced ​​together. When the action frame sequence plays to the static action, the motion data of the initial action is added to the static action until the corresponding audio is played.

[0041] Optionally, the second processing module is also used for:

[0042] Based on the chronological order, the target text and the target speech corresponding to the target text are input into the limb motion generation model to generate limb motions, resulting in the corresponding input target speech and target text motion frame sequence. The motion frames contained in the motion frame sequence represent the rotation angle of each limb joint.

[0043] Optionally, the second processing module is also used for:

[0044] Input the target text and the target speech corresponding to the target text into the arm motion generator to obtain the corresponding arm motion frame sequence of the input target speech and target text;

[0045] Input the target text, the target speech corresponding to the target text, and the arm sequence frame sequence into the finger action generator to obtain the corresponding finger action frame sequence of the input target speech and target text.

[0046] The action frame sequences corresponding to the target speech and target text are determined based on the arm frame sequence and the finger frame sequence.

[0047] Optionally, the second processing module is also used for:

[0048] Obtain training samples and the corresponding actual action frame sequences; whereby the training sample information includes training speech and training text corresponding to the training language;

[0049] The training samples and actual action frame sequences are input into the adversarial generative network. In the adversarial generative network, the training samples and actual action frame sequences are processed by sliding windows based on a set starting offset. The resulting window data is recorded in the replay buffer.

[0050] The first window data corresponding to the training samples is sampled from the replay buffer, and the predicted action frame sequence corresponding to the first window data is obtained through the action generator. The predicted action frame sequence is recorded in the replay buffer and the memory unit. The memory unit can store the training samples and the corresponding predicted action frame sequence, as well as the target text and the corresponding action frame sequence.

[0051] The second window data corresponding to the actual action frame sequence is sampled from the replay buffer, and the reconstructed action frame sequence corresponding to the second window data is obtained through the action feature network.

[0052] The training loss value is determined based on the predicted action frame sequence, the actual action frame sequence, and the reconstructed action frame sequence.

[0053] Adjust the model parameters of the action generator and / or action feature network based on the training loss value;

[0054] Once training is complete, the trained motion generator will be designated as the limb motion generation model.

[0055] Optionally, the acquisition module is also used for:

[0056] Collect action sets and audio files from recorded motion data, and determine multiple basic actions and static actions based on the action sets;

[0057] Determine the training text, determine the training speech based on the training text, and align the training text and the corresponding training speech according to the CTC caption alignment algorithm;

[0058] The training action frame sequence corresponding to the training sample is synthesized based on the training text, training speech, multiple basic actions, and static actions.

[0059] Thirdly, this application provides a digital human limb movement driving device, including: a processor, and a memory communicatively connected to the processor, including:

[0060] The memory stores the instructions that the computer executes;

[0061] The processor executes computer execution instructions stored in memory to implement the digital human's limb movement driving method of the first aspect.

[0062] Fourthly, a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the limb movement driving method of the digital human of the first aspect.

[0063] Fifthly, a computer program product includes a computer program that, when executed by a processor, implements the method for driving the limb movements of a digital human in the first aspect.

[0064] This application provides a method, device, and storage medium for driving the limb movements of a digital human, relating to the field of artificial intelligence technology. The method includes: acquiring a speech to be played and its corresponding text; dividing the text into multiple target text segments based on punctuation, and segmenting the speech to be played into target speech corresponding to each target text segment; for each target text segment, generating a sequence of action frames for the corresponding target speech and target text based on the target text and its corresponding target speech; and synchronizing the playback of the target speech and its corresponding action frame sequence during the playback of the multiple target speech segments to drive the digital human in real time. The method of this application achieves the technical effect of improving the coherence of language and movements in digital humans. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0066] Figure 1 The flowchart of the digital human limb movement driving method provided in the embodiments of this application Figure 1 ;

[0067] Figure 2 The flowchart of the digital human limb movement driving method provided in the embodiments of this application Figure 2 ;

[0068] Figure 3 A schematic diagram of the limb movement driving method for digital humans provided in this application embodiment. Figure 1 ;

[0069] Figure 4 A schematic diagram of the limb movement driving method for digital humans provided in this application embodiment. Figure 2 ;

[0070] Figure 5 The flowchart of the digital human limb movement driving method provided in the embodiments of this application Figure 3 ;

[0071] Figure 6 A schematic diagram of the limb movement driving method for digital humans provided in this application embodiment. Figure 4 ;

[0072] Figure 7 A schematic diagram of the limb movement driving method for digital humans provided in this application embodiment. Figure 5 ;

[0073] Figure 8 A schematic diagram of the limb movement driving method for digital humans provided in this application embodiment. Figure 6 ;

[0074] Figure 9 A schematic diagram of the limb movement driving method for digital humans provided in this application embodiment. Figure 7 ;

[0075] Figure 10 A schematic diagram of the limb movement driving method for digital humans provided in this application embodiment. Figure 8 ;

[0076] Figure 11 This is a schematic diagram of the structure of a cross-device login device provided in an embodiment of this application;

[0077] Figure 12 This is a hardware structure diagram of a cross-device login device provided in an embodiment of this application.

[0078] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0079] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0080] Because the output of motion frame sequences of digital humans in existing technologies is usually at a fixed frame rate, while in practical applications the frame rate may be dynamic and there may be intervals between each speech segment, this can lead to problems such as motion stuttering and disjointed speech-motion.

[0081] This application provides a method, device, and storage medium for driving the limb movements of a digital human. The method includes: acquiring the speech to be played and the corresponding text; dividing the text into multiple target text segments based on punctuation, and segmenting the speech to be played into target speech corresponding to each target text segment; generating a sequence of action frames for the target speech and target text in the multiple target text segments based on the target text and the target speech corresponding to the target text; and synchronizing the playback of the target speech and the corresponding action frame sequence during the playback of the multiple target speech segments to drive the digital human in real time. This automatically divides the text into multiple paragraphs based on punctuation marks, and correspondingly segments the speech to be played into speech segments corresponding to each target text segment. Based on the text content and the corresponding target speech, a series of action frame sequences are generated. During the playback of the multiple target speech segments, the system ensures that the target speech and the corresponding action frame sequence played in real time are consistent through a synchronization mechanism. Thus, when the digital human plays a certain speech segment, it is driven to perform the corresponding action in real time according to the action frame sequence corresponding to that speech segment, thereby achieving the continuity of the digital human's speech movements and achieving a smooth movement effect.

[0082] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0083] First, let me explain the terms used in this application:

[0084] Automatic Speech Recognition (ASR) is a technology service that converts human speech into text.

[0085] Natural Language Processing (NLP) refers to the technology of using computers to process the form, sound, and meaning of natural language, that is, the operation and processing of input, output, recognition, analysis, understanding, and generation of characters, words, sentences, and texts.

[0086] Text-to-Speech (TTS) technology uses computer technology to convert text information into human-understandable speech output.

[0087] The Connectionist Temporal Classification (CTC) algorithm is an algorithm used in fields such as speech recognition and text recognition. It is mainly used to solve the problem of inconsistent input and output sequence lengths and difficulty in alignment.

[0088] 6D rotation representation describes a three-dimensional rotation using six dimensions. This representation method has the characteristics of continuity and no singularity. A common form of 6D rotation representation is to take the six elements from the first two rows or the first three columns of the rotation matrix, but there may be other forms, such as a simplified representation based on quaternions.

[0089] Long Short-Term Memory (LSTM) network generator is a neural network model based on LSTM units used to generate sequential data, such as text and time series.

[0090] A Gated Recurrent Unit (GRU) is a sequence generation model. Compared to LSTM, GRU has a simpler structure with only two gates: an update gate and a reset gate. These two gates jointly control the flow of information, enabling the model to retain important historical information while discarding irrelevant information.

[0091] A Retentive Network (RetNet) is a neural network model that improves inference efficiency by introducing a retention mechanism.

[0092] Non-Linear Animation (NLA track) is a feature in 3D modeling and animation software used to organize and edit animation clips.

[0093] The LSTM (Long Short-Term Memory) layer is a special type of recurrent neural network layer in deep learning, used to process and predict important events with very long intervals and delays in time series data.

[0094] Figure 1 The flow chart of the digital human limb movement driving method provided in the embodiments of this application Figure 1 .like Figure 1 As shown, the digital human limb movement driving method provided in this embodiment includes:

[0095] S101. Obtain the audio to be played and the corresponding text.

[0096] In this embodiment, in the scenario of digital human interaction, the first and crucial step is to accurately acquire the speech to be played and its corresponding text. This process typically begins from multiple sources, including but not limited to user-uploaded audio files, real-time recorded speech streams, or speech synthesized directly from text. When acquiring the speech to be played, ASR technology can be used to convert human speech signals into digital signals, then analyze and process the digital signals, and finally convert them into text information that computers can understand.

[0097] S102. Divide the text into multiple target text segments according to punctuation marks, and divide the audio to be played into target audio corresponding to each target text segment;

[0098] In this embodiment, a text segmentation algorithm from NLP technology can be used to scan and segment the text based on a predefined set of punctuation marks. Optionally, during the segmentation process, the logical relationships between sentences can also be considered to further optimize the segmentation results. Simultaneously, the speech to be played is segmented based on the target text. An ASR (Automatic Speech Reduction) technology can be used to process the continuous speech stream. By analyzing the acoustic features of the speech signal (such as energy, pitch, and frequency) and features such as pauses and rhythmic changes in the speech, sentence boundaries in the speech can be identified. This embodiment divides the text into multiple target text segments, which helps the system understand the text content more accurately. This helps the system generate more accurate and appropriate responses or actions, improving the accuracy and effectiveness of the interaction.

[0099] S103. For target text in multiple target text segments, generate action frame sequences for corresponding target speech and target text based on the target text and the target speech corresponding to the target text.

[0100] In this embodiment, a series of action frame sequences that precisely match each target text paragraph can be generated, thereby achieving a high degree of synchronization and consistency between visual and auditory content.

[0101] S104. During the playback of multiple target audio segments, the playback of the target audio segments played before and after and the corresponding action frame sequences are synchronized to drive the digital human in real time.

[0102] In this embodiment, during the playback of multiple target audio segments, a voice-action synchronization mechanism is used to ensure that each target audio segment and its corresponding action frame sequence can drive the digital human's performance instantly and accurately. This method aims to achieve highly fluid audiovisual interaction, enabling the digital human's movements, expressions, and even overall performance to closely match the playing audio content, providing viewers with an immersive experience. Specifically, when multiple target audio segments are played sequentially, the system analyzes the content of each audio segment in real time and quickly retrieves the pre-associated action frame sequence. As the audio plays, the system instantly triggers the corresponding action frame sequence based on the currently playing audio segment. During this process, the system has intelligent adjustment capabilities, fine-tuning the playback speed of the action frame sequence according to the audio to further improve the accuracy and expressiveness of synchronization.

[0103] This application provides a method, device, and storage medium for driving the limb movements of a digital human. The method includes: acquiring the speech to be played and the corresponding text; dividing the text into multiple target text segments based on punctuation, and segmenting the speech to be played into target speech corresponding to each target text segment; generating a sequence of action frames for the target speech and target text in the multiple target text segments based on the target text and the target speech corresponding to the target text; and synchronizing the playback of the target speech and the corresponding action frame sequence during the playback of the multiple target speech segments to drive the digital human in real time. This automatically divides the text into multiple paragraphs based on punctuation marks, and correspondingly segments the speech to be played into speech segments corresponding to each target text segment. Based on the text content and the corresponding target speech, a series of action frame sequences are generated. During the playback of the multiple target speech segments, the system ensures that the target speech and the corresponding action frame sequence played in real time are consistent through a synchronization mechanism. Thus, when the digital human plays a certain speech segment, it is driven to perform the corresponding action in real time according to the action frame sequence corresponding to that speech segment, thereby achieving the continuity of the digital human's speech movements and achieving a smooth movement effect.

[0104] Figure 2 The flow chart of the digital human limb movement driving method provided in the embodiments of this application Figure 2 This embodiment is... Figure 1 Based on the embodiments, the method for driving the limb movements of a digital human is described in detail. In this embodiment, acquiring the target text and target speech is achieved through steps S201-S202. Generating the corresponding action frame sequence of the target speech and target text based on the target text and the target speech corresponding to the target text is achieved through step 3. During the playback of multiple target speech segments, synchronizing the playback of the target speech and the corresponding action frame sequence played before and after is achieved through step 4. Figure 2 As shown, the digital human limb movement driving method provided in this embodiment includes:

[0105] S201. Obtain the voice to be played and the text corresponding to the voice to be played;

[0106] S202. Divide the text into multiple segments of target text according to punctuation marks, and split the voice to be played into target voices corresponding to each segment of target text;

[0107] In this embodiment, Figure 3 is a schematic diagram of a method for driving the body movements of a digital human provided by an embodiment of the present application Figure 1 . As Figure 3 shown, after obtaining multiple segments of target text and corresponding multiple segments of target voices, align the voice and subtitles for each segment of target text and target voice. The alignment method can adopt the CTC algorithm. At the same time, after aligning the voice and subtitles for each segment of target text and target voice, the time interval of each word can be obtained. Further, extract the voice features of the voice through the pre-trained voice unsupervised model Hubert, and extract the features of the text through the text unsupervised model RoBerta. Models such as the voice unsupervised model Hubert and the text unsupervised model RoBerta that have been trained on a large amount of audio and text can effectively extract the features of voice text. Models trained based on these features usually have good generalization performance.

[0108] Specifically, after determining the features of the target text and target voice, since the sequence lengths of the voice features and text features are different, interpolation and alignment need to be performed on the voice features and text features. Among them, the frame rate of the voice feature sequence output by the voice unsupervised model Hubert is 49, and the length of the feature sequence output by the text unsupervised model RoBerta depends on the length of the input text. Therefore, the alignment can be performed according to the following method: preset the output frame rate according to the actual situation. For example, if the frame rate is preset to 20fps, the voice feature sequence (49fps) is downsampled to this rate (20fps). The downsampling method can include but is not limited to the nearest neighbor interpolation algorithm. After calculating the start time of each frame after downsampling, the frame closest to this time point in the original feature sequence can be found as the feature of the new frame. Further, according to the time interval of each word, calculate the number of frames that each word lasts, and extend the text feature sequence to the same output frame rate as well. For example, the time interval of a word "home" is [1.0s, 1.5s], and it lasts for 0.5 seconds, which is 10 frames. Then, the feature vector of the word "home" is repeated 10 times in the time dimension. After aligning the voice features and text features, they can be input into the action generator.

[0109] S203. Based on the chronological order, input the target text and the target speech corresponding to the target text into the arm motion generator to obtain the corresponding arm motion frame sequence of the input target speech and target text; input the target text, the target speech corresponding to the target text, and the arm sequence frame sequence into the finger motion generator to obtain the corresponding finger motion frame sequence of the input target speech and target text; determine the motion frame sequence corresponding to the target speech and target text based on the arm frame sequence and the finger frame sequence.

[0110] In this embodiment, speech features and text features are input into the limb motion generation model to generate limb movements. The limb motion generation model includes an arm motion generator and a finger motion generator. The motion frame sequence contains motion frames representing the rotation angles of various limb joints. The rotation angles are represented using a 6D three-dimensional rotation representation method, specifically a 6D rotation representation of joint points. This method is not only easy to train the model but also easy to convert to other representations, such as Euler angles or quaternions, making it more suitable for driving 3D models.

[0111] Specifically, the action generator can be a sequence generator with memory units, including but not limited to LSTM, GRU or Retentive Network. At the same time, the model's memory units are used to retain information about the previous text and speech as well as the generated action sequence. This ensures that the response time of each segment is within a reasonable range, and the digital human can perform reasoning for the next sentence while speaking the current sentence.

[0112] S204. Determine whether the ending action frame sequence in the action frame sequence corresponding to the first target text is a static action; if so, determine the percentage information of the ending action frame sequence in the action frame sequence, and perform frame interpolation processing on the ending action frame sequence according to the percentage information; if not, continue to determine whether there is a static action in the action frame sequence corresponding to other target texts after the first target text, until a static action is found, then concatenate the first target text and the action frame sequence before the static action appears, and when the action frame sequence plays to the static action, supplement the static action with the initial action action data, until the corresponding audio is played.

[0113] In this embodiment, it is necessary to determine whether the ending action of the action sequence corresponding to each sentence is a stationary action. The determination rule for a stationary action is as follows:

[0114] During the training of a limb motion generator, the motion sequence needs to be standardized (mean 0, variance 1). This is a common step in training neural networks. By using the standardized motion sequence output by the limb motion generator, the standardized static motion is calculated, as shown in the following formula:

[0115]

[0116] Among them, M idle For a standardized action sequence, μ is the mean of the actions in the training data; σ is the standard deviation of the actions in the training data.

[0117] Furthermore, the mean absolute error between the output motion frame sequence and the static motion during the application of the limb motion generator is calculated. A preset threshold is set for the mean absolute error. If the mean absolute error is less than or equal to the preset threshold, the motion is determined to be a static motion. If it is greater than the preset threshold, it is not a static motion.

[0118] Figure 4 A schematic diagram of a digital human limb movement driving method provided in this application embodiment. Figure 2 .like Figure 4 As shown, synthesizing and generating the required speech and actions all at once can result in significant delays. Therefore, this application synthesizes based on punctuation marks. However, when a game engine plays a sentence-by-sentence, there are time intervals between playing one sentence and moving on to the next. The algorithm outputs actions continuously without considering the pauses between sentences. In this case, the frame sequence lacks data for these pauses, leading to stuttering, such as... Figure 4 To address this issue, when the ending action frame sequence in the action frame sequence corresponding to the first target text is determined to be a static action, it is necessary to calculate which frame of the current action should be played based on the speech duration and then perform interpolation. For example, if the speech plays to the 2nd second, the action needs to play the 40th frame (if the action frame rate is 20fps). Therefore, frame interpolation can be performed each time a static action is reached. This ensures continuity without accumulating errors. If the ending action frame sequence in the action frame sequence corresponding to the first target text is determined not to be a static action, then the ending actions in the action frame sequences of other sentences following that sentence are queried until the next static action is reached. The action of that sentence is then concatenated with the queried actions to form an action sequence. When the corresponding audio is played, the action sequence is played without considering pauses, according to the playback frame rate. When the action sequence finishes playing and returns to a static action, the initial action data is supplemented to the digital human until the corresponding audio playback is complete.

[0119] This application discloses a method for driving the limb movements of a digital human. The method involves: acquiring the speech to be played and its corresponding text; dividing the text into multiple target text segments based on punctuation, and segmenting the speech to be played into target speech corresponding to each target text segment; inputting the target text and its corresponding target speech into an arm movement generator based on chronological order to obtain arm movement frame sequences for the input target speech and target text; inputting the target text, its corresponding target speech, and the arm movement frame sequences into a finger movement generator to obtain finger movement frame sequences for the input target speech and target text; and determining the target speech based on the arm and finger frame sequences. The system generates action frame sequences corresponding to the target speech and text. It then determines whether the ending action frame sequence in the first target text's action frame sequence is a static action. If so, it determines the percentage of the ending action frame sequence within the total action frame sequence and performs frame interpolation based on this percentage. If not, it continues to determine whether static actions exist in the action frame sequences corresponding to other target texts after the first target text, until a static action is found. In this case, it concatenates the first target text with the action frame sequences preceding the static action. When the action frame sequence reaches a static action, it supplements the static action with the initial action's motion data until the corresponding audio playback is complete. This achieves automated and intelligent generation of digital human body movements, automatically creating smooth and natural body movements that match the input speech and text content, especially fine arm and finger movements. This not only improves the realism and fluency of digital human interaction.

[0120] Figure 5 The flow chart of the digital human limb movement driving method provided in the embodiments of this application Figure 3 .like Figure 5 As shown, the digital human limb movement driving method provided in this embodiment includes:

[0121] S501: Collect the action set and voice file from the recorded action data, and determine multiple basic actions and static actions based on the action set;

[0122] In this embodiment, Figure 6 A schematic diagram of a digital human limb movement driving method provided in this application embodiment. Figure 3 .like Figure 6As shown, the first step is to collect corpus information. Collection methods on the internet include, but are not limited to, collecting news articles, speeches, introductions, and tour guide scripts, ensuring the collected corpus information meets a preset threshold in terms of word count. Next, actions are designed based on the corpus information, such as waving, shaking hands, making a fist, shaking one's head, and bowing. These actions include, but are not limited to, rhythm-related actions, such as spreading one's hands, or rhythmically chopping to express parallel relationships, such as adding three consecutive chopping motions with the right hand to the three parallel nouns in the sentence "I can be a guide in various venues, scenic spots, and museums." The corpus is then annotated with the designed actions, with the designed actions noted in parentheses after the text. Finally, a portion of the script is recorded according to a preset ratio. For example, if the corpus information contains 100,000 words, only 1 / 20 of the content needs to be recorded, but the recorded content must include all designed action types. Actors wear motion capture equipment for recording. Simultaneously, voice recording is performed using a voice capture device, resulting in motion capture files and corresponding voice files. Specifically, this process involves defining a static pose—the actor must maintain the same posture for any unannotated movement, returning to this static pose after each movement. The recorded motion capture files are imported into 3D animation software for basic motion extraction, resulting in multiple basic movements. Each basic movement is a continuous sequence of actions, beginning and ending in a static state. The extracted basic movements can be logically categorized, with each movement having different rhythms and positions.

[0123] Specifically, during motion capture, although the actor needs to return to a standard still position after each movement, slight variations in the equipment placement can lead to inconsistencies in the data. Therefore, each audio segment needs to be calibrated based on the still position. This can be achieved using the following formula:

[0124]

[0125] Where R is the rotation matrix of the unprocessed action frame sequence; R idle This is the first frame (still motion) of the action sequence; R standardidle A custom standard static motion rotation matrix, where the standard static motion is determined based on the mean of the first frame of all motion sequences; R new This is the standardized sequence of action frames.

[0126] S502. Determine the training text, determine the training speech based on the training text, and align the training text and the corresponding training speech according to the CTC subtitle alignment algorithm.

[0127] In this embodiment, based on the collected corpus information, the corresponding training text samples can be determined. The training text samples are synthesized into corresponding speech using TTS technology. The speech and text are aligned using the CTC subtitle alignment algorithm to obtain the training speech and the training text corresponding to the training language.

[0128] S503. Synthesize the training action frame sequence corresponding to the training sample based on the training text, training speech, multiple basic actions and static actions.

[0129] In this embodiment, it can be referred to Figure 6 The data is synthesized in 3D animation software based on training speech, training text, multiple basic actions, and action annotations on the training text. The specific operations are as follows: import audio, import a sequence of still motion frames, insert the still animation in the non-linear animation editor and make its duration the same as the audio, import basic actions, and insert the basic actions at an appropriate position above the still layer in the non-linear animation editor based on the speech and the annotated actions. Figure 7 A schematic diagram of a digital human limb movement driving method provided in this application embodiment. Figure 4 ,like Figure 7 As shown, the speed of synthesized actions needs to be fine-tuned according to the speech. In the diagram of 3D animation production, the upper layer of NLAtrack will cover the lower layer, and the two slanted sides of the trapezoid represent blend in and blend out, meaning that the animation will automatically transition. The duration of the basic action can be modified according to the actual situation (speech length), or the action curve can be modified to ensure the matching of action and speech.

[0130] S504. Input the training samples and actual action frame sequences into the adversarial generation network. In the adversarial generation network, the training samples and actual action frame sequences are processed by sliding windows based on a set starting offset. The obtained window data is recorded in the replay buffer.

[0131] In this embodiment, since each sequence segment has a different length, batch processing requires filling large gaps in shorter sequences. This problem can be solved by using a sliding window sampling method. The sliding window is a time segment, from the start time to the end time, and corresponding segments are extracted from the original data based on these two time points. The sliding window has a fixed size, and the two windows are closely connected. However, if windowing is only performed once during the data preparation stage, the limited training data cannot be utilized to the maximum extent. Therefore, before each round of training, this application randomly sets the starting offset of the sliding window, records the window data obtained by the new window in the replay buffer, and then samples the training data for that round from the buffer.

[0132] S505. Sample the first window data corresponding to the training samples from the replay buffer, and obtain the predicted action frame sequence corresponding to the first window data through the action generator. Record the predicted action frame sequence into the replay buffer and memory unit.

[0133] In this embodiment, Figure 8 A schematic diagram of a digital human limb movement driving method provided in this application embodiment. Figure 5 , Figure 9 A schematic diagram of a digital human limb movement driving method provided in this application embodiment. Figure 6 ,like Figure 8 and Figure 9 As shown, the first window data is obtained by sampling from the replay buffer, and the first window data is input into the action generator to obtain the corresponding predicted action frame sequence. The predicted action frame sequence is recorded in the replay buffer and memory unit. The memory unit can store training samples and corresponding predicted action frame sequences, as well as target text and corresponding action frame sequences. Specifically, the memory unit is the output of the previous window. For example, the replay buffer of the third window is the memory unit output obtained by inputting the features of the second window into the model, as well as the speech and semantic features and actions of the third window. At the same time, the replay buffer has an upper limit on the number of windows. This upper limit can be preset. If the upper limit is reached after adding more, a certain number of elements are randomly deleted from it so that the number of windows does not exceed the upper limit.

[0134] S506. Sample the second window data corresponding to the actual action frame sequence from the replay buffer, and obtain the reconstructed action frame sequence corresponding to the second window data through the action feature network; determine the training loss value based on the predicted action frame sequence, the actual action frame sequence and the reconstructed action frame sequence.

[0135] In this embodiment, Figure 10 A schematic diagram of a digital human limb movement driving method provided in this application embodiment. Figure 7 ,like Figure 10 As shown, according to the sliding window, the sampled actual action frame sequence is divided into multiple segments of equal length and stored in the action replay buffer. The replay buffer here only contains the sampled actual action frame sequence. During training, a batch of action data is sampled from the buffer and determined as the second window data. The action feature vector is obtained by the transformer encoder, and the discretized action feature vector is obtained by the VQVAE. Finally, the feature vector is used to obtain the reconstructed action frame sequence by the transformer decoder. The training loss value is determined based on the predicted action frame sequence, the actual action frame sequence and the reconstructed action frame sequence.

[0136] S507. Adjust the model parameters of the action generator and / or action feature network according to the training loss value. If the training is completed, determine the trained action generator as the limb action generation model.

[0137] In this embodiment, it can be referred to Figure 8 The training of the limb motion generation model can employ a generative adversarial network (GAN). First-window data corresponding to training samples is sampled from the replay buffer, and a motion generator generates the predicted motion frame sequence corresponding to the first-window data. This predicted motion frame sequence is then recorded in the replay buffer and memory unit. Based on the predicted motion frame sequence, the actual motion frame sequence, and the reconstructed motion frame sequence, the training loss value is determined using the following formula:

[0138] L rec =Huber loss(M pred M true )

[0139] Among them, L rec M represents the training loss value. pred For predicting action frame sequences; M true This is the actual action frame sequence.

[0140] The predicted action frame sequence and the actual action frame sequence are respectively input into the discriminator in the Generative Adversarial Network (GAN) unit. The action discriminator's input is the action frame sequence, and its output is a decimal between 0 and 1; the closer to 1, the more realistic the input action. In this embodiment, the discriminator can consist of four convolutional layers and one LSTM layer. The output of the LSTM is passed through a sigmoid function to obtain the final output. The predicted realism and training realism are obtained, and the adversarial loss value is determined based on these two values, calculated as follows:

[0141] L adv =-mean(log(M) pred ))

[0142] Among them, L adv To counteract the loss value.

[0143] The predicted action frame sequence and the actual action frame sequence are input into the action feature extraction network to obtain the features of the predicted action frame sequence and the actual action frame sequence, respectively. The perceptual loss value is determined based on the features of the predicted action frame sequence and the actual action frame sequence, and the calculation formula is as follows:

[0144] L prec =MSE(Encoder(M pred ),Encoder(M true ))

[0145] Among them, L prec To perceive the loss value.

[0146] Based on this, it is determined whether the training loss value, adversarial loss value, and perceptual loss value are less than or equal to the preset threshold. If so, the training ends and the trained motion generator is determined as the limb motion generation model.

[0147] This application provides a method for driving the limb movements of a digital human. It involves collecting a set of movements from recorded motion data, determining multiple basic movements based on the set, identifying training samples, aligning training text and corresponding training speech using the CTC caption alignment algorithm, and synthesizing training action sequences corresponding to the training samples based on multiple basic movement information. Further, the training samples and actual action frame sequences are input into a generative adversarial network (GAN). In the GAN, sliding windows with set starting offsets are used to process the training samples and actual action frame sequences, recording the resulting window data in a replay buffer. First window data corresponding to the training samples is sampled from the replay buffer, and a predicted action frame sequence corresponding to the first window data is obtained through an action generator. This predicted action frame sequence is recorded in the replay buffer and memory unit. Second window data corresponding to the actual action frame sequence is sampled from the replay buffer, and a reconstructed action frame sequence corresponding to the second window data is obtained through an action feature network. A training loss value is determined based on the predicted action frame sequence, the actual action frame sequence, and the reconstructed action frame sequence. The model parameters of the action generator and / or the action feature network are adjusted based on the training loss value. If training is complete, the trained action generator is determined as the limb movement generation model. The limb motion data preparation process provided in this application involves recording a small amount of motion capture data based on choreographed movements, and then synthesizing realistic and easily modifiable motion data based on this motion capture data. This reduces the cost of data acquisition and modification and improves the efficiency of data preparation. At the same time, an adversarial generative network is used to train the limb motion generator. Through mutual adversarial learning between the generator and the discriminator, the limb motion generator eventually outputs the most accurate sequence of motion frames.

[0148] Figure 11 This is a schematic diagram of the structure of a digital human limb movement driving device provided in an embodiment of this application. Figure 11 As shown in the embodiment of this application, a digital human limb movement driving device 1100 is provided. The device includes: an acquisition module 1101, a first processing module 1102, a second processing module 1103 and a third processing module 1104.

[0149] The acquisition module 1101 is used to acquire the audio to be played and the text corresponding to the audio to be played.

[0150] The first processing module 1102 is used to divide the text into multiple target text segments according to punctuation marks, and to divide the speech to be played into target speech corresponding to each target text segment;

[0151] The second processing module 1103 is used to generate an action frame sequence of the target speech and the target text corresponding to the target text in multiple target text segments, based on the target text and the target speech corresponding to the target text.

[0152] The third processing module 1104 is used to synchronize the playback of the target audio and the corresponding action frame sequence played before and after the playback of multiple target audio segments, thereby driving the digital human in real time.

[0153] Optionally, the third processing module 1104 is also used for:

[0154] Each segment of target speech corresponds to an action frame sequence including a start action frame sequence, multiple intermediate action frame sequences, and an end action frame sequence.

[0155] Determine whether the ending action frame sequence in the action frame sequence corresponding to the first target text is a static action; if so, determine the percentage information of the ending action frame sequence in the action frame sequence, and perform frame interpolation processing on the ending action frame sequence according to the percentage information.

[0156] If not, continue to determine whether there is a static action in the action frame sequence corresponding to other target texts after the first target text. Until a static action is found, the first target text and the action frame sequence before the static action appear are spliced ​​together. When the action frame sequence plays to the static action, the motion data of the initial action is added to the static action until the corresponding audio is played.

[0157] Optionally, the second processing module 1103 is also used for:

[0158] Based on the chronological order, the target text and the target speech corresponding to the target text are input into the limb motion generation model to generate limb motions, resulting in the corresponding input target speech and target text motion frame sequence. The motion frames contained in the motion frame sequence represent the rotation angle of each limb joint.

[0159] Optionally, the second processing module 1103 is also used for:

[0160] Input the target text and the target speech corresponding to the target text into the arm motion generator to obtain the corresponding arm motion frame sequence of the input target speech and target text;

[0161] Input the target text, the target speech corresponding to the target text, and the arm sequence frame sequence into the finger action generator to obtain the corresponding finger action frame sequence of the input target speech and target text.

[0162] The action frame sequences corresponding to the target speech and target text are determined based on the arm frame sequence and the finger frame sequence.

[0163] Optionally, the second processing module 1103 is also used for:

[0164] Obtain training samples and the corresponding actual action frame sequences; whereby the training sample information includes training speech and training text corresponding to the training language;

[0165] The training samples and actual action frame sequences are input into the adversarial generative network. In the adversarial generative network, the training samples and actual action frame sequences are processed by sliding windows based on a set starting offset. The resulting window data is recorded in the replay buffer.

[0166] Sample the first window data corresponding to the training samples from the replay buffer, and obtain the predicted action frame sequence corresponding to the first window data through the action generator. Record the predicted action frame sequence into the replay buffer and memory unit.

[0167] The second window data corresponding to the actual action frame sequence is sampled from the replay buffer, and the reconstructed action frame sequence corresponding to the second window data is obtained through the action feature network.

[0168] The training loss value is determined based on the predicted action frame sequence, the actual action frame sequence, and the reconstructed action frame sequence.

[0169] Adjust the model parameters of the action generator and / or action feature network based on the training loss value;

[0170] Once training is complete, the trained motion generator will be designated as the limb motion generation model.

[0171] Optionally, the acquisition module 1101 is also used for:

[0172] Collect a set of actions from the recorded motion data, and determine multiple basic actions based on the set of actions;

[0173] The training samples are determined, and the training speech and the corresponding training text are aligned according to the CTC caption alignment algorithm.

[0174] The training action sequence corresponding to the training sample is synthesized based on multiple basic action information.

[0175] This application discloses a digital human limb movement driving device. It collects and records a set of movements from motion data, determines multiple basic movements based on the set, identifies training samples, aligns the training text and its corresponding training speech using the CTC caption alignment algorithm, and synthesizes a training action sequence corresponding to the training samples based on multiple basic movement information. Further, the training samples and actual action frame sequences are input into a generative adversarial network for training. After training, the trained action generator is defined as the limb movement generation model. This model obtains the speech to be played and its corresponding text; divides the text into multiple target text segments based on punctuation, and segments the speech to be played into target speech corresponding to each target text segment; based on chronological order, the target text and its corresponding target speech are input into the arm movement generator to obtain the corresponding input target speech and target text arm movement frame sequence. The process involves inputting the target text, its corresponding target speech, and arm sequence frames into a finger motion generator to obtain corresponding finger motion frame sequences for the input target speech and text. Based on the arm and finger frame sequences, the generator determines the motion frame sequences corresponding to the target speech and text. It then checks whether the ending motion frame sequence in the first segment of the target text is a static motion. If so, it determines the percentage of the ending motion frame sequence within the total motion frame sequence and performs frame interpolation based on this percentage. If not, it continues to check whether static motions exist in the motion frame sequences of other target texts following the first segment, continuing until a static motion is found. Then, it concatenates the first segment of the target text with the motion frame sequences preceding the static motion. When the motion frame sequence reaches a static motion, it supplements the static motion with the initial motion data until the corresponding audio playback is complete. This achieves automated and intelligent generation of digital human body movements, automatically creating smooth and natural body movements that match the input speech and text content, especially fine arm and finger movements. This not only improves the realism and fluency of digital human interaction.

[0176] Figure 12 A schematic diagram of the hardware structure of the digital human limb movement driving method device provided in this application. Figure 12 As shown, the electronic device 1200 provided in this embodiment includes at least one processor 1201 and a memory 1202. Optionally, the device 1200 further includes a communication component 1203. The processor 1201, the memory 1202, and the communication component 1203 are connected via a bus 1204.

[0177] In a specific implementation, at least one processor 1201 executes computer execution instructions stored in memory 1202, causing at least one processor 1201 to perform the above-described method.

[0178] The specific implementation process of processor 1201 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0179] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0180] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0181] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0182] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0183] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0184] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0185] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0186] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0189] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0191] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for driving the limb movements of a digital human, characterized in that, include: Obtain the audio to be played and the corresponding text. The text is divided into multiple target text segments based on punctuation marks, and the audio to be played is segmented into target audio corresponding to each target text segment; For the target text in the multiple target text segments, an action frame sequence corresponding to the target speech and the target text is generated based on the target text and the target speech corresponding to the target text. During the playback of multiple target audio segments, the playback of the target audio segments played before and after and the corresponding action frame sequences are synchronized to drive the digital human in real time.

2. The method for driving limb movements of a digital human according to claim 1, characterized in that, The step of synchronizing the playback of the target audio and the corresponding action frame sequence played before and after, and driving the digital human in real time, includes: Each segment of target speech corresponds to an action frame sequence including a start action frame sequence, multiple intermediate action frame sequences, and an end action frame sequence. Determine whether the ending action frame sequence in the action frame sequence corresponding to the first target text is a static action; if so, determine the percentage information of the ending action frame sequence in the action frame sequence, and perform frame interpolation processing on the ending action frame sequence according to the percentage information. If not, continue to determine whether there is a static action in the action frame sequence corresponding to other target texts after the first target text, until a static action is found. Then, concatenate the first target text and the action frame sequence before the static action appears. When the action frame sequence plays to the static action, supplement the static action with the initial action action data until the corresponding audio playback is complete.

3. The method according to claim 2, characterized in that, The step of generating an action frame sequence corresponding to the target speech and the target text based on the target text and the target speech corresponding to the target text includes: Based on the chronological order, the target text and the target speech corresponding to the target text are input into the limb motion generation model to generate limb motions, resulting in a sequence of motion frames for the corresponding input target speech and target text. The motion frames in the sequence represent the rotation angles of each limb joint.

4. The method according to claim 3, characterized in that, The limb movement generation model includes an arm movement generator and a finger movement generator. The step of inputting the target text and the corresponding target speech into the limb movement generation model to generate limb movements, resulting in a sequence of action frames for the input target speech and target text, includes: The target text and the target speech corresponding to the target text are input into the arm motion generator to obtain the corresponding arm motion frame sequence of the input target speech and the target text; The target text, the target speech corresponding to the target text, and the arm sequence frame sequence are input into the finger action generator to obtain the corresponding input finger action frame sequence of the target speech and the target text. The action frame sequence corresponding to the target speech and the target text is determined based on the arm frame sequence and the finger frame sequence.

5. The method according to claim 4, characterized in that, The limb movement generation model was trained in the following way: Obtain training samples and the corresponding actual action frame sequences; wherein, the training sample information includes training speech and training text corresponding to the training language; The training samples and the actual action frame sequences are input into the adversarial generation network. In the adversarial generation network, the training samples and the actual action frame sequences are processed by sliding windows based on a set starting offset. The obtained window data is recorded in the replay buffer. The first window data corresponding to the training sample is sampled from the replay buffer, and the predicted action frame sequence corresponding to the first window data is obtained through the action generator. The predicted action frame sequence is recorded in the replay buffer and the memory unit. The memory unit can store the training sample and the corresponding predicted action frame sequence, as well as the target text and the corresponding action frame sequence. The second window data corresponding to the actual action frame sequence is sampled from the replay buffer, and the reconstructed action frame sequence corresponding to the second window data is obtained through the action feature network. The training loss value is determined based on the predicted action frame sequence, the actual action frame sequence, and the reconstructed action frame sequence; Based on the training loss value, adjust the model parameters of the action generator and / or the action feature network; If the training is complete, the trained motion generator will be identified as the limb motion generation model.

6. The method according to claim 5, characterized in that, The training samples and the actual action frame sequence were obtained in the following way: Collect a set of actions and audio files from the recorded action data, and determine multiple basic actions and static actions based on the set of actions; The training text is determined, the training speech is determined based on the training text, and the training text and the corresponding training speech are aligned according to the CTC subtitle alignment algorithm. The training action frame sequence corresponding to the training sample is synthesized based on the training text, the training speech, the multiple basic actions, and the static actions.

7. A digital human limb movement driving device, characterized in that, include: The acquisition module is used to acquire the audio to be played and the text corresponding to the audio. The first processing module is used to divide the text into multiple target text segments according to punctuation marks, and to segment the speech to be played into target speech corresponding to each target text segment; The second processing module is used to generate an action frame sequence corresponding to the target speech and the target text based on the target text and the target speech corresponding to the target text in the multiple target text segments; The third processing module is used to synchronize the playback of the target audio and the corresponding action frame sequence played before and after the playback of multiple target audio segments, thereby driving the digital human in real time.

8. A digital human limb movement driving device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement a digital human limb movement driving method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the limb movement driving method for a digital human as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the limb movement driving method of any one of claims 1 to 6.

Images