Digital human limb motion generation method and system based on deep learning model

Abstract

This invention provides a method and system for generating digital human body movements based on a deep learning model, comprising: acquiring deep learning information and audio / video data; separating and processing the audio / video data, and obtaining body movement classification data and text data based on the deep learning information; establishing a tuple that associates text with corresponding body movement tags and storing it in a database; when the digital human broadcasts, determining whether a body movement tag corresponding to the text can be found in the database; if so, using the body movement tag to drive the digital human to display the corresponding body movement; if not, associating the text with the corresponding body movement tag in the tuple based on semantic similarity, driving the digital human to display the corresponding body movement, and updating the database. This invention can adaptively guide the generation of digital human body movements through a deep learning model and automatically improve the recognition effect.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for generating digital human limb movements based on a deep learning model. Background Technology

[0002] Digital humans refer to digital AI robots that use computer digital modeling technology to simulate human figures through graphics and graphics rendering, and use motion and facial capture technologies to simulate the facial expressions and features of real people. They are formed by using computer means such as deep learning, speech recognition, semantic understanding, and speech synthesis, and have multiple human characteristics (physical features, human performance abilities, human interaction abilities, etc.).

[0003] Patent document CN114972923A discloses a method for generating virtual digital human limb interaction actions based on self-supervised learning. The method includes: processing a dataset of limb action segments using the K3M human 3D skeleton extraction toolkit to obtain a temporal sequence of human 3D skeleton frames; training a pre-constructed self-supervised model using a contrastive training method and the temporal sequence of human 3D skeleton frames to obtain a 3D skeleton self-supervised model; training the 3D skeleton self-supervised model on the dataset based on limb interaction action sequences to obtain a limb interaction sequence autoencoder model; and using the limb interaction sequence autoencoder model to perform semantic feature encoding and action interaction decoding on the action commands issued by the virtual digital human to obtain a virtual digital human response action sequence.

[0004] Current digital human technology can generate voice broadcasts and corresponding digital human appearances based on pre-configured scripts. However, it lacks the ability to automatically generate corresponding body movements based on the broadcast text. Adding body movements that match the broadcast content can greatly enhance the expressiveness of language. Most current solutions use random sequences of body movements or insert pre-defined body movement markers at fixed positions in the configured script during digital human broadcasting. This approach requires manual pre-configuration, and since there is no clear correspondence between the broadcast text and the corresponding body movements, the configurer can only configure the body movements based on their own intuition, which may result in different configurations from different personnel. Furthermore, it is impossible to pre-set corresponding body movement markers when dealing with scripts that need to be broadcast in real time. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for generating digital human limb movements based on a deep learning model.

[0006] A method for generating digital human limb movements based on a deep learning model, provided by the present invention, includes:

[0007] Step S1: Acquire deep learning information and audio / video data;

[0008] Step S2: Separate and process the audio and video data, and obtain limb movement classification data and text data based on the deep learning information;

[0009] Step S3: Match the body movement classification data and text data one by one, and create a binary tuple that associates the text and the corresponding body movement label, and store it in the database.

[0010] Step S4: When the digital human is broadcasting, determine whether the body action tag corresponding to the text can be found in the database. If yes, use the body action tag to drive the digital human to display the corresponding body action; otherwise, trigger step S5.

[0011] Step S5: Associate the text with the corresponding body movement labels in the binary tuple based on semantic similarity, drive the digital human to display the corresponding body movements, and update the database.

[0012] Preferably, the deep learning information includes a deep learning model;

[0013] The deep learning model is obtained by building a multi-level convolutional neural network model and training the convolutional neural network model. The multi-level convolutional neural network model has multiple convolutional layers and pooling layers;

[0014] The deep learning model is capable of classifying body movements.

[0015] Preferably, step S1 includes:

[0016] Step S1.1: Obtain the live broadcast image data and label the live broadcast images containing body movements with the corresponding body movement data;

[0017] Step S1.2: Construct the convolutional neural network model and initially set the weights of the pre-trained model parameters between each layer in the model;

[0018] Step S1.3: Train the convolutional neural network model based on the labeled body movement data, wherein the real-person broadcasting image containing body movements is used as the input of the convolutional neural network model, and the body movement data is used as the output of the convolutional neural network model.

[0019] Step S1.4: After validating the trained convolutional neural network model on sample data, determine the model parameters that have good classification performance, thereby obtaining the deep learning model.

[0020] Preferably, step S2 includes:

[0021] Step S2.1: Acquire the broadcast video and separate the audio, then extract keyframe images from the video.

[0022] Step S2.2: Extract text data from audio using speech recognition;

[0023] Step S2.3: Classify the body movements of the keyframe images obtained by frame extraction using the deep learning information model trained in the deep learning information to obtain body movement classification data.

[0024] Preferably, step S5 includes:

[0025] Step S5.1: Use word vectors to calculate the similarity between texts that were not found and texts that already have a relationship in the database;

[0026] Step S5.2: Associate the unqueried text with the body action tags corresponding to the text obtained from the similarity calculation to obtain the updated tuple.

[0027] Step S5.3: The limb movement tags drive the digital human to display corresponding limb movements, and at the same time, the updated tuples are stored in the database.

[0028] According to the present invention, a deep learning model-driven digital human limb motion generation system includes:

[0029] Module M1: Acquires deep learning information and audio / video data;

[0030] Module M2: Separates and processes the audio and video data, and obtains body movement classification data and text data based on the deep learning information;

[0031] Module M3: Matches the body movement classification data and text data one by one, and creates a binary tuple that associates the text and the corresponding body movement label, storing it in the database;

[0032] Module M4: When the digital human is broadcasting, it determines whether the corresponding body action tag for the text can be found in the database. If so, the body action tag is used to drive the digital human to display the corresponding body action; otherwise, module M5 is triggered.

[0033] Module M5: Associates the text with the corresponding body action tags in the binary tuple based on semantic similarity, drives the digital human to display the corresponding body actions, and updates the database.

[0034] Preferably, the deep learning information includes a deep learning model;

[0035] The deep learning model is obtained by building a multi-level convolutional neural network model and training the convolutional neural network model. The multi-level convolutional neural network model has multiple convolutional layers and pooling layers;

[0036] The deep learning model is capable of classifying body movements.

[0037] Preferably, module M1 includes:

[0038] Module M1.1: Acquires live-action broadcast image data and annotates the corresponding body movement data for live-action broadcast images containing body movements;

[0039] Module M1.2: Construct the convolutional neural network model and initially set the weights of the pre-trained model parameters between each layer in the model;

[0040] Module M1.3: Train the convolutional neural network model based on the labeled body movement data, wherein the real-person broadcast image containing body movements is used as the input of the convolutional neural network model, and the body movement data is used as the output of the convolutional neural network model;

[0041] Module M1.4: After validating the trained convolutional neural network model on sample data, determine the model parameters that have good classification performance, thereby obtaining a deep learning model.

[0042] Preferably, module M2 includes:

[0043] Module M2.1: Acquires the broadcast video and extracts the audio, then extracts keyframe images from the video.

[0044] Module M2.2: Extracts text data from audio using speech recognition;

[0045] Module M2.3: Classifies body movements in keyframe images obtained by frame extraction using a deep learning information model trained in deep learning information, and obtains body movement classification data.

[0046] Preferably, the module M5 includes:

[0047] Module M5.1: Uses word vectors to calculate the similarity between text not found and text with existing relationships in the database;

[0048] Module M5.2: Associate the unqueried text with the body action tags corresponding to the text obtained by calculating similarity to obtain the updated tuple;

[0049] Module M5.3: The limb movement tags drive the digital human to display corresponding limb movements, and at the same time store the updated tuples in the database.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. This invention trains a deep learning model that can classify body movements. When a digital human is broadcasting, it can adaptively guide the generation of body movements by means of the correlation between text and body movements, and can automatically improve the recognition effect, thereby improving the diversity and sensory appeal of the digital human's body movements.

[0052] 2. This invention introduces body language that matches the text being broadcast to digital humans, increasing the expressive capabilities of multimodal digital humans, making them more human-like, and also increasing the friendliness of human-computer interaction.

[0053] 3. This invention automatically establishes the correspondence between text and body action tags based on publicly available video data, solving the problem of requiring a large amount of manual annotation in the past, reducing costs while improving quality and enhancing the final presentation effect. Attached Figure Description

[0054] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0055] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0056] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0057] This invention relates to an adaptive generation method for multimodal digital human body movements. Specifically, it involves constructing a multi-level convolutional neural network, labeling publicly available data, and training the multi-level convolutional neural network with the labeled data to obtain a deep learning model capable of classifying body movements. The model classifies body movements on a large amount of publicly available data and associates them with the broadcast text. During the digital human's broadcast, the association between the text and body movements adaptively guides the generation of digital human body movements and can automatically improve the recognition effect.

[0058] Example 1

[0059] According to the present invention, a method for generating digital human limb movements based on a deep learning model is provided, such as... Figure 1 As shown, it includes:

[0060] Step S1: Obtain deep learning information and audio / video data; specifically, construct a multi-level convolutional neural network model, collect publicly available broadcast video data for data annotation, use the annotated data and model for parameter tuning, and finally train a model that can classify body movements.

[0061] The deep learning information includes a deep learning model; the deep learning model is obtained by building a multi-level convolutional neural network model and training the convolutional neural network model. The multi-level convolutional neural network model has multiple convolutional layers and pooling layers; the deep learning model is capable of classifying limb movements. Step S1 includes:

[0062] Step S1.1: Obtain the live-action broadcast image data and label the live-action broadcast images containing body movements with the corresponding body movement data.

[0063] Step S1.2: Construct the convolutional neural network model and initially set the weights of the pre-trained model parameters between each layer in the model.

[0064] Step S1.3: Train the convolutional neural network model based on the labeled body movement data, wherein the real-person broadcast image containing body movements is used as the input of the convolutional neural network model, and the body movement data is used as the output of the convolutional neural network model.

[0065] Step S1.4: After validating the trained convolutional neural network model on sample data, determine the model parameters that have good classification performance, thereby obtaining the deep learning model.

[0066] Step S2: Separate and process the audio and video data, and obtain body movement classification data and text data based on the deep learning information. Specifically, collect a large number of news broadcast videos from the public network, extract the audio from the videos for speech recognition to obtain text data, extract keyframe images from the videos, and automatically classify body movements using a trained model to obtain a large amount of classification data; Step S2 includes:

[0067] Step S2.1: Acquire the broadcast video and separate the audio, then extract keyframe images from the video.

[0068] Step S2.2: Extract text data from audio using speech recognition.

[0069] Step S2.3: Classify the body movements of the keyframe images obtained by frame extraction using the deep learning information model trained in the deep learning information to obtain body movement classification data.

[0070] Step S3: Match the body movement classification data and text data one by one, and establish a tuple that associates the text and the corresponding body movement label and store it in the database; specifically, use the body movement classification identified by the model to associate speech text to obtain a large number of text and body movement tuples, and establish a tuple that associates the broadcast text with the body movement based on the identified body movement and the result of the recognition at the corresponding speech of the body movement and store it in the database.

[0071] Step S4: When the digital human is broadcasting, determine whether the body action tag corresponding to the text can be found in the database. If so, use the body action tag to drive the digital human to display the corresponding body action; otherwise, trigger step S5. Specifically, when generating the digital human broadcasting video, for the text that needs to be broadcast by the digital human, search in the database where the relationship has been established whether the text already has a corresponding body action tag. If so, directly use the body action corresponding to the text to guide the digital human to generate the corresponding body action.

[0072] Step S5: Associate the text with the corresponding body movement tag in the binary tuple based on semantic similarity, drive the digital human to display the corresponding body movement, and update the database. Specifically, assuming no text is found (W1), calculate the similarity and obtain text (W2), where text W2 is a text with an existing association in the database. Further, if no body movement tag corresponding to text W1 is found in the database, use word vectors to calculate the text W2 that is closest to text with an existing association with text W1, use the body movement tag corresponding to text W2, and update the association between text and body movement tag in the database. Step S5 includes:

[0073] Step S5.1: Use word vectors to calculate the similarity between the text that was not found and the text that already has a relationship in the database.

[0074] Step S5.2: Associate the unqueried text with the body action tags corresponding to the text obtained by calculating similarity to obtain the updated tuple.

[0075] Step S5.3: The limb movement tags drive the digital human to display corresponding limb movements, and at the same time, the updated tuples are stored in the database.

[0076] This method employs a convolutional neural network with multiple convolutional and pooling layers, trains the model using publicly available data, and obtains a deep learning model capable of recognizing body movements. This model is then associated with the text accompanying the recognized body movements to establish a text-body movement relationship. During subsequent digital human broadcasting, this well-structured relationship guides the digital human to generate corresponding body movements, thereby achieving better expressive effects.

[0077] Example 2

[0078] This invention also provides a deep learning model-driven digital human body motion generation system. Those skilled in the art can implement the deep learning model-driven digital human body motion generation system by executing the steps of the deep learning model-driven digital human body motion generation method. That is, the deep learning model-driven digital human body motion generation method can be understood as a preferred embodiment of the deep learning model-driven digital human body motion generation system.

[0079] According to the present invention, a deep learning model-driven digital human limb motion generation system includes:

[0080] Module M1: Acquires deep learning information and audio / video data; the deep learning information includes a deep learning model; the deep learning model is obtained by building a multi-level convolutional neural network model and training the convolutional neural network model. The multi-level convolutional neural network model has multiple convolutional layers and pooling layers; the deep learning model is capable of classifying body movements. Module M1 includes:

[0081] Module M1.1: Acquires live-action broadcast image data and annotates the corresponding body movement data for live-action broadcast images containing body movements.

[0082] Module M1.2: Construct the convolutional neural network model and initially set the weights of the pre-trained model parameters between each layer in the model.

[0083] Module M1.3: Trains the convolutional neural network model based on labeled body movement data, wherein the real-person broadcast image containing body movements is used as the input of the convolutional neural network model, and the body movement data is used as the output of the convolutional neural network model.

[0084] Module M1.4: After validating the trained convolutional neural network model on sample data, determine the model parameters that have good classification performance, thereby obtaining a deep learning model.

[0085] Module M2: Separates and processes the audio and video data, and obtains limb movement classification data and text data based on the deep learning information; Module M2 includes:

[0086] Module M2.1: Acquires the broadcast video and extracts the audio, then extracts keyframe images from the video. Module M2.2: Recognizes text data from the audio using speech recognition. Module M2.3: Classifies the extracted keyframe images into body movements using a deep learning information model trained on the deep learning information, obtaining body movement classification data.

[0087] Module M3: Matches the body movement classification data and text data one by one, and creates a binary tuple that associates the text and the corresponding body movement label, storing it in the database.

[0088] Module M4: When the digital human is broadcasting, it determines whether the corresponding body action tag for the text can be found in the database. If so, the body action tag is used to drive the digital human to display the corresponding body action; otherwise, module M5 is triggered.

[0089] Module M5: Associates the text with the corresponding body movement tags in the binary tuple based on semantic similarity, drives the digital human to display the corresponding body movement, and updates the database. Module M5 includes: Module M5.1: Calculates the similarity between unqueried text and text with existing associations in the database using word vectors; Module M5.2: Associates the unqueried text with the body movement tags corresponding to the text obtained from the similarity calculation, obtaining an updated binary tuple; Module M5.3: The body movement tags drive the digital human to display the corresponding body movement, and the updated binary tuple is stored in the database.

[0090] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0091] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for generating digital human limb motion based on a deep learning model, characterized in that, The method comprises the following steps: Step S1: obtaining deep learning information and audio-video data; Step S2: separating and processing the audio-video data, and obtaining limb action classification data and text data according to the deep learning information; Step S3: matching the limb action classification data and the text data one by one, and storing a binary tuple related to a text-limb action corresponding limb action label into a database; Step S4: when a digital person broadcasts, judging whether the text corresponding limb action label can be queried in the database, if yes, driving the digital person to display the corresponding limb action by using the limb action label; if no, triggering step S5; Step S5: according to semantic similarity, associating the text with the corresponding limb action label in the binary tuple, driving the digital person to display the corresponding limb action, and updating the database; Step S1 comprises the following steps: Step S1.1: obtaining real person broadcasting picture data, and labeling the corresponding limb action data of the real person broadcasting picture containing limb action; Step S1.2: constructing a convolutional neural network model, and preliminarily setting the weight of the pre-trained model parameters between each level in the model; Step S1.3: training the convolutional neural network model according to the labeled limb action data, wherein the real person broadcasting picture containing limb action is used as the input of the convolutional neural network model, and the limb action data is used as the output of the convolutional neural network model; Step S1.4: determining the model parameters with good classification effect after verifying the trained convolutional neural network model on sample data, thereby obtaining a deep learning model; Step S2 comprises the following steps: Step S2.1: obtaining a broadcasting video and separating the audio, and obtaining key frame pictures by frame extraction from the video; Step S2.2: identifying text data from the audio by voice recognition; Step S2.3: classifying the key frame pictures obtained by frame extraction into limb action by using the trained deep learning information model in the deep learning information, thereby obtaining limb action classification data.

2. The method of claim 1, wherein the method is based on a deep learning model. The deep learning information comprises a deep learning model; The deep learning model is obtained by using a convolutional neural network, constructing a multi-level convolutional neural network model, and training the convolutional neural network model; The multi-level convolutional neural network model has multiple convolutional layers and pooling layers; The deep learning model can classify limb action.

3. The method of claim 1, wherein the method is based on a deep learning model. The step S5 comprises the following steps: Step S5.1: calculating the similarity of the unqueried text and the text with an associated relationship in the database by using a word vector; Step S5.2: associating the unqueried text with the limb action label corresponding to the text obtained by calculating the similarity, thereby obtaining an updated binary tuple; Step S5.3: the limb action label drives the digital person to display the corresponding limb action, and the updated binary tuple is stored into the database.

4. A digital human limb motion generation system based on a deep learning model, characterized in that, The method comprises the following steps: Module M1: obtaining deep learning information and audio-video data; Module M2: separating and processing the audio-video data, and obtaining limb action classification data and text data according to the deep learning information; Module M3: match the body action classification data and the text data one by one, and store the binary tuple of the text and the body action corresponding body action label into the database; Module M4: when the digital person broadcasts, judge whether the text corresponding body action label can be queried in the database, if yes, use the body action label to drive the digital person to show the corresponding body action; if not, trigger module M5; Module M5: according to the semantic similarity, associate the text with the corresponding body action label in the binary tuple, drive the digital person to show the corresponding body action, and update the database; Module M1 includes: Module M1.1: obtain real person broadcast picture data, and label the corresponding body action data of the real person broadcast picture containing body action; Module M1.2: construct a convolutional neural network model, and preliminarily set the weight of the pre-trained model parameters between each level in the model; Module M1.3: train the convolutional neural network model according to the labeled body action data, wherein the real person broadcast picture containing body action is used as the input of the convolutional neural network model, and the body action data is used as the output of the convolutional neural network model; Module M1.4: after the trained convolutional neural network model is verified on sample data, the model parameters with good classification effect are determined, so as to obtain a deep learning model; Module M2 includes: Module M2.1: obtain a broadcast video and separate the audio, and obtain key frame pictures by frame extraction from the video; Module M2.2: identify the text data from the audio by voice recognition; Module M2.3: classify the body action of the key frame pictures obtained by frame extraction by the deep learning information model trained in the deep learning information, to obtain body action classification data.

5. The deep learning model based digital human limb motion generation system of claim 4, wherein, The deep learning information includes a deep learning model; The deep learning model is obtained by establishing a multi-level convolutional neural network model and training the convolutional neural network model through the convolutional neural network; The multi-level convolutional neural network model has multiple convolutional layers and pooling layers; The deep learning model can classify body actions.

6. The deep learning model based digital human limb motion generation system of claim 4, wherein, The module M5 includes: Module M5.1: calculate the similarity between the unqueried text and the text with associated relationship in the database using word vector; Module M5.2: associate the unqueried text with the body action label corresponding to the text obtained by calculating the similarity to obtain updated binary tuples; Module M5.3: the body action label drives the digital person to show the corresponding body action, and the updated binary tuples are stored into the database.

Images