Digital human face action generation method and device and electronic equipment

Abstract

The invention discloses a digital human face action generation method and device and electronic equipment, and relates to the technical field of computers. The method is applied to a server and comprises the following steps: receiving driving information and prompt action information sent by a client; inputting the driving information and the prompt action information into a facial action generation model, and outputting a facial action coefficient sequence; taking the facial action coefficient sequence as a facial action curve parameter, and generating a digital human facial action video frame; and sending the digital human face action video frame to a client. According to the embodiment of the invention, the driving information can be converted into the face action sequence based on the prompt action information sent by the client, so that the digital human has the ability of simulating different styles of figures to speak, the user can realize a customized and personalized digital human face driving effect through the prompt action information, and the use requirements of different users are met.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and electronic equipment for generating digital human facial movements. Background Art

[0002] In the virtual world, letting digital humans imitate real people's speech according to action prompts allows digital humans to participate in performances in a controllable manner, which can greatly enrich the performance details.

[0003] However, the effect of existing speech imitation technology is greatly affected by the professional level of animators, and even if the same speaker has the same style, his facial movements are still different in different states and emotions. When making stylized facial animations, animators need to take into account both the speaker's style and the rationality of the emotional movements. This is difficult to achieve and cannot meet the needs of different customers. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and electronic device for generating digital human facial movements, so as to solve the problem that existing digital human facial movements cannot meet the usage requirements of different customers.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating digital human facial movements, which is applied to a server and includes:

[0006] Receive the driving information and prompt action information sent by the client;

[0007] Inputting the driving information and the prompting action information into a facial action generation model, and outputting a facial action coefficient sequence;

[0008] Using the facial action coefficient sequence as facial action curve parameters to generate digital human facial action video frames;

[0009] The digital human facial action video frame is sent to the client.

[0010] Optionally, the training process of the facial action generation model includes:

[0011] Acquire an action sample database; wherein each driving source sample in the action sample database corresponds to an action coefficient sequence sample;

[0012] Performing data segmentation on each driving source sample and action coefficient sequence sample in the action sample database to obtain driving source segments, target action coefficient sequence segments, and prompt action coefficient sequence segments;

[0013] A diffusion model is trained according to the driving source segment, the target action coefficient sequence segment and the prompt action coefficient sequence segment to obtain the facial action generation model; wherein the diffusion model includes a cascaded Z-layer noise regression structure and an attention module, and Z is a positive integer.

[0014] Optionally, the step of performing data segmentation on each driving source sample and action coefficient sequence sample in the action sample database to obtain driving source segments, target action coefficient sequence segments and prompt action coefficient sequence segments comprises:

[0015] The i-th driving source sample is divided into a first driving source segment and a second driving source segment; the i-th driving source sample is any one in the action sample database, and i is a positive integer;

[0016] The i-th action coefficient sequence sample corresponding to the i-th driving source sample is divided into a first action coefficient sequence segment and a second action coefficient sequence segment; wherein the first driving source segment corresponds to the first action coefficient sequence segment, and the second driving source segment corresponds to the second action coefficient sequence segment;

[0017] If the first driving source segment is used as the driving source, the second action coefficient sequence segment is used as the prompt action coefficient sequence segment, and the first action coefficient sequence segment is used as the target action coefficient sequence segment;

[0018] If the second driving source segment is used as the driving source, the first action coefficient sequence segment is used as the prompt action coefficient sequence segment, and the second action coefficient sequence segment is used as the target action coefficient sequence segment.

[0019] Optionally, the training of a diffusion model according to the driving source segment, the target action coefficient sequence segment and the prompt action coefficient sequence segment to obtain the facial action generation model comprises:

[0020] Using the driving source fragment and the target action coefficient sequence fragment, the predicted noise output by the noise regression structure of the jth layer is obtained; Z≥j≥1, j is a positive integer;

[0021] Using the fine-tuning amount output by the attention module, the predicted noise of the j-th layer is adjusted to obtain a noise estimation value corresponding to the j-th layer; wherein the input of the attention module is the prompt action coefficient sequence and the predicted noise of the j-th layer; using the fine-tuning amount, the predicted noise corresponding to the j-th layer is adjusted to obtain a noise estimation value corresponding to the j-th layer;

[0022] Accumulate the noise estimation values ​​corresponding to the noise regression structure of the Z layer to obtain a noise prediction result;

[0023] The diffusion model is iteratively trained according to the loss value between the noise signal and the noise prediction result to obtain the facial action generation model.

[0024] Optionally, receiving prompt action information sent by the client includes:

[0025] receiving prompt action information sent by the client, wherein the prompt action information is information of a target action snapshot, and the target action snapshot is one of at least one action snapshot displayed by the client; or,

[0026] The prompt action information sent by the client is received, where the prompt action information is a facial action video recorded by the user.

[0027] In a second aspect, an embodiment of the present invention further provides a method for generating digital human facial movements, which is applied to a client and includes:

[0028] Send drive information and prompt action information to the server;

[0029] Receiving digital human facial action video frames sent by the server;

[0030] Play the digital human facial action video frame.

[0031] Optionally, the above digital human facial action generation method includes:

[0032] displaying at least one action snapshot;

[0033] receiving a user input operation on a target action snapshot among the at least one action snapshot;

[0034] The sending of the prompt action information to the server includes: in response to the input operation, determining a prompt action coefficient sequence sample associated with the target action snapshot as the prompt action information, and sending the information of the target action snapshot to the server.

[0035] Optionally, sending the prompt action information to the server includes:

[0036] The facial action video obtained by the user through the camera is used as the prompt action information, and the facial action video is sent to the server.

[0037] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the method for generating digital human facial movements as described in the first or second aspect above.

[0038] In a fourth aspect, an embodiment of the present invention further provides a digital human facial action generation device, comprising:

[0039] A first receiving module, used for receiving driving information and prompting action information sent by the client;

[0040] A first acquisition module, used for inputting the driving information and the prompting action information into a facial action generation model, and outputting a facial action coefficient sequence;

[0041] A generating module, used for using the facial action coefficient sequence as facial action curve parameters to generate a digital human facial action video frame;

[0042] The first sending module is used to send the digital human facial action video frame to the client.

[0043] In a fifth aspect, an embodiment of the present invention further provides a digital human facial action generation device, comprising:

[0044] The second sending module is used to send driving information and prompt action information to the server;

[0045] A second receiving module is used to receive the digital human facial action video frame sent by the server;

[0046] The display module is used to play the digital human facial action video frame.

[0047] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the digital human facial action generation method as described above are implemented.

[0048] In a seventh aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the digital human facial action generation method as described above.

[0049] In an embodiment of the present invention, the server receives the driving information and prompt action information sent by the client; inputs the driving information and prompt action information into the facial action generation model, outputs a facial action coefficient sequence; uses the facial action coefficient sequence as a facial action curve parameter to generate a digital human facial action video frame; and sends the digital human facial action video frame to the client. In this way, based on the prompt action information sent by the client, the driving information can be converted into a facial action coefficient sequence, so that the digital human has the ability to imitate different styles of characters to speak, and also has the ability to quickly imitate different emotions of the same person, without the need for large labor costs and time costs to produce action templates. Users can achieve customized and personalized digital human facial driving effects through prompt action information to meet the usage needs of different users. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0051] Figure 1 This is one of the flow charts of the method for generating digital human facial movements provided by an embodiment of the present invention;

[0052] Figure 2 is a schematic diagram of a training architecture of an extended model provided in an embodiment of the present invention;

[0053] Figure 3 It is one of the schematic diagrams of alignment of the driving source and the action coefficient sequence provided by the embodiment of the present invention;

[0054] Figure 4 This is the second schematic diagram of alignment of the driving source and the action coefficient sequence provided by the embodiment of the present invention;

[0055] Figure 5 is a schematic diagram of a digital human facial action generation architecture provided by an embodiment of the present invention;

[0056] Figure 6 is one of the facial action schematic diagrams provided by an embodiment of the present invention;

[0057] Figure 7 This is the second facial action schematic diagram provided by the embodiment of the present invention;

[0058] Figure 8 This is the second flow chart of the method for generating digital human facial movements provided by an embodiment of the present invention;

[0059] Fig. 9 This is one of the structural block diagrams of the digital human facial action generation device provided by the embodiment of the present invention;

[0060] Fig.10 This is the second structural block diagram of the digital human facial action generation device provided by the embodiment of the present invention;

[0061] Fig.11 is a schematic diagram of the hardware structure of a server provided in an embodiment of the present invention;

[0062] Fig.12 It is a schematic diagram of the hardware structure of the client device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] See also Figure 1 , Figure 1 is a flow chart of a method for generating digital human facial movements provided by an embodiment of the present invention. Figure 1 As shown, the digital human facial action generation method is applied to the server, including the following steps:

[0065] Step 101: receiving driving information and prompting action information sent by the client.

[0066] Optionally, the driving information is text information or audio information, such as spoken text or voice entered by the user.

[0067] Optionally, the prompt action information is a short video of facial action recorded by the user or a target action snapshot selected by the user; wherein the user selects a target action snapshot based on at least one action snapshot displayed by the client, and the action snapshot is a dynamic image generated based on some key frames of action coefficient sequence samples in an action sample database, which is used to show characteristic facial actions.

[0068] Optionally, the action snapshot may be generated by the server according to the action coefficient sequence samples in the action sample database and sent to the client, and the number of frames included in the action snapshot is specified by the user or is a default value.

[0069] Step 102: Input the driving information and the prompting action information into a facial action generation model, and output a facial action coefficient sequence.

[0070] The facial action coefficient sequence refers to a vector with a dimension of R, where R elements are real numbers between 0 and 1. Each element of the dimension in the vector represents the action of a certain position in the facial features, and the corresponding value represents the amplitude of the action. For example, 0.5 jaw opening means that the amplitude of the jaw opening action in the facial action is 0.5.

[0071] The prompt action information is used to determine the prompt action coefficient sequence, the facial action coefficient sequence is generated by imitating the prompt action coefficient sequence, and the facial action coefficient sequence has similar facial style features and / or emotional features as the prompt action.

[0072] Optionally, the facial action generation model is trained based on a diffusion model.

[0073] Step 103: Using the facial action coefficient sequence as facial action curve parameters to generate digital human facial action video frames.

[0074] In specific implementation, the user selects the digital human to be driven and reports it to the server. The server obtains the facial action coefficient sequence, uses the facial action coefficient sequence as the facial action curve parameter, generates the facial action control curve, and maps the facial action control curve to the binding controller corresponding to the digital human to realize the facial action driving of the digital human. The 3D rendering engine (such as Maya or UE or Unity or Blender) synthesizes the digital human facial action video frame by frame.

[0075] It should be pointed out that the binding controller corresponding to the digital human can be understood as the three-dimensional model corresponding to the digital human, such as the 3D head model of the digital human.

[0076] Step 104: Send the digital human facial action video frame to the client.

[0077] The digital human facial action video frame is sent to the client so that the client plays the digital human facial action video frame.

[0078] In the above embodiment, the server can convert the driving information into a facial action coefficient sequence based on the prompt action information sent by the client, so that the digital human has the ability to imitate the speech of characters with different styles, and also has the ability to quickly imitate different emotions of the same person. Users can use the prompt action information to achieve customized and personalized digital human facial driving effects to meet the usage needs of different users without requiring large labor costs and time costs to produce action templates.

[0079] In some embodiments, in the above step 102, the step of inputting the driving information and the prompting action information into a facial action generation model and outputting a facial action coefficient sequence includes:

[0080] Preprocessing the driving information to generate a driving source;

[0081] Acquire a prompt action coefficient sequence corresponding to the prompt action information;

[0082] The driving source and the prompt action coefficient sequence are input into a facial action generation model, and the facial action coefficient sequence output by the facial action generation model is obtained.

[0083] Optionally, if the driving information is audio information, a frequency spectrum feature (such as a Mel spectrum) corresponding to the audio information is used as a driving source; if the driving information is text information, a phoneme corresponding to the text information and a sound length corresponding to the phoneme are used as a driving source;

[0084] For example, Figure 2 In , the process of outputting facial action coefficient sequence based on facial action generation model includes:

[0085] Step S1: Obtain prompt action coefficient sequence and driving source.

[0086] Optionally, if the prompt action information is a short video of facial movements recorded by the user, a corresponding prompt action coefficient sequence is generated based on the short video of facial movements; if the prompt action information is a target action snapshot selected by the user, based on the correspondence between the target action snapshot and the action coefficient sequence samples in the action sample database, the action coefficient sequence samples corresponding to the target action snapshot are selected, and key frame fragments are captured from the target action coefficient sequence samples as the prompt action coefficient sequence.

[0087] Step S2: randomly synthesize initial noise according to the driving source.

[0088] Specifically, Figure 3 In , when the driving information is speech, the speech is converted into a frame-by-frame Mel spectrum as the driving source. Figure 4 In the example, when the driving information is text, the phonemes and corresponding pronunciation durations of the text are converted into frame-by-frame phoneme information. Assuming that the total number of frames of the driving source is N, and N is a positive integer, a noise signal noise is randomly generated through Gaussian distribution, and the length of the noise signal is consistent with the total number of frames N.

[0089] Step S3: Input the driving source and the prompt action coefficient sequence into the facial action generation model, perform denoising and reconstruction of the action sequence through the facial action generation model, and output the reconstructed facial action coefficient sequence.

[0090] Specifically, the facial action generation model performs denoising and reconstruction, including: denoising one by one from the maximum number of noise steps T. Starting from the Tth denoising, the initial noise is Assume that the current denoising is the tth time, and the current noise data is The current noise scheduling weight is α′ t , β′ t .but:

[0091]

[0092]

[0093] Among them, noise pred To predict noise, diffusion is the diffusion model, cond is the driving source, and prompt is the prompt action coefficient sequence. As the input of the t-1th denoising, until t = 1; the final is the sequence of facial action coefficients to be reconstructed (imitated).

[0094] In the above embodiment, by adding noise and denoising the facial action generation model, in the process of iterative denoising of the noise signal, the facial action generation model can learn the facial feature style based on the driving source and the prompt action coefficient sequence, realize facial action reconstruction, and finally generate a facial action coefficient sequence similar to the prompt action coefficient sequence. In this way, it is possible to drive the digital human to imitate the prompt action coefficient sequence to speak. Moreover, the user can achieve customized and personalized digital human facial driving effects through the prompt action information, meet the use needs of different users, and do not require large labor costs and time costs.

[0095] In some embodiments, the training process of the facial action generation model includes:

[0096] Acquire an action sample database; wherein each driving source sample in the action sample database corresponds to an action coefficient sequence sample;

[0097] Performing data segmentation on each driving source sample and action coefficient sequence sample in the action sample database to obtain driving source segments, target action coefficient sequence segments, and prompt action coefficient sequence segments;

[0098] A diffusion model is trained according to the driving source segment, the target action coefficient sequence segment and the prompt action coefficient sequence segment to obtain the facial action generation model; wherein the diffusion model includes a cascaded Z-layer noise regression structure and an attention module, and Z is a positive integer.

[0099] Exemplarily, the step of obtaining the action sample database includes: one-to-one correspondence between the driving source samples and the action coefficient sequence samples. The driving source samples can be Mel spectrum or the phonemes and pronunciation intervals of the text. The driving source samples and the action coefficient sequence are one-to-one correspondence according to the number of frames on the time axis, and finally the aligned driving source and action coefficient sequence are obtained. Figure 3 , which shows a schematic diagram of the frame-by-frame alignment of the audio, Mel spectrum and motion coefficient sequences, as shown in Figure 4 As shown, it is a schematic diagram of frame-by-frame alignment of text, phonemes, sound lengths and action coefficient sequences.

[0100] For example, Figure 5In the model training process, the following steps are involved: obtaining an action sample database; segmenting the action sample database to obtain driving source segments, target action coefficient sequences, and prompt action coefficient sequences; inputting the driving source segments, target action coefficient sequences plus noise signals, and prompt action coefficient sequences into a diffusion model to output facial action coefficient sequences and noise prediction results; and performing loss training based on the noise prediction results and input noise. Specifically, the diffusion model is iteratively trained based on the loss value between the noise prediction results and the input noise, and finally a trained diffusion model is obtained as a facial action generation model.

[0101] The action coefficient sequence sample is a vector with a dimension of R. The R elements in the vector are all real numbers between 0 and 1. The element at each position in the vector represents an action at a certain position in the facial features, and the corresponding value represents the amplitude of the action. For example, if the chin is opened 0.5, it means that the movement amplitude of the chin opening action in the facial action is 0.5.

[0102] Among them, the target action coefficient sequence fragment and the prompt action coefficient sequence fragment corresponding to the first driving source fragment belong to the first action coefficient sequence sample, and the first driving source fragment is aligned with the target action coefficient sequence fragment frame by frame. That is, after the first action coefficient sequence sample is segmented, some fragments are used as target action coefficient sequence fragments, and some fragments are used as prompt action coefficient sequence fragments, which are used as training data to train the diffusion model. Since the target action coefficient sequence fragment and the prompt action coefficient sequence fragment belong to the same action coefficient sequence sample, the facial action style is similar. In this way, the diffusion model can learn facial action sequences similar to the target action coefficient sequence fragment by iterative training based on the driving source fragment and the prompt action coefficient sequence fragment.

[0103] In some embodiments, the data segmentation of each driving source sample and action coefficient sequence sample in the action sample database to obtain driving source segments, target action coefficient sequence segments and prompt action coefficient sequence segments includes:

[0104] The i-th driving source sample is divided into a first driving source segment and a second driving source segment; the i-th driving source sample is any one in the action sample database, and i is a positive integer;

[0105] The i-th action coefficient sequence sample corresponding to the i-th driving source sample is divided into a first action coefficient sequence segment and a second action coefficient sequence segment; wherein the first driving source segment corresponds to the first action coefficient sequence segment, and the second driving source segment corresponds to the second action coefficient sequence segment;

[0106] If the first driving source segment is used as the driving source, the second action coefficient sequence segment is used as the prompt action coefficient sequence segment, and the first action coefficient sequence segment is used as the target action coefficient sequence segment;

[0107] If the second driving source segment is used as the driving source, the first action coefficient sequence segment is used as the prompt action coefficient sequence segment, and the second action coefficient sequence segment is used as the target action coefficient sequence segment.

[0108] For example, let the action database be D = {(c i ,a i ,l i )|i=1,2,…,n}, with a total of n data. The driving source of the i-th data is c i , the corresponding facial action coefficient sequence is a i , the length of the corresponding action coefficient sequence is l i (l i is an integer); for sequence length l i Randomly split into two parts (the minimum length of the subsequence is l min ), and get the corresponding two sets of new sequence lengths l i1 ,l i2 (Integer and satisfy l i1 > min ,l i2 > min ,l i1 +l i2 ≤l i ). According to the segmentation, the corresponding driving source c is obtained. i1 , target action sequence a i1 And the prompt action coefficient sequence fragment p i1 ,satisfy:

[0109] c i1 =c i [0:l i1 ];

[0110] a i1 =a i [0:l i1 ];

[0111] p i1 =a i [l i1 :l i ];

[0112] Among them, c i1 is the driving source fragment, a i1 is the target action coefficient sequence fragment, p i1 To indicate the action coefficient sequence fragment, [0:li1 ] to 0 to l i1 This part, [l i1 :l i ] refers to l i1 To i This part (i.e. i2 this paragraph).

[0113] In some embodiments, the step of training a diffusion model according to the driving source segment, the target action coefficient sequence segment, and the prompt action coefficient sequence segment to obtain the facial action generation model includes:

[0114] Using the driving source fragment and the target action coefficient sequence fragment, the predicted noise output by the noise regression structure of the jth layer is obtained; Z≥j≥1, j is a positive integer;

[0115] Using the fine-tuning amount output by the attention module, the predicted noise of the j-th layer is adjusted to obtain a noise estimation value corresponding to the j-th layer; wherein the input of the attention module is the prompt action coefficient sequence and the predicted noise of the j-th layer; using the fine-tuning amount, the predicted noise corresponding to the j-th layer is adjusted to obtain a noise estimation value corresponding to the j-th layer;

[0116] Accumulate the noise estimation values ​​corresponding to the noise regression structure of the Z layer to obtain a noise prediction result;

[0117] The diffusion model is iteratively trained according to the loss value between the noise signal and the noise prediction result to obtain the facial action generation model.

[0118] In the specific implementation, assume that the target action sequence segment is x, the driving source segment is cond, and the prompt action coefficient sequence segment is prompt. Assume that the diffusion model is diffsuion, which includes a noise regression structure (WaveNet) and an attention prompt learning module (attention); the entire diffusion model has a total of Z layers. The specific steps of training the diffusion model may include:

[0119] Step 1: adding a random noise signal to the target action sequence segment to obtain noisy data;

[0120] Specifically, assuming that the number of noise scheduling steps is randomly generated, the maximum number of noise scheduling steps is T, and the corresponding noise coefficient is α t , β t The noise signal noise is randomly generated through Gaussian distribution, and the tensor form of the noise signal is consistent with the tensor form of the target action sequence fragment.

[0121] x noise =α t x+β tnoise;

[0122] Among them, α t , β t It is a method for calculating the noise step schedule coefficient of a diffusion model, which is used to add noise signal noise to data x; noise is noisy data.

[0123] Step 2: using the driving source fragment and the noisy data as inputs of the noise regression structure, performing noise prediction layer by layer, and obtaining the predicted noise corresponding to each layer;

[0124] Specifically, the diffusion model uses a cascaded noise regression structure (WaveNet) to predict noise and calculate noise layer by layer. Suppose the current layer is the jth layer, with a total of Z layers; the predicted noise of the jth layer is for:

[0125]

[0126] Where cond is the driving source fragment, t is the number of noise scheduling steps, is the noisy data output by the j-1th layer.

[0127] Step 3: Fine-tune the prediction noise of the j-th layer output by prompting the action sequence fragment.

[0128] Specifically, the prompt action sequence fragment and the predicted noise output by the jth layer are input into the attention module to obtain the output fine-tuning amount; the predicted noise corresponding to the jth layer is adjusted using the fine-tuning amount to obtain the noise estimation value corresponding to the jth layer. That is:

[0129]

[0130] Among them, q, k and v are the three input vectors of the attention model. Prompt and prompt are assigned to the three vectors q, k and v respectively to obtain the fine-tuning amount output by the attention model; the sum of the fine-tuning amount and the predicted noise corresponding to the j-th layer is used as the noise estimation value corresponding to the j-th layer.

[0131] Furthermore, according to the formula Get the noisy data of the current jth layer Enter the next layer and repeat the calculation until all calculations of the Z layer are completed.

[0132] Step 4: Estimated noise value of each layer in all Z layers Accumulate and get the noise prediction result: Simplified to:

[0133] noisepred = diffusion(x noise ,cond,prompt);

[0134] Among them, noise pred is the noise prediction result, cond is the driving source segment, and prompt is the prompt action coefficient sequence segment.

[0135] Step 5: Calculate the loss based on the noise prediction result and the input noise to achieve motion data distribution learning; repeat steps 1 to 4 above until the loss converges.

[0136] Among them, the loss uses the mean square error function: Loss MSE =(noise-noise pred ) 2 , Loss MSE is the loss value, noise is the input noise signal, pred The noise prediction result.

[0137] In the above embodiment, by fine-tuning the predicted noise by prompting the action coefficient sequence, the accuracy of noise prediction can be improved, which helps to improve the imitation similarity of the diffusion model.

[0138] In some embodiments, receiving the prompt action information sent by the client includes:

[0139] receiving prompt action information sent by the client, wherein the prompt action information is information of a target action snapshot, and the target action snapshot is one of at least one action snapshot displayed by the client; or,

[0140] The prompt action information sent by the client is received, where the prompt action information is a facial action video recorded by the user.

[0141] Optionally, the user can record voice and corresponding facial video clips of different emotional actions to build his own action prompt library, and the user can select corresponding facial action short videos from the action prompt library as prompt action information.

[0142] In the specific implementation, the user performs speaking with different amplitudes and emotions in front of the camera through a client (such as an APP that can transmit images and audio), and the background transmits facial action picture data and audio data to the server. The server obtains the corresponding driving source and facial action coefficient sequence through facial action capture software.

[0143] In the above embodiment, the user can select or record prompt action information that meets his or her needs through the client, and the server obtains the prompt action coefficient sequence based on the prompt action information sent by the client; thereby, based on the driving source and the prompt action coefficient sequence, the server drives the digital human to imitate the prompt action coefficient sequence to speak.

[0144] In some optional embodiments, if the facial action coefficient sequence output by the facial action generation model does not meet the user's needs, the facial video clips recorded by the user are added to the action sample database, and the facial action generation model is transferred and learned based on the updated action sample database to optimize the parameters and obtain a user-specific facial action generation model.

[0145] It should be pointed out that the action sample database in this application is obtained based on facial action videos of multiple actors, and can be used as a basic model to provide user-customized rapid transfer learning capabilities.

[0146] The following is combined with Figure 6 and 7 The actual operating effect of the embodiment of the present application is described.

[0147] See also Figure 6 , which shows the effect of imitating facial movements with similar degrees of imitation.

[0148] exist Figure 6 In each subgraph (i.e. Figure 6 DEMO1, DEMO2, DEMO3, DEMO4, DEMO5 in the figure include three facial actions, of which the left one is the facial action of a real actor when he speaks a specified line, and the right one is a similar prompt action imitating the left one (not directly using the left action); the middle one is the synthetic effect of the facial action generation model. That is, the input is the facial action and prompt action when the actor speaks the specified line, and the output is the imitation action.

[0149] The prompt segmentation of each group of sub-images is different. Among them, DEMO1 has a slight upward movement of the mouth corners and downward pressure on the eyebrows. The predicted data generates downward eyebrows and stretched mouth corners, but there is no obvious upward movement of the mouth corners. In DEMO2, the synthetic action shows the frowning and downward movement of the prompt action, which is relatively in line with expectations. In DEMO3, due to the difference between the prompt action and the real action itself, the synthetic action is more inclined to the prompt action, which is different from the real action. In DEMO4, the small mouth shape and frowning are the focus, which is basically in line with expectations. In DEMO5, this is a set of actions with smile lines. In terms of the chin amplitude, it tends to be in the prompt action style, accompanied by cheek lifting, and is even better than the real action data.

[0150] See also Figure 7, which shows a schematic diagram of facial movements for different interpretations of the same sentence.

[0151] Figure 7 In the same dialogue of the same actor, three different types of prompt actions were used to generate three different styles of speech sequences. Style 1 is a dull effect, with dull eyes and mainly chin movements, imitating the speech performance of drunkenness for action synthesis. Style 2 is a faint smile effect, with the corners of the mouth raised, the cheeks lifted, and a smile inadvertently while speaking. Style 3 is a disgust effect (the model's eyebrows are not obvious), with the whole frown and lowered eyebrows, and the corners of the mouth pulled down, showing a feeling of disgust.

[0152] From the above operation results, it can be seen that in the embodiment of the present application, the input is the driving source and the action prompt information, and the output is the facial action coefficient sequence. The facial action generation model can convert the driving source into a facial action coefficient sequence based on the characteristics learned from the action sample database, so that the digital human has the ability to quickly imitate the speaking styles of different people, and also has the ability to quickly imitate different emotions of the same person.

[0153] That is, the above embodiment can quickly generate a facial action coefficient sequence with a style similar to the action prompt through the driving source and action prompt information. By deepening the number of layers of the facial action generation model, it can support the learning of ultra-large-scale data sets and perform facial action style imitation under 2 or 3 seconds of action prompts. On the one hand, it can effectively replace the task of animators to manually make key frames, and transform the traditional key frame making task into a refinement task for animators based on synthetic data; on the other hand, it can effectively replace the animators to summarize the action style of the imitation target, and realize fast style imitation through a large-scale action library. In addition, users can record voice and corresponding different emotional action clips, build their own action prompt library, and realize digital humans to imitate user actions and emotions to speak according to the user's action prompt library.

[0154] See also Figure 8 The present application provides a method for generating digital human facial movements, which is applied to a client and includes:

[0155] Step 201: Send drive information and prompt action information to the server.

[0156] Optionally, the driving information is text information or audio information, such as spoken text or voice entered by the user.

[0157] Optionally, the prompt action information is a short video of facial action recorded by the user or a target action snapshot selected by the user; wherein the user selects a target action snapshot based on at least one action snapshot displayed by the client, and the action snapshot is a dynamic image generated based on some key frames of action coefficient sequence samples in an action sample database, which is used to show characteristic facial actions.

[0158] Step 202: receiving the digital human facial action video frame sent by the server.

[0159] Among them, the digital human facial action video frame is generated based on the facial action coefficient sequence output by the facial action generation model.

[0160] The facial action coefficient sequence refers to a vector with a dimension of R, where R elements are real numbers between 0 and 1. Each element of the dimension in the vector represents the action of a certain position in the facial features, and the corresponding value represents the amplitude of the action. For example, 0.5 jaw opening means that the amplitude of the jaw opening action in the facial action is 0.5.

[0161] Step 203: Play the digital human facial action video frame.

[0162] In the above embodiment, the server can convert the driving information into a facial action coefficient sequence based on the prompt action information sent by the client, so that the digital human has the ability to imitate the speech of characters with different styles, and also has the ability to quickly imitate different emotions of the same person. Users can use the prompt action information to achieve customized and personalized digital human facial driving effects to meet the usage needs of different users without requiring large labor costs and time costs.

[0163] In some embodiments, the digital human facial action generation method further includes:

[0164] displaying at least one action snapshot;

[0165] receiving a user input operation on a target action snapshot among the at least one action snapshot;

[0166] In the above step 201, the sending of prompt action information to the server includes: in response to the input operation, determining the prompt action coefficient sequence sample associated with the target action snapshot as the prompt action information, and sending the target action snapshot information to the server.

[0167] Optionally, the action snapshot may be generated by the server according to the action coefficient sequence samples in the action sample database and sent to the client, and the number of frames included in the action snapshot is specified by the user or is a default value.

[0168] For example, the user can view and select an action snapshot through the function portal displayed on the client, and based on the selection operation of the target action snapshot, trigger the sending of information of the target action snapshot to the server, so that based on the correspondence between the target action snapshot and the action coefficient sequence sample in the action sample database, the action coefficient sequence sample corresponding to the target action snapshot is selected, and the key frame segment is intercepted from the target action coefficient sequence sample as the prompt action coefficient sequence.

[0169] In some embodiments, in the above step 201, sending the prompt action information to the server includes:

[0170] The facial action video obtained by the user through the camera is used as the prompt action information, and the facial action video is sent to the server.

[0171] Accordingly, the server generates a corresponding prompt action coefficient sequence based on the facial action short video.

[0172] Optionally, the user can record voice and corresponding facial video clips of different emotional actions to build his own action prompt library, and the user can select corresponding facial action short videos from the action prompt library as prompt action information.

[0173] In the above embodiment, the user can select or record prompt action information that meets his or her needs through the client, and the server obtains the prompt action coefficient sequence based on the prompt action information sent by the client; thereby, based on the driving source and the prompt action coefficient sequence, the server drives the digital human to imitate the prompt action coefficient sequence to speak.

[0174] like Fig. 9 The embodiment of the present application provides a digital human facial action generation device 900, which is applied to a server, and includes:

[0175] The first receiving module 901 is used to receive the driving information and prompt action information sent by the client;

[0176] A first acquisition module 902, used for inputting the driving information and the prompting action information into a facial action generation model, and outputting a facial action coefficient sequence;

[0177] A generating module 903 is used to use the facial action coefficient sequence as facial action curve parameters to generate a digital human facial action video frame;

[0178] The first sending module 904 is used to send the digital human facial action video frame to the client.

[0179] Optionally, the digital human facial action generation device 900 further includes: a model training module, which includes:

[0180] A first training submodule is used to obtain an action sample database; wherein each driving source sample in the action sample database corresponds to an action coefficient sequence sample;

[0181] The second training submodule is used to perform data segmentation on each driving source sample and action coefficient sequence sample in the action sample database to obtain driving source segments, target action coefficient sequence segments and prompt action coefficient sequence segments;

[0182] The third training submodule is used to train a diffusion model according to the driving source segment, the target action coefficient sequence segment and the prompt action coefficient sequence segment to obtain the facial action generation model; wherein the diffusion model includes a cascaded Z-layer noise regression structure and an attention module, and Z is a positive integer.

[0183] Optionally, the second training submodule includes:

[0184] The first training unit is used to divide the i-th driving source sample into a first driving source segment and a second driving source segment; the i-th driving source sample is any one in the action sample database, and i is a positive integer;

[0185] A second training unit is used to divide the i-th action coefficient sequence sample corresponding to the i-th driving source sample into a first action coefficient sequence segment and a second action coefficient sequence segment; wherein the first driving source segment corresponds to the first action coefficient sequence segment, and the second driving source segment corresponds to the second action coefficient sequence segment;

[0186] a third training unit, configured to use the second action coefficient sequence segment as the prompt action coefficient sequence segment and the first action coefficient sequence segment as the target action coefficient sequence segment if the first driving source segment is used as the driving source;

[0187] The fourth training unit is used to use the first action coefficient sequence segment as the prompt action coefficient sequence segment and the second action coefficient sequence segment as the target action coefficient sequence segment if the second driving source segment is used as the driving source.

[0188] Optionally, the third training submodule includes:

[0189] A fifth training unit is used to obtain the predicted noise output by the noise regression structure of the jth layer by using the driving source segment and the target action coefficient sequence segment; Z≥j≥1, j is a positive integer;

[0190] A sixth training unit is used to adjust the predicted noise of the j-th layer using the fine-tuning amount output by the attention module to obtain a noise estimation value corresponding to the j-th layer; wherein the input of the attention module is the prompt action coefficient sequence and the predicted noise of the j-th layer; the predicted noise corresponding to the j-th layer is adjusted using the fine-tuning amount to obtain a noise estimation value corresponding to the j-th layer;

[0191] A seventh training unit is used to accumulate the noise estimation values ​​corresponding to the noise regression structure of the Z layer to obtain a noise prediction result;

[0192] The eighth training unit is used to iteratively train the diffusion model according to the loss value between the noise signal and the noise prediction result to obtain the facial action generation model.

[0193] Optionally, the first receiving module 901 includes:

[0194] A first receiving submodule is configured to receive prompt action information sent by the client, wherein the prompt action information is information of a target action snapshot, and the target action snapshot is one of at least one action snapshot displayed by the client; or

[0195] The second receiving submodule is used to receive the prompt action information sent by the client, where the prompt action information is a facial action video recorded by the user.

[0196] Since the principle of solving the problem by the digital human facial action generation device 900 is similar to the digital human facial action generation method applied to the server side in the embodiment of the present invention, the implementation of the digital human facial action generation device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0197] like Fig.10 The embodiment of the present application provides a digital human facial action generation device 1000, which is applied to a client and includes:

[0198] The second sending module 1001 is used to send driving information and prompt action information to the server;

[0199] The second receiving module 1002 is used to receive the digital human facial action video frame sent by the server;

[0200] The display module 1003 is used to play the digital human facial action video frame.

[0201] Optionally, the digital human facial action generation device 1000 further includes:

[0202] A display module, configured to display at least one action snapshot;

[0203] A third receiving module, configured to receive a user input operation on a target action snapshot in the at least one action snapshot;

[0204] The second sending module 1001 includes:

[0205] The first sending submodule is used to determine the prompt action coefficient sequence samples associated with the target action snapshot as the prompt action information in response to the input operation, and send the information of the target action snapshot to the server.

[0206] The second sending module 1001 includes:

[0207] The first sending submodule is used to use the facial action video obtained by the user through the camera as the prompt action information, and send the facial action video to the server.

[0208] Since the principle of solving the problem by the digital human facial action generation device 1000 is similar to the digital human facial action generation method applied to the client side in the embodiment of the present invention, the implementation of the digital human facial action generation device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0209] like Fig.11 As shown, the electronic device of an embodiment of the present invention is a server, including: a processor 1100; and a memory 1120 connected to the processor 1100 via a bus interface, the memory 1120 is used to store programs and data used by the processor 1100 when performing operations, and the processor 1100 calls and executes the programs and data stored in the memory 1120.

[0210] The transceiver 1110 is connected to the bus interface and is used to receive and send data under the control of the processor 1100; the processor 1100 is used to read the program in the memory 1120 and execute the following process:

[0211] Receive the driving information and prompt action information sent by the client;

[0212] Inputting the driving information and the prompting action information into a facial action generation model, and outputting a facial action coefficient sequence;

[0213] Using the facial action coefficient sequence as facial action curve parameters to generate digital human facial action video frames;

[0214] The digital human facial action video frame is sent to the client.

[0215] The transceiver 1110 is configured to receive and send data under the control of the processor 1100 .

[0216] Among them, Fig.11In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1100 and memory represented by memory 1120. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1110 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. The processor 1100 is responsible for managing the bus architecture and general processing, and the memory 1120 may store data used by the processor 1100 when performing operations.

[0217] The processor 1100 is responsible for managing the bus architecture and general processing, and the memory 1120 can store data used by the processor 1100 when performing operations.

[0218] Optionally, the processor 1100 is further configured to read a program in the memory 1120 and execute the following process:

[0219] Acquire an action sample database; wherein each driving source sample in the action sample database corresponds to an action coefficient sequence sample;

[0220] Performing data segmentation on each driving source sample and action coefficient sequence sample in the action sample database to obtain driving source segments, target action coefficient sequence segments, and prompt action coefficient sequence segments;

[0221] A diffusion model is trained according to the driving source segment, the target action coefficient sequence segment and the prompt action coefficient sequence segment to obtain the facial action generation model; wherein the diffusion model includes a cascaded Z-layer noise regression structure and an attention module, and Z is a positive integer.

[0222] Optionally, the processor 1100 is further configured to read a program in the memory 1120 and execute the following process:

[0223] The i-th driving source sample is divided into a first driving source segment and a second driving source segment; the i-th driving source sample is any one in the action sample database, and i is a positive integer;

[0224] The i-th action coefficient sequence sample corresponding to the i-th driving source sample is divided into a first action coefficient sequence segment and a second action coefficient sequence segment; wherein the first driving source segment corresponds to the first action coefficient sequence segment, and the second driving source segment corresponds to the second action coefficient sequence segment;

[0225] If the first driving source segment is used as the driving source, the second action coefficient sequence segment is used as the prompt action coefficient sequence segment, and the first action coefficient sequence segment is used as the target action coefficient sequence segment;

[0226] If the second driving source segment is used as the driving source, the first action coefficient sequence segment is used as the prompt action coefficient sequence segment, and the second action coefficient sequence segment is used as the target action coefficient sequence segment.

[0227] Optionally, the processor 1100 is further configured to read a program in the memory 1120 and execute the following process:

[0228] Using the driving source fragment and the target action coefficient sequence fragment, the predicted noise output by the noise regression structure of the jth layer is obtained; Z≥j≥1;

[0229] Using the fine-tuning amount output by the attention module, the predicted noise of the j-th layer is adjusted to obtain a noise estimation value corresponding to the j-th layer; wherein the input of the attention module is the prompt action coefficient sequence and the predicted noise of the j-th layer; using the fine-tuning amount, the predicted noise corresponding to the j-th layer is adjusted to obtain a noise estimation value corresponding to the j-th layer;

[0230] Accumulate the noise estimation values ​​corresponding to the noise regression structure of the Z layer to obtain a noise prediction result;

[0231] The diffusion model is iteratively trained according to the loss value between the noise signal and the noise prediction result to obtain the facial action generation model.

[0232] Optionally, the processor 1100 is further configured to read a program in the memory 1120 and execute the following process:

[0233] receiving prompt action information sent by the client, wherein the prompt action information is information of a target action snapshot, and the target action snapshot is one of at least one action snapshot displayed by the client; or,

[0234] The prompt action information sent by the client is received, where the prompt action information is a facial action video recorded by the user.

[0235] The server provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be repeated in this embodiment.

[0236] like Fig.12As shown, the electronic device of an embodiment of the present invention is a client device, including: a processor 1200; and a memory 1220 connected to the processor 1200 via a bus interface, the memory 1220 being used to store programs and data used by the processor 1200 when performing operations, and the processor 1200 calls and executes the programs and data stored in the memory 1220.

[0237] The transceiver 1210 is connected to the bus interface and is used to receive and send data under the control of the processor 1200; the processor 1200 is used to read the program in the memory 1220 and execute the following process:

[0238] Send drive information and prompt action information to the server;

[0239] Receiving digital human facial action video frames sent by the server;

[0240] Play the digital human facial action video frame.

[0241] The transceiver 1210 is configured to receive and send data under the control of the processor 1200 .

[0242] Among them, Fig.12 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by the processor 1200 and various circuits of the memory represented by the memory 1220 are linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1210 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. For different terminals, the user interface 1230 may also be an interface capable of externally connecting to required internal devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like. The processor 1200 is responsible for managing the bus architecture and general processing, and the memory 1220 may store data used by the processor 1200 when performing operations.

[0243] The processor 1200 is responsible for managing the bus architecture and general processing, and the memory 1220 can store data used by the processor 1200 when performing operations.

[0244] Optionally, the processor 1200 is further configured to read a program in the memory 1220 and execute the following process:

[0245] displaying at least one action snapshot;

[0246] receiving a user input operation on a target action snapshot among the at least one action snapshot;

[0247] In response to the input operation, a prompt action coefficient sequence sample associated with the target action snapshot is determined as the prompt action information, and the information of the target action snapshot is sent to the server.

[0248] Optionally, the processor 1200 is further configured to read a program in the memory 1220 and execute the following process:

[0249] The facial action video obtained by the user through the camera is used as the prompt action information, and the facial action video is sent to the server.

[0250] In addition, the computer-readable storage medium of the embodiment of the present invention is used to store a computer program, and the computer program can be executed by a processor to implement the steps of generating digital human facial movements as described above.

[0251] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0252] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0253] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0254] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0255] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating digital human facial movements, characterized in that: Applied to the server, including: Receive the driving information and prompt action information sent by the client; Inputting the driving information and the prompting action information into a facial action generation model, and outputting a facial action coefficient sequence; Using the facial action coefficient sequence as facial action curve parameters to generate digital human facial action video frames; The digital human facial action video frame is sent to the client.

2. The method for generating digital human facial movements according to claim 1, characterized in that: The training process of the facial action generation model includes: Acquire an action sample database; wherein each driving source sample in the action sample database corresponds to an action coefficient sequence sample; Performing data segmentation on each driving source sample and action coefficient sequence sample in the action sample database to obtain driving source segments, target action coefficient sequence segments, and prompt action coefficient sequence segments; A diffusion model is trained according to the driving source segment, the target action coefficient sequence segment and the prompt action coefficient sequence segment to obtain the facial action generation model; wherein the diffusion model includes a cascaded Z-layer noise regression structure and an attention module, and Z is a positive integer.

3. The method for generating digital human facial movements according to claim 2, characterized in that: The step of segmenting each driving source sample and each action coefficient sequence sample in the action sample database to obtain driving source segments, target action coefficient sequence segments, and prompt action coefficient sequence segments includes: The i-th driving source sample is divided into a first driving source segment and a second driving source segment; the i-th driving source sample is any one in the action sample database, and i is a positive integer; The i-th action coefficient sequence sample corresponding to the i-th driving source sample is divided into a first action coefficient sequence segment and a second action coefficient sequence segment; wherein the first driving source segment corresponds to the first action coefficient sequence segment, and the second driving source segment corresponds to the second action coefficient sequence segment; If the first driving source segment is used as the driving source, the second action coefficient sequence segment is used as the prompt action coefficient sequence segment, and the first action coefficient sequence segment is used as the target action coefficient sequence segment; If the second driving source segment is used as the driving source, the first action coefficient sequence segment is used as the prompt action coefficient sequence segment, and the second action coefficient sequence segment is used as the target action coefficient sequence segment.

4. The method for generating digital human facial movements according to claim 2, characterized in that: The step of training a diffusion model according to the driving source segment, the target action coefficient sequence segment and the prompt action coefficient sequence segment to obtain the facial action generation model comprises: Using the driving source fragment and the target action coefficient sequence fragment, the predicted noise output by the noise regression structure of the jth layer is obtained; Z≥j≥1, j is a positive integer; Using the fine-tuning amount output by the attention module, the predicted noise of the j-th layer is adjusted to obtain a noise estimation value corresponding to the j-th layer; wherein the input of the attention module is the prompt action coefficient sequence and the predicted noise of the j-th layer; using the fine-tuning amount, the predicted noise corresponding to the j-th layer is adjusted to obtain a noise estimation value corresponding to the j-th layer; Accumulate the noise estimation values ​​corresponding to the noise regression structure of the Z layer to obtain a noise prediction result; The diffusion model is iteratively trained according to the loss value between the noise signal and the noise prediction result to obtain the facial action generation model.

5. A method for generating digital human facial movements, characterized in that: Applied to the client, including: Send drive information and prompt action information to the server; Receiving digital human facial action video frames sent by the server; Play the digital human facial action video frame.

6. A digital human facial action generation device, characterized in that: include: A first receiving module, used for receiving driving information and prompting action information sent by the client; A first acquisition module, used for inputting the driving information and the prompting action information into a facial action generation model, and outputting a facial action coefficient sequence; A generating module, used for using the facial action coefficient sequence as facial action curve parameters to generate a digital human facial action video frame; The first sending module is used to send the digital human facial action video frame to the client.

7. A digital human facial action generation device, characterized in that: include: The second sending module is used to send driving information and prompt action information to the server; A second receiving module is used to receive the digital human facial action video frame sent by the server; The display module is used to play the digital human facial action video frame.

8. An electronic device comprising: A transceiver, a memory, a processor and a computer program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the method for generating digital human facial movements as described in any one of claims 1 to 4, or the steps in the method for generating digital human facial movements as described in claim 5.

9. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the digital human facial action generation method according to any one of claims 1 to 4 and the steps of the digital human facial action generation method according to claim 5 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the method for generating digital human facial movements as claimed in any one of claims 1 to 4, and the steps in the method for generating digital human facial movements as claimed in claim 5.

Images