A three-dimensional face animation generation method and device based on audio driving and a medium

Abstract

The application discloses a three-dimensional face animation generation method and device based on audio driving and a medium, wherein the method comprises the following steps: acquiring input audio and extracting audio features; predicting the speaking style of a current speaker according to the audio features; inputting the obtained speaking style and audio features into a self-recurrence coefficient prediction model to predict the three-dimensional face model coefficients corresponding to the audio frame by frame; mapping the obtained three-dimensional face model coefficients to three-dimensional vertex coordinates; calculating the loss according to the obtained three-dimensional vertex coordinates and real coordinates, and training the model by using a mouth shape and face key point loss function. The application solves the problems of weak generalization and low mouth shape matching degree of the three-dimensional face animation generation method by using the self-recurrence coefficient prediction algorithm, effectively improves the mouth shape matching degree of the generated animation and the driving ability of the model for new faces or new audio. The application can be widely applied to the technical field of three-dimensional face driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional face driving, and particularly relates to a three-dimensional face animation generation method and device based on audio driving and a medium. BACKGROUND

[0002] In virtual instructors, online customer service, film production and many other scenarios, the production and application of three-dimensional face animation have extensive needs. However, the previous way of producing three-dimensional face animation is very time-consuming and laborious, and the production quality greatly depends on the experience of animators. In order to improve the efficiency of producing three-dimensional face animation and improve the lip matching degree of three-dimensional face animation, researchers have begun to pay extensive attention to the method of generating three-dimensional face animation based on audio driving.

[0003] The generation of three-dimensional face animation based on audio driving refers to obtaining a corresponding three-dimensional face animation sequence for a piece of audio through a specific technical means, and keeping the mouth shape consistent with the audio in vision. The existing method of generating three-dimensional face animation based on audio driving mainly includes two categories: rule-based method and deep learning-based method. The rule-based method uses the inherent correspondence between pronunciation and mouth shape to establish rule constraints, which is equivalent to constructing a mouth shape query table. According to the rule, the mouth shape for each pronunciation is determined and spliced to finally obtain the three-dimensional face animation. However, this method has the following problems: 1) poor generalization. Since everyone has different styles when speaking, special rules need to be established; 2) low efficiency. The final animation can only be obtained with the participation of manpower. Therefore, researchers have begun to explore deep learning-based methods.

[0004] The deep learning-based generation method uses an audio-three-dimensional face animation dataset to train a deep model, so that it can implicitly model the relationship between audio and three-dimensional face. After training is completed, the deep model can extract features from the audio and map them to three-dimensional face vertex coordinates to obtain three-dimensional face animation. However, the existing deep learning-based method has two problems: 1) difficult data acquisition. A three-dimensional face generally consists of thousands of vertices and face pieces. In order to ensure data accuracy, the acquisition environment is generally very complex and difficult to build; 2) weak model generalization. Most existing methods directly map audio features to three-dimensional face vertices, which contain identity information, which leads to the model giving a mouth movement style that does not belong to the new face itself when driving the new face. SUMMARY

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a three-dimensional face animation generation method and device based on audio driving and a medium.

[0006] The technical solution adopted by the present application is:

[0007] A three-dimensional face animation generation method based on audio driving, comprising the following steps:

[0008] Obtaining input audio and extracting audio features;

[0009] Predicting the speaking style of the current speaker according to the audio features;

[0010] Inputting the obtained speaking style and audio features into an autoregressive coefficient prediction model to predict the three-dimensional face model coefficients corresponding to the audio frame by frame;

[0011] Mapping the obtained three-dimensional face model coefficients to three-dimensional vertex coordinates;

[0012] Calculating the loss according to the obtained three-dimensional vertex coordinates and real coordinates, and training the model using a mouth shape and face key point loss function.

[0013] Further, the obtaining input audio and extracting audio features comprises:

[0014] Converting the input audio into a one-dimensional vector X according to a preset sampling rate a , extracting audio features f a from the one-dimensional vector X a ;

[0015] Using linear interpolation to interpolate the audio features to a preset frame rate, so that the frame rate of the audio features is consistent with the frame rate of the real three-dimensional face animation.

[0016] Further, the predicting the speaking style of the current speaker according to the audio features comprises:

[0017] Using a linear rectifier layer, a batch normalization layer, and a Softmax layer to construct a style encoding model;

[0018] Inputting the audio features into the style encoding model for prediction, and outputting style encoding as the speaking style of the current speaker.

[0019] Further, the style encoding model adds a style consistency loss to the style encoding when training:

[0020] Let S i represent the style encoding of the i-th frame audio feature, and the definition of the style consistency loss is as follows:

[0021]

[0022] Where, represents the average value of the style encoding of all frames, and the style consistency loss constrains each frame of the style encoding to be as similar as possible.

[0023] Furthermore, the autoregressive coefficient prediction model includes a Transformer decoder layer and two linear rectifier layers;

[0024] The step of inputting the obtained speaking style and audio features into the autoregressive coefficient prediction model to predict the 3D face model coefficients corresponding to the audio frame by frame includes:

[0025] A1. When predicting the coefficients of the 3D face model corresponding to the first frame of audio, the speaking style S and the audio features of the first frame of audio are used. As input to the Transformer decoder, feature F is extracted. 1 Using the first linear rectifier layer based on feature F 1 The predicted coefficients C of the first frame are obtained. 1 ;

[0026] A2. Using the second linear rectifier layer based on the first frame coefficient C 1 Extract the lip-shape animation feature A from the first frame. 1 As the current lip-shape animation feature A = A 1 ;

[0027] A3, in predicting the coefficient C of the second frame 2 At that time, the speaking style S and the current lip-sync animation feature A will be used. 1 Add them together, combining the audio features of the first two frames. The coefficients C of the first two frames are predicted using the Transformer decoder and the first-layer linear rectifier. 1 ,2 ;

[0028] A4. Using the second linear rectifier layer based on the predicted second frame coefficients C 2 Obtain the second frame's lip-shape animation feature A 2 And update the current lip-shape animation feature to A = concat(A 1 A 2 );

[0029] A5. Predicting the 3D face model coefficients C corresponding to the i-th frame of audio. i At that time, based on the audio features of the previous i frames The current lip-sync animation feature A is obtained by concatenating the speaking style S and the lip-sync animation features of the previous i-1 frames. 1 ,…,A i-1 The predicted coefficients C of the first i frames are obtained. 1,…,i Finally, the coefficients of the entire audio segment are obtained.

[0030] Furthermore, mapping the obtained 3D face model coefficients to 3D vertex coordinates includes:

[0031] The obtained three-dimensional face model coefficients C are mapped to corresponding three-dimensional face sequences through a blendshape coefficient mapping function of a three-dimensional face model.

[0032] Further, the model is trained according to a loss calculated from obtained three-dimensional vertex coordinates and real coordinates, and a mouth shape and face key point loss function, including:

[0033] The mean square error loss between the face vertex V and the real vertex The mean square error loss between the vertices of the mouth region And the minimum absolute error loss of the mouth key point And the style encoding model and the autoregressive coefficient prediction model are updated using stochastic gradient descent.

[0034] Further, the final loss function is defined as:

[0035]

[0036]

[0037]

[0038]

[0039] Wherein, M mouth Indicates the mask of the three-dimensional face model mouth region, (i,j) indicates the key point pair corresponding to the upper and lower lips, E indicates the set of all key point pairs, k and sΠ(·) respectively indicate the real key point and the mapping function of the key point coordinates obtained according to the predicted vertex mapping, V i Indicates the i-th frame three-dimensional face vertex coordinates, and λ indicates the weight of the corresponding loss function.

[0040] An audio-driven three-dimensional face animation generation device, comprising:

[0041] At least one processor;

[0042] At least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0044] Another technical solution adopted by the present application is:

[0045] A computer readable storage medium, wherein a processor executable program is stored, the processor executable program is used to execute the method as described above when executed by the processor.​

[0046] The application has the beneficial effects that the application solves the problems of weak generalization and low mouth shape matching degree of the three-dimensional face animation generation method by using the autoregressive coefficient prediction algorithm, and effectively improves the mouth shape matching degree of the generated animation and the driving ability of the model for new faces or new audio. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0048] Figure 1 is a schematic diagram of a three-dimensional face animation generation method based on audio driving in an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of a style encoding model in an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of an autoregressive coefficient prediction model in an embodiment of the present application;

[0051] Figure 4 is a schematic diagram of a three-dimensional face mouth region mask (left) and key points (right) in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0054] In the description of the present application, the meaning of one or more is one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, within, etc. are understood as including the number. If it is described as first, second, it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the sequence of indicated technical features.

[0055] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical scheme.

[0056] As shown in Figure 1 , the embodiment provides a three-dimensional face animation generation method based on audio driving, comprising the following steps:

[0057] S1, acquiring input audio and extracting audio features.

[0058] Using a pre-trained audio processing model E a to extract audio features f a ∈N×D a , where N represents the number of audio frames, D a represents the dimension of the audio features. In order to enable the neural network to process audio, the input audio needs to be first converted into a one-dimensional vector X a ∈1×L, where L represents the total number of audio sampling points, and then the features f a can be extracted. a = E a (X

[0059] It should be noted that since the frame rate of the extracted audio features may not be consistent with the frame rate of the real three-dimensional face animation, linear interpolation is needed to interpolate the audio features to the specified frame rate.

[0060] S2, predicting the speaking style of the current speaker according to the audio features.

[0061] Referring to Figure 2 , a style encoding model is constructed using a linear rectifier layer, a batch normalization layer and a Softmax layer to map the audio features f a to the speaking style encoding S. It should be noted that the style encoding model will predict a style encoding for each frame of audio features, but the speaking style in the same speech should be fixed or approximate. Therefore, when training the model, a style consistency loss needs to be added to the style encoding.

[0062] Let S iLet style encoding represent the audio features of the i-th frame. The style consistency loss is defined as follows:

[0063]

[0064] in, This represents the average style encoding across all frames. Style consistency loss is used to constrain the style encoding of each frame to remain as similar as possible.

[0065] S3. Input the obtained speaking style and audio features into the autoregressive coefficient prediction model, and predict the 3D face model coefficients corresponding to the audio frame by frame.

[0066] See Figure 3 This embodiment proposes an autoregressive 3D face model coefficient prediction module based on a Transformer decoder T and a linear rectifier layer. This module combines previously predicted coefficient features, speech style encoding S, and audio features f. a Perform coefficient prediction. Predict the coefficients C of the 3D face model corresponding to the first frame of audio. 1 At that time, style-coded S and the audio features of the first frame were used. The final features are extracted from the input of the Transformer decoder. Then, the first linear rectifier layer is used according to F. 1 The predicted coefficients for the first frame are:

[0067] C 1 =FC1(F 1 )

[0068] Then use the second linear rectifier layer according to C 1 Extract the lip-shape animation feature A from the first frame. 1 =FC2(C 1 As the current lip-shape animation feature A = A 1 ; In predicting the coefficient C of the second frame 2 When that happens, S and the current lip-shape animation feature A will be used. 1 Add them together, combining the audio features of the first two frames. The coefficients for the first two frames are predicted using the Transformer decoder and the first-layer linear rectified layer:

[0069]

[0070] Then, the second linear rectifier layer is used based on the predicted second frame coefficient C. 2 Obtain the second frame's lip-shape animation feature A 2 =FC2(C 2 ), and update the current lip-shape animation feature to A = concat(A 1 A 2); and so on, in predicting the 3D face model coefficients C corresponding to the i-th frame of audio. i At that time, based on the audio features of the previous i frames The current lip animation feature A is obtained by concatenating the style code S and the lip animation features of the previous i-1 frames. 1 ,…,A i-1 The coefficients of the first i frames are predicted as follows:

[0071]

[0072] The process of predicting subsequent coefficients is repeated in this way, and finally the coefficients of the entire audio segment are obtained in an autoregressive manner.

[0073] S4. Map the obtained 3D face model coefficients to 3D vertex coordinates.

[0074] Using the blendshape coefficient mapping function G(·)→(V,E) of the 3D face model, all predicted coefficients C are mapped to the corresponding 3D face sequences, where vertices V∈N×n v ×3,n v E represents the number of vertices in a 3D face model, E∈N×n f ×3,n f Indicates the number of pieces of dough.

[0075] S5. Calculate the loss based on the obtained 3D vertex coordinates and the real coordinates, and train the model using the mouth shape and facial key point loss functions.

[0076] See Figure 4 Calculate the predicted face vertex V and the actual vertex V. Mean square error loss between Mean square error loss between vertices in the mouth region and the minimum absolute value error loss of key points of the mouth The style encoding model and the autoregressive coefficient prediction model are updated using stochastic gradient descent. The final loss function is defined as:

[0077]

[0078]

[0079]

[0080]

[0081] Among them, M mouthMask representing the mouth region of the three-dimensional face model, (i,j) represents a key point pair corresponding to the upper and lower lips, E represents the set of all key point pairs, k and sΠ(·) represent the mapping function of the real key point and the key point coordinates obtained by mapping according to the predicted vertex, V i represents the i-th frame three-dimensional face vertex coordinates, and λ represents the weight of the corresponding loss function. The mouth loss and key point loss introduced by us can make the model pay more attention to the movement of the mouth, thereby improving the mouth shape matching degree of the generated three-dimensional face animation.

[0082] In summary, the present embodiment has at least the following advantages and beneficial effects compared to the prior art:

[0083] (1) The embodiment of the present application proposes a style encoding model, which can more accurately describe the speaking style of the speaker.

[0084] (2) The embodiment of the present application proposes an autoregressive coefficient prediction model, which converts the face driving problem into an autoregressive coefficient prediction problem, uses an autoregressive prediction mode to improve the timing of the three-dimensional face animation, and uses an attention mechanism to improve the matching degree between the audio and the mouth animation modal.

[0085] (3) The embodiment of the present application introduces a three-dimensional face model, and obtains three-dimensional face vertices based on predicted coefficients.

[0086] (4) The embodiment of the present application proposes a mouth shape loss and a face key point loss, which further optimizes the mouth shape matching degree of the three-dimensional face animation generation result.

[0087] The embodiment also provides a three-dimensional face animation generation device based on audio driving, comprising:

[0088] At least one processor;

[0089] At least one memory for storing at least one program;

[0090] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method.

[0091] The three-dimensional face animation generation device based on audio driving of the embodiment can execute the three-dimensional face animation generation method based on audio driving provided by the method embodiment of the present application, and can execute the steps of any combination of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0092] The embodiment of the present application further discloses a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method shown in the method. Figure 1

[0093] The embodiment further provides a storage medium storing instructions or programs for performing the method for generating a three-dimensional face animation based on an audio drive.

[0094] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, and the purpose is to provide a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0095] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is unnecessary for an understanding of the present application. Rather, consideration of the properties, functions and internal relationships of the various functional modules disclosed in the devices herein, within the context of the ordinary skill of the engineer, will be sufficient to understand the actual implementation of the module. Therefore, those skilled in the art can implement the present application as set forth in the claims without undue experimentation using ordinary skill. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0096] ​If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0097] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution systems, apparatus or devices. For the purpose of this specification, the "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0098] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be electronically obtained and then stored in the computer memory.

[0099] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their equivalents, can be employed for implementation: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0100] In the above description of the present specification, the description of the terms "one embodiment / one example", "another embodiment / another example", or "certain embodiments / certain examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0101] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents.

[0102] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A method for generating 3D face animation based on audio-driven methods, characterized in that, Includes the following steps: Acquire the input audio and extract its features; The current speaker's speaking style is predicted based on the audio features. The obtained speaking style and audio features are input into the autoregressive coefficient prediction model to predict the coefficients of the three-dimensional face model corresponding to the audio frame by frame. Map the obtained 3D face model coefficients to 3D vertex coordinates; The loss is calculated based on the obtained 3D vertex coordinates and the real coordinates, and the model is trained using mouth shape and facial key point loss functions; The autoregressive coefficient prediction model includes a Transformer decoder layer and two linear rectifier layers. The step of inputting the obtained speaking style and audio features into the autoregressive coefficient prediction model to predict the 3D face model coefficients corresponding to the audio frame by frame includes: When predicting the coefficients of the 3D face model corresponding to the first frame of audio, the speech style S and the audio features of the first frame of audio are used. As input to the Transformer decoder, feature F is extracted. 1 Using the first linear rectifier layer based on feature F 1 The predicted coefficients C of the first frame are obtained. 1 ; Using the second linear rectifier layer based on the first frame coefficient C 1 Extract the mouth animation feature A from the first frame. 1 As the current lip-shape animation feature A = A 1 ; Predicting the coefficient C in the second frame 2 At that time, the speaking style S and the current lip-sync animation feature A will be used. 1 Add them together, combining the audio features of the first two frames. The coefficients C of the first two frames are predicted using the Transformer decoder and the first-layer linear rectifier. 1,2 ; Using the second linear rectifier layer, based on the predicted second frame coefficient C... 2 Obtain the second frame's lip-shape animation feature A 2 And update the current lip-shape animation feature to A = concat(A 1 A 2 ); In predicting the 3D face model coefficients C corresponding to the i-th frame of audio. i At that time, based on the audio features of the previous i frames The current lip-sync animation feature A is obtained by concatenating the speech style S and the lip-sync animation features of the previous i-1 frames. 1 ,…,A i-1 The predicted coefficients C of the first i frames are obtained. 1,…,i Finally, the coefficients of the entire audio segment are obtained.

2. The method for generating 3D face animation based on audio-driven technology according to claim 1, characterized in that, The process of acquiring input audio and extracting audio features includes: The input audio is converted into a one-dimensional vector X according to a preset sampling rate. a According to the one-dimensional vector X a Extracting audio features f a ; Linear interpolation is used to interpolate audio features to a preset frame rate so that the frame rate of the audio features is consistent with the frame rate of the real 3D face animation.

3. The method for generating 3D face animation based on audio-driven technology according to claim 1, characterized in that, The step of predicting the current speaker's speaking style based on the audio features includes: A style coding model is constructed using a linear rectifier layer, a batch normalization layer, and a softmax layer. The audio features are input into the style coding model for prediction, and the style code is output as the current speaker's speaking style.

4. The method for generating 3D face animation based on audio-driven technology according to claim 3, characterized in that, During training, the style encoding model incorporates a style consistency loss term into the style encoding process. Let S i Let style encoding represent the audio features of the i-th frame. The style consistency loss is defined as follows: in, This represents the average style encoding of all frames, and the style encoding of each frame is constrained to remain as similar as possible through style consistency loss.

5. The method for generating 3D face animation based on audio-driven technology according to claim 1, characterized in that, The process of mapping the obtained 3D face model coefficients to 3D vertex coordinates includes: The obtained 3D face model coefficients C are mapped to the corresponding 3D face sequence using the blendshape coefficient mapping function of the 3D face model.

6. The method for generating 3D face animation based on audio-driven technology according to claim 1, characterized in that, The process of calculating the loss based on the obtained 3D vertex coordinates and the true coordinates, and training the model using mouth shape and facial key point loss functions, includes: Calculate the face vertex V and the real vertex V based on the obtained 3D vertex coordinates and the real vertex. Mean square error loss between Mean square error loss between vertices in the mouth region and the minimum absolute value error loss of key points of the mouth The stochastic gradient descent style coding model and the autoregressive coefficient prediction model are updated.

7. An audio-driven 3D face animation generation device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1-6.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-6.

Images