Real-time mouth shape animation generation method and device, storage medium and computer equipment

By predicting and correcting the deformation coefficient of facial expression units of the speech stream frame by frame, the problem of mismatch between speech and lip shape is solved, and the sound and picture synchronization of animation is improved.

CN119068085APending Publication Date: 2024-12-03SHANGHAI KAITUO HUANYU TECH DEV CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310640836.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, when speech and lip simultaneous rendering, the difference in speech speed leads to lip simultaneous mismatch, low sensitivity, and poor sound and picture synchronization.

Method used

By predicting the deformation coefficient of facial expression units of the speech stream frame by frame, the difference between the deformation coefficients of the target frame and the historical reference frame is calculated, and the scaling correction is performed to improve the sensitivity and sound and picture synchronization of lip animation.

Benefits of technology

It improves the sensitivity of lip animation, enhances the synchrony of sound and picture, and solves the problem of mismatch between voice and lip shape.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068085A_ABST
    Figure CN119068085A_ABST
Patent Text Reader

Abstract

The invention discloses a real-time mouth shape animation generation method and device, a storage medium and computer equipment, and the method comprises the steps: carrying out the frame-by-frame deformation coefficient prediction of a facial expression unit of a voice stream in response to a voice stream input signal, and obtaining the deformation coefficient of each frame; for any target frame deformation coefficient in the frame deformation coefficients, acquiring a reference frame deformation coefficient of a historical reference frame corresponding to the target frame, calculating a difference between the target frame deformation coefficient and the reference frame deformation coefficient as a target frame deformation coefficient gradient, and performing scaling calculation on the target frame deformation coefficient gradient to obtain a target frame deformation coefficient gradient; correcting the deformation coefficient of the target frame by using a target frame zooming gradient calculated by zooming; and according to the corrected deformation coefficient of each frame, driving the facial expression units on the target role model frame by frame to obtain the mouth shape animation of the target role. According to the invention, the method helps to improve the sensitivity of the mouth shape animation in a voice-driven mouth shape animation scene, and further improves the voice and picture synchronism of the animation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of animation production technology, and in particular to a real-time lip animation generation method and device, a storage medium, and a computer device. Background Art

[0002] Virtual character modeling and rendering technology is widely used in industries such as animation, games, and movies. Enabling virtual characters to speak naturally and fluently and with synchronized lip movements is the key to improving user experience. In a real-time system, it is necessary to play the audio acquired in real time in the form of a stream and the synchronously rendered virtual character image synchronously. In this process, the synchronization between the audio and the character's lip movements must be ensured.

[0003] At present, some applications have been made by training models with speech samples, using speech as a driving source to predict the deformation coefficient of expression units using models. However, when the predicted speech and speech sample have a large difference in speech speed, the prediction results of this method will have problems such as mismatch between speech and mouth shape, low mouth shape sensitivity, and poor synchronization between sound and picture. Summary of the invention

[0004] In view of this, the present application provides a real-time lip animation generation method and apparatus, a storage medium, and a computer device, which help to improve the sensitivity of lip animation in the scene of voice-driven lip animation, thereby improving the synchronization of audio and video of the animation.

[0005] According to one aspect of the present application, a real-time lip animation generation method is provided, the method comprising:

[0006] In response to a voice stream input signal, predicting deformation coefficients of facial expression units of the voice stream frame by frame to obtain deformation coefficients of each frame;

[0007] For any target frame deformation coefficient among the frame deformation coefficients, obtain the reference frame deformation coefficient of the historical reference frame corresponding to the target frame, calculate the difference between the target frame deformation coefficient and the reference frame deformation coefficient as the target frame deformation coefficient gradient, perform scaling calculation on the target frame deformation coefficient gradient, and correct the target frame deformation coefficient using the scaled target frame scaling gradient calculated by scaling;

[0008] According to the corrected deformation coefficients of each frame, the facial expression unit on the target character model is driven frame by frame to obtain the lip animation of the target character.

[0009] According to another aspect of the present application, a real-time lip animation generation device is provided, characterized in that the device comprises:

[0010] A prediction module, for predicting the deformation coefficient of the facial expression unit of the voice stream frame by frame in response to the voice stream input signal, and obtaining the deformation coefficient of each frame;

[0011] A correction module, for obtaining, for any target frame deformation coefficient among the frame deformation coefficients, a reference frame deformation coefficient of a historical reference frame corresponding to the target frame, calculating a difference between the target frame deformation coefficient and the reference frame deformation coefficient as a target frame deformation coefficient gradient, performing scaling calculation on the target frame deformation coefficient gradient, and correcting the target frame deformation coefficient using the scaled target frame scaling gradient calculated by scaling;

[0012] The driving module is used to drive the facial expression unit on the target character model frame by frame according to the corrected deformation coefficients of each frame to obtain the lip animation of the target character.

[0013] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned real-time lip animation generation method is implemented.

[0014] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned real-time lip animation generation method when executing the program.

[0015] By means of the above technical scheme, the present application provides a real-time lip animation generation method and device, storage medium, and computer equipment, which predict the input voice stream frame by frame, obtain the deformation coefficients of each frame of the facial expression unit, correct the deformation coefficients of each frame, and then drive the facial expression unit on the target character model to deform, and obtain the lip animation of the target character, wherein, when correcting any target frame deformation coefficient, the target frame deformation coefficient gradient is calculated in combination with the reference frame deformation coefficient, and the target frame deformation coefficient gradient is scaled within a certain range, and the scaled gradient is used to correct the target frame deformation coefficient. The present application uses the reference frame deformation coefficient to calculate the target frame deformation coefficient gradient and scales it within a certain range, and then corrects the target frame deformation coefficient, which helps to improve the sensitivity of the lip animation in the scene where real-time voice drives the lip animation, and then improves the synchronization of the animation with sound and picture.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A schematic diagram of a flow chart of a real-time lip animation generation method provided in an embodiment of the present application is shown;

[0019] Figure 2 A schematic diagram showing a deformation coefficient correction effect provided by an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram showing a flow chart of another real-time lip animation generation method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the structure of a real-time lip animation generation device provided in an embodiment of the present application is shown;

[0022] Figure 5 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0024] In this embodiment, a real-time lip animation generation method is provided. Figure 1 As shown, the method includes:

[0025] Step 101, in response to a voice stream input signal, predicting the deformation coefficient of the facial expression unit of the voice stream frame by frame to obtain the deformation coefficient of each frame.

[0026] The embodiment of the present application can be applied to the application scenario of using real-time voice to drive the model expression to generate lip animation. The deformation coefficient of each facial expression unit is predicted based on the input voice stream, and after the deformation coefficient is corrected, the facial expression unit is driven to deform using the corrected deformation coefficient, thereby obtaining the lip animation. The facial expression unit can be specifically a Blendshape, which refers to the deformation target of the model vertex animation. It can make the character model vertex displace according to the specified deformation target without binding the bones. For example, it can be specifically 52 Blendshapes corresponding to the ARKit workflow defined by Apple.

[0027] Step 102, for any target frame deformation coefficient in each frame deformation coefficient, obtain the reference frame deformation coefficient of the historical reference frame corresponding to the target frame, calculate the difference between the target frame deformation coefficient and the reference frame deformation coefficient as the target frame deformation coefficient gradient, and perform scaling calculation on the target frame deformation coefficient gradient, and use the scaled target frame scaling gradient to correct the target frame deformation coefficient.

[0028] In this embodiment, the input voice stream is predicted frame by frame to obtain the deformation coefficient of each frame. In order to improve the synchronization effect of sound and picture, the deformation coefficient of each frame can be corrected before driving the facial expression unit to generate animation. Specifically, when correcting the deformation coefficient of any target frame, first determine the historical reference frame corresponding to the target frame, and obtain the deformation coefficient of the historical reference frame. For example, the historical reference frame can specifically select the previous frame of the target frame, and the reference frame deformation coefficient is the deformation coefficient of the frame before the target frame. It should be noted here that the reference frame deformation coefficient is preferably selected from the deformation coefficient of the participating frame after correction. Next, the difference between the target frame deformation coefficient and the reference frame deformation coefficient is calculated as the target frame deformation coefficient gradient. The target frame deformation coefficient gradient is used to reflect the degree of change of the deformation coefficient of the target frame relative to the historical reference frame, and the target frame deformation coefficient gradient is further scaled to a certain extent. For example, in a scenario with a fast speech flow, the target frame deformation coefficient gradient is multiplied by a coefficient greater than 1 (such as 1.2) for scaling, and the target frame deformation coefficient is corrected by the target frame scaling gradient obtained after scaling.

[0029] In an embodiment of the present application, optionally, in step 102, the target frame deformation coefficient gradient is scaled and calculated, and the target frame deformation coefficient is corrected using the scaled and calculated target frame scaling gradient, including: calculating the target frame scaling scale according to the target frame deformation coefficient and a preset scaling parameter; obtaining the reference frame deformation coefficient gradient corresponding to the historical reference frame, and mixing the target frame deformation coefficient gradient and the reference frame deformation coefficient gradient to obtain a target frame mixed gradient; scaling and calculating the target frame mixed gradient according to the target frame scaling scale to obtain a target frame scaling gradient, and calculating the corrected target frame deformation coefficient based on the reference frame deformation coefficient and the target frame scaling gradient.

[0030] In this embodiment, the target frame scaling scale scale used in the scaling calculation is calculated using the preset scaling parameters and the target frame deformation coefficient. The target frame deformation coefficient gradient dx and the reference frame deformation coefficient gradient dx_prev corresponding to the historical reference frame are mixed, and the above-mentioned target frame scaling scale is used to perform scaling calculation on the mixed target frame mixed gradient. Specifically, the preset gradient weighting coefficient alpha_dx can be used to weight the sum of the two gradients, the target frame mixed gradient dx_hat=dx_prev*(1-alpha_dx)+dx*alpha_dx, and the target frame scaling gradient dx_scale=dx_hat*scale. The target frame deformation coefficient is further corrected according to the reference frame deformation coefficient and the target frame scaling gradient, for example, the corrected target frame deformation coefficient x_hat=x_prev+dx_scale. By setting the specific value of the preset scaling parameter, the size of the deformation coefficient within a certain range can be adjusted, making the correction more flexible and further improving the sensitivity of the deformation coefficient in the local area.

[0031] Further, in an embodiment of the present application, optionally, calculating the target frame scaling according to the target frame deformation coefficient and the preset scaling parameters includes: calculating an initial scaling of the target frame according to the target frame deformation coefficient and the preset scaling parameters, wherein the preset scaling parameters include a corrected range center value center, a corrected range size sigma, and a scaling coefficient amp, and the initial scaling = amp*exp(-(x–center)^2 / sigma); determining a scaling range corresponding to the scaling coefficient, truncating the initial scaling within the scaling range, and obtaining the target frame scaling.

[0032] In this embodiment, a preliminary scaling can be calculated first, and the preliminary scaling can be further limited to a certain range to avoid overcorrection. Specifically, the initial scaling y = amp*exp(-(x–center)^2 / sigma). Further, according to the scaling coefficient in the scaling parameter, the scaling range is determined. For example, if the scaling coefficient amp is greater than 1, the scaling range can be determined to be [1, amp]. If the preliminary scaling is less than 1, the target scaling is 1. If the preliminary scaling is greater than 1, the target scaling is amp. If the preliminary scaling falls within the range, it remains unchanged. For example, the scaling coefficient corresponding to the corrected speech speed range slower than the standard speech speed range is less than 1, and the scaling coefficient corresponding to the corrected speech speed range faster than the standard speech speed range is greater than 1. In the scenario where real-time speech drives the lip shape, it is sometimes necessary to adjust the sensitivity of the lip shape in real time (such as a fast-speech-speed speaking scenario), especially the sensitivity within a local range. By calculating the scaling of the target frame and correcting the deformation coefficient, the sensitivity of the local lip shape coefficient can be adjusted.

[0033] In the above embodiment, optionally, the method also includes: determining a target speaking rate range in which the target frame speaking rate falls within a plurality of corrected speaking rate ranges, wherein the plurality of corrected speaking rate ranges and the standard speaking rate range do not overlap with each other, and the corrected speaking rate range at least includes a range greater than the standard speaking rate range; and obtaining a preset scaling parameter corresponding to the target speaking rate range.

[0034] In this embodiment, by setting preset scaling parameters for different speech speed ranges, it is possible to achieve different degrees of correction of the deformation coefficient prediction values ​​corresponding to speech at different speech speeds. Specifically, the speech speed of the speech sample sampled when training the deformation coefficient prediction model can be used as the standard speech speed, and the standard speech speed range can be set based on the standard speech speed. Further, based on the standard speech speed range, multiple correction speech speed ranges that do not overlap with the standard speech speed range are set, and corresponding preset scaling parameters are set for each correction speech speed range. For example, in the scene where the deformation coefficient is corrected for a scene with a faster speech speed, amp is greater than 1, and the target frame scaling scale scale = clip (y, a_min = 1, a_max = amp), and y is truncated to the [1, amp] interval. For another example, in the scene where the deformation coefficient is corrected for a scene with a faster speech speed, amp is less than 1, and the target frame scaling scale scale = clip (y, a_min = amp, a_max = 1), and y is truncated to the [amp, 1] interval.

[0035] In the embodiment of the present application, the sum of the reference frame deformation coefficient and the target frame scaling gradient can be directly used as the corrected target frame deformation coefficient. To further improve the correction effect, a secondary correction can be performed. Optionally, the calculation of the corrected target frame deformation coefficient based on the reference frame deformation coefficient and the target frame scaling gradient includes: calculating the sum of the reference frame deformation coefficient and the target frame scaling gradient as the target frame intermediate deformation coefficient; mixing the target frame intermediate deformation coefficient and the reference frame deformation coefficient to obtain the corrected target frame deformation coefficient.

[0036] In this embodiment, the sum of the reference frame deformation coefficient and the target frame scaling gradient is taken as the preliminary correction result, that is, the target frame intermediate deformation coefficient. The reference frame deformation coefficient and the preliminary correction result are further mixed to obtain the final target frame deformation coefficient after secondary correction. Specifically, the reference frame deformation coefficient and the target frame intermediate deformation coefficient can be weighted and summed using the preset deformation coefficient weighting coefficient alpha_x, and the corrected target frame deformation coefficient x_final = x_prev*(1–alpha_x)+x_hat*alpha_x.

[0037] Step 103, according to the corrected deformation coefficients of each frame, the facial expression unit on the target character model is driven frame by frame to obtain the lip animation of the target character.

[0038] Finally, after correcting the deformation coefficient frame by frame, the corrected deformation coefficient is used to drive the facial expression unit on the target character model to deform frame by frame, so as to obtain the lip animation of the target character, improve the sensitivity of the target character's lip shape, help synchronize the sound and picture, and solve the problem of mismatch between speech and lip shape.

[0039] By applying the technical solution of this embodiment, the input voice stream is predicted frame by frame to obtain the deformation coefficients of each frame of the facial expression unit, and the deformation coefficients of each frame are corrected to drive the facial expression unit on the target character model to deform, so as to obtain the lip animation of the target character, wherein, when correcting any target frame deformation coefficient, the target frame deformation coefficient gradient is calculated in combination with the reference frame deformation coefficient, and the target frame deformation coefficient gradient is scaled within a certain range, and the scaled gradient is used to correct the target frame deformation coefficient. The present application uses the reference frame deformation coefficient to calculate the target frame deformation coefficient gradient and scales it within a certain range, and then corrects the target frame deformation coefficient, which helps to improve the sensitivity of the lip animation in the scene where the voice drives the lip animation, and then improves the synchronization of the animation with sound and picture.

[0040] In a specific application scenario, the correction scheme for the deformation coefficient of any frame can be described as follows:

[0041] 1. Define parameters: x: represents the deformation coefficient of the facial expression unit. dx: represents the difference between the current frame x and the previous frame x. alpha_x: the weighted coefficient of the deformation coefficient between the current frame x and the previous frame x. alpha_dx: the gradient weighted coefficient between the current frame gradient dx and the previous frame gradient dx. center: the center value of the range to be locally adjusted (the center value of the corrected range). sigma: the size of the range to be locally adjusted (the size of the corrected range). amp: the maximum adjustment multiple.

[0042] 2. Correction process of the deformation coefficient predicted frame by frame:

[0043] 1) Initialize the first frame parameters: x_prev = x; dx_prev = 0

[0044] 2) Filter and correct the current frame

[0045] a. Calculate the scale: y = amp*exp(-(x–center)^2 / sigma), where exp is an exponential function; scale = clip(y, a_min = 1, a_max = amp), truncate y to the interval [1, amp]. In an example, center = 0.18, sigma = 0.01, amp = 1.2. The corresponding deformation coefficient correction effect diagram is as follows: Figure 2 ,Depend on Figure 2 It can be seen that scale scaling will only affect the local interval near the center, and adjust the deformation coefficient within a certain range to achieve the adjustment of the sensitivity of the local lip coefficient.

[0046] b. Calculate the current frame x gradient: dx = x – x_prev

[0047] c. Calculate the weighted gradient dx_hat = dx_prev*(1-alpha_dx)+dx*alpha_dx

[0048] d. Calculate the scaled gradient dx_scale = dx_hat * scale

[0049] e. Calculate the initial filtering result of the current frame x x_hat = x_prev + dx_scale

[0050] f. Calculate the weighted final result x_final = x_prev*(1–alpha_x)+x_hat*

[0051] alpha_x

[0052] g. Update the first frame parameters: x_prev = x_hat; dx_prev = dx_hat

[0053] Return x_final as the corrected deformation parameter. For the next frame, return to a.

[0054] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another real-time lip animation generation method is provided, such as Figure 3 As shown, the method includes:

[0055] Step 301, collecting the facial expression data of the simulated character through a motion capture system.

[0056] Step 302: If the expression data is valid, the facial expression unit on the target character model is driven based on the expression data to obtain the lip animation of the target character.

[0057] Step 303: if the expression data is invalid, a voice stream is acquired, a real-time speech rate of the voice stream is identified in response to a voice stream input signal, and a deformation coefficient prediction model is used to predict the voice stream frame by frame to obtain deformation coefficients of each frame.

[0058] Step 304 , determining whether the real-time speaking speed falls within a standard speaking speed range, wherein the standard speaking speed range is determined based on the sample speaking speed of the speech sample used when training the deformation coefficient prediction model.

[0059] Step 305, when the real-time speech rate falls within the standard speech rate range, directly drive the facial expression unit on the target character model according to the predicted deformation coefficients of each frame to obtain the lip animation of the target character.

[0060] Step 306, when the real-time speech rate does not fall within the standard speech rate range, the deformation coefficients of each frame are corrected, and the facial expression units on the target character model are driven frame by frame according to the corrected deformation coefficients of each frame to obtain the lip animation of the target character.

[0061] In this embodiment, the solution of driving lip animation with voice stream can be used as a fault-tolerant solution for the motion capture system to drive lip animation. For example, during the live broadcast of a virtual person, the expression data of a real person is collected, and the collected expression data is used to drive the expression of the virtual person. When the expression data collected by the motion capture system is valid, the collected expression data is directly used to calculate the deformation coefficient of the facial expression unit and perform animation driving. When the expression data collected by the motion capture system is invalid, the input voice stream is used to drive the lip animation. Furthermore, if speech rate recognition can also be performed based on the voice stream, if the real-time speech rate of the voice stream falls within the standard speech rate range, then the deformation coefficient predicted by the deformation coefficient prediction model can be directly used to drive the lip animation. If the real-time speech rate of the voice stream does not fall within the standard speech rate range, then it can be based on the following: Figure 1The deformation coefficient correction scheme provided by the illustrated embodiment uses the real-time corrected deformation parameters to drive the lip animation. The standard speech rate range is specifically the sample speech rate of the speech sample used when training the deformation coefficient prediction model, for example, it can be a standard broadcasting speech rate.

[0062] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a real-time lip animation generation device, such as Figure 4 As shown, the device comprises:

[0063] A prediction module, for predicting the deformation coefficient of the facial expression unit of the voice stream frame by frame in response to the voice stream input signal, and obtaining the deformation coefficient of each frame;

[0064] A correction module, for obtaining, for any target frame deformation coefficient among the frame deformation coefficients, a reference frame deformation coefficient of a historical reference frame corresponding to the target frame, calculating a difference between the target frame deformation coefficient and the reference frame deformation coefficient as a target frame deformation coefficient gradient, performing scaling calculation on the target frame deformation coefficient gradient, and correcting the target frame deformation coefficient using the scaled target frame scaling gradient calculated by scaling;

[0065] The driving module is used to drive the facial expression unit on the target character model frame by frame according to the corrected deformation coefficients of each frame to obtain the lip animation of the target character.

[0066] Optionally, the correction module is further used to:

[0067] Calculating a target frame scaling scale according to the target frame deformation coefficient and a preset scaling parameter;

[0068] Acquire a reference frame deformation coefficient gradient corresponding to the historical reference frame, and mix the target frame deformation coefficient gradient and the reference frame deformation coefficient gradient to obtain a target frame mixed gradient;

[0069] The target frame mixed gradient is scaled and calculated according to the target frame scaling scale to obtain the target frame scaling gradient, and the corrected target frame deformation coefficient is calculated based on the reference frame deformation coefficient and the target frame scaling gradient.

[0070] Optionally, the correction module is further used to:

[0071] Calculating the sum of the reference frame deformation coefficient and the target frame scaling gradient as the target frame intermediate deformation coefficient;

[0072] The target frame intermediate deformation coefficient and the reference frame deformation coefficient are mixed to obtain a corrected target frame deformation coefficient.

[0073] Optionally, the correction module is further used to:

[0074] Calculate the initial scaling scale of the target frame according to the target frame deformation coefficient and the preset scaling parameters, wherein the preset scaling parameters include the center value of the corrected range center, the corrected range size sigma and the scaling coefficient amp, and the initial scaling scale = amp*exp(-(x–center)^2 / sigma);

[0075] A scaling scale range corresponding to the scaling scale coefficient is determined, and the initial scaling scale is truncated within the scaling scale range to obtain the target frame scaling scale.

[0076] Optionally, the prediction module is further used to: in response to the voice stream input signal, identify the real-time speech speed of the voice stream, and use the deformation coefficient prediction model to predict the voice stream frame by frame to obtain the deformation coefficient of each frame; determine whether the real-time speech speed falls within the standard speech speed range, wherein the standard speech speed range is determined based on the sample speech speed of the voice sample used when training the deformation coefficient prediction model;

[0077] The driving module is further used to: when the real-time speech rate falls within the standard speech rate range, directly drive the facial expression unit on the target character model according to the predicted deformation coefficients of each frame to obtain the lip animation of the target character;

[0078] The correction module is also used for: when the real-time speech rate does not fall within the standard speech rate range, continuing to execute the step of obtaining the reference frame deformation coefficient of the historical reference frame corresponding to the target frame for any target frame deformation coefficient among the frame deformation coefficients.

[0079] Optionally, the correction module is further used to:

[0080] Determine a target speech rate range in which the target frame speech rate falls among a plurality of corrected speech rate ranges, wherein the plurality of corrected speech rate ranges and the standard speech rate range do not overlap with each other, and the corrected speech rate range at least includes a range greater than the standard speech rate range;

[0081] Obtain a preset scaling parameter corresponding to the target speech rate range.

[0082] Optionally, the device further comprises:

[0083] Motion capture module, used to collect the facial expression data of the simulated character through the motion capture system;

[0084] The driving module is further used to: if the expression data is valid, drive the facial expression unit on the target character model based on the expression data to obtain the lip animation of the target character;

[0085] The prediction module is further used to: if the expression data is invalid, obtain the voice stream.

[0086] It should be noted that for other corresponding descriptions of the functional units involved in the real-time lip animation generation device provided in the embodiment of the present application, reference can be made to Figures 1 to 3 The corresponding description in the method will not be repeated here.

[0087] The present application also provides a computer device, which may be a personal computer, a server, a network device, etc. Figure 5 As shown, the computer device includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.

[0088] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0090] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0091] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0092] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0093] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A real-time lip animation generation method, characterized in that: The method comprises: In response to a voice stream input signal, predicting deformation coefficients of facial expression units of the voice stream frame by frame to obtain deformation coefficients of each frame; For any target frame deformation coefficient among the frame deformation coefficients, obtain the reference frame deformation coefficient of the historical reference frame corresponding to the target frame, calculate the difference between the target frame deformation coefficient and the reference frame deformation coefficient as the target frame deformation coefficient gradient, perform scaling calculation on the target frame deformation coefficient gradient, and correct the target frame deformation coefficient using the scaled target frame scaling gradient calculated by scaling; According to the corrected deformation coefficients of each frame, the facial expression unit on the target character model is driven frame by frame to obtain the lip animation of the target character.

2. The method according to claim 1, characterized in that: The step of scaling the target frame deformation coefficient gradient and correcting the target frame deformation coefficient using the target frame scaling gradient obtained by scaling, comprises: Calculating a target frame scaling scale according to the target frame deformation coefficient and a preset scaling parameter; Acquire a reference frame deformation coefficient gradient corresponding to the historical reference frame, and mix the target frame deformation coefficient gradient and the reference frame deformation coefficient gradient to obtain a target frame mixed gradient; The target frame mixed gradient is scaled and calculated according to the target frame scaling scale to obtain the target frame scaling gradient, and the corrected target frame deformation coefficient is calculated based on the reference frame deformation coefficient and the target frame scaling gradient.

3. The method according to claim 2, characterized in that The step of calculating the corrected target frame deformation coefficient based on the reference frame deformation coefficient and the target frame scaling gradient comprises: Calculating the sum of the reference frame deformation coefficient and the target frame scaling gradient as the target frame intermediate deformation coefficient; The target frame intermediate deformation coefficient and the reference frame deformation coefficient are mixed to obtain a corrected target frame deformation coefficient.

4. The method according to claim 2, characterized in that: The step of calculating the target frame scaling scale according to the target frame deformation coefficient and the preset scaling parameter includes: Calculate the initial scaling scale of the target frame according to the target frame deformation coefficient and the preset scaling parameters, wherein the preset scaling parameters include the center value of the corrected range center, the corrected range size sigma and the scaling coefficient amp, and the initial scaling scale = amp*exp(-(x–center)^2 / sigma); A scaling scale range corresponding to the scaling scale coefficient is determined, and the initial scaling scale is truncated within the scaling scale range to obtain the target frame scaling scale.

5. The method according to any one of claims 2 to 4, characterized in that The method predicts the deformation coefficient of the facial expression unit of the voice stream frame by frame in response to the voice stream input signal to obtain the deformation coefficient of each frame, including: In response to a voice stream input signal, the real-time speech rate of the voice stream is identified, and the voice stream is predicted frame by frame using a deformation coefficient prediction model to obtain deformation coefficients of each frame; Determining whether the real-time speaking speed falls within a standard speaking speed range, wherein the standard speaking speed range is determined based on a sample speaking speed of a speech sample used when training the deformation coefficient prediction model; When the real-time speech rate falls within the standard speech rate range, the facial expression unit on the target character model is driven directly according to the predicted deformation coefficients of each frame to obtain the lip animation of the target character; When the real-time speech rate does not fall within the standard speech rate range, the step of obtaining the reference frame deformation coefficient of the historical reference frame corresponding to the target frame for any target frame deformation coefficient among the frame deformation coefficients is continued.

6. The method according to claim 5, characterized in that Before calculating the target frame scaling scale according to the target frame deformation coefficient and the preset scaling parameter, the method further includes: Determine a target speech rate range in which the target frame speech rate falls among a plurality of corrected speech rate ranges, wherein the plurality of corrected speech rate ranges and the standard speech rate range do not overlap with each other, and the corrected speech rate range at least includes a range greater than the standard speech rate range; Obtain a preset scaling parameter corresponding to the target speech rate range.

7. The method according to any one of claims 1 to 4, characterized in that Before predicting the deformation coefficient of the facial expression unit on the voice stream frame by frame, the method further comprises: Collect the facial expression data of the imitated character through the motion capture system; If the expression data is valid, then based on the expression data, the facial expression unit on the target character model is driven to obtain the lip animation of the target character; If the expression data is invalid, then the voice stream is obtained.

8. A real-time lip animation generation device, characterized in that: The device comprises: A prediction module, for predicting the deformation coefficient of the facial expression unit of the voice stream frame by frame in response to the voice stream input signal, and obtaining the deformation coefficient of each frame; A correction module, for obtaining, for any target frame deformation coefficient among the frame deformation coefficients, a reference frame deformation coefficient of a historical reference frame corresponding to the target frame, calculating a difference between the target frame deformation coefficient and the reference frame deformation coefficient as a target frame deformation coefficient gradient, performing scaling calculation on the target frame deformation coefficient gradient, and correcting the target frame deformation coefficient using the scaled target frame scaling gradient calculated by scaling; The driving module is used to drive the facial expression unit on the target character model frame by frame according to the corrected deformation coefficients of each frame to obtain the lip animation of the target character.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Digital human voice mouth shape synchronous control system based on multi-modal feature dynamic fusion

    CN120319262A