A method and system for generating digital human lips

Abstract

The application discloses a digital human lip shape generation method and system, and belongs to the technical field of virtual digital human driving and computer animation. The method comprises the following steps: input audio is framed, MFCC features are extracted, and initial phoneme labels are recognized; a standard phoneme-lip shape mapping library is called to obtain multi-dimensional basic lip shape parameters; dynamic offset correction coefficients are generated in combination with the audio speed and tone color two-dimensional features to perform individual correction on the basic lip shape parameters; then, target digital human lip region topology information is automatically recognized, vertex displacement mapping weights are generated through six-region partition calculation, and the lip shape parameters are converted into model drivable vertex animation sequences; finally, a three-order Bezier curve smoothing interpolation with front and rear frame constraints is performed, and continuous lip shape animation is output. The application controls the lip sound synchronization error within 35ms, can adapt to mainstream topology structure digital human models without manual parameter adjustment, can realize real-time lip shape generation on a consumer-level GPU, and is suitable for live broadcast, online education and film and television production scenes.

Description

Technical Field

[0001] This invention relates to the field of digital human driving technology, and in particular to a method and system for generating digital human lip shapes. Background Technology

[0002] With the widespread application of virtual digital humans in fields such as live streaming, online education, and film and television production, the synchronization between lip movements and speech has become one of the core indicators for measuring the realism of digital humans. Existing mainstream digital human lip-shape generation schemes are mainly divided into two categories: one is a rule-based phoneme mapping method, which pre-establishes a standard lip shape parameter table corresponding to each phoneme and directly retrieves the corresponding lip shape based on the phoneme recognition results of the input audio; the other is an end-to-end generation method based on deep learning, which trains a neural network through a large amount of audio-lip-shape pairing data to directly predict lip shape sequences from audio features.

[0003] Existing technologies suffer from two common minor drawbacks: First, rule-based methods produce abrupt lip transitions, prone to abrupt changes in lip shape during continuous pronunciation, and unable to adapt to personalized pronunciation habits with different timbres and speaking speeds. Second, while end-to-end deep learning methods offer smooth transitions, they are prone to "asynchrony" issues when encountering uncommon pronunciations or accented speech, with lip shapes lagging behind the audio, typically exhibiting synchronization errors exceeding 80ms, far exceeding the 40ms error threshold perceptible to the human eye. Furthermore, most existing solutions do not consider the topological differences in digital human facial models; the same generation algorithm exhibits vastly different adaptation effects on digital human models with varying face counts and lip region weights, requiring extensive manual parameter tuning before it can be used.

[0004] Therefore, there is an urgent need for a lightweight digital human lip shape generation method and system with lower synchronization error and stronger model adaptability, which has become a technical problem that needs to be solved in this field. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a method and system for generating digital human lip shapes.

[0006] In a first aspect, embodiments of the present invention provide a method for generating digital human lip shapes, including:

[0007] S100. Acquire the input audio signal, perform frame segmentation on the audio signal, extract the Mel-frequency cepstral coefficients (MFCC) features of each frame of audio, and output the initial phoneme label corresponding to each frame of audio through a pre-trained phoneme recognition model.

[0008] S200. Retrieve a pre-constructed standard phoneme-lip shape mapping library, which stores the basic lip shape parameter vectors corresponding to each phoneme;

[0009] S300. Calculate the speech rate features and timbre features of the current input audio, generate dynamic offset correction coefficients based on the speech rate features and timbre features, and use the dynamic offset correction coefficients to offset and correct the basic lip shape parameter vector to obtain the corrected lip shape parameters;

[0010] S400. Obtain the lip region topology information of the target digital human model, automatically calculate the mapping weight from lip shape parameters to model vertex displacement, and convert the corrected lip shape parameters into a lip region vertex animation sequence that can be directly driven by the digital human model through the mapping weight;

[0011] S500. Performs smooth interpolation on the lip region vertex animation sequence of adjacent frames to output a continuous final lip animation.

[0012] Furthermore, in S200, the lip shape parameter vector includes at least normalized values ​​for four dimensions: upper lip height, lower lip height, mouth width, and lip tip protrusion.

[0013] Furthermore, in S300, a dynamic offset correction coefficient is generated based on the speech rate characteristics and timbre characteristics. The specific method includes: statistically analyzing the phoneme switching frequency of a continuous preset number of audio frames to calculate the current speech rate value V; setting the lip opening amplitude correction coefficient corresponding to the standard speech rate value V0 to 1; when V > V0, for every first preset percentage increase in speech rate, the lip opening amplitude parameter is correspondingly reduced by a second preset percentage; when V ≤ V0, for every third preset percentage decrease in speech rate, the lip opening amplitude parameter is correspondingly increased by a fourth preset percentage.

[0014] Furthermore, in S300, dynamic offset correction coefficients are generated based on timbre characteristics. The specific method includes: extracting the fundamental frequency mean of the audio as timbre characteristics;

[0015] When the average fundamental frequency is lower than the first threshold, the lip tip protrusion parameter is increased by the fifth preset ratio; when the average fundamental frequency is higher than the second threshold, the mouth width parameter is increased by the sixth preset ratio.

[0016] Furthermore, in S400, the vertex displacement mapping weights are automatically calculated, specifically through the following methods:

[0017] The spatial coordinates and UV information of all vertices in the lip region of the digital human model are identified, and the lip region is divided into 6 independent regions: upper lip left region, upper lip middle region, upper lip right region, lower lip left region, lower lip middle region, and lower lip right region. Based on the Euclidean distance from the vertex in each region to the center point of the mouth, the displacement weight of each vertex corresponding to each dimension of the lip shape parameter is automatically generated to complete the automatic adaptation of different topological models.

[0018] Furthermore, in S500, the smooth interpolation process uses third-order Bézier curve interpolation, and introduces motion prediction constraints for one forward frame and one backward frame to make the speed and acceleration of the lip shape change continuous.

[0019] Secondly, this invention also discloses a digital human lip shape generation system, comprising: an audio processing module, a storage module, a dynamic correction module, a model adaptation module, and an animation output module; wherein:

[0020] The audio processing module is used to perform frame-by-frame processing on the input audio signal, extract the MFCC features of each frame, and output the initial phoneme label corresponding to each frame through a pre-trained phoneme recognition model.

[0021] The storage module is used to store a pre-built standard phoneme-lip shape mapping library, which contains multi-dimensional basic lip shape parameter vectors corresponding to each phoneme;

[0022] The dynamic correction module is used to calculate the speech rate and timbre features of the input audio, generate dynamic offset correction coefficients, and perform offset correction on the basic lip shape parameter vector to obtain the corrected lip shape parameters.

[0023] The model adaptation module is used to obtain the lip region topology information of the target digital human model, automatically calculate the mapping weight from lip shape parameters to model vertex displacement, and convert the corrected lip shape parameters into a lip region vertex animation sequence.

[0024] The animation output module is used to perform smooth interpolation on the lip region vertex animation sequence and output the final continuous lip animation.

[0025] Furthermore, the animation output module supports offline lip-shape animation file output, and also supports real-time output of lip-shape driving parameters via UDP protocol, with end-to-end latency lower than a preset value.

[0026] Thirdly, the present invention discloses an electronic device, comprising:

[0027] One or more processors;

[0028] Memory, used to store one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0030] A computer-readable medium having a computer program stored thereon, the computer program implementing the steps of the method when executed by a processor.

[0031] This invention provides a method and system for generating lip shapes in a digital human. The method includes: acquiring an input audio signal; performing frame-by-frame processing on the audio signal; extracting the Mel-frequency cepstral coefficients (MFCC) features of each frame of audio; and outputting initial phoneme labels corresponding to each frame of audio through a pre-trained phoneme recognition model; retrieving a pre-constructed standard phoneme-lip shape mapping library, wherein the mapping library stores basic lip shape parameter vectors corresponding to each phoneme; calculating the speech rate and timbre features of the current input audio; generating dynamic offset correction coefficients based on the speech rate and timbre features; using the dynamic offset correction coefficients to perform offset correction on the basic lip shape parameter vectors to obtain corrected lip shape parameters; acquiring the lip region topology information of the target digital human model; automatically calculating the mapping weights from the lip shape parameters to the model vertex displacements; converting the corrected lip shape parameters into a lip region vertex animation sequence that can be directly driven by the digital human model through the mapping weights; and performing smooth interpolation processing on the lip region vertex animation sequences of adjacent frames to output continuous final lip shape animations. This invention controls the lip-sound synchronization error to within 35ms, adapts to mainstream digital human models without manual parameter tuning, and has a single-frame processing time of less than 2ms. It can achieve real-time lip-shape generation on consumer-grade GPUs and is suitable for scenarios such as live streaming, online education, and film and television production.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. For the first time, a dynamic offset correction mechanism with two dimensions of speech rate and timbre was introduced. Compared with the traditional fixed mapping lip shape generation method, the lip-sound synchronization error was reduced from 80ms to less than 35ms, which is completely lower than the synchronization error threshold that can be perceived by the human eye, and greatly improved the realism of digital human videos.

[0034] 2. A new automatic lip region topology adaptation function has been added, eliminating the need for manual weight adjustment for different digital human models. The model adaptation efficiency has been improved by more than 90%, and ordinary operators can quickly deploy and use it without professional animation experience.

[0035] 3. The overall algorithm is lightweight, with a single frame processing time of less than 2ms. Real-time lip-shape generation can be achieved on ordinary consumer-grade GPUs, fully meeting the usage requirements of low-latency scenarios such as live streaming. Attached Figure Description

[0036] Figure 1 A flowchart illustrating a digital human lip shape generation method provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the dynamic offset correction coefficient generation logic provided in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of lip region segmentation provided in an embodiment of the present invention;

[0039] Figure 4 A structural block diagram of a digital human lip shape generation system provided in an embodiment of the present invention;

[0040] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0042] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0043] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0044] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0045] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0046] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0047] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a method and system for generating digital human lip shapes.

[0048] This implementation discloses a method for generating digital human lip shapes, such as... Figure 1 ,include:

[0049] S100. Acquire the input audio signal, perform frame segmentation processing, extract the Mel-Frequency Cepstral Coefficients (MFCC) features of each frame, and output the initial phoneme label corresponding to each frame through a pre-trained phoneme recognition model. The Mel-Frequency Cepstral Coefficients (MFCC) are acoustic feature parameters extracted based on human auditory perception characteristics and are one of the most widely used standard features in speech recognition, speaker recognition, and audio signal processing. Their core function is to convert the original time-domain audio waveform into a low-dimensional, high-information-density feature vector for subsequent model inference, feature matching, and computation.

[0050] For example, a mono audio signal with an input sampling rate of 16kHz is acquired, and the audio is processed by using the standard audio framing method with a frame length of 25ms and a frame shift of 10ms. 13-dimensional MFCC features are extracted from each frame, and a phoneme recognition model pre-trained based on the CNN+CTC structure is called to obtain the phoneme label corresponding to each frame. The frame rate of phoneme recognition is consistent with the audio frame rate to ensure that a phoneme recognition result is output every 10ms.

[0051] S200. Retrieve a pre-constructed standard phoneme-lip shape mapping library, which stores basic lip shape parameter vectors corresponding to each phoneme; in S200 of this embodiment, the lip shape parameter vectors include at least normalized values ​​of four dimensions: upper lip height, lower lip height, mouth width, and lip tip protrusion.

[0052] Specifically, a pre-built standard phoneme-lip shape mapping library is invoked, which covers all phonemes of Mandarin Chinese and the phoneme combinations corresponding to 412 toned syllables. Each phoneme corresponds to a set of 4-dimensional basic lip shape parameter vectors, with parameters including upper lip height, lower lip height, mouth width, and lip tip protrusion. All parameters are normalized to the range [0,1], where 0 represents a closed / no displacement state and 1 represents a maximum open / displaced state.

[0053] For example, the basic lip shape parameter vector corresponding to the phoneme / a / is: upper lip height 0.32, lower lip height 0.45, mouth width 0.78, and lip tip protrusion 0.21; the basic lip shape parameter vector corresponding to the phoneme / i / is: upper lip height 0.12, lower lip height 0.15, mouth width 0.35, and lip tip protrusion 0.42. Based on the phoneme label of each frame, the corresponding basic lip shape parameter vector is retrieved from the mapping library to generate the initial lip shape parameter sequence frame by frame.

[0054] S300. Calculate the speech rate features and timbre features of the current input audio, generate dynamic offset correction coefficients based on the speech rate features and timbre features, and use the dynamic offset correction coefficients to offset and correct the basic lip shape parameter vector to obtain the corrected lip shape parameters;

[0055] In S300 of this implementation, a dynamic offset correction coefficient is generated based on the speech rate characteristics and timbre characteristics. The specific method includes: statistically analyzing the phoneme switching frequency of a continuous preset number of audio frames to calculate the current speech rate value V; setting the lip opening amplitude correction coefficient corresponding to the standard speech rate value V0 to 1; when V > V0, for every first preset percentage increase in speech rate, the lip opening amplitude parameter is correspondingly decreased by a second preset percentage; when V ≤ V0, for every third preset percentage decrease in speech rate, the lip opening amplitude parameter is correspondingly increased by a fourth preset percentage.

[0056] Preferably, the number of phoneme switching times within 100 consecutive audio frames is counted to calculate the current speech rate value V, in units of phonemes per second. The standard speech rate V0 is set to 8 phonemes per second, with a corresponding correction coefficient base value of 1 for lip opening and closing amplitude.

[0057] The correction rules are as follows: When V > V0, for every 10% increase in speaking speed, the opening and closing amplitude parameters of the upper lip height and lower lip height will be reduced by 5% to avoid excessive exaggeration of lip opening and closing during fast pronunciation; when V ≤ V0, for every 10% decrease in speaking speed, the opening and closing amplitude parameters will be increased by 3% to avoid excessively weak lower lip movements during slow speech.

[0058] For example, if the current speech rate is 12 phonemes / second, which is 50% higher than the standard speech rate, the overall opening and closing amplitude parameter is reduced by 25%, so that the lip movements at fast speech rate are more in line with real physiological performance.

[0059] In S300 of this implementation, a dynamic offset correction coefficient is generated based on the timbre characteristics. The specific method includes: extracting the fundamental frequency mean of the audio as the timbre characteristic; when the fundamental frequency mean is lower than the first threshold, increasing the lip tip protrusion parameter by a fifth preset ratio; when the fundamental frequency mean is higher than the second threshold, increasing the mouth width parameter by a sixth preset ratio.

[0060] Preferred, such as Figure 2 When the average fundamental frequency is below 150Hz (typically a deep tone in adult males), the lip tip protrusion parameter is increased by 8% to enhance the thickness of the lips and match the visual characteristics of the deep tone; when the average fundamental frequency is above 300Hz (typically a sharp tone in children's voices or high-pitched female voices), the mouth width parameter is increased by 6% to match the mouth force characteristics of high-frequency tones.

[0061] The speech rate correction coefficient and the timbre correction coefficient are superimposed on the basic lip shape parameter vector to finally obtain the corrected lip shape parameter sequence.

[0062] S400. Obtain the lip region topology information of the target digital human model, automatically calculate the mapping weight from lip shape parameters to model vertex displacement, and convert the corrected lip shape parameters into a lip region vertex animation sequence that can be directly driven by the digital human model through the mapping weight;

[0063] In this implementation of S400, the vertex displacement mapping weights are automatically calculated, specifically through the following methods:

[0064] The spatial coordinates and UV information of all vertices in the lip region of the digital human model are identified, and the lip region is divided into 6 independent regions: upper lip left region, upper lip middle region, upper lip right region, lower lip left region, lower lip middle region, and lower lip right region. Based on the Euclidean distance from the vertex in each region to the center point of the mouth, the displacement weight of each vertex corresponding to each dimension of the lip shape parameter is automatically generated to complete the automatic adaptation of different topological models.

[0065] Specifically, the S300 enables unmanned adaptation of digital human models with different topologies, and the lip segmentation method is as follows: Figure 3 As shown, the specific process is as follows:

[0066] Lip region vertex localization: Read the mesh data of the target digital human 3D model, and automatically locate all vertices of the lip region through the UV coordinate range of the lip region and facial semantic tags to obtain the 3D spatial coordinates of each vertex.

[0067] Functional area division: The lip area is divided into 6 independent functional areas, namely the upper lip left area, upper lip middle area, upper lip right area, lower lip left area, lower lip middle area, and lower lip right area; with the center point of the mouth as the reference point, the Euclidean distance from each vertex to the reference point is calculated.

[0068] Displacement weight calculation: For the four dimensions of the lip shape parameters, displacement weights are calculated for each vertex: vertices closer to the center of the mouth receive higher displacement weights; the weights of vertices farther away decrease with a gradient. Wherein:

[0069] The upper lip height parameter mainly affects three areas of the upper lip;

[0070] The lower lip height parameter mainly affects three areas of the lower lip;

[0071] The mouth width parameter mainly affects the left and right lateral lip areas;

[0072] The lip tip protrusion parameter mainly affects the middle area of ​​the upper and lower lips.

[0073] Vertex animation generation: Through weight mapping, the normalized lip shape parameters are converted into the actual 3D displacement of each vertex, generating a frame-by-frame sequence of lip region vertex animations.

[0074] The entire adaptation process is completed automatically, without the need for manual keyframe setting or weight adjustment, and can adapt to digital human face models with different face counts and topological structures.

[0075] S500. Perform smooth interpolation processing on the lip region vertex animation sequence of adjacent frames to output a continuous final lip animation. In S500 of this embodiment, the smooth interpolation processing adopts third-order Bézier curve interpolation, and introduces motion prediction constraints for one forward frame and one backward frame to make the speed and acceleration of lip shape change continuous.

[0076] Specifically, to avoid inter-frame lip movements, the generated vertex animation sequence is smoothed by interpolation. A third-order Bézier curve is used to interpolate the vertex displacements of adjacent frames. At the same time, the motion states of the previous frame and the next frame are introduced as constraints to ensure the continuity of the velocity and acceleration of the lip movement and eliminate the feeling of mechanical jumps.

[0077] The final output supports two formats: one is offline baking output of animation files in mainstream formats such as FBX and BVH, which is suitable for film and television production and recording scenarios; the other is real-time output of vertex displacement driving parameters via UDP protocol, which is suitable for low-latency scenarios such as live streaming and real-time interaction.

[0078] This invention provides a digital human lip-shape generation method, comprising: acquiring an input audio signal; performing frame-by-frame processing on the audio signal; extracting the Mel-frequency cepstral coefficients (MFCC) features of each frame of audio; outputting the initial phoneme label corresponding to each frame of audio through a pre-trained phoneme recognition model; retrieving a pre-constructed standard phoneme-lip shape mapping library, wherein the mapping library stores the basic lip shape parameter vectors corresponding to each phoneme; calculating the speech rate features and timbre features of the current input audio; generating dynamic offset correction coefficients based on the speech rate features and timbre features; using the dynamic offset correction coefficients to perform offset correction on the basic lip shape parameter vectors to obtain corrected lip shape parameters; acquiring the lip region topology information of the target digital human model; automatically calculating the mapping weights from the lip shape parameters to the model vertex displacement; converting the corrected lip shape parameters into a lip region vertex animation sequence that can be directly driven by the digital human model through the mapping weights; performing smooth interpolation processing on the lip region vertex animation sequences of adjacent frames to output a continuous final lip shape animation. This invention controls the lip-sound synchronization error to within 35ms, adapts to mainstream digital human models without manual parameter tuning, and has a single-frame processing time of less than 2ms. It can achieve real-time lip-shape generation on consumer-grade GPUs and is suitable for scenarios such as live streaming, online education, and film and television production.

[0079] Based on the same inventive concept, embodiments of the present invention also provide a digital human lip shape generation system, such as... Figure 4 It includes: an audio processing module, a storage module, a dynamic correction module, a model adaptation module, and an animation output module; among which:

[0080] The audio processing module is used to perform frame-by-frame processing on the input audio signal, extract the MFCC features of each frame, and output the initial phoneme label corresponding to each frame through a pre-trained phoneme recognition model.

[0081] The storage module is used to store a pre-built standard phoneme-lip shape mapping library, which contains multi-dimensional basic lip shape parameter vectors corresponding to each phoneme;

[0082] The dynamic correction module is used to calculate the speech rate and timbre features of the input audio, generate dynamic offset correction coefficients, and perform offset correction on the basic lip shape parameter vector to obtain the corrected lip shape parameters.

[0083] The model adaptation module is used to obtain the lip region topology information of the target digital human model, automatically calculate the mapping weight from lip shape parameters to model vertex displacement, and convert the corrected lip shape parameters into a lip region vertex animation sequence.

[0084] The animation output module is used to perform smooth interpolation on the lip region vertex animation sequence and output the final continuous lip animation.

[0085] In actual deployment, the audio processing module is implemented using an open-source audio library that supports MFCC analysis, the storage module uses a local database to store the phoneme-lip shape mapping library, the correction module and model adaptation module are written in C++ to achieve high-performance computing, and the animation output module supports direct output of lip shape animation files, and can also output driving parameters in real time via UDP protocol to meet the low latency requirements of live streaming.

[0086] To verify the practical effect of this invention, 100 audio clips of different types were selected for comparative testing, covering multiple scenarios such as fast-paced news broadcasts, slow-paced emotional recitations, and daily conversations with slight accents. The test results showed that the average lip-sync error of the method of this invention was 31ms, which is much lower than the 87ms of the traditional rule-based method and the 62ms of the ordinary end-to-end deep learning method. At the same time, adaptation tests were conducted on 12 mainstream digital human models with different topologies. All of them could directly generate usable lip shapes without manual adjustment, and the adaptation success rate reached 100%, which fully verified the advanced nature and practicality of the solution of this invention.

[0087] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0088] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0089] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0090] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0091] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0092] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described method.

[0093] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0094] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0095] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0096] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0097] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0098] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0100] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0102] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for generating digital human lip shapes, characterized in that, include: S100. Acquire the input audio signal, perform frame segmentation on the audio signal, extract the Mel-frequency cepstral coefficients (MFCC) features of each frame of audio, and output the initial phoneme label corresponding to each frame of audio through a pre-trained phoneme recognition model. S200. Retrieve a pre-constructed standard phoneme-lip shape mapping library, which stores the basic lip shape parameter vectors corresponding to each phoneme; S300. Calculate the speech rate features and timbre features of the current input audio, generate dynamic offset correction coefficients based on the speech rate features and timbre features, and use the dynamic offset correction coefficients to offset and correct the basic lip shape parameter vector to obtain the corrected lip shape parameters; S400. Obtain the lip region topology information of the target digital human model, automatically calculate the mapping weight from lip shape parameters to model vertex displacement, and convert the corrected lip shape parameters into a lip region vertex animation sequence that can be directly driven by the digital human model through the mapping weight; S500. Performs smooth interpolation on the lip region vertex animation sequence of adjacent frames to output a continuous final lip animation.

2. The method according to claim 1, characterized in that, In S200, the lip shape parameter vector includes at least normalized values ​​for four dimensions: upper lip height, lower lip height, mouth width, and lip tip protrusion.

3. The method according to claim 1, characterized in that, In S300, a dynamic offset correction coefficient is generated based on the speech rate characteristics and timbre characteristics. The specific method includes: statistically analyzing the phoneme switching frequency of a continuous preset number of audio frames to calculate the current speech rate value V; setting the lip opening amplitude correction coefficient corresponding to the standard speech rate value V0 to 1; when V > V0, for every first preset percentage increase in speech rate, the lip opening amplitude parameter is correspondingly reduced by a second preset percentage; when V ≤ V0, for every third preset percentage decrease in speech rate, the lip opening amplitude parameter is correspondingly increased by a fourth preset percentage.

4. The method according to claim 1, characterized in that, In S300, dynamic offset correction coefficients are generated based on timbre characteristics. The specific method includes: extracting the fundamental frequency mean of the audio as timbre characteristics; When the average fundamental frequency is lower than the first threshold, the lip tip protrusion parameter is increased by the fifth preset ratio; when the average fundamental frequency is higher than the second threshold, the mouth width parameter is increased by the sixth preset ratio.

5. The method according to claim 1, characterized in that, In S400, vertex displacement mapping weights are automatically calculated. Specific methods include: The spatial coordinates and UV information of all vertices in the lip region of the digital human model are identified, and the lip region is divided into 6 independent regions: upper lip left region, upper lip middle region, upper lip right region, lower lip left region, lower lip middle region, and lower lip right region. Based on the Euclidean distance from the vertex in each region to the center point of the mouth, the displacement weight of each vertex corresponding to each dimension of the lip shape parameter is automatically generated to complete the automatic adaptation of different topological models.

6. The method according to claim 1, characterized in that, In S500, the smooth interpolation process uses third-order Bézier curve interpolation, and introduces motion prediction constraints for one forward frame and one backward frame to make the speed and acceleration of the lip shape change continuous.

7. A digital human lip shape generation system, characterized in that, include: The module includes an audio processing module, a storage module, a dynamic correction module, a model adaptation module, and an animation output module; among which: The audio processing module is used to perform frame-by-frame processing on the input audio signal, extract the MFCC features of each frame, and output the initial phoneme label corresponding to each frame through a pre-trained phoneme recognition model. The storage module is used to store a pre-built standard phoneme-lip shape mapping library, which contains multi-dimensional basic lip shape parameter vectors corresponding to each phoneme; The dynamic correction module is used to calculate the speech rate and timbre features of the input audio, generate dynamic offset correction coefficients, and perform offset correction on the basic lip shape parameter vector to obtain the corrected lip shape parameters. The model adaptation module is used to obtain the lip region topology information of the target digital human model, automatically calculate the mapping weight from lip shape parameters to model vertex displacement, and convert the corrected lip shape parameters into a lip region vertex animation sequence. The animation output module is used to perform smooth interpolation on the lip region vertex animation sequence and output the final continuous lip animation.

8. The system according to claim 1, characterized in that, The animation output module supports offline lip-shape animation file output, and also supports real-time output of lip-shape driving parameters via UDP protocol, with end-to-end latency lower than the preset value.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images