3D digital human lip shape driving method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a 3D digital human lip shape driving method and device, electronic equipment and storage medium, which relates to the technical field of computer vision. The method comprises: obtaining input text information; converting the text information into a phoneme sequence, audio data and timestamp information based on text-to-speech (TTS) technology; deleting corresponding silent phonemes in the phoneme sequence according to the timestamp information; performing a preset multiple sampling on the phoneme sequence after the deletion processing to obtain a bs animation coefficient sequence; and generating a 3D digital human lip shape animation according to the bs animation coefficient sequence, the audio data, a preset phoneme lip shape mapping table and a preset optimization of special phonemes. The present disclosure improves the robustness and fluency of 3D digital human lip shape driving.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a 3D digital human lip-syncing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the advent of the artificial intelligence era, the application of AI-based human-computer interaction methods is becoming increasingly common. Visual voice technology is one such next-generation human-computer interaction method.

[0003] Visualized speech technology refers to an interactive method that combines technologies such as deep learning-based speech recognition, speech synthesis, and virtual objects. It tightly integrates ordinary users with computers through natural language recognition, understanding, and synthesis. A common approach is to achieve human-computer interaction through 3D (three-dimensional) virtual humans. These 3D virtual humans can simulate the movements and expressions of real people and can engage in voice communication with them.

[0004] 3D digital human lip-syncing technology is one of the key technologies required for the deployment of 3D digital human products. Currently, most similar products adopt deep learning-based solutions. On the one hand, this type of solution heavily relies on GPU (graphics processing unit) computing resources. On the other hand, the cost of acquiring such training data is high and difficult to obtain. However, the model's inference performance is highly dependent on the comprehensiveness of the training data; if it encounters data not covered during training, the robustness of the lip-syncing effect will be significantly challenged.

[0005] Therefore, improving the robustness of lip-syncing in 3D digital humans has become a pressing technical problem that needs to be solved.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] This disclosure provides a 3D digital human lip-syncing method, apparatus, electronic device, and storage medium, which at least to some extent overcomes the problem of low robustness of existing 3D digital human lip-syncing methods in related technologies.

[0008] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0009] According to one aspect of this disclosure, a 3D digital human lip-syncing method is provided, comprising: acquiring input text information; converting the text information into a phoneme sequence, audio data, and timestamp information based on TTS (Text To Speech) technology, wherein the timestamp information includes the timestamps of each element in the phoneme sequence; deleting corresponding silent phonemes in the phoneme sequence according to the timestamp information, sampling the deleted phoneme sequence at a preset multiple to obtain a bs animation coefficient sequence; and rendering and generating a 3D digital human lip-syncing animation based on the bs animation coefficient sequence, the audio data, a preset phoneme lip-syncing mapping table, and preset optimization of special phonemes, wherein the preset phoneme lip-syncing mapping table includes mapping relationships between various phonemes and the lip shapes of the 3D digital human, and the special phonemes include bilabial consonants, labiodental consonants, and dental fricatives.

[0010] In one embodiment of this disclosure, the preset phoneme lip shape mapping table is constructed by the following steps: obtaining all single phonemes; sequentially mapping each single phoneme to a complete lip shape animation to obtain the preset phoneme lip shape mapping table.

[0011] In one embodiment of this disclosure, before rendering and generating the lip animation of a 3D digital human based on the bs animation coefficient sequence, the audio data, the preset phoneme lip shape mapping table, and the preset optimization of special phonemes, the method further includes: performing cubic spline interpolation sampling on the bs animation coefficient sequence.

[0012] In one embodiment of this disclosure, the bilabial consonant includes at least one of b, p, and m; the labiodental consonant includes at least one of f and v; and the dental fricative includes at least one of z, c, s, j, q, x, zh, ch, and sh.

[0013] In one embodiment of this disclosure, the preset optimization of special phonemes is configured through the following steps: setting the upper lip and lower lip of the 3D digital human to contact each other before the 3D digital human pronounces bilabial sounds; setting the lower lip of the 3D digital human to touch the upper teeth before the 3D digital human pronounces labiodental sounds; setting the distance between the upper and lower teeth of the 3D digital human to meet a preset distance when the 3D digital human pronounces dental fricatives.

[0014] In one embodiment of this disclosure, before rendering and generating the lip-shape animation of a 3D digital human based on the bs animation coefficient sequence, the audio data, the preset phoneme lip-shape mapping table, and the preset optimization of special phonemes, the method further includes: attenuating the bs animation coefficient sequence to obtain an attenuated bs animation coefficient sequence.

[0015] In one embodiment of this disclosure, before rendering and generating the lip animation of a 3D digital human based on the bs animation coefficient sequence, the audio data, the preset phoneme lip shape mapping table, and the preset optimization of special phonemes, the method further includes: smoothing the bs animation coefficient sequence.

[0016] In one embodiment of this disclosure, before rendering and generating the lip-shape animation of a 3D digital human based on the bs animation coefficient sequence, the audio data, the preset phoneme lip-shape mapping table, and the preset optimization of special phonemes, the method further includes: adding a silent phoneme at a preset position in the bs animation coefficient sequence, and smoothing the bs animation coefficient sequence after adding the silent phoneme, wherein the preset position is before the first element and after the last element of the bs animation coefficient sequence.

[0017] According to another aspect of this disclosure, a 3D digital human lip-shape driving device is provided, comprising: a text information acquisition module for acquiring input text information; a TTS module for converting the text information into a phoneme sequence, audio data, and timestamp information based on TTS technology, wherein the timestamp information includes the timestamp of each element in the phoneme sequence; a phoneme preprocessing module for deleting corresponding silent phonemes in the phoneme sequence according to the timestamp information, and sampling the deleted phoneme sequence at a preset multiple to obtain a bs animation coefficient sequence; and a lip-shape postprocessing module for rendering and generating a 3D digital human lip-shape animation based on the bs animation coefficient sequence, the audio data, a preset phoneme lip-shape mapping table, and preset optimization of special phonemes, wherein the preset phoneme lip-shape mapping table includes mapping relationships between various phonemes and the lip shape of the 3D digital human, and the special phonemes include bilabial consonants, labiodental consonants, and dental fricatives.

[0018] In one embodiment of this disclosure, the apparatus further includes a mapping table construction module, which is used to acquire all monophones and sequentially map each monophone to a complete lip-shape animation to obtain the preset phoneme lip-shape mapping table.

[0019] In one embodiment of this disclosure, the above-described apparatus further includes a difference sampling module, which is used to perform cubic spline difference sampling on the bs animation coefficient sequence.

[0020] In one embodiment of this disclosure, the lip-shape post-processing module is further configured to: set the upper lip of the 3D digital human to contact the lower lip before the 3D digital human pronounces bilabial sounds; set the lower lip of the 3D digital human to touch the upper teeth before the 3D digital human pronounces labiodental sounds; and set the distance between the upper and lower teeth of the 3D digital human to satisfy a preset distance when the 3D digital human pronounces dental fricatives.

[0021] In one embodiment of this disclosure, the lip-shape post-processing module is further configured to attenuate the bs animation coefficient sequence to obtain an attenuated bs animation coefficient sequence.

[0022] In one embodiment of this disclosure, the lip-shaping post-processing module is further configured to smooth the BS animation coefficient sequence.

[0023] In one embodiment of this disclosure, the lip-shaping post-processing module is further configured to add a silence phoneme to a preset position in the BS animation coefficient sequence and to smooth the BS animation coefficient sequence after adding the silence phoneme, wherein the preset position is before the first element and after the last element of the BS animation coefficient sequence.

[0024] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described 3D digital human lip-syncing method by executing the executable instructions.

[0025] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described 3D digital human lip-syncing method.

[0026] This disclosure provides a 3D digital human lip-syncing method, apparatus, electronic device, and storage medium. The 3D digital human lip-syncing method includes: acquiring input text information; converting the text information into a phoneme sequence, audio data, and timestamp information based on text-to-speech (TTS) technology, wherein the timestamp information includes the timestamps of each element in the phoneme sequence; deleting corresponding silent phonemes from the phoneme sequence according to the timestamp information, sampling the deleted phoneme sequence at a preset multiple to obtain a bs animation coefficient sequence; and rendering a 3D digital human lip-syncing animation based on the bs animation coefficient sequence, the audio data, a preset phoneme lip-syncing mapping table, and preset optimizations for special phonemes. The preset phoneme lip-syncing mapping table includes mapping relationships between various phonemes and 3D digital human lip shapes, and the special phonemes include bilabial consonants, labiodental consonants, and dental fricatives. This disclosure does not rely on a GPU, consumes less computing power, and improves the robustness and smoothness of the 3D digital human lip-syncing driver.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0029] Figure 1 This diagram illustrates a communication system architecture according to an embodiment of the present disclosure;

[0030] Figure 2 This diagram illustrates a flowchart of a 3D digital human lip-shape driving method according to an embodiment of the present disclosure.

[0031] Figure 3 This diagram illustrates another 3D digital human lip-driving method according to an embodiment of the present disclosure.

[0032] Figure 4 This diagram illustrates another 3D digital human lip-driving method according to an embodiment of the present disclosure.

[0033] Figure 5 This diagram illustrates a 3D digital human lip-shaping method according to an embodiment of the present disclosure.

[0034] Figure 6 A schematic diagram of a 3D digital human lip-shaped actuator is shown in an embodiment of this disclosure; and

[0035] Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0036] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0037] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0038] As mentioned in the background section, voice / text-driven 3D digital human lip-syncing technology is one of the most critical technologies in customer service-oriented 3D digital humans. With customer service-oriented digital humans primarily used in mobile applications, the computational demands on lip-syncing technology are high. Simultaneously, acquiring 3D real-person lip-syncing-voice / text datasets for model training is costly and difficult. However, model training relies heavily on a large amount of training data; a lack of data significantly reduces the robustness of the lip-syncing effect.

[0039] Based on this, the present disclosure provides a 3D digital human lip-sync driving method, device, electronic device and storage medium. The 3D digital human lip-sync driving is achieved by preset phoneme lip-sync mapping table and preset optimization of special phonemes, which reduces the problems of high computing power requirements, poor driving effect and lack of robustness of existing 3D digital human lip-sync technology.

[0040] Figure 1 A schematic diagram of an exemplary system architecture that can be applied to the 3D digital human lip-syncing method or 3D digital human lip-syncing device in the embodiments of this disclosure is shown.

[0041] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105.

[0042] Network 104 is a medium used to provide a communication link between terminal devices 101, 102, 103 and server 105, and can be a wired network or a wireless network.

[0043] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0044] Terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0045] Optionally, the client applications installed on different terminal devices 101, 102, and 103 may be the same, or clients of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also differ; for example, the application client may be a mobile client, a PC client, etc.

[0046] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0047] Optionally, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0048] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.

[0049] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.

[0050] First, this disclosure provides a 3D digital human lip-shape driving method, which can be applied to the above-mentioned... Figure 1 An open system can be executed by any electronic device with computing power.

[0051] Figure 2 This diagram illustrates a flowchart of a 3D digital human lip-syncing method according to an embodiment of the present disclosure, as follows: Figure 2 As shown, the 3D digital human lip-shape driving method provided in this embodiment includes the following steps:

[0052] S202, Obtain the input text information;

[0053] It should be noted that the text information can be text entered by the user, or text information entered by an electronic device with text input function after the text is collected. For example, the text information can be "Hello".

[0054] S204, based on text-to-speech (TTS) technology, converts text information into phoneme sequences, audio data, and timestamp information, wherein the timestamp information contains the timestamps of each element in the phoneme sequence;

[0055] It should be noted that based on the text-to-speech (TTS) technology, text information is converted into a phoneme sequence, audio data, and timestamp information. For example, when the text information is "Hello", the text information is converted into a phoneme sequence, audio data, and timestamp information. The phoneme sequence is [n, iii3, h, ao4], the audio data is the voice form of "Hello", and the timestamp information is [20, 20, 10, 35]. Here, the elements in the timestamp information correspond one-to-one with the elements in the phoneme sequence.

[0056] S206, according to the timestamp information, delete the corresponding silent phonemes in the phoneme sequence, and perform a preset multiple sampling on the phoneme sequence after the deletion process to obtain a bs animation coefficient sequence;

[0057] It should be noted that in order to prevent the 3D digital human lip shape from closing the mouth during a short pause, it is possible to first delete the silent phonemes in the phoneme sequence whose duration does not meet the preset duration according to the duration of each phoneme in the timestamp information. The preset duration can be any duration, such as 0.1 second, 0.05 second, or 0.03 second, etc. The preset duration can be freely set, and no specific limit is imposed on the size of the preset duration here. Then, the phoneme sequence can be sampled at a preset multiple to ensure that each phoneme lasts at least four frames to completely cover the lip shape change process of each phoneme. Here, the preset multiple can be any multiple, such as n times, where n is a positive number.

[0058] S208, render and generate the lip animation of the 3D digital human according to the bs animation coefficient sequence, audio data, preset phoneme lip shape mapping table, and preset optimization of special phonemes. The preset phoneme lip shape mapping table contains the mapping relationships between various phonemes and the lip shapes of the 3D digital human. The special phonemes include bilabial sounds, labiodental sounds, and fricative sounds.

[0059] The 3D digital human lip shape driving method provided by the embodiments of the present disclosure obtains the input text information; based on the text-to-speech (TTS) technology, converts the text information into a phoneme sequence, audio data, and timestamp information, where the timestamp information contains the timestamps of each element in the phoneme sequence; according to the timestamp information, deletes the corresponding silent phonemes in the phoneme sequence, and performs a preset multiple sampling on the phoneme sequence after the deletion process to obtain a bs animation coefficient sequence; renders and generates the lip animation of the 3D digital human according to the bs animation coefficient sequence, audio data, preset phoneme lip shape mapping table, and preset optimization of special phonemes. The present disclosure does not rely on the GPU, consumes less computing power, and improves the robustness and smoothness of the 3D digital human lip shape driving.

[0060] In one embodiment of this disclosure, before rendering and generating the lip-shape animation of a 3D digital human based on the BS animation coefficient sequence, audio data, a preset phoneme lip-shape mapping table, and preset optimizations for specific phonemes, the method further includes: performing cubic spline interpolation sampling on the BS animation coefficient sequence. This disclosure ensures that the frame rate of the phoneme sequence and the BS animation coefficient sequence is consistent by performing cubic spline interpolation sampling on the BS animation coefficient sequence, thereby ensuring that the sampled BS animation coefficient sequence maintains the lip-shape change process of each phoneme to the maximum extent, while also maintaining the smoothness of the BS animation coefficient sequence.

[0061] In one embodiment of this disclosure, before rendering and generating the lip animation of the 3D digital human based on the BS animation coefficient sequence, audio data, a preset phoneme lip shape mapping table, and preset optimization of special phonemes, the method further includes: attenuating the BS animation coefficient sequence to obtain an attenuated BS animation coefficient sequence. This disclosure attenuates the BS animation coefficient sequence to limit the amplitude of certain lip movements, making the lip movements of the 3D digital human more aesthetically pleasing. The degree of attenuation of the BS animation coefficient sequence is strongly correlated with the digital assets and can be freely adjusted according to actual conditions; this disclosure does not impose specific limitations on this.

[0062] In one embodiment of this disclosure, before rendering and generating the lip-shape animation of a 3D digital human based on the BS animation coefficient sequence, audio data, a preset phoneme lip-shape mapping table, and preset optimizations of special phonemes, the method further includes: smoothing the BS animation coefficient sequence. Smoothing the BS animation coefficient sequence may include: smoothing between individual phonemes in the BS animation coefficient sequence; and performing global smoothing on the BS animation coefficient sequence. Specifically, this disclosure eliminates the problem of lip jitter between different phonemes by smoothing between individual phonemes in the BS animation coefficient sequence. This disclosure also ensures smoothness within the same phoneme by performing global smoothing on the BS animation coefficient sequence.

[0063] In one embodiment of this disclosure, before rendering and generating the lip-sync animation of the 3D digital human based on the BS animation coefficient sequence, audio data, a preset phoneme lip-sync map, and preset optimization of special phonemes, the method further includes: adding silence phonemes at preset positions in the BS animation coefficient sequence, and smoothing the BS animation coefficient sequence after adding silence phonemes, wherein the preset positions are before the first element and after the last element of the BS animation coefficient sequence. This disclosure ensures the continuity of the 3D digital human's speech at the beginning and end by adding silence phonemes at preset positions in the BS animation coefficient sequence and performing smoothing processing.

[0064] In one embodiment of this disclosure, bilabial consonants include at least one of b, p, and m; labiodental consonants include at least one of f and v; and dental fricatives include at least one of z, c, s, j, q, x, zh, ch, and sh.

[0065] In one embodiment of this disclosure, participants Figure 3 The flowchart of another 3D digital human lip-syncing method shown can be configured with preset optimizations for special phonemes through the following steps:

[0066] S302, set the 3D digital human to make contact between the upper and lower lips before the bilabial sound is pronounced.

[0067] S304, set the 3D digital human's lower lip to touch the upper teeth before the labiodental sound is pronounced.

[0068] S306, when the 3D digital human pronounces dental fricatives, the distance between the upper and lower teeth of the 3D digital human meets the preset distance.

[0069] It should be noted that the preset distance can be any distance, such as k centimeters or millimeters, where k is any positive number.

[0070] This disclosure improves the fluency of lip movements and enhances the visual appeal of 3D digital humans by configuring preset optimizations for special phonemes.

[0071] In one embodiment of this disclosure, participants Figure 4 The flowchart of another 3D digital human lip-shape driving method shown below constructs a preset phoneme lip-shape mapping table through the following steps:

[0072] S402, retrieve all monophones.

[0073] S404, sequentially maps each single phoneme to a complete lip-shape animation to obtain a preset phoneme lip-shape mapping table.

[0074] This disclosure can construct a mapping relationship between all monophones and 3D digital human lip shapes by using facial motion capture tools, wherein a complete pronunciation process is recorded for each monophone.

[0075] In one embodiment of this disclosure, the method may further include: constructing a blinking pattern for a real speaking process, and outputting the blinking pattern to a sequence of BS animation coefficients, so that the 3D digital human blinks naturally when speaking.

[0076] In one embodiment of this disclosure, see Figure 5The diagram illustrates a 3D digital human lip-syncing method. When the input text is "Hello", the TTS module executes step S204, converting the text into audio data, a phoneme sequence [n, iii3, h, ao4], and a timestamp information [20, 20, 10, 35]. The TTS module sends the audio data, phoneme sequence, and timestamp information to the phoneme preprocessing module. The phoneme preprocessing module executes step S206, processing the phoneme sequence to obtain the bs animation coefficient sequence. The phoneme preprocessing module sends the audio data and the bs animation coefficient sequence to the lip-syncing (bs) postprocessing module. The lip-syncing (bs) postprocessing module executes step S208, rendering and generating the lip-syncing animation of the 3D digital human based on the bs animation coefficient sequence, audio data, a preset phoneme lip-syncing mapping table, and preset optimizations for special phonemes. The rendered lip-syncing animation of the 3D digital human is then sent to the rendering engine.

[0077] Based on the same inventive concept, this disclosure also provides a 3D digital human lip-shaped driving device, as shown in the following embodiment. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiment, the implementation of this device embodiment can refer to the implementation of the above-described method embodiment, and repeated details will not be elaborated further.

[0078] Figure 6 This diagram illustrates a 3D digital human lip-shaped actuator according to an embodiment of the present disclosure. Figure 6 As shown, the device includes:

[0079] The text information acquisition module 610 is used to acquire the input text information;

[0080] The TTS module 620 is used to convert text information into phoneme sequences, audio data and timestamp information based on TTS technology. The timestamp information includes the timestamps of each element in the phoneme sequence.

[0081] The phoneme preprocessing module 630 is used to delete the corresponding silent phonemes in the phoneme sequence according to the timestamp information, and to sample the deleted phoneme sequence by a preset multiple to obtain the bs animation coefficient sequence.

[0082] The lip-shape post-processing module 640 is used to render and generate lip-shape animation of 3D digital human based on the bs animation coefficient sequence, audio data, preset phoneme lip-shape mapping table and preset optimization of special phonemes. The preset phoneme lip-shape mapping table contains the mapping relationship between various phonemes and 3D digital human lip shapes. The special phonemes include bilabial consonants, labiodental consonants and dental fricatives.

[0083] In one embodiment of this disclosure, the apparatus further includes a mapping table construction module, which is used to acquire all monophones and sequentially map each monophone to a complete lip-shape animation to obtain a preset phoneme lip-shape mapping table.

[0084] In one embodiment of this disclosure, the above-described apparatus further includes a difference sampling module, which is used to perform cubic spline difference sampling on the bs animation coefficient sequence.

[0085] In one embodiment of this disclosure, the lip-shaped post-processing module 640 is further configured to: set the upper lip of the 3D digital human to contact the lower lip before the 3D digital human pronounces bilabial sounds; set the lower lip of the 3D digital human to touch the upper teeth before the 3D digital human pronounces labiodental sounds; and set the distance between the upper and lower teeth of the 3D digital human to meet a preset distance when the 3D digital human pronounces dental fricatives.

[0086] In one embodiment of this disclosure, the lip-shape post-processing module 640 is further configured to attenuate the BS animation coefficient sequence to obtain an attenuated BS animation coefficient sequence.

[0087] In one embodiment of this disclosure, the lip-shaping post-processing module is further configured to smooth the BS animation coefficient sequence.

[0088] In one embodiment of this disclosure, the lip-shaping post-processing module 640 is further configured to add a silence phoneme at a preset position in the BS animation coefficient sequence and to smooth the BS animation coefficient sequence after adding the silence phoneme, wherein the preset position is before the first element and after the last element of the BS animation coefficient sequence.

[0089] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0090] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0091] like Figure 7As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including storage unit 720 and processing unit 710).

[0092] The storage unit stores program code, which can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 710 can perform the following steps of the above method embodiments: acquiring input text information; converting the text information into a phoneme sequence, audio data, and timestamp information based on text-to-speech (TTS) technology, wherein the timestamp information includes the timestamps of each element in the phoneme sequence; deleting corresponding silent phonemes in the phoneme sequence according to the timestamp information, sampling the deleted phoneme sequence at a preset multiple to obtain a bs animation coefficient sequence; and rendering and generating a 3D digital human's lip animation based on the bs animation coefficient sequence, audio data, a preset phoneme lip shape mapping table, and preset optimization of special phonemes, wherein the preset phoneme lip shape mapping table contains mapping relationships between various phonemes and the lip shapes of the 3D digital human, and the special phonemes include bilabial consonants, labiodental consonants, and dental fricatives.

[0093] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include a read-only memory (ROM) 7203.

[0094] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0095] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0096] Electronic device 700 can also communicate with one or more external devices 740 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0097] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0098] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0099] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0100] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0101] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0102] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0103] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0104] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0105] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0106] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A 3D digital human lip-shape driving method, characterized in that, include: Obtain the input text information; Based on text-to-speech (TTS) technology, the text information is converted into phoneme sequences, audio data, and timestamp information, wherein the timestamp information includes the timestamps of each element in the phoneme sequence. Based on the timestamp information, the corresponding silent phonemes in the phoneme sequence are deleted, and the deleted phoneme sequence is sampled at a preset multiple to obtain the lip-sync animation coefficient sequence. Based on the lip animation coefficient sequence, the audio data, the preset phoneme lip shape mapping table, and the preset optimization of special phonemes, a 3D digital human lip animation is rendered and generated. The preset phoneme lip shape mapping table contains the mapping relationship between various phonemes and the 3D digital human lip shape. The special phonemes include bilabial consonants, labiodental consonants, and dental fricatives.

2. The 3D digital human lip-shape driving method according to claim 1, characterized in that, The preset phoneme lip shape mapping table is constructed through the following steps: Get all monophones; Each single phoneme is mapped sequentially to a complete lip-shape animation to obtain the preset phoneme lip-shape mapping table.

3. The 3D digital human lip-shape driving method according to claim 1, characterized in that, Before rendering and generating the lip animation of a 3D digital human based on the lip animation coefficient sequence, the audio data, the preset phoneme lip mapping table, and preset optimizations for special phonemes, the method further includes: The lip animation coefficient sequence is subjected to cubic spline interpolation sampling.

4. The 3D digital human lip-shape driving method according to claim 1, characterized in that, The bilabial consonant includes at least one of b, p, and m; The labiodental consonants include at least one of f and v; The dental fricatives include at least one of z, c, s, j, q, x, zh, ch, and sh.

5. The 3D digital human lip-shape driving method according to claim 4, characterized in that, Configure preset optimizations for special phonemes using the following steps: Before the 3D digital human pronounces a bilabial sound, its upper lip and lower lip are in contact. Before the 3D digital human pronounces a labiodental sound, the lower lip of the 3D digital human touches the upper teeth. When the 3D digital human pronounces dental fricatives, the distance between the upper and lower teeth of the 3D digital human meets a preset distance.

6. The 3D digital human lip-shape driving method according to claim 1, characterized in that, Before rendering and generating the lip animation of a 3D digital human based on the lip animation coefficient sequence, the audio data, the preset phoneme lip mapping table, and preset optimizations for special phonemes, the method further includes: The lip animation coefficient sequence is attenuated to obtain the attenuated lip animation coefficient sequence.

7. The 3D digital human lip-shape driving method according to claim 1, characterized in that, Before rendering and generating the lip animation of a 3D digital human based on the lip animation coefficient sequence, the audio data, the preset phoneme lip mapping table, and preset optimizations for special phonemes, the method further includes: The lip-shape animation coefficient sequence is smoothed.

8. The 3D digital human lip-shape driving method according to claim 1, characterized in that, Before rendering and generating the lip animation of a 3D digital human based on the lip animation coefficient sequence, the audio data, the preset phoneme lip mapping table, and preset optimizations for special phonemes, the method further includes: A silence phoneme is added to a preset position in the lip-shape animation coefficient sequence, and the lip-shape animation coefficient sequence after adding the silence phoneme is smoothed. The preset position is before the first element and after the last element of the lip-shape animation coefficient sequence.

9. A 3D digital human lip-shaped driving device, characterized in that, include: The text information acquisition module is used to acquire the input text information; The TTS module is used to convert the text information into phoneme sequences, audio data and timestamp information based on TTS technology, wherein the timestamp information includes the timestamps of each element in the phoneme sequence; The phoneme preprocessing module is used to delete the corresponding silent phonemes in the phoneme sequence according to the timestamp information, and to sample the deleted phoneme sequence by a preset multiple to obtain the lip animation coefficient sequence. The lip shape post-processing module is used to render and generate lip shape animation of 3D digital human based on the lip shape animation coefficient sequence, the audio data, the preset phoneme lip shape mapping table, and the preset optimization of special phonemes. The preset phoneme lip shape mapping table contains the mapping relationship between various phonemes and 3D digital human lip shapes. The special phonemes include bilabial consonants, labiodental consonants, and dental fricatives.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the 3D digital human lip-driving method of any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the 3D digital human lip-syncing method according to any one of claims 1 to 8.

Images