Mouth shape generation method and device, equipment and storage medium

Abstract

The present application relates to a kind of mouth shape generation method, device, equipment and storage medium, wherein mouth shape generation method includes: obtaining the original audio of digital person;According to original audio, determine the preset emotional mouth shape graph and preset original mouth shape graph corresponding to original audio;According to preset original mouth shape graph and preset emotional mouth shape graph, determine the target mouth shape graph corresponding to original audio.It can be first according to the original audio of digital person, determine its corresponding preset emotional mouth shape graph and preset original mouth shape graph, then again based on the above preset emotional mouth shape graph and preset original mouth shape graph, obtain the target mouth shape graph corresponding to original audio, so that target mouth shape graph can be with emotion, to better ensure the matching degree between target mouth shape graph and original audio, improve use experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a mouth shape generation method, apparatus, device, and storage medium. Background Technology

[0002] Currently, AI (Artificial Intelligence) synthesized virtual character videos represent a new field of AI technology application. This includes advanced technologies such as speech synthesis, speech recognition, machine translation, facial expression recognition, human motion recognition, and high-definition image processing, enabling customized virtual character solutions. AI-generated virtual characters can be used in many human-interaction scenarios, such as news broadcasting, classroom education, elderly care, and human-computer interaction. In the video industry, different character images can be customized, making interactive video content more vivid and engaging.

[0003] Virtual digital humans are human models created using computer vision or computer graphics technologies to resemble real people or cartoon characters. Users can interact with these digital humans through voice, text, and other means. The virtual digital humans use algorithms to drive changes in facial expressions, lip movements, and body language, combined with voice input, to interact with and respond to users. Currently, digital humans are widely used in government, finance, scenic spots, e-commerce, and other scenarios, such as providing guided tours in scenic spots and offering customer consultation services on e-commerce websites.

[0004] However, existing technologies often suffer from technical problems when creating virtual digital humans, such as mismatch between the virtual digital human's lip movements and voice, and inconsistency between the virtual digital human's lip movements and emotions, making it difficult to meet user needs. Summary of the Invention

[0005] One objective of this application is to provide a lip-shape generation method that can make the lip shape of a digital human more closely match its speech and emotions; a second objective of this application is to provide a lip-shape generation device; a third objective of this application is to provide an electronic device; and a fourth objective of this application is to provide a storage medium.

[0006] To achieve the above objectives, firstly, this application provides a mouth shape generation method, the mouth shape generation method comprising:

[0007] Obtain the original audio of the digital human;

[0008] Based on the original audio, determine the preset emotional lip shape diagram and the preset original lip shape diagram corresponding to the original audio;

[0009] Based on the preset original lip shape diagram and the preset emotional lip shape diagram, determine the target lip shape diagram corresponding to the original audio.

[0010] Further, determining the preset emotional lip-sync diagram and the preset original lip-sync diagram corresponding to the original audio based on the original audio includes:

[0011] Based on the original audio, determine the phoneme data corresponding to the original audio; wherein, the phoneme data includes multiple phonemes with a time sequence;

[0012] Based on the phoneme data, the preset original lip shape diagram and the preset emotional lip shape diagram are determined.

[0013] Further, determining the preset original lip-sync pattern and the preset emotional lip-sync pattern based on the phoneme data includes:

[0014] Obtain a mouth shape database; wherein the mouth shape database includes the correspondence between multiple phonemes and key mouth shape parameters;

[0015] Based on the mouth shape database and the phoneme data, multiple preset mouth shape key parameters corresponding to the original audio are determined; wherein, the multiple preset mouth shape key parameters correspond one-to-one with multiple phonemes in the phoneme data;

[0016] The preset original mouth shape diagram is determined based on multiple preset mouth shape key parameters and the phoneme data.

[0017] Further, determining the preset original lip-sync pattern and the preset emotional lip-sync pattern based on the phoneme data includes:

[0018] When it is determined that the original audio needs to be emotionally processed, multiple preset emotional action parameters corresponding to the original audio are determined based on the lip shape database and the phoneme data; wherein, the lip shape database includes a correspondence between multiple phonemes and emotional action parameters, and the multiple preset emotional action parameters correspond one-to-one with multiple phonemes in the phoneme data;

[0019] The preset emotional lip-sync diagram is determined based on multiple preset emotional action parameters and the phoneme data.

[0020] Furthermore, the mouth shape database is determined in the following way:

[0021] Acquire real-person video recordings; wherein the real-person video recordings include lip-sync information and voice information, and the lip-sync information includes emotional lip-sync information corresponding to the voice information and original lip-sync information;

[0022] Based on the real-person video recording, the correspondence between multiple phonemes and key parameters of lip shape, as well as the correspondence between multiple phonemes and emotional action parameters, is determined to obtain the lip shape database.

[0023] Further, determining the phoneme data corresponding to the original audio based on the original audio includes:

[0024] Determine the initial text corresponding to the original audio;

[0025] Based on the initial text, determine the pinyin document;

[0026] Based on the Pinyin document, determine multiple phonemes corresponding to the original audio;

[0027] The phonemes corresponding to the original audio are aligned with the original audio to determine the phoneme data.

[0028] Further, determining the target lip shape diagram corresponding to the original audio based on the preset original lip shape diagram and the preset emotional lip shape diagram includes:

[0029] Based on an artificial intelligence engine, the original audio, the preset original lip-sync image, and the preset emotional lip-sync image are processed to determine the target lip-sync image.

[0030] To achieve the above objectives, secondly, this application also provides a mouth shape generating apparatus, the mouth shape generating apparatus comprising:

[0031] The acquisition module is used to acquire the raw audio.

[0032] The determining module is used to determine, based on the original audio, a preset emotion lip shape diagram and a preset original lip shape diagram corresponding to the original audio;

[0033] It is also used to determine the target lip shape diagram corresponding to the original audio based on the preset original lip shape diagram and the preset emotion lip shape diagram.

[0034] To achieve the above objectives, in a third aspect, this application also provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a control program stored in the memory to implement the mouth shape generation method as described above.

[0035] To achieve the above objectives, in a fourth aspect, this application also provides a storage medium storing one or more programs that can be executed by one or more processors to implement the mouth shape generation method as described above.

[0036] The beneficial effects of this application are:

[0037] In this invention, a preset emotional lip-sync image and a preset original lip-sync image can be determined based on the original audio of the digital human. Then, based on the preset emotional lip-sync image and the preset original lip-sync image, a target lip-sync image corresponding to the original audio can be obtained, so that the target lip-sync image can carry emotions, thereby better ensuring the matching degree between the target lip-sync image and the original audio and improving the user experience. Attached Figure Description

[0038] Figure 1 This diagram illustrates a flow chart of a mouth shape generation method provided in an embodiment of this application.

[0039] Figure 2 This diagram illustrates the structure of a mouth shape generating device according to an embodiment of this application.

[0040] Figure 3 This illustration shows a structural schematic diagram of an electronic device provided in an embodiment of this application;

[0041] in:

[0042] 10. Obtain module; 20. Confirm module;

[0043] 100. Electronic device; 101. Processor; 102. Memory; 1021. Operating system; 1022. Application program; 103. User interface; 104. Network interface; 105. Bus system. Detailed Implementation

[0044] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0045] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0046] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0047] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0048] This embodiment provides a mouth shape generation method that can be applied to electronic devices. (Reference) Figure 1 As shown, the method may include:

[0049] S110, Obtain the original audio of the digital human;

[0050] S120. Based on the original audio, determine the preset emotional lip shape diagram and the preset original lip shape diagram corresponding to the original audio.

[0051] S130. Based on the preset original lip shape diagram and the preset emotion lip shape diagram, determine the target lip shape diagram corresponding to the original audio.

[0052] In step S110, the original audio refers to the audio during the digital human's speech output process. This original audio can be a segment of audio corresponding to the digital human without lip movements. This original audio needs to be obtained before generating the digital human's lip movements.

[0053] In step S120, after the electronic device obtains the original audio, it can process the original audio to obtain a preset emotional lip shape diagram and a preset original lip shape diagram corresponding to the original audio.

[0054] Among them, the preset emotional lip-sync image refers to the lip-sync image corresponding to the original audio with emotion. The preset original lip-sync image refers to the lip-sync image corresponding to the original audio without emotion.

[0055] It should be noted that the mouth image in this method may include an image of the mouth, an image of the face around the mouth, or even an image of the entire face with the mouth included; there is no limitation on this.

[0056] In step S130, after the electronic device obtains the preset emotional lip shape image and the preset original lip shape image corresponding to the original audio, it can process the original audio, the preset original lip shape image and the preset emotional lip shape image based on the artificial intelligence engine, simulate facial muscle tissue, drive the lip shape of the portrait model with the algorithm, and obtain the target lip shape image corresponding to the original audio.

[0057] The AI ​​engine primarily includes various AI capabilities such as facial recognition, semantics, speech, and dialogue. Combined with a self-developed hyper-realistic, high-precision 3D digital employee-driven solution based on AI and computer graphics, it can simultaneously support synthesized speech and human voice input, generating accurately synchronized lip movements. The aforementioned algorithms drive the lip shape of the facial model and can also include the simulation of facial expressions and movements.

[0058] The artificial intelligence engine can be configured according to actual needs; it can be set before or after the electronic device leaves the factory, without limitation. Furthermore, once configured, the artificial intelligence engine can be modified to better ensure its reliability and meet diverse requirements.

[0059] In this method, the corresponding preset emotion lip shape diagram and preset original lip shape diagram can be determined first based on the original audio of the digital human. Then, based on the above preset emotion lip shape diagram and preset original lip shape diagram, the target lip shape diagram corresponding to the original audio can be obtained so that the target lip shape diagram can carry emotions, thereby better ensuring the matching degree between the target lip shape diagram and the original audio and improving the user experience.

[0060] Among them, the original audio, preset original lip shape image and preset emotional lip shape image are combined by the artificial intelligence engine to simulate the facial muscle tissue. The algorithm drives the lip shape of the portrait model to generate a portrait animation with accurate lip-sound synchronization and rich and realistic expressions. It has the characteristics of high generalization and low latency, avoiding the technical problem of inconsistency between the mouth shape and facial expression of virtual digital humans.

[0061] This embodiment provides a mouth shape generation method, which can be applied to electronic devices. The method may include:

[0062] S210, Obtain the original audio of the digital human;

[0063] S220. Based on the original audio, determine the phoneme data corresponding to the original audio; wherein, the phoneme data includes multiple phonemes with a time sequence;

[0064] S230. Based on the phoneme data, determine the preset original lip shape diagram and the preset emotional lip shape diagram;

[0065] S240. Based on the preset original lip shape diagram and the preset emotion lip shape diagram, determine the target lip shape diagram corresponding to the original audio.

[0066] Step S210 can refer to step S110 in other embodiments, and step S240 can refer to step S130 in other embodiments, and will not be described in detail here.

[0067] In step S220, after the electronic device obtains the original audio, it can process the original audio to obtain multiple phonemes with a time sequence corresponding to the original audio. These multiple phonemes with a time sequence can be denoted as phoneme data.

[0068] In other words, in this step, the original audio is split into phonemes to obtain all the phonemes corresponding to the original audio, and the timestamps corresponding to all the phonemes are obtained, thus obtaining all the phonemes with time sequence.

[0069] For example, a neural network model can be used to process raw audio to obtain its corresponding phoneme data. This neural network model can be trained on multiple training sample pairs. These training sample pairs can include the original audio sample and the corresponding phoneme data samples.

[0070] The specific structure of the neural network model can be set according to actual needs and is not limited thereto. The neural network model can be set before or after the electronic device leaves the factory; there is no limitation in this regard. After the neural network model is set, it can be modified later to better meet different needs.

[0071] It should be noted that, in addition to obtaining the phoneme data corresponding to the original audio through the methods described above, other methods can also be used to obtain the phoneme data corresponding to the original audio, and there are no limitations on this.

[0072] In step S230, after the electronic device obtains the phoneme data corresponding to the original audio, it can determine the sub-original lip shape map and sub-emotional lip shape map corresponding to each phoneme based on the phonemes in the phoneme data. Then, it integrates the sub-original lip shape maps corresponding to all phonemes based on the time series to obtain the preset original lip shape map corresponding to the phoneme data. Similarly, it integrates the sub-emotional lip shape maps corresponding to all phonemes based on the time series to obtain the preset emotional lip shape map corresponding to the phoneme data.

[0073] It should be noted that, in addition to obtaining the preset original lip shape diagram and preset emotional lip shape diagram corresponding to the phoneme data through the above methods, the preset original lip shape diagram and preset emotional lip shape diagram corresponding to the phoneme data can also be obtained through other methods, and there is no limitation on this.

[0074] In this method, multiple phonemes with time sequences corresponding to the original audio are first determined. Then, based on the multiple phonemes with time sequences, preset original lip-sync diagrams and preset emotional lip-sync diagrams corresponding to the original audio are obtained. This makes the matching degree between the preset original lip-sync diagram and the original audio better, and also makes the matching degree between the preset emotional lip-sync diagram and the original audio better. This can further improve the matching degree between the target lip-sync diagram and the original audio, and further improve the user experience.

[0075] This embodiment provides a lip-shape generation method that can be applied to electronic devices. The method involves determining the phoneme data corresponding to the original audio based on the original audio, and may include:

[0076] S310. Determine the initial text corresponding to the original audio;

[0077] S320. Determine the Pinyin document based on the initial text;

[0078] S330. Based on the Pinyin document, determine the multiple phonemes corresponding to the original audio.

[0079] S340. Align the multiple phonemes corresponding to the original audio with the original audio to determine the phoneme data.

[0080] In step S310, after the electronic device obtains the original audio, it can perform text conversion on the original audio to obtain the text corresponding to the original audio, which can be recorded as the initial text.

[0081] In step S320, the pinyin document is a document composed of pinyin during the pronunciation process of the text content in the initial text.

[0082] Once the electronic device obtains the initial text, it can extract the punctuation marks from the initial text and then convert the remaining text content into a pinyin document during its pronunciation process.

[0083] In step S330, after the electronic device obtains the pinyin document, it can separate the initials and finals of each pinyin in the pinyin document, and then use the preset phonetic symbol rules to mark the phonemes corresponding to each letter in the pinyin document, thereby obtaining multiple phonemes corresponding to the original audio.

[0084] It should be noted that, in addition to determining multiple phonemes through the above methods, other methods can also be used to determine multiple phonemes corresponding to the original audio, and there are no limitations on this.

[0085] In step S340, after the electronic device obtains the multiple phonemes corresponding to the original audio, it can process the original audio and its corresponding multiple phonemes to determine the multiple phonemes with time sequence, that is, to determine the phoneme data.

[0086] Among them, the original audio can be processed based on the timestamp in the original audio and its corresponding multiple phonemes to obtain phoneme data.

[0087] It should be noted that, in addition to the methods described above, phoneme data can also be determined by other methods, and there are no limitations on these methods.

[0088] This method involves splitting the initials and finals of the pinyin document corresponding to the original audio to determine multiple phonemes corresponding to the original audio. Then, the original audio and its corresponding multiple phonemes are processed to obtain multiple phonemes with a time sequence. This method can efficiently and accurately obtain the phoneme data corresponding to the original audio, thus providing reliable data support for the subsequent determination of the target lip shape image. This can further improve the reliability of the entire method, better ensure the matching degree between the target lip shape image and the original audio, and improve the user experience.

[0089] This embodiment provides a lip-shape generation method that can be applied to electronic devices. The method involves determining a preset original lip-shape image and a preset emotion lip-shape image based on phoneme data, and may include:

[0090] S410. Obtain the mouth shape database; wherein, the mouth shape database includes the correspondence between multiple phonemes and key mouth shape parameters;

[0091] S420. Based on the lip shape database and phoneme data, determine multiple preset lip shape key parameters corresponding to the original audio; wherein, the multiple preset lip shape key parameters correspond one-to-one with multiple phonemes in the phoneme data.

[0092] S430. Determine the preset original mouth shape diagram based on multiple preset mouth shape key parameters and phoneme data.

[0093] In step S410, the mouth shape database can be obtained based on real-person video recordings, thereby better ensuring the authenticity of the data in the mouth shape database.

[0094] The live video recording may include lip-sync information and voice information. Lip-sync information can be a lip-sync image or the corresponding parameters; there is no limitation in this regard. Voice information can be audio or phonemes or other speech-related information; there is no limitation in this regard either.

[0095] After obtaining the video recording of the real person, the video recording can be processed to obtain the correspondence between multiple phonemes and key parameters of mouth shape, so as to determine the corresponding key parameters of mouth shape based on the phonemes.

[0096] The mouth shape database can be set either before or after the electronic device leaves the factory; there is no limitation on this. Once the mouth shape database is set, it can be modified later to better meet different needs.

[0097] In step S420, after the electronic device obtains the lip shape database and phoneme data, it can select multiple lip shape key parameters corresponding to multiple phonemes from the lip shape database, and define the selected multiple lip shape key parameters as multiple preset lip shape key parameters. The multiple preset lip shape key parameters can correspond one-to-one with the multiple phonemes.

[0098] In step S430, after the electronic device obtains multiple preset mouth shape key parameters corresponding to multiple phonemes in the phoneme data, it can integrate the multiple preset mouth shape key parameters based on the time series of multiple phonemes in the phoneme data to obtain the preset original mouth shape diagram corresponding to the phoneme data.

[0099] This method uses a lip-shape database obtained from real-person video recordings to determine key lip-shape parameters corresponding to multiple phonemes in the phoneme data. This ensures the realism of the determined key lip-shape parameters, better guarantees the matching degree between the pre-generated original lip-shape image and the original audio, and further improves the reliability of the entire method. It also better guarantees the matching degree between the target lip-shape image and the original audio, thus enhancing the user experience.

[0100] This embodiment provides a lip-shape generation method that can be applied to electronic devices. The method involves determining a preset original lip-shape image and a preset emotion lip-shape image based on phoneme data, and may include:

[0101] S510. When it is determined that the original audio needs to be emotionalized, multiple preset emotional action parameters corresponding to the original audio are determined based on the lip shape database and phoneme data. The lip shape database includes the correspondence between multiple phonemes and emotional action parameters, and the multiple preset emotional action parameters correspond one-to-one with multiple phonemes in the phoneme data.

[0102] S520. Determine the preset emotional lip shape diagram based on multiple preset emotional action parameters and phoneme data.

[0103] In step S510, it can be determined whether the original audio needs to be emotionally processed based on the dialogue scenario or other information, which will not be elaborated here.

[0104] It should be noted that in this method, the mouth shape database can include not only the correspondence between multiple phonemes and key mouth shape parameters, but also the correspondence between multiple phonemes and emotional action parameters.

[0105] When building the lip-sync database, real-person video recordings can be obtained first. These recordings can include lip-sync information and voice information. Additionally, the lip-sync information can include emotional lip-sync information corresponding to the voice information and the original lip-sync information.

[0106] This includes the ability to directly capture lip-sync information (including facial expressions) and voice information from real people via facial and voice capture, and then integrate this information into a lip-sync database. Data acquisition from real-person video recordings can be achieved by collecting relevant information from various video sources, including still videos, recordings, and audiovisual materials.

[0107] This involves processing lip-sync and speech information from live-action video recordings to obtain the correspondence between multiple phonemes and emotional action parameters. It also involves processing the raw lip-sync and speech information from live-action video recordings to obtain the correspondence between multiple phonemes and key lip-sync parameters. This allows the creation of the desired lip-sync database based on live-action video recordings.

[0108] It should be noted that, in addition to the methods described above, other methods can also be used to determine the mouth shape database, and there are no limitations on this.

[0109] In this step, when it is determined that emotional processing is needed on the original audio, multiple emotional action parameters corresponding to multiple phonemes can be selected from the lip-shape database and phoneme data, and these selected emotional action parameters are defined as multiple preset emotional action parameters. Each preset emotional action parameter corresponds one-to-one with a specific phoneme.

[0110] In step S520, after the electronic device obtains multiple preset emotional action parameters corresponding to multiple phonemes in the phoneme data, it can integrate the multiple preset emotional action parameters based on the time series of multiple phonemes in the phoneme data to obtain the preset emotional lip shape diagram corresponding to the phoneme data.

[0111] This method uses a lip-shape database obtained from real-person video recordings to determine the emotional action parameters corresponding to multiple phonemes in the phoneme data. This ensures the realism of the determined emotional action parameters, better guarantees the matching degree between the pre-generated emotional lip-shape image and the original audio, and further improves the reliability of the entire method. It also better guarantees the matching degree between the target lip-shape image and the original audio, thus improving the user experience.

[0112] This embodiment provides a mouth shape generation system, which can be applied to electronic devices. This mouth shape generation system can be used to implement the mouth shape generation method described above. The mouth shape generation system may include:

[0113] Video recording module: used to collect mouth shape and voice information from real people, and to extract mouth shape data;

[0114] Data storage module: used to store multiple sets of mouth shape and sound data of the extracted mouth shape information and voice information;

[0115] Raw audio processing module: processes the raw audio of the digital human to obtain phoneme data consisting of multiple phonemes with a time sequence;

[0116] Original mouth shape generation module: used to determine the preset original mouth shape image;

[0117] Emotional lip-syncing generation module: used to determine the preset emotional lip-syncing image;

[0118] Mouth shape output module: Based on the acquired preset original mouth shape image and preset emotion mouth shape image, it integrates them through the AI ​​engine and simulates human facial muscle tissue. The algorithm drives the lip shape of the portrait model to generate the target mouth shape image.

[0119] This embodiment provides a mouth shape generation device, which can be applied to electronic devices. This device can be used to implement the mouth shape generation method described above. For example, refer to... Figure 2 As shown, the device may include an acquisition module 10 and a determination module 20. During the implementation of the above method, the device...

[0120] Module 10 is used to acquire the raw audio.

[0121] The determination module 20 is used to determine the preset emotional lip shape diagram and the preset original lip shape diagram corresponding to the original audio based on the original audio.

[0122] It is also used to determine the target lip shape corresponding to the original audio based on the preset original lip shape diagram and the preset emotion lip shape diagram.

[0123] This embodiment provides a mouth shape generating device that can be applied to electronic devices. (Reference) Figure 2 As shown, in this device, the determining module 20 can be used for:

[0124] Based on the original audio, determine the phoneme data corresponding to the original audio; wherein, the phoneme data includes multiple phonemes with a time sequence;

[0125] Based on the phoneme data, the preset original lip shape diagram and the preset emotional lip shape diagram are determined.

[0126] This embodiment provides a mouth shape generating device that can be applied to electronic devices. (Reference) Figure 2 As shown, in this device,

[0127] The acquisition module 10 can be used to acquire a mouth shape database; wherein, the mouth shape database includes the correspondence between multiple phonemes and key mouth shape parameters;

[0128] The determining module 20 can be used to determine multiple preset lip shape key parameters corresponding to the original audio based on the lip shape database and the phoneme data; wherein, the multiple preset lip shape key parameters correspond one-to-one with multiple phonemes in the phoneme data;

[0129] It can also be used to determine the preset original mouth shape diagram based on multiple preset mouth shape key parameters and the phoneme data.

[0130] This embodiment provides a mouth shape generating device that can be applied to electronic devices. (Reference) Figure 2As shown, in this device, the determining module 20 can be used for:

[0131] When it is determined that the original audio needs to be emotionally processed, multiple preset emotional action parameters corresponding to the original audio are determined based on the lip shape database and the phoneme data; wherein, the lip shape database includes a correspondence between multiple phonemes and emotional action parameters, and the multiple preset emotional action parameters correspond one-to-one with multiple phonemes in the phoneme data;

[0132] The preset emotional lip-sync diagram is determined based on multiple preset emotional action parameters and the phoneme data.

[0133] This embodiment provides a mouth shape generating device that can be applied to electronic devices. (Reference) Figure 2 As shown, in this device, the determining module 20 can be used for:

[0134] Determine the initial text corresponding to the original audio;

[0135] Based on the initial text, determine the pinyin document;

[0136] Based on the Pinyin document, determine multiple phonemes corresponding to the original audio;

[0137] The phonemes corresponding to the original audio are aligned with the original audio to determine the phoneme data.

[0138] This embodiment provides a mouth shape generating device that can be applied to electronic devices. (Reference) Figure 2 As shown, in this device, the determining module 20 can be used for:

[0139] Based on an artificial intelligence engine, the original audio, the preset original lip-sync image, and the preset emotional lip-sync image are processed to determine the target lip-sync image.

[0140] This embodiment provides an electronic device. The electronic device can be from various fields. For example, in the smart home field, it can include large-screen devices, artificial intelligence (AI) speakers, high-fidelity (HiFi) speakers, temperature sensors, humidity sensors, etc. In the smart terminal field, it can include mobile phones, tablets, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. In the smart manufacturing field, it can include robotic arms, cameras, joysticks, monitors, sensors, logistics vehicles, smart shelves, etc. This application embodiment does not impose any limitations on the specific type of electronic device.

[0141] refer to Figure 3 As shown, the electronic device 100 includes at least one processor 101, a memory 102, at least one network interface 104, and other user interfaces 103. The various components in the electronic device 100 are coupled together via a bus system 105. It is understood that the bus system 105 is used to implement communication between these components. In addition to a data bus, the bus system 105 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus system 105.

[0142] The user interface 103 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0143] It is understood that the memory 102 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 102 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0144] In some implementations, memory 102 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 1021 and application program 1022.

[0145] The operating system 1021 includes various system programs, such as a framework layer, a core library layer, and a driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 1022 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of this application embodiment can be included in the application program 1022.

[0146] In this embodiment of the application, the processor 101 executes the methods provided in each method embodiment by calling the program or instructions stored in the memory 102, specifically the program or instructions stored in the application program 1022.

[0147] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 101. The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the processor 101. The processor 101 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 102. Processor 101 reads the information in memory 102 and performs the above method in conjunction with its hardware.

[0148] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0149] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0150] This application also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0151] One or more programs in the storage medium can be executed by one or more processors to implement the mouth shape generation method described above when executed in an electronic device.

[0152] The processor is used to execute the control program of the electronic device stored in the memory to implement the mouth shape generation method described above in the electronic device.

[0153] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0154] It should be noted that the terms "one implementation," "embodiment," "exemplary embodiment," and "some embodiments" used in the specification indicate that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments, whether explicitly described or not, is within the knowledge scope of those skilled in the art.

[0155] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0156] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A method for generating a mouth shape, characterized in that, The mouth shape generation method includes: Obtain the original audio of the digital human; Based on the original audio, determine the preset emotional lip shape diagram and the preset original lip shape diagram corresponding to the original audio; Based on the preset original lip shape diagram and the preset emotional lip shape diagram, determine the target lip shape diagram corresponding to the original audio. The step of determining the preset emotional lip-sync diagram and the preset original lip-sync diagram corresponding to the original audio based on the original audio includes: Based on the original audio, determine the phoneme data corresponding to the original audio; wherein, the phoneme data includes multiple phonemes with a time sequence; Based on the phoneme data, the preset original lip shape diagram and the preset emotional lip shape diagram are determined; The step of determining the preset original lip shape diagram and the preset emotional lip shape diagram based on the phoneme data includes: Obtain a mouth shape database; wherein the mouth shape database includes the correspondence between multiple phonemes and key mouth shape parameters; Based on the mouth shape database and the phoneme data, multiple preset mouth shape key parameters corresponding to the original audio are determined; wherein, multiple phonemes with time sequence are denoted as the phoneme data, and the multiple preset mouth shape key parameters correspond one-to-one with the multiple phonemes in the phoneme data; Based on the time series of multiple phonemes in the phoneme data, multiple preset mouth shape key parameters are integrated to obtain the preset original mouth shape diagram corresponding to the phoneme data; The step of determining the preset original lip shape diagram and the preset emotional lip shape diagram based on the phoneme data includes: Determine whether the original audio needs to be emotionally processed based on the dialogue scenario; When it is determined that the original audio needs to be emotionally processed, multiple preset emotional action parameters corresponding to the original audio are determined based on the lip shape database and the phoneme data; wherein, the lip shape database includes a correspondence between multiple phonemes and emotional action parameters, and the multiple preset emotional action parameters correspond one-to-one with multiple phonemes in the phoneme data; Based on the time series of multiple phonemes in the phoneme data, multiple preset emotional action parameters are integrated to obtain the preset emotional lip shape diagram corresponding to the phoneme data. The mouth shape database is determined in the following way: Acquire real-person video recordings; wherein the real-person video recordings include lip-sync information and voice information, and the lip-sync information includes emotional lip-sync information corresponding to the voice information and original lip-sync information; Based on the real-person video recording, the correspondence between multiple phonemes and key parameters of mouth shape, as well as the correspondence between multiple phonemes and emotional action parameters, is determined to obtain the mouth shape database. The correspondence between the multiple phonemes in the mouth shape database and the key parameters of mouth shape, as well as the correspondence between the multiple phonemes and the parameters of emotional actions, are determined based on mouth shape information and voice information from the same real-person video recording. The step of determining the target lip shape diagram corresponding to the original audio based on the preset original lip shape diagram and the preset emotional lip shape diagram includes: Based on an artificial intelligence engine, the original audio, the preset original lip-sync image, and the preset emotional lip-sync image are processed to determine the target lip-sync image.

2. The mouth shape generation method according to claim 1, characterized in that, The step of determining the phoneme data corresponding to the original audio based on the original audio includes: Determine the initial text corresponding to the original audio; Based on the initial text, determine the pinyin document; Based on the Pinyin document, determine multiple phonemes corresponding to the original audio; The phonemes corresponding to the original audio are aligned with the original audio to determine the phoneme data.

3. A mouth-shaped generating device, characterized in that, The mouth shape generating device includes: The acquisition module is used to acquire the raw audio. The determining module is used to determine, based on the original audio, a preset emotion lip shape diagram and a preset original lip shape diagram corresponding to the original audio; It is also used to determine the target lip shape diagram corresponding to the original audio based on the preset original lip shape diagram and the preset emotion lip shape diagram; The step of determining the preset emotional lip-sync diagram and the preset original lip-sync diagram corresponding to the original audio based on the original audio includes: Based on the original audio, determine the phoneme data corresponding to the original audio; wherein, the phoneme data includes multiple phonemes with a time sequence; Based on the phoneme data, the preset original lip shape diagram and the preset emotional lip shape diagram are determined; The step of determining the preset original lip shape diagram and the preset emotional lip shape diagram based on the phoneme data includes: Obtain a mouth shape database; wherein the mouth shape database includes the correspondence between multiple phonemes and key mouth shape parameters; Based on the mouth shape database and the phoneme data, multiple preset mouth shape key parameters corresponding to the original audio are determined; wherein, multiple phonemes with time sequence are denoted as the phoneme data, and the multiple preset mouth shape key parameters correspond one-to-one with the multiple phonemes in the phoneme data; Based on the time series of multiple phonemes in the phoneme data, multiple preset mouth shape key parameters are integrated to obtain the preset original mouth shape diagram corresponding to the phoneme data; The step of determining the preset original lip shape diagram and the preset emotional lip shape diagram based on the phoneme data includes: Determine whether the original audio needs to be emotionally processed based on the dialogue scenario; When it is determined that the original audio needs to be emotionally processed, multiple preset emotional action parameters corresponding to the original audio are determined based on the lip shape database and the phoneme data; wherein, the lip shape database includes a correspondence between multiple phonemes and emotional action parameters, and the multiple preset emotional action parameters correspond one-to-one with multiple phonemes in the phoneme data; Based on the time series of multiple phonemes in the phoneme data, multiple preset emotional action parameters are integrated to obtain the preset emotional lip shape diagram corresponding to the phoneme data. The mouth shape database is determined in the following way: Acquire real-person video recordings; wherein the real-person video recordings include lip-sync information and voice information, and the lip-sync information includes emotional lip-sync information corresponding to the voice information and original lip-sync information; Based on the real-person video recording, the correspondence between multiple phonemes and key parameters of mouth shape, as well as the correspondence between multiple phonemes and emotional action parameters, is determined to obtain the mouth shape database. The correspondence between the multiple phonemes in the mouth shape database and the key parameters of mouth shape, as well as the correspondence between the multiple phonemes and the parameters of emotional actions, are determined based on mouth shape information and voice information from the same real-person video recording. The step of determining the target lip shape diagram corresponding to the original audio based on the preset original lip shape diagram and the preset emotional lip shape diagram includes: Based on an artificial intelligence engine, the original audio, the preset original lip-sync image, and the preset emotional lip-sync image are processed to determine the target lip-sync image.

4. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a control program stored in the memory to implement the mouth shape generation method of claim 1 or 2.

5. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the mouth shape generation method according to claim 1 or 2.

Images