Speech animation synthesis method and device, electronic equipment and storage medium

Abstract

This application provides a method, apparatus, electronic device, and storage medium for synthesizing speech animation, relating to the field of speech technology. The method includes: displaying an image acquisition control; in response to a trigger operation of the image acquisition control, acquiring a facial image of a target user to obtain a target facial image; displaying follow-up reading information and a follow-up reading control; in response to a trigger operation of the follow-up reading control, acquiring audio information input by the target user corresponding to the follow-up reading information; acquiring and displaying a speech animation including an image of the target user's lip shape changes, wherein the lip shape changes of the target user are synchronized with the content of the audio information, and the image of the target user's lip shape changes is obtained based on the lip shape in the target facial image and the audio information. This application embodiment can help users practice pronunciation and lip shape more accurately based on speech animation, improving the interest and efficiency of language learning.

Description

Technical Field

[0001] This application relates to the field of speech technology, and more specifically, to a method, apparatus, electronic device, and storage medium for synthesizing speech animation. Background Technology

[0002] Listening and reading are very important in language learning. Usually, when learning a language, you need to read aloud the sounds you hear in order to learn various sounds.

[0003] In many cases, learners cannot determine whether their pronunciation is standard on their own. Therefore, most language learning software also displays standard lip shape patterns for pronunciation to help learners adjust their lip shape to correct their pronunciation by referring to the standard lip shape patterns.

[0004] However, the standard lip patterns provided by existing technologies are either hand-drawn or based on models' lip patterns. Learners cannot accurately imitate the standard lip patterns, which affects the accuracy of pronunciation and reduces their enthusiasm for language learning. Summary of the Invention

[0005] The present invention provides a method, apparatus, electronic device, and storage medium for synthesizing voice animation that overcomes or at least partially solves the above problems.

[0006] Firstly, a method for synthesizing speech animation is provided, the method comprising:

[0007] Display the image acquisition control, respond to the trigger operation of the image acquisition control, acquire the facial image of the target user, and obtain the target facial image;

[0008] Display follow-up reading information and follow-up reading controls, and in response to the triggering operation of the follow-up reading controls, collect audio information input by the target user that corresponds to the follow-up reading information;

[0009] Acquire and display voice animations including images of the target user's lip movements, with the lip movements synchronized with the audio information content. The images of the target user's lip movements are obtained based on the lip shape and audio information in the target facial image.

[0010] One possible implementation method for obtaining voice animation includes:

[0011] Obtain at least one sample lip shape, which is used to express the pronunciation of at least one phoneme; update the lip shape of the target user in the target facial image based on the sample lip shape to obtain a synthetic facial image, in which the lip shape of the target user is used to express the pronunciation of the phoneme expressed by the corresponding sample lip shape;

[0012] Speech recognition is performed on audio information to obtain a phoneme sequence of the audio information, wherein the phoneme sequence includes at least one phoneme corresponding to a time point in the audio information;

[0013] Determine the synthetic facial image corresponding to each phoneme in the phoneme sequence, and obtain a synthetic facial image sequence of the target user's lip shape changes based on each synthetic facial image;

[0014] Obtain the audio frame sequence corresponding to the phoneme sequence in the audio information. Based on the time information of the audio frame sequence in the audio information, synchronize the audio frame sequence and the synthesized facial image sequence to generate a speech animation.

[0015] In one possible implementation, determining the synthesized facial image corresponding to each phoneme in the phoneme sequence includes:

[0016] Determine the lip shape corresponding to each phoneme in the phoneme sequence to obtain the lip shape sequence;

[0017] Determine the synthetic facial image corresponding to each lip shape in the lip shape sequence.

[0018] In one possible implementation, a synthetic facial image sequence is obtained, followed by:

[0019] The synthesized facial images in the synthesized facial image sequence are fused with preset source images to obtain a fused image sequence;

[0020] Synchronizing the audio frame sequence and the synthesized facial image sequence yields a speech animation, including:

[0021] By synchronizing the audio frame sequence and the fused image sequence, a voice animation is obtained.

[0022] In one possible implementation, speech recognition is performed on the audio information to obtain the phoneme sequence of the audio information, including:

[0023] Obtain the initial translated text of the audio information, and determine the language of the audio information based on the initial translated text;

[0024] If the language of the audio information is the target language, then obtain the word segmentation results to be corrected and the polyphonic characters in the word segmentation results from the initial translated text;

[0025] The correct polyphonic characters are selected from the polyphonic characters, and then filled into the word segmentation results to be corrected to obtain the correct word segmentation results.

[0026] Obtain the standard pronunciation of the correct word segmentation result, and perform phoneme recognition on the standard pronunciation through a preset acoustic model to obtain the phoneme sequence of audio information.

[0027] In one possible implementation, the initial translated text of the audio information is obtained, including:

[0028] The DC offset in the audio information is detected and eliminated, and the audio information after DC offset elimination is resampled to obtain the resampled audio information;

[0029] Human voice detection is performed on the resampled audio information to obtain human voice audio frames in the audio information;

[0030] Speech recognition is performed on human voice audio frames to obtain the initial translated text.

[0031] In one possible implementation, acquiring and displaying a speech animation that includes an image of the target user's lip movements includes:

[0032] Input the target facial image and audio information into the voice animation installation package running locally on the terminal, and obtain the voice animation output by the voice animation installation package;

[0033] The voice animation installation package is generated through the following steps:

[0034] Obtain the program code used to generate voice animation based on the target facial image and audio information;

[0035] The program code is compiled using a cross-toolchain to obtain a static library that runs on the target operating system. The cross-toolchain is a cross-compilation environment for the voice animation installation package to be generated.

[0036] Define the external interface and header files of the static library, and generate the voice animation installation package.

[0037] Secondly, a device for synthesizing voice-animation is provided, comprising:

[0038] The target facial image acquisition module is used to display the image acquisition control, and in response to the trigger operation of the image acquisition control, acquire the facial image of the target user to obtain the target facial image;

[0039] The audio information acquisition module is used to display the follow-up reading information and the follow-up reading control. In response to the trigger operation of the follow-up reading control, it collects the audio information entered by the target user that corresponds to the follow-up reading information.

[0040] The voice animation display module is used to acquire and display voice animations including images of the target user's lip movements. The target user's lip movements are synchronized with the content of the audio information, and the images of the target user's lip movements are obtained based on the lip shape and audio information in the target facial image.

[0041] In one possible implementation, the synthesis device further includes: a speech animation synthesis module, specifically, the speech animation synthesis module includes:

[0042] A synthetic face image generation submodule is used to obtain at least one sample lip shape, which is used to express the pronunciation of at least one phoneme; the lip shape of the target user in the target face image is updated according to the sample lip shape to obtain a synthetic face image, in which the lip shape of the target user is used to express the pronunciation of the phoneme expressed by the corresponding sample lip shape;

[0043] The speech recognition submodule is used to perform speech recognition on audio information and obtain the phoneme sequence of the audio information. The phoneme sequence includes at least one phoneme corresponding to a time point in the audio information.

[0044] The image sequence submodule is used to determine the synthetic facial image corresponding to each phoneme in the phoneme sequence, and to obtain a synthetic facial image sequence of the target user's lip shape changes based on each synthetic facial image.

[0045] The synchronization submodule is used to obtain the audio frame sequence corresponding to the phoneme sequence in the audio information, and synchronize the audio frame sequence and the synthesized facial image sequence according to the time information of the audio frame sequence in the audio information to generate a voice animation.

[0046] In one possible implementation, the image sequence submodule includes:

[0047] The lip shape sequence unit is used to determine the lip shape corresponding to each phoneme in the phoneme sequence to obtain the lip shape sequence.

[0048] The image correspondence unit is used to determine the synthetic facial image corresponding to each lip shape in the lip shape sequence.

[0049] In one possible implementation, the speech animation synthesis module also includes:

[0050] The fusion sequence submodule is used to fuse the synthesized facial images in the synthesized facial image sequence with preset source images to obtain a fused image sequence;

[0051] Correspondingly, the synchronization submodule is used to synchronize the audio frame sequence and the fused image sequence to obtain voice animation.

[0052] In one possible implementation, the speech recognition submodule includes:

[0053] The initial translation unit is used to obtain the initial translated text of the audio information and determine the language of the audio information based on the initial translated text.

[0054] The word segmentation unit is used to obtain the word segmentation results to be corrected and the polyphonic characters in the word segmentation results from the initial translated text if the language of the audio information is determined to be the target language.

[0055] The calibration unit is used to filter out the correct polyphonic characters from the polyphonic characters, fill the correct polyphonic characters into the word segmentation results to be corrected, and obtain the correct word segmentation results;

[0056] The phoneme recognition unit is used to obtain the standard pronunciation of the correct word segmentation result. It performs phoneme recognition on the standard pronunciation through a preset acoustic model to obtain the phoneme sequence of audio information.

[0057] In one possible implementation, the initial translation unit includes:

[0058] The preprocessing unit is used to detect and eliminate DC offset in the audio information and resample the audio information after DC offset elimination to obtain the resampled audio information.

[0059] The human voice detection unit is used to detect human voices in the resampled audio information and obtain human voice audio frames in the audio information.

[0060] The speech recognition unit is used to perform speech recognition on human voice audio frames to obtain the initial translated text.

[0061] In one possible implementation, the voice animation display module is specifically used to: input the target facial image and audio information into the voice animation installation package running locally on the terminal, and obtain the voice animation output by the voice animation installation package;

[0062] In one possible implementation, the synthesis apparatus further includes an installation package generation module, which includes:

[0063] The code module unit is used to obtain program code for generating voice animation based on the target facial image and audio information;

[0064] The compilation unit is used to compile program code using a cross-toolchain to obtain a static library that runs on the target operating system. The cross-toolchain is the cross-compilation environment corresponding to the voice animation installation package to be generated.

[0065] Define the unit, which is used to define the external interface and header files of the static library and generate the voice animation installation package.

[0066] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method provided in the first aspect.

[0067] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0068] Fifthly, embodiments of the present invention provide a computer program including computer instructions stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the computer device to perform steps implementing the method provided in the first aspect.

[0069] The speech animation synthesis method, apparatus, electronic device, and storage medium provided in this invention, through the display of an image acquisition control, in response to the trigger operation of the image acquisition control, acquires a facial image of the target user, laying the foundation for generating an image with the target user's lip shape number. After obtaining the target facial image, it displays follow-up reading information and a follow-up reading control. In response to the trigger operation of the follow-up reading control, it acquires audio information input by the target user corresponding to the follow-up reading information, and obtains and displays a speech animation including an image of the target user's lip shape changes. The lip shape changes of the target user are synchronized with the content of the audio information, enabling the user to see a speech animation in which their own lip shape changes synchronously with the content of the audio information during language learning. Since the lip shape in the speech animation is generated based on a preset sample lip shape corresponding to the standard pronunciation, the display effect of the speech animation is more realistic and the lip shape is closer to the lip shape during standard pronunciation. Therefore, the user can practice pronunciation and lip shape more accurately based on the speech animation, improving the interest and efficiency of language learning. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0071] Figure 1 A schematic diagram of the implementation environment provided for the embodiments of this application;

[0072] Figure 2 This is a flowchart illustrating a method for synthesizing voice animation according to an embodiment of this application;

[0073] Figure 3 This is a schematic diagram illustrating how the original lip shape is updated to a lip shape corresponding to different pronunciations in an embodiment of this application.

[0074] Figure 4a This is a schematic diagram of the interface for acquiring facial images as shown in an embodiment of this application;

[0075] Figure 4b This is a schematic diagram of the preparation interface before reading the follow-up information shown in the embodiments of this application;

[0076] Figure 4c This is a schematic diagram of the interface for displaying follow-up information in an embodiment of this application;

[0077] Figure 4dThis is a schematic diagram showing an interface for reading follow-up information according to another embodiment of this application;

[0078] Figure 4e This is a schematic diagram of the interface after the reading is completed, as shown in an embodiment of this application.

[0079] Figure 4f This is a schematic diagram of the interface for the voice animation shown in an embodiment of this application;

[0080] Figure 5 A schematic diagram of a model's face when uttering English phonemes according to an embodiment of this application;

[0081] Figure 6 This is a schematic diagram illustrating the process of obtaining a synthetic facial image sequence according to an embodiment of the application.

[0082] Figure 7 This application provides a schematic diagram of the structure of a speech animation synthesis device;

[0083] Figure 8 This application provides a schematic diagram of the structure of an electronic device. Detailed Implementation

[0084] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting the invention.

[0085] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0086] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0087] First, let's introduce and explain several terms used in this application:

[0088] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0089] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0090] 2) Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0091] 3) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0092] 4) Rhubarb Lip Sync is a command-line tool based on the CMU Sphinx speech recognition system that matches phonemes with lip shapes and outputs animations based on a timeline.

[0093] The speech animation synthesis method, apparatus, electronic device, and computer-readable storage medium provided in this application are intended to solve the above-mentioned technical problems of the prior art.

[0094] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0095] Please see Figure 1 The diagram illustrates an implementation environment provided in one embodiment of this application. This implementation environment may include: a terminal 110 and a server 120.

[0096] Terminal 110 installs and runs application 111, which can be a program for speech-to-animation synthesis. When terminal 110 runs application 111, the user interface of application 111 is displayed on the screen of terminal 110. Application 111 can be a language teaching program, a multimedia entertainment program, a photography program, a social communication program, etc. In this embodiment, application 111 is used as an example of a language teaching program. Terminal 110 is the terminal used by user 112. Application 111 displays an image acquisition control. In response to the trigger operation of the image acquisition control, it acquires the facial image of user 112, obtains the target facial image, displays follow-up reading information and follow-up reading control, and in response to the trigger operation of the follow-up reading control, acquires the audio information input by the user corresponding to the follow-up reading information, obtains and displays a speech animation including an image of the user's lip movements. The user's lip movements are synchronized with the content of the audio information, and the image of the user's lip movements is obtained based on the lip shape and audio information in the target facial image.

[0097] Optionally, terminal 110 can refer to one of multiple terminals, and this embodiment only uses terminal 110 as an example. The device types of terminal 110 include at least one of the following: smartphone, tablet computer, e-book reader, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, laptop computer, and desktop computer.

[0098] Figure 1 Only one terminal is shown in the diagram, but in different embodiments, multiple other terminals can access the server 120. Optionally, one or more terminals may also be terminals corresponding to developers, on which an application development and editing platform is installed. Developers can edit and update applications on these terminals and transmit the updated application installation package to the server 120 via wired or wireless network. Terminal 110 can download the application installation package from the server 120 to update the application.

[0099] The first terminal 110 and other terminals are connected to the server 120 via a wireless network or a wired network.

[0100] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0101] The server execution method in this application embodiment can be implemented in the form of cloud computing. Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing resources is called the "cloud." From the user's perspective, resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

[0102] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0103] Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

[0104] Server 120 is used to provide background services for applications. Optionally, server 120 undertakes the main computing work and the terminal undertakes the secondary computing work; or, server 120 undertakes the secondary computing work and the terminal undertakes the main computing work; or, server 120 and terminal use a distributed computing architecture for collaborative computing.

[0105] In an illustrative example, server 120 includes memory 121, processor 122, user account database 123, image processing service module 124, and user-facing input / output interface (I / O interface) 125. The processor 122 loads instructions stored in server 120 and processes data in user account database 123 and image processing service module 124. User account database 123 stores data about user accounts used by terminal 110 and other terminals, such as user account avatars, nicknames, learning records, and service areas. Image processing service module 124 provides multiple learning materials for users, such as oral English instruction, vocabulary memorization, and song learning. User-facing I / O interface 125 establishes communication and exchanges data with first terminal 110 and / or second terminal 130 via wireless or wired network.

[0106] Please see Figure 2 The illustration shows a flowchart of a speech animation synthesis method according to an embodiment of this application, including:

[0107] S101. Display the image acquisition control. In response to the trigger operation of the image acquisition control, acquire the facial image of the target user and obtain the target facial image.

[0108] This application embodiment provides an image acquisition control for user interaction. The method by which the user triggers the image acquisition control is not specifically limited; for example, it can be a single click, double click, long press, swipe, etc. When the user triggers the image acquisition control, an image acquisition device on or connected to the terminal, such as a camera or camera acquisition component, is invoked to acquire the target user's facial image in real time, serving as the target facial image. The target facial image refers to a facial image whose quality meets specified conditions; for example, the specified conditions could be that the facial clarity reaches a certain threshold, etc.

[0109] Optionally, to more quickly adjust the lip shape of the target user to correspond to different phonemes, the target facial image in this application embodiment can be taken when the target user is expressionless or smiling. Furthermore, after acquiring the target user's facial image, this application can also perform facial attribute detection on the target face in the target image. For example, the facial attributes may include attributes such as emotion. If it is determined that the target user's emotion is smiling, expressionless, etc., then it is determined that the lip shape in the facial image meets preset conditions.

[0110] S102, Display the follow-up reading information and the follow-up reading control, and in response to the trigger operation of the follow-up reading control, collect the audio information input by the target user that corresponds to the follow-up reading information.

[0111] This application embodiment does not specifically limit the execution order of steps S101 and S102. For example, step S101 may precede step S102, or step S102 may precede step S101, etc. By displaying the follow-up reading information, the target user can read the follow-up reading information aloud. Before reading aloud, in response to the target user's trigger operation on the follow-up reading space, the audio of the target user reading the follow-up reading information is captured by calling an audio acquisition device on the terminal or connected to the terminal, such as a microphone or other audio acquisition components.

[0112] S103. Acquire and display a speech animation including an image of the target user's lip movements. The target user's lip movements are synchronized with the content of the audio information. The image of the target user's lip movements is obtained based on the lip shape in the target facial image and the audio information.

[0113] By acquiring target facial images and audio information, computer vision techniques, such as Rhubarb LipSync, can be used to process the target facial images into images corresponding to different lip shapes. In these images, the main subject of the face is still the target user, but the lip shape is updated from the original lip shape in the target facial image to the lip shape corresponding to different pronunciations.

[0114] Please see Figure 3The illustration shows a schematic diagram of how the original lip shape is updated to a lip shape corresponding to different pronunciations in an embodiment of this application. As shown in the figure, the original lip shape is the lip shape when the user is not making a sound, and the lips are close to closed. Different pronunciation characteristics will result in different lip shapes. For example, when pronouncing the sound a (ah), the lips are naturally wide open. When pronouncing the sound o (oh), the lips are round. When pronouncing the sound e (e / yi), the mouth is slightly open and flat. When pronouncing the sound u (wu / yu), the lips are rounded and protrude into a small hole.

[0115] After acquiring images of different lip shapes of the target user, further synchronizing these images with the audio information will yield a voice animation of the target user's lip shape changes.

[0116] The speech animation synthesis method of this application embodiment, by displaying an image acquisition control, in response to the trigger operation of the image acquisition control, acquires the facial image of the target user, laying the foundation for generating an image with the target user's lip shape number, obtains the target facial image, displays follow-up reading information and a follow-up reading control, in response to the trigger operation of the follow-up reading control, acquires the audio information input by the target user corresponding to the follow-up reading information, obtains and displays a speech animation including an image of the target user's lip shape changes, and the lip shape changes of the target user are synchronized with the content of the audio information, so that when the user is learning a language, the speech animation can display the user's own lip shape changing synchronously with the content of the audio information. Since the lip shape in the speech animation is generated based on a preset sample lip shape corresponding to the standard pronunciation, the display effect of the speech animation is more realistic and the lip shape is closer to the lip shape during standard pronunciation. Therefore, the user can practice pronunciation and lip shape during pronunciation more accurately based on the speech animation, improving the interest and efficiency of language learning.

[0117] The following is combined with Figures 4a-4f To further understand the method for synthesizing voice animation in the embodiments of this application.

[0118] Please see Figure 4a The illustration shows a schematic diagram of an interface for capturing facial images, as shown in the embodiment of this application. As illustrated, the dashed box 401 in the interface guides the user to match the size and position of the face with the dashed box when capturing a facial image. This avoids problems such as failing to capture the user's complete face or capturing a face that is too small to clearly identify the user's lip shape. The interface also includes a camera switching control 403. By operating, for example, clicking this camera switching control 403, the user can switch between the front and rear cameras on the terminal. When switching to the front camera, the target user captured is generally the user operating the terminal; when switching to the rear camera, the target user captured is generally not the user operating the terminal. The user can complete the capture of the target facial image by clicking the image capture control 402 in the interface.

[0119] Figure 4b An exemplary diagram of the preparation interface before reading aloud, as shown in an embodiment of this application, is illustrated. The prompt message 404 in the diagram prompts the user whether they are ready to read aloud. The specific content of the prompt message can be "ready?" as shown in the diagram, or it can be something like "Are you ready?" or "Can we start?". The diagram also shows a confirmation control 405. When the user operates the confirmation control 405, it indicates that the user is ready to read aloud. In this embodiment, the confirmation control 405 can be operated by clicking, double-clicking, long-pressing, or sliding. Preferably, the confirmation control 405 is operated by long-pressing or sliding, which avoids accidental touches that could interrupt subsequent reading. In the diagram, the confirmation control 405 is triggered by sliding. When the user (represented by a hand in the diagram) presses and moves the confirmation control 405 from the left side to the right side of the sliding groove 406, it confirms that the user is ready to read aloud.

[0120] Optionally, the preparation interface before displaying the follow-up reading information in this embodiment of the application may not display a confirmation control, but instead enter the interface for displaying the follow-up reading information through a countdown. The specific time information of the countdown can be displayed in the preparation interface to remind the user to get ready to follow along as soon as possible.

[0121] Figure 4c An exemplary schematic diagram of the interface for displaying follow-up reading information in an embodiment of this application is shown. As shown in the figure, the follow-up reading information display area 417 in this interface is used to display follow-up reading information. Besides text, the follow-up reading information can also include information corresponding to the text, such as photos, pictures, and animations, thereby improving the user's cognitive level and enhancing the enjoyment of follow-up reading. For example, the text in the follow-up reading information display area in the figure is "dog," and a photo of a dog is also displayed. When the user triggers an operation on the follow-up reading control 407, the audio information corresponding to the follow-up reading information input by the target user begins to be collected. When there are multiple follow-up reading information items, the interface also includes a follow-up reading information item indicator area 408, used to display the real-time sorting of the follow-up reading information within the total follow-up reading information. Figure 4c For example, from the follow-up information item indicator area 408, it can be seen that the follow-up information "dog" is the first follow-up information.

[0122] Figure 4d A schematic diagram of an interface for reading aloud information, as exemplarily shown in another embodiment of this application, is illustrated. Figure 4d compared to Figure 4aThe new follow-up information "cat" and a cat pattern are displayed in the follow-up information display area 417. The corresponding follow-up information item indicator area 408 is also updated to the second follow-up information. Furthermore, for the first follow-up information, a five-pointed star pattern is used to indicate that the follow-up has been completed. It is understood that in practical applications, other forms can also be used to identify the follow-up information that has been completed. This application embodiment does not make specific limitations.

[0123] Figure 4e An exemplary schematic diagram of the interface after completing the follow-up reading as shown in the embodiment of this application is illustrated. As shown in the figure, the interface displays multiple display controls 409, each display control corresponding to a completed follow-up reading information. In addition to displaying the corresponding completed follow-up reading information, the display control 409 further includes a playback control 4091 and a first rereading control 4092. When the user triggers an operation on the playback control 4091, the audio information of the user reading the corresponding follow-up reading information will be played. When the user triggers an operation on the rereading control 4092, the interface for displaying the follow-up reading information will be further adjusted. When the user reads aloud again, the newly collected audio information of the user will overwrite the previously collected audio information. The interface also displays a second rereading control 410 and an animation generation control 411. When the user triggers the second rereading control 410, it means that the user needs to reread all the follow-up reading information, and then the user will be redirected to the interface of the first follow-up reading information. When the user triggers the animation generation control 411, a voice animation of the target user's lip movements is generated based on the target facial image and audio information. Since this process is time-consuming, a transition screen can be displayed before the user is redirected to the interface displaying the voice animation. The transition screen can display the words "Video is being generated" and the percentage of the video generation progress, so that the user can more accurately understand the generation progress of the audio animation.

[0124] Figure 4f An exemplary schematic diagram of the interface for a voice animation shown in an embodiment of this application is illustrated. As shown in the figure, the interface includes a playback area 412 for playing the voice animation. By triggering an operation on the playback area 412, the voice animation can be played or paused. As can be seen from the voice animation in the figure, the voice animation includes not only the user's facial image but also preset background materials. The figure shows background material of a virtual character with the target user's facial image dancing with a bear. This application integrates the image of the target user's lip movements with the background material to obtain a more engaging voice animation. Figure 4f The interface shown further includes a profile picture change control 413, a voice repeat control 414, a share control 415, and a save control 416. Specifically, when the profile picture change control 414 is triggered, it will redirect to... Figure 4aThe interface shown is used to recapture the avatar. When the reread voice control 414 is triggered, it jumps to the interface of the first follow-up information to recapture the audio information. When the share control 415 is triggered, it further displays other applications with sharing permissions to share the voice animation to other applications. When the save control 416 is triggered, it saves the voice animation to the local terminal.

[0125] Based on the above embodiments, as an optional embodiment, the method of obtaining voice animation includes:

[0126] S201. Obtain at least one sample lip shape, update the lip shape of the target user in the target facial image according to the sample lip shape, and obtain a synthetic facial image. The lip shape of the target user in the synthetic facial image is used to express the pronunciation of the phoneme expressed by the corresponding sample lip shape.

[0127] The sample lip shape in this embodiment is used to express the pronunciation of at least one phoneme. Specifically, a facial image of a model pronouncing a phoneme can be pre-captured, and then the lip shape region can be cropped from that facial image to obtain the sample image. Please refer to [link to relevant documentation]. Figure 5 It exemplarily illustrates a schematic diagram of a model's facial image when uttering English phonemes according to an embodiment of this application. Figure 5 A total of 9 illustrations of different lip shapes on the models were displayed, specifically:

[0128] Type A lip shape refers to the lip shape when pronouncing the consonants "P", "B", and "M". It is similar to type O lip shape, except that there is slight pressure between the lips.

[0129] Type B lip shape refers to the lip shape used when pronouncing consonants such as "K", "S", and "T". When pronouncing these sounds, the mouth needs to be open and the teeth clenched. It can also be used for some vowels, such as the "EE" in "bee".

[0130] Type C lip shape refers to the lip shape used when pronouncing vowels such as "EH" and "AE". Depending on the context, it is also used for some consonants. In addition, Type C lip shape is also used in the middle position when transitioning from Type A or Type B lip shape to Type D lip shape.

[0131] Type D lip shape refers to the lip shape used when pronouncing vowels such as "AA" in the word "father".

[0132] Type E lip shape is a slightly rounded mouth shape, used for sounds such as "AO" in "off" and "ER" in "bird". Type E lip shape involves a less open mouth than Type C lip shape. In addition, Type E lip shape is also used as an intermediate position when transitioning from Type C or Type D lip shape to Type F lip shape.

[0133] The F - type lip shape is a puckered lip shape, such as the lip shape when pronouncing "UW" in you, "OW" in show, and "W" in way.

[0134] The G - type lip shape, where the upper teeth touch the lower lip, such as the lip shape when pronouncing "F" in for and "V" in very.

[0135] The H - type lip shape is used to pronounce the long "L" sound. The tongue is raised behind the upper teeth. The mouth should be at least as wide as the C - type lip shape but less open than the D - type lip shape. It is the lip shape when pronouncing "P", "B", and "M".

[0136] The X - type lip shape generally appears in the idle position of pronunciation for pauses in speech. This lip shape is almost the same as the A - type lip shape, but the pressure between the lips is slightly less because the lips should be in a closed and relaxed state at this time.

[0137] Generally, a lip shape can include multiple key points, which are called "lip - shape key points" in the embodiments of this application. These multiple key points describe the contour of the lip shape. As an implementation, the key points can be distributed on the contour line of the lip shape, specifically at the two corners of the mouth, the outer edges of the upper and lower lips, and the inner edges of the lips. In addition to this example, other numbers of key points can also be used.

[0138] This application determines information such as the distance relationship, angular relationship, opening degree, and the proportion of the lips to the model's face among the lip - shape key points in the sample lip shape as the lip - shape features of the corresponding sample lip shape. Then, according to these lip - shape features, it correspondingly adjusts the lip - shape features of the lip - shape key points of the target user in the target facial image, so that the similarity between the adjusted lip - shape features of the target user and the lip - shape features of the sample lip shape meets the preset conditions, and thus a synthesized facial image can be obtained.

[0139] S202: Perform speech recognition on the audio information to obtain the phoneme sequence of the audio information. The phoneme sequence includes phonemes corresponding to at least one time point in the audio information.

[0140] In practice, a phoneme is the smallest speech unit divided according to the natural properties of speech. From an acoustic perspective, a phoneme is the smallest speech unit divided from the aspect of sound quality. Taking Chinese characters as an example, the Chinese syllable ā (ah) has one phoneme, ài (love) has two phonemes, dāi (stupid) has three phonemes, etc. Specifically, the speech signals collected in the embodiments of this application are in the time domain. To facilitate the analysis of speech signals, the speech signals in the time domain need to be converted into speech signals in the frequency domain. The acoustic features of this application can be acoustic features in the frequency domain, and the acoustic features can be represented by a spectrogram. Inputting the acoustic features into a pre - trained phoneme recognition model to obtain a phoneme sequence, which is a sequence composed of multiple phonemes.

[0141] S203. Determine the synthetic facial image corresponding to each phoneme in the phoneme sequence, and obtain a synthetic facial image sequence of the target user's lip shape changes based on each synthetic facial image.

[0142] Specifically, in this embodiment, the lip shape corresponding to each phoneme in the phoneme sequence can be determined to obtain the lip shape sequence. Then, the synthetic facial image corresponding to each lip shape in the lip shape sequence can be determined. The determined synthetic facial images are arranged according to the arrangement order of each phoneme in the phoneme sequence to obtain the synthetic facial image sequence of the target user's lip shape changes.

[0143] Please see Figure 6 The figure exemplifies a flowchart illustrating the process of obtaining a synthetic facial image sequence according to an embodiment of this application. As shown, each vertical bar (e.g., 611) represents a phoneme. All bars are sorted according to the time of appearance of the phonemes in the audio information, thus obtaining a phoneme sequence 610. The synthetic facial image set 620 includes corresponding synthetic facial images pre-obtained for each phoneme, including those based on… Figure 5 The nine sample lip shapes synthesized into synthetic facial images are represented as A'~H' and X'. Therefore, the corresponding synthetic facial image can be determined for each phoneme in the phoneme sequence 610, thereby obtaining the synthetic facial image sequence 630. The first synthetic facial image in this sequence is A', indicating that the synthetic facial image corresponding to the first phoneme in the phoneme sequence is A'.

[0144] S204. Obtain the audio frame sequence corresponding to the phoneme sequence in the audio information. Based on the time information of the audio frame sequence in the audio information, synchronize the audio frame sequence and the synthesized facial image sequence to generate a speech animation.

[0145] To ensure that the lip-shape number of the target user is synchronized with the sound, this embodiment of the application needs to determine the audio frame sequence corresponding to the phoneme sequence in the audio information. Generally, one audio frame in the audio information corresponds to one phoneme, so the audio frame sequence corresponding to the phoneme sequence is also unique. Furthermore, based on the time information of the audio frame sequence in the audio information, the duration of each lip shape of the target user can be determined, thereby obtaining the speech animation.

[0146] As an optional embodiment, step S201 described above can be performed by... Figure 1The server executes the steps shown, while steps S202-S204 are executed by the terminal. That is, after the terminal acquires the target facial image, it sends the target facial image to the server. The server updates the lip shape of the target user in the target facial image based on the pre-acquired sample lip shape, obtains a synthesized facial image, and returns it to the terminal. The terminal performs speech recognition on the audio information locally to obtain the phoneme sequence of the audio information, determines the synthesized facial image corresponding to each phoneme in the phoneme sequence, and obtains a sequence of synthesized facial images showing the lip shape changes of the target user based on each synthesized facial image. It also acquires the audio frame sequence corresponding to the phoneme sequence in the audio information, synchronizes the audio frame sequence and the synthesized facial image sequence based on the time information of the audio frame sequence in the audio information, and generates a speech animation. Since the most time-consuming step of obtaining the synthesized facial image is completed by the server, the processing pressure on the terminal side is smaller, and the total time for generating the speech animation is also relatively smaller. This is more suitable for the current situation where electronic devices (such as learning machines) in language teaching scenarios have low computing power.

[0147] Based on the above embodiments, as an optional embodiment, a synthetic facial image sequence is obtained, which then includes:

[0148] The synthesized facial images in the synthesized facial image sequence are fused with preset source images to obtain a fused image sequence.

[0149] In order to further enhance the fun of the voice animation, this embodiment of the application also requires that the synthesized facial images in the synthesized facial image sequence be fused with preset material images respectively. It can be understood that the specific fusion method can be to superimpose the synthesized facial image onto a preset position in the material image.

[0150] Synchronizing the audio frame sequence and the synthesized facial image sequence to obtain speech animation includes: synchronizing the audio frame sequence and the fused image sequence to obtain speech animation.

[0151] It should be understood that since each frame of the fused image sequence corresponds one-to-one with the synthesized facial image, the audio frame sequence and the fused image sequence can be synchronized using the above method to obtain voice animation. The embodiments of this application will not be described in detail.

[0152] Based on the above embodiments, as an optional embodiment, speech recognition is performed on the audio information to obtain the phoneme sequence of the audio information, including:

[0153] S301. Obtain the initial translated text of the audio information, and determine the language of the audio information based on the initial translated text;

[0154] This application embodiment can obtain the initial translated text of audio information through speech recognition technology. Since some languages ​​have characters / words with multiple pronunciations, which may lead to inaccurate translation results of audio information, this application further determines the language of the audio information. Optionally, the target language is English.

[0155] S302. If the language of the audio information is the target language, then obtain the word segmentation results to be corrected and the polyphonic characters in the word segmentation results from the initial translated text.

[0156] S303. Select the correct polyphonic characters from the polyphonic characters, fill the correct polyphonic characters into the word segmentation results to be corrected, and obtain the correct word segmentation results;

[0157] S304. Obtain the standard pronunciation of the correct word segmentation result, and perform phoneme recognition on the standard pronunciation through a preset acoustic model to obtain the phoneme sequence of the audio information.

[0158] The acoustic model is obtained by training on audio that meets the pronunciation conditions. The acoustic model has the ability to identify phonemes in speech, that is, the acoustic model calculates the posterior probability of acoustic features belonging to each phoneme.

[0159] The present invention also provides a method for creating an acoustic model, comprising:

[0160] S401. Obtain training samples, which are audio recordings of standard pronunciations. Standard pronunciations refer to audio recordings with good pronunciation conditions, such as those representing clarity of pronunciation, audio rate, etc.

[0161] S402. Segment the audio of the training samples into frames, and extract features from the segmented audio to obtain audio features.

[0162] S403, Generate phoneme labels for the audio of the training samples;

[0163] S404. Match the audio features with the phoneme labels to obtain the processed training samples;

[0164] S405. The acoustic model is obtained by iteratively training the training samples using a neural network model.

[0165] The training samples can be generated from several hours (e.g., 100 hours) of well-pronounced audio. The audio is first segmented into frames, and then audio features are extracted. For example, each frame is 25ms long, with a frame shift of 10ms, and the features are 40-dimensional Mel-scale Frequency Cepstral Coefficients (MFCCs).

[0166] After the audio features are extracted, the audio text is expanded into phonemes according to the dictionary. Each frame is divided into equal segments according to time and labeled with phoneme tags. After the audio features and phoneme tags are matched, the initial model is trained. When the iteration reaches a certain number of rounds, the training is stopped, and the final acoustic model is obtained.

[0167] Based on the above embodiments, as an optional embodiment, obtaining the initial translated text of the audio information includes:

[0168] S501. Detect and eliminate DC offset in audio information and resample the audio information after eliminating DC offset to obtain resampled audio information.

[0169] Audible artifacts are perceptible noise introduced into the sound output from an audio device, often caused by the operation of the audio device itself. Audible artifacts are generally undesirable and represent a deviation from the fidelity of the audio input to the device. Clicking is a specific type of audible artifact, an unpleasant audible artifact produced by a speaker, typically caused by abrupt transient voltage changes (e.g., DC offset across the speaker that may occur when an audio power amplifier switches between operating modes, such as power-off and power-on modes). Embodiments of this application can eliminate audible artifacts, including clicking, by detecting DC offset in the audio information, resulting in clearer audio information. Specifically, embodiments of this application can utilize digital filters to eliminate DC offset in the audio information. For example, an infinite-response high-pass filter can be used to eliminate DC offset.

[0170] S502. Perform human voice detection on the resampled audio information to obtain human voice audio frames in the audio information.

[0171] Specifically, embodiments of this application can use a pre-trained voice detection model to detect human information in resampled audio information. For example, audio data is input into the voice detection model, which performs voice detection on the audio data and outputs the detection result. The detection result may include the audio time period in which the voice appears and the audio frames of the voice, etc. In practical applications, other parameters may also be included as needed, which are not limited in this embodiment.

[0172] S503. Perform speech recognition on the human voice audio frame to obtain the initial translated text.

[0173] Based on the above embodiments, as an optional embodiment, acquiring and displaying a speech animation including an image of the target user's lip movements includes:

[0174] Input the target facial image and audio information into the voice animation installation package running locally on the terminal to obtain the voice animation output by the voice animation installation package.

[0175] This application embodiment compiles a voice animation installation package and runs it locally on the terminal, enabling the voice animation of the target user's lip movements to be generated locally on the terminal. This overcomes the drawbacks of Rhubarb Lip Sync, which is a command-line tool, and cannot be applied to mobile devices, is suitable for developers, and has a high learning curve.

[0176] The voice animation installation package is generated through the following steps:

[0177] S601. Obtain program code for obtaining voice animation based on target facial image and audio information;

[0178] S602. Use the cross-toolchain to compile the program code to obtain a static library that runs on the target operating system. The cross-toolchain is the cross-compilation environment corresponding to the voice animation installation package to be generated.

[0179] The cross-toolchain in this application embodiment is a cross-compilation environment for the voice animation installation package to be generated, and the target platform may include Android and iOS platforms. That is, the cross-toolchain in this embodiment of the invention implements cross-platform compilation.

[0180] During the compilation process, all functions and code are completed on a Linux machine, which is a Unix operating system. The user terminal is an Android operating platform or an iOS platform. Since they are based on Android and iOS systems, cross-platform compilation is required.

[0181] Cross-platform compilation first requires setting up a cross-compilation environment, which involves installing and configuring a cross-compilation toolchain. Within this environment, the operating system, applications, and other components required for the embedded Linux system are compiled, and then uploaded to the target machine. Different versions of Android and iOS each require their own dedicated cross-compilation environment.

[0182] Define the external interface and header files of the static library, and generate the voice animation installation package.

[0183] The program code for voice animation is compiled into a static library that can run on both Android and iOS development platforms. Then, the interface parts for Android and iOS are designed, the external interface and header files of the static library are defined, and the localized voice animation synthesis function is called from Android and iOS.

[0184] In this embodiment of the invention, after generating the voice animation installation package, it needs to be tested. Then, based on the test results, it is determined whether to release the installation package or to debug and modify it. This testing process involves comparing the test results obtained when the voice animation installation package is executed on a local terminal with the test results from the server to determine whether the installation package is usable.

[0185] The testing section primarily determines the usability of the local speech-animation synthesis function. Usability is assessed in three aspects: stability, accuracy, and low latency. Stability means the program should not crash mid-run. Local speech-animation synthesis was performed on one thousand target encoded images and speech information, and no crashes were observed. Accuracy is evaluated by ensuring there is no deviation between the locally synthesized speech-animation and the server-synthesized speech-animation. Testing with one thousand speech-animations yielded an average score difference within tolerable limits. Latency testing involves statistically analyzing the latency difference between local and server-synthesized speech-animation. The local average latency was significantly lower than the server's average latency. In this embodiment of the invention, the generated speech-animation installation package can run on user terminals without relying on a network.

[0186] This application provides a speech animation synthesis apparatus, such as... Figure 7 As shown, the device may include: a target facial image acquisition module 101, an audio information acquisition module 102, and a voice animation display module 103, specifically:

[0187] The target facial image acquisition module 101 is used to display an image acquisition control, and in response to the trigger operation of the image acquisition control, acquires the facial image of the target user and obtains the target facial image;

[0188] The audio information acquisition module 102 is used to display the follow-up reading information and the follow-up reading control, and in response to the trigger operation of the follow-up reading control, to collect the audio information entered by the target user that corresponds to the follow-up reading information.

[0189] The voice animation display module 103 is used to acquire and display voice animations including images of the target user's lip shape changes. The target user's lip shape changes are synchronized with the content of the audio information. The images of the target user's lip shape changes are obtained based on the lip shape and audio information in the target facial image.

[0190] The speech animation synthesis apparatus provided in this embodiment of the invention specifically executes the process described in the above-described method embodiment. For details, please refer to the content of the speech animation synthesis method embodiment described above; further details will not be repeated here. The speech animation synthesis apparatus provided in this embodiment of the invention, by displaying an image acquisition control, in response to the trigger operation of the image acquisition control, acquires a facial image of the target user, laying the foundation for generating an image with the target user's lip shape number. After obtaining the target facial image, it displays follow-up reading information and a follow-up reading control. In response to the trigger operation of the follow-up reading control, it acquires the audio information input by the target user corresponding to the follow-up reading information, obtains and displays a speech animation including an image of the target user's lip shape changes. The lip shape changes of the target user are synchronized with the content of the audio information, enabling the user to see a speech animation in which their own lip shape changes synchronously with the content of the audio information during language learning. Since the lip shape in the speech animation is generated based on a preset sample lip shape corresponding to the standard pronunciation, the display effect of the speech animation is more realistic and the lip shape is closer to the lip shape during standard pronunciation. Therefore, the user can practice pronunciation and lip shape more accurately based on the speech animation, improving the interest and efficiency of language learning.

[0191] Based on the above embodiments, as an optional embodiment, the synthesis device further includes: a speech animation synthesis module, specifically, the speech animation synthesis module includes:

[0192] A synthetic face image generation submodule is used to obtain at least one sample lip shape, which is used to express the pronunciation of at least one phoneme; the lip shape of the target user in the target face image is updated according to the sample lip shape to obtain a synthetic face image, in which the lip shape of the target user is used to express the pronunciation of the phoneme expressed by the corresponding sample lip shape;

[0193] The speech recognition submodule is used to perform speech recognition on audio information and obtain the phoneme sequence of the audio information. The phoneme sequence includes at least one phoneme corresponding to a time point in the audio information.

[0194] The image sequence submodule is used to determine the synthetic facial image corresponding to each phoneme in the phoneme sequence, and to obtain a synthetic facial image sequence of the target user's lip shape changes based on each synthetic facial image.

[0195] The synchronization submodule is used to obtain the audio frame sequence corresponding to the phoneme sequence in the audio information, and synchronize the audio frame sequence and the synthesized facial image sequence according to the time information of the audio frame sequence in the audio information to generate a voice animation.

[0196] Based on the above embodiments, as an optional embodiment, the image sequence submodule includes:

[0197] The lip shape sequence unit is used to determine the lip shape corresponding to each phoneme in the phoneme sequence to obtain the lip shape sequence.

[0198] The image correspondence unit is used to determine the synthetic facial image corresponding to each lip shape in the lip shape sequence.

[0199] Based on the above embodiments, as an optional embodiment, the speech animation synthesis module further includes:

[0200] The fusion sequence submodule is used to fuse the synthesized facial images in the synthesized facial image sequence with preset source images to obtain a fused image sequence;

[0201] Correspondingly, the synchronization submodule is used to synchronize the audio frame sequence and the fused image sequence to obtain voice animation.

[0202] Based on the above embodiments, as an optional embodiment, the speech recognition submodule includes:

[0203] The initial translation unit is used to obtain the initial translated text of the audio information and determine the language of the audio information based on the initial translated text.

[0204] The word segmentation unit is used to obtain the word segmentation results to be corrected and the polyphonic characters in the word segmentation results from the initial translated text if the language of the audio information is determined to be the target language.

[0205] The calibration unit is used to filter out the correct polyphonic characters from the polyphonic characters, fill the correct polyphonic characters into the word segmentation results to be corrected, and obtain the correct word segmentation results;

[0206] The phoneme recognition unit is used to obtain the standard pronunciation of the correct word segmentation result. It performs phoneme recognition on the standard pronunciation through a preset acoustic model to obtain the phoneme sequence of audio information.

[0207] Based on the above embodiments, as an optional embodiment, the initial translation unit includes:

[0208] The preprocessing unit is used to detect and eliminate DC offset in the audio information and resample the audio information after DC offset elimination to obtain the resampled audio information.

[0209] The human voice detection unit is used to detect human voices in the resampled audio information and obtain human voice audio frames in the audio information.

[0210] The speech recognition unit is used to perform speech recognition on human voice audio frames to obtain the initial translated text.

[0211] Based on the above embodiments, as an optional embodiment, the voice animation display module is specifically used to: input the target facial image and audio information into the voice animation installation package running locally on the terminal, and obtain the voice animation output by the voice animation installation package;

[0212] Based on the above embodiments, as an optional embodiment, the synthesis apparatus further includes an installation package generation module, which includes:

[0213] The code module unit is used to obtain program code for generating voice animation based on the target facial image and audio information;

[0214] The compilation unit is used to compile program code using a cross-toolchain to obtain a static library that runs on the target operating system. The cross-toolchain is the cross-compilation environment corresponding to the voice animation installation package to be generated.

[0215] Define the unit, which is used to define the external interface and header files of the static library and generate the voice animation installation package.

[0216] This application provides an electronic device comprising: a memory and a processor; at least one program stored in the memory, which, when executed by the processor, can achieve the following compared to the prior art: by displaying an image acquisition control, in response to a trigger operation of the image acquisition control, acquiring a facial image of a target user, laying the foundation for generating an image with the target user's lip shape number, obtaining the target facial image, displaying follow-up reading information and a follow-up reading control, in response to a trigger operation of the follow-up reading control, acquiring audio information input by the target user corresponding to the follow-up reading information, acquiring and displaying a speech animation including an image of the target user's lip shape changes, wherein the target user's lip shape changes are synchronized with the content of the audio information, so that when the user is learning a language, a speech animation showing the user's own lip shape changing synchronously with the content of the audio information can be displayed. Since the lip shape in the speech animation is generated based on a preset sample lip shape corresponding to the standard pronunciation, the display effect of the speech animation is more realistic and the lip shape is closer to the lip shape during standard pronunciation. Therefore, the user can practice pronunciation and lip shape more accurately based on the speech animation, improving the interest and efficiency of language learning.

[0217] In one alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of this electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0218] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0219] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0220] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0221] The memory 4003 stores application code that executes the scheme of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0222] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments. Compared with the prior art, by displaying an image acquisition control, in response to the triggering operation of the image acquisition control, a facial image of the target user is acquired, laying the foundation for generating an image with the target user's lip shape number. The target facial image is obtained, and follow-up reading information and a follow-up reading control are displayed. In response to the triggering operation of the follow-up reading control, audio information input by the target user corresponding to the follow-up reading information is acquired, and a speech animation including an image of the target user's lip shape changes is obtained and displayed. The lip shape changes of the target user are synchronized with the content of the audio information, allowing the user to see a speech animation in which their own lip shape changes synchronously with the content of the audio information during language learning. Since the lip shape in the speech animation is generated based on a preset sample lip shape corresponding to the standard pronunciation, the display effect of the speech animation is more realistic and the lip shape is closer to the lip shape during standard pronunciation. Therefore, users can practice pronunciation and lip shape more accurately based on the speech animation, improving the interest and efficiency of language learning.

[0223] This application provides a computer program that includes computer instructions stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the computer device to perform the content shown in the foregoing method embodiments. Compared to existing technologies, this method utilizes an image acquisition control to capture facial images of the target user in response to its triggering action. This lays the foundation for generating images with the target user's lip shape number. After obtaining the target facial image, the method displays follow-up reading information and a follow-up reading control. In response to the follow-up reading control's triggering action, it captures audio information input by the target user corresponding to the follow-up reading information. It then acquires and displays a speech animation including images of the target user's lip shape changes. The target user's lip shape changes are synchronized with the audio information, allowing the user to experience a speech animation where their lip shape changes synchronously with the audio content during language learning. Since the lip shape in the speech animation is generated based on preset sample lip shapes corresponding to standard pronunciation, the display effect is more realistic and the lip shape is closer to the standard pronunciation. Therefore, users can practice pronunciation and lip shape more accurately based on the speech animation, improving their interest and efficiency in language learning.

[0224] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0225] The above are only some embodiments of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method of synthesizing speech animation, characterized by, The method is executed by a terminal and includes the following steps: displaying an image collection control, collecting a face image of a target user in response to a triggering operation of the image collection control, obtaining a target face image, and sending the target face image to a server; displaying follow-up reading information and a follow-up reading control, collecting audio information corresponding to the follow-up reading information and input by the target user in response to a triggering operation of the follow-up reading control; obtaining a synthesized face image corresponding to each phoneme from the server, and displaying a speech animation including an image of target user lip movement according to the synthesized face image corresponding to each phoneme, the target user lip movement being synchronized with the content of the audio information, and the image of the target user lip movement being obtained according to the lip shape in the target face image and the audio information; wherein the synthesized face image is obtained by the server through the following steps: determining lip shape features of a plurality of sample lip shapes, each sample lip shape being used to express pronunciation of at least one phoneme, the lip shape features including angle relationships of each lip shape key point and a proportion of a lip part to a sample face; adjusting the lip shape features of the lip shape key points of the target user in the target face image according to the lip shape features of the sample lip shapes, and obtaining the synthesized face image.

2. The method of speech animation synthesis of claim 1, wherein, The manner of obtaining the speech animation includes: performing speech recognition on the audio information to obtain a phoneme sequence of the audio information, the phoneme sequence including phonemes corresponding to at least one time point in the audio information; determining a synthesized face image corresponding to each phoneme in the phoneme sequence, and obtaining a synthesized face image sequence of target user lip movement according to each synthesized face image; obtaining an audio frame sequence corresponding to the phoneme sequence in the audio information, synchronizing the audio frame sequence and the synthesized face image sequence according to time information of the audio frame sequence in the audio information, and generating the speech animation.

3. The method of speech animation synthesis of claim 2, wherein, The determining of the synthesized face image corresponding to each phoneme in the phoneme sequence includes: determining a lip shape corresponding to each phoneme in the phoneme sequence to obtain a lip shape sequence; determining a synthesized face image corresponding to each lip shape in the lip shape sequence.

4. The method of speech animation synthesis of claim 2, wherein, The obtaining of the synthesized face image sequence further includes: fusing the synthesized face images in the synthesized face image sequence with preset material images respectively to obtain a fused image sequence; The synchronizing of the audio frame sequence and the synthesized face image sequence to obtain the speech animation includes: synchronizing the audio frame sequence and the fused image sequence to obtain the speech animation.

5. The method of speech animation synthesis of claim 2, wherein, The speech recognition on the audio information to obtain the phoneme sequence of the audio information includes: obtaining an initial translation text of the audio information, and determining a language of the audio information according to the initial translation text; if it is determined that the language of the audio information is a target language, obtaining a word segmentation result to be corrected and multi-phonetic characters in the word segmentation result from the initial translation text; selecting correct multi-phonetic characters from the multi-phonetic characters, filling the correct multi-phonetic characters into the word segmentation result to be corrected, and obtaining a correct word segmentation result; The standard pronunciation of the correct word segmentation result is obtained, and the standard pronunciation is subjected to phoneme recognition through a preset acoustic model to obtain a phoneme sequence of the audio information.

6. The method of speech animation synthesis of claim 5, wherein, The obtaining of the initial translation text of the audio information includes: Detecting and eliminating the direct current offset in the audio information, and resampling the audio information after eliminating the direct current offset to obtain resampled audio information; Performing voice detection on the resampled audio information to obtain a voice audio frame in the audio information; Performing speech recognition on the voice audio frame to obtain the initial translation text.

7. The speech animation synthesis method according to any one of claims 1 to 6, characterized by, The speech animation including the image of the lip shape change of the target user includes: Inputting the target face image and the audio information into a speech animation installation package running locally on the terminal to obtain the speech animation output by the speech animation installation package; The speech animation installation package is generated through the following steps: Obtaining program code for obtaining the speech animation according to the target face image and the audio information; Compiling the program code using a cross-tool chain to obtain a static library running on a target operating system, the cross-tool chain being a cross-compilation environment corresponding to the speech animation installation package to be generated; Defining the external interface and header file of the static library to generate the speech animation installation package.

8. A speech animation synthesizing apparatus characterized by comprising: The synthesis device applied to a terminal includes: A target face image acquisition module configured to display an image acquisition control, acquire a face image of a target user in response to a trigger operation of the image acquisition control, obtain a target face image, and send the target face image to a server; An audio information acquisition module configured to display follow-up reading information and a follow-up reading control, and acquire audio information corresponding to the follow-up reading information and input by the target user in response to a trigger operation of the follow-up reading control; A speech animation display module configured to acquire a synthesized face image corresponding to each phoneme from the server, and display a speech animation including an image of a lip shape change of a target user according to the synthesized face image corresponding to each phoneme, the lip shape change of the target user being synchronized with the content of the audio information, and the image of the lip shape change of the target user being obtained according to the lip shape in the target face image and the audio information. The synthesized face image is obtained by the server through the following manner: Determining lip shape features of a plurality of sample lip shapes, each sample lip shape being used to express pronunciation of at least one phoneme, the lip shape features including angle relationships of key points of the lip shape and proportions of the lip part to a sample face; Adjusting the lip shape features of the key points of the lip shape of the target user in the target face image according to the lip shape features of the sample lip shapes to obtain the synthesized face image.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the speech animation synthesis method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions cause the computer to perform the steps of the speech animation synthesis method according to any one of claims 1 to 7.

Images