Mouth shape generation method and device of virtual character and electronic equipment

Abstract

The invention discloses a mouth shape generation method and device of a virtual character and electronic equipment. Relates to the field of audio and video, and comprises the following steps: obtaining target audio information broadcast through a virtual character, and identifying the target audio information to obtain sound feature information and text information of the target audio information; identifying emotional feature information of the target audio information according to the sound feature information and the text information; and determining the mouth shape animation of the virtual character according to the sound feature information, the text information and the emotion feature information. Through the method and the device, the problem of relatively low accuracy of generating the mouth shape of the virtual character by identifying phoneme information in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of audio and video, and in particular to a method, device and electronic device for generating lip shapes of a virtual character. Background Art

[0002] With the advancement of technology and the rapid development of the financial industry, virtual characters have gradually become an important part of the intelligent services of banks and other financial institutions. Virtual characters can interact with customers and provide basic banking business processing and consulting services. When virtual characters interact with users, lip animations need to be generated during the interaction. Through lip animations, virtual characters can communicate with customers more naturally and enhance customer experience.

[0003] In the related art, lip animation is usually generated based on audio information. When generating lip animation based on audio information, it is mainly through identifying phoneme information, and then further matching the corresponding lip information and generating the corresponding lip animation according to the sound features contained in the phoneme information.

[0004] However, in the process of generating lip-sync animation, the above-mentioned existing technology only considers the phoneme information in the audio, without comprehensive factors such as emotional information, resulting in low accuracy of the generated lip-sync animation and affecting the matching degree between the lip-sync animation and the audio content.

[0005] With regard to the problem of low accuracy in generating the lip shape of a virtual character by recognizing phoneme information in the related art, no effective solution has been proposed so far. Summary of the invention

[0006] The main purpose of the present application is to provide a method, device and electronic device for generating lip shapes of a virtual character, so as to solve the problem of low accuracy in generating lip shapes of a virtual character by recognizing phoneme information in the related art.

[0007] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for generating lip shapes of a virtual character is provided. The method comprises: obtaining target audio information broadcasted by a virtual character, and identifying the target audio information to obtain sound feature information and text information of the target audio information; identifying emotional feature information of the target audio information according to the sound feature information and text information; and determining the lip shape animation of the virtual character according to the sound feature information, text information and emotional feature information.

[0008] Optionally, identifying the target audio information to obtain sound feature information and text information of the target audio information includes: preprocessing the target audio information to obtain processed audio information; inputting the processed audio information into a text recognition model to obtain text information, wherein the text recognition model is trained by multiple first sample data, and the first sample data includes sample audio information and sample text information corresponding to the sample audio information; performing feature extraction on the processed audio information to obtain sound feature information, wherein the sound feature information includes at least one of the following: pitch, sound intensity, and sound length.

[0009] Optionally, identifying emotional characteristic information of target audio information based on sound characteristic information and text information includes: inputting the sound characteristic information and text information into an emotion recognition model to obtain the emotional characteristic information of the target audio information, wherein the emotion recognition model is trained by multiple second sample data, and the second sample data includes sample sound characteristic information and sample text information, and corresponding emotional characteristic information.

[0010] Optionally, determining the lip animation of the virtual character based on the sound feature information, text information and emotional feature information includes: generating an initial lip animation of the virtual character based on the text information; correcting the initial lip animation based on the sound feature information to obtain a candidate lip animation; and correcting the candidate lip animation based on the emotional feature information to obtain the lip animation of the virtual character.

[0011] Optionally, generating an initial lip animation of a virtual character based on text information includes: segmenting the text information to obtain multiple words, and obtaining a lip animation corresponding to each word in a lip animation library to obtain multiple first lip animations, and splicing the multiple first lip animations according to the order of the multiple words in the text information to obtain the initial lip animation of the virtual character; or, inputting the text information into a lip animation recognition model to obtain an initial lip animation, wherein the lip animation recognition model is trained by multiple third sample data, and the third sample data includes sample text information, and sample lip animation corresponding to the sample text information.

[0012] Optionally, the initial lip animation is corrected according to the sound feature information to obtain a candidate lip animation, including: determining a first feature of the lip animation according to the sound feature information, and identifying whether the first feature exists in the initial lip animation; when the first feature exists in the initial lip animation, determining the initial lip animation as a candidate lip animation; when the first feature does not exist in the initial lip animation, adjusting the initial lip animation according to the first feature to obtain a candidate lip animation; correcting the candidate lip animation according to the emotion feature information to obtain the lip animation of the virtual character includes: determining a second feature of the lip animation according to the emotion feature information, and adjusting the candidate lip animation according to the second feature to obtain the lip animation of the virtual character.

[0013] Optionally, identifying emotional characteristic information of target audio information based on sound characteristic information and text information includes: determining whether there is contextual audio information for the target audio information; and if there is contextual audio information for the target audio information, identifying the emotional characteristic information of the target audio information based on the contextual audio information, sound characteristic information and text information.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a device for generating lip shapes of a virtual character is provided. The device comprises: an acquisition unit, which is used to acquire target audio information broadcasted by the virtual character, and recognize the target audio information to obtain sound feature information and text information of the target audio information; a recognition unit, which is used to recognize the emotional feature information of the target audio information according to the sound feature information and text information; and a determination unit, which is used to determine the lip shape animation of the virtual character according to the sound feature information, text information and emotional feature information.

[0015] In an embodiment of the present application, a method is adopted in which target audio information broadcasted by a virtual character is obtained, and the target audio information is identified to obtain sound feature information and text information of the target audio information; emotional feature information of the target audio information is identified based on the sound feature information and text information; and the lip animation of the virtual character is determined based on the sound feature information, text information and emotional feature information. The sound feature information and text information in the audio information are extracted, and the emotional feature information of the audio is determined by using an emotion recognition model. Then, the sound feature information, text information and emotional feature information are comprehensively considered to obtain a lip animation with a higher accuracy. This solves the problem of low accuracy in generating the lip shape of a virtual character by identifying phoneme information in related technologies, thereby achieving the technical effect of improving the vividness of the virtual character's emotional expression and ensuring the accuracy of the lip shape generation of the virtual character. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating lip shapes of a virtual character is shown;

[0018] Figure 2 is a flow chart of a method for generating lip shapes of a virtual character according to Embodiment 1 of the present application;

[0019] Figure 3 is a schematic diagram of a lip shape generation device for a virtual character provided in Example 2 of the present application;

[0020] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] It should be noted that the method, device and electronic device for generating lip shapes of virtual characters determined in the present disclosure can be used in the audio and video field, and can also be used in any field other than the audio and video field. The application field of the method, device and electronic device for generating lip shapes of virtual characters determined in the present disclosure is not limited.

[0025] It should be noted that the collected information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) used in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated. Corresponding operation entrances are provided for users to choose to authorize use or refuse use. If the user chooses to refuse, the expert decision-making process is entered. For example, an interface is set between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0026] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0027] Example 1

[0028] According to an embodiment of the present application, a method embodiment of generating lip shapes of a virtual character is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating lip shapes of a virtual character is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0030] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the lip shape generation method of the virtual character in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the lip shape generation method of the virtual character described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0034] Under the above operating environment, this application provides Figure 2 The method for generating lip shapes of a virtual character is shown. Figure 2 It is a flowchart of the method for generating lip shapes of a virtual character according to Example 1 of the present application.

[0035] Step S201, obtaining target audio information broadcasted by a virtual character, and identifying the target audio information to obtain sound feature information and text information of the target audio information.

[0036] Specifically, the executor of this embodiment may be a lip-sync generation system. During the lip-sync generation process of a virtual character, the system may first obtain target audio information broadcast by the virtual character. The target audio information may be audio pre-recorded by bank staff, and the target audio information may be collected through a microphone or other audio input device.

[0037] Furthermore, after collecting the target audio information, the acquired target audio information can be preprocessed. For example, the original target audio information can be processed by noise reduction, audio segmentation, volume standardization, format conversion, etc., thereby improving the quality, readability and usability of the audio information, thereby making the audio information easier to be analyzed and processed by machine learning models or other analysis tools.

[0038] Furthermore, the pre-processed audio information can be subjected to sound feature recognition and text recognition through existing audio analysis and processing tools such as speech recognition technology or recognition models to obtain sound feature information and text information of the target audio information, wherein the sound feature information may include feature information such as pitch, loudness, duration, rhythm, timbre, and speed, and the text information is a text description of the speech content contained in the audio.

[0039] Step S202: identifying emotional feature information of the target audio information based on the sound feature information and the text information.

[0040] Specifically, the emotional features of the target audio information can be emotional categories and emotional levels, etc. The emotional categories can express the types of emotions contained in the sounds in the audio information, such as happiness, sadness, surprise, etc. The emotional levels can express the degree of a certain emotional category, such as mild, medium, strong, etc. Different sound feature information or text information corresponds to different emotional feature information. For example, a higher pitch can indicate happiness or surprise, and a lower pitch can indicate sadness or calmness. Therefore, the sound feature information and text information can be combined to improve the accuracy of identifying the emotional feature information of the target audio information.

[0041] Furthermore, in the process of identifying the emotional feature information of the target audio information based on the sound feature information and the text information, the emotional feature information can be obtained by performing identification and prediction using a pre-trained emotion recognition model based on the sound feature information and the text information.

[0042] It should be noted that in the process of identifying the emotional characteristic information of the target audio information based on the sound characteristic information and text information, in addition to considering the sound characteristic information and text information of the current segment of the target audio information, the contextual audio information of the target audio information can also be considered. By combining the contextual audio information, sound characteristic information and text information, the emotional characteristic information of the target audio information can be comprehensively determined, thereby improving the accuracy of identifying the emotional characteristic information of the target audio information.

[0043] Step S203, determining the lip animation of the virtual character according to the sound feature information, text information and emotion feature information.

[0044] Specifically, after determining the emotional feature information of the target audio information, the sound feature information, text information and emotional feature information can be fused and grouped according to the text information content. For example, if the text information content of the target audio information is "I wish you a nice day", it can be divided into four groups, namely, "I wish you", "you", "a day" and "happy". The text information, sound feature information and emotional feature information of each group are fused and input into a pre-trained lip animation recognition model to obtain the lip animation data corresponding to each group of text content. For example, the lip animation data may include the degree of lip opening and closing, the shape of the lips, the position and angle of the corners of the mouth, the position of the tongue, etc. The lip animation data corresponding to each group of text content are then combined and spliced ​​to finally obtain the lip animation of the virtual character.

[0045] It should be noted that the lip-sync animation recognition model can adopt a regression model, a neural network, etc., and the model can be trained through a large amount of historical text information, sound feature information, emotional feature information, and the corresponding lip-sync animation data. The parameters of the model can be optimized through training, so that the model can learn the mapping rules from historical text information, sound feature information and emotional feature information to lip-sync animation, thereby improving the accuracy of the model in recognizing lip-sync animation.

[0046] The method for generating lip shapes of a virtual character provided in an embodiment of the present application adopts the method of acquiring target audio information broadcasted by a virtual character, and identifying the target audio information to obtain sound feature information and text information of the target audio information; identifying the emotional feature information of the target audio information based on the sound feature information and the text information; and determining the lip shape animation of the virtual character based on the sound feature information, the text information and the emotional feature information. The method extracts the sound feature information and the text information from the audio information, and uses an emotion recognition model to determine the emotional feature information of the audio. Then, the sound feature information, the text information and the emotional feature information are comprehensively considered to obtain a lip shape animation with a high accuracy. The method solves the problem of low accuracy in generating lip shapes of virtual characters by identifying phoneme information in related technologies, thereby achieving the technical effect of improving the vividness of the emotional expression of the virtual character and ensuring the accuracy of the lip shape generation of the virtual character.

[0047] Optionally, in the lip shape generation method of a virtual character provided in an embodiment of the present application, target audio information is recognized to obtain sound feature information and text information of the target audio information, including: preprocessing the target audio information to obtain processed audio information; inputting the processed audio information into a text recognition model to obtain text information, wherein the text recognition model is trained by multiple first sample data, and the first sample data includes sample audio information and sample text information corresponding to the sample audio information; and performing feature extraction on the processed audio information to obtain sound feature information, wherein the sound feature information includes at least one of the following: pitch, sound intensity, and sound length.

[0048] It should be noted that the target audio information is audio information broadcast by a virtual character. After the target audio information is collected, the target audio information can be preprocessed first to improve the quality of the target audio information. For example, noise reduction processing can be performed to reduce background noise in the target audio information and improve the clarity of the audio information. Volume standardization processing can also be performed to control the volume of the target audio information within an appropriate range to avoid distortion or unclear audio caused by excessive or insufficient audio. By preprocessing the target audio information, the audio information can be more easily analyzed and processed by machine learning models such as text recognition models or other analysis tools.

[0049] Furthermore, the preprocessed target audio information can be input into a text recognition model, and the text information corresponding to the target audio information can be obtained through the output result of the text recognition model, wherein the text recognition model can be a neural network model, etc., by using multiple groups of first sample data, that is, multiple groups of sample audio information and their corresponding sample text information for training until the recognition accuracy of the model reaches the expected value, the final text recognition model can be obtained, thereby ensuring the accuracy of text recognition.

[0050] Furthermore, feature extraction can be performed on the preprocessed target audio information, and existing audio processing tools or feature extraction models can be used to extract sound-related features from the target audio information to obtain sound feature information, wherein the sound feature information may include feature information such as pitch, loudness, length, rhythm, timbre, and speed of sound. The accuracy of virtual character lip shape generation can then be improved based on the sound feature information.

[0051] This embodiment preprocesses the target audio information and then performs text recognition and sound feature extraction on the preprocessed target audio information, thereby improving the accuracy of the virtual character's lip shape generation.

[0052] Optionally, in the lip shape generation method of a virtual character provided in an embodiment of the present application, identifying emotional characteristic information of target audio information based on sound characteristic information and text information includes: inputting the sound characteristic information and text information into an emotion recognition model to obtain the emotional characteristic information of the target audio information, wherein the emotion recognition model is trained by multiple second sample data, and the second sample data includes sample sound characteristic information and sample text information, and corresponding emotional characteristic information.

[0053] It should be noted that after obtaining the sound feature information and text information of the target audio information, the emotional feature information of the target audio information can be determined based on the identified sound feature information and text information. The sound feature information and text information can be input into a pre-trained emotion recognition model to perform emotional feature extraction to obtain the emotional feature information of the target audio information, wherein the emotional feature information may include the emotional category and emotional level of the target audio information. The emotional category may include happy, sad, surprised, etc., and the emotional level may include mild, medium, strong, etc. For example, when the target audio information is welcoming you, the emotional category may represent happy, and the emotional level may be strong.

[0054] Specifically, the emotion recognition model can be obtained through machine learning training based on a large amount of second sample data, including sample sound feature information and sample text information, as well as corresponding emotion feature information. During the training process, the emotion recognition model can learn the correspondence between sound feature information, text information and emotion feature information. At the same time, the model performance can be evaluated through indicators such as accuracy and recall rate. Based on the evaluation results, the emotion recognition model can be adjusted and optimized, such as increasing the amount of data, adjusting hyperparameters, using more complex models, etc. Through continuous iteration and optimization, a good performance emotion recognition model is finally obtained. Through this emotion recognition model, the emotion feature information in the target audio information can be efficiently and accurately identified.

[0055] This embodiment is based on a large amount of sound feature information and text information, and uses a pre-trained emotion recognition model to identify the emotion feature information corresponding to the target audio information, thereby ensuring the accuracy of emotion feature recognition.

[0056] Optionally, in the method for generating lip movements of a virtual character provided in an embodiment of the present application, determining the lip movement animation of the virtual character based on sound feature information, text information and emotional feature information includes: generating an initial lip movement animation of the virtual character based on the text information; correcting the initial lip movement animation based on the sound feature information to obtain a candidate lip movement animation; and correcting the candidate lip movement animation based on the emotional feature information to obtain the lip movement animation of the virtual character.

[0057] It should be noted that after obtaining the emotional feature information of the target audio information, the lip animation of the virtual character can be comprehensively determined based on the sound feature information, text information and emotional feature information of the target audio information. First, the initial lip animation of the virtual character can be generated according to the text information corresponding to the target audio information, through an existing text and lip animation comparison library or a pre-trained neural network recognition model. Among them, the initial lip animation can only reflect the lip shape corresponding to the text, and cannot achieve good synchronization with the target audio information. The vividness of the virtual character's emotional expression is also relatively low.

[0058] Furthermore, the initial lip animation of the virtual character obtained above can be corrected according to the sound feature information of the target audio information, by obtaining the pitch, volume, length, rhythm, timbre, speed and other features of the target audio information, and integrating these feature information into the initial lip animation to obtain candidate lip animations, wherein the candidate lip animations can correspond to the text content of the target audio information and achieve basic synchronization with the target audio information, but the vividness of the virtual character's emotional expression is still poor.

[0059] Furthermore, the candidate lip animations of the virtual character obtained above can be corrected according to the emotional feature information of the target audio information, by obtaining the emotional features such as the emotional category and emotional level of the target audio information, and integrating the obtained emotional features with the candidate lip animations, and finally obtaining the lip animation of the virtual character. The obtained lip animation of the virtual character can not only accurately correspond to the text content of the target audio information, but also achieve good synchronization with the target audio information, and at the same time can make the emotional expression of the virtual character vivid, thereby improving the customer's perception and experience.

[0060] This embodiment corrects the initial lip animation generated by using text information through the sound feature information and emotion feature information of the target audio information, thereby improving the accuracy of the virtual character's lip animation and the vividness of the emotional expression.

[0061] Optionally, in the lip shape generation method for a virtual character provided in an embodiment of the present application, generating an initial lip shape animation of the virtual character based on text information includes: segmenting the text information to obtain multiple words, and obtaining the lip shape animation corresponding to each word in a lip shape animation library to obtain multiple first lip shape animations, and splicing the multiple first lip shape animations according to the order of the multiple words in the text information to obtain the initial lip shape animation of the virtual character; or, inputting the text information into a lip shape animation recognition model to obtain an initial lip shape animation, wherein the lip shape animation recognition model is obtained by training multiple third sample data, and the third sample data includes sample text information, and sample lip shape animation corresponding to the sample text information.

[0062] It should be noted that, according to the text information corresponding to the target audio information, the initial lip animation of the virtual character can be generated by two methods, namely, by using an existing lip animation library, or by using a pre-trained lip animation recognition model.

[0063] Specifically, in the process of generating the initial lip animation of a virtual character using an existing lip animation library, the text information recognized based on the target audio information can be first segmented. For example, for the text information "welcome your arrival", the following can be obtained through word segmentation: welcome, you, your, arrival. By segmenting the text information, it is convenient to individually recognize the lip animation of each word.

[0064] Furthermore, based on the multiple words obtained by word segmentation, the lip animation corresponding to each word can be obtained from an existing lip animation library that contains the correspondence between words and lip animations, to obtain multiple first lip animations, and then the first lip animation corresponding to each word can be spliced ​​according to the order of the words in the text information to obtain the initial lip animation of the virtual character, thereby ensuring the accurate correspondence between the lip animation and the text information.

[0065] Specifically, in the process of generating the initial lip animation of a virtual character using a lip animation recognition model, the lip animation recognition model may be a regression model or a neural network, etc., and may be trained using a large amount of third sample data, i.e., a large amount of sample text information, and sample lip animations corresponding to the sample text information. During the training process, the model performance may be evaluated using indicators such as accuracy, and then continuously adjusted and optimized to improve the accuracy of the lip animation recognition model.

[0066] Furthermore, after the model training is completed, the final lip animation recognition model can be obtained, which can complete the conversion from text information to lip animation. The text information corresponding to the target audio information can be input into the lip animation recognition model. Through the output results of the model, the initial lip animation of the virtual character can be obtained, ensuring the accurate correspondence between the lip animation and the text information.

[0067] This embodiment realizes the conversion from text information to lip animation by using the existing lip animation library or lip animation recognition model, thereby ensuring the accurate correspondence between the initial lip animation and the text information.

[0068] Optionally, in the lip shape generation method for a virtual character provided in an embodiment of the present application, the initial lip shape animation is corrected according to the sound feature information to obtain a candidate lip shape animation, including: determining a first feature of the lip shape animation according to the sound feature information, and identifying whether the first feature exists in the initial lip shape animation; when the first feature exists in the initial lip shape animation, determining the initial lip shape animation as a candidate lip shape animation; when the first feature does not exist in the initial lip shape animation, adjusting the initial lip shape animation according to the first feature to obtain a candidate lip shape animation; correcting the candidate lip shape animation according to the emotion feature information to obtain the lip shape animation of the virtual character includes: determining a second feature of the lip shape animation according to the emotion feature information, and adjusting the candidate lip shape animation according to the second feature to obtain the lip shape animation of the virtual character.

[0069] It should be noted that although the initial lip animation can reflect the lip shape corresponding to the text information, it cannot achieve good synchronization with the target audio information, and the vividness of the virtual character's emotional expression is also relatively low. Therefore, the initial lip animation can be corrected using sound feature information to obtain candidate lip animations, and then the candidate lip animations can be corrected using emotional feature information, and finally a lip animation of a virtual character with rich emotional expression can be obtained.

[0070] Specifically, in the process of correcting the initial lip animation using sound feature information, the first feature of the lip animation can be determined based on the sound feature information obtained above, that is, the degree of opening and closing of the lips, the position and angle of the corners of the mouth, the shape of the mouth, the speed of change of the mouth shape and other features of the lip animation can be determined through the sound features of the target audio information such as pitch, volume, length, rhythm, timbre, and speed of sound. The process of determining the first feature of the lip animation based on the sound feature information can be achieved through existing speech analysis tools or pre-trained models.

[0071] Furthermore, by identifying whether the aforementioned first feature exists in the initial lip animation, if the first feature exists in the initial lip animation, the initial lip animation can be determined as a candidate lip animation; if the first feature does not exist in the initial lip animation, the initial lip animation can be adjusted according to the first feature, and the initial lip animation can be merged with the obtained first feature to finally obtain a candidate lip animation.

[0072] Specifically, in the process of correcting the candidate lip animation using emotional feature information, the second feature of the lip animation can be first determined based on the emotional feature information obtained above, that is, the degree of opening and closing of the lips, the position and angle of the corners of the mouth, the shape of the mouth, the speed of change of the mouth shape and other features of the lip animation can be determined through emotional features such as the emotional type and emotional level of the target audio information. The process of determining the second feature of the lip animation based on the emotional feature information can also be achieved through existing speech analysis tools or pre-trained models.

[0073] Furthermore, by identifying whether the aforementioned second feature exists in the candidate lip-sync animation, if the second feature exists in the candidate lip-sync animation, the candidate lip-sync animation can be determined as the lip-sync animation of the virtual character; if the second feature does not exist in the candidate lip-sync animation, the candidate lip-sync animation can be adjusted according to the second feature, and the candidate lip-sync animation can be merged with the second feature to ultimately obtain a lip-sync animation of a virtual character with rich emotional expression.

[0074] This embodiment adjusts and corrects the initial lip animation by utilizing the sound feature information and emotion feature information of the target audio information, thereby improving the accuracy of the virtual character's lip animation and the vividness of the emotional expression.

[0075] Optionally, in the lip shape generation method for a virtual character provided in an embodiment of the present application, identifying emotional characteristic information of target audio information based on sound characteristic information and text information includes: determining whether there is contextual audio information for the target audio information; and if there is contextual audio information for the target audio information, identifying the emotional characteristic information of the target audio information based on the contextual audio information, sound characteristic information and text information.

[0076] It should be noted that, in the process of identifying the emotional characteristic information of the target audio information based on the sound characteristic information and the text information, the emotional characteristic information in the target audio information can also be determined in combination with the contextual audio information of the target audio information. First, it can be determined whether the target audio information has contextual audio information. For example, for an audio whose text content is "Hello, welcome to institution A, what service do you need?", when the text information corresponding to the target audio information is "Welcome to institution A", the target audio information has context, namely "Hello" and "What service do you need?"

[0077] Furthermore, when there is contextual audio information in the target audio information, the text information and sound feature information of the contextual audio information can be combined with the sound feature information and text information of the target audio information and input into the emotion recognition model to comprehensively determine the emotional feature information of the target audio information. For example, for "Hello, welcome to institution A, what service can you provide?", when determining the emotional feature information of "Welcome to institution A", in addition to its own text information and recognized sound feature information, the text information and sound feature information of "Hello" and "What service can you provide?" can also be combined to identify the emotional features.

[0078] This embodiment comprehensively determines the emotional feature information of the target audio information by combining the context audio information of the target audio information, thereby improving the accuracy of identifying the emotional features of the target audio information.

[0079] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0080] Example 2

[0081] The embodiment of the present application also provides a lip shape generation device for a virtual character. It should be noted that the lip shape generation device for a virtual character in the embodiment of the present application can be used to execute the lip shape generation method for a virtual character provided in the embodiment of the present application. The lip shape generation device for a virtual character provided in the embodiment of the present application is introduced below.

[0082] According to an embodiment of the present application, a device for implementing the above-mentioned method for generating lip shapes of a virtual character is also provided. Figure 3 is a schematic diagram of a lip shape generation device for a virtual character provided in Example 2 of the present application, such as Figure 3 As shown, the device comprises:

[0083] The acquisition unit 31 is used to acquire the target audio information broadcasted by the virtual character, and identify the target audio information to obtain the sound feature information and text information of the target audio information.

[0084] The recognition unit 32 is used to recognize the emotional characteristic information of the target audio information according to the sound characteristic information and the text information.

[0085] The determination unit 33 is used to determine the lip animation of the virtual character according to the sound feature information, text information and emotion feature information.

[0086] The lip shape generation device for a virtual character provided in the embodiment of the present application adopts an acquisition unit 31 to acquire target audio information broadcasted by the virtual character, and recognizes the target audio information to obtain sound feature information and text information of the target audio information; the recognition unit 32 recognizes the emotional feature information of the target audio information based on the sound feature information and the text information; the determination unit 33 determines the lip shape animation of the virtual character based on the sound feature information, the text information and the emotional feature information, and determines the emotional feature information of the audio by extracting the sound feature information and the text information from the audio information and using the emotion recognition model, and then comprehensively considers the sound feature information, the text information and the emotional feature information to obtain a lip shape animation with a higher accuracy, thereby solving the problem of low accuracy in generating the lip shape of a virtual character by recognizing phoneme information in the related art, thereby achieving the technical effect of improving the vividness of the emotional expression of the virtual character and ensuring the accuracy of the lip shape generation of the virtual character.

[0087] Optionally, in the lip shape generation device for a virtual character provided in an embodiment of the present application, the acquisition unit 31 includes: a processing module, used to pre-process the target audio information to obtain processed audio information; a first input module, used to input the processed audio information into a text recognition model to obtain text information, wherein the text recognition model is obtained by training a plurality of first sample data, and the first sample data includes sample audio information and sample text information corresponding to the sample audio information; an extraction module, used to perform feature extraction on the processed audio information to obtain sound feature information, wherein the sound feature information includes at least one of the following: pitch, sound intensity, and sound length.

[0088] Optionally, in the lip shape generation device for a virtual character provided in an embodiment of the present application, the recognition unit 32 includes: a second input module, used to input sound feature information and text information into an emotion recognition model to obtain emotion feature information of the target audio information, wherein the emotion recognition model is trained by multiple second sample data, and the second sample data includes sample sound feature information and sample text information, as well as corresponding emotion feature information.

[0089] Optionally, in the lip shape generation device for a virtual character provided in an embodiment of the present application, the determination unit 33 includes: a generation module, used to generate an initial lip shape animation of the virtual character based on text information; a first correction module, used to correct the initial lip shape animation based on sound feature information to obtain a candidate lip shape animation; a second correction module, used to correct the candidate lip shape animation based on emotion feature information to obtain the lip shape animation of the virtual character.

[0090] Optionally, in the lip shape generation device for a virtual character provided in an embodiment of the present application, a generation module includes: a word segmentation submodule, used to segment text information to obtain multiple words, and obtain the lip shape animation corresponding to each word in a lip shape animation library to obtain multiple first lip shape animations, and splice the multiple first lip shape animations according to the order of the multiple words in the text information to obtain the initial lip shape animation of the virtual character; or, an input submodule, used to input the text information into a lip shape animation recognition model to obtain an initial lip shape animation, wherein the lip shape animation recognition model is obtained by training multiple third sample data, and the third sample data includes sample text information, and sample lip shape animation corresponding to the sample text information.

[0091] Optionally, in the lip shape generation device for a virtual character provided in an embodiment of the present application, the first correction module includes: a first determination submodule, used to determine a first feature of a lip shape animation according to sound feature information, and identify whether the first feature exists in an initial lip shape animation; a second determination submodule, used to determine the initial lip shape animation as a candidate lip shape animation when the first feature exists in the initial lip shape animation; an adjustment submodule, used to adjust the initial lip shape animation according to the first feature to obtain a candidate lip shape animation when the first feature does not exist in the initial lip shape animation; the second correction module includes: a third determination submodule, used to determine a second feature of the lip shape animation according to emotion feature information, and adjust the candidate lip shape animation according to the second feature to obtain the lip shape animation of the virtual character.

[0092] Optionally, in the lip shape generation device for a virtual character provided in an embodiment of the present application, the recognition unit 32 includes: a judgment module for judging whether there is contextual audio information for the target audio information; and a recognition module for identifying, when there is contextual audio information for the target audio information, emotional feature information of the target audio information based on the contextual audio information, sound feature information and text information.

[0093] It should be noted that the acquisition unit 31, identification unit 32, and determination unit 33 correspond to steps S201 to S203 in Example 1, and the examples and application scenarios implemented by the acquisition unit 31, identification unit 32, and determination unit 33 are the same as those of the corresponding steps, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0094] Example 3

[0095] An embodiment of the present application may provide an electronic device, Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0096] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the lip shape generation method and device of the virtual character in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the lip shape generation method of the virtual character mentioned above is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0097] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain the target audio information broadcast by the virtual character, and identify the target audio information to obtain the sound feature information and text information of the target audio information; identify the emotional feature information of the target audio information based on the sound feature information and text information; determine the lip animation of the virtual character based on the sound feature information, text information and emotional feature information.

[0098] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: pre-process the target audio information to obtain processed audio information; input the processed audio information into a text recognition model to obtain text information, wherein the text recognition model is obtained by training multiple first sample data, and the first sample data includes sample audio information and sample text information corresponding to the sample audio information; extract features from the processed audio information to obtain sound feature information, wherein the sound feature information includes at least one of the following: pitch, sound intensity, and sound length.

[0099] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: input the sound feature information and text information into the emotion recognition model to obtain the emotion feature information of the target audio information, wherein the emotion recognition model is trained by multiple second sample data, and the second sample data includes sample sound feature information and sample text information, as well as corresponding emotion feature information.

[0100] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: generate an initial lip animation of the virtual character based on the text information; modify the initial lip animation based on the sound feature information to obtain a candidate lip animation; modify the candidate lip animation based on the emotion feature information to obtain the lip animation of the virtual character.

[0101] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: segment the text information to obtain multiple words, and obtain the lip animation corresponding to each word in the lip animation library to obtain multiple first lip animations, and splice the multiple first lip animations according to the order of the multiple words in the text information to obtain the initial lip animation of the virtual character; or, input the text information into a lip animation recognition model to obtain an initial lip animation, wherein the lip animation recognition model is obtained by training multiple third sample data, and the third sample data includes sample text information and sample lip animation corresponding to the sample text information.

[0102] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine the first feature of the lip animation based on the sound feature information, and identify whether the first feature exists in the initial lip animation; when the first feature exists in the initial lip animation, determine the initial lip animation as a candidate lip animation; when the first feature does not exist in the initial lip animation, adjust the initial lip animation according to the first feature to obtain a candidate lip animation; determine the second feature of the lip animation based on the emotional feature information, and adjust the candidate lip animation according to the second feature to obtain the lip animation of the virtual character.

[0103] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine whether there is contextual audio information in the target audio information; if there is contextual audio information in the target audio information, identify the emotional feature information of the target audio information based on the contextual audio information, sound feature information and text information.

[0104] It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 4 The structure of the electronic device is not limited. Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 4 Different configurations shown.

[0105] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0106] Example 4

[0107] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for generating the lip shape of a virtual character provided in the first embodiment.

[0108] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0109] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the method for generating lip shapes of a virtual character.

[0110] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0111] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0116] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating lip shapes of a virtual character, characterized in that: include: Acquire target audio information broadcast by a virtual character, and identify the target audio information to obtain sound feature information and text information of the target audio information; Identifying emotional feature information of the target audio information based on the sound feature information and the text information; The lip animation of the virtual character is determined according to the sound feature information, the text information and the emotion feature information.

2. The method according to claim 1, characterized in that Identifying the target audio information to obtain sound feature information and text information of the target audio information includes: Preprocessing the target audio information to obtain processed audio information; Inputting the processed audio information into a text recognition model to obtain the text information, wherein the text recognition model is trained by a plurality of first sample data, the first sample data including sample audio information and sample text information corresponding to the sample audio information; The processed audio information is subjected to feature extraction to obtain the sound feature information, wherein the sound feature information includes at least one of the following: pitch, intensity, and duration.

3. The method according to claim 1, characterized in that Identifying the emotional feature information of the target audio information according to the sound feature information and the text information includes: The sound feature information and the text information are input into an emotion recognition model to obtain the emotion feature information of the target audio information, wherein the emotion recognition model is obtained by training a plurality of second sample data, and the second sample data includes sample sound feature information and sample text information, as well as corresponding emotion feature information.

4. The method according to claim 1, wherein Determining the lip animation of the virtual character according to the sound feature information, the text information, and the emotion feature information includes: generating an initial lip animation of the virtual character according to the text information; Modifying the initial lip animation according to the sound feature information to obtain a candidate lip animation; The candidate lip animation is modified according to the emotional feature information to obtain the lip animation of the virtual character.

5. The method according to claim 4, characterized in that Generating the initial lip animation of the virtual character according to the text information includes: Segmenting the text information to obtain a plurality of words, obtaining a lip animation corresponding to each word from a lip animation library to obtain a plurality of first lip animations, and splicing the plurality of first lip animations in the order of the plurality of words in the text information to obtain an initial lip animation of the virtual character; or The text information is input into a lip animation recognition model to obtain the initial lip animation, wherein the lip animation recognition model is trained by a plurality of third sample data, wherein the third sample data includes sample text information and sample lip animation corresponding to the sample text information.

6. The method according to claim 4, characterized in that Modifying the initial lip animation according to the sound feature information to obtain a candidate lip animation includes: determining a first feature of the lip animation according to the sound feature information, and identifying whether the first feature exists in the initial lip animation; If the first feature exists in the initial lip animation, determining the initial lip animation as the candidate lip animation; When the first feature does not exist in the initial lip animation, adjusting the initial lip animation according to the first feature to obtain the candidate lip animation; Modifying the candidate lip animation according to the emotional feature information to obtain the lip animation of the virtual character includes: A second feature of the lip animation is determined according to the emotional feature information, and the candidate lip animation is adjusted according to the second feature to obtain the lip animation of the virtual character.

7. The method according to claim 1, characterized in that Identifying the emotional feature information of the target audio information according to the sound feature information and the text information includes: Determining whether the target audio information has contextual audio information; In a case where the target audio information contains the context audio information, the emotional feature information of the target audio information is identified according to the context audio information, the sound feature information and the text information.

8. A device for generating lip shapes of a virtual character, characterized in that: include: an acquiring unit, configured to acquire target audio information broadcasted by a virtual character, and identify the target audio information to obtain sound feature information and text information of the target audio information; an identification unit, configured to identify emotional characteristic information of the target audio information based on the sound characteristic information and the text information; A determination unit is used to determine the lip animation of the virtual character based on the sound feature information, the text information and the emotion feature information.

9. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method for generating the lip shape of a virtual character according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory storing an executable program; A processor is used to run the program, wherein when the program is run, the method for generating the lip shape of a virtual character according to any one of claims 1 to 7 is executed.

Images