Emotion expression method, system and equipment for voice-driven 3D virtual human

Abstract

The invention relates to the technical field of 3D virtual people, and provides an emotion expression method, system and device of a voice-driven 3D virtual human, which are used for improving the emotion expression precision of the 3D virtual human. According to the method, driving parameters of all expression bases are obtained from audio clips obtained after target audio is segmented, corresponding target emotions are extracted according to keyword sets extracted from text paragraphs obtained from the target audio by adopting an emotion extraction model, and then driving parameters of core expression bases corresponding to the target emotions are corrected, so that the target emotions are obtained. The speech content and context of the 3D virtual human are matched with emotions, self-adaptive dynamic adjustment of the emotions is achieved, the naturalness and authenticity of voice driving are improved, an emotion extraction model based on machine learning training has certain learning ability and generalization, and for keywords which are not stored in a database, the speech recognition efficiency is improved. Under the condition that the complexity of the model is not increased, the corresponding emotion can still be accurately extracted, so that the emotion expression precision of the 3D virtual human is improved.

Description

Background Art

[0002] Compared with 2D virtual humans, 3D virtual humans can bring a stronger sense of reality and immersion to the audience, so they have more application fields and scopes, such as VR games, 3D animation, etc. At the same time, the production of 3D virtual humans requires higher technology, and the driving principle of 3D virtual humans is completely different from that of 2D virtual humans. Among them, 2D virtual humans are driven by images, and 3D virtual humans are driven by three-dimensional models. Therefore, the objects driven by the two and the driving methods are different. In this way, the real-time emotional expression method based on text in the field of voice drive of 2D virtual humans cannot be applied to the emotional control of 3D virtual humans when voice is driven.

[0003] At present, in the field of voice-driven 3D virtual humans, the focus is only on the mouth of the 3D virtual human. Usually, the voice is matched with the mouth shape of the 3D virtual human, and some random parameters such as blinking and head posture are added, while the emotional expression of the 3D virtual human is ignored, making the driven 3D virtual human not natural and real enough, and even looking dull and sluggish. Summary of the invention

[0004] The embodiments of the present application provide a method, system and device for expressing emotions of a 3D virtual human driven by voice, which are used to improve the accuracy of the emotional expression of a 3D virtual human.

[0005] On the one hand, an embodiment of the present application provides a method for expressing emotions of a 3D virtual human driven by voice, comprising:

[0006] Get the target audio of the 3D virtual person;

[0007] Segmenting the target audio according to a preset frame rate, and determining first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment;

[0008] Convert the target audio into a target text, and segment the target text;

[0009] For each text paragraph, extract keywords related to emotions in the text paragraph to obtain a keyword set, and input the keyword set into a pre-trained emotion extraction model; wherein the emotion extraction model is trained based on a database storing a plurality of pairs of corresponding relationships between emotions and keywords, and each emotion corresponds to an expression base set;

[0010] Using the emotion extraction model, obtaining the target emotion corresponding to the 3D virtual person in the text paragraph, wherein the target emotion has or does not have a corresponding keyword in the database;

[0011] According to the target emotion, adjusting the first driving parameters of each core expression base in the corresponding expression base set related to the target emotion to obtain a second driving parameter;

[0012] According to the second driving parameters of each core expression base and the first driving parameters of other expression bases, a 3D virtual human with emotional expression is obtained and displayed.

[0013] On the other hand, an embodiment of the present application provides a voice-driven 3D virtual human emotion expression system, including:

[0014] An audio acquisition module is used to acquire the target audio of the 3D virtual human;

[0015] An audio segmentation module, used to segment the target audio according to a preset frame rate;

[0016] An expression base parameter estimation module, used for determining first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment;

[0017] A text conversion module, used for converting the target audio into a target text;

[0018] A segmentation and word segmentation module is used to segment the target text, extract emotion-related keywords from each text paragraph, obtain a keyword set, and input the keyword set into a pre-trained emotion extraction model; wherein the emotion extraction model is trained based on a database storing a plurality of pairs of corresponding relationships between emotions and keywords, and each emotion corresponds to an expression base set;

[0019] An emotion extraction module, used to use the emotion extraction model to obtain the target emotion corresponding to the 3D virtual person in the text paragraph, wherein the target emotion has or does not have a corresponding keyword in the database;

[0020] An expression base parameter correction module, used for adjusting the first driving parameters of each core expression base related to the target emotion in the corresponding expression base set according to the target emotion to obtain a second driving parameter;

[0021] The expression base driving module is used to obtain and display a 3D virtual human with emotional expression according to the second driving parameters of each core expression base and the first driving parameters of other expression bases.

[0022] Optionally, the emotion extraction module is specifically used for:

[0023] For each keyword in the keyword set, perform the following operations:

[0024] Using the emotion extraction model to extract the emotion features of the keywords;

[0025] If the keyword exists in the database for training the emotion extraction model, the probability that the keyword corresponds to various emotions in the database is calculated according to the emotion feature, and the emotion corresponding to the highest probability among the probabilities is used as the candidate emotion corresponding to the keyword;

[0026] If the keyword does not exist in the database for training the emotion extraction model, predicting various emotions corresponding to the keyword according to the emotion feature, and determining candidate emotions corresponding to the keyword according to the comparison result of the probability of the various emotions and the preset emotion reaching threshold;

[0027] A target emotion corresponding to the 3D virtual person in the text paragraph is determined from candidate emotions corresponding to each keyword in the keyword set.

[0028] Optionally, the emotion extraction module is specifically used for:

[0029] Compare the probabilities of the various emotions with the preset emotion reaching thresholds respectively;

[0030] If the probabilities of the various emotions are all less than the preset emotion reaching threshold, the keyword is eliminated; otherwise, the emotion corresponding to the maximum probability greater than or equal to the preset emotion reaching threshold is used as the candidate emotion of the keyword.

[0031] Optionally, the emotion extraction module is further used to:

[0032] Counting the number of correspondences between the keyword and the emotion represented by the maximum probability greater than or equal to the preset emotion reaching threshold;

[0033] If the number of correspondences exceeds a preset number threshold, the correspondence between the keyword and the emotion represented by the maximum probability is stored in the database to update the database.

[0034] Optionally, the system further includes a model training module, which is used to:

[0035] Counting the number of updates of the database;

[0036] If the number of updates reaches a preset update threshold, the emotion extraction model is retrained according to the updated database.

[0037] Optionally, the expression base parameter correction module is specifically used for:

[0038] Adding a control coefficient to the intensity of the target emotion, wherein the intensity is determined based on the frequency of occurrence of the target emotion in various candidate emotions corresponding to the keyword set;

[0039] For each core expression base, respectively determining a first influence value of the control coefficient on the intensity of the target emotion and a second influence value of the control coefficient on the first driving parameter of the core expression base;

[0040] The first influence value and the second influence value are fused to obtain a second driving parameter of the core expression base.

[0041] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute any voice-driven 3D virtual human emotion expression method provided in the embodiment of the present application.

[0042] The beneficial effects of the method, system and device for expressing emotions of a voice-driven 3D virtual human provided by the embodiments of the present application are as follows:

[0043] The first driving parameters of all expression bases of the 3D virtual human are determined by dividing each audio segment according to a preset frame rate, so that the 3D virtual human can be animated according to the preset frame rate. At the same time, in order to improve the accuracy of the 3D virtual human driving during the animation process, the target text after the target audio is converted is segmented, and for each text paragraph, a set of keywords expressing emotions is extracted, so that the target emotion expressed by the 3D virtual human in each text paragraph is determined by using the keyword set, so that the target emotion is adapted to the lip shape, and the authenticity of the voice driving is improved. In addition, considering that the categories of emotions are limited and the keywords expressing emotions are relatively wide, it is difficult for the preset database to cover all the keywords. Therefore, when determining the target emotion of the text paragraph, an emotion extraction model trained with a database containing multiple pairs of keyword and emotion correspondences is used to output the target emotion. Since the emotion extraction model can learn the correspondence between keywords and emotions, for keywords not stored in the database, the corresponding emotions can still be accurately input, thereby improving the accuracy of emotional expression. On the other hand, since each emotion in the database corresponds to a set of core expression bases, the driving parameters of each core expression base determined by the target emotional voice can be used to make corrections, making the control of the target emotion on the 3D virtual human more delicate and realistic, further improving the authenticity of voice driving.

[0044] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A schematic diagram of a method for driving expression of a 3D virtual human in the prior art provided in an embodiment of the present application;

[0047] Figure 2 The expression effect diagrams of two 3D virtual humans with different emotions provided in the embodiments of the present application;

[0048] Figure 3 The overall framework diagram of the emotional expression of a voice-driven 3D virtual human provided in the embodiment of the present application;

[0049] Figure 4 A flow chart of a method for expressing emotions of a 3D virtual human driven by voice provided in an embodiment of the present application;

[0050] Figure 5 52 basic expression diagrams of a 3D virtual human provided in an embodiment of the present application;

[0051] Figure 6 A flow chart of an emotion acquisition method based on an emotion extraction model provided in an embodiment of the present application;

[0052] Figure 7 A flow chart of a method for correcting expression base parameters provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of expression-based weighting provided in an embodiment of the present application;

[0054] Fig. 9 A schematic diagram of a model of different expressions provided in an embodiment of the present application;

[0055] Fig.10 A structural diagram of a voice-driven 3D virtual human emotion expression system provided in an embodiment of the present application;

[0056] Fig.11 A structural diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0058] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0059] It should be understood that the terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0060] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0061] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0062] The following is an overview of the design concept of the embodiments of the present application in conjunction with application scenarios.

[0063] The human face is an important part of human emotion expression. In the three-dimensional virtual world, the virtual face is the basis of 3D virtual human expression. The purpose of the research on 3D virtual human expression is to make 3D virtual human appear like a real person. With the development of computer vision and computer graphics, voice-driven technology has given 3D virtual human the ability to speak like a real person, making 3D virtual human widely used in various scenarios such as live broadcast, short video generation, digital cloning, customer service, education, finance, video conferencing, etc.

[0064] At present, the voice-driven technology of 3D virtual humans has achieved high-precision matching of audio and mouth shape, achieving a lip synchronization effect close to that of a real person. However, when a 3D virtual human speaks, only the mouth moves, which appears stiff and unnatural. Therefore, some technologies have added a small number of movements to the 3D virtual human, such as blinking and head shaking, to make the 3D virtual human look more realistic when speaking.

[0065] In practical applications, the emotions expressed by people when speaking (such as anger, happiness, surprise, disgust, sadness, fear, etc.) are an important part of the human face. Different voices often contain different emotions, and people's facial expressions will be very different under different emotions. At present, most voice-driven solutions for 3D virtual humans do not have emotional expression, that is, the 3D virtual human defaults to a natural expression. The lack of emotion makes the 3D virtual human seem to lack a "soul", and the effect presented is very different from that of a real person and is not natural enough.

[0066] In order to improve the authenticity of 3D virtual human voice driving, some technologies specify emotions for 3D virtual humans by inputting them in advance, such as happiness, anger, excitement, etc., and then use them together with audio as input to estimate the expression base parameters, obtain the driving parameters of the expression base, and thus obtain the voice driving effect of the 3D virtual human with the specified emotions, such as Figure 1 As shown, the input method of the emotion includes but is not limited to text, code, reference video and other forms.

[0067] In practical applications, 3D virtual humans may need different emotional expressions in different contexts and scenarios, such as Figure 2 The figure shows the expression of two emotions: happy and angry. However, this method can only set fixed emotions for a certain audio segment, and cannot adaptively change according to the context and voice content of the 3D virtual person. It has a low level of intelligence and cannot meet the needs of the 3D virtual person for different emotional expressions in different voice content or contexts during real-time conversation or driving. Especially in the long audio driving scenario, a segment of audio may contain multiple emotions. This method is prone to the problem of contradiction between the spoken content and the expressed emotions, which seriously affects the driving effect of the 3D virtual person. Therefore, the authenticity and naturalness of the 3D virtual person's speech still needs to be improved.

[0068] For example, in a depressed passage, the 3D virtual person still uses the input happy emotion, or, in a joyful passage, the 3D virtual person continues to maintain a depressed emotion. Due to the incompatibility of emotion and context, the presentation effect of the 3D virtual person will be affected, which will cause a great sense of disharmony to the audience.

[0069] In view of this, the embodiment of the present application provides a method for expressing emotions of a 3D virtual person driven by speech. In the process of voice driving the 3D virtual person, the method considers that the input target audio may be long and the emotions of the 3D virtual person may change. Therefore, the target audio is converted into a target text and then segmented so that each text paragraph can express an emotion, thereby improving the accuracy of the emotion expression. At the same time, a database containing the correspondence between keywords and emotions is used to train the emotion extraction model, so that the emotion extraction model can fully learn the correspondence between keywords and emotions, thereby improving the generalization of the model. In this way, for the keywords in the keyword set extracted from each text paragraph that are not stored in the database, the emotion extraction model can still output the corresponding emotions, thereby further improving the accuracy of the emotion expression of the 3D virtual person. After determining the target emotion corresponding to each text paragraph, the expression of the target emotion is displayed by controlling the movement of expression bases such as the mouth, cheeks, eyes, eyebrows, eyeballs, head, and chin, and is fused with the expression base determined by the speech in real time, thereby realizing the dynamic change of the emotion following the speech adaptation in the process of voice driving the 3D virtual person, thereby improving the authenticity and naturalness of the speech driving of the 3D virtual person. This method only needs to input a piece of audio during the entire voice-driven process, without the need for additional references such as images or videos. It realizes end-to-end adaptive dynamic emotion 3D virtual human voice driving, improves the coordination and consistency of mouth movements with other facial position movements and head movements, makes the 3D virtual human look more natural, and does not require subsequent processing of emotions. It uses fewer resources and can meet the needs of real-time voice-driven scenarios.

[0070] In the embodiments of the present application, the head of the 3D virtual human is constructed by a parametric head model (such as Skinned Multi-Person Linear, referred to as SLMP), which is composed of shape parameters, posture parameters and expression parameters. The shape parameters of the same person remain basically unchanged, the posture parameters mainly control the rigid movement of the head, and the expression parameters mainly control facial changes. The parameterized 3D virtual human contains multiple expression bases. Therefore, the changes in the face or head posture of the 3D virtual human can be driven by controlling the driving parameters of the expression bases. Since the embodiments of the present application mainly study the expression of facial emotions when driven by voice, and reasonable expression parameters can enable the 3D virtual human to show different emotions through changes in the eyes, mouth, mandible, eyebrows, cheeks, etc., therefore, the driving parameters of the embodiments of the present application mainly refer to the expression parameters that can drive the face and the posture parameters that can drive the head.

[0071] The overall framework diagram of a voice-driven 3D virtual human emotion expression method provided in an embodiment of the present application is as follows: Figure 3As shown, it mainly includes three parts: voice-driven 3D virtual human emotion adaptive adjustment, 3D virtual human driving and 3D virtual human display. Among them, the driving part of the 3D virtual human is mainly the driving of the head and face, and the changes of the facial expressions and head postures of the 3D virtual human are realized by weighting the driving parameters of each expression base; the display part of the 3D virtual human is to present the 3D virtual human to the user through the screen; the voice-driven 3D virtual human emotion adaptive adjustment part is the key content of the embodiment of this application, including audio input, audio segmentation, text conversion, segmentation and word segmentation, emotion extraction model training based on machine learning, emotion extraction, expression base parameter estimation, expression base parameter correction and expression base parameter output and other processes.

[0072] The emotion extraction model is trained based on a pre-built database that stores the correspondence between keywords and emotions, as well as the core expression base set corresponding to each emotion. Through machine learning of the correspondence, the emotion extraction model can obtain the emotions corresponding to the keywords stored in the database and the emotions corresponding to the keywords not stored in the database, with high generalization, thereby improving the accuracy of 3D virtual human emotion expression.

[0073] See also Figure 4 , which is a flow chart of the method for expressing emotions of a 3D virtual human driven by voice provided in an embodiment of the present application, mainly includes the following steps:

[0074] S401: Acquire target audio of the 3D virtual human.

[0075] In an optional embodiment, the target audio associated with the 3D virtual person can be obtained from the outside or locally, including but not limited to any one of speech recordings, audio obtained using a text-to-speech tool, and text-converted audio automatically generated based on a language model (such as an intelligent question-and-answer scenario).

[0076] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the language type of the target audio, which can be Chinese, English, Japanese, etc.

[0077] S402: Segment the target audio according to a preset frame rate, and determine first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment.

[0078] The target audio itself is segmented into multiple audio segments according to a preset frame rate, and each audio segment is used to extract expression-based driving parameters, so that each audio segment drives the 3D virtual human to animate at a preset frame rate.

[0079] In the specific implementation, for each audio clip, the audio clip is input into a pre-trained parameter generation network to extract speech features, and the first driving parameters of all expression bases of the 3D virtual person corresponding to the audio clip are output according to the extracted speech features. Among them, all expression bases include expression bases for controlling head posture and expression bases for controlling facial expressions. Therefore, the first driving parameters can control the movements of the mouth, cheeks, eyes, eyebrows, eyeballs, head, chin, etc. of the 3D virtual person's face, and can also control the swing, rotation, pitch and other movements of the 3D virtual person's head, so as to achieve synchronization between the input voice and the lip shape of the 3D virtual person, realize the natural movement of the face and head, and improve the authenticity of voice driving.

[0080] Take the example of using ARKit 52 expression bases to control the facial expressions of 3D virtual people. The information of these 52 expression bases is as follows: Figure 5 shown.

[0081] Generally, the preset frame rate of the target audio is greater than 25 frames per second. Optionally, the embodiment of the present application is set to 30 frames per second. The size can be adjusted according to actual needs, and the embodiment of the present application does not make any restrictive requirements.

[0082] S403: Convert the target audio into a target text, and segment the target text.

[0083] In an optional implementation, a speech-to-text tool is used to convert the complete target audio into a target text, so as to extract keywords expressing emotions from the table text.

[0084] In actual applications, when the input target audio is long, the emotions of the 3D virtual human may change during the voice-driven process. In order to achieve adaptive adjustment of the emotions of the 3D virtual human with the voice content, the target text can be divided into multiple text paragraphs, so that the emotions of the 3D virtual human can be extracted in units of text paragraphs, making the emotion control more precise.

[0085] In specific implementation, the target text can be segmented using periods as markers, with each sentence being a text paragraph. In this way, the 3D virtual human can flexibly express emotions through each sentence, thereby improving the accuracy of adaptive emotional expression.

[0086] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the segmentation method of the target text. For example, line breaks can be used for segmentation, or audio clips can be used for segmentation.

[0087] S404: For each text paragraph, extract keywords related to emotions in the text paragraph, obtain a keyword set, and input the keyword set into a pre-trained emotion extraction model.

[0088] In an optional implementation, for each text paragraph, the text paragraph is segmented according to certain semantics to obtain a set of multiple words and phrases that represent emotions. Considering that the semantics expressed by a single word vary greatly, and most words do not have clear meanings and cannot accurately express real emotions, for example, the semantics expressed by a single word "smile" can be a real smile, a fake smile, or a bad smile. Therefore, the embodiment of the present application removes the single word extracted from the text paragraph and uses the remaining multiple phrases to generate a keyword set, recorded as Among them, N v The total number of keywords.

[0089] In the embodiment of the present application, a database is pre-constructed, which stores the correspondence between multiple pairs of keywords and emotions, and each emotion corresponds to an expression base set, and the format is as follows: {keyword-emotion-(expression base 1, expression base 2, ...)}. Among them, the emotions corresponding to different keywords are the same or different.

[0090] For example, the keywords "heart bursting with joy" and "feeling relaxed and happy" correspond to the emotion "happy", and the keywords "furious" and "discouraged" correspond to the emotions "angry" and "depressed", respectively.

[0091] The expression base set corresponding to each emotion includes all expression bases of the 3D virtual human head, but the core expression bases related to the emotion only account for a part. Therefore, other expression bases in the expression base set that control the head posture can be represented by 0.

[0092] In the database of the embodiment of the present application, different emotions are associated with specific core expression bases through an expression base set. In this way, the expression of emotions can be displayed by controlling the movements of core expression bases such as the mouth, cheeks, eyes, eyebrows, eyeballs, head, and chin, thereby achieving the effect of adjusting the emotional expression of the 3D virtual human by adjusting the driving parameters of the core expression base, making the emotional expression of the 3D virtual human more delicate and realistic.

[0093] Furthermore, based on the constructed database, the emotion extraction model can be trained by using a machine learning algorithm. During the training process, the machine learning algorithm gives the emotion extraction model a certain learning ability by learning the corresponding relationship between keywords and emotions, and can output the corresponding emotions according to the input keywords.

[0094] In the embodiments of the present application, since the types of human emotions are limited, such as natural, happy, angry, sad, frustrated, excited, fearful, surprised, and pleasant, these emotions can cover most of the emotions of 3D virtual humans, which are recorded as emotion set E. By storing these emotions in a database, the 3D virtual human can adaptively and dynamically adjust the emotions without making the types of emotions of the 3D virtual human too complicated, which helps to improve the expression of the emotions of the 3D virtual human.

[0095] It should be noted that the embodiments of the present application do not impose restrictive requirements on machine learning algorithms, for example, including but not limited to random forest algorithms, support vector machine algorithms, logistic regression algorithms, etc.

[0096] S405: Using an emotion extraction model, obtaining a target emotion corresponding to the 3D virtual human in the text paragraph.

[0097] The same meaning can be expressed using different phrases, such as cheerful, joyful, and jubilant, all of which represent happy emotions. In this way, the constructed database may not cover all emotion-related keywords. However, since the emotion extraction model has acquired a certain learning ability and generalization after training, the emotion extraction model can accurately output the emotion corresponding to the keyword in the keyword set input into the emotion extraction model, regardless of whether the keyword has a corresponding emotion in the database.

[0098] In the specific implementation, for each text paragraph, the process of using the emotion extraction model to determine the target emotion of the 3D virtual human is as follows: Figure 6 As shown, it mainly includes the following steps:

[0099] S4051: For each keyword in the keyword set corresponding to the text paragraph, the emotion extraction model is used to extract the emotion feature of the keyword.

[0100] S4052: Determine whether the keyword exists in the database for training the emotion extraction model. If so, execute S4053; otherwise, execute S4054.

[0101] Since the same semantics can be expressed using different phrases, the keywords extracted from the text paragraph may exist in the database used to train the sentiment extraction model, denoted by v in , may not exist in the database used to train the emotion extraction model, denoted as v out .

[0102] S4053: Calculate the probability of the keyword corresponding to various emotions in the database according to the emotion characteristics, and take the emotion corresponding to the highest probability among the probabilities as the candidate emotion corresponding to the keyword.

[0103] Assume that all keywords stored in the database are denoted as s known , when the keyword exists in the database, that is, v in ∈s known , there must be an emotion in the database that corresponds to the keyword. In this way, among the various probabilities output by the model, the probability of the emotion corresponding to the keyword is usually higher. Therefore, the emotion corresponding to the highest probability among the various probabilities output by the model can be directly used as the candidate emotion corresponding to the keyword. The formula is expressed as:

[0104]

[0105] Among them, p mαx =max(p), p=(..., p e (v in ), ..), e∈E, represents an emotion in the emotion set E, p e (v in ) indicates the keyword v in The probability corresponding to emotion e, φ is the function of finding the index corresponding to the maximum value, that is, the output is the emotion corresponding to the maximum probability.

[0106] S4054: Predict various emotions corresponding to the keywords according to the emotion characteristics, and determine candidate emotions corresponding to the keywords according to the comparison results of the probabilities of various emotions and the preset emotion reaching thresholds.

[0107] When the keyword does not exist in the database, If it is not possible to determine whether the corresponding emotion exists in the database, you can set an emotion threshold to determine whether the corresponding emotion exists. The specific judgment process is as follows:

[0108] S4054_1: Determine whether the probability of each emotion determined by the emotion feature is less than a preset emotion reaching threshold. If so, execute S40542; otherwise, execute S40543.

[0109] Since the emotion set E in the database contains most of the emotions of the 3D virtual human, and the trained emotion extraction model has certain learning ability and generalization, after the keyword that does not exist in the database is input into the emotion extraction model, the emotion extraction model can predict the probability that the keyword belongs to various emotions in the emotion set E based on the extracted emotion features. The formula is expressed as:

[0110] p=(..., p e (v out ),...) Formula 2

[0111] Among them, e∈E represents an emotion in the emotion set E, p e (v out) indicates the keyword v out The probability corresponding to emotion e, φ is the function of finding the index corresponding to the maximum value, that is, the output is the emotion corresponding to the maximum probability.

[0112] However, since the keyword does not exist in the database, it may not correspond to any emotion in the emotion set E, resulting in smaller probabilities output by the emotion extraction model. Therefore, it is necessary to compare each probability with the preset emotion threshold.

[0113] S4054_2: Remove the keyword.

[0114] When the probabilities of various emotions output by the emotion extraction model are all less than the preset emotion threshold, it indicates that the keyword does not correspond to any emotion, and the result of the keyword output by the emotion extraction model is invalid. Therefore, the keyword can be eliminated, that is, the keyword is not used as the emotional expression of the text paragraph.

[0115] S4054_3: The emotion corresponding to the maximum probability that is greater than or equal to the preset emotion reaching threshold is used as the candidate emotion of the keyword.

[0116] When the probability of various emotions output by the emotion extraction model is greater than or equal to the preset emotion reaching threshold, it indicates that the emotion with the highest probability corresponds to the keyword, and the keyword and the emotion and probability it expresses are recorded.

[0117] Specifically, for keywords that do not exist in the database, the sentiment determination formula is as follows:

[0118]

[0119] Among them, ζ represents the preset emotion reaching threshold.

[0120] In the embodiment of the present application, by inputting the keyword set extracted from each text paragraph into the emotion extraction model, the emotion stored in the database corresponding to each keyword can be obtained, thereby realizing the emotion extraction of the input text. Compared with directly querying the correspondence between each keyword and emotion from the database, this method can not only accurately extract emotions through learning, but also has a certain degree of generalization, and can infer the corresponding emotions for keywords not stored in the database, and has stronger versatility.

[0121] S4055: Determine the target emotion corresponding to the 3D virtual person in the text paragraph from the candidate emotions corresponding to each keyword in the keyword set.

[0122] After traversing each keyword in the keyword set corresponding to the text paragraph, multiple candidate emotions corresponding to the text paragraph can be obtained, which are recorded as set s EBy counting the frequency of occurrence of each candidate emotion, the intensity of each candidate emotion can be obtained, and then a target emotion can be selected from multiple candidate emotions according to the intensity. The specific process is as follows:

[0123] S4055_1: Count the frequency of occurrence of each candidate emotion among the obtained candidate emotions, and determine the strength of the corresponding candidate program according to each frequency.

[0124] For this text paragraph, the higher the frequency of each emotion, the more the emotion can represent the context of the text paragraph. Therefore, the candidate emotion set s can be counted. E The frequency of each candidate emotion in the equation is used to determine the intensity of the corresponding candidate emotion. The formula is as follows:

[0125]

[0126] Among them, N e represents the frequency of candidate emotion e, D e Indicates the intensity of candidate emotions, N v Indicates the total number of candidate emotions, that is, the total number of keywords.

[0127] For example, in the candidate emotion set s E ={happy, happy, worried}, the intensity of "happy" is 2 / 3, and the intensity of "worried" is 1 / 3.

[0128] S4055_2: The candidate emotion corresponding to the highest intensity is used as the target emotion of the 3D virtual human.

[0129] Still taking the candidate emotion set s E = {happy, happy, worried} as an example, "happy" has the highest intensity, so "happy" is used as the target emotion of the 3D virtual person in the text paragraph. The target emotion may or may not have a corresponding keyword in the database.

[0130] In the embodiments of the present application, the overall emotion of the 3D virtual human in each text paragraph is reflected by the target emotion, thereby avoiding the negative impact of individual keywords in some sentences on the entire context, making the expression of emotions more accurate.

[0131] In an optional embodiment, when determining the target emotion corresponding to each text paragraph, for each keyword that does not exist in the database, when determining that it has a corresponding emotion in the database through a preset emotion reaching threshold, the number of correspondences between the keyword and the emotion represented by the maximum probability greater than or equal to the preset emotion reaching threshold can be counted, and the number of correspondences can be compared with the preset number threshold N. out Compare, if the corresponding number exceeds the preset number threshold N out, then the corresponding relationship between the keyword and the emotion represented by the maximum probability is stored in the database to update the database.

[0132] In an optional embodiment, the number of times the database is updated can also be counted. If the number of updates reaches a preset update threshold, it indicates that there are many newly stored keyword and emotion pairs in the database. Therefore, the emotion extraction model can be re-trained based on the updated database to improve the accuracy of the model's emotion extraction.

[0133] In the embodiments of the present application, the emotion extraction model based on machine learning training can accurately extract the emotions corresponding to the stored keywords, and can also perform good emotion extraction for the unstored keywords, thereby dynamically expanding the database storing the correspondence between keywords and emotions, so that the trained emotion extraction model has higher accuracy, greater generalization and versatility.

[0134] S406: adjusting first driving parameters of each core expression base related to the target emotion in the corresponding expression base set according to the target emotion to obtain second driving parameters.

[0135] Since each emotion corresponds to a set of expression bases in the database, and the core expression bases related to the emotion are represented by non-zero, the core expression bases corresponding to the target emotion corresponding to each text paragraph can be obtained from the database. Since the expression of emotion is independent of the expression bases that control the head posture, the core expression bases can be fused with some expression bases related to facial expressions in all expression bases determined by voice to modify the driving parameters of each core expression base, thereby obtaining the driving parameters of dynamic adaptive emotion, realizing the adaptive dynamic adjustment of the emotion of the voice-driven 3D virtual human, and improving the authenticity and naturalness of the 3D virtual human.

[0136] The fusion process can be seen in Figure 7 , mainly includes the following steps:

[0137] S4061: Add a control factor for the intensity of the target emotion.

[0138] For each text paragraph, after determining the intensity of the target emotion based on the frequency of occurrence of various candidate emotions corresponding to the keyword set, a control coefficient is added to the intensity of the target emotion. The control coefficient is used to control the intensity of the target emotion. The larger the control coefficient, the stronger the expression effect of the target emotion.

[0139] Optionally, the control coefficient has a value range of [0, 1], which can be set according to the actual application scenario. Different control coefficients are used in different scenarios, and the embodiment of the present application does not impose any restrictive requirements. By default, the value is 0.5.

[0140] S4062: For each core expression base, respectively determine a first influence value of the control coefficient on the intensity of the target emotion and a second influence value of the control coefficient on the first driving parameter of the core expression base.

[0141] S4063: Fusing the first influence value and the second influence value to obtain a second driving parameter of the core expression base.

[0142] Specifically, the fusion process formula of the core expression base is expressed as follows:

[0143]

[0144] Among them, δ represents the control coefficient, Indicates the target emotion main The strength of j represents the first driving parameter of the jth core expression base, represents the first impact value, (1-δ)b j Indicates the second impact value, b' j Represents the second driving parameter of the jth core expression base.

[0145] In an embodiment of the present application, the first driving parameter in the core expression base determined by the voice is corrected through dynamic adaptive emotions, so as to realize the control of the swing, rotation and pitch of the head, and at the same time realize the precise driving and delicate emotional expression of the mouth, cheeks, eyes, eyebrows, eyeballs, head and chin. Compared with adding changes such as glasses and eyebrows on the basis of voice-driven lip shape to control the expression of emotions, the coordination and consistency of the mouth movements with other facial positions and head movements are improved, making the effect of voice-driven 3D virtual people more natural.

[0146] S407: According to the second driving parameters of each core expression base and the first driving parameters of other expression bases, a 3D virtual human with emotional expression is obtained and displayed.

[0147] By combining the modified second driving parameters of each core expression base and the first driving parameters of other expression bases, the control parameters of all expression bases of the parametric head model of the 3D virtual human can be obtained, so as to directly drive the face and head of the 3D virtual human, obtain a 3D virtual human with emotional expression and display the animation of the 3D virtual human. Since the expression base occupies fewer parameters (generally <100), it consumes fewer resources, and a small number of parameters can be used to realize the dynamic expression of 3D virtual human emotions and the control of head posture, which is faster and has better real-time performance.

[0148] It should be noted that the embodiments of the present application do not impose any restrictive requirements on the display method of the 3D virtual human. For example, the 3D virtual human driving effect can be displayed on a two-dimensional plane through a screen, or the 3D virtual human driving effect can be displayed in a three-dimensional space based on three-dimensional display technology and VR technology.

[0149] In an optional embodiment, the process of driving the facial expression and head posture movement of the 3D virtual human by the driving parameters of the expression base is a process of linearly weighting the driving parameters of each expression base, which is expressed as follows:

[0150]

[0151] Among them, α h represents the driving parameter of the hth expression base, ω h represents the weight corresponding to the hth expression base, w is the driving parameter after all expression bases are weighted, and F(w) represents the model driven by all expression bases, that is, the model obtained by performing expression migration based on the basic model in the natural state (such as the model composed of 52 expression bases of the standard Arkit).

[0152] Taking four expression bases as an example, the linear weighted schematic diagram of each expression base is as follows: Figure 8 As shown in , by weighting, new expressions can be obtained. It should be noted that Figure 8 The number of expression bases in is only an example and is not a limitation of the expression bases used for weighting.

[0153] like Fig. 9 As shown, 48 expression models are obtained after the expression migration of the basic model in the natural state composed of 52 expression bases.

[0154] In the embodiment of the present application, when the 3D virtual human is voice-driven, not only the mouth is driven, but on the basis of lip synchronization, the driving parameters of the expression base that controls the entire facial movements (such as the movements of the mouth, cheeks, eyes, eyebrows, eyeballs, head, chin, etc.) and head movements (such as the swinging, turning, and pitching of the head) are directly extracted from each audio segment after the voice segmentation, so that the overall facial expression is rich, the facial control is more precise, and the dullness of the voice-driven 3D virtual human is avoided; at the same time, the input audio is converted into text and then segmented, and the emotions expressed by the keyword set extracted from each text segment are determined by the emotion extraction model, so that the speech content and context of the 3D virtual human are adapted to the emotions, and dynamic adaptive adjustment of emotions is achieved, thereby improving the naturalness and authenticity of the voice-driven 3D virtual human, and the emotion extraction model based on machine learning training has a certain learning ability and generalization. For keywords not stored in the database, the corresponding emotions can still be accurately extracted without increasing the complexity of the model, thereby improving the accuracy of the emotional expression of the 3D virtual human. The entire voice-driven process only requires voice input to achieve end-to-end driving of the 3D virtual human face and head, without the need for additional emotional input or provision of emotional references (such as videos, pictures, etc.), and is more intelligent.

[0155] Based on the same technical concept, an embodiment of the present application provides a voice-driven 3D virtual human emotion expression system, which can implement the steps of any of the above-mentioned voice-driven 3D virtual human emotion expression methods and achieve the same technical effects.

[0156] See also Fig.10 , audio acquisition module 1001, audio segmentation module 1002, expression base parameter estimation module 1003, text conversion module 1004, segmentation and word segmentation module 1005, emotion extraction module 1006, expression base parameter correction module 1007 and expression base driving module 1008, wherein:

[0157] The audio acquisition module 1001 is used to acquire the target audio of the 3D virtual human;

[0158] The audio segmentation module 1002 is used to segment the target audio according to a preset frame rate;

[0159] An expression base parameter estimation module 1003 is used to determine first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment;

[0160] A text conversion module 1004, used to convert the target audio into a target text;

[0161] The segmentation and word segmentation module 1004 is used to segment the target text, extract emotion-related keywords in each text paragraph, obtain a keyword set, and input the keyword set into a pre-trained emotion extraction model; wherein the emotion extraction model is trained based on a database storing a plurality of corresponding relationships between emotions and keywords, and each emotion corresponds to an expression base set;

[0162] An emotion extraction module 1006 is used to use the emotion extraction model to obtain a target emotion corresponding to the 3D virtual person in the text paragraph, where the target emotion has a corresponding keyword in the database or not;

[0163] The expression base parameter correction module 1007 is used to adjust the first driving parameters of each core expression base related to the target emotion in the corresponding expression base set according to the target emotion to obtain the second driving parameters;

[0164] The expression base driving module 1008 is used to obtain and display a 3D virtual human with emotional expression according to the second driving parameters of each core expression base and the first driving parameters of other expression bases.

[0165] Optionally, the emotion extraction module 1006 is specifically used for:

[0166] For each keyword in the keyword set, perform the following operations:

[0167] Using the emotion extraction model to extract the emotion features of the keywords;

[0168] If the keyword exists in the database for training the emotion extraction model, the probability that the keyword corresponds to various emotions in the database is calculated according to the emotion feature, and the emotion corresponding to the highest probability among the probabilities is used as the candidate emotion corresponding to the keyword;

[0169] If the keyword does not exist in the database for training the emotion extraction model, predicting various emotions corresponding to the keyword according to the emotion feature, and determining candidate emotions corresponding to the keyword according to the comparison result of the probability of the various emotions and the preset emotion reaching threshold;

[0170] A target emotion corresponding to the 3D virtual person in the text paragraph is determined from candidate emotions corresponding to each keyword in the keyword set.

[0171] Optionally, the emotion extraction module 1006 is specifically used for:

[0172] Compare the probabilities of the various emotions with the preset emotion reaching thresholds respectively;

[0173] If the probabilities of the various emotions are all less than the preset emotion reaching threshold, the keyword is eliminated; otherwise, the emotion corresponding to the maximum probability greater than or equal to the preset emotion reaching threshold is used as the candidate emotion of the keyword.

[0174] Optionally, the emotion extraction module 1006 is further used for:

[0175] Counting the number of correspondences between the keyword and the emotion represented by the maximum probability greater than or equal to the preset emotion reaching threshold;

[0176] If the number of correspondences exceeds a preset number threshold, the correspondence between the keyword and the emotion represented by the maximum probability is stored in the database to update the database.

[0177] Optionally, the system further includes a model training module 1009, which is used to:

[0178] Counting the number of updates of the database;

[0179] If the number of updates reaches a preset update threshold, the emotion extraction model is retrained according to the updated database.

[0180] Optionally, the expression base parameter correction module 1007 is specifically used for:

[0181] Adding a control coefficient to the intensity of the target emotion, wherein the intensity is determined based on the frequency of occurrence of the target emotion in various candidate emotions corresponding to the keyword set;

[0182] For each core expression base, respectively determining a first influence value of the control coefficient on the intensity of the target emotion and a second influence value of the control coefficient on the first driving parameter of the core expression base;

[0183] The first influence value and the second influence value are fused to obtain a second driving parameter of the core expression base.

[0184] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0185] After introducing the text method and apparatus based on active learning according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0186] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system".

[0187] Based on the same inventive concept as the above-mentioned method embodiment, an electronic device is also provided in the embodiment of the present application, which includes but is not limited to desktop computers, laptop computers, wearable devices (such as AR devices, VR devices, etc.), game terminals, etc.

[0188] In an optional embodiment, the structure of the electronic device can be as follows: Fig.11 As shown, it includes a memory 1101, a display 1103 and one or more processors 1102. Among them:

[0189] The memory 1101 is used to store the computer program executed by the processor 1102. The memory 1101 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system and the programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory 1101 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1101 may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1101 may be any other medium that can be used to carry or store the desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1101 may be a combination of the above memories.

[0190] The processor 1102 may include one or more central processing units (CPU), GPU or digital processing unit, etc. The processor 1102 is used to implement any of the above-mentioned steps of the voice-driven 3D virtual human emotion expression method when calling the computer program stored in the memory 1101.

[0191] The display 1103 is used to display a 3D virtual human.

[0192] The specific connection medium between the memory 1101, the display 1103 and the processor 1102 is not limited in the embodiment of the present application. Fig.11 In the embodiment, the memory 1101, the display 1103 and the processor 1102 are connected via a bus 1104. Fig.11 The connections between the other components are described with bold lines, which are only for illustration and are not intended to be limiting. The bus 1104 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.11 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0193] It should be noted that Fig.11 This is just an example, and provides the necessary hardware for the electronic device to execute the steps of the voice-driven 3D virtual human emotion expression method provided in the embodiment of the present application. If not shown, the electronic device may also include conventional hardware such as a speaker, a pickup, a communication interface, and a power supply.

[0194] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete any of the voice-driven 3D virtual human emotion expression methods in the aforementioned embodiments.

[0195] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute any one of the voice-driven 3D virtual human emotion expression methods in the aforementioned embodiments.

[0196] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0197] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0200] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for expressing emotions of a 3D virtual human driven by speech, characterized in that: include: Get the target audio of the 3D virtual person; Segmenting the target audio according to a preset frame rate, and determining first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment; Convert the target audio into a target text, and segment the target text; For each text paragraph, extract keywords related to emotions in the text paragraph to obtain a keyword set, and input the keyword set into a pre-trained emotion extraction model; wherein the emotion extraction model is trained based on a database storing a plurality of pairs of corresponding relationships between emotions and keywords, and each emotion corresponds to an expression base set; Using the emotion extraction model, obtaining the target emotion corresponding to the 3D virtual person in the text paragraph, wherein the target emotion has or does not have a corresponding keyword in the database; According to the target emotion, adjusting the first driving parameters of each core expression base in the corresponding expression base set related to the target emotion to obtain a second driving parameter; According to the second driving parameters of each core expression base and the first driving parameters of other expression bases, a 3D virtual human with emotional expression is obtained and displayed.

2. The method according to claim 1, characterized in that The adopting the emotion extraction model to obtain the target emotion corresponding to the 3D virtual person in the text paragraph includes: For each keyword in the keyword set, perform the following operations: Using the emotion extraction model to extract the emotion features of the keywords; If the keyword exists in the database for training the emotion extraction model, the probability that the keyword corresponds to various emotions in the database is calculated according to the emotion feature, and the emotion corresponding to the highest probability among the probabilities is used as the candidate emotion corresponding to the keyword; If the keyword does not exist in the database for training the emotion extraction model, predicting various emotions corresponding to the keyword according to the emotion feature, and determining candidate emotions corresponding to the keyword according to the comparison result of the probability of the various emotions and the preset emotion reaching threshold; A target emotion corresponding to the 3D virtual person in the text paragraph is determined from candidate emotions corresponding to each keyword in the keyword set.

3. The method according to claim 2, characterized in that The step of determining the candidate emotion corresponding to the keyword according to the comparison result of the probability of the various emotions and the preset emotion reaching threshold value comprises: Compare the probabilities of the various emotions with the preset emotion reaching thresholds respectively; If the probabilities of the various emotions are all less than the preset emotion reaching threshold, the keyword is eliminated; otherwise, the emotion corresponding to the maximum probability greater than or equal to the preset emotion reaching threshold is used as the candidate emotion of the keyword.

4. The method according to claim 3, characterized in that The method further comprises: Counting the number of correspondences between the keyword and the emotion represented by the maximum probability greater than or equal to the preset emotion reaching threshold; If the number of correspondences exceeds a preset number threshold, the correspondence between the keyword and the emotion represented by the maximum probability is stored in the database to update the database.

5. The method according to claim 4, characterized in that The method further comprises: Counting the number of updates of the database; If the number of updates reaches a preset update threshold, the emotion extraction model is retrained according to the updated database.

6. The method according to claim 1, characterized in that The step of adjusting the first driving parameters of each core expression base in the corresponding expression base set related to the target emotion according to the target emotion to obtain the second driving parameters includes: Adding a control coefficient to the intensity of the target emotion, wherein the intensity is determined based on the frequency of occurrence of the target emotion in various candidate emotions corresponding to the keyword set; For each core expression base, respectively determining a first influence value of the control coefficient on the intensity of the target emotion and a second influence value of the control coefficient on the first driving parameter of the core expression base; The first influence value and the second influence value are fused to obtain a second driving parameter of the core expression base.

7. A voice-driven 3D virtual human emotion expression system, characterized in that: include An audio acquisition module is used to acquire the target audio of the 3D virtual human; An audio segmentation module, used to segment the target audio according to a preset frame rate; An expression base parameter estimation module, used for determining first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment; A text conversion module, used for converting the target audio into a target text; A segmentation and word segmentation module is used to segment the target text, extract emotion-related keywords from each text paragraph, obtain a keyword set, and input the keyword set into a pre-trained emotion extraction model; wherein the emotion extraction model is trained based on a database storing a plurality of pairs of corresponding relationships between emotions and keywords, and each emotion corresponds to an expression base set; An emotion extraction module, used to use the emotion extraction model to obtain the target emotion corresponding to the 3D virtual person in the text paragraph, wherein the target emotion has or does not have a corresponding keyword in the database; An expression base parameter correction module, used for adjusting the first driving parameters of each core expression base related to the target emotion in the corresponding expression base set according to the target emotion to obtain a second driving parameter; The expression base driving module is used to obtain and display a 3D virtual human with emotional expression according to the second driving parameters of each core expression base and the first driving parameters of other expression bases.

8. The system according to claim 7, characterized in that The emotion extraction module is specifically used for: For each keyword in the keyword set, perform the following operations: Using the emotion extraction model to extract the emotion features of the keywords; If the keyword exists in the database for training the emotion extraction model, the probability that the keyword corresponds to various emotions in the database is calculated according to the emotion feature, and the emotion corresponding to the highest probability among the probabilities is used as the candidate emotion corresponding to the keyword; If the keyword does not exist in the database for training the emotion extraction model, predicting various emotions corresponding to the keyword according to the emotion feature, and determining candidate emotions corresponding to the keyword according to the comparison result of the probability of the various emotions and the preset emotion reaching threshold; A target emotion corresponding to the 3D virtual person in the text paragraph is determined from candidate emotions corresponding to each keyword in the keyword set.

9. The system according to claim 8, characterized in that The emotion extraction module is specifically used for: Compare the probabilities of the various emotions with the preset emotion reaching thresholds respectively; If the probabilities of the various emotions are all less than the preset emotion reaching threshold, the keyword is eliminated; otherwise, the emotion corresponding to the maximum probability greater than or equal to the preset emotion reaching threshold is used as the candidate emotion of the keyword.

10. An electronic device, characterized in that: It includes a processor, a memory and a display, wherein the display, the memory and the processor are connected via a bus; The display is used to display a 3D virtual person; The memory stores a computer program, and the processor executes the method according to any one of claims 1 to 6.

Images