3D virtual human expression driving method, system and equipment based on keyword matching

Abstract

The invention relates to the technical field of 3D virtual humans, and provides a keyword matching-based 3D virtual human expression driving method, system and device, which are used for improving the precision of 3D virtual human emotion expression during voice driving. According to the method, first driving parameters of all expression bases are obtained by utilizing audio clips obtained after target audio is segmented, and a preset database is inquired by utilizing a keyword set extracted from a target text obtained after the target audio is converted to obtain a target emotion, so that the emotion of the 3D virtual human adapts to the lip shape; the first driving parameters of the core expression base corresponding to the target emotion are corrected, self-adaptive adjustment of the emotion along with the voice is achieved, the expression effect of the emotion of the 3D virtual human is enhanced, a more natural and real voice driving effect of the 3D virtual human is presented in combination with the first driving parameters of other expression bases, and the user experience is improved. Meanwhile, the intelligent degree of the 3D virtual human is improved, and compared with a complex emotion extraction network, the occupied computing resources are few, and the real-time performance is higher.

Description

Background Art

[0002] Compared with 2D virtual humans, 3D virtual humans can bring a stronger sense of reality and immersion to the audience, so they have more application fields and scopes, such as VR games, 3D animation, etc. At the same time, the production of 3D virtual humans requires higher technology, and the driving principle of 3D virtual humans is completely different from that of 2D virtual humans. Among them, 2D virtual humans are driven by images, and 3D virtual humans are driven by three-dimensional models. Therefore, the objects driven by the two and the driving methods are different. In this way, the text-based emotion control method of 2D virtual humans in the field of voice drive cannot achieve voice drive to control the emotions of 3D virtual humans.

[0003] At present, in the field of voice-driven 3D virtual humans, the focus is only on the mouth of the 3D virtual human. Usually, the voice is matched with the mouth shape of the 3D virtual human, and some random parameters such as blinking and head posture are added, while the expression of the 3D virtual human's emotions is ignored, making the driven 3D virtual human not natural and realistic enough. Summary of the invention

[0004] The embodiments of the present application provide a 3D virtual human expression driving method, system and device based on keyword matching, which are used to improve the accuracy of 3D virtual human emotional expression.

[0005] On the one hand, an embodiment of the present application provides a 3D virtual human expression driving method based on keyword matching, comprising:

[0006] Acquire target audio of the 3D virtual human, segment the target audio according to a preset frame rate, and convert the target audio into target text;

[0007] Determining first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment;

[0008] Extract keywords from the target text to obtain a keyword set, and query a preset database according to the keyword set to obtain a target emotion of the 3D virtual person and a core expression base set corresponding to the target emotion; wherein the preset database stores a plurality of pairs of corresponding relationships between keywords and emotions, and each emotion corresponds to a core expression base set;

[0009] Adjusting the first driving parameter of each core expression base in the core expression base set according to the target emotion to obtain a second driving parameter;

[0010] According to the second driving parameters of each core expression base and the first driving parameters of other expression bases, a 3D virtual human with emotional expression is obtained and displayed.

[0011] On the other hand, the embodiment of the present application provides a 3D virtual human expression driving system based on keyword matching, including:

[0012] An audio acquisition module is used to acquire the target audio of the 3D virtual human;

[0013] An audio segmentation module, used to segment the target audio according to a preset frame rate;

[0014] A text conversion module, used for converting the target audio into a target text;

[0015] An expression base parameter estimation module, used for determining first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment;

[0016] A keyword extraction module, used to extract keywords from the target text to obtain a keyword set;

[0017] A keyword matching module is used to query a preset database according to the keyword set to obtain the target emotion of the 3D virtual person and the core expression base set corresponding to the target emotion; wherein the preset database stores a plurality of pairs of corresponding relationships between keywords and emotions, and each emotion corresponds to a core expression base set;

[0018] An expression base parameter correction module, used for adjusting the first driving parameter of each core expression base in the core expression base set according to the target emotion to obtain a second driving parameter;

[0019] The expression base driving module is used to obtain and display a 3D virtual human with emotional expression according to the second driving parameters of each core expression base and the first driving parameters of other expression bases.

[0020] Optionally, the keyword matching module is specifically used for:

[0021] Querying the preset database according to each keyword in the keyword set to obtain candidate emotions corresponding to the keyword;

[0022] Count the frequency of occurrence of each candidate emotion among the obtained candidate emotions;

[0023] The candidate emotion corresponding to the highest frequency is used as the target emotion of the 3D virtual person;

[0024] According to the target emotion, a corresponding core expression base set is obtained.

[0025] Optionally, the keyword extraction module is specifically used for:

[0026] Segmenting the target text to obtain a combination of multiple words and phrases representing emotions;

[0027] Individual words are removed, and the keyword set is generated using the remaining multiple phrases.

[0028] Optionally, the expression base parameter correction module is specifically used for:

[0029] Adding a control factor for the intensity of the target emotion;

[0030] For each core expression base in the core expression base set, determining an influence value of the target emotion with the control coefficient added thereto on a first driving parameter of the core expression base;

[0031] The influence value is superimposed on the first driving parameter of the core expression base to obtain the second driving parameter of the core expression base.

[0032] Optionally, the expression base parameter correction module is also used to:

[0033] For each core expression base, determine whether the second driving parameter of the core expression base is greater than a preset parameter upper limit; if greater, use the preset parameter upper limit as the second driving parameter of the core expression base.

[0034] Optionally, the target audio includes any one of text-converted audio and speech recording.

[0035] Optionally, the emotions corresponding to different keywords are the same or different.

[0036] On the other hand, an embodiment of the present application provides an electronic device, including a processor, a memory, and a display, wherein the display, the memory, and the processor are connected via a bus;

[0037] The display is used to display a 3D virtual person;

[0038] The memory stores a computer program, and the processor executes any step of the 3D virtual human expression driving method based on keyword matching according to the computer program.

[0039] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a 3D virtual human expression driving method based on keyword matching provided in an embodiment of the present application.

[0040] The beneficial effects of a 3D virtual human expression driving method, system and device based on keyword matching provided by the embodiments of the present application are as follows:

[0041] The first driving parameters of all expression bases of the 3D virtual person are determined for each audio segment divided according to the preset frame rate, so that the 3D virtual person can be animated according to the preset frame rate. In order to improve the accuracy of the 3D virtual person driving during the animation process, the target text after the target audio conversion is extracted by keywords, and the preset database is queried according to the keyword set, so as to accurately obtain the overall target emotion of the 3D virtual person, so that the target emotion is adapted to the lip shape, and the authenticity of the voice drive is improved. In addition, this method does not need to design a complex emotion extraction network, occupies less computing resources, and can meet the needs of real-time driving scenes. On the other hand, since each emotion corresponds to a core expression base set, the first driving parameters of each core expression base in the core expression base set corresponding to the target emotion can be used to adjust the expression of the 3D virtual person and the lip shape. At the same time, by associating the expression with the core expression base set, the control of the emotion on the 3D virtual person is more delicate and real. In this way, when the 3D virtual person with emotional expression is obtained in combination with the first driving parameters of other expression bases, the authenticity of the voice drive is further improved, and the 3D virtual person looks more natural.

[0042] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A schematic diagram of a method for driving expression of a 3D virtual human in the prior art provided in an embodiment of the present application;

[0045] Figure 2 The expression effect diagrams of two 3D virtual humans with different emotions provided in the embodiments of the present application;

[0046] Figure 3 The overall framework diagram of 3D virtual human expression driving based on keyword matching provided in the embodiment of the present application;

[0047] Figure 4 A flowchart of a 3D virtual human expression driving method based on keyword matching provided in an embodiment of the present application;

[0048] Figure 5 52 basic expression diagrams of a 3D virtual human provided in an embodiment of the present application;

[0049] Figure 6 A flow chart of a method for acquiring emotions based on keyword matching provided in an embodiment of the present application;

[0050] Figure 7 A flow chart of a method for modifying driving parameters of a core expression base corresponding to a target emotion provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of expression-based weighting provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of a model of different expressions provided in an embodiment of the present application;

[0053] Figure 1 0 is a structural diagram of a 3D virtual human expression driving system based on keyword matching provided in an embodiment of the present application;

[0054] Fig.11 A structural diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0056] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0057] It should be understood that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0058] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0059] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0060] The following is an overview of the design concept of the embodiments of the present application in conjunction with application scenarios.

[0061] The human face is an important part of human emotion expression. In the three-dimensional virtual world, the virtual face is the basis for 3D virtual human expression. The research purpose of 3D virtual human expression drive is to make 3D virtual human appear like a real person. With the development of computer vision and computer graphics, voice-driven technology has given 3D virtual human the ability to speak like a real person, making 3D virtual human widely used in various scenarios such as live broadcast, short video generation, digital cloning, customer service, education, finance, video conferencing, etc.

[0062] At present, the voice-driven technology of 3D virtual humans has achieved high-precision matching of audio and mouth shape, achieving a lip synchronization effect close to that of a real person. However, when a 3D virtual human speaks, only the mouth moves, which appears stiff and unnatural. Therefore, some technologies have added actions such as blinking and head shaking to the 3D virtual human to make the 3D virtual human look more realistic when speaking.

[0063] In practical applications, the emotions expressed by people when speaking (such as anger, happiness, surprise, disgust, sadness, fear, etc.) are an important part of the human face. Different voices often contain different emotions, and people's facial expressions will be very different under different emotions. At present, most voice-driven solutions for 3D virtual humans do not have emotional expression, that is, the 3D virtual human defaults to a natural expression. The lack of emotion makes the 3D virtual human seem to lack a "soul", and the effect presented is very different from that of a real person and is not natural enough.

[0064] In order to improve the authenticity of 3D virtual human voice driving, some technologies specify emotions for 3D virtual humans by inputting them in advance, such as happiness, anger, excitement, etc., and then use them together with audio as input to estimate the expression base parameters, obtain the driving parameters of the expression base, and thus obtain the voice driving effect of the 3D virtual human with the specified emotions, such as Figure 1However, in different contexts and scenes, 3D virtual humans may need different emotional expressions, such as Figure 2 The figure shows the expression of two emotions, happiness and anger. However, this method can only correspond to a single emotion in a natural state or a fixed emotion set in advance. It cannot adaptively change according to the context and audio of the 3D virtual person. It is difficult to meet the needs of the 3D virtual person for different emotions under different voice content or context during real-time dialogue or driving. It is easy to have the problem of contradiction between what is said and the emotion expressed, which seriously affects the driving effect of the 3D virtual person.

[0065] For example, in a depressing scene, the 3D virtual person still uses a fixed happy emotion, or, in a joyful scene, the 3D virtual person continues to maintain a depressed emotion. Due to the incompatibility of emotions and scenes, the display effect of the 3D virtual person will be affected, which will cause a great sense of disharmony to the audience.

[0066] In view of this, the embodiment of the present application provides a 3D virtual human expression driving method based on keyword matching, which uses keywords extracted from the target text after the target audio is converted to query a preset database, thereby obtaining the target emotion corresponding to the target audio, so that the 3D virtual human can express emotions according to the current speech content or context adaptively, improve the accuracy of the 3D virtual human's emotional expression, and do not need to design a complex emotion extraction network, occupy less computing resources, and can meet the needs of real-time driving scenes. In addition, an association relationship between each emotion and a core expression base set is established, so that the first driving parameters of each core expression base in the corresponding core expression base set are adjusted by using the obtained target emotion, and the target emotion is integrated into the driving parameters of the expression base of the 3D virtual human, so that the expression of the 3D virtual human is synchronized with the lip shape, and a more accurate 3D virtual human voice driving is achieved. At the same time, by associating the expression with the core expression base set, the control of the emotion on the 3D virtual human is also made more delicate and real. In this way, when the 3D virtual human with emotional expression is obtained in combination with the first driving parameters of other expression bases, the authenticity of the voice driving and the intelligence of the 3D virtual human are improved, so that the 3D virtual human looks more natural.

[0067] In the embodiments of the present application, the head of the 3D virtual human is constructed by a parametric head model (such as Skinned Multi-Person Linear, referred to as SLMP), which is composed of shape parameters, posture parameters and expression parameters. The shape parameters of the same person remain basically unchanged, the posture parameters mainly control the rigid movement of the head, and the expression parameters mainly control facial changes. The parameterized 3D virtual human contains multiple expression bases. Therefore, the changes in the face or head posture of the 3D virtual human can be driven by controlling the driving parameters of the expression bases. Since the embodiments of the present application mainly study the expression of facial emotions when driven by voice, and reasonable expression parameters can enable the 3D virtual human to show different emotions through changes in the eyes, mouth, mandible, eyebrows, cheeks, etc., therefore, the driving parameters of the embodiments of the present application mainly refer to the expression parameters that can drive the face and the posture parameters that can drive the head.

[0068] The overall framework diagram of 3D virtual human expression driving based on keyword matching provided in the embodiment of the present application is as follows: Figure 3 As shown, it mainly includes three parts: emotion and voice adaptive adjustment based on keyword matching, 3D virtual human driving and 3D virtual human display. Among them, the driving part of the 3D virtual human is mainly to drive the head and face, and the change of the 3D virtual human face is achieved by weighting the driving parameters of each expression base; the display part of the 3D virtual human is to present the 3D virtual human to the user through the screen; the emotion and voice adaptive adjustment part based on keyword matching is the key content of the embodiment of this application, including audio input, audio segmentation, text conversion, expression base parameter estimation, keyword extraction, keyword matching, expression base parameter correction and expression base parameter output and other processes.

[0069] See also Figure 4 , which is a flow chart of a 3D virtual human expression driving method based on keyword matching provided in an embodiment of the present application, mainly includes the following steps:

[0070] S401: Acquire the target audio of the 3D virtual human, segment the target audio according to a preset frame rate, and convert the target audio into a target text.

[0071] In an optional embodiment, the target audio associated with the 3D virtual human may be obtained from the outside or locally, including but not limited to any one of speech recordings and audio obtained using a text-to-speech tool.

[0072] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the language type of the target audio, which can be Chinese, English, Japanese, etc.

[0073] In order to improve the accuracy of emotional expression of 3D virtual humans during voice-driven operation, after acquiring the target audio, the target audio was processed in terms of both speech and text.

[0074] At the voice level, the acquired target audio is divided into multiple audio segments according to a preset frame rate, so that the 3D virtual human can be driven to animate at a preset frame rate according to each audio segment. Optionally, the preset frame rate is 30 frames per second, and its size can be adjusted according to actual needs, and the embodiment of the present application does not impose a restrictive requirement.

[0075] At the text level, a speech-to-text tool is used to convert the acquired target audio into target text for extracting keywords expressing emotions. Optionally, the multiple audio segments after the target audio is segmented can be converted into text segments respectively, and the entire target audio can also be converted into an entire text.

[0076] S402: Determine first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment.

[0077] The head of a 3D virtual human is a parametric head model composed of multiple expression bases. Therefore, the face and head posture of the 3D virtual human can be driven by controlling the driving parameters of the expression bases.

[0078] In an optional embodiment, the driving parameters for controlling the expression base can be obtained from the target audio. Specifically, for each segmented audio segment, the audio segment is input into a pre-trained parameter generation network to extract speech features, and the first driving parameters of all expression bases of the 3D virtual human corresponding to the audio segment are output according to the extracted speech features. The first driving parameters can control the facial features of the 3D virtual human and achieve synchronization between the input speech and the lips of the 3D virtual human.

[0079] Take the example of using ARKit 52 expression bases to control the facial expressions of 3D virtual people. The information of these 52 expression bases is as follows: Figure 5 shown.

[0080] S403: extract keywords from the target text to obtain a keyword set, and query a preset database according to the keyword set to obtain the target emotion of the 3D virtual human and the core expression base set corresponding to the target emotion.

[0081] In the embodiment of the present application, a database is pre-built, which stores the correspondence between multiple pairs of keywords and emotions, and each emotion corresponds to a core expression base set, and the format is as follows: {keyword-emotion-(core expression base 1, core expression base 2, ...)}. Among them, the emotions corresponding to different keywords are the same or different.

[0082] For example, the keywords "heart bursting with joy" and "feeling relaxed and happy" correspond to the emotion "happy", and the keywords "furious" and "discouraged" correspond to the emotions "angry" and "depressed", respectively.

[0083] In an embodiment of the present application, a certain emotion is corresponded to a core expression base set composed of specific core expression bases in a preset database, so that the display of different emotions is associated with the core expression bases that control the head posture, cheek muscles, eyebrows, eyeballs, wrinkles, etc. of the 3D virtual human, thereby achieving the effect of adjusting the emotional expression of the 3D virtual human by adjusting the driving parameters of the core expression base, making the emotional expression of the 3D virtual human more delicate and realistic.

[0084] In an optional embodiment, when extracting keywords from a text, the text segment converted from each audio segment can be used as the target text to extract keywords, or the entire text segment converted from the target audio can be used as the target text to extract keywords.

[0085] In order to avoid the negative impact of individual keywords in a partial text on the context, the embodiment of the present application uses the entire text as the target text to extract keywords.

[0086] In the specific implementation, the target text is segmented according to certain semantics to obtain multiple words, phrases and sets that represent emotions. Considering that the semantics expressed by a single word are very different, and most words do not have a clear meaning. For example, the semantics expressed by a single word "smile" can be a real smile, a fake smile, or a bad smile, which cannot accurately express the real emotion. Therefore, the embodiment of the present application removes the extracted single words and uses the remaining multiple phrases to generate a keyword set, which is recorded as Among them, N k The total number of keywords.

[0087] Based on the pre-built database, by matching the extracted keywords with the keywords in the database, the target emotion of the 3D virtual person and the core expression base set corresponding to the target emotion can be obtained. Figure 6 , mainly includes the following steps:

[0088] S4031: Query a preset database according to each keyword in the keyword set to obtain candidate emotions corresponding to the keyword.

[0089] For each keyword k in the keyword set K i , query keyword k from the preset database i The corresponding candidate emotions. After the query is completed, N k The candidate emotion set E corresponding to the keyword.

[0090] Taking the keyword set K = {smiling, crying with joy, flattered} as an example, by querying the preset database, the candidate emotion set E = {happy, happy, worried} can be obtained.

[0091] S4032: Count the frequency of occurrence of each candidate emotion among the obtained candidate emotions.

[0092] In the entire text, the higher the frequency of each emotion, the more the emotion can represent the context of the entire audio. Therefore, the frequency of each candidate emotion in the candidate emotion set E can be counted to determine the intensity of the corresponding candidate emotion. The formula is as follows:

[0093]

[0094] in, Represents the candidate emotion e main The frequency of occurrence, Indicates the intensity of candidate emotions, N k Indicates the total number of candidate emotions, that is, the total number of keywords.

[0095] For example, in the candidate emotion set E={happy, happy, worried}, the intensity of "happy" is 2 / 3, and the intensity of "worried" is 1 / 3.

[0096] S4033: Taking the candidate emotion corresponding to the highest intensity as the target emotion of the 3D virtual human.

[0097] Still taking the candidate emotion set E={happy, happy, worried} as an example, “happy” has the highest intensity, so “happy” is used as the target emotion of the 3D virtual human in the entire audio segment.

[0098] S4034: According to the target emotion, obtain the corresponding core expression base set.

[0099] After obtaining the target emotion, the database is queried again to obtain the core expression base set I corresponding to the target emotion. main The corresponding core expression base set I contains the expression e main In this way, when adjusting the emotions of the 3D virtual human in the subsequent driving process, it is only necessary to adjust the core expression bases in the core expression base set I without adjusting the driving parameters of all expression bases, which makes the emotional expression of the 3D virtual human more accurate and improves the driving efficiency of the 3D virtual human face.

[0100] In an embodiment of the present application, the target emotion of the 3D virtual person is obtained by querying a preset database through keywords extracted from the entire text, so that the target emotion can reflect the emotional expression of the 3D virtual person in the entire audio, avoiding the negative impact of individual keywords in some texts on the overall context, and making the emotional control of the 3D virtual person more accurate; at the same time, using the keyword matching method, there is no need to design a complex emotion extraction model or network, and the emotion query is directly performed through the mapping relationship between keywords and emotions recorded in the preset database. The implementation process consumes less resources, is efficient, and has strong real-time performance, thereby improving the real-time performance of emotion control. In this way, in scenarios such as real-time interactive question and answer, the 3D virtual person is supported to adaptively adjust his emotions according to the voice content, thereby improving the real-time performance and intelligence of the 3D virtual person's voice drive.

[0101] S404: adjusting the first driving parameter of each core expression base in the core expression base set according to the target emotion to obtain a second driving parameter.

[0102] After obtaining the core expression base set I, the first driving parameters of all expression bases determined for each audio clip and the intensity of the target emotion are combined Correct the first driving parameter of the core expression base in the core expression base set I. For the specific correction process, see Figure 7 , mainly includes the following steps:

[0103] S4041: Add a control factor for the intensity of the target emotion.

[0104] Among them, the control coefficient is used to control the intensity of the target emotion. The larger the control coefficient, the stronger the expression effect of the target emotion.

[0105] Optionally, the control coefficient has a value range of [0, 2], which can be set according to the actual application scenario. Different control coefficients are used in different scenarios, and the embodiment of the present application does not impose any restrictive requirements. By default, its value is 1.

[0106] S4042: For each core expression base in the core expression base set, determine the influence value of the target emotion to which the control coefficient is added on the first driving parameter of the core expression base.

[0107] S4043: Superimpose the influence value and the first driving parameter of the core expression base to obtain the second driving parameter of the core expression base.

[0108] Among them, the correction formula of the core expression base parameters is expressed as follows:

[0109]

[0110] Where j∈I,p j represents the first driving parameter of the jth core expression base, represents the intensity of the target emotion, δ represents the control coefficient of the intensity of the target emotion, and p′ j Represents the second driving parameter of the jth core expression base after correction.

[0111] In an optional embodiment, for each core expression base, after determining the second driving parameter after correction of the core expression base, the second driving parameter is compared with a preset parameter upper limit. If the second driving parameter is greater than the preset parameter upper limit, the preset parameter upper limit is used as the second driving parameter of the core expression base, otherwise, the second driving parameter is kept unchanged. By setting a parameter upper limit for the core expression base, the expression of the target emotion is not overly exaggerated, thereby improving the authenticity of the emotional expression of the 3D virtual human during voice drive.

[0112] In the embodiments of the present application, for the face of a 3D virtual human constructed by a parametric model, the driving parameters of the core expression base corresponding to the target emotion are corrected, thereby achieving precise movement control of the expression bases associated with the head posture, cheek muscles, eyebrows, eyeballs, wrinkles, etc. of the 3D virtual human in three-dimensional space and lip synchronization of the mouth, making the effect of the 3D virtual human voice driving more natural and realistic; at the same time, these core expression bases have fewer corresponding parameters in the head parametric model (generally less than 100), so the emotional adjustment and facial driving of the 3D virtual human can be achieved with a small number of parameters, and the control and transmission speeds are faster, with better real-time performance.

[0113] S405: According to the second driving parameters of each core expression base and the first driving parameters of other expression bases, a 3D virtual human with emotional expression is obtained and displayed.

[0114] By combining the corrected second driving parameters of each core expression base with the first driving parameters of other expression bases, control parameters of all expression bases of the 3D virtual human head parametric model can be obtained, so as to directly drive the face of the 3D virtual human, obtain a 3D virtual human with emotional expression, and display the animation of the 3D virtual human on the screen.

[0115] In an optional embodiment, the process of driving the facial movement of the 3D virtual human through the driving parameters of the expression base is a process of linearly weighting the driving parameters of each expression base, which is expressed as follows:

[0116]

[0117] Among them, α h represents the driving parameter of the hth expression base, ω hrepresents the weight corresponding to the hth expression base, w is the driving parameter after all expression bases are weighted, and F(w) represents the model driven by all expression bases, that is, the model obtained by performing expression migration based on the basic model in the natural state (such as the model composed of 52 expression bases of the standard Arkit).

[0118] Taking four expression bases as an example, the linear weighted schematic diagram of each expression base is as follows: Figure 8 As shown in , by weighting, new expressions can be obtained. It should be noted that Figure 8 The number of expression bases in is only an example and is not a limitation of the expression bases used for weighting.

[0119] like Fig. 9 As shown, 48 expression models are obtained after the expression migration of the basic model in the natural state composed of 52 expression bases.

[0120] In an embodiment of the present application, the first driving parameters of all expression bases are obtained by using the audio segments after the target audio is segmented, and the target emotion is queried by using the keyword set extracted from the target text after the target audio is converted, and the first driving parameters of the core expression base in the core expression base set corresponding to the target emotion are corrected to achieve adaptive adjustment of the emotion with the voice, thereby enhancing the expression effect of the 3D virtual human emotion; and then combined with the driving parameters of other uncorrected expression bases, the driving parameters of all expression bases of the 3D virtual human's head parametric model are obtained, thereby presenting a more natural and realistic 3D virtual human voice driving effect, while improving the intelligence level of the 3D virtual human.

[0121] Based on the same technical concept, an embodiment of the present application provides a 3D virtual human expression driving system based on keyword matching, which can implement the steps of any of the above-mentioned 3D virtual human expression driving methods based on keyword matching and achieve the same technical effect.

[0122] See also Fig.10 The system includes an audio acquisition module 1001, an audio segmentation module 1002, a text conversion module 1003, an expression base parameter estimation module 1004, a keyword extraction module 1005, a keyword matching module 1006, an expression base parameter correction module 1007 and an expression base driving module 1008, wherein:

[0123] The audio acquisition module 1001 is used to acquire the target audio of the 3D virtual human;

[0124] The audio segmentation module 1002 is used to segment the target audio according to a preset frame rate;

[0125] A text conversion module 1003, used for converting the target audio into a target text;

[0126] An expression base parameter estimation module 1004 is used to determine first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment;

[0127] A keyword extraction module 1005 is used to extract keywords from the target text to obtain a keyword set;

[0128] The keyword matching module 1006 is used to query the preset database according to the keyword set to obtain the target emotion of the 3D virtual human and the core expression base set corresponding to the target emotion; wherein the preset database stores a plurality of pairs of corresponding relationships between keywords and emotions, and each emotion corresponds to a core expression base set;

[0129] An expression base parameter correction module 1007, configured to adjust the first driving parameter of each core expression base in the core expression base set according to the target emotion to obtain a second driving parameter;

[0130] The expression base driving module 1008 is used to obtain and display a 3D virtual human with emotional expression according to the second driving parameters of each core expression base and the first driving parameters of other expression bases.

[0131] Optionally, the keyword matching module 1006 is specifically used for:

[0132] Querying the preset database according to each keyword in the keyword set to obtain candidate emotions corresponding to the keyword;

[0133] Count the frequency of occurrence of each candidate emotion among the obtained candidate emotions;

[0134] The candidate emotion corresponding to the highest frequency is used as the target emotion of the 3D virtual person;

[0135] According to the target emotion, a corresponding core expression base set is obtained.

[0136] Optionally, the keyword extraction module 1005 is specifically used for:

[0137] Segmenting the target text to obtain a combination of multiple words and phrases representing emotions;

[0138] Individual words are removed, and the keyword set is generated using the remaining multiple phrases.

[0139] Optionally, the expression base parameter correction module 1007 is specifically used for:

[0140] Adding a control factor for the intensity of the target emotion;

[0141] For each core expression base in the core expression base set, determining an influence value of the target emotion with the control coefficient added thereto on a first driving parameter of the core expression base;

[0142] The influence value is superimposed on the first driving parameter of the core expression base to obtain the second driving parameter of the core expression base.

[0143] Optionally, the expression base parameter correction module 1007 is further used for:

[0144] For each core expression base, determining whether a second driving parameter of the core expression base is greater than a preset parameter upper limit;

[0145] If it is greater, the preset parameter upper limit is used as the second driving parameter of the core expression base.

[0146] Optionally, the target audio includes any one of text-converted audio and speech recording.

[0147] Optionally, the emotions corresponding to different keywords are the same or different.

[0148] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0149] After introducing the text method and apparatus based on active learning according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0150] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0151] Based on the same inventive concept as the above-mentioned method embodiment, an electronic device is also provided in the embodiment of the present application, which includes but is not limited to desktop computers, laptop computers, wearable devices (such as AR devices, VR devices, etc.), game terminals, etc.

[0152] In an optional embodiment, the structure of the electronic device can be as follows: Fig.11 As shown, it includes a memory 1101, a display 1103 and one or more processors 1102. Among them:

[0153] The memory 1101 is used to store the computer program executed by the processor 1102. The memory 1101 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system and the programs required to run the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory 1101 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1101 may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1101 may be any other medium that can be used to carry or store the desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1101 may be a combination of the above memories.

[0154] The processor 1102 may include one or more central processing units (CPU), GPU or digital processing unit, etc. The processor 1102 is used to implement any of the steps of the above-mentioned 3D virtual human expression driving method based on keyword matching when calling the computer program stored in the memory 1101.

[0155] The display 1103 is used to display a 3D virtual human.

[0156] The specific connection medium between the memory 1101, the display 1103 and the processor 1102 is not limited in the embodiment of the present application. Fig.11 In the embodiment, the memory 1101, the display 1103 and the processor 1102 are connected via a bus 1104. The bus 1104 is connected to the processor 1102 via a bus 1104. Fig.11 The connections between the other components are described with bold lines, which are only for illustration and are not intended to be limiting. The bus 1104 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.11 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0157] It should be noted that Fig.11 This is just an example, and provides the necessary hardware for an electronic device to execute the steps of the 3D virtual human expression driving method based on keyword matching provided in the embodiment of the present application. If not shown, the electronic device may also include conventional hardware such as a speaker, a pickup, a communication interface, and a power supply.

[0158] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete any one of the 3D virtual human expression driving methods based on keyword matching in the aforementioned embodiments.

[0159] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute any one of the 3D virtual human expression driving methods based on keyword matching in the aforementioned embodiments.

[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A 3D virtual human expression driving method based on keyword matching, characterized in that: include: Acquire target audio of the 3D virtual human, segment the target audio according to a preset frame rate, and convert the target audio into target text; Determining first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment; Extract keywords from the target text to obtain a keyword set, and query a preset database according to the keyword set to obtain a target emotion of the 3D virtual person and a core expression base set corresponding to the target emotion; wherein the preset database stores a plurality of pairs of corresponding relationships between keywords and emotions, and each emotion corresponds to a core expression base set; Adjusting the first driving parameter of each core expression base in the core expression base set according to the target emotion to obtain a second driving parameter; According to the second driving parameters of each core expression base and the first driving parameters of other expression bases, a 3D virtual human with emotional expression is obtained and displayed.

2. The method according to claim 1, characterized in that The step of querying a preset database according to the keyword set to obtain the target emotion of the 3D virtual person and the core expression base set corresponding to the target emotion includes: Querying the preset database according to each keyword in the keyword set to obtain candidate emotions corresponding to the keyword; According to the frequency of occurrence of each candidate emotion, the intensity of the corresponding candidate emotion is obtained; The candidate emotion corresponding to the highest intensity is used as the target emotion of the 3D virtual human; According to the target emotion, a corresponding core expression base set is obtained.

3. The method according to claim 1, characterized in that The step of extracting keywords from the target text to obtain a keyword set includes: Segmenting the target text to obtain a combination of multiple words and phrases representing emotions; Individual words are removed, and the keyword set is generated using the remaining multiple phrases.

4. The method according to claim 1, characterized in that The step of adjusting the first driving parameter of each core expression base in the core expression base set according to the target emotion to obtain the second driving parameter includes: Adding a control factor for the intensity of the target emotion; For each core expression base in the core expression base set, determining an influence value of the target emotion with the control coefficient added thereto on a first driving parameter of the core expression base; The influence value is superimposed on the first driving parameter of the core expression base to obtain the second driving parameter of the core expression base.

5. The method according to claim 1, characterized in that After obtaining the second expression parameters of each core expression base, the method further includes: For each core expression base, determining whether a second driving parameter of the core expression base is greater than a preset parameter upper limit; If it is greater, the preset parameter upper limit is used as the second driving parameter of the core expression base.

6. The method according to any one of claims 1 to 5, characterized in that The target audio includes any one of text-converted audio and speech recording.

7. The method according to any one of claims 1 to 5, characterized in that Different keywords may correspond to the same or different emotions.

8. A 3D virtual human expression driving system based on keyword matching, characterized in that: include: An audio acquisition module is used to acquire the target audio of the 3D virtual human; An audio segmentation module, used to segment the target audio according to a preset frame rate; A text conversion module, used for converting the target audio into a target text; An expression base parameter estimation module, used for determining first driving parameters of all expression bases of the 3D virtual human according to each segmented audio segment; A keyword extraction module, used to extract keywords from the target text to obtain a keyword set; A keyword matching module is used to query a preset database according to the keyword set to obtain the target emotion of the 3D virtual person and the core expression base set corresponding to the target emotion; wherein the preset database stores a plurality of pairs of corresponding relationships between keywords and emotions, and each emotion corresponds to a core expression base set; An expression base parameter correction module, used for adjusting the first driving parameter of each core expression base in the core expression base set according to the target emotion to obtain a second driving parameter; The expression base driving module is used to obtain and display a 3D virtual human with emotional expression according to the second driving parameters of each core expression base and the first driving parameters of other expression bases.

9. The system according to claim 8, characterized in that The keyword matching module is specifically used for: Querying the preset database according to each keyword in the keyword set to obtain candidate emotions corresponding to the keyword; Count the frequency of occurrence of each candidate emotion among the obtained candidate emotions; The candidate emotion corresponding to the highest frequency is used as the target emotion of the 3D virtual person; According to the target emotion, a corresponding core expression base set is obtained.

10. An electronic device, characterized in that: It includes a processor, a memory and a display, wherein the display, the memory and the processor are connected via a bus; The display is used to display a 3D virtual person; The memory stores a computer program, and the processor executes the method according to any one of claims 1 to 7.

Images