Voice action synchronization method and device of virtual character, equipment and storage medium

Abstract

The embodiment of the invention discloses a voice action synchronization method and device for a virtual character, equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the voice generation of an answer text, obtaining an answer voice sequence and the attribute information of each phoneme, determining a standard mouth shape identifier of the corresponding phoneme based on the phoneme identifier of each phoneme, and obtaining a voice action synchronization result; generating a mouth shape animation sequence based on the standard mouth shape identifier of each phoneme according to the time range of each phoneme; determining an emotional action sequence and an emotional action starting timestamp corresponding to the emotional label of the answer text, and generating a supplementary animation sequence based on the emotional action sequence and the emotional action starting timestamp; and synchronizing the answer voice sequence, the mouth shape animation sequence and the supplementary animation sequence according to the time sequence to obtain a synchronization relationship, and driving the virtual response character of the user based on the synchronization relationship, the mouth shape animation sequence and the supplementary animation sequence when the answer voice sequence is played, so that the voice action synchronization function of the virtual character is realized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of artificial intelligence, and in particular, to a virtual character speech action synchronization method and device, equipment and a storage medium. BACKGROUND

[0002] At present, interaction with a user can be realized through a virtual character. For example, a car machine carries on a dialogue with a user through a virtual character, that is, the car machine outputs an answer voice to a user question through the virtual character, and controls the mouth of the virtual character to perform simple and repetitive opening and closing actions. However, the virtual character has a single expression when carrying on a dialogue with the user, and the mouth movement is also single, which cannot realize accurate synchronization of speech, mouth movement and expression actions, and affects the user experience. SUMMARY

[0003] Embodiments of the present application provide a virtual character speech action synchronization method, device, equipment and storage medium, which realizes the speech action synchronization function of the virtual character, to solve the problem that accurate synchronization of speech, mouth movement and expression actions cannot be realized in the prior art.

[0004] In a first aspect, embodiments of the present application provide a virtual character speech action synchronization method, which comprises:

[0005] obtaining an answer text corresponding to a question of a user;

[0006] generating speech from the answer text to obtain an answer voice sequence and attribute information of each phoneme, the attribute information including a phoneme identifier and a time range, determining a standard mouth movement identifier of the corresponding phoneme based on the phoneme identifier of each phoneme, and generating a mouth movement animation sequence based on the standard mouth movement identifier of each phoneme according to the time range of each phoneme;

[0007] performing sentiment analysis on the answer text to obtain a sentiment label, determining a sentiment action sequence and a sentiment action start timestamp corresponding to the sentiment label, and generating a supplementary animation sequence based on the sentiment action sequence and the sentiment action start timestamp;

[0008] synchronizing the answer voice sequence, the mouth movement animation sequence and the supplementary animation sequence in time sequence to obtain a synchronization relationship, and driving a virtual answering character of the user based on the synchronization relationship, the mouth movement animation sequence and the supplementary animation sequence when playing the answer voice sequence.

[0009] In the embodiments of the present application, when generating the answer speech sequence corresponding to the answer text, attribute information of each phoneme, i.e., phoneme identifier and time range, can be generated, then the standard mouth shape identifier of the corresponding phoneme is determined based on the phoneme identifier of each phoneme, and the standard mouth shape identifier corresponding to each phoneme is combined in time sequence based on the time range of each phoneme, so that the mouth shape animation sequence corresponding to the answer speech sequence can be accurately generated, so that the mouth shape animation sequence corresponds to the pronunciation mouth shape of the answer speech sequence, instead of the simple and repetitive opening and closing action of the mouth, and then the mouth action of the virtual answering character changes with the answer speech sequence, and the mouth action of the virtual answering character is enriched; then, the emotional label is obtained by performing emotional analysis on the answer text, and the emotional action sequence and the emotional action start timestamp corresponding to the emotional label are determined, then the supplementary animation sequence is further generated, so that the expression action of the virtual answering character can change with the answer speech sequence, and then the emotional action (such as expression action) of the virtual answering character is enriched; then, the answer speech sequence, the mouth shape animation sequence and the supplementary animation sequence are synchronized in time sequence to obtain the synchronization relationship, and when the answer speech sequence is played, the virtual answering character of the user is driven based on the synchronization relationship, the mouth shape animation sequence and the supplementary animation sequence, so that the played speech, the mouth shape of the virtual answering character and the supplementary action (i.e., emotional action) of the virtual answering character are synchronized, and then the accurate synchronization of speech, mouth shape and expression action is realized, thereby improving the use experience of the virtual character for the user.

[0010] In a second aspect, the embodiments of the present application provide a virtual character speech action synchronization device, which comprises:

[0011] The acquisition module is configured to acquire an answer text corresponding to a question of a user;

[0012] The first generation module is configured to perform speech generation on the answer text to obtain an answer speech sequence and attribute information of each phoneme, the attribute information comprising a phoneme identifier and a time range, determine a standard mouth shape identifier of the corresponding phoneme based on the phoneme identifier of each phoneme, and generate a mouth shape animation sequence based on the standard mouth shape identifier corresponding to each phoneme according to the time range of each phoneme;

[0013] The second generation module is configured to perform emotional analysis on the answer text to obtain an emotional label, determine an emotional action sequence and an emotional action start timestamp corresponding to the emotional label, and generate a supplementary animation sequence based on the emotional action sequence and the emotional action start timestamp;

[0014] The synchronization module is configured to synchronize the answer speech sequence, the mouth shape animation sequence and the supplementary animation sequence in time sequence to obtain a synchronization relationship, and drive a virtual answering character of the user based on the synchronization relationship, the mouth shape animation sequence and the supplementary animation sequence when the answer speech sequence is played.

[0015] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0016] at least one processor; and a memory connected with the at least one processor in communication;

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for synchronizing speech actions of a virtual character according to any of the embodiments of the present application.

[0018] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method for synchronizing speech actions of a virtual character according to any of the embodiments of the present application.

[0019] The description of the second aspect, the third aspect and the fourth aspect of the present application can refer to the detailed description of the first aspect, and the beneficial effects of the second aspect, the third aspect and the fourth aspect can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0020] In the present application, the name of the virtual character speech action synchronization device does not constitute a limitation on the device or the function module itself, and in actual implementation, these devices or function modules can appear with other names. As long as the functions of each device or function module are similar to those of the present application, they belong to the scope of the claims of the present application and their equivalents.

[0021] These aspects or other aspects of the present application will be more apparent in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0023] Figure 1 is a flowchart of the method for synchronizing speech actions of a virtual character provided by the embodiments of the present application;

[0024] Figure 2 is another flowchart of the method for synchronizing speech actions of a virtual character provided by the embodiments of the present application;

[0025] Figure 3 is a structural diagram of the virtual character speech action synchronization device provided by the embodiments of the present application;

[0026] Figure 4is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work should belong to the protection scope of the present application.

[0028] It should be noted that the terms “first”, “second”, “target” and “original” and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms “include”, “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] Figure 1 is a flowchart of a virtual character speech action synchronization method provided by an embodiment of the present application. The embodiment can be applied to a scene in which the speech and action of a virtual character need to be synchronized. The virtual character speech action synchronization method provided by the embodiment can be executed by a virtual character speech action synchronization device provided by the embodiment. The device can be implemented in software and / or hardware. In a specific embodiment, the virtual character speech action synchronization device can be integrated in an electronic device, which can be integrated in a vehicle or other device that utilizes a virtual character to interact with a user, for example, the electronic device can be a computer or other device. The execution subject of the method can be an electronic device. Referring to Figure 1 , the virtual character speech action synchronization method of the embodiment includes but is not limited to the following steps:

[0030] S110, obtaining a reply text corresponding to a question of a user.

[0031] The reply text is a text for answering the question of the user in a question and answer scene; the reply text can be a paragraph of text or a sentence of text.

[0032] Specifically, the answer text corresponding to the question of the user can be obtained, that is, the question in a voice version input by the user through voice can be obtained, and the question in the voice version is converted into a question in a text version, or the question in the text version input by the user through a keyboard or the like is obtained, and then the question in the text version is input into a pre-trained question and answer model to obtain the answer text corresponding to the question, wherein the question and answer model is a pre-trained neural network model used to predict the answer corresponding to the question.

[0033] In S120, speech generation is performed on the answer text to obtain an answer speech sequence and attribute information of each phoneme, a standard mouth shape identifier of a corresponding phoneme is determined based on a phoneme identifier of each phoneme, and a mouth shape animation sequence is generated based on the standard mouth shape identifier of each phoneme according to a time range of each phoneme.

[0034] The answer speech sequence is speech corresponding to the answer text. The phoneme is the smallest linear speech unit divided according to the natural attributes of speech, which is a physical entity of pronunciation action and can distinguish different pronunciations from the perspective of voice quality. The attribute information of the phoneme can include a phoneme identifier and a time range. The phoneme identifier is a unique identifier of the phoneme. The time range is the actual pronunciation duration of a single phoneme in the answer speech sequence, and the time range is a relative time range. The time range of the phoneme includes a start time stamp of the phoneme and an end time stamp of the phoneme.

[0035] The standard mouth shape is a visible action or posture of the mouth corresponding to a specific phoneme during pronunciation. The standard mouth shape identifier is a unique identifier of the standard mouth shape. The mouth shape animation sequence is a combination of the standard mouth shapes corresponding to each phoneme in the answer speech sequence.

[0036] Specifically, after obtaining the answer text, speech generation can be performed on the answer text to obtain an answer speech sequence and attribute information of each phoneme, that is, the answer text can be input into a pre-trained speech generation model. At this time, the speech generation model analyzes the answer text, identifies the phoneme sequence corresponding to the answer text and the attribute information of each phoneme, such as the phoneme identifier and the time range, and then generates speech based on the phoneme sequence and the attribute information of each phoneme to obtain the answer speech sequence. The speech generation model is a pre-trained neural network model used to generate speech corresponding to the text.

[0037] Then, the standard mouth shape identifier of the corresponding phoneme can be determined based on the phoneme identifier of each phoneme, for example, for a current phoneme in a plurality of phonemes of the answer speech sequence, the standard mouth shape identifier corresponding to the current phoneme can be obtained by querying a first mapping relationship based on the phoneme identifier of the current phoneme, wherein the first mapping relationship includes the corresponding relationship between the phoneme identifier and the standard mouth shape identifier, and then each phoneme is traversed to repeat the above process to obtain the standard mouth shape identifier corresponding to each phoneme in the answer speech sequence.

[0038] Afterwards, the mouth animation sequence can be generated based on the standard mouth shape identifier corresponding to each phoneme according to the time range of each phoneme, that is, the standard mouth shape identifier of each phoneme can be mapped to the corresponding Blendshape parameter set, that is, one phoneme corresponds to one Blendshape parameter set to express the mouth deformation required by the pronunciation of different phonemes, and then the Blendshape parameter set corresponding to each phoneme is interpolated and combined in time sequence based on the time range of each phoneme to form the mouth animation sequence.

[0039] Further, the standard mouth shape identifier corresponding Blendshape parameter can be obtained by querying the preset parameter mapping table based on the standard mouth shape identifier, wherein the obtained Blendshape parameter can be one or multiple, so as to obtain the Blendshape parameter set corresponding to the standard mouth shape identifier; wherein the preset parameter mapping table includes the corresponding relationship between the standard mouth shape identifier and the Blendshape parameter; BlendShape is a core technology for realizing dynamic deformation of a model in three-dimensional computer graphics, also known as Morph Target (Morph Target) or Shape Key (Shape Key), and its basic principle is to generate a smooth transition effect by weighted fusion of a basic model and multiple target morphologies; the BlendShape parameter is a variable parameter set used to describe and realize the shape change of a model in computer graphics, mainly used to capture and represent the shape change of a model in different states, for example, the BlendShape parameter can include JawOpen, MouthSmile and MouthPucker, etc.; the value of the BlendShape parameter is a weight value, which is a key factor for controlling the influence degree of each target morphology on the final result, and the value range is [0, 1].

[0040] S130, performing sentiment analysis on the answer text to obtain a sentiment label, determining a sentiment action sequence and a sentiment action start timestamp corresponding to the sentiment label, and generating a supplementary animation sequence based on the sentiment action sequence and the sentiment action start timestamp.

[0041] Wherein, the sentiment label is the sentiment corresponding to the answer text, such as happy, sad, angry and fear, etc. The sentiment action sequence is a series of observable external behavior or action combinations corresponding to the sentiment label; for example, the sentiment action sequence can include facial expressions and / or body movements, such as smiling and dancing, etc.

[0042] The emotional action start timestamp is the start time of driving the virtual answering character to perform the emotional action sequence. The virtual character is a three-dimensional digital image created by computer graphics, animation technology, artificial intelligence and other technical means, which has human appearance, behavior characteristics and certain personality attributes. The virtual answering character is a virtual character that can respond to various questions or topics.

[0043] The supplementary animation sequence is an animation sequence such as other limb actions and expression actions that the virtual answering character needs to perform in addition to the lip animation sequence, which is used to supplement the lip animation sequence to make the behavior of the virtual answering character closer to the real behavior of humans, and facilitate the user to understand the answer text.

[0044] Specifically, after generating the lip animation sequence, the answer text can be subjected to emotional analysis to obtain an emotional label, that is, if the answer text is a text, the answer text is divided, each sentence of the text is divided as a whole to obtain a plurality of subtexts, and each subtext is input into a pre-trained emotional analysis model. At this time, the emotional analysis model analyzes the subtext and outputs the emotional label corresponding to the subtext, that is, one subtext corresponds to one emotional label; if the answer text is a sentence, the answer text is directly input into the pre-trained emotional analysis model, and the emotional analysis model analyzes the answer text and outputs the emotional label of the answer text. The emotional analysis model is a pre-trained neural network model used to predict the emotion corresponding to the text.

[0045] Then, the emotional action sequence corresponding to the emotional label can be determined, for example, the emotional action sequence corresponding to the emotional label can be obtained by querying a preset identifier mapping table based on the emotional label, and obtaining the action sequence corresponding to the emotional action identifier from the pre-stored action sequence set. The preset identifier mapping table includes the corresponding relationship between the emotional label and the emotional action identifier, and the emotional action identifier is the unique identifier of the emotional action.

[0046] Then, the emotional action start timestamp corresponding to the emotional label can be determined, for example, the emotional action start timestamp of the emotional label can be set as the end timestamp of the corresponding answer speech sequence. For example, when the answer text includes subtext 1, subtext 2 and subtext 3, and the answer speech sequence includes sub-speech sequence 1, sub-speech sequence 2 and sub-speech sequence 3, the emotional action start timestamp of the emotional label of the subtext 1 is the end timestamp of the sub-speech sequence 1 corresponding to the subtext 1, the emotional action start timestamp of the emotional label of the subtext 2 is the end timestamp of the sub-speech sequence 2 corresponding to the subtext 2, and the emotional action start timestamp of the emotional label of the subtext 3 is the end timestamp of the sub-speech sequence 3 corresponding to the subtext 3.

[0047] Afterwards, the supplementary animation sequence can be generated based on the emotional action sequence and the emotional action start timestamp, for example, if the answer text is a paragraph of text, there are multiple emotional labels, at this time, based on the emotional action start timestamp of each emotional label, the emotional action sequence of each emotional label is combined in chronological order to form the supplementary animation sequence; if the answer text is a sentence of text, there is one emotional label, at this time, the emotional action sequence is determined as the supplementary animation sequence; at the same time, the emotional action start timestamp corresponding to the emotional action sequence is marked in the supplementary animation sequence, so as to drive the virtual answering character to perform the corresponding emotional action sequence at a specific time in the future.

[0048] S140, synchronizing the answer speech sequence, the lip movement animation sequence and the supplementary animation sequence in chronological order to obtain a synchronization relationship, and driving the virtual answering character of the user based on the synchronization relationship, the lip movement animation sequence and the supplementary animation sequence when playing the answer speech sequence.

[0049] The synchronization relationship is a kind of coordinated and consistent, mutually matching association characteristics between the answer speech sequence, the lip movement animation sequence and the supplementary animation sequence in the time dimension, which means that at each time point in the time advancing process, the respective corresponding states (i.e. the vocalization condition of the speech, the presentation form of the lip movement and the action performance of the supplementary animation) of the three sequences can closely cooperate to achieve the effect of natural and smooth overall performance, which conforms to the habit of human communication.

[0050] Specifically, after obtaining the supplementary animation sequence, the answer speech sequence, the lip movement animation sequence and the supplementary animation sequence can be associated according to the timestamp to form a synchronization relationship associated with the timestamp, ensuring that the answer speech sequence, the lip movement animation sequence and the supplementary animation sequence of the same timestamp are mutually synchronized, then the answer speech sequence can be played, and the virtual answering character of the user can be driven to perform the lip movement animation of the same timestamp in the lip movement animation sequence and the supplementary animation of the same timestamp in the supplementary animation sequence based on the synchronization relationship, so that the played speech, the lip movement of the virtual answering character and the supplementary action of the virtual answering character are synchronized.

[0051] It should be noted that the synchronization relationship of S140 is the synchronization relationship of the answer text corresponding to each sentence of the answer text, the lip movement animation sequence and the supplementary animation sequence; if the answer text is a paragraph of text, the answer text is divided, each sentence of the text is divided into a whole, and a plurality of subtexts are obtained, then S140 is executed based on each subtext in time sequence, that is, when playing the answer voice sequence corresponding to the first subtext, the virtual answering character of the user is driven based on the synchronization relationship, the lip movement animation sequence and the supplementary animation sequence corresponding to the first subtext (that is, after the virtual answering character performs the supplementary animation sequence corresponding to the first subtext), and then the answer voice sequence corresponding to the next subtext is played, and the virtual answering character of the user is driven based on the synchronization relationship, the lip movement animation sequence and the supplementary animation sequence corresponding to the next subtext.

[0052] For example, if the answer text is "today is a good weather, very suitable for spring outing with friends", the subtext 1 is "today is a good weather", the subtext 2 is "very suitable for spring outing with friends", the emotional label 1 is happy, the emotional label 2 is happy, and the emotional action sequence is smile, first, the answer voice sequence corresponding to the subtext 1 is played, when the voice "today" is played, the virtual answering character performs the lip movement animation sequence corresponding to "today", when the voice "weather" is played, the virtual answering character performs the supplementary animation sequence corresponding to smile, and then after the virtual answering character performs the supplementary animation sequence corresponding to smile, the answer voice sequence corresponding to the subtext 2 is played.

[0053] The technical solution of the embodiment of the application can generate attribute information of each phoneme, i.e., a phoneme identifier and a time range, when generating an answer speech sequence corresponding to the answer text, then determine a standard mouth shape identifier of each phoneme based on the phoneme identifier of each phoneme, and combine the standard mouth shape identifier corresponding to each phoneme in time sequence based on the time range of each phoneme, so as to accurately generate a mouth shape animation sequence corresponding to the answer speech sequence, so that the mouth shape animation sequence corresponds to the pronunciation mouth shape of the answer speech sequence, instead of simple and repetitive opening and closing actions of the mouth, and thus the mouth action of the virtual answering character changes with the answer speech sequence, and the mouth action of the virtual answering character is enriched; then, the emotion label is obtained by performing emotion analysis on the answer text, and the emotion action sequence and the emotion action start timestamp corresponding to the emotion label are determined, then a supplementary animation sequence is further generated, so that the expression action of the virtual answering character can change with the answer speech sequence, and thus the emotion action (such as the expression action) of the virtual answering character is enriched; then, the answer speech sequence, the mouth shape animation sequence and the supplementary animation sequence are synchronized in time sequence to obtain a synchronization relationship, and the virtual answering character of the user is driven based on the synchronization relationship, the mouth shape animation sequence and the supplementary animation sequence when the answer speech sequence is played, so as to ensure that the played speech, the mouth shape of the virtual answering character and the supplementary action (i.e., the emotion action) of the virtual answering character are synchronized, and thus the accurate synchronization of the speech, the mouth shape and the expression action is realized, and thus the use experience of the virtual character for the user is improved.

[0054] A virtual character speech action synchronization method provided in the embodiment of the application is further described below, Figure 2 is another flowchart of the virtual character speech action synchronization method provided in the embodiment of the application. The embodiment of the application is optimized on the basis of the above-mentioned embodiments. Referring to Figure 2 , the method of the embodiment includes but is not limited to the following steps:

[0055] S201, obtaining an answer text corresponding to a question of a user.

[0056] S202, performing speech generation on the answer text to obtain an answer speech sequence and attribute information of each phoneme.

[0057] The attribute information of the phoneme can further include a sound intensity.

[0058] S203, classifying the omission attribute of the phoneme based on the time range and the sound intensity of each phoneme by using a classification model to obtain an omission classification result corresponding to the phoneme.

[0059] The omission attribute of the phoneme is used to represent whether the mouth movement sequence corresponding to the phoneme is omitted when the answer speech sequence is broadcast, so that the generated mouth movement animation sequence is closer to the real mouth movement of a human. The classification model is a pre-trained neural network model used to classify the omission attribute of the phoneme. The omission classification result is obtained by classifying the omission attribute of the phoneme by the classification model, and includes omission and non-omission.

[0060] Specifically, after obtaining the answer speech sequence and the attribute information of each phoneme, for a current phoneme in the plurality of phonemes of the answer speech sequence, the time range and the sound intensity of the current phoneme can be input into the classification model, and at this time, the classification model can classify the omission attribute of the current phoneme based on the time range and the sound intensity of the current phoneme, and output the omission classification result corresponding to the current phoneme, so that the omission classification result of the phoneme with a short time range and / or a weak sound intensity in the answer speech sequence is omission, and the omission classification result of the phoneme with a long time range and a strong sound intensity is non-omission. The mouth movement that is not prominent in pronunciation in the answer speech sequence can be omitted, so that the generated mouth movement animation sequence is closer to the real mouth movement of a human.

[0061] S204, determining the standard mouth movement identifier of the corresponding phoneme based on the omission classification result and the phoneme identifier of each phoneme.

[0062] Specifically, after obtaining the omission classification result of each phoneme, it can be determined whether the phoneme is pronounced by a human when broadcasting the answer speech sequence according to the omission classification result of the phoneme, and then the standard mouth movement identifier corresponding to the phoneme is determined based on the phoneme identifier of the pronounced phoneme.

[0063] Further, for a current phoneme in the plurality of phonemes of the answer speech sequence, when the omission classification result of the current phoneme is omission, it indicates that the current phoneme is not pronounced by a human when broadcasting the answer speech sequence, and at this time, it is not necessary to determine the standard mouth movement identifier corresponding to the current phoneme; when the omission classification result of the current phoneme is non-omission, it indicates that the current phoneme is pronounced by a human when broadcasting the answer speech sequence, and at this time, the standard mouth movement identifier corresponding to the current phoneme can be obtained by querying the first mapping table based on the phoneme identifier of the current phoneme. Then, each phoneme in the answer speech sequence is traversed, and the above process is repeated to obtain the standard mouth movement identifier corresponding to the pronounced phoneme. By using the omission classification result, the phonemes that need to be pronounced in the answer speech sequence can be accurately identified, and then the standard mouth movement identifier corresponding to the pronounced phoneme is determined, which can improve the calculation efficiency and reduce the implementation complexity, and provides an accurate data basis for generating the mouth movement animation sequence subsequently, and then the mouth movement that is not prominent in pronunciation in the answer speech sequence can be omitted, so that the generated mouth movement animation sequence is closer to the real mouth movement of a human, thereby improving the mouth movement simulation reality of the virtual answering character.

[0064] S205, generating a mouth shape animation sequence based on the standard mouth shape corresponding to each phoneme according to the time range of each phoneme.

[0065] S206, performing sentiment analysis on the answer text to obtain a sentiment label.

[0066] S207, querying a second mapping table based on the sentiment label to obtain a sentiment action sequence corresponding to the sentiment label.

[0067] The second mapping table can include a corresponding relationship between the sentiment label and the sentiment action sequence.

[0068] Specifically, if the answer text is a text, one answer text corresponds to one sentiment label, i.e., there are multiple sentiment labels, and the sentiment action sequence obtained thereby has multiple, i.e., one sentiment label corresponds to one sentiment action sequence; if the answer text is a text, there is one sentiment label, and the sentiment action sequence obtained thereby has one.

[0069] S208, identifying the sentiment action sequence to determine whether the sentiment action sequence belongs to a mouth action, and obtaining an identification result.

[0070] The identification result is the result of identifying whether the action in the sentiment action sequence belongs to a mouth action.

[0071] Specifically, after obtaining the sentiment action sequence, the sentiment actions in the sentiment action sequence can be analyzed to identify which part of the action, such as hand action, head action, eye action, shoulder action or mouth action, the sentiment actions belong to. If the sentiment actions belong to the mouth action, it is determined that the identification result is the mouth action; if the sentiment actions do not belong to the mouth action, it is determined that the identification result is not the mouth action. It should be noted that in the embodiments of the present application, the sentiment actions in a sentiment action sequence belong to the action of one part.

[0072] S209, determining a sentiment action start time stamp corresponding to the sentiment label based on the identification result.

[0073] Specifically, after obtaining the identification result, the sentiment action start time stamp corresponding to the sentiment label can be determined according to the identification result of the sentiment action sequence, so as to drive the virtual answering character to trigger and execute the corresponding sentiment action sequence at the sentiment action start time stamp.

[0074] Further, when the recognition result is a mouth action, it indicates that the emotional action sequence will conflict with the lip animation sequence, and in this case, the emotional action start timestamp can be determined as the end timestamp of the answer speech sequence, where the answer speech sequence is the answer speech sequence of a sentence of the answer text; if the answer text is a paragraph of text, the emotional action start timestamp of the emotional label is the end timestamp of the answer speech sequence corresponding to the sentence of the answer text corresponding to the emotional label.

[0075] When the recognition result is not a mouth action, it indicates that the emotional action sequence will not conflict with the lip animation sequence, and in this case, the emotional action start timestamp can be determined as the middle timestamp of the answer speech sequence, where the answer speech sequence is the answer speech sequence of a sentence of the answer text; if the answer text is a paragraph of text, the emotional action start timestamp of the emotional label is the middle timestamp of the answer speech sequence corresponding to the sentence of the answer text; where the middle timestamp is a pre-set timestamp, which is used to represent a timestamp at a middle node of the answer speech sequence, and can be adjusted and set according to actual needs, for example, the middle timestamp can be a timestamp of a one-third node or a one-half node of the answer speech sequence.

[0076] For example, when the answer text includes subtext 1, subtext 2 and subtext 3, the answer speech sequence includes sub-speech sequence 1, sub-speech sequence 2 and sub-speech sequence 3, and the recognition result of all subtexts of the answer text is not a mouth action, the emotional action start timestamp of the emotional label of the subtext 1 is the middle timestamp of the sub-speech sequence 1 corresponding to the subtext 1, the emotional action start timestamp of the emotional label of the subtext 2 is the middle timestamp of the sub-speech sequence 2 corresponding to the subtext 2, and the emotional action start timestamp of the emotional label of the subtext 3 is the middle timestamp of the sub-speech sequence 3 corresponding to the subtext 3.

[0077] In the embodiments of the present application, by determining whether the recognition result is a mouth action, the emotional action start timestamp is determined as the end timestamp of the answer speech sequence or the middle timestamp of the answer speech sequence, which can improve the calculation efficiency, reduce the implementation complexity, avoid the conflict between the emotional action sequence and the lip action sequence, and further improve the driving accuracy and efficiency of the lip animation sequence and the supplementary animation sequence.

[0078] S210, generating a supplementary animation sequence based on the emotional action sequence and the emotional action start timestamp.

[0079] The supplementary animation sequence can include an emotional animation sequence and an explanation animation sequence; the emotional animation sequence is a sequence of actions for expressing emotions; the explanation animation sequence is a sequence of actions for expressing the explanation effect of the answer text, for example, the explanation animation sequence can belong to hand actions, etc.

[0080] Specifically, after obtaining the emotional action start timestamp, the emotional animation sequence can be generated based on the emotional action sequence and the emotional action start timestamp. For example, if the answer text is a paragraph of text, there are multiple emotional tags. At this time, based on the emotional action start timestamp of each emotional tag, the emotional action sequence of each emotional tag is combined in chronological order to form an emotional animation sequence. If the answer text is a sentence of text, there is one emotional tag. At this time, the emotional action sequence is determined as the emotional animation sequence. At the same time, the emotional action start timestamp corresponding to the emotional action sequence in the emotional animation sequence is marked, so as to drive the virtual answering character to perform the corresponding emotional action sequence at a specific time subsequently.

[0081] Then, the answer text can be subjected to semantic analysis to obtain a keyword. Then, it is determined whether the keyword corresponding explanation action is triggered based on the question and answer scene and the keyword of the answer text. When it is determined that the keyword corresponding explanation action is triggered, the keyword corresponding explanation animation sequence is generated. The start timestamp of the explanation animation sequence is the start timestamp of the answer voice sequence.

[0082] Specifically, the answer text can be subjected to semantic analysis, and it is determined whether the answer text contains a word in a preset keyword list. If the answer text contains a word in the preset keyword list, the word is determined as the keyword of the answer text. Then, the previous question and answer text of the answer text is obtained, and the previous question and answer text is subjected to semantic analysis to determine the question and answer scene of the answer text. Then, it is determined whether the keyword and the question and answer scene belong to a trigger explanation relationship, i.e., the keyword is queried based on a preset trigger relationship to obtain the trigger scene corresponding to the keyword. If the question and answer scene is consistent with the trigger scene, it is determined that the keyword and the question and answer scene belong to the trigger explanation relationship, and the keyword corresponding explanation action is determined to be triggered. If the question and answer scene is not consistent with the trigger scene, it is determined that the keyword and the question and answer scene do not belong to the trigger explanation relationship, and the keyword corresponding explanation action is determined not to be triggered. The preset keyword list is used to save the words corresponding to the explanation action. The preset trigger relationship includes the corresponding relationship between the keyword and the trigger scene.

[0083] When it is determined that the keyword corresponding explanation action is not triggered, the explanation animation sequence does not need to be determined, and the supplementary animation sequence only includes the emotional animation sequence.

[0084] When determining the explanation action corresponding to the trigger keyword, the explanation action sequence corresponding to the keyword can be obtained from the pre-saved action sequence set, that is, one keyword corresponds to one explanation action sequence, and in the embodiment of the present application, it is defaulted that one sentence of answer text corresponds to one keyword, that is, one sentence of answer text corresponds to one explanation action sequence; then the starting timestamp corresponding to the explanation action sequence is determined to be the starting timestamp of the answer voice sequence, and the answer voice sequence here is the answer voice sequence of one sentence of answer text; if the answer text is a paragraph of text, the starting timestamp of the explanation action sequence is determined to be the starting timestamp of the answer voice sequence corresponding to the sentence of answer text; then the explanation animation sequence corresponding to the keyword is generated based on the explanation action sequence and its corresponding starting timestamp, and the starting timestamp corresponding to the explanation action sequence is marked in the explanation animation sequence, and the answer voice sequence of one sentence of answer text corresponds to one explanation animation sequence.

[0085] In an embodiment of the present application, an emotional animation sequence is generated based on the emotional action sequence and the emotional action start timestamp, which can enrich the emotional actions of the virtual responding character; when the explanation action corresponding to the keyword of the answer text is determined to be triggered in the question-and-answer scenario based on the answer text, the explanation animation sequence corresponding to the keyword is generated, which can avoid arbitrarily driving the virtual responding character to perform the explanation action, thereby improving the generation accuracy of the explanation animation sequence.

[0086] S211. Synchronize the answer voice sequence, lip animation sequence, and supplementary animation sequence in chronological order to obtain a synchronization relationship, and when playing the answer voice sequence, drive the user's virtual answer character based on the synchronization relationship, lip animation sequence, and supplementary animation sequence.

[0087] Optionally, when the length of the answer voice sequence is greater than the length of the explanation animation sequence, the way to drive the user's virtual response character based on the explanation animation sequence is a circular drive, that is, when the length of the answer voice sequence is greater than the length of the corresponding explanation animation sequence, it indicates that the virtual response character has completed the explanation animation sequence, but the corresponding answer voice sequence has not yet been played. At this time, the end timestamp of the explanation animation sequence can be used as the starting timestamp, and the virtual response character can continue to be driven based on the explanation animation sequence until the corresponding answer voice sequence is finished playing.

[0088] When the length of the answer voice sequence is less than the length of the corresponding explanation animation sequence, the end timestamp of the explanation animation sequence is the end timestamp of the corresponding answer voice sequence.

[0089] Optionally, when the emotional action sequence belongs to a mouth action and the answer text is a paragraph of text, that is, the emotional action start timestamp is the end timestamp of the answer voice sequence, at this time, the mouth shape action sequence of the last phoneme in the current sentence answer text and the emotional action sequence corresponding to the current sentence answer text are interpolated and combined, and the emotional action sequence corresponding to the current sentence answer text and the mouth shape action sequence of the first phoneme in the next sentence answer text are interpolated and combined, so as to make the mouth action smooth transition.

[0090] Optionally, in the case where the emotional action start timestamp is the middle timestamp of the answer voice sequence, if the emotional action sequence and the explanation action sequence belong to the same part of the action, that is, the emotional action sequence and the explanation action sequence conflict, when the emotional action start timestamp is reached, the virtual answering character is preferentially driven based on the emotional animation sequence, and after the virtual answering character performs the emotional animation sequence, the virtual answering character is then continued to be driven based on the explanation animation sequence, while interpolation processing is performed to achieve a smooth transition effect.

[0091] The technical scheme of the embodiment of the present application can obtain the answer text corresponding to the user's question, generate speech for the answer text, obtain the answer voice sequence and the attribute information of each phoneme, then classify the omission attribute of each phoneme based on the time range and sound intensity of each phoneme using a classification model, obtain the omission classification result corresponding to the phoneme, and determine the standard mouth shape identifier of the corresponding phoneme based on the omission classification result and the phoneme identifier of each phoneme. The mouth shape that is not prominent in pronunciation in the answer voice sequence can be omitted, providing an accurate data basis for subsequent generation of mouth shape animation sequence, and further making the generated mouth shape animation sequence closer to the real mouth shape of humans, thereby improving the mouth shape simulation reality of the virtual answering character. Then, the mouth shape animation sequence is generated based on the standard mouth shape identifier corresponding to each phoneme according to the time range of each phoneme, the answer text is analyzed for emotion, and an emotional label is obtained, then the second mapping table is queried based on the emotional label to obtain the emotional action sequence corresponding to the emotional label, which can improve the calculation efficiency and reduce the implementation complexity, providing an accurate data basis for subsequent determination of the recognition result. The emotional action sequence is then recognized to determine whether the emotional action sequence belongs to a mouth action, and a recognition result is obtained, and the emotional action start timestamp corresponding to the emotional label is determined based on the recognition result, which can avoid conflicts between the emotional action sequence and the mouth shape action sequence, providing an accurate data basis for subsequent generation of the supplementary animation sequence, and further improving the driving accuracy and efficiency of the mouth shape animation sequence and the supplementary animation sequence.

[0092] Afterwards, a supplementary animation sequence is generated based on the emotional action sequence and the emotional action start timestamp, the answer speech sequence, the lip animation sequence and the supplementary animation sequence are synchronized in time sequence to obtain a synchronization relationship, and the virtual answering character of the user is driven based on the synchronization relationship, the lip animation sequence and the supplementary animation sequence when the answer speech sequence is played, which can ensure that the played speech, the lip of the virtual answering character and the supplementary action (i.e. the emotional action and the explanation action) of the virtual answering character are synchronized, and thus the precise synchronization of the speech, the lip, the expression action and the explanation action is realized, the emotional action and the explanation action of the virtual answering character are enriched, and thus the use experience of the virtual character by the user is improved.

[0093] Figure 3 is a structural schematic diagram of a virtual character speech action synchronization device provided by the embodiment of the present application, referring to Figure 3 The virtual character speech action synchronization device can include:

[0094] The acquisition module 310 is configured to acquire an answer text corresponding to a question of a user.

[0095] The first generation module 320 is configured to perform speech generation on the answer text to obtain an answer speech sequence and attribute information of each phoneme, the attribute information including a phoneme identifier and a time range, determine a standard lip shape identifier of a corresponding phoneme based on the phoneme identifier of each phoneme, and generate a lip animation sequence based on the standard lip shape identifier of each phoneme corresponding to each phoneme according to the time range of each phoneme.

[0096] The second generation module 330 is configured to perform emotional analysis on the answer text to obtain an emotional label, determine an emotional action sequence and an emotional action start timestamp corresponding to the emotional label, and generate a supplementary animation sequence based on the emotional action sequence and the emotional action start timestamp.

[0097] The synchronization module 340 is configured to synchronize the answer speech sequence, the lip animation sequence and the supplementary animation sequence in time sequence to obtain a synchronization relationship, and drive a virtual answering character of the user based on the synchronization relationship, the lip animation sequence and the supplementary animation sequence when the answer speech sequence is played.

[0098] In an embodiment, the attribute information further includes a sound intensity, and the first generation module 320 determines the standard lip shape identifier of the corresponding phoneme based on the phoneme identifier of each phoneme, including:

[0099] The omission attribute of the phoneme is classified based on the time range and the sound intensity of each phoneme by using a classification model to obtain an omission classification result of the corresponding phoneme.

[0100] The standard lip shape identifier of the corresponding phoneme is determined based on the omission classification result and the phoneme identifier of each phoneme.

[0101] In an embodiment, the first generation module 320 determines the standard mouth shape identifier of the corresponding phoneme based on the omission classification result and the phoneme identifier of each phoneme, including:

[0102] For each phoneme, when the omission classification result of the current phoneme is not omitted, the standard mouth shape identifier corresponding to the current phoneme is obtained by querying the first mapping table based on the phoneme identifier of the current phoneme, and the first mapping table includes the corresponding relationship between the phoneme identifier and the standard mouth shape identifier.

[0103] In an embodiment, the second generation module 330 determines the emotional action sequence corresponding to the emotional label and the emotional action start timestamp, including:

[0104] Querying the second mapping table based on the emotional label to obtain the emotional action sequence corresponding to the emotional label, and the second mapping table includes the corresponding relationship between the emotional label and the emotional action sequence;

[0105] Identifying the emotional action sequence to determine whether the emotional action sequence belongs to the mouth action, and obtaining an identification result;

[0106] Determining the emotional action start timestamp corresponding to the emotional label based on the identification result.

[0107] In an embodiment, the second generation module 330 determines the emotional action start timestamp corresponding to the emotional label based on the identification result, including:

[0108] When the identification result is the mouth action, the emotional action start timestamp is determined as the end timestamp of the answer voice sequence;

[0109] When the identification result is not the mouth action, the emotional action start timestamp is determined as the intermediate timestamp of the answer voice sequence.

[0110] In an embodiment, the supplementary animation sequence includes an emotional animation sequence and an explanation animation sequence, and the second generation module 330 generates the supplementary animation sequence based on the emotional action sequence and the emotional action start timestamp, including:

[0111] Generating the emotional animation sequence based on the emotional action sequence and the emotional action start timestamp;

[0112] Performing semantic analysis on the answer text to obtain a keyword;

[0113] Determining whether to trigger the explanation action corresponding to the keyword based on the question and answer scene of the answer text and the keyword, and generating the explanation animation sequence corresponding to the keyword when it is determined to trigger the explanation action corresponding to the keyword.

[0114] In one embodiment, the starting timestamp of the explanation animation sequence in the second generation module 330 is the starting timestamp of the answer voice sequence. When the length of the answer voice sequence in the synchronization module 340 is greater than the length of the explanation animation sequence, the method of driving the user's virtual response character based on the explanation animation sequence is a circular drive.

[0115] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0116] The device for synchronizing the speech and movement of a virtual character provided in this embodiment can be applied to the method for synchronizing the speech and movement of a virtual character provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0117] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 4 A block diagram of an exemplary electronic device 11 suitable for implementing embodiments of the present application is shown. Figure 4 The electronic device 11 shown is only an example and should not bring any limitation to the functions and scope of use of this embodiment.

[0118] like Figure 4 As shown, electronic device 11 is implemented as a general-purpose computing electronic device. Components of electronic device 11 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 connecting various system components (including system memory 28 and processing unit 16).

[0119] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0120] The electronic device 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 11, including volatile and non-volatile media, removable and non-removable media.

[0121] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 4 Not shown, often called a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present application.

[0122] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0123] The electronic device 11 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the electronic device 11, and / or any device that enables the electronic device 11 to communicate with one or more other computing devices (e.g., a network card and a modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 11 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20.

[0124] like Figure 4 As shown, the network adapter 20 communicates with other modules of the electronic device 11 via the bus 18. ​ Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] The processing unit 16 performs various function applications and page displays by running programs stored in the system memory 28, such as implementing a voice action synchronization method of a virtual character according to any of the embodiments of the present application.

[0126] The embodiments of the present application provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement a voice action synchronization method of a virtual character according to any of the embodiments of the present application.

[0127] The computer storage medium of the embodiments can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0128] The computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a computer readable program code is carried. Such a propagated data signal can take on many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can be used to carry or store a program for use by or in connection with an instruction execution system, apparatus or device.

[0129] The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, cable, RF, etc., or any suitable combination thereof.

[0130] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0131] Those skilled in the art will appreciate that the modules or steps of the present application described above can be implemented using general computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and can be implemented using computer device executable program codes, which can be stored in storage devices and executed by computing devices, or can be implemented by manufacturing each module into an integrated circuit module or manufacturing multiple modules or steps into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0132] In addition, the acquisition, storage, use, processing and the like of data in the technical solutions of the present application comply with relevant provisions of laws and regulations.

[0133] Note that the above are only preferred embodiments of the present application and the technical principles applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the inventive concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for synchronizing a virtual character's voice and action, characterized in that: The method comprises: Get the answer text corresponding to the user's question; Performing speech generation on the answer text to obtain an answer speech sequence and attribute information of each phoneme, the attribute information including a phoneme identifier and a time range, determining a standard lip shape identifier corresponding to each phoneme based on the phoneme identifier of each phoneme, and generating a lip shape animation sequence based on the standard lip shape identifier corresponding to each phoneme according to the time range of each phoneme; Performing sentiment analysis on the answer text to obtain a sentiment tag, determining an emotion action sequence and an emotion action start timestamp corresponding to the emotion tag, and generating a supplementary animation sequence based on the emotion action sequence and the emotion action start timestamp; The answer voice sequence, the lip animation sequence and the supplementary animation sequence are synchronized in chronological order to obtain a synchronization relationship, and when playing the answer voice sequence, the user's virtual answer character is driven based on the synchronization relationship, the lip animation sequence and the supplementary animation sequence.

2. The method for synchronizing the voice and movement of a virtual character according to claim 1, wherein: The attribute information further includes sound intensity. The determining of the standard lip shape identifier of the corresponding phoneme based on the phoneme identifier of each phoneme includes: Using the classification model, classifying the omission attributes of the phonemes based on the time range and sound intensity of each phoneme, and obtaining an omission classification result of the corresponding phoneme; A standard lip shape identifier of the corresponding phoneme is determined based on the omitted classification result and the phoneme identifier of each phoneme.

3. The method for synchronizing the voice and movement of a virtual character according to claim 2, wherein: The determining of the standard lip shape identifier of the corresponding phoneme based on the omitted classification result and the phoneme identifier of each phoneme includes: For the current phoneme in each of the phonemes, when the omission classification result of the current phoneme is not omitted, a first mapping table is queried based on the phoneme identifier of the current phoneme to obtain the standard lip shape identifier corresponding to the current phoneme, and the first mapping table includes the correspondence between the phoneme identifier and the standard lip shape identifier.

4. The method for synchronizing the voice and movement of a virtual character according to claim 1, wherein: The determining of the emotion action sequence and the emotion action start timestamp corresponding to the emotion tag includes: Based on the emotion label, a second mapping table is searched to obtain an emotion action sequence corresponding to the emotion label, wherein the second mapping table includes a correspondence between the emotion label and the emotion action sequence; Identifying the emotion action sequence, determining whether the emotion action sequence belongs to a mouth action, and obtaining an identification result; The emotional action start timestamp corresponding to the emotional tag is determined based on the recognition result.

5. The method for synchronizing the voice and movement of a virtual character according to claim 4, wherein: The determining, based on the recognition result, a start timestamp of the emotional action corresponding to the emotional tag, includes: When the recognition result is a mouth movement, determining the start timestamp of the emotional movement as the end timestamp of the answering speech sequence; When the recognition result is not a mouth movement, the emotional movement start timestamp is determined to be a middle timestamp of the answering speech sequence.

6. The method for synchronizing the voice and movement of a virtual character according to claim 1, wherein: The supplementary animation sequence includes an emotional animation sequence and an explanation animation sequence, and the generating of the supplementary animation sequence based on the emotional action sequence and the emotional action start timestamp includes: generating an emotional animation sequence based on the emotional action sequence and the emotional action start timestamp; Performing semantic analysis on the answer text to obtain keywords; Based on the question-answering scenario of the answer text and the keyword, it is determined whether to trigger the explanation action corresponding to the keyword, and when it is determined that the explanation action corresponding to the keyword is triggered, an explanation animation sequence corresponding to the keyword is generated.

7. The method for synchronizing the voice and movement of a virtual character according to claim 6, wherein: The starting timestamp of the explanation animation sequence is the starting timestamp of the answer voice sequence. When the length of the answer voice sequence is greater than the length of the explanation animation sequence, the method of driving the user's virtual answer character based on the explanation animation sequence is circular driving.

8. A device for synchronizing a virtual character's voice and action, characterized in that: The device comprises: The acquisition module is used to obtain the answer text corresponding to the user's question; a first generating module configured to perform speech generation on the answer text to obtain an answer speech sequence and attribute information of each phoneme, the attribute information including a phoneme identifier and a time range, determine a standard lip shape identifier corresponding to each phoneme based on the phoneme identifier, and generate a lip shape animation sequence based on the standard lip shape identifier corresponding to each phoneme according to the time range of each phoneme; A second generation module is configured to perform sentiment analysis on the answer text to obtain a sentiment tag, determine a sentiment action sequence and a sentiment action start timestamp corresponding to the sentiment tag, and generate a supplementary animation sequence based on the sentiment action sequence and the sentiment action start timestamp; A synchronization module is used to synchronize the answer voice sequence, the lip animation sequence and the supplementary animation sequence in chronological order to obtain a synchronization relationship, and when playing the answer voice sequence, drive the user's virtual answer character based on the synchronization relationship, the lip animation sequence and the supplementary animation sequence.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the voice and action synchronization method of the virtual character as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for synchronizing the voice and action of a virtual character as described in any one of claims 1 to 7 is implemented.

Images