Digital human driving method and device, electronic equipment and storage medium

Abstract

The invention relates to a digital human driving method and device, electronic equipment and a storage medium. The method comprises the following steps: receiving an instruction in a text form input by a user; responding to the received instruction in the text form, and utilizing a large model to obtain a reply text for replying the instruction in the text form; wherein the reply text carries a driving mark, and the driving mark is used for indicating the position, where the digital human needs to generate the action and / or expression, in the reply text; the large model is obtained through pre-training; generating an audio based on the reply text, and obtaining an audio playing time point corresponding to the driving mark; and playing the audio, and driving the digital human to generate actions and / or expressions at the audio playing time point corresponding to the driving mark. According to the embodiment of the invention, the efficiency of driving the digital human is improved, and the real-time performance is good.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a digital human driving method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of digital humans are becoming more and more extensive, especially in real-time dialogue and text broadcasting. In real-time dialogue scenarios, digital humans can serve as intelligent customer service to provide people with 24-hour services and solve various problems; in virtual scenarios such as the metaverse, digital humans can also serve as virtual characters to communicate and interact with users in real time; in text broadcasting scenarios, digital humans can be used in video creation, cartoon production, and games. Through the lip shape, expression, and movement of digital humans, the text content can be presented more vividly, improving the audience's viewing experience.

[0003] For the action and expression driving of digital humans, the current general practice is for animators to pre-produce animation assets, and then use additional algorithms to detect whether there are elements in text or voice that trigger actions and expressions during real-time operation, thereby triggering animation assets. However, this additional detection algorithm requires more computing power, has low efficiency, and poor real-time performance. Summary of the invention

[0004] The present disclosure proposes a digital human driving technology solution.

[0005] According to one aspect of the present disclosure, a digital human driving method is provided, comprising:

[0006] Receive text-based instructions from the user;

[0007] In response to receiving the instruction in text form, using the large model to obtain a reply text for replying to the instruction in text form; wherein the reply text carries a driving mark, and the driving mark is used to indicate a position in the reply text where the digital human needs to generate an action and / or an expression; the large model is obtained by pre-training;

[0008] Generate audio based on the reply text, and obtain the audio playback time point corresponding to the drive mark;

[0009] The audio is played, and at the audio playing time point corresponding to the driving mark, the digital human is driven to generate actions and / or expressions.

[0010] In a possible implementation, generating audio based on the reply text and obtaining the audio playback time point corresponding to the driving mark includes:

[0011] Removing the driving mark in the reply text to obtain the target text;

[0012] Performing a text-to-speech operation on the target text to obtain an audio file and a time point of each phoneme in the audio;

[0013] Based on the phonemes of the text at the position indicated by the driving mark and the time points of each phoneme in the audio, the audio playback time point corresponding to the driving mark is determined.

[0014] In a possible implementation, the large model is trained based on sample data, and the labels in the sample data include action-driven tags; the generation process of the sample data includes:

[0015] Using sample examples containing action-driven tags, a sample generation model based on contextual learning is trained to obtain a sample generation model;

[0016] The sample generation model is used to generate sample data containing action-driven tags.

[0017] In a possible implementation, the large model is trained based on sample data, the labels in the sample data include expression-driven tags, and the generation process of the sample data includes:

[0018] Split the text to obtain text fragments;

[0019] Perform emotion recognition on each text segment to obtain the emotion of each text segment;

[0020] An expression-driven tag corresponding to the emotion is added to each text segment, and a label of sample data containing the expression-driven tag is generated.

[0021] In a possible implementation, the large model is trained based on sample data, and the generation process of the sample data includes:

[0022] Split the text to obtain text fragments;

[0023] Detect emotional keywords in the text fragments to obtain emotional keywords in the text fragments;

[0024] Corresponding expression-driven tags are added to the detected emotional keywords, and labels of sample data containing expression-driven tags are generated.

[0025] In a possible implementation, the large model is trained based on sample data, and the generation process of the sample data includes:

[0026] For a text containing an action-driven mark, an expression-driven mark corresponding to the action-driven mark is determined according to a preset action-emotion dictionary, wherein the action-emotion dictionary contains a correspondence between the action-driven mark and the expression-driven mark;

[0027] The expression-driven mark is added to the position corresponding to the action-driven mark in the text to generate a label containing sample data of the expression-driven mark.

[0028] In a possible implementation, the sample data includes sample pairs consisting of instruction samples in text form and reply text samples, wherein the reply text samples are labels containing action-driven tags and / or expression-driven tags, and the training process of the large model includes:

[0029] Taking the instruction sample in text form as input of the large model to obtain the estimated text output by the large model;

[0030] Using the reply text sample as the expected output of the large model, determining the loss between the estimated text and the expected output;

[0031] The large model is tuned using the loss.

[0032] According to one aspect of the present disclosure, a digital human driving device is provided, comprising:

[0033] An instruction receiving unit, used for receiving an instruction in text form input by a user;

[0034] a reply generating unit, configured to, in response to receiving the instruction in the form of text, obtain a reply text for replying to the instruction in the form of text by using a large model; wherein the reply text carries a driving mark, and the driving mark is used to indicate a position in the reply text where the digital human needs to generate an action and / or an expression; and the large model is obtained by pre-training;

[0035] An audio generation unit, used to generate audio based on the reply text and obtain the audio playback time point corresponding to the drive mark;

[0036] The playing unit is used to play the audio and drive the digital human to produce actions and / or expressions at the audio playing time point corresponding to the driving mark.

[0037] In a possible implementation, the audio generation unit is used to:

[0038] Removing the driving mark in the reply text to obtain the target text;

[0039] Performing a text-to-speech operation on the target text to obtain an audio file and a time point of each phoneme in the audio;

[0040] Based on the phonemes of the text at the position indicated by the driving mark and the time points of each phoneme in the audio, the audio playback time point corresponding to the driving mark is determined.

[0041] In a possible implementation, the large model is trained based on sample data, and the labels in the sample data include action-driven tags; the generation process of the sample data includes:

[0042] Using sample examples containing action-driven tags, a sample generation model based on contextual learning is trained to obtain a sample generation model;

[0043] The sample generation model is used to generate sample data containing action-driven tags.

[0044] In a possible implementation, the large model is trained based on sample data, the labels in the sample data include expression-driven tags, and the generation process of the sample data includes:

[0045] Split the text to obtain text fragments;

[0046] Perform emotion recognition on each text segment to obtain the emotion of each text segment;

[0047] An expression-driven tag corresponding to the emotion is added to each text segment, and a label of sample data containing the expression-driven tag is generated.

[0048] In a possible implementation, the large model is trained based on sample data, and the generation process of the sample data includes:

[0049] Split the text to obtain text fragments;

[0050] Detect emotional keywords in the text fragments to obtain emotional keywords in the text fragments;

[0051] Corresponding expression-driven tags are added to the detected emotional keywords, and labels of sample data containing expression-driven tags are generated.

[0052] In a possible implementation, the large model is trained based on sample data, and the generation process of the sample data includes:

[0053] For a text containing an action-driven mark, an expression-driven mark corresponding to the action-driven mark is determined according to a preset action-emotion dictionary, wherein the action-emotion dictionary contains a correspondence between the action-driven mark and the expression-driven mark;

[0054] The expression-driven mark is added to the position corresponding to the action-driven mark in the text to generate a label containing sample data of the expression-driven mark.

[0055] In a possible implementation, the sample data includes sample pairs consisting of instruction samples in text form and reply text samples, wherein the reply text samples are labels containing action-driven tags and / or expression-driven tags, and the training process of the large model includes:

[0056] Taking the instruction sample in text form as input of the large model to obtain the estimated text output by the large model;

[0057] Using the reply text sample as the expected output of the large model, determining the loss between the estimated text and the expected output;

[0058] The large model is tuned using the loss.

[0059] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0060] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0061] In the disclosed embodiment, after receiving the text-form instruction input by the user, in response to receiving the text-form instruction, a reply text for replying to the text-form instruction is obtained by using a pre-trained large model, and the reply text carries a driving mark to indicate the position in the reply text where the digital human needs to produce an action and / or an expression; then audio is generated based on the reply text, and the audio playback time point corresponding to the driving mark is obtained; the audio is played, and the digital human is driven to produce an action or an expression at the audio playback time point corresponding to the driving mark. Thus, the reply text obtained by the pre-trained large model already has a driving mark to indicate the position in the reply text where the digital human needs to produce an action and / or an expression, and no additional algorithm is required to analyze and detect the text output by the large model, so that after the large model outputs the text, the text is directly converted into speech, and the digital human is driven to produce an action or an expression according to the driving mark in the text, so that the result can be displayed to the user in a timely manner, with high efficiency and good real-time performance.

[0062] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0064] Figure 1 A flow chart of a digital human driving method according to an embodiment of the present disclosure is shown.

[0065] Figure 2 A flow chart of adding expression-driven tags according to an embodiment of the present disclosure is shown.

[0066] Figure 3 A block diagram of a digital human driving device according to an embodiment of the present disclosure is shown.

[0067] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0068] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0069] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0070] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0071] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0072] The application scenarios of digital humans are becoming increasingly diverse, especially in real-time dialogue and text broadcasting, which shows great potential. In real-time dialogue scenarios, digital humans can serve as intelligent customer service, provide round-the-clock services, solve various problems, and improve user experience. In virtual environments such as the metaverse, digital humans can serve as virtual characters to communicate and interact with users in real time, enhancing the immersion in the virtual world. In the field of text broadcasting, digital humans are also widely used. Through the lip shape, expression and movement of digital humans, text content can be vividly presented to enhance the audience's viewing experience.

[0073] In interactive scenarios such as dialogue, large models are often used to output the answers required by users. Large models can be neural network models with ultra-large-scale parameters (usually more than one billion). Large models are widely used in human-computer interaction in the field of natural language processing. Large models can be large-scale deep learning models, such as large-scale language models, multimodal models, etc., with large-scale model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters.

[0074] In related technologies, after receiving the answer output by the large model, an additional algorithm is used to detect whether the text of the answer contains words that trigger actions and expressions, and then trigger animation assets when the text is broadcast. However, this additional detection algorithm requires more computing power, has low efficiency, and poor real-time performance.

[0075] In the disclosed embodiment, after receiving the text-form instruction input by the user, in response to receiving the text-form instruction, a reply text for replying to the text-form instruction is obtained by using a pre-trained large model, and the reply text carries a driving mark to indicate the position in the reply text where the digital human needs to produce an action and / or an expression; then audio is generated based on the reply text, and the audio playback time point corresponding to the driving mark is obtained; the audio is played, and the digital human is driven to produce an action or an expression at the audio playback time point corresponding to the driving mark. Thus, the reply text obtained by the pre-trained large model already has a driving mark to indicate the position in the reply text where the digital human needs to produce an action and / or an expression, and no additional algorithm is required to analyze and detect the text output by the large model, so that after the large model outputs the text, the text is directly converted into speech, and the digital human is driven to produce an action or an expression according to the driving mark in the text, so that the result can be displayed to the user in a timely manner, with high efficiency and good real-time performance.

[0076] Figure 1 A flowchart of a digital human driving method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes:

[0077] In step S11, receiving a text instruction input by a user;

[0078] In a conversation between digital humans, text instructions are used to instruct the digital humans to play voice. Text instructions can be words directly input by the user, or they can be voice input by the user, and then get text instructions based on voice recognition. The reply text is the reply content generated by the digital human according to the text instructions.

[0079] In a conversation between digital humans, users will ask questions to the digital humans, and the big model will be responsible for generating text answers; or users can issue instructions to the big model to generate text for the digital human to read. Then, the instructions in text form can include: questions issued by the user to the big model; instructions issued by the user to instruct the digital human to read text orally.

[0080] For example, in a questioning scenario, the textual instruction input by the user may be: "How is the weather today?" In a scenario for instructing a digital human to read a text, the textual instruction input by the user may be: "Please read aloud this text: XXXXXX."

[0081] The digital human's reply text is usually generated by the big model based on text instructions, conversation history, user portraits and other information. Based on this information, the big model will generate a reply text that conforms to the conversation context and user needs.

[0082] In the digital human dialogue, text-based instructions and reply texts are interrelated, and together they constitute the complete process of the dialogue. Through the interaction of text-based instructions and reply texts, the digital human and the user can have a natural and smooth dialogue.

[0083] In step S12, in response to receiving the instruction in text form, a reply text for replying to the instruction in text form is obtained using the large model; wherein the reply text carries a driving mark, and the driving mark is used to indicate the position in the reply text where the digital human needs to generate an action and / or an expression; the large model is obtained by pre-training;

[0084] After receiving the text-based instruction input by the user, the text-based instruction can be input into the pre-trained large model, and the large model can output a reply text for replying to the text-based instruction, and the reply text contains a driving mark for indicating that the digital human needs to generate an action and / or expression in the reply text. The large model can be trained using sample data with driving marks, and the sample data with driving marks can be used to fine-tune the known pre-trained large model, so that the training of the large model can be quickly completed. For specific training methods, please refer to the possible implementation methods provided in the present disclosure, which will not be described in detail here.

[0085] The driving mark includes an action driving mark and an expression driving mark, wherein the action driving mark is used to indicate the position in the reply text where the digital human needs to generate an action, and the expression driving mark is used to indicate the position in the reply text where the digital human needs to generate an expression.

[0086] The driving tag can be embedded in the reply text and marked with a specific symbol. For example, the reply text obtained by the large model is "I [point to myself] to explain [happy], you [point to the audience] listen carefully", where the content in [] is the driving tag. The driving tag [point to myself] is used to instruct the digital human to make an action pointing to himself, the driving tag [point to the audience] is used to instruct the digital human to make an action pointing to the audience, and the driving tag [happy] is used to instruct the digital human to have a happy facial expression.

[0087] For example, in a questioning scenario, the user may input a textual instruction: "How is the weather today?", and the large model may output a reply text: "Today's weather is sunny [happy]." In a scenario for instructing a digital human to read a text, the user may input a textual instruction: "Please read this text aloud: XXXXXX," and the model may output a reply text: "XX [pointing to oneself] XXXX [pointing to the audience]."

[0088] In step S13, audio is generated based on the reply text, and the audio playback time point corresponding to the driving mark is obtained;

[0089] Text-to-speech technology can be used to generate audio based on the reply text. Specifically, text-to-speech technology (TTS) can be used to convert the reply text after removing the driving mark into audio, and then perform speech synthesis by performing text preprocessing on the reply text. When performing text preprocessing, the reply text will be processed by word segmentation, part-of-speech tagging, syntactic analysis, etc. to facilitate subsequent speech synthesis. When performing speech synthesis, rule-based or statistical-based methods are usually used to convert the processed text into audio data.

[0090] After the audio is obtained, the audio has a certain time length, and the driving mark is the position in the marked reply text, so it needs to be converted into a time point corresponding to the audio, which is the time point of the text corresponding to the position of the play mark in the audio file. The specific conversion process can be referred to the possible implementation method provided by the present disclosure, which will not be described here.

[0091] In step S14, the audio is played, and at the audio playing time point corresponding to the driving mark, the digital human is driven to generate movement or expression.

[0092] After obtaining the audio, the audio can be loaded for playback, and then the digital human can be driven to produce corresponding actions and / or expressions according to the audio playback time point corresponding to the driving mark, so that the actions and / or expressions of the digital human match the audio playback content.

[0093] Specifically, the actual triggering time of each action and / or expression can be obtained according to the audio playback time point corresponding to the driving mark, and the animation curve of the action and / or expression can be generated. The animation curve describes the changes of the expression and / or action of the digital human over time during the speaking process, for example, the change of the digital human's mouth shape from open to closed over time. When the audio is played, when the actual triggering time of the driving mark is played, according to the generated animation curve of the action and / or expression, the digital human is driven to generate the action and / or expression corresponding to the driving mark, and the expression and / or action of the digital human will change synchronously according to the progress of the audio playback, so that the digital human will synchronously generate the corresponding action when the audio is played.

[0094] In a possible implementation, the generating audio based on the reply text and obtaining the audio playback time point corresponding to the driving mark includes: removing the driving mark in the reply text to obtain a target text; performing a text-to-speech operation on the target text to obtain an audio file and the time point of each phoneme in the audio; and determining the audio playback time point corresponding to the driving mark based on the phonemes of the text at the position indicated by the driving mark and the time point of each phoneme in the audio.

[0095] The driving mark in the reply text may be embedded in the body of the reply text. In this case, before the text is converted to speech, the driving mark in the reply text may be removed first, and then the text conversion is performed. Of course, the driving mark in the reply text may not be included in the main content (body) of the reply text. For example, the driving mark may include an identifier for indicating the position of the text. Exemplarily, the identifier may be a number indicating the number of characters in the reply text, thereby indicating the position in the text.

[0096] In the case where the driver tag in the reply text is embedded in the body of the reply text, the driver tag in the reply text can be removed first. The driver tag is often marked with a specific symbol, so the driver tag can be removed by a simple regular expression. Regular Expression (usually abbreviated as regex, regexp or regexen) is a text processing tool that uses specific patterns to match or operate character sequences in a string. Regular expressions can match, search and delete driver tags in the reply text, that is, the driver tags in the reply text are deleted to obtain the target text.

[0097] It should be noted that while deleting the drive mark, the position of each drive mark in the text can also be obtained. For example, for a reply text with a drive mark: "I [point to myself] to explain [happy], you [point to the audience] listen carefully", after deleting the drive mark, you can get the text "I will explain, you listen carefully", and each drive mark corresponds to the position in the text {1: [point to myself], 8: [point to the audience]}, {3: [happy]}, among which, "1: [point to myself]" indicates that the action drive mark "point to myself" is located after the first character; "8: [point to the audience]}" indicates that the action drive mark "point to the audience" is located after the eighth character; "{3: [happy]}" indicates that the expression drive mark "happy" is located after the third character.

[0098] After obtaining the target text, the target text can be converted to speech to obtain an audio file and the time point of each phoneme in the audio. The time point of each phoneme in the audio is the intermediate result of the text-to-speech operation. As mentioned above, when converting text to speech, the reply text will first be lexically analyzed to split it into individual words or morphemes, which usually involves operations such as word segmentation and part-of-speech tagging of the text; then speech mapping will be performed, that is, on the basis of lexical analysis, each word or morpheme will be mapped to its corresponding speech representation. This process will involve speech units of different levels such as phonemes, syllables, and rhymes; after the speech mapping is completed, speech synthesis will be performed, that is, using a speech synthesis engine to convert the mapped speech representation into an actual audio signal. The speech synthesis engine usually generates a corresponding audio file based on the input speech representation and its corresponding phoneme duration; finally, the phoneme time point extraction is performed. During the speech synthesis process, the time point information of the phoneme will be recorded, and the time point information of the phoneme can be extracted by querying the intermediate results in the synthesis process or using a specific function.

[0099] After determining the time point information of the phoneme, the phoneme of the text at the position indicated by the driving mark can be matched with the time point of the factor in the audio to accurately determine the audio playback time point corresponding to the driving mark.

[0100] In the disclosed embodiment, the target text is obtained by removing the driving mark in the reply text, and then the target text is converted to speech to obtain an audio file and the time point of each phoneme in the audio. Then, based on the phonemes of the text at the position indicated by the driving mark and the time point of each phoneme in the audio, the audio playback time point corresponding to the driving mark is determined. Since the phoneme is an intermediate result of speech synthesis, the determination of the time point of the phoneme can be obtained directly through query without consuming more processing resources. The audio playback time point corresponding to the driving mark is determined by matching the phonemes of the text at the position indicated by the driving mark with the time point of the factor in the audio. The audio playback time point corresponding to the driving mark can be accurately determined, so that the subsequent text playback is more consistent with the action expression of the digital human, and the user experience is better.

[0101] In one possible implementation, the large model is trained based on sample data, and the labels in the sample data include action-driven tags; the sample data generation process includes: using sample examples including action-driven tags to train a sample generation model based on context learning to obtain a sample generation model; using the sample generation model to generate sample data including action-driven tags.

[0102] The sample examples containing action-driven tags can be text containing action-driven tags. In this implementation, the sample generation model can directly generate text with action-driven tags. The sample generation model based on context learning can be trained by the sample examples containing action-driven tags. When training it, the sample generation model can use the learning method based on context information (In-Context Learning, ICL) to combine context information and sample examples to learn and train the model.

[0103] The sample generation model can be a pre-trained large model. When training it, it is not necessary to start learning from scratch. Instead, the previously learned knowledge (i.e., contextual information) and sample examples with action-driven tags are used to fine-tune the large model. The fine-tuned model can output text with action-driven tags.

[0104] When fine-tuning the large model, sample examples can be directly input into the large model through dialogue, and the large model can be informed that it should output data in the form of sample examples. For example, a sample example is given, which is a text with action-driven tags inserted into the text. The sample example is input into the large model through dialogue, and the large model is informed that when receiving instructions in text form, it should output reply text with action-driven tags inserted in the corresponding positions, so as to train the large model.

[0105] In the disclosed embodiments, the general knowledge that has been learned by the large language model can be used to fine-tune the large model using sample examples containing action-driven tags, and the large model can be applied to the generation of sample data containing action-driven tags, thereby achieving more efficient and accurate sample data generation. Moreover, this implementation does not require a large amount of labeled data, but only a small number of examples for the model to learn related tasks, thus saving a lot of labeling costs and time.

[0106] It should be noted that the sample data generated by the sample generation model can be further manually checked to ensure its correctness, and the action-driven tags with incorrect predictions can be manually modified.

[0107] In one possible implementation, the label in the sample data includes an expression-driven tag, and the generation process of the sample data includes: splitting the text to obtain text fragments; performing emotion recognition on each text fragment to obtain the emotion of each text fragment; adding an expression-driven tag corresponding to the emotion to each text fragment, and generating a label of the sample data including the expression-driven tag.

[0108] By splitting the text, multiple text fragments can be obtained. The text here can be any text used to generate samples, and the present disclosure does not limit this. When the text is split, the text will be split into smaller, meaningful fragments or clauses. In one example, the text can be split by sentence, that is, the text is split according to the boundaries of the sentences to form independent sentence fragments; in another example, the text can be split by paragraphs, that is, the text is split according to the boundaries of paragraphs or chapters to form independent paragraphs or chapter fragments. In addition, it can also be split according to specific rules, that is, the text is split according to specific rules or patterns, such as splitting according to specific punctuation marks, keywords or patterns. The present disclosure does not specifically limit the details of the specific splitting.

[0109] It should be noted that the context and semantics of the text need to be considered when splitting the text, so that the split segments have complete semantics and coherence, so that emotion recognition can be performed based on semantics later.

[0110] Emotion recognition can identify the speaker's emotions based on text, such as happiness, sadness, anger, fear, surprise, disgust, etc.

[0111] When performing emotion recognition on a text segment, feature extraction can be performed on the text segment to extract emotion-related features from the text segment, and then the extracted features can be classified using a machine learning or deep learning model to identify the emotion in the text segment. Commonly used models include support vector machines, naive Bayes, neural networks, etc.

[0112] Since people's expressions are often different under different emotions, based on the recognized emotions, expression-driven tags can be added to the text, and the corresponding expression-driven tags can be annotated to the corresponding text segments according to the recognized emotion results. For example, for the happy emotion, a smiling expression-driven tag can be added to the corresponding text segment.

[0113] In one example, an expression-driven tag within "{}" may be added before each text snippet, such as "{happy} Eat healthy: Maintain a balanced diet, {happy} Eat more vegetables and fruits, {disgusted} Eat less greasy and high-sugar foods.

[0114] In the disclosed embodiment, the text is split to obtain text segments, and then emotion recognition is performed on each text segment to obtain the emotion of each text segment, and an expression-driven tag corresponding to the emotion is added to each text segment as a label of the sample data. Thus, by adding expression-driven tags to text segments based on emotion recognition, expression-driven tags can be automatically added to the text, and sample data containing tags can be automatically generated, which is highly efficient.

[0115] In one possible implementation, the sample data generation process includes: splitting the text to obtain text fragments; detecting emotion keywords in the text fragments to obtain emotion keywords in the text fragments; adding corresponding expression-driven tags to the detected emotion keywords to generate labels for the sample data containing the expression-driven tags.

[0116] Emotional keywords are words or phrases that can express specific emotions in a text segment. These words or phrases are usually related to emotions, emotional states, emotional reactions, etc., such as "happy", "sad", "angry", "fear", etc. Emotional keywords can reflect the emotions and emotional states expressed by the text, so that expression-driven tags can be added to the text more accurately.

[0117] When detecting emotional keywords in text fragments, emotion-related features can be extracted from the text fragments, and then the extracted features can be classified using machine learning or deep learning models to identify emotional keywords in the text fragments. Commonly used models include support vector machines, naive Bayes, neural networks, etc. Alternatively, the emotional keywords in the text fragments can be searched for by matching the pre-set emotional keywords.

[0118] After the detected emotional keywords, the corresponding expression-driven tags can be added before the emotional keywords.

[0119] In the disclosed embodiment, the emotion keywords in the text segment are obtained by detecting emotion keywords in the text segment, and then corresponding expression-driven tags are added to the detected emotion keywords. Since emotion keywords can reflect the emotion and emotional state expressed by the text, expression-driven tags can be added to the text more accurately without the need for users to manually add expression-driven tags, thereby improving the efficiency of adding tags to sample data.

[0120] In one possible implementation, the large model is trained based on sample data, and the generation process of the sample data includes: for a text containing an action-driven tag, determining an expression-driven tag corresponding to the action-driven tag according to a preset action emotion dictionary, wherein the action emotion dictionary contains a correspondence between the action-driven tag and the expression-driven tag; adding the expression-driven tag to a position corresponding to the action-driven tag in the text, and generating a label for the sample data containing the expression-driven tag.

[0121] In the case where action-driven tags have been added to the text in advance, since some actions can also reflect emotions, expression-driven tags can be added based on the user's actions. The action-driven tags here can be added manually by the user, or can be added based on a model, such as by using the implementation method provided in the present disclosure, which will not be described in detail in the present disclosure.

[0122] In the case where the text already contains an action-driven tag, the expression-driven tag can be determined based on the action-driven tag. Specifically, an action emotion dictionary can be pre-set, and the correspondence between actions and expressions is pre-set in the dictionary. For example, when a person feels dissatisfied or angry, he may clench his fists, and at this time, he may also make a frowning expression. Based on this, a "clenched fist-frown" correspondence can be established; when a person feels sad or depressed, he may lower his head and droop his shoulders, and at this time, he may make a depressed expression. Based on this, a "lowering head and drooping shoulders-depression" correspondence can be established; when a person feels confident or satisfied, he may raise his head and chest, and at this time, he may make a smiling expression. Based on this, a "raise his head and chest-smile" correspondence can be established.

[0123] Based on the preset action emotion dictionary, corresponding expression-driven tags are added to the action-driven tags in the text.

[0124] In the disclosed embodiment, the expression driven tag corresponding to the action driven tag is determined according to the preset action emotion dictionary; and the expression driven tag is added to the position corresponding to the action driven tag in the text. Thus, when adding the expression driven tag to the sample data, the relationship between the action and the expression is considered, and the expression driven tag is added based on the action driven tag, so that the expression driven tag can be accurately added to the label in the sample data.

[0125] Figure 2 A flowchart of adding expression driven tags according to an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the text with action-driven tags can be processed into sentences to obtain text fragments, and then text emotion recognition can be performed on the text fragments to obtain text emotion categories, and expression-driven tags can be added to the text fragments based on the text emotion categories; further, emotion keywords can be detected based on the emotion dictionary, and expression-driven tags can be added before the detected emotion keywords; further, expression matching can be performed based on the action-driven tags, and expression-driven tags can be added to the positions corresponding to each action-driven tag. Finally, the added expression-driven tags can be proofread by manual proofreading.

[0126] In one possible implementation, the sample data includes a sample pair consisting of a text-based instruction sample and a reply text sample, wherein the reply text sample is a label containing an action-driven tag and / or an expression-driven tag, and the training process of the large model includes: taking the text-based instruction sample as the input of the large model to obtain an estimated text output by the large model; using the reply text sample as the expected output of the large model to determine the loss between the estimated text and the expected output; and using the loss to tune the large model.

[0127] After obtaining the text containing action-driven tags and / or expression-driven tags, the text can be used as the expected output of the large model, that is, as a label in the sample data, that is, the large model is expected to output text containing action-driven tags and / or expression-driven tags. Since the large model is in the form of a conversation, the training sample also includes samples of instructions in the form of text input by the user; and its expected output is called a reply text sample. That is, the sample data includes a sample pair consisting of a text-form instruction sample and a reply text sample, wherein the reply text sample is a reply to the text-form instruction sample, and the sample data corresponding to the training task is constructed into the format of "user: [human instruction] bot: [model reply]".

[0128] When training the big model, the instruction samples in text form are used as the input of the big model to obtain the estimated text output by the big model. The big model will use the loss between the estimated text and the reply text samples to tune the model so that the output reply text can contain the driving mark.

[0129] After the training of the large model is completed, the large model can be used to drive the digital human. For details, please refer to the possible implementation methods provided in the present disclosure, which will not be described here.

[0130] In a possible implementation, the digital human driving method can be executed by electronic devices such as terminal devices and servers. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be executed by a processor calling computer-readable instructions stored in a memory.

[0131] In addition, the present disclosure also provides a digital human driving device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any digital human driving method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0132] Figure 3 A block diagram of a digital human driving device according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the device 20 comprises:

[0133] The instruction receiving unit 21 is used to receive the instruction in the form of text input by the user;

[0134] The reply generation unit 22 is used for obtaining a reply text for replying to the text-form instruction by using the large model in response to receiving the text-form instruction; wherein the reply text carries a driving mark, and the driving mark is used to indicate a position in the reply text where the digital human needs to generate an action and / or an expression; the large model is obtained by pre-training;

[0135] An audio generation unit 23, configured to generate audio based on the reply text and obtain an audio playback time point corresponding to the drive mark;

[0136] The playing unit 24 is used to play the audio and drive the digital human to generate actions and / or expressions at the audio playing time point corresponding to the driving mark.

[0137] In a possible implementation, the audio generation unit is used to:

[0138] Removing the driving mark in the reply text to obtain the target text;

[0139] Performing a text-to-speech operation on the target text to obtain an audio file and a time point of each phoneme in the audio;

[0140] Based on the phonemes of the text at the position indicated by the driving mark and the time points of each phoneme in the audio, the audio playback time point corresponding to the driving mark is determined.

[0141] In a possible implementation, the large model is trained based on sample data, and the labels in the sample data include action-driven tags; the generation process of the sample data includes:

[0142] Using sample examples containing action-driven tags, a sample generation model based on contextual learning is trained to obtain a sample generation model;

[0143] The sample generation model is used to generate sample data containing action-driven tags.

[0144] In a possible implementation, the large model is trained based on sample data, the labels in the sample data include expression-driven tags, and the generation process of the sample data includes:

[0145] Split the text to obtain text fragments;

[0146] Perform emotion recognition on each text segment to obtain the emotion of each text segment;

[0147] An expression-driven tag corresponding to the emotion is added to each text segment, and a label of sample data containing the expression-driven tag is generated.

[0148] In a possible implementation, the large model is trained based on sample data, and the generation process of the sample data includes:

[0149] Split the text to obtain text fragments;

[0150] Detect emotional keywords in the text fragments to obtain emotional keywords in the text fragments;

[0151] Corresponding expression-driven tags are added to the detected emotional keywords, and labels of sample data containing expression-driven tags are generated.

[0152] In a possible implementation, the large model is trained based on sample data, and the generation process of the sample data includes:

[0153] For a text containing an action-driven mark, an expression-driven mark corresponding to the action-driven mark is determined according to a preset action-emotion dictionary, wherein the action-emotion dictionary contains a correspondence between the action-driven mark and the expression-driven mark;

[0154] The expression-driven mark is added to the position corresponding to the action-driven mark in the text to generate a label containing sample data of the expression-driven mark.

[0155] In a possible implementation, the sample data includes sample pairs consisting of instruction samples in text form and reply text samples, wherein the reply text samples are labels containing action-driven tags and / or expression-driven tags, and the training process of the large model includes:

[0156] Taking the instruction sample in text form as input of the large model to obtain the estimated text output by the large model;

[0157] Using the reply text sample as the expected output of the large model, determining the loss between the estimated text and the expected output;

[0158] The large model is tuned using the loss.

[0159] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0160] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0161] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0162] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0163] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0164] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0165] Figure 4 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 4 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0166] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0167] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0168] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0169] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0170] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0171] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0172] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0173] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0174] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0175] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0176] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0177] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0178] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0179] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0180] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A digital human driving method, It is characterized in that include: Receive text-based instructions from the user; In response to receiving the instruction in text form, using the large model to obtain a reply text for replying to the instruction in text form; wherein the reply text carries a driving mark, and the driving mark is used to indicate a position in the reply text where the digital human needs to generate an action and / or an expression; the large model is obtained by pre-training; Generate audio based on the reply text, and obtain the audio playback time point corresponding to the drive mark; The audio is played, and at the audio playing time point corresponding to the driving mark, the digital human is driven to generate actions and / or expressions.

2. The method according to claim 1, It is characterized in that The step of generating audio based on the reply text and obtaining the audio playback time point corresponding to the driving mark includes: Removing the driving mark in the reply text to obtain the target text; Performing a text-to-speech operation on the target text to obtain an audio file and a time point of each phoneme in the audio; Based on the phonemes of the text at the position indicated by the driving mark and the time points of each phoneme in the audio, the audio playback time point corresponding to the driving mark is determined.

3. The method according to claim 1, It is characterized in that The large model is trained based on sample data, and the labels in the sample data include action-driven tags; the generation process of the sample data includes: Using sample examples containing action-driven tags, a sample generation model based on contextual learning is trained to obtain a sample generation model; The sample generation model is used to generate sample data containing action-driven tags.

4. The method according to claim 1, It is characterized in that The large model is trained based on sample data, the labels in the sample data include expression-driven tags, and the generation process of the sample data includes: Split the text to obtain text fragments; Perform emotion recognition on each text segment to obtain the emotion of each text segment; An expression-driven tag corresponding to the emotion is added to each text segment, and a label of sample data containing the expression-driven tag is generated.

5. The method according to claim 1, It is characterized in that The large model is obtained by training based on sample data, and the generation process of the sample data includes: Split the text to obtain text fragments; Detect emotional keywords in the text fragments to obtain emotional keywords in the text fragments; Corresponding expression-driven tags are added to the detected emotional keywords, and labels of sample data containing expression-driven tags are generated.

6. The method according to claim 1, It is characterized in that The large model is obtained by training based on sample data, and the generation process of the sample data includes: For a text containing an action-driven mark, an expression-driven mark corresponding to the action-driven mark is determined according to a preset action-emotion dictionary, wherein the action-emotion dictionary contains a correspondence between the action-driven mark and the expression-driven mark; The expression-driven mark is added to the position corresponding to the action-driven mark in the text to generate a label containing sample data of the expression-driven mark.

7. The method according to any one of claims 3 to 6, It is characterized in that The sample data includes sample pairs consisting of instruction samples in text form and reply text samples, wherein the reply text samples are labels containing action-driven tags and / or expression-driven tags. The training process of the large model includes: Taking the instruction sample in text form as input of the large model to obtain the estimated text output by the large model; Using the reply text sample as the expected output of the large model, determining the loss between the estimated text and the expected output; The large model is tuned using the loss.

8. A digital human driving device, It is characterized in that include: An instruction receiving unit, used for receiving an instruction in text form input by a user; a reply generating unit, configured to, in response to receiving the instruction in the form of text, obtain a reply text for replying to the instruction in the form of text by using a large model; wherein the reply text carries a driving mark, and the driving mark is used to indicate a position in the reply text where the digital human needs to generate an action and / or an expression; and the large model is obtained by pre-training; An audio generation unit, used to generate audio based on the reply text and obtain the audio playback time point corresponding to the drive mark; The playing unit is used to play the audio and drive the digital human to produce actions and / or expressions at the audio playing time point corresponding to the driving mark.

9. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images