Virtual digital human estrus sharing ability enhancement method, device, equipment, medium and product

Abstract

The invention discloses a method, a device, equipment, a medium and a product for enhancing the estrus sharing ability of a virtual digital human, and relates to the technical field of human-computer interaction and virtual digital humans. The method comprises the following steps: firstly, based on dialogue data of a user, extracting to obtain multi-modal feature information for reflecting a dialogue understanding result of the user, voice emotion of the user and facial emotion of the user, and then importing the multi-modal feature information into a user real intention recognition model obtained by pre-training based on a machine learning algorithm; and finally, according to the big language model prompt word bank, combining the recognition result and the dialogue content, generating a response prompt word, and according to the prompt word, controlling the virtual digital person to carry out big language model-based common situation response to the user, so that the response content of the digital person is accurate and appropriate, and the user experience is improved. And the mood, expression and the like of the digital human can be kept consistent with those of the user, so that more real common-situation interaction experience is realized.

Description

Technical Field

[0001] The present invention belongs to the field of human-computer interaction and virtual digital human technology, and specifically relates to a method, device, equipment, medium and product for enhancing the empathy ability of virtual digital humans. Background Art

[0002] Currently, intelligent interactive systems that combine audio and video processing, natural language processing (NLP), and computer vision (CV) technologies have been widely used in a variety of fields, including customer service, online education, telemedicine, and virtual anchors. For example, in the customer service field, many companies' customer service systems have introduced audio and video-based intelligent customer service robots. These robots can recognize users' voice commands and facial expressions and provide targeted answers and services. In the field of online education, online education platforms use audio and video interaction technology to achieve real-time interaction between virtual teachers and students, improving teaching effectiveness and students' learning interest. In the field of telemedicine, telemedicine systems use audio and video interaction technology to enable doctors to remotely diagnose and treat patients. This technology can play a huge role during special lockdowns. In the field of virtual anchors, in the entertainment industry, audio and video-based digital human technology is used to create virtual anchors. These virtual anchors can simulate the words and behaviors of real anchors and interact with users.

[0003] Although the above existing technologies have made significant progress, they still have some defects that limit their application in certain scenarios:

[0004] (1) Insufficient accuracy in intent recognition: Although existing audio and video processing technologies and NLP algorithms can better understand the content and tone of users' conversations, they are still insufficient in recognizing the users' true intentions. In particular, when users express themselves in metaphors, sarcasm, or complex emotions, the system often has difficulty accurately understanding them.

[0005] (2) Limited ability to recognize facial muscle changes: Although computer vision technology can already recognize some basic facial expressions, it still needs to improve in recognizing more subtle facial muscle changes, which limits the system's in-depth understanding of the user's true emotions;

[0006] (3) Lack of personalization of empathic responses: Existing intelligent interactive systems often use preset response templates or strategies, which makes it difficult to provide personalized empathic responses based on individual differences and situational changes of users, thus reducing the user's satisfaction and immersion in the interaction with the system;

[0007] (4) New challenges in technology integration and compatibility: Integrating audio and video processing, NLP, and CV technologies into one system requires resolving compatibility and collaborative work issues among multiple technology modules, which increases system complexity and development costs.

[0008] (5) Privacy and security risks: When acquiring and processing users’ audio and video information, existing intelligent interactive systems may face privacy leaks and security risks, which requires that the system must strictly comply with relevant laws and regulations and privacy protection principles during the design and implementation process.

[0009] In summary, while existing audio-visual-based intelligent interaction systems have made significant progress, they still have shortcomings in areas such as intent recognition accuracy, the ability to identify facial muscle changes, and personalized empathetic responses. These shortcomings limit their effectiveness in certain scenarios and provide room for further development and optimization of this application solution. Summary of the Invention

[0010] The purpose of the present invention is to provide a method, device, computer equipment, computer-readable storage medium and computer program product for enhancing the empathy ability of virtual digital humans, so as to solve the problems of low intention recognition accuracy, limited ability to recognize facial muscle changes and lack of personalized empathetic response in existing audio and video-based intelligent interaction systems.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions:

[0012] In a first aspect, a method for enhancing the empathy ability of a virtual digital human is provided, comprising:

[0013] Applying human-computer interaction technology to collect user conversation data, wherein the conversation data contains synchronized user conversation audio data and user facial video data;

[0014] Converting the user conversation audio data into user conversation content, and applying natural language processing technology to perform semantic understanding on the user conversation content to obtain a first feature reflecting the result of user conversation understanding;

[0015] Applying a speech emotion recognition technology based on a first deep learning algorithm to the user conversation audio data to obtain a second feature reflecting the user's speech emotion;

[0016] Applying a facial emotion recognition technology based on a second deep learning algorithm based on the user facial video data to obtain a third feature for reflecting the user's facial emotion;

[0017] Importing the first feature, the second feature, and the third feature into a user true intention recognition model pre-trained based on a machine learning algorithm, and outputting a user true intention recognition result;

[0018] Based on the preset large language model prompt word library, combined with the user's true intention recognition result and the user's conversation content, generate response prompt words to help the virtual digital human respond to the user in a natural and appropriate manner;

[0019] According to the response prompt word, the virtual digital human is controlled to make an empathic response to the user based on a large language model.

[0020] Based on the above invention content, a new solution for empathic response of virtual digital humans based on user audio and video data is provided. That is, first, based on the user's conversation data, multimodal feature information reflecting the user's conversation understanding results, user voice emotions and user facial emotions is extracted, and then the multimodal feature information is imported into the user's true intention recognition model pre-trained based on the machine learning algorithm, and the user's true intention recognition result is output. Finally, according to the large language model prompt vocabulary, combined with the recognition result and the conversation content, response prompt words are generated, and according to the prompt words, the virtual digital human is controlled to make an empathic response to the user based on the large language model. In this way, not only can the digital human's response content be accurate and appropriate, but also the digital human's tone and expression can be consistent with the user, achieving a more realistic empathic interactive experience, which is convenient for practical application and promotion.

[0021] In one possible design, the first deep learning algorithm adopts an LSTM-based deep learning algorithm, and the user voice emotion includes emotions expressed based on user speaking speed, user volume and / or user voice intonation changes.

[0022] In one possible design, the second deep learning algorithm adopts a deep learning algorithm based on the Transformer architecture, and the user facial emotions include emotions expressed based on user expressions, user eyes and / or user smiles.

[0023] In one possible design, the machine learning algorithm adopts a support vector machine algorithm.

[0024] In one possible design, the response prompt word includes the user conversation content and the user's current emotion among multiple basic emotions, the user's current muscle movement among multiple facial muscle movements, the emotion presented by the user conversation content and / or the emotion presented by the user conversation movement, wherein the multiple basic emotions include happiness, surprise, anger, disgust, fear, neutrality and / or sadness, and the multiple facial muscle movements include raising the inner eyebrows, raising the outer eyebrows, lowering the eyebrows, raising the upper eyelids, raising the cheeks, tightening the eyelids, wrinkling the nose, raising the upper lip, pulling up the lip corners, revealing dimples, pressing down the corners of the mouth, raising the chin, stretching the lips, tightening the lips, pouting, separating the lips and / or loosening the jaw.

[0025] In one possible design, according to the response prompt word, controlling the virtual digital human to respond empathetically to the user based on the large language model includes:

[0026] Inputting the response prompt words into the large language model to obtain the response content;

[0027] Determining an empathically matching reply expression and tone based on the user's current emotion in the response prompt, the user's current muscle movement, the emotion presented by the user's conversation content, and / or the emotion presented by the user's conversation movement;

[0028] The virtual digital human is controlled to present the response content to the user using the reply expression and tone.

[0029] In the second aspect, a device for enhancing the empathy ability of a virtual digital human is provided, which includes a dialogue data acquisition unit, a dialogue semantic understanding unit, a voice emotion recognition unit, a facial emotion recognition unit, a true intention recognition unit, a prompt word generation unit, and a digital human control unit;

[0030] The conversation data collection unit is used to collect user conversation data using human-computer interaction technology, wherein the conversation data contains synchronized user conversation audio data and user facial video data;

[0031] The conversation semantic understanding unit is communicatively connected to the conversation data collection unit and is configured to convert the user conversation audio data into user conversation content, and apply natural language processing technology to perform semantic understanding on the user conversation content to obtain a first feature reflecting the result of user conversation understanding;

[0032] The voice emotion recognition unit is communicatively connected to the conversation data acquisition unit and is configured to apply a voice emotion recognition technology based on a first deep learning algorithm to the user conversation audio data to obtain a second feature reflecting the user's voice emotion;

[0033] The facial emotion recognition unit is communicatively connected to the conversation data acquisition unit and is configured to apply a facial emotion recognition technology based on a second deep learning algorithm to the user facial video data to obtain a third feature reflecting the user's facial emotion;

[0034] The true intention recognition unit is communicatively connected to the conversation semantic understanding unit, the voice emotion recognition unit, and the facial emotion recognition unit, respectively, and is configured to import the first feature, the second feature, and the third feature into a user true intention recognition model pre-trained based on a machine learning algorithm, and output a user true intention recognition result;

[0035] The prompt word generation unit is communicatively connected to the conversation semantic understanding unit and the true intention recognition unit, and is used to generate a response prompt word for helping the virtual digital human respond to the user in a natural and appropriate manner based on a preset large language model prompt word library, combined with the user's true intention recognition result and the user's conversation content;

[0036] The digital human control unit is communicatively connected to the prompt word generation unit, and is used to control the virtual digital human to make an empathic response to the user based on the large language model according to the response prompt word.

[0037] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a transceiver communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the method for enhancing the empathy ability of a virtual digital human as described in the first aspect or any possible design of the first aspect.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon. When the instructions are run on a computer, the method for enhancing the empathy ability of a virtual digital human as described in the first aspect or any possible design of the first aspect is executed.

[0039] In a fifth aspect, the present invention provides a computer program product, comprising a computer program or instructions, which, when executed by a computer, implements the method for enhancing the empathy ability of a virtual digital human as described in the first aspect or any possible design of the first aspect.

[0040] Beneficial effects of the above scheme:

[0041] (1) The present invention creatively provides a new solution for empathic response of virtual digital humans based on user audio and video data, namely, firstly, based on the user's conversation data, multimodal feature information reflecting the user's conversation understanding results, user's voice emotions and user's facial emotions is extracted, and then the multimodal feature information is imported into the user's true intention recognition model pre-trained based on the machine learning algorithm, and the user's true intention recognition result is output. Finally, according to the large language model prompt word library, combined with the recognition result and the conversation content, a response prompt word is generated, and according to the prompt word, the virtual digital human is controlled to make an empathic response to the user based on the large language model. In this way, not only can the response content of the digital human be accurate and appropriate, but also the tone and expression of the digital human can be consistent with the user, thereby achieving a more realistic empathic interaction experience;

[0042] (2) It can also improve the naturalness and fluency of the interactive experience: that is, by deeply analyzing the user's conversation content, voice tone, and facial muscle changes, this solution can accurately judge the user's true intention and generate appropriate responses accordingly. This process requires not only the accuracy of the digital human's response content, but also the consistency of its tone and expression with the user, thereby greatly improving the naturalness and fluency of the interactive experience. When users interact with the digital human, they can feel a more real and appropriate response, enhancing the immersion and satisfaction of the interaction;

[0043] (3) It can also enhance the empathy of digital humans: Empathy refers to the ability to understand and identify with the thoughts, feelings, and ideas of others, and to consciously think from their perspective. This solution simulates the human empathy process, enabling digital humans to more accurately understand the user's intentions and emotions and respond in an appropriate manner. This process not only enhances the empathy of digital humans, but also enables them to better adapt to the needs and changes of users, provide more personalized and intelligent services, and facilitate practical application and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of a method for enhancing the empathy ability of a virtual digital human provided in an embodiment of the present application.

[0046] Figure 2 This is a structural diagram of the device for enhancing the empathy ability of a virtual digital human provided in an embodiment of the present application.

[0047] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structures of the drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0049] It should be understood that although the terms first, second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are merely used to distinguish one object from another. For example, a first object can be referred to as a second object, and similarly, a second object can be referred to as a first object without departing from the scope of the exemplary embodiments of the present invention.

[0050] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that there may be three relationships. For example, A and / or B can indicate three situations: A exists alone, B exists alone, or A and B exist at the same time. For another example, A, B and / or C can indicate the existence of any one of A, B and C or any combination of them. The term " / and" that may appear in this document describes another type of association object relationship, indicating that there may be two relationships. For example, A / and B can indicate two situations: A exists alone or A and B exist at the same time. In addition, the character " / " that may appear in this document generally indicates that the previous and next associated objects are in an "or" relationship.

[0051] Example

[0052] like Figure 1 As shown, the method for enhancing the empathy ability of virtual digital humans provided in the first aspect of this embodiment can be, but is not limited to, executed by a computer device with certain computing resources, such as a cloud server, a personal computer (PC, a multi-purpose computer with a size, price, and performance suitable for personal use; desktops, laptops, small laptops, tablets, and ultrabooks are all personal computers), a smart phone, a personal digital assistant (PDA), or a wearable device. Figure 1As shown, the method for enhancing the empathy ability of a virtual digital human may include, but is not limited to, the following steps S1 to S7.

[0053] S1. Apply human-computer interaction technology to collect user conversation data, where the conversation data includes but is not limited to synchronized user conversation audio data and user facial video data.

[0054] In step S1, the human-computer interaction technology (Human-Computer Interaction Techniques) refers to the technology of realizing human-computer dialogue in an effective manner through computer input and output devices, that is, it can specifically be but not limited to obtaining synchronized user dialogue audio data and user facial video data through conventional audio and video acquisition devices (such as microphones and cameras), so as to serve as the data basis for subsequent user intention analysis.

[0055] S2. Convert the user conversation audio data into user conversation content, and apply natural language processing technology to perform semantic understanding on the user conversation content to obtain a first feature that reflects the result of user conversation understanding.

[0056] In step S2, the specific process of content conversion can be conventionally implemented using existing speech and text recognition technology. Natural language processing technology is a key research area in the field of artificial intelligence, integrating knowledge from multiple disciplines, including linguistics, computer science, machine learning, mathematics, and cognitive psychology. It is an interdisciplinary subject that integrates computer science, artificial intelligence, and linguistics. It encompasses two main aspects: natural language understanding and natural language generation. Its research encompasses multiple levels, including characters, words, phrases, sentences, paragraphs, and chapters, serving as a bridge between machine language and human language. Therefore, the user conversation content can first be imported into a deep learning model for natural language understanding, and then the final layer output of the deep learning model can be used as the first feature reflecting the user conversation understanding result.

[0057] S3. Based on the user conversation audio data, apply the voice emotion recognition technology based on the first deep learning algorithm to obtain a second feature for reflecting the user's voice emotion.

[0058] In step S3, deep learning specifically refers to machine learning based on deep neural network models and methods; specifically, the first deep learning algorithm can be, but is not limited to, a deep learning algorithm based on LSTM (Long Short-Term Memory), and the user voice emotion includes, but is not limited to, emotions expressed based on user speaking speed, user volume, and / or user voice changes. Therefore, the second feature can reflect the user's true emotions and psychological state from the voice dimension, and the user conversation audio data can be first imported into the LSTM model for realizing voice emotion recognition, and then the last layer output of the LSTM model is used as the second feature for reflecting the user's voice emotion.

[0059] S4. Based on the user facial video data, apply facial emotion recognition technology based on the second deep learning algorithm to obtain a third feature for reflecting the user's facial emotion.

[0060] In step S4, specifically, the second deep learning algorithm can be but is not limited to a deep learning algorithm based on the Transformer architecture, and the user's facial emotions include but are not limited to emotions expressed based on user expressions, user eyes and / or user smiles, so the third feature can reflect the user's true emotions and psychological state from the dimension of facial muscle changes, and the user's facial video data can be first imported into the Transformer model for facial emotion recognition, and then the last layer output of the Transformer model can be used as the third feature for reflecting the user's facial emotions.

[0061] S5. Import the first feature, the second feature, and the third feature into a user true intention recognition model pre-trained based on a machine learning algorithm, and output the user true intention recognition result.

[0062] In step S5, the first feature, the second feature, and the third feature are used as multimodal information to be imported into the user's true intention recognition model as model input. The machine learning algorithm is a core artificial intelligence algorithm that specifically studies how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. It is the fundamental way to make computers intelligent. Specifically, the machine learning algorithm can be, but is not limited to, a support vector machine algorithm to quickly and accurately find patterns in the data. Thus, based on a certain amount of sample data (i.e., the model input is multimodal information, and the model output is the user's true intention label corresponding to the multimodal information), a user's true intention recognition model that has been verified can be trained through conventional calibration and verification modeling methods (the specific process includes a model calibration process and a verification process, i.e., first comparing the model simulation results with the measured data, and then adjusting the model parameters based on the comparison results to make the simulation results consistent with the actual process).

[0063] S6. Based on the preset large language model prompt word library, combined with the user's true intention recognition result and the user conversation content, generate response prompt words to help the virtual digital human respond to the user in a natural and appropriate manner.

[0064] In step S6, the large language model is a deep learning model with billions to hundreds of billions of parameters. It uses unsupervised learning to capture language statistics from massive amounts of text and simulate the human language cognitive process. Its core capabilities include text generation, semantic understanding, and contextual reasoning. Prompts are questions, instructions, or requests sent by users to the large language model, which clearly inform the model of the problem the user wants to solve or the task they want to complete. They are the basis for the large language model to understand user needs and generate relevant and accurate answers or content accordingly. The large language model prompt vocabulary can be pre-determined by developers based on natural and appropriate requirements to ensure that the response prompts are appropriate and natural. Specifically, the response prompt words include but are not limited to the user conversation content and the user's current emotion among multiple basic emotions, the user's current muscle movements among multiple facial muscle movements, the emotions presented by the user conversation content and / or the emotions presented by the user conversation movements, etc., wherein the multiple basic emotions include but are not limited to happiness, surprise, anger, disgust, fear, neutrality and / or sadness, etc., and the multiple facial muscle movements include but are not limited to raising the inner eyebrows, raising the outer eyebrows, lowering the eyebrows, raising the upper eyelids, raising the cheeks, tightening the eyelids, wrinkling the nose, raising the upper lip, pulling up the lip corners, revealing dimples, pressing down the corners of the mouth, raising the chin, stretching the lips, tightening the lips, pouting, separating the lips and / or loosening the jaw, etc. For example, the response prompt words are composed as follows: "Which of the seven basic emotions is the current emotional state of the user (i.e., the user's current emotion), which of the 17 facial muscle movements is the facial muscle movement (i.e., the user's current muscle movement), the user's words are: *** (i.e., the user's conversation content / and the user's conversation content presents emotions), the user's speaking emotions are: *** (i.e., the user's conversation actions present emotions), please analyze the user's true intentions and answer."

[0065] S7. According to the response prompt word, control the virtual digital human to make an empathic response to the user based on the large language model.

[0066] In step S7, the specific process of constructing the virtual digital human is based on existing technology, preferably aiming for a highly realistic virtual digital human. Furthermore, the matching virtual digital human is preferably constructed based on the user's personal preferences. For example, if the user prefers a certain type of anime character, then that type of anime character can be constructed as a matching virtual digital human. Specifically, based on the response prompt, the virtual digital human is controlled to provide an empathic response to the user based on the large language model, including but not limited to the following steps S71 to S73.

[0067] S71. Input the response prompt word into the large language model to obtain the response content.

[0068] In step S71, the large language model can be implemented by, but is not limited to, a BERT (Bidirectional Encoder Representations from Transformers) series model or a T5 series model.

[0069] S72. Determine an empathically matching reply expression and tone based on the user's current emotion in the response prompt words, the user's current muscle movements, the emotions presented by the user's conversation content, and / or the emotions presented by the user's conversation movements.

[0070] In step S72, since the user's current emotion, the user's current muscle movements, the emotion displayed in the user's conversation content, and / or the emotion displayed in the user's conversational actions reflect the user's current expression and tone, an expression and tone that is identical or similar to these may be used as a reply expression and tone to achieve empathy matching. For example, if the user's current expression and tone is crying, the reply expression and tone may be a sad expression and tone.

[0071] S73. Control the virtual digital human to present the response content to the user using the response expression and tone.

[0072] In step S73, the specific process of controlling the virtual digital human to present its response content is based on existing technical means. For example, a TTS (Text-To-Speech) voice broadcast algorithm is first applied to convert the response content into a response voice signal that matches the response expression and tone. The response voice signal is then presented by the virtual digital human in spoken form and / or with gestures. This not only ensures that the digital human's response content is accurate and relevant, but also aligns its tone and expression with the user, achieving a more authentic empathetic interaction experience.

[0073] Therefore, based on the method for enhancing the empathy ability of virtual digital humans described in the aforementioned steps S1 to S7, a new solution for empathic response of virtual digital humans based on user audio and video data is provided. That is, first, based on the user's conversation data, multimodal feature information reflecting the user's conversation understanding results, user voice emotions and user facial emotions is extracted, and then the multimodal feature information is imported into the user's true intention recognition model pre-trained based on the machine learning algorithm, and the user's true intention recognition result is output. Finally, based on the large language model prompt vocabulary, combined with the recognition results and the conversation content, response prompt words are generated, and according to the prompt words, the virtual digital human is controlled to make an empathic response to the user based on the large language model. This not only makes the digital human's response content accurate and appropriate, but also makes the digital human's tone and expression consistent with the user, achieving a more realistic empathic interaction experience, which is convenient for practical application and promotion.

[0074] like Figure 2 As shown, the second aspect of this embodiment provides a virtual device for implementing the method for enhancing the empathy ability of a virtual digital human described in the first aspect, comprising a dialogue data acquisition unit, a dialogue semantic understanding unit, a voice emotion recognition unit, a facial emotion recognition unit, a true intention recognition unit, a prompt word generation unit, and a digital human control unit;

[0075] The conversation data collection unit is used to collect user conversation data using human-computer interaction technology, wherein the conversation data contains synchronized user conversation audio data and user facial video data;

[0076] The conversation semantic understanding unit is communicatively connected to the conversation data collection unit and is configured to convert the user conversation audio data into user conversation content, and apply natural language processing technology to perform semantic understanding on the user conversation content to obtain a first feature reflecting the result of user conversation understanding;

[0077] The voice emotion recognition unit is communicatively connected to the conversation data acquisition unit and is configured to apply a voice emotion recognition technology based on a first deep learning algorithm to the user conversation audio data to obtain a second feature reflecting the user's voice emotion;

[0078] The facial emotion recognition unit is communicatively connected to the conversation data acquisition unit and is configured to apply a facial emotion recognition technology based on a second deep learning algorithm to the user facial video data to obtain a third feature reflecting the user's facial emotion;

[0079] The true intention recognition unit is communicatively connected to the conversation semantic understanding unit, the voice emotion recognition unit, and the facial emotion recognition unit, respectively, and is configured to import the first feature, the second feature, and the third feature into a user true intention recognition model pre-trained based on a machine learning algorithm, and output a user true intention recognition result;

[0080] The prompt word generation unit is communicatively connected to the conversation semantic understanding unit and the true intention recognition unit, and is used to generate a response prompt word for helping the virtual digital human respond to the user in a natural and appropriate manner based on a preset large language model prompt word library, combined with the user's true intention recognition result and the user's conversation content;

[0081] The digital human control unit is communicatively connected to the prompt word generation unit, and is used to control the virtual digital human to make an empathic response to the user based on the large language model according to the response prompt word.

[0082] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be found in the method for enhancing the empathy ability of virtual digital humans described in the first aspect, and will not be repeated here.

[0083] like Figure 3 As shown, the third aspect of this embodiment provides a computer device for executing the method for enhancing the empathy ability of a virtual digital human as described in the first aspect, comprising a memory, a processor, and a transceiver connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the method for enhancing the empathy ability of a virtual digital human as described in the first aspect. For example, the memory may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a first-in-first-out memory (FIFO), and / or a first-in-last-out memory (FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. In addition, the computer device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0084] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be found in the method for enhancing the empathy ability of virtual digital humans described in the first aspect, and will not be repeated here.

[0085] A fourth aspect of this embodiment provides a computer-readable storage medium storing instructions including the method for enhancing the empathy ability of a virtual digital human as described in the first aspect. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, execute the method for enhancing the empathy ability of a virtual digital human as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device.

[0086] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be referred to the method for enhancing the empathy ability of virtual digital humans as described in the first aspect, and will not be repeated here.

[0087] A fifth aspect of this embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the method for enhancing the empathy ability of a virtual digital human as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0088] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for enhancing the empathy ability of a virtual digital human, characterized by: include: Applying human-computer interaction technology to collect user conversation data, wherein the conversation data contains synchronized user conversation audio data and user facial video data; Converting the user conversation audio data into user conversation content, and applying natural language processing technology to perform semantic understanding on the user conversation content to obtain a first feature reflecting the result of user conversation understanding; Applying a speech emotion recognition technology based on a first deep learning algorithm to the user conversation audio data to obtain a second feature reflecting the user's speech emotion; Applying a facial emotion recognition technology based on a second deep learning algorithm based on the user facial video data to obtain a third feature for reflecting the user's facial emotion; Importing the first feature, the second feature, and the third feature into a user true intention recognition model pre-trained based on a machine learning algorithm, and outputting a user true intention recognition result; Based on the preset large language model prompt word library, combined with the user's true intention recognition result and the user's conversation content, generate response prompt words to help the virtual digital human respond to the user in a natural and appropriate manner; According to the response prompt word, the virtual digital human is controlled to make an empathic response to the user based on a large language model.

2. The method for enhancing the empathy ability of a virtual digital human according to claim 1, characterized in that: The first deep learning algorithm adopts an LSTM-based deep learning algorithm, and the user voice emotion includes emotions expressed based on user speaking speed, user volume and / or user voice intonation changes.

3. The method for enhancing the empathy ability of a virtual digital human according to claim 1, characterized in that: The second deep learning algorithm adopts a deep learning algorithm based on the Transformer architecture, and the user facial emotions include emotions expressed based on user expressions, user eyes and / or user smiles.

4. The method for enhancing the empathy ability of a virtual digital human according to claim 1, characterized in that: The machine learning algorithm adopts the support vector machine algorithm.

5. The method for enhancing the empathy ability of a virtual digital human according to claim 1, characterized in that: The response prompt words include the user conversation content and the user's current emotion among multiple basic emotions, the user's current muscle movement among multiple facial muscle movements, the emotion presented by the user conversation content and / or the emotion presented by the user conversation movement, wherein the multiple basic emotions include happiness, surprise, anger, disgust, fear, neutrality and / or sadness, and the multiple facial muscle movements include raising the inner eyebrows, raising the outer eyebrows, lowering the eyebrows, raising the upper eyelids, raising the cheeks, tightening the eyelids, wrinkling the nose, raising the upper lip, pulling up the lip corners, revealing dimples, pressing down the corners of the mouth, raising the chin, stretching the lips, tightening the lips, pouting, separating the lips and / or loosening the jaw.

6. The method for enhancing the empathy ability of a virtual digital human according to claim 5, characterized in that: According to the response prompt word, controlling the virtual digital human to respond empathetically to the user based on the large language model includes: Inputting the response prompt words into the large language model to obtain the response content; Determining an empathically matching reply expression and tone based on the user's current emotion in the response prompt, the user's current muscle movement, the emotion presented by the user's conversation content, and / or the emotion presented by the user's conversation movement; The virtual digital human is controlled to present the response content to the user using the reply expression and tone.

7. A device for enhancing the empathy ability of a virtual digital human, characterized in that: It includes a dialogue data acquisition unit, a dialogue semantic understanding unit, a voice emotion recognition unit, a facial emotion recognition unit, a true intention recognition unit, a prompt word generation unit and a digital human control unit; The conversation data collection unit is used to collect user conversation data using human-computer interaction technology, wherein the conversation data contains synchronized user conversation audio data and user facial video data; The conversation semantic understanding unit is communicatively connected to the conversation data collection unit and is configured to convert the user conversation audio data into user conversation content, and apply natural language processing technology to perform semantic understanding on the user conversation content to obtain a first feature reflecting the result of user conversation understanding; The voice emotion recognition unit is communicatively connected to the conversation data acquisition unit and is configured to apply a voice emotion recognition technology based on a first deep learning algorithm to the user conversation audio data to obtain a second feature reflecting the user's voice emotion; The facial emotion recognition unit is communicatively connected to the conversation data acquisition unit and is configured to apply a facial emotion recognition technology based on a second deep learning algorithm to the user facial video data to obtain a third feature reflecting the user's facial emotion; The true intention recognition unit is communicatively connected to the conversation semantic understanding unit, the voice emotion recognition unit, and the facial emotion recognition unit, respectively, and is configured to import the first feature, the second feature, and the third feature into a user true intention recognition model pre-trained based on a machine learning algorithm, and output a user true intention recognition result; The prompt word generation unit is communicatively connected to the conversation semantic understanding unit and the true intention recognition unit, and is used to generate a response prompt word for helping the virtual digital human respond to the user in a natural and appropriate manner based on a preset large language model prompt word library, combined with the user's true intention recognition result and the user's conversation content; The digital human control unit is communicatively connected to the prompt word generation unit, and is used to control the virtual digital human to make an empathic response to the user based on the large language model according to the response prompt word.

8. A computer device, characterized in that: It includes a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the method for enhancing the empathy ability of a virtual digital human as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the computer, the method for enhancing the empathy ability of a virtual digital human as described in any one of claims 1 to 6 is executed.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the method for enhancing the empathy ability of a virtual digital human as claimed in any one of claims 1 to 6 is implemented.

Images