Deep learning-based emotional response generation device with emoticons

Abstract

The present invention provides a deep learning-based device for generating emotional responses with emoticons. The device comprises: a dialogue module for generating responses based on a conversational utterance and a specified emotion; an emoticon module for generating a specified emoticon set based on the conversational utterance, the specified emotion, and an emoticon library; a location module for generating sequence annotation results based on the responses and the specified emoticon set; and an integration module for integrating the sequence annotation results, the specified emoticon set, and the responses to generate an emotional response. In short, this method can generate emotionally rich and accurate responses in conversations.

Description

Technical Field

[0001] The present invention relates to the application of natural language processing and other technologies in open domain dialogue generation, and in particular to a deep learning-based emotional response generation device with emoticons. Background Art

[0002] Emotional conversation is an essential component of human communication. Humans possess a rich emotional landscape, with emotions like joy, anger, sorrow, and happiness easily revealed in conversation. During communication, people not only express their own emotions but also understand the speaker's emotions and respond appropriately. Therefore, conversations lacking emotion often feel hollow and lack impact, failing to evoke empathy and resonance. Related research indicates that analyzing emotion in casual conversations not only helps generate appropriate responses but also helps regulate the emotions of both parties, a crucial function of human companionship. Emotional conversation models serve more than just entertainment; they also have important implications for emotional and psychological counseling in mental health treatment, calming negative customer attitudes in online intelligent customer service, and providing customized personalized assistants. Artificial intelligence can only engage in common-sense reasoning and thinking if it understands emotions. If a model fails to learn the user's emotions, its responses will be grammatically correct but illogical, significantly reducing the quality of the conversation.

[0003] Emotional conversational robots hold significant value in areas such as psychotherapy, providing companionship for elderly people living alone, and soothing consumers. The quality of emotional conversations has also improved with the continuous innovation of researchers. However, most existing methods focus solely on text construction, which still suffers from insufficient conversational information and ambiguous emotional expression. Pure text expressions rely entirely on abstract and vague terms, which has limitations in expressing emotions.

[0004] With the booming popularity of emojis on social platforms like Weibo and WeChat, it's no secret that they're a more effective way to express emotions. Facial expressions influence communication, and emojis, graphical representations of facial expressions, body language, and gestures, can convey complex messages through simple symbols, helping both parties in text-based online communication express their opinions and emotions more clearly and accurately.

[0005] Initial approaches to generating emotional conversations were primarily rule-based, recursively executing pattern matchers to generate responses. Currently, however, most chatbot technologies are built using neural networks. Existing research is focusing on improving the diversity of responses and the rationality of emotional expression in emotional conversation models through various approaches. In real conversations, a sentence may not express a single emotion, so capturing the nuance and diversity of human emotions remains a challenge. In fact, many symbolic texts are more capable of expressing complex, multifaceted emotions, a fact that has been largely overlooked by existing research. Summary of the Invention

[0006] The present invention is made to solve the above-mentioned problems, and its purpose is to provide an emotional response generation device with emoticons based on deep learning.

[0007] The present invention provides an emotional response generation device with emoticons based on deep learning, which is used to obtain corresponding emotional responses according to dialogue discourse, specified emotions and an emoticon library, and has the following characteristics: a dialogue module, which is used to obtain a response reply according to the dialogue discourse and the specified emotion; an emoticon module, which is used to obtain a specified emoticon set according to the dialogue discourse, the specified emotion and the emoticon library; a position module, which is used to obtain a sequence labeling result according to the response reply and the specified emoticon set; an integration module, which is used to integrate the sequence labeling result, the specified emoticon set and the response reply to obtain the emotional response, wherein the emoticon module includes an emotion detection submodule, an emotion quantification submodule and an emoticon calculation submodule, the emotion detection submodule is used to obtain corresponding multiple emotional states according to the dialogue discourse, the emotion quantification submodule is used to quantify the specified emotion and all emotional states respectively to obtain corresponding emotion vectors, and the emoticon calculation submodule is used to calculate according to the emotion vector of the specified emotion, the emotion vectors of each emotional state and the emoticon library to obtain multiple emoticons in the emoticon library as the specified emoticon set.

[0008] In the deep learning-based emotional response generation device with emoticons provided by the present invention, it can also have the following features: wherein the dialogue module includes a trained conditional variational autoencoder CVAE, which generates an emotional response based on the existing speech X and emotion set E and the corresponding original response Y. ′ The training set is used to train the initial conditional variational autoencoder CVAE to obtain the trained conditional variational autoencoder CVAE. The loss function used in the training process is The calculation formula is: Where z is the latent variable, Y is the maximized response, and C x,e For given conditions, C x,e =[ox ;e′],o x is the output of the encoder of the conditional variational autoencoder CVAE after processing the speech X, e′ is the embedding vector of the emotion category specified from the emotion set E, and p D (Y|z,C x,e ) is the prior model used to sample z from the prior distribution, q R (z|Y,C x,e ) is the recognition network used to approximate the posterior distribution of the latent variable z, p P (z|C x,e ) is the variational lower bound of maximizing the given conditional likelihood obtained by approximate calculation, and KL is the KL divergence.

[0009] In the deep learning-based emotional response generation device with emoticons provided by the present invention, it can also have the following features: wherein, the emotion detection submodule includes an Embedding layer, a Bi-LSTM layer, an attention layer and a Softmax layer, the Embedding layer is used to map each word in the dialogue discourse into a corresponding standardized vector, the Bi-LSTM layer is used to encode the standardized vector to obtain hidden layer features, the attention layer is used to process the hidden layer features and the standardized vector according to the Attention mechanism to obtain a vector representation of the dialogue discourse, and the Softmax layer is used to perform a Softmax function operation on the vector representation to obtain the emotional state.

[0010] The deep learning-based emotional response generation device with emoticons provided by the present invention may also have the following features: wherein the emotion quantification submodule quantifies the specified emotions and emotional states according to the emoticon quantification rules of the emoticon library.

[0011] The deep learning-based emotional response generation device with emoticons provided by the present invention may also have the following features: in the emoticon calculation submodule, the emotion vector V corresponding to the emotional state is calculated based on the emotion vector of the specified emotion and the emotion vector of the emotional state, and then the Euclidean distance between the emotion vector V and the vector of each emoticon in the emoticon library is calculated, and the emoticon corresponding to the smallest Euclidean distance is selected as the emoticon in the specified emoticon set.

[0012] The deep learning-based emotional response generation device with emoticons provided by the present invention may also have the following feature: when there is no emoticon in the specified emoticon set, the emotional response is a response response.

[0013] In the deep learning-based emotional response generation device with emoticons provided by the present invention, it can also have the following characteristics: wherein, the sequence labeling result is a judgment result of whether an emoticon from a specified emoticon set is inserted after each word in the response reply, and the judgment result is 0 or 1, 0 means that no emoticon is inserted after the corresponding word, and 1 means that an emoticon is inserted after the corresponding word.

[0014] In the deep learning-based emotional response generation device with emoticons provided by the present invention, it can also have the following characteristics: wherein, all judgment results of the sequence labeling results can be 0 at the same time, that is, the response reply without inserting any emoticons is regarded as an emotional response.

[0015] The deep learning-based emotional response generation device with emoticons provided by the present invention may also have the following features: wherein, the integration module includes emoticon gating, and the emoticon gating is used to insert the corresponding emoticons in the emoticon set into the corresponding position of the response reply according to the sequence labeling result to obtain an emotional response.

[0016] Functions and effects of the invention

[0017] According to the deep learning-based emotional reply generation device with emoticons involved in the present invention, because the emotional state in the dialogue discourse is extracted through the emoticon module, and then the emotion is quantified in combination with the specified emotion to obtain the emotion vector, and then the Euclidean distance is calculated with the emoticons in the emoticon library to obtain the emoticon that best matches the emotion vector, and the position module is used to determine the position where the emoticon needs to be inserted in the response reply, thereby generating an emotional reply with emoticons. Therefore, the deep learning-based emotional reply generation device with emoticons of the present invention can generate rich and accurate responses in the conversation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the framework and working principle of the emotional response generation device in an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of the framework of the emoticon module in an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of the framework of the emotion quantification submodule in an embodiment of the present invention;

[0021] Figure 4 4 is a heat diagram of three groups of emoticons that interchange emotion vector values ​​in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the emotional response generation device with emoticons based on deep learning of the present invention.

[0023] In this embodiment, the emotional response generation device with emoticons based on deep learning of the present invention is used to obtain corresponding emotional responses based on dialogue discourse, specified emotions and emoticon library, and the specified emotions are emotional category labels.

[0024] Figure 1 It is a schematic diagram of the framework and working principle of the emotional response generation device in an embodiment of the present invention.

[0025] like Figure 1 As shown, the emotional response generation device 100 includes a dialogue module 10, an emoticon module 20, a position module 30 and an integration module 40. Its working principle is: the dialogue discourse and the specified emotion are input into the dialogue module 10 to obtain a response reply, and then the dialogue discourse and the specified emotion are input into the emoticon module 20 to obtain a specified emoticon set from the emoticon library, and then the response reply and the specified emoticon set are input into the position module 30 to obtain a sequence labeling result, and finally the sequence labeling result, the specified emoticon set and the response reply are input into the integration module 40 to obtain the corresponding emotional response.

[0026] The dialogue module 10 is used to obtain a response reply based on the dialogue discourse and the specified emotion.

[0027] The dialogue module 10 includes a conditional variational autoencoder (CVAE). In this embodiment, the encoder of the conditional variational autoencoder (CVAE) is a Bi-GRU, and the specified emotion is the label information input into the conditional variational autoencoder (CVAE).

[0028] According to the existing discourse X and emotion set e and the corresponding original response Y ′ The training set is used to train the initial conditional variational autoencoder CVAE to obtain the trained conditional variational autoencoder CVAE. The initial conditional variational autoencoder CVAE adds a recognition network and a priori network on the basis of the traditional Seq2Seq model, and introduces the latent variable z to increase the diversity of the response. The loss function used in the training process is The calculation formula is:

[0029]

[0030] Where z is the latent variable, Y is the maximized response, and C x,e For given conditions, C x,e =[o x ;e′],o xis the output of the encoder of the conditional variational autoencoder CVAE after processing the speech X, e′ is the embedding vector of the emotion category specified from the emotion set E, and p D (Y|z,C x,e ) is the prior model used to sample z from the prior distribution, q R (z|Y,C x,e ) is the recognition network used to approximate the posterior distribution of the latent variable z, p P (z|C x,e ) is the variational lower bound of maximizing the given conditional likelihood obtained by approximate calculation, and KL is the KL divergence.

[0031] During training, the latent variable z is passed from the recognition network to the decoder to decode and generate the response p(Y|C x,e ) and is trained by the prior network until it is close to z′:

[0032] p(Y|C x,e )=∫p(Y|z,C x,e )p(z|C x,e )dz.

[0033] Figure 2 2 is a schematic diagram of the framework of the emoticon module in an embodiment of the present invention.

[0034] like Figure 2 As shown, the emoticon module 20 includes an emotion detection submodule 201, an emotion quantification submodule 202 and an emoticon calculation submodule 203, which is used to obtain a specified emoticon set based on dialogue discourse, specified emotions and an emoticon library. In this embodiment, the emoticon library is EmoTag-1200.

[0035] The emotion detection submodule 201 is used to obtain corresponding multiple emotional states according to the dialogue discourse.

[0036] Figure 3 Schematic diagram of the framework of the emotion quantification submodule in an embodiment of the present invention.

[0037] like Figure 3 As shown, the emotion quantization submodule 202 includes an Embedding layer 2021, a Bi-LSTM layer 2022, an attention layer 2023 and a Softmax layer 2024, which are used to quantify the specified emotion and all emotional states respectively to obtain corresponding emotion vectors.

[0038] The Embedding layer 2021 is used to map each word in the dialogue into a corresponding standardized vector. In this embodiment, during the mapping process, each Embedding dimension is mapped to [-1, 1] using a bitangent function, so that the value range of each value in the vector is [-1, 1], thereby realizing vector standardization processing.

[0039] The Bi-LSTM layer 2022 is used to encode the normalized vector to obtain hidden layer features.

[0040] The attention layer 2023 is used to process the hidden layer features and the normalized vector according to the Attention mechanism to obtain a vector representation of the dialogue utterance. In this embodiment, the attention layer 2023 calculates the importance score of each word based on the hidden layer features and the normalized vector, and then uses the importance score as a weight to weight the sum of all time steps to obtain a representation vector, i.e., a vector representation, of the entire dialogue utterance.

[0041] The Softmax layer 2024 is used to perform a Softmax function operation on the vector representation to obtain the emotional state.

[0042] Among them, the emotion quantification submodule 202 quantifies the specified emotions and emotional states according to the emoticon quantification rules of the emoticon library. In this embodiment, the emoticon quantification rules of EmoTag-1200 are based on the eight emotion combinations of Plutchik's "Wheel of Emotions", namely anger, expectation, disgust, fear, happiness, sadness, surprise and trust. Therefore, the quantified specified emotions and emotional states have eight dimensions.

[0043] The emoticon calculation submodule 203 is used to perform calculations based on the emotion vector of the designated emotion, the emotion vectors of each emotional state, and the emoticon library to obtain multiple emoticons in the emoticon library as the designated emoticon set.

[0044] Among them, in the emoticon calculation submodule 203, the specific process of obtaining the emoticon is: according to the emotion vector of the specified emotion and the emotion vector of the emotional state, the emotion vector V corresponding to the emotional state is calculated, and then the Euclidean distance between the emotion vector V and the vector of each emoticon in the emoticon library is calculated, and finally the emoticon corresponding to the smallest Euclidean distance is selected as the emoticon in the specified emoticon set.

[0045] The expression of the emotion vector V is:

[0046]

[0047] In the formula is the value corresponding to the i-th dimension of the emotion vector of the specified emotion, is the value corresponding to the i-th dimension of the emotion vector of the emotional state, α and β are both hyperparameters, and N is the total number of dimensions of the emotion vector. In this embodiment, when the emoticon library is EmoTag-1200, N=8.

[0048] The mathematical expression for getting the emojis in the specified emoji set is:

[0049]

[0050] emoji p =argmin{dist j},

[0051] Where m j For emojis in the emoji library, V u is the value corresponding to the u-th dimension of the emotion vector V, m ju For emoticon m j The value corresponding to the u-th dimension of the corresponding emotion vector, n is the total number of dimensions, dist j (m j ,V) is the Euclidean distance between the emotion vector V and each emoticon in the emoticon library, argmin{dist j} is the minimum value among all Euclidean distances, emoji p The emoji corresponding to the minimum Euclidean distance is taken as the emoji corresponding to the emotional state.

[0052] The location module 30 is used to obtain a sequence labeling result according to the response reply and the specified emoticon set.

[0053] Among them, the sequence labeling result is the judgment result of whether an emoticon from the specified emoticon set is inserted after each word in the response reply. The judgment result is 0 or 1, 0 means no emoticon is inserted after the corresponding word, and 1 means an emoticon is inserted after the corresponding word.

[0054] The integration module 40 is used to integrate the sequence labeling results, the specified emoticon set and the response reply to obtain an emotional reply.

[0055] Among them, the integration module 40 includes emoticon gating, which is used to insert the corresponding emoticon in the emoticon set into the corresponding position of the response reply according to the sequence labeling result to obtain an emotional reply.

[0056] In this embodiment, the dialogue words "you let me down due to your comments, but still wish you well." and the specified emotion The process of generating an emotional response by the emotional response generating device 100 is described in detail:

[0057] First, combine the dialogue utterance "you let me down due to your comments, but still wish you well." with the specified emotion Input the dialogue module 10 to generate the response reply "i was hacked it would mean the world if you refollowed me". Secondly, input the dialogue utterance and the specified emotion into the emoticon module 20 to obtain the expression containing and Then, the response reply and the specified emoticon set are input into the position module 30 to obtain the sequence labeling result [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1]. Finally, the integration module 40 obtains the emotional reply "i was hacked" according to the input sequence labeling result, the specified emoticon set and the response reply. it would mean the world if you followed me ”, inserted in the emotional response and The emotional response is basically consistent with the previous and subsequent contexts and emotions, which enables it to imitate human reactions, not only visually enhancing the expression of emotions, but also adjusting the atmosphere and relaxing the rhythm of language.

[0058] In this embodiment, it is not mandatory to add emoticons to every sentence, which inevitably leads to the abuse of emoticons, stiffness and deliberateness, visual fatigue, a perfunctory feeling to the interlocutor, and an unpleasant chatting experience. In order to be more in line with real social scenarios, the specific setting of the emotional reply generation device 100 is: when there are no emoticons in the specified emoticon set, the emotional reply is a response reply; all judgment results of the sequence labeling results can be 0 at the same time, that is, the response reply without inserting any emoticons is regarded as an emotional reply.

[0059] In this embodiment, the emotional vectors, i.e., the values ​​of the emotion vectors, corresponding to some emoticons in the emoticon library are adjusted according to the currently popular Internet language and people's usage habits. Specifically, the emotion vector values ​​of multiple emotion symbols with similar emotional characteristics are interchanged, thereby constructing a new emoticon library for the emotional response generation device 100, thereby improving the expression accuracy of the emotional response. This is because the frequency of appearance of the adjusted emoticons is more in line with the application frequency in the real world.

[0060] Figure 4 4 is a heat diagram of three groups of emoticons that interchange emotion vector values ​​in an embodiment of the present invention.

[0061] like Figure 4 As shown, and For the first set of swapped emojis, and Swap emojis for the second set, and This is the third group of interchangeable emojis. The horizontal axis represents the quantitative value of each emoji for the eight basic emotions, namely anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. The bar on the right indicates that the color of the corresponding color block darkens as the quantitative value increases. It can be seen that the emojis in each group that can exchange emotional vector values ​​have similar characteristics in the quantitative values ​​of each basic emotion.

[0062] In this embodiment, in order to verify the effect of the emotional response generated by the emotional response generation device 100, the existing Attn-S2S model, CVAE model, Mojitalk model and CARE model are used as comparison models to compare the generation effects under the conditions of different dialogue discourses and specified emotions. The generation effect comparison results are shown in the following table:

[0063]

[0064]

[0065] The first row in the table above is different dialogue words, the second row is different specified emotions, the third to sixth rows are the responses generated by the Attn-S2S model, the CVAE model, the Mojitalk model and the CARE model respectively, and the seventh row is the emotional response generated by the emotional response generation device 100. For example, the cell in the third column of the fourth row represents the combination of the dialogue words "I shall not suffer alone!" and the specified emotion. The response obtained by inputting into the CVAE model is “I don’t want to lie. I’m not going to lie.” As can be seen from the above table, the emotional response generating device 100 can generate better emotional responses by inserting appropriate emoticons in appropriate contexts compared to other existing models. Although other models randomly generate a small number of emoticons, they ignore more factors, resulting in unnatural and unreasonable responses. For example, the Mojitalk model responds to the conversation content “I don’t want to suffer alone!” with “I’m by your side now.” ", however the specified emotion is Sad and heartbroken, The expression of happiness is inconsistent, that is, no reasonable response with emoticons is generated based on the specified emotion and the conversation content. In addition, for the conversation content "This work starts tonight and reaches its peak tomorrow night," the corresponding response generated by the emotional response generation device 100 does not insert emoticons. This is because the conversation scene involves more serious topics or rational discussions, such as work, plans, and academics. To avoid excessive entertainment, the insertion of emoticons is reduced.

[0066] Functions and Effects of the Embodiments

[0067] According to the deep learning-based emotional response generation device with emoticons involved in this embodiment, the emoticon module extracts the emotional state in the conversational speech, and then combines it with the specified emotion to obtain the emotion vector. The Euclidean distance is then calculated with the emoticons in the emoticon library to obtain the emoticon that best matches the emotion vector. The position module is then used to determine the position in the response where the emoticon needs to be inserted, thereby generating an emotional response with emoticons. In short, this method can generate rich and accurate responses in the conversation.

[0068] The above embodiments are preferred examples of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A deep learning-based emotional response generation device with emoticons, used to obtain corresponding emotional responses based on dialogue discourse, specified emotions and emoticon libraries, characterized in that: include: A dialogue module, configured to obtain a response reply based on the dialogue utterance and the specified emotion; The dialogue module includes a trained conditional variational autoencoder (CVAE). According to the existing discourse X and emotion set E and the corresponding original response Y ′ The training set is used to train the initial conditional variational autoencoder CVAE to obtain the trained conditional variational autoencoder CVAE. The loss function used in the training process is The calculation formula is: Where z is the latent variable, Y is the maximized response, and C x,e For given conditions, C x,e =[o x ;e′],o x is the output of the encoder of the conditional variational autoencoder CVAE after processing the speech X, e′ is the embedding vector of the emotion category specified from the emotion set E, and p D (Y|z,C x,e ) is the prior model used to sample z from the prior distribution, q R (z|Y,C x,e ) is the recognition network used to approximate the posterior distribution of the latent variable z, p P (z|C x,e ) is the variational lower bound of maximizing the given conditional likelihood obtained by approximate calculation, and KL is the KL divergence; An emoticon module, configured to obtain a specified emoticon set according to the dialogue discourse, the specified emotion, and the emoticon library; a location module, configured to obtain a sequence labeling result according to the response reply and the specified emoticon set; An integration module is used to integrate the sequence labeling result, the specified emoticon set and the response reply to obtain the emotional reply, The emoticon module includes an emotion detection submodule, an emotion quantification submodule and an emoticon calculation submodule. The emotion detection submodule is used to obtain corresponding multiple emotional states according to the dialogue words. The emotion quantification submodule is used to quantify the specified emotion and all the emotional states respectively to obtain corresponding emotion vectors. The emoticon calculation submodule is used to perform calculations based on the emotion vector of the designated emotion, the emotion vectors of each of the emotional states, and the emoticon library to obtain multiple emoticons in the emoticon library as the designated emoticon set.

2. The deep learning-based emotional response generation device with emoticons according to claim 1, characterized in that: in, The emotion detection submodule includes an Embedding layer, a Bi-LSTM layer, an attention layer, and a Softmax layer. The Embedding layer is used to map each word in the dialogue into a corresponding normalized vector. The Bi-LSTM layer is used to encode the normalized vector to obtain hidden layer features. The attention layer is used to process the hidden layer features and the normalized vector according to the Attention mechanism to obtain the vector representation of the dialogue utterance. The Softmax layer is used to perform a Softmax function operation on the vector representation to obtain the emotional state.

3. The deep learning-based emotional response generation device with emoticons according to claim 1, characterized in that: in, The emotion quantification submodule quantifies the designated emotion and the emotional state according to the emoticon quantification rules of the emoticon library.

4. The deep learning-based emotional response generation device with emoticons according to claim 1, characterized in that: in, In the emoticon calculation submodule, the emotion vector V corresponding to the emotional state is calculated based on the emotion vector of the specified emotion and the emotion vector of the emotional state, and then the Euclidean distance between the emotion vector V and the vector of each emoticon in the emoticon library is calculated, and the emoticon corresponding to the smallest Euclidean distance is selected as the emoticon in the specified emoticon set.

5. The deep learning-based emotional response generation device with emoticons according to claim 1, characterized in that: in, When the designated emoticon set does not contain any of the emoticons, the emotional reply is the response reply.

6. The deep learning-based emotional response generation device with emoticons according to claim 1, characterized in that: in, The sequence labeling result is a judgment result of whether an emoticon from the specified emoticon set is inserted after each word in the response reply, and the judgment result is 0 or 1, 0 indicates that the emoticon is not inserted after the corresponding word, and 1 indicates that the emoticon is inserted after the corresponding word.

7. The deep learning-based emotional response generation device with emoticons according to claim 6, characterized in that: in, All the judgment results of the sequence labeling result can be 0 at the same time, that is, the response reply without inserting any emoticon is regarded as the emotional reply.

8. The deep learning-based emotional response generation device with emoticons according to claim 1, characterized in that: in, The integration module includes an emoticon gating, and the emoticon gating is used to insert the corresponding emoticon in the emoticon set into the corresponding position of the response reply according to the sequence labeling result to obtain the emotional reply.

Images