Method and device for identifying user emotions in human-computer dialogue

Abstract

Embodiments of the present disclosure provide a method and apparatus for identifying user emotions in human-computer dialogue. The method includes: obtaining first voice data, wherein the first voice data is the user's voice data in the current round; extracting a first audio feature representing the emotion and a first text feature representing the emotion from the first voice data; obtaining at least one of a second text feature and a second audio feature, wherein the second text feature is a weighted sum of the user's text features in each of the previous N rounds, and the second audio feature is a weighted sum of the user's audio features in each of the previous N rounds, where N is a positive integer; concatenating the first audio feature, the first text feature, and at least one of the second text feature and the second audio feature into a concatenated feature; and identifying the user's first emotion in the current round based on the concatenated feature using a trained multimodal model.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a method and apparatus for identifying user emotions in human-computer dialogue. Background Art

[0002] In psychology, human emotions can be categorized into seven main types: joy, anger, sadness, fear, disgust, surprise, and envy. When a user interacts with a robot, the robot can identify the user's current emotions based on the text data they input and provide appropriate interactive content. This improves the user experience during the interaction. Summary of the Invention

[0003] The embodiments described herein provide a method, apparatus, and computer-readable storage medium storing a computer program for identifying user emotions in a human-computer dialogue.

[0004] According to a first aspect of the present disclosure, a method for identifying user emotions in human-computer dialogue is provided. The method includes: obtaining first speech data, wherein the first speech data is speech data of the user in the current round; extracting a first audio feature representing the emotion and a first text feature representing the emotion from the first speech data; obtaining at least one of a second text feature and a second audio feature, wherein the second text feature is a weighted sum of text features representing the emotion of the user in each of the previous N rounds, and the second audio feature is a weighted sum of audio features representing the emotion of the user in each of the previous N rounds, where N is a positive integer; concatenating the first audio feature, the first text feature, and at least one of the second text feature and the second audio feature into a concatenated feature; and identifying the first emotion of the user in the current round based on the concatenated feature using a trained multimodal model.

[0005] In some embodiments of the present disclosure, the weights of the various text features in the second text features increase in chronological order.

[0006] In some embodiments of the present disclosure, the weights of the respective audio features in the second audio features are increased in chronological order.

[0007] In some embodiments of the present disclosure, the method further includes: obtaining first video data, wherein the first video data includes facial image data of the user in the current round; extracting first image features representing emotions from the first video data; and splicing the first image features into spliced features.

[0008] In some embodiments of the present disclosure, the method further includes: obtaining a second image feature, wherein the second image feature is a weighted sum of image features representing the user's emotions in each of the first N rounds; and splicing the second image feature into the spliced feature.

[0009] In some embodiments of the present disclosure, the method also includes: obtaining first physiological characteristic data, wherein the first physiological characteristic data is the physiological characteristic data of the user in this round; extracting a first physiological characteristic representing emotion from the first physiological characteristic data; and splicing the first physiological characteristic into the spliced feature.

[0010] In some embodiments of the present disclosure, the method further includes: obtaining a second physiological feature, wherein the second physiological feature is a weighted sum of the physiological features representing the user's emotions in each of the first N rounds; and splicing the second physiological feature into the spliced feature.

[0011] In some embodiments of the present disclosure, user emotions include emotion categories and emotion intensity, and the emotion intensity includes multiple emotion intensity levels. The method also includes: obtaining multiple second emotions of the user in each of the first N rounds; in response to the emotion categories of the multiple second emotions being the same as the emotion categories of the first emotion, determining an emotion intensity level change curve according to the emotion intensity levels of the multiple second emotions; and in response to the determined emotion intensity level change curve being the same as one of the preset multiple emotion intensity level change curves, predicting the user's emotion intensity in this round according to the determined emotion intensity level change curve, and adjusting the emotion intensity of the first emotion based on the predicted emotion intensity.

[0012] In some embodiments of the present disclosure, the method also includes: in response to the emotion category of multiple second emotions being the same as the emotion category of the first emotion, predicting the emotion intensity development trend based on the emotion intensity levels of the multiple second emotions and the emotion intensity level of the first emotion; and providing a reply to the user based on the user's first emotion and the predicted emotion intensity development trend.

[0013] According to a second aspect of the present disclosure, a device for identifying user emotions in human-computer dialogue is provided. The device includes at least one processor; and at least one memory storing a computer program. When the computer program is executed by the at least one processor, the device obtains first voice data, wherein the first voice data is the voice data of the user in this round; extracts a first audio feature representing the emotion and a first text feature representing the emotion from the first voice data; obtains at least one of a second text feature and a second audio feature, wherein the second text feature is a weighted sum of the text features representing the emotion of the user in each of the first N rounds, and the second audio feature is a weighted sum of the audio features representing the emotion of the user in each of the first N rounds, wherein N is a positive integer; concatenates the first audio feature, the first text feature, and at least one of the second text feature and the second audio feature into a concatenated feature; and identifies the first emotion of the user in this round based on the concatenated feature using a trained multimodal model.

[0014] In some embodiments of the present disclosure, the computer program, when executed by at least one processor, causes the device to further: obtain first video data, wherein the first video data includes facial image data of the user in the current round; extract first image features representing emotions from the first video data; and splice the first image features into spliced features.

[0015] In some embodiments of the present disclosure, the computer program, when executed by at least one processor, causes the device to further: obtain a second image feature, wherein the second image feature is a weighted sum of image features representing the user's emotions in each of the first N rounds; and splice the second image feature into the spliced feature.

[0016] In some embodiments of the present disclosure, when the computer program is executed by at least one processor, the device further: obtains first physiological characteristic data, wherein the first physiological characteristic data is the physiological characteristic data of the user in this round; extracts a first physiological characteristic representing an emotion from the first physiological characteristic data; and splices the first physiological characteristic into a spliced feature.

[0017] In some embodiments of the present disclosure, the computer program, when executed by at least one processor, causes the device to further: obtain a second physiological feature, wherein the second physiological feature is a weighted sum of the physiological features of the user representing the emotions in each of the first N rounds; and splice the second physiological feature into the spliced feature.

[0018] In some embodiments of the present disclosure, user emotions include emotion categories and emotion intensity, and the emotion intensity includes multiple emotion intensity levels. When the computer program is executed by at least one processor, the device causes the device to further: obtain multiple second emotions of the user in each of the first N rounds; in response to the emotion categories of the multiple second emotions being the same as the emotion categories of the first emotion, determine an emotion intensity level change curve according to the emotion intensity levels of the multiple second emotions; and in response to the determined emotion intensity level change curve being the same as one of the preset multiple emotion intensity level change curves, predict the user's emotion intensity in this round according to the determined emotion intensity level change curve, and adjust the emotion intensity of the first emotion based on the predicted emotion intensity.

[0019] In some embodiments of the present disclosure, the computer program, when executed by at least one processor, causes the device to also: in response to the emotion category of the multiple second emotions being the same as the emotion category of the first emotion, predict an emotion intensity development trend based on the emotion intensity levels of the multiple second emotions and the emotion intensity level of the first emotion; and provide a response to the user based on the user's first emotion and the predicted emotion intensity development trend.

[0020] According to a third aspect of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein the computer program implements the steps of the method according to the first aspect of the present disclosure when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be noted that the drawings described below only relate to some embodiments of the present disclosure and are not intended to limit the present disclosure.

[0022] Figure 1 is an exemplary flow chart of a method for identifying user emotions in a human-computer dialogue according to an embodiment of the present disclosure;

[0023] Figure 2 is an exemplary flow chart of further steps for identifying user emotions in a human-computer dialogue according to an embodiment of the present disclosure; and

[0024] Figure 3 4 is a schematic block diagram of an apparatus for recognizing user emotions in human-computer dialogue according to an embodiment of the present disclosure.

[0025] Elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work also fall within the scope of protection of the present disclosure.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the subject matter of the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the specification and the relevant art, and will not be interpreted in an idealized or overly formal manner unless otherwise explicitly defined herein. In addition, terms such as "first" and "second" are only used to distinguish one component (or a portion of a component) from another component (or another portion of a component).

[0028] In human-computer interaction, if a robot determines a user's emotion based solely on text input, misunderstandings are inevitable. For example, "The weather is so nice today" can be a compliment or a resigned expression of dismal weather. Therefore, in some human-computer interaction scenarios, the user can be allowed to directly express their desired expression. This allows the robot to obtain not only the user's spoken content but also the user's spoken voice data, thereby extracting the user's audio features. The robot can determine the user's textual emotion label based on the user's spoken content, then determine the user's spoken voice emotion label based on the user's spoken voice, and then combine the textual and spoken emotion labels to determine a composite emotion label. However, in this solution, the textual emotion label determination process and the spoken emotion label determination process are independent, and the correlation between textual and spoken features is not utilized to determine the composite emotion label. In some cases, the difference between the textual emotion label and the spoken emotion label can be significant. If either model identifies an inaccurate emotion label, the composite emotion label may also be inaccurate. Furthermore, these solutions often only analyze the emotion expressed in a single sentence, failing to analyze the emotions expressed in the context of the human-computer interaction. Therefore, the identified emotion may also be inaccurate.

[0029] Figure 1 An exemplary flow chart of a method for identifying user emotions in a human-machine dialogue according to an embodiment of the present disclosure is shown. A human-machine dialogue may include multiple rounds of dialogue. Each round of dialogue includes a user's speech and a robot's response.

[0030] exist Figure 1 At block S102, first voice data is obtained, where the first voice data is the user's voice data in the current round. In some embodiments of the present disclosure, the first voice data may be collected in real time, for example, using a microphone. In other embodiments of the present disclosure, the first voice data may be obtained, for example, by reading an audio file sent by the user.

[0031] At box S104, a first audio feature representing an emotion and a first text feature representing an emotion are extracted from the first speech data. In some embodiments of the present disclosure, the first audio feature representing an emotion and the first text feature representing an emotion in the first speech data can be simultaneously extracted, for example, by a multimodal feature extractor. The multimodal feature extractor is, for example, a Transformer model. In other embodiments of the present disclosure, the first audio feature representing an emotion in the first speech data can be extracted, for example, by an audio feature extractor. Natural language processing (NLP) technology is used to convert the first speech data into first text data. Then, a text feature extractor is used to extract the first text feature representing the emotion in the first text data.

[0032] In some embodiments of the present disclosure, the following operations may be performed on the first speech data to extract the first audio feature: pre-emphasis, framing, windowing, discrete Fourier transform (FFT), passing through a Mel filter, logarithmic operation, and discrete cosine transform (DCT).

[0033] The first audio feature is, for example, one or more of the following: energy (E), zero crossing rate (ZCR), fundamental frequency (F0), first formant (FF), Mel-frequency cepstral coefficient (MFCC), linear prediction coefficient (LPC), short-time average amplitude (SAM), etc., and their statistical information, such as maximum value (Max), minimum value (Min), mean, variance (Var), first-order difference (FOD), rate of change (RC), amplitude disturbance parameter, etc. In one example, the first audio feature includes at least MFCC features extracted from the fundamental frequency curve and amplitude disturbance parameters extracted from the short-time average amplitude curve.

[0034] In some embodiments of the present disclosure, the first text feature is, for example, a word vector formed by words expressing emotions.

[0035] At block S106, at least one of a second text feature and a second audio feature is obtained. The second text feature is the weighted sum of the text features representing the user's emotions in each of the first N rounds, and the second audio feature is the weighted sum of the audio features representing the user's emotions in each of the first N rounds. N is a positive integer. In one example, the value of N can be an empirical value. The value of N can be selected based on a trade-off between computational complexity and emotion recognition accuracy.

[0036] In some embodiments of the present disclosure, the weights of the various text features in the second text features are increased in chronological order. This allows the second text features to take into account more of the emotions expressed in the content of recent conversations. In some embodiments of the present disclosure, the weights of the various audio features in the second audio features are increased in chronological order. This allows the second audio features to take into account more of the emotions expressed in the tone of recent conversations.

[0037] At block S108, at least one of the first audio feature, the first text feature, the second text feature, and the second audio feature is concatenated into a concatenated feature. In some embodiments of the present disclosure, the first audio feature, the first text feature, and the second text feature are concatenated into a concatenated feature. In other embodiments of the present disclosure, the first audio feature, the first text feature, and the second audio feature are concatenated into a concatenated feature. In still other embodiments of the present disclosure, the first audio feature, the first text feature, the second text feature, and the second audio feature are concatenated into a concatenated feature.

[0038] In some embodiments of the present disclosure, in the concatenated features, the order of at least one of the first audio feature, the first text feature, the second text feature, and the second audio feature can be arbitrary or set based on empirical values. In some embodiments of the present disclosure, the dimensions of the first audio feature, the dimensions of the first text feature, and the dimensions of at least one of the second text feature and the second audio feature can be different.

[0039] Furthermore, in addition to determining the user's emotions through the audio features and text features of the user's speech, the user's emotions may also be determined through the user's facial expressions and / or physiological features.

[0040] In some embodiments of the present disclosure, first video data may be acquired. The first video data includes facial image data of the user during the current round. Then, first image features representing emotions are extracted from the first video data. For example, when a user is happy, the corners of their mouth may rise, and when angry, their brows may furrow. Therefore, these image features can represent the user's emotions. The first image features may be spliced into spliced features. The spliced features thus further include the first image features.

[0041] In some embodiments of the present disclosure, a second image feature can be obtained. The second image feature is the weighted sum of the image features representing the user's emotions in each of the first N rounds. The second image feature can then be concatenated into the concatenated feature. In some embodiments of the present disclosure, the weights of the individual image features in the second image feature are assigned increasing values chronologically. This allows the second image feature to better consider the emotions expressed by the user's facial expressions in recent rounds of conversation.

[0042] In some embodiments of the present disclosure, first physiological characteristic data may be obtained. The first physiological characteristic data is the physiological characteristic data of the user during the current round. A first physiological feature representing emotion is extracted from the first physiological characteristic data. For example, when a user is emotionally intense, their heart rate and blood pressure increase, while when they are emotionally calm, their heart rate and blood pressure are normal. Therefore, physiological characteristics such as heart rate and blood pressure can represent the user's emotion. The first physiological feature can be spliced into the spliced feature.

[0043] In some embodiments of the present disclosure, a second physiological feature can be obtained. This second physiological feature is the weighted sum of the user's physiological features representing emotions in each of the first N rounds. The second physiological feature can then be concatenated into the concatenated feature. In some embodiments of the present disclosure, the weights of the individual physiological features in the second physiological feature are assigned increasing chronologically. This allows the second physiological feature to better reflect the emotions reflected by the user's physiological features in the most recent rounds of conversation.

[0044] At block S110 , the user's first emotion in the current round is identified using a trained multimodal model based on the concatenated features. In some embodiments of the present disclosure, the multimodal model is a multimodal emotion classification model. The multimodal model can be trained using a supervised training method.

[0045] As described above, the spliced features may include: a first audio feature, a first text feature, and at least one of a second text feature and a second audio feature. In this way, the user's tone and semantics in the current round, as well as their tone and / or semantics in the previous N rounds, can be used to identify the user's emotion in the current round. Furthermore, the spliced features may also include one or more of the first video data, the second video data, the first physiological characteristic data, and the second physiological characteristic data. In this way, the user's facial expressions and / or physiological characteristics in the current round of conversation and in previous conversations can also be used to identify emotions.

[0046] The embodiments of the present disclosure can integrate the multimodal features of a user to identify the user's emotions. These features are correlated with each other, so the user's emotions can be identified more accurately.

[0047] Figure 2 An exemplary flow chart illustrating further steps for identifying user emotions in human-computer dialogue according to an embodiment of the present disclosure. In some embodiments of the present disclosure, user emotions may include emotion categories and emotion intensity. Emotion categories include, for example, joy, anger, sadness, fear, disgust, surprise, envy, etc. Emotion intensity may include multiple levels of emotion intensity. For example, emotion intensity may include five levels: a, b, c, d, and e. The emotion intensity represented by these five levels increases in order from a to e. Level a represents the weakest intensity, and level e represents the strongest intensity.

[0048] exist Figure 2 At block S202, a plurality of second emotions of the user in each of the first N rounds may be obtained. The second emotions may be understood as the user's historical emotions.

[0049] At block S204, a determination is made as to whether the emotion categories of the plurality of second emotions are the same as the emotion category of the user's first emotion in this round. If the emotion categories of the plurality of second emotions are the same as the emotion category of the user's first emotion in this round ("yes" at block S204), it indicates that the user's emotions are substantially the same as before. The process proceeds to block S206, where an emotion intensity level change curve is determined based on the emotion intensity levels of the plurality of second emotions. The determined emotion intensity level change curve may be, for example: a, a, b, c; a, c, b, a; e, d, c, b; e, a, c, a, and so on.

[0050] Based on statistical information about user emotions manually recorded during a large number of human-computer conversations, multiple emotion intensity level change curves can be pre-set. Examples of these multiple emotion intensity level change curves include: a, b, c, d; a, b, c, e; b, c, d, e; c, d, e, d, and so on. In one example, the multiple preset emotion intensity level change curves can be change curves consisting of all permutations and combinations of the multiple emotion intensity levels included in the emotion intensity. Based on this statistical information, the emotion intensity level corresponding to each preset emotion intensity level change curve in the next round can be determined.

[0051] At block S208, a determination is made as to whether the determined emotion intensity level change curve is identical to (matches) one of the plurality of preset emotion intensity level change curves. If the determined emotion intensity level change curve is identical to one of the plurality of preset emotion intensity level change curves ("yes" at block S208), then at block S210, the user's emotion intensity in the current round may be predicted based on the determined emotion intensity level change curve. The predicted emotion intensity may be the emotion intensity level corresponding to the matched emotion intensity level change curve in the next round.

[0052] Then, at block S212, the emotional intensity of the first emotion may be adjusted based on the predicted emotional intensity. For example, the emotional intensity of the first emotion may be adjusted to a weighted sum of the emotional intensity of the first emotion and the predicted emotional intensity. In one example, if the emotional intensity of the first emotion is e and the predicted emotional intensity is c, the emotional intensity of the first emotion may be adjusted to a value d intermediate between e and c.

[0053] If the emotion categories of the plurality of second emotions are different from the emotion category of the user's first emotion in this round ("No" at block S204), it indicates that the user's emotions change too quickly and it is not appropriate to use the user's second emotions to adjust the user's first emotion. In this case, the process proceeds to block S214 and the process ends.

[0054] If the determined emotion intensity level change curve is different from any of the preset emotion intensity level change curves (“Yes” at block S208 ), the process proceeds to block S214 and the process ends.

[0055] Furthermore, in some embodiments of the present disclosure, if the emotion category of multiple second emotions is the same as the emotion category of the first emotion, the emotion intensity development trend is predicted based on the emotion intensity levels of the multiple second emotions and the emotion intensity level of the first emotion. In one example, the emotion intensity development trend can be predicted based on statistical information. In another example, the difference between the emotion intensity levels of each two (in time) adjacent emotion in the multiple second emotions and the first emotion can be calculated, and the emotion intensity development trend can be predicted by the average value of the difference. If the average value is a positive number, it indicates that the emotion intensity will increase. If the average value is a negative number, it indicates that the emotion intensity will decrease.

[0056] Then, a response can be provided to the user based on the user's first emotion and the predicted trend of emotion intensity. In one example, if the user's first emotion is anger at level c, and the predicted trend of emotion intensity is increasing, then more soothing expressions should be added to the response to prevent the user's anger from escalating.

[0057] Figure 3 FIG. 3 is a schematic block diagram of an apparatus 300 for identifying user emotions in a human-computer dialogue according to an embodiment of the present disclosure. Figure 3 As shown, the apparatus 300 may include a processor 310 and a memory 320 storing a computer program. When the computer program is executed by the processor 310, the apparatus 300 may perform the following operations: Figure 1 The steps of method 100 shown. In one example, the device 300 can be a computer device or a cloud computing node. The device 300 can obtain first voice data. The first voice data is the voice data of the user in this round. The device 300 can extract a first audio feature representing an emotion and a first text feature representing an emotion from the first voice data. The device 300 can obtain at least one of a second text feature and a second audio feature. The second text feature is the weighted sum of the text features representing the emotion of the user in each of the first N rounds, and the second audio feature is the weighted sum of the audio features representing the emotion of the user in each of the first N rounds. N is a positive integer. The device 300 can splice the first audio feature, the first text feature, and at least one of the second text feature and the second audio feature into a spliced feature. The device 300 can identify the first emotion of the user in this round based on the spliced feature through a trained multimodal model.

[0058] In some embodiments of the present disclosure, device 300 may obtain first video data. The first video data may include facial image data of a user during a current round. Device 300 may extract first image features representing emotions from the first video data. Device 300 may splice the first image features into spliced features.

[0059] In some embodiments of the present disclosure, the device 300 may obtain a second image feature, which is a weighted sum of the image features representing the user's emotions in each of the first N rounds. The device 300 may splice the second image feature into the spliced feature.

[0060] In some embodiments of the present disclosure, device 300 may obtain first physiological characteristic data. The first physiological characteristic data may be physiological characteristic data of the user during the current round. Device 300 may extract a first physiological feature representing emotion from the first physiological characteristic data. Device 300 may splice the first physiological feature into a spliced feature.

[0061] In some embodiments of the present disclosure, the device 300 may obtain a second physiological feature, wherein the second physiological feature is a weighted sum of the physiological features representing the user's emotions in each of the first N rounds. The device 300 may splice the second physiological feature into the spliced feature.

[0062] In some embodiments of the present disclosure, user emotions include emotion categories and emotion intensity, and the emotion intensity includes multiple emotion intensity levels. The device 300 can obtain multiple second emotions of the user in each of the first N rounds. In response to the emotion categories of the multiple second emotions being the same as the emotion category of the first emotion, the device 300 can determine an emotion intensity level change curve based on the emotion intensity levels of the multiple second emotions. In response to the determined emotion intensity level change curve being the same as one of the multiple preset emotion intensity level change curves, the device 300 can predict the user's emotion intensity in the current round based on the determined emotion intensity level change curve, and adjust the emotion intensity of the first emotion based on the predicted emotion intensity.

[0063] In some embodiments of the present disclosure, in response to the emotion category of multiple second emotions being the same as the emotion category of the first emotion, the device 300 may predict the emotion intensity development trend based on the emotion intensity levels of the multiple second emotions and the emotion intensity level of the first emotion, and provide a response to the user based on the user's first emotion and the predicted emotion intensity development trend.

[0064] In an embodiment of the present disclosure, the processor 310 may be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, etc. The memory 320 may be any type of memory implemented using data storage technology, including but not limited to random access memory, read-only memory, semiconductor-based memory, flash memory, disk storage, etc.

[0065] Furthermore, in embodiments of the present disclosure, apparatus 300 may also include input devices 330, such as a microphone, keyboard, mouse, camera, or physiological characteristic detection instrument, for inputting the user's audio, video, and physiological characteristic data. Furthermore, apparatus 300 may also include output devices 340, such as a loudspeaker or display, for outputting the robot's responses.

[0066] In other embodiments of the present disclosure, a computer-readable storage medium storing a computer program is further provided, wherein the computer program can achieve the following when executed by a processor: Figures 1 to 2 The steps of the method are shown.

[0067] In summary, the embodiments of the present disclosure can identify the user's emotions by integrating the user's multimodal features. These features are interrelated, so the user's emotions can be identified more accurately. Furthermore, the embodiments of the present disclosure can also predict the user's emotional intensity in this round based on the user's emotional intensity level change curve, thereby adjusting the identified user emotions from more dimensions to further improve the accuracy of user emotion recognition. In addition, the embodiments of the present disclosure can also predict the development trend of the user's emotional intensity, and reply to the user based on the user's emotions and the predicted emotional intensity development trend to provide a more thoughtful human-computer dialogue.

[0068] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0069] Unless the context clearly indicates otherwise, as used herein and in the appended claims, the singular includes the plural, and vice versa. Thus, when referring to the singular, the plural of the corresponding term is generally included. Similarly, the words "include" and "comprising" are to be interpreted as inclusive rather than exclusive. Likewise, the terms "include" and "or" should be interpreted as inclusive unless such interpretation is expressly prohibited herein. Where the term "example" is used herein, particularly when it follows a group of terms, the "example" is merely exemplary and illustrative and should not be considered exclusive or comprehensive.

[0070] Further aspects and scope of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended to be illustrative only and are not intended to limit the scope of the present application.

[0071] Several embodiments of the present disclosure have been described in detail above, but it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The scope of protection of the present disclosure is defined by the appended claims.

Claims

1. A method for identifying user emotions in a human-computer dialogue, comprising: Acquire first voice data, wherein the first voice data is the user's voice data in this round; Extracting a first audio feature representing an emotion and a first text feature representing an emotion from the first speech data; A multimodal feature extractor is used to simultaneously extract a first audio feature representing an emotion and a first text feature representing an emotion from the first speech data; an audio feature extractor is used to extract a first audio feature representing an emotion from the first speech data, wherein the first audio feature comprises at least a Mel-frequency cepstral coefficient feature extracted from a basic frequency curve and an amplitude perturbation parameter extracted from a short-time average amplitude curve; Obtaining at least one of a second text feature and a second audio feature, wherein the second text feature is a weighted sum of text features representing emotions of the user in each of the first N rounds, and the second audio feature is a weighted sum of audio features representing emotions of the user in each of the first N rounds, where N is a positive integer; concatenating the first audio feature, the first text feature, and the at least one of the second text feature and the second audio feature into a concatenated feature; and identifying the first emotion of the user in this round based on the spliced features using a trained multimodal model; Obtain multiple second emotions of the user in each of the first N rounds, where the second emotions are used to represent the user's historical emotions; determine whether the emotion categories of the multiple second emotions are the same as the emotion category of the user's first emotion in this round, If the emotion categories of the multiple second emotions are different from the emotion category of the user's first emotion in this round, the judgment data that the user's emotion changes too quickly is obtained, and the data that the user's second emotion is not suitable for adjusting the user's first emotion is obtained, and the emotion prediction ends; If the emotion categories of the plurality of second emotions are the same as the emotion category of the user's first emotion in this round, an emotion intensity level change curve is determined according to the emotion intensity levels of the plurality of second emotions; and whether the determined emotion intensity level change curve is the same as one of the plurality of preset emotion intensity level change curves is determined. If the determined emotion intensity level change curve is the same as one of the preset multiple emotion intensity level change curves; predicting the user's emotion intensity in the current round based on the determined emotion intensity level change curve; If the determined emotion intensity level change curve is different from any of the preset emotion intensity level change curves, the emotion prediction is terminated.

2. The method according to claim 1, wherein The weight of each text feature in the second text feature increases in chronological order, and / or the weight of each audio feature in the second audio feature increases in chronological order.

3. The method according to claim 1, further comprising: Acquire first video data, wherein the first video data includes facial image data of the user in this round; Extracting a first image feature representing emotion from the first video data; and The first image feature is stitched into the stitched feature.

4. The method according to claim 3, further comprising: Obtaining a second image feature, wherein the second image feature is a weighted sum of image features representing the user's emotions in each of the first N rounds; and The second image feature is stitched into the stitched feature.

5. The method according to any one of claims 1 to 4, further comprising: Acquiring first physiological characteristic data, wherein the first physiological characteristic data is the physiological characteristic data of the user in this round; Extracting a first physiological feature representing emotion from the first physiological feature data; and The first physiological feature is stitched into the stitching feature.

6. The method according to claim 5, further comprising: Acquire a second physiological feature, wherein the second physiological feature is a weighted sum of the physiological features representing the emotions of the user in each of the first N rounds; and The second physiological feature is stitched into the stitching feature.

7. The method according to claim 1, wherein The user emotion includes an emotion category and an emotion intensity, and the emotion intensity includes multiple emotion intensity levels. The method further includes: Obtaining multiple second emotions of the user in each of the first N rounds; In response to the emotion categories of the plurality of second emotions being the same as the emotion category of the first emotion, determining an emotion intensity level change curve according to the emotion intensity levels of the plurality of second emotions; and In response to the determined emotion intensity level change curve being the same as one of a plurality of preset emotion intensity level change curves, the emotion intensity of the user in this round is predicted according to the determined emotion intensity level change curve, and the emotion intensity of the first emotion is adjusted based on the predicted emotion intensity.

8. The method according to claim 7, further comprising: In response to the emotion category of the plurality of second emotions being the same as the emotion category of the first emotion, predicting an emotion intensity development trend based on the emotion intensity levels of the plurality of second emotions and the emotion intensity level of the first emotion; as well as A response is provided to the user based on the first emotion of the user and the predicted emotion intensity development trend.

9. A device for identifying user emotions in a human-computer dialogue, comprising: at least one processor; as well as at least one memory storing a computer program; Wherein, when the computer program is executed by the at least one processor, the device is caused to perform the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, wherein: The computer program implements the steps of the method according to any one of claims 1 to 8 when executed by a processor.

Images