Emotion recognition method, device, computer equipment and storage medium

Abstract

The embodiment of the present invention discloses an emotion recognition method, apparatus, computer equipment and storage medium, the method comprising: obtaining voice data and text data of a user to be recognized; performing binary classification processing of N kinds of fine-grained emotion labels on the voice data and text data respectively, and outputting the user's first emotion set and second emotion set; determining the user's third emotion set based on a preset emotion probability threshold and the credibility of the first emotion set and the second emotion set; and determining the user's target emotion set using the first credibility emotion set, the second credibility emotion set, the emotion-related probability map and the opposing emotion table. Through the above method, the recognition of N kinds of fine-grained emotions can be achieved, the richness of emotion categories can be improved, and the target emotion set can be obtained through the emotion-related probability map and the opposing emotion table to clarify the user's true emotions. Compared with directly outputting the emotion recognition results, the accuracy of emotion recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to an emotion recognition method, device, computer equipment and storage medium. Background Art

[0002] Current emotion recognition technology primarily relies on machine learning within the field of artificial intelligence (AI), and is often based on deep learning techniques in computer vision, natural language processing, and speech technology. Deep learning is a groundbreaking area of ​​current machine learning technology.

[0003] Current emotion recognition systems mostly identify a person's emotions by analyzing facial expressions, voice, and conversation content. Some solutions also analyze emotions through physiological signals such as brain waves and electrodermal signals. However, these systems generally rely on text-based recognition of the six major emotions (anger, surprise, frustration, happiness, fear, and sadness) to ensure high accuracy.

[0004] However, the above method can identify fewer types of emotions. In AI dialogue, using only 6 emotions to configure dialogue strategies will make the robot's response strategy more rigid, which is not conducive to using AI robots to provide psychological counseling and preliminary psychological intervention for people with psychological distress.

[0005] Therefore, a solution that can recognize a wider range of emotions is urgently needed to better understand the user's current emotional state during conversations and help people with severe psychological distress. This is of great significance in alleviating the situation in which people with severe psychological distress may be reluctant to communicate with real people and may even engage in extreme behavior. Summary of the Invention

[0006] The main purpose of the present invention is to provide an emotion recognition method, apparatus, computer equipment and storage medium, which can solve the problem in the prior art of lacking a solution that can recognize more emotions.

[0007] To achieve the above objectives, the present invention provides, in a first aspect, a method for identifying emotions, the method comprising:

[0008] Acquire voice data and text data of a user to be recognized, wherein the text data is obtained by performing voice recognition on the voice data;

[0009] Inputting the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and outputting a first emotion set of the user; and inputting the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and outputting a second emotion set of the user, wherein the emotion set includes a correspondence between the emotion labels and the credibility;

[0010] Determining a third emotion set of the user based on the credibility of the first emotion set and the second emotion set, and a preset emotion probability threshold, wherein the third emotion set includes the first credibility emotion set and the second credibility emotion set;

[0011] The target emotion set of the user is determined using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map and the opposing emotion table. The emotion correlation probability map is used to reflect the emotion correlation probability between N fine-grained emotion tags. The emotion correlation probability is used to reflect the probability that one emotion also exists when another emotion exists. The opposing emotion table is used to reflect the irrelevant emotions corresponding to each emotion tag in the N fine-grained emotion tags. The irrelevant emotions refer to emotions that must not exist when one emotion exists.

[0012] In one feasible implementation, if the credibility of the first credibility emotion set is greater than the credibility of the second credibility emotion set, then determining the user's third emotion set based on the credibility of the first emotion set and the second emotion set and a preset emotion probability threshold includes:

[0013] Determining the first emotion labels in the first emotion set whose credibility is greater than or equal to the first emotion probability threshold as a first high-probability emotion subset;

[0014] Determining the second emotion labels in the second emotion set whose credibility probability is greater than or equal to the second emotion probability threshold as a second high probability emotion subset;

[0015] The first credibility emotion set is determined according to the first high-probability emotion subset and the second high-probability emotion subset.

[0016] In one possible implementation, the method further includes:

[0017] Determine the first emotion labels in the first emotion set whose credibility probability is greater than or equal to the third emotion probability threshold and less than or equal to the first emotion probability threshold as a first low-probability emotion subset, and the first emotion probability threshold is greater than the third emotion probability threshold;

[0018] Determine the second emotion labels in the second emotion set whose credibility probability is greater than or equal to the fourth emotion probability threshold and less than or equal to the second emotion probability threshold as a second low-probability emotion subset, and the second emotion probability threshold is greater than the fourth emotion probability threshold;

[0019] The second credibility emotion set is obtained by using the first low-probability emotion subset and the second low-probability emotion subset.

[0020] In a feasible implementation, determining the target emotion set of the user by using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map, and the opposing emotion table includes:

[0021] Determining whether there is an opposing emotion in the first credibility emotion set using the opposing emotion table;

[0022] When there are opposing emotions in the first credibility emotion set, determining the credibility difference between the opposing emotions;

[0023] If there is a first opposing emotion whose credibility difference is greater than or equal to a preset credibility difference threshold, deleting the emotion label with low credibility probability in the first opposing emotion from the first credibility emotion set, and updating the first credibility emotion set;

[0024] If there is a second opposing emotion whose credibility difference is less than a preset credibility difference threshold, determining whether there is a related emotion of the first highest credibility emotion label in the second opposing emotion by using the emotion correlation probability map and the first highest credibility emotion label in the first credibility emotion set;

[0025] If any emotion label in the second opposing emotions is a related emotion of the first highest credibility emotion label, then the emotion labels in the second opposing emotions other than the related emotions are deleted from the first credibility emotion set, and the first credibility emotion set is updated; if any emotion label in the second opposing emotions does not have a related emotion of the first highest credibility emotion label, then the emotion labels in the second opposing emotions other than the second highest credibility emotion label are deleted from the first credibility emotion set, and the first credibility emotion set is updated;

[0026] When there is no opposing emotion in the first credibility emotion set, deleting the emotion tags other than the emotion tag with the highest credibility in the first credibility emotion set, and updating the first credibility emotion set;

[0027] A target emotion set is determined using the second credibility emotion set, the updated first credibility emotion set, and the emotion-related probability map.

[0028] In a feasible implementation, determining the target emotion set by using the second credibility emotion set, the updated first credibility emotion set, and the emotion-related probability map includes:

[0029] Determining, using the emotion correlation probability graph, whether the first emotion label in the second credibility emotion set and the second emotion label in the updated first credibility emotion set are related emotions;

[0030] When the first emotion label and the second emotion label are related emotions, searching the emotion correlation probability map for the emotion correlation probability of the first emotion label corresponding to the second emotion label;

[0031] If the product of the emotion correlation probability and the credibility of the second emotion label in the updated first credibility emotion set is greater than or equal to a preset correlation probability threshold, retaining the first emotion label in the second credibility emotion set;

[0032] If the product of the emotion correlation probability and the credibility of the second emotion label in the updated first credibility emotion set is less than a preset correlation probability threshold, or when the first emotion label and the second emotion label are unrelated emotions, deleting the first emotion label from the second credibility emotion set and updating the second credibility emotion set;

[0033] The union of the updated second credibility emotion set and the updated first credibility emotion set is used as the target emotion set.

[0034] In a feasible implementation, the method of determining the target emotion set of the user by using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map, and the opposing emotion table further includes:

[0035] Acquire a training sample data set, wherein the training sample data set includes a plurality of emotion sample data for training, each of the emotion sample data corresponds to an emotion label set, and each emotion label set includes multiple emotion labels;

[0036] Counting a target emotion tag set including an nth emotion tag and a first set number of the target emotion tag set, where n belongs to N fine-grained emotion tags and the value of n ranges from 1 to N;

[0037] For each third emotion label, counting the number of second sets of all target emotion label sets that include the third emotion label, where the third emotion label is any fine-grained emotion label other than the nth emotion label;

[0038] Using the ratio of the number of the first set to the number of the second set as the emotion correlation probability between the emotion of the third emotion label and the nth emotion label, obtaining the emotion correlation probability between each third emotion label and the nth emotion label;

[0039] The third emotion tag whose emotion-related probability is greater than a preset related probability threshold is used as the related emotion of the nth emotion tag;

[0040] The emotion-related probability map is generated using the related emotions of each fine-grained emotion tag and the emotion-related probabilities of the related emotions.

[0041] In one feasible implementation, N is 40, and the steps of inputting the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels and outputting the user's first emotion set, and inputting the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels and outputting the user's second emotion set include:

[0042] The speech data is converted into a feature vector using the speech feature conversion algorithm of the speech emotion recognition model; the feature vector is input into the CNN model of the speech emotion recognition model to extract speech features and determine target speech features;

[0043] Based on the BERT migration algorithm of the text emotion recognition model and the emotion data, text feature extraction is performed on the text data to obtain target text features of the text data;

[0044] The target speech features and the target text features are respectively input into a fully connected layer with a sigmoid activation function to perform binary classification processing of 40 fine-grained emotion labels, and the first emotion set and the second emotion set of the user are output.

[0045] To achieve the above-mentioned object, the second aspect of the present invention provides an emotion recognition device, the device comprising:

[0046] Data acquisition module: used to acquire the voice data and text data of the user to be recognized, wherein the text data is obtained by performing voice recognition on the voice data;

[0047] Emotion recognition module: used to input the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the first emotion set of the user; and input the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the second emotion set of the user, wherein the emotion set includes the correspondence between emotion labels and credibility;

[0048] An emotion screening module is configured to determine a third emotion set of the user based on the credibility of the first emotion set and the second emotion set, and a preset emotion probability threshold, wherein the third emotion set includes the first credibility emotion set and the second credibility emotion set;

[0049] Emotion determination module: used to determine the target emotion set of the user by using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map and the opposing emotion table, the emotion correlation probability map is used to reflect the emotion correlation probability between N fine-grained emotion tags, the emotion correlation probability is used to reflect the probability that one emotion also exists when another emotion exists, and the opposing emotion table is used to reflect the irrelevant emotions corresponding to each emotion tag in the N fine-grained emotion tags, and the irrelevant emotions refer to emotions that must not exist when one emotion exists.

[0050] To achieve the above-mentioned objectives, the third aspect of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps shown in the first aspect and any feasible implementation method.

[0051] To achieve the above-mentioned objectives, the fourth aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps shown in the first aspect and any feasible implementation method.

[0052] The embodiments of the present invention have the following beneficial effects:

[0053] The present invention provides an emotion recognition method, which includes: obtaining voice data and text data of a user to be recognized, where the text data is obtained by performing voice recognition on the voice data; inputting the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and outputting a first emotion set of the user; and inputting the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and outputting a second emotion set of the user, where the emotion set includes a correspondence between emotion labels and credibility; determining a third emotion set of the user based on the credibility in the first emotion set and the second emotion set, and a preset emotion probability threshold, where the third emotion set includes a first credibility emotion set and a second credibility emotion set; and determining a target emotion set of the user using the first credibility emotion set, the second credibility emotion set, an emotion correlation probability map, and an opposing emotion table, where the emotion correlation probability map is used to reflect the emotion correlation probability between the N fine-grained emotion labels, the emotion correlation probability is used to reflect the probability that one emotion also exists when another emotion exists, and the opposing emotion table is used to reflect irrelevant emotions corresponding to each emotion label in the N fine-grained emotion labels, where irrelevant emotions refer to emotions that must not exist when one emotion exists. Through the above method, N kinds of fine-grained emotions can be recognized, the richness of emotion categories can be improved, and the target emotion set can be obtained through the emotion-related probability map and the opposing emotion table to clarify the user's true emotions. Compared with directly outputting the emotion recognition results, the accuracy of emotion recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] in:

[0056] Figure 1 This is a flow chart of an emotion recognition method according to an embodiment of the present invention;

[0057] Figure 2 is another flow chart of an emotion recognition method according to an embodiment of the present invention;

[0058] Figure 3 This is a structural block diagram of an emotion recognition device according to an embodiment of the present invention;

[0059] Figure 4 4 is a structural block diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of an emotion recognition method according to an embodiment of the present invention. Figure 1 The method shown can be applied to a server or a terminal. The following description will be based on the application to the terminal. Figure 1 The method shown includes the following steps:

[0062] 101. Acquire voice data and text data of the user to be recognized;

[0063] The text data is obtained by performing speech recognition on the speech data.

[0064] It should be noted that the terminal includes but is not limited to a desktop terminal or a mobile terminal, and the mobile terminal can specifically be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server can be implemented as an independent server or a server cluster composed of multiple servers. When the method is applied to the server, the server establishes communication with the terminal through a network connection, wherein, when applied to the terminal, the terminal directly performs voice collection through the voice collection device on the terminal to obtain the voice data of the user to be identified, wherein the voice collection device includes but is not limited to a microphone. Further, speech recognition (Automatic Speech Recognition, ASR) is performed on the collected voice data, and the voice data is converted into text data, and the user's emotions are recognized through the two dimensions of voice data and text data. Among them, speech recognition can be obtained by converting voice data through ASR.

[0065] 102. Input the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the first emotion set of the user; and input the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the second emotion set of the user;

[0066] Furthermore, the speech data and text data to be recognized are input into an emotion recognition model. The speech data is input into the speech emotion recognition model, and the text data is input into the text emotion recognition model. Emotion recognition processing is performed simultaneously on both data to determine the possible presence of emotions in the speech and text. Specifically, the speech data is input into the speech emotion recognition model for binary classification of N fine-grained emotion labels, outputting a first set of user emotions, and the text data is input into the text emotion recognition model for binary classification of N fine-grained emotion labels, outputting a second set of user emotions. It should be noted that the terms "first" and "second" are merely used for differentiation and do not constitute specific limitations on technical features. The emotion set includes a correspondence between emotion labels and credibility. Emotion labels include, but are not limited to, joy, anger, sadness, happiness, etc. Confidence is the probability that such an emotion label exists in the speech data or text data. Fine-grainedness can reflect the degree of object segmentation. Applied to the above-mentioned emotion recognition model, this can be understood as subdividing the emotion labels in the emotion recognition model, thereby obtaining a more scientific and reasonable emotion recognition model, thereby outputting multiple emotion labels and improving the richness of user emotion recognition. For example, in this embodiment, 40 emotions can be classified, so N can be set to 40. Then, step 102 can be to perform binary classification processing on the 40 fine-grained emotion labels. The 40 fine-grained emotions are divided into 3 major categories, 11 medium categories, and 26 minor categories.

[0067] Furthermore, speech emotion recognition models include but are not limited to deep learning models (CNN), and text emotion recognition models include but are not limited to natural language processing (NLP) models. The emotion recognition model traverses each emotion label to determine the probability of each emotion label existing in the speech data and text emotion.

[0068] 103. Determine a third emotion set of the user based on the credibility of the first emotion set and the second emotion set, and a preset emotion probability threshold;

[0069] It should be noted that, through the above step 102, the emotion sets corresponding to the voice data and the text data can be obtained, and the credibility of each emotion label in the voice data and the text emotions can be obtained. Furthermore, the third emotion set can be determined by the credibility of the emotion label in the voice data or text data and the preset emotion probability threshold. The third emotion set can be understood as being obtained by screening the credibility of the first emotion set and the second emotion set through the preset emotion probability threshold. Furthermore, the third emotion set includes the first credibility emotion set and the second credibility emotion set, wherein the first credibility emotion set and the second credibility emotion set can be obtained by dividing them through the preset emotion probability threshold.

[0070] 104. Determine the target emotion set of the user by using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map, and the opposing emotion table;

[0071] After obtaining the above-mentioned third emotion set, the user's target emotion set can be further determined by the first credibility emotion set and the second credibility emotion set in the third emotion set to obtain the final emotion recognition result. Specifically, the user's target emotion set is determined using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map, and the opposing emotion table. Among them, the emotion correlation probability map is used to reflect the emotion correlation probability between N kinds of fine-grained emotion tags, the emotion correlation probability is used to reflect the probability that an emotion exists when another emotion exists, and the opposing emotion table is used to reflect the irrelevant emotions corresponding to each emotion tag in the N kinds of fine-grained emotion tags. Irrelevant emotions refer to emotions that must not exist when one emotion exists.

[0072] Exemplarily, the emotion correlation probability graph consists of each of N fine-grained emotion labels, its associated emotion, and its emotion correlation probability Y. For example, if the N fine-grained emotion labels include {emotion 1, emotion 2, emotion 3, emotion 4, emotion 5, ..., emotion 40}, then the emotion correlation probability graph includes the related emotions corresponding to emotion 1: emotion 2 (Y = 90%) and emotion 3 (Y = 80%); the related emotions corresponding to emotion 2: emotion 3 (Y = 65%) and emotion 40 (Y = 50%); the related emotions corresponding to emotion 3: emotion 5 (Y = 70%) and emotion 8 (Y = 65%), ... and the related emotions corresponding to emotion 40: emotion 29 (Y = 70%) and emotion 27 (Y = 65%). Furthermore, opposing emotions can be understood as sadness and happiness, relaxation and tension, and so on.

[0073] The present invention provides a method for identifying user emotions, the method comprising: obtaining voice data and text data of a user to be identified, wherein the text data is obtained by performing voice recognition on the voice data; inputting the voice data into a voice emotion recognition model for binary classification of N fine-grained emotion labels, and outputting a first emotion set of the user; and inputting the text data into a text emotion recognition model for binary classification of N fine-grained emotion labels, and outputting a second emotion set of the user, wherein the emotion set includes a correspondence between emotion labels and credibility; determining a third emotion set of the user based on the credibility in the first emotion set and the second emotion set, and a preset emotion probability threshold, wherein the third emotion set includes a first credibility emotion set and a second credibility emotion set; and determining a target emotion set of the user using the first credibility emotion set, the second credibility emotion set, an emotion correlation probability map, and an opposing emotion table, wherein the emotion correlation probability map is used to reflect the emotion correlation probability between the N fine-grained emotion labels, the emotion correlation probability is used to reflect the probability that one emotion also exists when another emotion exists, and the opposing emotion table is used to reflect irrelevant emotions corresponding to each emotion label in the N fine-grained emotion labels, wherein irrelevant emotions refer to emotions that are necessarily absent when one emotion exists. Through the above method, N kinds of fine-grained emotions can be recognized, the richness of emotion categories can be improved, and the target emotion set can be obtained through the emotion-related probability map and the opposing emotion table to clarify the user's true emotions. Compared with directly outputting the emotion recognition results, the accuracy of emotion recognition can be improved.

[0074] See also Figure 2 , Figure 2 is another flow chart of an emotion recognition method according to an embodiment of the present invention, as shown in FIG. Figure 2 The method shown includes the following steps:

[0075] 201. Acquire voice data and text data of a user to be recognized;

[0076] 202. Input the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the first emotion set of the user; and input the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the second emotion set of the user;

[0077] It should be noted that steps 201 and 202 are Figure 1 The contents of steps 101 and 102 are similar, and are not described here in detail to avoid repetition. Figure 1 The contents of steps 101 and 102 are shown.

[0078] In a feasible implementation, N may be 40, and step 202 includes steps A1-A3:

[0079] A1. Converting the speech data into a feature vector using the speech feature conversion algorithm of the speech emotion recognition model; inputting the feature vector into a CNN model of the speech emotion recognition model to extract speech features and determine target speech features;

[0080] It can be understood that after the voice data is input into the voice emotion recognition model, the voice feature conversion algorithm can be used to convert the voice data into a feature vector that can be collected, and then the convolutional neural network model (Convolutional Neural Network, CNN) can be used to extract voice features and determine the target voice features, where the target voice features include but are not limited to pitch, loudness, duration, timbre and other features that reflect voice emotions.

[0081] A2. Based on the BERT migration algorithm of the text emotion recognition model and the emotion data, extract text features from the text data to obtain target text features of the text data;

[0082] It is understood that after the text data is input into the text emotion recognition model, the BERT transfer algorithm and sentiment data can be used to extract features from the text data to obtain the target text features of the text data. The target text features include but are not limited to semantics, entities, and dependencies between entities, which can reflect the text emotion. Specifically, the target text features can be determined by using the BERT-based transfer algorithm on the text data and performing secondary training using sentiment data.

[0083] A3. Input the target speech features and the target text features into a fully connected layer with a sigmoid activation function to perform binary classification of 40 fine-grained emotion labels, and output the first emotion set and the second emotion set of the user.

[0084] It should be noted that after obtaining the target speech features and target text features, emotion recognition can be performed using the target speech features and target text features. Specifically, the target speech features and target text features are respectively input into the fully connected layer with the activation function of sigmoid to perform binary classification processing of 40 fine-grained emotion labels, and output the user's first emotion set and second emotion set.

[0085] 203. Determine a third emotion set of the user based on the credibility of the first emotion set and the second emotion set, and a preset emotion probability threshold;

[0086] It should be noted that step 203 and Figure 1 The contents of step 103 are similar, and will not be described here to avoid repetition. For details, please refer to the aforementioned Figure 1 The contents of step 103 are shown.

[0087] In one feasible implementation, the first credibility set is a set of emotion labels of speech data and text data with higher credibility, and the second credibility set is a set of emotion labels of speech data and text data with lower credibility. Therefore, the credibility in the first credibility emotion set is greater than the credibility in the second credibility emotion set. Step 203 may include the following steps B1-B3 to determine the first credibility emotion set, and C1-C3 to determine the second credibility emotion set. Specifically:

[0088] B1. Determine the first emotion labels in the first emotion set whose credibility is greater than or equal to the first emotion probability threshold as a first high-probability emotion subset;

[0089] B2. Determine the second emotion labels in the second emotion set whose credibility probability is greater than or equal to the second emotion probability threshold as a second high-probability emotion subset;

[0090] B3. Determine the first credibility emotion set based on the first high-probability emotion subset and the second high-probability emotion subset;

[0091] The first emotion set and the second emotion set are screened by setting a first emotion probability threshold and a second emotion probability threshold to obtain a first high-probability emotion subset of the first emotion set and a second high-probability emotion subset of the second emotion set. The first high-probability emotion subset and the second high-probability emotion subset constitute the first credibility emotion set. For example, the first emotion probability threshold and the second emotion probability threshold can be the same or different and can be set according to actual needs, which is not limited here.

[0092] C1. Determine the first emotion labels in the first emotion set whose credibility probability is greater than or equal to the third emotion probability threshold and less than or equal to the first emotion probability threshold as a first low-probability emotion subset;

[0093] Wherein, the first emotion probability threshold is greater than the third emotion probability threshold;

[0094] C2. Determine the second emotion labels in the second emotion set whose credibility probability is greater than or equal to the fourth emotion probability threshold and less than or equal to the second emotion probability threshold as a second low-probability emotion subset;

[0095] Wherein, the second emotion probability threshold is greater than the fourth emotion probability threshold;

[0096] C3. Obtain the second credibility emotion set by using the first low-probability emotion subset and the second low-probability emotion subset.

[0097] Among them, by setting the third emotion probability threshold and the fourth emotion probability threshold, the first emotion set and the first emotion set are screened again to obtain the first low-probability emotion subset of the first emotion set and the second low-probability emotion subset of the second emotion set. The first low-probability emotion subset and the second low-probability emotion subset are the second credibility emotion set. Exemplarily, the third emotion probability threshold and the fourth emotion probability threshold can be the same or different and can be set according to actual needs, and are not limited here. However, the first emotion probability threshold is greater than the third emotion probability threshold, and the second emotion probability threshold is greater than the fourth emotion probability threshold.

[0098] For example, the speech data is converted into a collectible feature vector using the LogFBank algorithm, and the CNN model is used for feature extraction. Using a fully connected layer with a sigmoid activation function, the speech data is classified into 40 fine-grained emotion labels. Two thresholds A and B (where A>B) are set, and the generated label list ListA (the first high-probability emotion subset) with a probability greater than A is output. The generated label list ListB (the first low-probability emotion subset) with a probability between A and B is output. Labels with a probability less than B are discarded.

[0099] The text data is trained using a BERT-based transfer algorithm and secondary training with sentiment data, followed by a fully connected layer using a sigmoid activation function. The text data is then binary-classified into 40 fine-grained sentiment labels. Two thresholds, C and D (where C > D), are set. The generated labels with a probability greater than C (ListC) (the second-highest probability subset) are output, while the generated labels with a probability between C and D (ListD) (the second-lowest probability subset) are output. Labels with a probability less than D are discarded.

[0100] In a feasible implementation, after obtaining two emotion sets through probability, the emotion labels in the two sets can be selected and rejected respectively to further clarify the user's true emotions. The first credibility emotion set, the second credibility emotion set, the emotion-related probability map and the opposing emotion table are used to determine the user's target emotion set, which can include steps 204-2010 to screen the emotion labels in the first credibility emotion set, that is, the high-probability emotion set. For details, please refer to the following instructions.

[0101] 204. Determine whether there is an opposing emotion in the first credibility emotion set using the opposing emotion table;

[0102] The opposing emotion table can be provided by professionals such as psychological counselors. The table provides opposing relationships for 40 fine-grained emotions, such as happiness -> sadness, pain, and irritability. This indicates that happiness cannot occur simultaneously with the following emotions (sadness, pain, and irritability). Furthermore, the opposing emotion table is used to determine whether an opposing emotion exists in the first credibility emotion set. If an opposing emotion exists, step 205 is executed; if not, step 209 is executed.

[0103] 205. When opposing emotions exist in the first credibility emotion set, determining a credibility difference between the opposing emotions;

[0104] Furthermore, when opposing emotions exist in the first credibility emotion set, the trade-off between the opposing emotions can be determined based on the credibility difference between them. If the credibility difference is greater than or equal to a preset credibility difference threshold, step 206 is executed; if the credibility difference is less than the preset credibility difference threshold, step 207 is executed. It will be appreciated that there can be one or more opposing emotions, and different opposing emotion detection results can be obtained based on different emotion recognition results.

[0105] 206. If there is a first opposing emotion whose credibility difference is greater than or equal to a preset credibility difference threshold, delete the emotion label with low credibility probability in the first opposing emotion from the first credibility emotion set, and update the first credibility emotion set;

[0106] Among them, the first credibility emotion set is updated in different ways through the relationship between the preset credibility difference threshold and the credibility difference, such as performing different processing logic through the results of size comparison. Among them, if there is a first opposing emotion in the opposing emotions whose credibility difference is greater than or equal to the preset credibility difference threshold, then the emotion label with low credibility probability in the first opposing emotion can be deleted from the first credibility emotion set, and the first credibility emotion set is updated, retaining the emotion label with high credibility probability in the first opposing emotion. It can be understood that the high credibility probability and the low credibility probability are determined based on the size relationship of the credibility of the emotion labels in the opposing emotions.

[0107] 207. If there is a second opposing emotion whose credibility difference is less than a preset credibility difference threshold, determine whether the second opposing emotion has a related emotion of the first highest credibility emotion label by using the emotion-related probability map and the first highest credibility emotion label in the first credibility emotion set;

[0108] Furthermore, if there is a second opposing emotion in the opposing emotions whose credibility difference is less than a preset credibility difference threshold, then the first credibility emotion set can be updated through another processing logic. Therefore, if there is a second opposing emotion with a credibility difference less than the preset credibility difference threshold, the emotion-related probability map and the first highest credibility emotion label in the first credibility emotion set are used to determine whether there is a related emotion of the first highest credibility emotion label in the opposing emotions, and the second opposing emotion is selected and discarded based on the first highest credibility emotion label, which has the highest credibility among all labels in the first credibility emotion set.

[0109] 208. If any emotion label in the second opposing emotions is a related emotion of the first highest credibility emotion label, then delete the emotion labels in the second opposing emotions except the related emotions from the first credibility emotion set, and update the first credibility emotion set. If no emotion label in the second opposing emotions has a related emotion of the first highest credibility emotion label, then delete the emotion labels in the second opposing emotions except the second highest credibility emotion label from the first credibility emotion set, and update the first credibility emotion set.

[0110] Specifically, if any emotion in the second opposing emotions has an emotion label that is a related emotion of the first highest credibility emotion label, then the emotion labels in the second opposing emotions except the related emotions are deleted from the first credibility emotion set, and the first credibility emotion set is updated, that is, the related emotions are retained and the irrelevant emotions are deleted.

[0111] For example, if the second opposing emotion includes the opposing emotions of emotion label 1 and emotion label 2, and the emotion of emotion label 2 is the related emotion of the first highest credibility emotion label, then emotion label 1 is deleted from the first credibility emotion set, and emotion label 2 is retained. Conversely, if the emotion of emotion label 1 is the related emotion of the first highest credibility emotion label, then emotion label 2 is deleted from the first credibility emotion set, and emotion label 1 is retained.

[0112] Furthermore, if the emotion tag in the second opposing emotion does not contain any related emotions for the emotion tag with the first highest credibility, that is, the emotion tags in the second opposing emotion are unrelated to the emotion tag with the first highest credibility, then the emotion tags in the second opposing emotion other than the emotion tag with the second highest credibility are deleted from the first credibility emotion set, and the first credibility emotion set is updated, wherein the emotion tag with the second highest credibility is the emotion tag corresponding to the highest credibility in the second opposing emotion. It is understandable that although the second opposing emotion does not contain any related emotions for the emotion tag with the first highest credibility, one reason is that the emotion correlation probability map in this embodiment only retains relatively relevant emotions, and therefore it may also be the user's true emotion. In order to improve recognition accuracy, it is necessary to retain the emotion tag of the emotion tag with the second highest credibility in the second opposing emotion to ensure the accuracy of the target emotion set.

[0113] For example, the second opposing emotion includes the opposing emotions of emotion label 1 and emotion label 2, where neither emotion label 1 nor emotion label 2 is related to the first highest credibility emotion label, and the credibility of emotion label 1 is less than the credibility of emotion label 2. In this case, emotion label 1 is deleted from the first credibility emotion set, and emotion label 2 is retained. Conversely, if the credibility of emotion label 1 is greater than the credibility of emotion label 2, then emotion label 2 is deleted from the first credibility emotion set, and emotion label 1 is retained.

[0114] 209. When there is no opposing emotion in the first credibility emotion set, delete the emotion tags other than the emotion tag with the highest credibility in the first credibility emotion set, and update the first credibility emotion set;

[0115] It should be noted that the emotions described as not having opposing emotions are not opposing and the emotions are relatively close. Therefore, only the emotion label with the highest credibility may be retained to update the first credibility emotion set.

[0116] 2010. Determine a target emotion set by using the second credibility emotion set, the updated first credibility emotion set, and the emotion-related probability map.

[0117] It should be noted that, in this embodiment, the updated first credibility emotion set is obtained and the second credibility emotion set can be further updated, that is, the first credibility emotion set is screened. Specifically, the second credibility emotion set can be screened by the updated first credibility emotion set and the emotion correlation probability graph to determine the user's final emotion set. When generating fine-grained labels before, the algorithm considers each label as an independent label. But in fact, fine-grained emotions are related. The emergence of an emotion will also affect the probability of the emergence of other emotions. Through the N-gram method, a fine-grained emotion correlation probability map is established from the training data set. (For example, the 2-gram method is used to represent the two emotions [sadness, self-blame], that is, the probability of self-blame appearing when sadness appears is calculated; the probability of sadness appearing when self-blame appears).

[0118] In a feasible implementation, before step 2010, it is necessary to pre-set an emotion-related probability map. That is, before step 2010, the following steps are further included:

[0119] D1. Obtain a training sample data set, wherein the training sample data set includes a plurality of emotion sample data for training;

[0120] In this embodiment, when training a speech emotion recognition model and a text emotion recognition model, a large amount of emotion sample data for training is used, and a plurality of emotion sample data constitute a training sample data set. Each emotion sample data corresponds to an emotion label set, and each emotion label set includes multiple emotion labels.

[0121] D2. Counting the target emotion tag set including the nth emotion tag and the number of the first set of the target emotion tag set;

[0122] It should be noted that n belongs to N fine-grained emotion tags, and n ranges from 1 to N. The value range of N can be determined according to the total number of emotion tags. If the granularity is 40, the maximum value of N is 40 and the minimum value is 1.

[0123] Furthermore, a target emotion tag set including the nth emotion tag is counted, that is, the target emotion tag set including the nth emotion tag in the emotion tag set of the emotion sample data is determined, and a first set number of the target emotion tag set is determined.

[0124] D3. For each third emotion label, count the number of second sets including the third emotion label in all target emotion label sets;

[0125] Furthermore, the number of occurrences of each emotion tag except the nth emotion tag in the target emotion tag set is counted, that is, in the target emotion tag set, the third emotion tag is any fine-grained emotion tag except the nth emotion tag.

[0126] D4. Using the ratio of the number of the first set to the number of the second set as the emotion correlation probability between the emotion of the third emotion label and the nth emotion label, obtain the emotion correlation probability between each third emotion label and the nth emotion label;

[0127] Finally, the ratio between the number of the first set (the total number of sets in which the nth emotion label appears) and the number of the second set (the total number of the nth emotion label and any emotion label other than the nth emotion label) is used as the ratio of the nth emotion label to other emotion labels, and the emotion correlation probability of each third emotion label and the nth emotion label is obtained.

[0128] D5. Taking the third emotion tag whose emotion-related probability is greater than a preset related probability threshold as the related emotion of the nth emotion tag;

[0129] Furthermore, the related emotions of each emotion can be screened by using the emotion related probability, and the third emotion tag with a probability greater than a preset related probability threshold is used as the related emotion of the nth emotion tag.

[0130] D6. Generate the emotion-related probability graph using the related emotions of each fine-grained emotion tag and the emotion-related probabilities of the related emotions.

[0131] It can be understood that the final emotion-related probability map is composed of all emotions and related emotions as well as emotions and related emotion probabilities.

[0132] For example, 10,000 labeled sample data are prepared. The sample data consists of speech, text, and labeled emotion tags. The speech data can be a wav file. Each text data and tag is represented as follows:

[0133] 000001.wav - I had an argument with my boyfriend a few days ago and cursed him out. What I said was really harsh and a bit personal. --[Regret, self-blame, worry]

[0134] 000002.wav--The roller coaster is really fun. I want to go there again next time. --[happy, excited, thrilled]

[0135] 000003.wav--I shouldn't have been so careless and broke Xiao Ming's pencil. --[self-blame, guilt, sadness]

[0136] The probability graph is generated by counting the occurrence probability relationship between each fine-grained label and other labels in 10,000 data points:

[0137] Regret: Self-blame 60%, worry 45%. ——> Indicates the probability of self-blame and worry appearing when regret occurs;

[0138] Happy: Excited 70%, Thrilled 50%. ——> Indicates the probability of excitement and thrill appearing when happy appears;

[0139] Self-blame: Guilt 90%, Sadness 60%. ——> Indicates the probability of guilt and sadness occurring when self-blame occurs. This statistical analysis was performed on 40 fine-grained emotions, yielding the two most correlated emotions for each emotion.

[0140] It is understandable that the final emotion with the highest emotion correlation can be determined by a preset correlation probability threshold, or the emotions ranked in the top N, for example, the top 2, can be obtained by arranging the related emotion probabilities from high to low.

[0141] The following is an example of steps 204-2010: 1.1 Set A represents the second emotion label set generated by the text data, and a0 to a39 represent 40 emotions. The output of the text emotion recognition model is:

[0142] A={a0-->91%, a1-->65%, a2-->53%, a3-->42%, a4-->75%....a 39 -->0%};

[0143] 1.2 Take the emotions with a probability greater than 50%, then A = {a0-->91%, a4-->75%, a2-->53%}.

[0144] 2.1 Use B set to represent the first emotion label set generated by speech data, a0 to a 39 Representing 40 emotions, the output of the speech emotion recognition model is:

[0145] B={a0-->20%, a1-->71%, a2-->62%, a3-->71%, a4-->35%....a 39 -->0%};

[0146] 2.2 Take the emotions with a probability greater than 50%, then B = {a3-->71%, a1-->71%, a2-->62%}.

[0147] 3. Let C be the final output probability label of the fusion model, which is initially empty.

[0148] 4. Take the probability greater than 70% (the first credibility emotion set) and add it to C: C = {a0, a1, a3, a4}.

[0149] 5. According to the opposing emotions table, there are different ways to judge whether a0, a1, a3, and a4 have opposing emotions:

[0150] 5.1 First, sort the emotions in C according to their credibility: C = {a0-->91%, a4-->75%, a1-->71%, a3-->71%}.

[0151] 6. Opposition detection: first check a0, and then determine whether a1, a2, and a3 are its opposing emotions.

[0152] 7. Assume that a3 is the opposite of a0. Since the probability difference is greater than 15%, a3 is discarded directly. In this case, C = {a0, a4, a1}.

[0153] 8. Check a4 again to see if a1 is its opposite emotion. If so, since the probability difference is relatively small, determine which of a4 and a1 is the correlated emotion of a0 in the emotion correlation probability graph. If a1 is correlated and a4 is not, discard a4. C = {a0, a1}. If both a1 and a4 are not correlated with a0, select a4, which has a higher probability, and C = {a0, a4}.

[0154] Finally, the target emotion set is determined using the second credibility emotion set, the updated first credibility emotion set, and the emotion-related probability map.

[0155] In a feasible implementation, after obtaining the updated first credibility emotion set, the second credibility emotion set can be further updated, which can specifically include steps E1-E5 to filter the emotion labels in the second credibility emotion set, i.e., the low-probability emotion set. For details, please refer to the following description.

[0156] That is, step 2010 may include E1-E2:

[0157] E1. Determine, using the emotion correlation probability graph, whether the first emotion label in the second credibility emotion set and the second emotion label in the updated first credibility emotion set are related emotions;

[0158] It should be noted that the updated first credibility emotion set can be used to update the second credibility emotion set, for example, by determining whether there are related emotions in the two sets. Specifically, the emotion correlation probability graph is used to determine whether the first emotion label in the second credibility emotion set and the second emotion label in the updated first credibility emotion set are related emotions. The first emotion label is any emotion label in the second credibility emotion set, and the second emotion label is any emotion label in the updated first credibility emotion set. The emotion correlation probability graph is used to determine whether it is a related emotion. If the first emotion label and the second emotion label are related emotions, step E2 is executed; if the first emotion label and the second emotion label are unrelated emotions, step E5 is executed.

[0159] E2. When the first emotion label and the second emotion label are related emotions, searching the emotion correlation probability map for the first emotion label corresponding to the second emotion label;

[0160] It should be noted that if the first emotion label and the second emotion label are related emotions, the emotion correlation probability graph can be used to obtain the related emotions and their corresponding emotion correlation probabilities corresponding to the second emotion label in the updated first credibility emotion set. The emotion correlation probability graph can then be used to find the emotion correlation probability of the first emotion label corresponding to the second emotion label. For example, if regret (the second emotion label) has self-blame 60% and worry 45%, and the first emotion label is self-blame, then the emotion correlation probability of the first emotion label corresponding to the second emotion label is 60%.

[0161] E3. If the product of the emotion correlation probability and the credibility of the second emotion label in the updated first credibility emotion set is greater than or equal to a preset correlation probability threshold, retaining the first emotion label in the second credibility emotion set;

[0162] Furthermore, after obtaining the emotion correlation probability of the first emotion label corresponding to the second emotion label, the correlation between the two emotion labels in this emotion recognition can be further determined. The correlation can be determined by multiplying the credibility and the emotion correlation probability. The correlation is proportional to the result of the product. The larger the product, the higher the correlation. Specifically, the credibility of the second emotion label identified by the emotion recognition model is multiplied by the emotion correlation probability of the statistical first emotion label. If the product is greater than or equal to the preset correlation probability threshold, it indicates that the correlation exceeds the threshold range, and the first emotion label in the second credibility emotion set is a true related emotion, so the first emotion label in the second credibility emotion set is retained. On the contrary, if the product between the emotion correlation probability and the credibility of the second emotion label in the updated first credibility emotion set is less than the preset correlation probability threshold, step E4 is executed.

[0163] E4. If the product of the emotion correlation probability and the credibility of the second emotion label in the updated first credibility emotion set is less than a preset correlation probability threshold, or when the first emotion label and the second emotion label are unrelated emotions, deleting the first emotion label from the second credibility emotion set and updating the second credibility emotion set;

[0164] E5. Taking the union of the updated second credibility emotion set and the updated first credibility emotion set as the target emotion set.

[0165] Among them, if the product of the emotion correlation probability and the credibility of the second emotion label in the updated first credibility emotion set is less than the preset correlation probability threshold, it means that the two are not really correlated in this emotion recognition, so the first emotion label is deleted from the second credibility emotion set and the second credibility emotion set is updated. It can be understood that the relevant emotions of the second emotion label in the emotion correlation probability graph do not have the first and second credibility emotion sets, that is, the first emotion label and the second emotion label are unrelated emotions, so the first emotion label is deleted from the second credibility emotion set and the second credibility emotion set is updated. Finally, the updated second credibility emotion set and the updated first credibility emotion set are taken as the union to obtain the final target emotion set.

[0166] Continuing with the above example, let's assume the updated first credibility emotion set is C = {a0, a4}. We take emotion a2, whose probability is greater than 50% and less than or equal to 70%, and determine whether a0 and a4 are related emotions to a2 by multiplying the probabilities. If the product of a0's credibility and a2's emotion-related probability is greater than 25%, a2 is considered to be the emotion of the statement. In this case, C = {a0, a2, a4}.

[0167] The present invention provides a method for identifying user emotions, which includes: obtaining voice data and text data of a user to be identified, where the text data is obtained by performing voice recognition on the voice data; inputting the voice data and text data into a voice emotion recognition model and a text emotion recognition model respectively to perform binary classification processing of N fine-grained emotion labels, and outputting a first emotion set and a second emotion set of the user, where the emotion set includes a correspondence between the emotion label and the credibility; determining a third emotion set including the first credibility emotion set and the second credibility emotion set based on the credibility in the first emotion set and the second emotion set, and a preset emotion probability threshold; wherein the first credibility emotion set is a high-credibility emotion set and the second credibility emotion set is a low-credibility emotion set, and by first screening the emotions at the high-credibility emotion set level, using the updated first credibility emotion set to screen the second credibility emotion set, and using an emotion-related probability map and an opposing emotion table as a judgment reference, the user's target emotion set is determined, so that when performing N fine-grained emotion recognition, a relatively accurate emotion recognition result can also be obtained, so that the determination of the target emotion set is more accurate. Through the above method, not only can the recognition of N kinds of fine-grained emotions be achieved and the richness of emotion categories be improved, but the first credibility emotion set can be updated through the emotion-related probability map and the opposing emotion table, and the second credibility emotion set can be updated using the updated first credibility emotion set to obtain the final target emotion set, clarify the user's true emotions, and effectively improve the accuracy of emotion recognition.

[0168] See also Figure 3 , Figure 3 FIG. 1 is a structural block diagram of an emotion recognition device according to an embodiment of the present invention. Figure 3 The apparatus shown comprises:

[0169] Data acquisition module 301: used to acquire voice data and text data of a user to be recognized, wherein the text data is obtained by performing voice recognition on the voice data;

[0170] Emotion recognition module 302: used to input the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the first emotion set of the user; and input the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the second emotion set of the user, wherein the emotion set includes a correspondence between emotion labels and credibility;

[0171] Emotion screening module 303: used to determine a third emotion set of the user based on the credibility of the first emotion set and the second emotion set, and a preset emotion probability threshold, wherein the third emotion set includes the first credibility emotion set and the second credibility emotion set;

[0172] Emotion determination module 304: used to determine the target emotion set of the user by using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map and the opposing emotion table, the emotion correlation probability map is used to reflect the emotion correlation probability between N fine-grained emotion tags, the emotion correlation probability is used to reflect the probability that one emotion also exists when another emotion exists, and the opposing emotion table is used to reflect the irrelevant emotions corresponding to each emotion tag in the N fine-grained emotion tags, and the irrelevant emotions refer to emotions that must not exist when one emotion exists.

[0173] The present invention provides a device for identifying user emotions, which includes: a data acquisition module: used to acquire voice data and text data of a user to be identified, where the text data is obtained by performing voice recognition on the voice data; an emotion recognition module: used to input the voice data into a voice emotion recognition model for binary classification of N fine-grained emotion labels, and output a first emotion set of the user; and input the text data into a text emotion recognition model for binary classification of N fine-grained emotion labels, and output a second emotion set of the user, where the emotion set includes a correspondence between emotion labels and credibility; an emotion screening module: used to classify the first emotion set and the second emotion set based on the credibility of the first emotion set and the second emotion set, and A preset emotion probability threshold is used to determine the user's third emotion set, which includes a first credibility emotion set and a second credibility emotion set; an emotion determination module is used to determine the user's target emotion set using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map, and the opposing emotion table. The emotion correlation probability map is used to reflect the emotion correlation probability between N fine-grained emotion tags. The emotion correlation probability is used to reflect the probability that one emotion also exists when another emotion exists. The opposing emotion table is used to reflect the irrelevant emotions corresponding to each emotion tag in the N fine-grained emotion tags. An irrelevant emotion refers to an emotion that must not exist when one emotion exists. Through the above method, the recognition of N fine-grained emotions can be achieved, the richness of emotion categories can be improved, and the target emotion set can be obtained through the emotion correlation probability map and the opposing emotion table to clarify the user's true emotions. Compared with directly outputting the emotion recognition results, the accuracy of emotion recognition can be improved.

[0174] Figure 4 FIG1 shows an internal structure diagram of a computer device in an embodiment. The computer device can be a terminal or a server. Figure 4As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. It will be understood by those skilled in the art that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0175] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following Figure 1 or Figure 2 Steps of the method shown.

[0176] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor is caused to execute the following Figure 1 or Figure 2 Steps of the method shown.

[0177] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0178] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for identifying emotions, characterized in that: The method comprises: Acquire voice data and text data of a user to be recognized, wherein the text data is obtained by performing voice recognition on the voice data; Inputting the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and outputting a first emotion set of the user; and inputting the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and outputting a second emotion set of the user, wherein the emotion set includes a correspondence between the emotion labels and the credibility; Determining a third emotion set of the user based on the credibility of the first emotion set and the second emotion set, and a preset emotion probability threshold, wherein the third emotion set includes the first credibility emotion set and the second credibility emotion set; The target emotion set of the user is determined using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map and the opposing emotion table. The emotion correlation probability map is used to reflect the emotion correlation probability between N fine-grained emotion tags. The emotion correlation probability is used to reflect the probability that one emotion also exists when another emotion exists. The opposing emotion table is used to reflect the irrelevant emotions corresponding to each emotion tag in the N fine-grained emotion tags. The irrelevant emotions refer to emotions that must not exist when one emotion exists.

2. The method according to claim 1, characterized in that If the credibility of the first credibility emotion set is greater than the credibility of the second credibility emotion set, determining the third emotion set of the user according to the credibility of the first emotion set and the second emotion set and a preset emotion probability threshold includes: Determine the first emotion label in the first emotion set whose credibility is greater than the first emotion probability threshold as a first high-probability emotion subset; Determining the second emotion labels in the second emotion set whose credibility probability is greater than the second emotion probability threshold as a second high probability emotion subset; The first credibility emotion set is determined according to the first high-probability emotion subset and the second high-probability emotion subset.

3. The method according to claim 2, characterized in that The method further comprises: Determine the first emotion labels in the first emotion set whose credibility probability is greater than or equal to the third emotion probability threshold and less than or equal to the first emotion probability threshold as a first low-probability emotion subset, and the first emotion probability threshold is greater than the third emotion probability threshold; Determine the second emotion labels in the second emotion set whose credibility probability is greater than or equal to the fourth emotion probability threshold and less than or equal to the second emotion probability threshold as a second low-probability emotion subset, and the second emotion probability threshold is greater than the fourth emotion probability threshold; The second credibility emotion set is obtained by using the first low-probability emotion subset and the second low-probability emotion subset.

4. The method according to claim 1, wherein The determining the target emotion set of the user by using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map, and the opposing emotion table includes: determining whether there are opposing emotions in the first credibility emotion set using the opposing emotion table; and determining the credibility difference between the opposing emotions when there are opposing emotions in the first credibility emotion set; If there is a first opposing emotion whose credibility difference is greater than or equal to a preset credibility difference threshold, deleting the emotion label with low credibility probability in the first opposing emotion from the first credibility emotion set, and updating the first credibility emotion set; If there is a second opposing emotion whose credibility difference is less than a preset credibility difference threshold, determining whether there is a related emotion of the first highest credibility emotion label in the second opposing emotion by using the emotion correlation probability map and the first highest credibility emotion label in the first credibility emotion set; If any emotion label in the second opposing emotions is a related emotion of the first highest credibility emotion label, then the emotion labels in the second opposing emotions other than the related emotions are deleted from the first credibility emotion set, and the first credibility emotion set is updated; if any emotion label in the second opposing emotions does not have a related emotion of the first highest credibility emotion label, then the emotion labels in the second opposing emotions other than the second highest credibility emotion label are deleted from the first credibility emotion set, and the first credibility emotion set is updated; When there is no opposing emotion in the first credibility emotion set, deleting the emotion tags other than the emotion tag with the highest credibility in the first credibility emotion set, and updating the first credibility emotion set; A target emotion set is determined using the second credibility emotion set, the updated first credibility emotion set, and the emotion-related probability map.

5. The method according to claim 4, characterized in that The determining of the target emotion set by using the second credibility emotion set, the updated first credibility emotion set, and the emotion-related probability map includes: Determining, using the emotion correlation probability graph, whether the first emotion label in the second credibility emotion set and the second emotion label in the updated first credibility emotion set are related emotions; When the first emotion label and the second emotion label are related emotions, searching the emotion correlation probability map for the emotion correlation probability of the first emotion label corresponding to the second emotion label; If the product of the emotion correlation probability and the credibility of the second emotion label in the updated first credibility emotion set is greater than or equal to a preset correlation probability threshold, retaining the first emotion label in the second credibility emotion set; If the product of the emotion correlation probability and the credibility of the second emotion label in the updated first credibility emotion set is less than a preset correlation probability threshold, or when the first emotion label and the second emotion label are unrelated emotions, deleting the first emotion label from the second credibility emotion set and updating the second credibility emotion set; The union of the updated second credibility emotion set and the updated first credibility emotion set is used as the target emotion set.

6. The method according to claim 1, wherein The method of determining the target emotion set of the user by using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map, and the opposing emotion table may also include: Acquire a training sample data set, wherein the training sample data set includes a plurality of emotion sample data for training, each of the emotion sample data corresponds to an emotion label set, and each emotion label set includes multiple emotion labels; Counting a target emotion tag set including an nth emotion tag and a first set number of the target emotion tag set, where n belongs to N fine-grained emotion tags and the value of n ranges from 1 to N; For each third emotion label, counting the number of second sets of all target emotion label sets that include the third emotion label, where the third emotion label is any fine-grained emotion label other than the nth emotion label; Using the ratio of the number of the first set to the number of the second set as the emotion correlation probability between the emotion of the third emotion label and the nth emotion label, obtaining the emotion correlation probability between each third emotion label and the nth emotion label; The third emotion tag whose emotion relevance probability is greater than a preset relevance probability threshold is used as the relevant emotion of the nth emotion tag; and the emotion relevance probability graph is generated using the relevant emotions of each fine-grained emotion tag and the emotion relevance probability of the relevant emotions.

7. The method according to claim 1, characterized in that If N is 40, the speech data is input into the speech emotion recognition model for binary classification of N fine-grained emotion labels, and the first emotion set of the user is output. The text data is input into a text emotion recognition model for binary classification of N fine-grained emotion labels, and a second emotion set of the user is output, including: The speech data is converted into a feature vector using the speech feature conversion algorithm of the speech emotion recognition model; the feature vector is input into the CNN model of the speech emotion recognition model to extract speech features and determine target speech features; Based on the BERT migration algorithm of the text emotion recognition model and the emotion data, text feature extraction is performed on the text data to obtain target text features of the text data; The target speech features and the target text features are respectively input into a fully connected layer with a sigmoid activation function to perform binary classification processing of 40 fine-grained emotion labels, and the first emotion set and the second emotion set of the user are output.

8. An emotion recognition device, characterized in that: The device comprises: Data acquisition module: used to acquire the voice data and text data of the user to be recognized, wherein the text data is obtained by performing voice recognition on the voice data; Emotion recognition module: used to input the voice data into a voice emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the first emotion set of the user; and input the text data into a text emotion recognition model to perform binary classification processing of N fine-grained emotion labels, and output the second emotion set of the user, wherein the emotion set includes the correspondence between emotion labels and credibility; An emotion screening module is configured to determine a third emotion set of the user based on the credibility of the first emotion set and the second emotion set, and a preset emotion probability threshold, wherein the third emotion set includes the first credibility emotion set and the second credibility emotion set; Emotion determination module: used to determine the target emotion set of the user by using the first credibility emotion set, the second credibility emotion set, the emotion correlation probability map and the opposing emotion table, the emotion correlation probability map is used to reflect the emotion correlation probability between N fine-grained emotion tags, the emotion correlation probability is used to reflect the probability that one emotion also exists when another emotion exists, and the opposing emotion table is used to reflect the irrelevant emotions corresponding to each emotion tag in the N fine-grained emotion tags, and the irrelevant emotions refer to emotions that must not exist when one emotion exists.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images