Voice user emotion recognition method based on AI intelligent neural network

Abstract

The present invention provides a method for identifying emotions of speech users based on an AI intelligent neural network, comprising: obtaining speech feature information of each short sentence in a to-be-identified conversation content; processing the speech feature information of each short sentence based on a first long short-term memory network model to obtain contextual semantic features of each short sentence; inputting the speech feature information of a plurality of short sentences into a second long short-term memory network model to extract speech state features of a sender of the short sentences; determining a switch state of the sender of each short sentence based on the sender of the corresponding previous short sentence; and determining an emotion recognition result of each short sentence according to the contextual speech features of each short sentence, the speech state features of the sender of the short sentence, and the switch state of the sender.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to a method for voice user emotion recognition based on AI intelligent neural network. Background Art

[0002] Speech emotion recognition refers to extracting the speaker's emotional state from the speaker's speech, that is, judging the emotions of humans when they speak. What people usually refer to as speech recognition is automatic speech recognition, which refers to mapping human speech to a series of text, while speech emotion recognition maps human speech to emotional labels. The current common practice is to first extract audio features from human speech and input them into traditional machine learning algorithms or deep neural networks for classification, thereby identifying the emotions in speech.

[0003] In the prior art, most methods only consider the emotional information carried by the voice information itself, and do not perform in-depth recognition of the voice information for specific speaking scenarios. Based on this, the present invention provides a voice user emotion recognition method based on an AI intelligent neural network, which uses contextual emotion simulation of the speaking scene to determine the user's emotional results, which is beneficial to improving recognition accuracy. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method for voice user emotion recognition based on AI intelligent neural network to solve the above problems.

[0005] A method for recognizing speech user emotions based on an AI intelligent neural network, comprising: obtaining speech feature information of each short sentence in a conversation content to be recognized;

[0006] The speech feature information of each short sentence is processed based on the first long short-term memory network model to obtain the contextual semantic features of each short sentence;

[0007] Inputting speech feature information of multiple short sentences into the second long short-term memory network model to extract the speaking state features of the speaker of the short sentences;

[0008] Determining a switch state of a sender of each short sentence based on the sender of each short sentence and the sender of the corresponding previous short sentence;

[0009] The emotion recognition result of each short sentence is determined according to the context speech features of each short sentence, the speaking state features of the speaker of the short sentence and the switching state of the speaker.

[0010] As an embodiment of the present invention, the speech feature information includes voice feature information, speech rate feature information, text feature information, semantic feature information and intonation amplitude change feature information.

[0011] As an embodiment of the present invention, the speech feature information of each short sentence is processed based on the first long short-term memory network model to obtain the contextual semantic features of each short sentence, including:

[0012] Sort all the short sentences in the conversation content according to the order in which the conversation was sent, and generate a sorting result;

[0013] Acquire text feature information and semantic feature information of any short sentence in the conversation content to obtain integrated feature information, input the integrated feature information into the first long short-term memory network model, and output a first memory result;

[0014] For each short sentence, the first memory result of the previous short sentence that is adjacent to the current short sentence and the integrated feature information of the current short sentence are input into the first long short-term memory network model, and the contextual semantic features of the current short sentence are output.

[0015] As an embodiment of the present invention, the speech feature information of a plurality of short sentences is input into the second long short-term memory network model, and the speaking state features of the speaker of the short sentences are extracted, including:

[0016] Step 1, initializing the speaking state feature of the person who sends the short sentence, and obtaining the initial speaking state feature of the person who sends the short sentence;

[0017] Step 2, obtaining the voice feature information, speech rate feature information and intonation amplitude change feature information of the first short sentence sent by the short sentence sender, obtaining the second integrated feature information, inputting the second integrated feature information and the corresponding initial speaking state feature into the second long short-term memory network model, and outputting the second speaking state feature;

[0018] Step 3, obtaining the second integrated feature information of the second short sentence issued by the same short sentence issuer adjacent to the first short sentence, inputting the second integrated feature information of the second short sentence and the second speaking state feature into the second long short-term memory network model, and outputting the third speaking state feature;

[0019] Step 4, obtaining the second integrated feature information of the n+1th sentence issued by the same sentence issuer adjacent to the nth sentence, inputting the second integrated feature information of the n+1th sentence and the n+1th speaking state feature into the second long short-term memory network model, and outputting the n+2th speaking state feature, wherein n is an integer greater than 1;

[0020] Step 5: until n is equal to the total number of short sentences issued by the current short sentence issuer, extract the (n+1)th speaking state feature as the speaking state feature of the short sentence issuer.

[0021] As an embodiment of the present invention, determining the switch state of the sender of each short sentence based on the sender of each short sentence and the sender of the corresponding previous short sentence includes:

[0022] Integrate all short sentences, sort all short sentences in the conversation content according to the order in which the conversation was issued, and generate a sorting result;

[0023] For each short sentence, if the sender of the current short sentence is the same as the sender of the previous short sentence in the sorting result, the preset first switching state will be used as the sender switching state of the current short sentence; if the sender of the current short sentence is different from the sender of the previous short sentence in the sorting result, the preset second switching state will be used as the sender switching state of the current short sentence.

[0024] As an embodiment of the present invention, the emotion recognition result of each short sentence is determined according to the context speech features of each short sentence, the speaking state features of the speaker of the short sentence, and the switching state of the speaker, including:

[0025] For each short sentence, the contextual speech features of the short sentence, the speaking state features of the speaker of the short sentence, and the switching state of the speaker are connected to form a new vector;

[0026] Input the new vector to the fully connected layer, and output the probability of each emotion label category of the current short sentence based on the normalized exponential function to determine the probability distribution of the emotion label category of each short sentence in the dialogue content to be identified;

[0027] According to the probability distribution of the emotion label category of each short sentence, the emotion recognition result of each short sentence is determined.

[0028] As an embodiment of the present invention, an emotion recognition method based on user intonation also includes: if it is determined that the user's intonation feature information or the second intonation feature information is not negative intonation feature information, taking the first user emotion as the main emotion and combining the second user emotion as the secondary emotion, and determining the user's current emotion based on the second preset emotion combined with analysis logic.

[0029] As an embodiment of the present invention, the emotion recognition result of each short sentence is the probability of each emotion label category corresponding to the short sentence.

[0030] As an embodiment of the present invention, a method for recognizing speech user emotions based on an AI intelligent neural network also includes:

[0031] Obtain all short sentences currently uttered by the caller as short sentences to be predicted;

[0032] Obtaining the emotion recognition result of each short sentence to be predicted as the emotion data to be predicted;

[0033] Sorting all the emotion data to be predicted according to the generation time of the short sentences to be predicted to obtain a second sorting result;

[0034] Determine whether there are two adjacent emotion data to be predicted whose emotion fluctuation data is greater than a preset emotion fluctuation data range in the second sorting result, and if so, extract the short sentences to be predicted corresponding to the two adjacent emotion data to be predicted and record them as emotion fluctuation points;

[0035] Obtain all emotional fluctuation points and extract entity keywords in the short sentences to be predicted corresponding to all emotional fluctuation points;

[0036] Determine whether the similarity of entity keywords corresponding to adjacent emotional fluctuation points is greater than a preset similarity, and if so, determine that there is a correlation relationship between adjacent emotional fluctuation points;

[0037] Determine whether the emotion conversion relationship of adjacent emotion fluctuation points with an associated relationship is reversed, and if so, determine the entity keyword corresponding to the adjacent emotion fluctuation point as the target keyword;

[0038] When the emotion recognition result of the caller triggers the manual voice switch, all target keywords are integrated and combined with the caller's consultation items obtained in advance to predict the key information that the caller wants to consult;

[0039] Feedback the key information that the caller wants to consult to the manual customer service at the time of the manual voice switch.

[0040] As an embodiment of the present invention, determining whether the emotion conversion relationship of adjacent emotion fluctuation points having an associated relationship is opposite and generating a determination result includes:

[0041] Obtaining emotion conversion relationships of adjacent emotion fluctuation points that have an associated relationship, wherein the emotion conversion relationships include positive-negative emotion conversion relationships, negative-positive emotion conversion relationships, positive-positive emotion conversion relationships, and negative-negative emotion conversion relationships;

[0042] It is determined whether the emotion conversion relationship of the adjacent emotion fluctuation points with associated relationship includes both the positive-negative emotion conversion relationship and the negative-positive emotion conversion relationship. If so, it is determined that the emotion conversion relationship of the adjacent emotion fluctuation points with associated relationship is opposite, and a judgment result is generated.

[0043] As an embodiment of the present invention, all target keywords are integrated, combined with the pre-acquired consultation items of the caller, and the key information to be consulted by the caller is predicted, including:

[0044] Obtain all target keywords and perform data cleaning on all target keywords to obtain keywords to be predicted; wherein data cleaning includes removing repeated target keywords from all target keywords and assigning weights to corresponding target keywords according to the number of repetitions, where the more repeated times a target keyword has, the higher its weight;

[0045] Obtain the pre-obtained caller consultation items, extract the caller consultation items by item according to the matching degree between the keywords to be predicted and each sub-item in the caller consultation items, and predict the specific items consulted by the caller; among which, the specific items consulted by the caller have the highest matching degree with the keywords to be predicted;

[0046] According to the weight of each target keyword in the keywords to be predicted, the project information in the specific project consulted by the caller is classified into importance levels to obtain the importance level of each project information; wherein, the higher the total weight of the target keyword corresponding to each project information, the higher the importance level of the project information;

[0047] According to the importance level of project information, all project information in a specific project is sorted, and a project table is generated based on a preset table template, and the key information that the caller wants to consult is determined based on the project table.

[0048] The beneficial effects of the present invention are:

[0049] The present invention provides a method for voice user emotion recognition based on AI intelligent neural network, which adopts the method of context emotion simulation of speaking scene to determine the emotion result of the user, which is beneficial to improving the recognition accuracy.

[0050] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0051] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0053] Figure 1 This is a method flow chart of a method for voice user emotion recognition based on AI intelligent neural network in an embodiment of the present invention;

[0054] Figure 2 A flowchart for determining contextual semantic features of each short sentence in a method for identifying speech user emotions based on an AI intelligent neural network in an embodiment of the present invention;

[0055] Figure 3 The present invention is a flowchart for determining the speaking state characteristics of a short sentence sender in a method for voice user emotion recognition based on an AI intelligent neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0057] See also Figure 1 , a method for voice user emotion recognition based on AI intelligent neural network, comprising:

[0058] S101, obtaining voice feature information of each short sentence in the conversation content to be recognized;

[0059] S102, processing the speech feature information of each short sentence based on the first long short-term memory network model to obtain contextual semantic features of each short sentence;

[0060] S103, inputting speech feature information of a plurality of short sentences into a second long short-term memory network model, and extracting speech state features of the speaker of the short sentences;

[0061] S104, determining a switch state of a sender of each short sentence based on the sender of each short sentence and the sender of the corresponding previous short sentence;

[0062] S105, determining the emotion recognition result of each short sentence according to the context speech features of each short sentence, the speaking state features of the speaker of the short sentence, and the switching state of the speaker;

[0063] The working principle of the above technical solution is as follows: the present application provides a method for voice user emotion recognition based on AI intelligent neural network, which is preferably applied to customer service call scenarios; first, the voice feature information of each short sentence in the conversation content to be recognized is obtained; the voice feature information of each short sentence is processed based on the first long short-term memory network model to obtain the contextual semantic features of each short sentence; the voice feature information of multiple short sentences is input into the second long short-term memory network model to extract the speaking state features of the sender of the short sentence; the switching state of the sender of each short sentence is determined based on the sender of each short sentence and the sender of the corresponding previous short sentence; the emotion recognition result of each short sentence is determined according to the contextual voice features of each short sentence, the speaking state features of the sender of the short sentence and the switching state of the sender;

[0064] The beneficial effect of the above technical solution is: through the above technical solution, the emotional result of the user is determined by simulating the contextual emotions of the speaking scene, which is beneficial to improving the recognition accuracy.

[0065] In one embodiment, the speech feature information includes voice feature information, speech rate feature information, text feature information, semantic feature information, and intonation amplitude change feature information;

[0066] The working principle and beneficial effects of the above technical solution are as follows: voice feature information is used to participate in the analysis of user emotions. Some data show that when the user is excited, impatient, etc., the speech rate feature information will change from the initial user speech rate to a faster speech rate, and the tone feature information will also change. At the same time, different voice feature information can characterize the user's current emotions to a certain extent. For example, a sharp voice usually gives people a harsh and unpleasant feeling, which is suitable for expressing emotions such as dissatisfaction and anger; a pleasant voice usually gives people a pleasant and comfortable feeling, which is suitable for occasions such as singing and reciting that need to express beauty; a fuzzy voice usually gives people a vague and unclear feeling, which is suitable for expressing confusion and uncertainty; a sad voice usually gives people a sad and resentful feeling, which is suitable for expressing sadness, loss and other emotions; in addition, it also includes several different voice features, which will not be repeated here one by one; it can be seen from the above technical solution that by detailed definition of voice feature information, the accuracy of emotion recognition for voice data can be improved.

[0067] See also Figure 2 In one embodiment, the speech feature information of each short sentence is processed based on the first long short-term memory network model to obtain the contextual semantic features of each short sentence, including:

[0068] S201, sorting all the short sentences in the conversation content according to the order in which the conversations were sent, and generating a sorting result;

[0069] S202, obtaining text feature information and semantic feature information of any short sentence in the conversation content, obtaining integrated feature information, inputting the integrated feature information into a first long short-term memory network model, and outputting a first memory result;

[0070] S203, for each short sentence, inputting the first memory result of the previous short sentence that is adjacent to the current short sentence and the integrated feature information of the current short sentence into the first long short-term memory network model, and outputting the contextual semantic features of the current short sentence;

[0071] The working principle and beneficial effects of the above technical solution are as follows: sorting all the short sentences in the conversation content according to the order in which the conversation is issued to generate a sorting result; obtaining the text feature information and semantic feature information of any short sentence in the conversation content to obtain integrated feature information, inputting the integrated feature information into the first long short-term memory network model, and outputting a first memory result; for each short sentence, inputting the first memory result of the previous short sentence that is adjacent to the current short sentence in order and the integrated feature information of the current short sentence into the first long short-term memory network model, and outputting the contextual semantic features of the current short sentence. The above technical solution is beneficial to improving the accuracy of the contextual semantic features of the current short sentence and providing data support for subsequent emotion recognition.

[0072] See also Figure 3 In one embodiment, the speech feature information of multiple short sentences is input into the second long short-term memory network model, and the speaking state features of the speaker of the short sentences are extracted, including:

[0073] Step 1, initializing the speaking state feature of the person who sends the short sentence, and obtaining the initial speaking state feature of the person who sends the short sentence;

[0074] Step 2, obtaining the voice feature information, speech rate feature information and intonation amplitude change feature information of the first short sentence sent by the short sentence sender, obtaining the second integrated feature information, inputting the second integrated feature information and the corresponding initial speaking state feature into the second long short-term memory network model, and outputting the second speaking state feature;

[0075] Step 3, obtaining the second integrated feature information of the second short sentence issued by the same short sentence issuer adjacent to the first short sentence, inputting the second integrated feature information of the second short sentence and the second speaking state feature into the second long short-term memory network model, and outputting the third speaking state feature;

[0076] Step 4, obtaining the second integrated feature information of the n+1th sentence issued by the same sentence issuer adjacent to the nth sentence, inputting the second integrated feature information of the n+1th sentence and the n+1th speaking state feature into the second long short-term memory network model, and outputting the n+2th speaking state feature, wherein n is an integer greater than 1;

[0077] Step 5, until n is equal to the total number of short sentences issued by the current short sentence issuer, extract the n+1th speaking state feature as the speaking state feature of the short sentence issuer;

[0078] The working principle of the above technical solution is as follows: initialize the speaking state characteristics of the speaker of the short sentence to obtain the initial speaking state characteristics of the speaker of the short sentence; obtain the voice line characteristic information, speech speed characteristic information and intonation amplitude change characteristic information of the first short sentence issued by the speaker of the short sentence to obtain the second integrated characteristic information, input the second integrated characteristic information and the corresponding initial speaking state characteristics into the second long short-term memory network model, and output the second speaking state characteristics; obtain the second integrated characteristic information of the second short sentence issued by the speaker of the same short sentence adjacent to the first short sentence, input the second integrated characteristic information and the second speaking state characteristics of the second short sentence into the second long short-term memory network model, and output the third speaking state characteristics. state feature; obtaining the second integrated feature information of the n+1th sentence issued by the same sentence issuer adjacent to the nth sentence, inputting the second integrated feature information of the n+1th sentence and the n+1th speaking state feature into the second long short-term memory network model, and outputting the n+2th speaking state feature, wherein n is an integer greater than 1; until n is equal to the total number of sentences issued by the current sentence issuer, extracting the n+1th speaking state feature as the speaking state feature of the sentence issuer; for each sentence, updating the speaker state feature of the sentence speaker based on the speech feature information of the sentence, wherein the speaker state feature before the update is obtained based on the speech features of all previous sentences of the speaker;

[0079] The beneficial effect of the above technical solution is: through the above technical solution, the speaking state characteristics of the person who sends the short sentence are determined, providing data support for subsequent emotion recognition.

[0080] In one embodiment, determining the issuer switching state of a short sentence based on the issuer of each short sentence and the issuer of the corresponding previous short sentence includes:

[0081] Integrate all short sentences, sort all short sentences in the conversation content according to the order in which the conversation was issued, and generate a sorting result;

[0082] For each short sentence, if the sender of the current short sentence is the same as the sender of the previous short sentence in the sorting result, the preset first switching state is used as the sender switching state of the current short sentence; if the sender of the current short sentence is different from the sender of the previous short sentence in the sorting result, the preset second switching state is used as the sender switching state of the current short sentence;

[0083] The working principle and beneficial effects of the above technical solution are: integrating all short sentences, sorting all short sentences in the conversation content according to the order in which the conversation is issued, and generating a sorting result; for each short sentence, if the issuer of the current short sentence is the same as the issuer of the previous short sentence in the sorting result, the preset first switching state is used as the issuer switching state of the current short sentence; if the issuer of the current short sentence is different from the issuer of the previous short sentence in the sorting result, the preset second switching state is used as the issuer switching state of the current short sentence; through the above technical solution, the issuer switching state of the short sentence is determined to provide data support for subsequent emotion recognition.

[0084] In one embodiment, the emotion recognition result of each short sentence is determined according to the contextual speech features of each short sentence, the speaking state features of the speaker of the short sentence, and the switching state of the speaker, including:

[0085] For each short sentence, the contextual speech features of the short sentence, the speaking state features of the speaker of the short sentence, and the switching state of the speaker are connected to form a new vector;

[0086] Input the new vector to the fully connected layer, and output the probability of each emotion label category of the current short sentence based on the normalized exponential function to determine the probability distribution of the emotion label category of each short sentence in the dialogue content to be identified;

[0087] Determine the emotion recognition result of each short sentence according to the probability distribution of the emotion label category of each short sentence;

[0088] Among them, the emotion recognition result of each short sentence is the probability of each emotion label category of the corresponding short sentence;

[0089] The working principle and beneficial effects of the above technical solution are as follows: for each short sentence, the contextual speech features of the short sentence, the speaking state features of the sentence sender, and the switching state of the sender are connected to form a new vector; the new vector is input to the fully connected layer, and at the same time, the probability of each emotion label category of the current short sentence is output based on the normalized exponential function to determine the probability distribution of the emotion label category of each short sentence in the dialogue content to be identified; according to the probability distribution of the emotion label category of each short sentence, the emotion recognition result of each short sentence is determined; through the above technical solution, the emotion recognition of the dialogue content to be identified is completed.

[0090] In one embodiment, a method for voice user emotion recognition based on AI intelligent neural network also includes:

[0091] Obtain all short sentences currently uttered by the caller as short sentences to be predicted;

[0092] Obtaining the emotion recognition result of each short sentence to be predicted as the emotion data to be predicted;

[0093] Sorting all the emotion data to be predicted according to the generation time of the short sentences to be predicted to obtain a second sorting result;

[0094] Determine whether there are two adjacent emotion data to be predicted whose emotion fluctuation data is greater than a preset emotion fluctuation data range in the second sorting result, and if so, extract the short sentences to be predicted corresponding to the two adjacent emotion data to be predicted and record them as emotion fluctuation points;

[0095] Obtain all emotional fluctuation points and extract entity keywords in the short sentences to be predicted corresponding to all emotional fluctuation points;

[0096] Determine whether the similarity of entity keywords corresponding to adjacent emotional fluctuation points is greater than a preset similarity, and if so, determine that there is a correlation relationship between adjacent emotional fluctuation points;

[0097] Determine whether the emotion conversion relationship of adjacent emotion fluctuation points with an associated relationship is reversed, and if so, determine the entity keyword corresponding to the adjacent emotion fluctuation point as the target keyword;

[0098] When the emotion recognition result of the caller triggers the manual voice switch, all target keywords are integrated and combined with the caller's consultation items obtained in advance to predict the key information that the caller wants to consult;

[0099] Feedback the key information that the caller wants to consult to the manual customer service at the time of manual voice switching;

[0100] The working principle of the above technical solution is as follows: This application provides a voice user emotion recognition method based on AI intelligent neural network, which is preferably applied to customer service call scenarios. When facing customer service call scenarios, in order to improve service efficiency and reduce the service pressure of customer service personnel, intelligent voice customer service is usually provided to communicate with users first. When the intelligent voice customer service cannot complete the corresponding business, it is transferred to manual customer service. At present, in order to improve user satisfaction, emotion recognition is used to actively transfer users to manual customer service when the preset emotion mode is triggered. However, this transfer method is often triggered when the user is in a certain emotion. When the manual customer service asks the user about specific matters, because the user has communicated with the intelligent voice customer service many times before, the emotion is already at a critical point. To prevent users from having too many negative emotions, this solution provides key consulting information to manual customer service, reduces users' negative emotions, and improves customer service efficiency. Specifically, when the intelligent voice customer service is connected, all short sentences currently uttered by the incoming caller are obtained as short sentences to be predicted, and the emotion recognition result of each short sentence to be predicted is obtained as the emotion data to be predicted. All the emotion data to be predicted are sorted according to the generation time of the short sentence to be predicted to obtain a second sorting result. The order of each emotion data to be predicted in the second sorting result is determined by the generation time of the corresponding short sentence to be predicted. Preferably, the order of the emotion data to be predicted corresponding to the short sentence to be predicted with an earlier generation time is first. Then, it is determined whether there is any emotion data in the second sorting result. If there are two adjacent emotion data to be predicted whose emotion fluctuation data is greater than the preset emotion fluctuation data range, extract the short sentences to be predicted corresponding to the two adjacent emotion data to be predicted and record them as emotion fluctuation points; wherein, the method for judging whether there are two adjacent emotion data to be predicted whose emotion fluctuation data is greater than the preset emotion fluctuation data range in the second sorting result is preferably to judge whether the change in the probability distribution of the emotion label categories corresponding to the two adjacent emotion data to be predicted exceeds the preset change range; after determining the emotion fluctuation points, obtain all emotion fluctuation points, extract entity keywords in the short sentences to be predicted corresponding to all emotion fluctuation points, such as specific project information, etc. If the current call scenario is a mobile communication scenario, etc., the specific project information preferably includes Package A and Package B. Entity keywords such as; whether the similarity of entity keywords corresponding to adjacent emotion fluctuation points is greater than a preset similarity, if so, it is determined that there is a correlation between adjacent emotion fluctuation points; it is worth noting that when judging the text similarity of entity keywords, it is preferred to make a comprehensive judgment on all entity keywords in the short sentence corresponding to the current adjacent emotion fluctuation point to determine the total similarity; then, it is determined whether the emotion conversion relationship of adjacent emotion fluctuation points with a correlation is reversed, if so, the entity keyword corresponding to the adjacent emotion fluctuation point is determined to be the target keyword, and when the emotion recognition result of the caller triggers the manual voice switching, all target keywords are integrated, combined with the caller consultation items obtained in advance, to predict the key information to be consulted by the caller;Feedback the key information that the caller wants to consult to the manual customer service at the time of manual voice switching;

[0101] The beneficial effects of the above technical solution are: through the above technical solution, based on voice user emotion recognition, the user's emotional fluctuation points are determined, and the user's consultation focus is confirmed based on the emotional fluctuation points, thereby helping the manual customer service to grasp the service that the user wants to consult as soon as possible, improve user satisfaction, and at the same time improve service efficiency.

[0102] In one embodiment, determining whether the emotion conversion relationship of adjacent emotion fluctuation points having an associated relationship is opposite and generating a determination result includes:

[0103] Obtaining emotion conversion relationships of adjacent emotion fluctuation points that have an associated relationship, wherein the emotion conversion relationships include positive-negative emotion conversion relationships, negative-positive emotion conversion relationships, positive-positive emotion conversion relationships, and negative-negative emotion conversion relationships;

[0104] Determine whether the emotion conversion relationship of the adjacent emotion fluctuation points with associated relationship includes both the positive-negative emotion conversion relationship and the negative-positive emotion conversion relationship. If so, determine that the emotion conversion relationship of the adjacent emotion fluctuation points with associated relationship is opposite, and generate a judgment result.

[0105] The working principle and beneficial effects of the above technical solution are: obtaining the emotion conversion relationship of adjacent emotion fluctuation points with an associated relationship, wherein the emotion conversion relationship includes a positive-negative emotion conversion relationship, a negative-positive emotion conversion relationship, a positive-positive emotion conversion relationship and a negative-negative emotion conversion relationship; for example, the positive-negative emotion conversion relationship is a conversion relationship from positive emotion to negative emotion, the negative-positive emotion conversion relationship is a conversion relationship from negative emotion to positive emotion, the positive-positive emotion conversion relationship is a conversion relationship from positive emotion to positive emotion, and the negative-negative emotion conversion relationship is a conversion relationship from negative emotion to negative emotion; judging whether the emotion conversion relationship of adjacent emotion fluctuation points with an associated relationship includes both a positive-negative emotion conversion relationship and a negative-positive emotion conversion relationship. Conversion relationship, if so, it is determined that the emotion conversion relationship of adjacent emotion fluctuation points with associated relationship is reversed, and a judgment result is generated. Several experimental data show that, under normal circumstances, due to the limitations of intelligent voice customer service, with the emergence of specific entity keywords, the user's emotions generally change from negative-positive emotion conversion relationship to positive-negative emotion conversion relationship; it is worth noting that in order to increase the acquisition breadth of target keywords, it is preferred to include the negative-negative emotion conversion relationship in the reverse range of emotion conversion relationship, that is, it also includes any two of the positive-negative emotion conversion relationship, the negative-negative emotion conversion relationship and the negative-positive emotion conversion relationship; through the above scheme, it is determined whether the user has a large change in emotion due to the triggering of specific entity keywords.

[0106] In one embodiment, all target keywords are integrated and combined with the pre-acquired consultation items of the caller, and the key information to be consulted by the caller is predicted, including:

[0107] Obtain all target keywords and perform data cleaning on all target keywords to obtain keywords to be predicted; wherein data cleaning includes removing repeated target keywords from all target keywords and assigning weights to corresponding target keywords according to the number of repetitions, where the more repeated times a target keyword has, the higher its weight;

[0108] Obtain the pre-obtained caller consultation items, extract the caller consultation items by item according to the matching degree between the keywords to be predicted and each sub-item in the caller consultation items, and predict the specific items consulted by the caller; among which, the specific items consulted by the caller have the highest matching degree with the keywords to be predicted;

[0109] According to the weight of each target keyword in the keywords to be predicted, the project information in the specific project consulted by the caller is classified into importance levels to obtain the importance level of each project information; wherein, the higher the total weight of the target keyword corresponding to each project information, the higher the importance level of the project information;

[0110] According to the importance level of project information, all project information in a specific project is sorted, and a project table is generated based on a preset table template, and the key information to be consulted by the caller is determined based on the project table;

[0111] The working principle of the above technical solution is: obtain all target keywords, and perform data cleaning on all target keywords to obtain keywords to be predicted; wherein, data cleaning includes eliminating repeated target keywords from all target keywords, and assigning weights to corresponding target keywords according to the number of repetitions, the target keywords with more repetitions have higher weights; the sum of all weights is 1; obtain the pre-acquired caller consultation items, extract the caller consultation items by sub-items according to the matching degree between the keywords to be predicted and the sub-items in the caller consultation items, and predict the specific items consulted by the caller; wherein, the specific items consulted by the caller have the highest matching degree with the keywords to be predicted; usually, in order to improve the management of data, the use of classification management is beneficial to improve management efficiency. When faced with consultation content, most users can only explain the major categories of items they want to consult, such as major categories of items such as package changes and network charges in mobile call scenarios, and major categories of items such as product after-sales in online shopping scenarios. Project categories, therefore, through the above technical solution, the subcategories of projects that users want to consult, that is, sub-projects, are predicted in advance, so as to provide reliable data support for the generation of subsequent project tables; according to the weight of each target keyword in the keywords to be predicted, the project information in the specific project consulted by the caller is divided into important levels to obtain the importance level of each project information; wherein, the higher the total weight of the target keywords corresponding to each project information, the higher the importance level of the project information; according to the number of times the user mentions the project information during the prediction process, the user's pre-consultation probability for each type of project information in the sub-project is confirmed, thereby generating a more reliable and detailed project table; according to the importance level of the project information, all project information in the specific project is sorted, and a project table is generated based on a preset table template, and the key information to be consulted by the caller is determined according to the project table; preferably, the generation template of the project table can be determined according to the actual needs of the customer service staff, so that the customer service staff can quickly extract useful information;

[0112] The beneficial effect of the above technical solution is: through the above technical solution, the consulting information of the incoming caller is predicted to generate a project table with key information for reference by the manual customer service, which helps the manual customer service to grasp the service that the user wants to consult as soon as possible, improve user satisfaction, and at the same time improve service efficiency.

[0113] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for voice user emotion recognition based on AI intelligent neural network, characterized in that: include: Obtaining voice feature information of each short sentence in the conversation content to be recognized; The speech feature information of each short sentence is processed based on the first long short-term memory network model to obtain the contextual semantic features of each short sentence; Inputting speech feature information of multiple short sentences into the second long short-term memory network model to extract the speaking state features of the speaker of the short sentences; Determining a switch state of a sender of each short sentence based on the sender of each short sentence and the sender of the corresponding previous short sentence; Determine the emotion recognition result of each short sentence according to the context speech features of each short sentence, the speaking state features of the speaker of the short sentence, and the switching state of the speaker; Also includes: Obtain all short sentences currently uttered by the caller as short sentences to be predicted; Obtaining the emotion recognition result of each short sentence to be predicted as the emotion data to be predicted; Sorting all the emotion data to be predicted according to the generation time of the short sentences to be predicted to obtain a second sorting result; Determine whether there are two adjacent emotion data to be predicted whose emotion fluctuation data is greater than a preset emotion fluctuation data range in the second sorting result, and if so, extract the short sentences to be predicted corresponding to the two adjacent emotion data to be predicted and record them as emotion fluctuation points; Obtain all emotional fluctuation points and extract entity keywords in the short sentences to be predicted corresponding to all emotional fluctuation points; Determine whether the similarity of entity keywords corresponding to adjacent emotional fluctuation points is greater than a preset similarity, and if so, determine that there is a correlation relationship between adjacent emotional fluctuation points; Determine whether the emotion conversion relationship of adjacent emotion fluctuation points with an associated relationship is reversed, and if so, determine the entity keyword corresponding to the adjacent emotion fluctuation point as the target keyword; When the emotion recognition result of the caller triggers the manual voice switch, all target keywords are integrated and combined with the caller's consultation items obtained in advance to predict the key information that the caller wants to consult; Feedback the key information that the caller wants to consult to the manual customer service at the time of the manual voice switch.

2. According to claim 1, a method for voice user emotion recognition based on AI intelligent neural network is characterized in that: The speech feature information includes voice feature information, speech rate feature information, text feature information, semantic feature information and intonation amplitude change feature information.

3. The method for voice user emotion recognition based on AI intelligent neural network according to claim 1 is characterized in that: The speech feature information of each short sentence is processed based on the first long short-term memory network model to obtain the contextual semantic features of each short sentence, including: Sort all the short sentences in the conversation content according to the order in which the conversation was sent, and generate a sorting result; Acquire text feature information and semantic feature information of any short sentence in the conversation content to obtain integrated feature information, input the integrated feature information into the first long short-term memory network model, and output a first memory result; For each short sentence, the first memory result of the previous short sentence that is adjacent to the current short sentence and the integrated feature information of the current short sentence are input into the first long short-term memory network model, and the contextual semantic features of the current short sentence are output.

4. The method for voice user emotion recognition based on AI intelligent neural network according to claim 1 is characterized in that: Input the speech feature information of multiple short sentences into the second long short-term memory network model to extract the speaking state features of the speaker of the short sentences, including: Step 1, initializing the speaking state feature of the person who sends the short sentence, and obtaining the initial speaking state feature of the person who sends the short sentence; Step 2, obtaining the voice feature information, speech rate feature information and intonation amplitude change feature information of the first short sentence sent by the short sentence sender, obtaining the second integrated feature information, inputting the second integrated feature information and the corresponding initial speaking state feature into the second long short-term memory network model, and outputting the second speaking state feature; Step 3, obtaining the second integrated feature information of the second short sentence issued by the same short sentence issuer adjacent to the first short sentence, inputting the second integrated feature information of the second short sentence and the second speaking state feature into the second long short-term memory network model, and outputting the third speaking state feature; Step 4, obtaining the second integrated feature information of the n+1th sentence issued by the same sentence issuer adjacent to the nth sentence, inputting the second integrated feature information of the n+1th sentence and the n+1th speaking state feature into the second long short-term memory network model, and outputting the n+2th speaking state feature, wherein n is an integer greater than 1; Step 5: until n is equal to the total number of short sentences issued by the current short sentence issuer, extract the (n+1)th speaking state feature as the speaking state feature of the short sentence issuer.

5. The method for voice user emotion recognition based on AI intelligent neural network according to claim 1 is characterized in that: Determining the switch state of the sender of each short sentence based on the sender of each short sentence and the sender of the corresponding previous short sentence includes: Integrate all short sentences, sort all short sentences in the conversation content according to the order in which the conversation was issued, and generate a sorting result; For each short sentence, if the sender of the current short sentence is the same as the sender of the previous short sentence in the sorting result, the preset first switching state will be used as the sender switching state of the current short sentence; if the sender of the current short sentence is different from the sender of the previous short sentence in the sorting result, the preset second switching state will be used as the sender switching state of the current short sentence.

6. According to the method for voice user emotion recognition based on AI intelligent neural network in claim 1, the emotion recognition result of each short sentence is determined according to the contextual speech features of each short sentence, the speaking state features of the speaker of the short sentence and the switching state of the speaker, including: For each short sentence, the contextual speech features of the short sentence, the speaking state features of the speaker of the short sentence, and the switching state of the speaker are connected to form a new vector; Input the new vector to the fully connected layer, and output the probability of each emotion label category of the current short sentence based on the normalized exponential function to determine the probability distribution of the emotion label category of each short sentence in the dialogue content to be identified; According to the probability distribution of the emotion label category of each short sentence, the emotion recognition result of each short sentence is determined.

7. The method for voice user emotion recognition based on AI intelligent neural network according to claim 6 is characterized in that: The emotion recognition result of each short sentence is the probability of each emotion label category of the corresponding short sentence.

8. The method for voice user emotion recognition based on AI intelligent neural network according to claim 1, characterized in that: Determine whether the emotion conversion relationship of adjacent emotion fluctuation points with a correlation relationship is opposite, and generate a judgment result, including: Obtaining emotion conversion relationships of adjacent emotion fluctuation points that have an associated relationship, wherein the emotion conversion relationships include positive-negative emotion conversion relationships, negative-positive emotion conversion relationships, positive-positive emotion conversion relationships, and negative-negative emotion conversion relationships; It is determined whether the emotion conversion relationship of the adjacent emotion fluctuation points with associated relationship includes both the positive-negative emotion conversion relationship and the negative-positive emotion conversion relationship. If so, it is determined that the emotion conversion relationship of the adjacent emotion fluctuation points with associated relationship is opposite, and a judgment result is generated.

9. The method for voice user emotion recognition based on AI intelligent neural network according to claim 1, characterized in that: Integrate all target keywords and combine them with the pre-acquired caller inquiry items to predict the key information that the caller is going to inquire about, including: Obtain all target keywords and perform data cleaning on all target keywords to obtain keywords to be predicted; wherein data cleaning includes removing repeated target keywords from all target keywords and assigning weights to corresponding target keywords according to the number of repetitions, where the more repeated times a target keyword has, the higher its weight; Obtain the pre-obtained caller consultation items, extract the caller consultation items by item according to the matching degree between the keywords to be predicted and each sub-item in the caller consultation items, and predict the specific items consulted by the caller; among which, the specific items consulted by the caller have the highest matching degree with the keywords to be predicted; According to the weight of each target keyword in the keywords to be predicted, the project information in the specific project consulted by the caller is classified into importance levels to obtain the importance level of each project information; wherein, the higher the total weight of the target keyword corresponding to each project information, the higher the importance level of the project information; According to the importance level of project information, all project information in a specific project is sorted, and a project table is generated based on a preset table template, and the key information that the caller wants to consult is determined based on the project table.

Images