A decision-making method for transferring human services from an intelligent voice robot
By combining large language model and multimodal comprehensive model to analyze user intentions, emotions and voice emotions, and calculate the transfer index, the problem that intelligent voice robots cannot accurately understand user intentions is solved, more accurate transfer decisions are achieved, and users are improved Experience and efficiency.
Patent Information
- Application Number
- CN202410807062.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-06-21
AI Technical Summary
Existing intelligent voice robots cannot accurately understand user intentions when dealing with complex problems, resulting in a decline in user experience. In addition, traditional manual transfer decision-making methods cannot accurately identify whether manual transfer operations are required, affecting user experience and increasing labor costs.
Using an intelligent voice robot to transfer artificial decision-making method, by converting user audio into speech text and passing it into a large language model, identifying user intentions, searching similar corpus texts from the knowledge base, and analyzing user emotions with multimodal comprehensive model and voice sentiment, calculate the transfer index, and decide whether to transfer manual operation.
It improves the accuracy of transfer decisions, reduces the dependence of manual operators, saves labor costs, avoids unnecessary transfers or delays, and enhances user experience and satisfaction.
Smart Images

Figure CN118588082B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and in particular to a decision method for transferring speech from an intelligent speech robot to a human being. Background Art
[0002] Intelligent voice robots are widely used in daily life, and are able to communicate with users and provide services through voice interaction. However, current intelligent voice robots still have some limitations in handling complex problems or specific situations. For example, in some cases, intelligent voice robots may not accurately understand the user's questions or provide satisfactory answers. This may lead to a decline in user experience and may require the conversation to be transferred to a human operator to resolve the issue.
[0003] The traditional decision-making method for manual transfer is mainly based on preset rules or thresholds, which causes the intelligent voice robot to transfer to manual operation too early or too frequently in some cases, affecting the user experience. This method cannot accurately identify whether manual transfer is required, resulting in a serious imbalance in the number of transfers. In addition, manual operators are required to continuously monitor and intervene, which increases labor costs and workload.
[0004] In addition, the traditional method prompts users to transfer to manual operation through voice or button, but does not fully consider the user's emotional state and voice emotion, and thus cannot accurately grasp the timing of the user's need for manual operation, resulting in the user being unable to find the entrance when they need manual support. Summary of the invention
[0005] In view of the above analysis, an embodiment of the present invention aims to provide a decision method for transferring calls by an intelligent voice robot to a human being, so as to solve the problem that the existing method does not take user emotions and feelings into consideration, resulting in inaccurate manual transfer decisions.
[0006] The embodiment of the present invention provides a decision method for transferring a call between an intelligent voice robot and a human being, comprising the following steps:
[0007] The received user audio is converted into speech text and then passed into the large language model to identify the user's intention and retrieve multiple corpus texts that are most similar to the user's intention from the knowledge base;
[0008] If the highest similarity of multiple corpus texts does not exceed the threshold, the call is transferred to manual service; otherwise, multiple corpus texts are randomly sampled and the mean similarity is calculated. When the mean similarity is equal to 0, the call is transferred to manual service; when the mean similarity is greater than 0, the MFCC features of the user audio and the word embedding vector of the speech text are passed into the multimodal comprehensive model, and the speech emotion value and emotional state value of each word in the speech text are output; the transfer index is calculated based on the weight, speech emotion value, emotional state value and mean similarity of each word. When the transfer index is less than 0, the call is transferred to manual service.
[0009] Based on the further improvement of the above method, multiple corpus texts are randomly sampled and the similarity mean is calculated. For each corpus text, a corresponding random number is generated from the uniform distribution uniform(0,1). If the generated random number is less than the similarity of the corpus text, the similarity of the corpus text is sampled; the average similarity of all sampled corpus texts is calculated to obtain the similarity mean.
[0010] Based on the further improvement of the above method, the transfer index is calculated by the following formula according to the weight, voice emotion value, emotional state value and similarity mean of each word:
[0011]
[0012] Where r represents the transfer index, σ j represents the weight of the jth word, M represents the total number of words in the speech text, and p j and q j They represent the speech emotion value and emotional state value of the jth word respectively, μ represents the mean similarity, and λ represents the penalty coefficient.
[0013] Based on the further improvement of the above method, the multimodal comprehensive model includes a first BILSTM module, a second BILSTM module, a multi-head attention module, a first CNN module and a second CNN module; the first BILSTM module is used to receive the MFCC features of the user audio, and output the audio feature vector to the multi-head attention module; the second BILSTM module is used to receive the word embedding vector of the speech text, and output the word feature vector to the multi-head attention module; the multi-head attention model is used to learn the attention weights between the audio feature vector and the word feature vector using the cross-attention mechanism, and weightedly obtain the speech alignment feature vector of each word; the first CNN module and the second CNN module are used to receive the feature vector spliced from the word feature vector and the corresponding speech alignment feature vector, respectively classify the speech emotion and emotional state, and output the speech emotion value and emotional state value of each word.
[0014] Based on a further improvement of the above method, the audio feature vector output by the first BILSTM module is obtained by adding the feature vectors of the hidden layers of the first forward LSTM and the first backward LSTM therein; the word feature vector output by the second BILSTM module is obtained by adding the feature vectors of the hidden layers of the second forward LSTM and the second backward LSTM therein.
[0015] Based on the further improvement of the above method, the multi-head attention model uses the cross-attention mechanism to learn the attention weights between the audio feature vector and the word feature vector, including:
[0016] For each attention head, the query matrix of each word is calculated based on the query weight and the feature vector of each frame of audio; the key matrix of each frame of audio is calculated based on the key weight and the feature vector of each word; the dot product results of the query matrix and the key matrix are normalized by the softmax function to obtain the attention weight between each word feature vector of each attention head and the feature vector of each frame of audio.
[0017] Based on the further improvement of the above method, the attention weight between the audio feature vector and the word feature vector is calculated by the following formula:
[0018]
[0019] in, represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, W Q,t and W K,t denote the query weight and key weight in the t-th attention head, respectively, and s i represents the audio feature vector of the i-th frame, h j represents the j-th word feature vector, T represents the transpose operation of the matrix, and softmax(·) represents the normalization function.
[0020] Based on the further improvement of the above method, the speech alignment feature vector of each word is obtained by weighting according to the attention weight through the following formula:
[0021]
[0022] in, represents the speech-aligned feature vector of the jth word in the tth attention head, represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, W V,t represents the value weight in the tth attention head, s i Represents the audio feature vector of the i-th frame.
[0023] Based on the further improvement of the above method, after the first CNN module and the second CNN module classify the intensity of emotion and the positive and negative emotion respectively, the classification results are respectively obtained using sigmoid and tanh activation functions to obtain the speech emotion value and emotional state value.
[0024] Based on the further improvement of the above method, the weight of each word is calculated by the following formula:
[0025]
[0026] Among them, σ j represents the weight of the jth word, and M represents the total number of words in the speech text.
[0027] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0028] 1. Provide a manual transfer decision method that integrates a large language model, knowledge base retrieval, sentiment analysis and voice emotion analysis technology. This method combines the generation ability and semantic understanding of the large language model. While using intent similarity to query the answer, it comprehensively considers the user's emotional state and voice emotion. By integrating intent similarity with emotional state and voice emotion values to calculate the transfer index, it identifies whether manual transfer is required. This not only reduces the dependence on manual operators, saves labor costs, and improves work efficiency; it also avoids unnecessary transfers or delays and improves the accuracy of transfer decisions.
[0029] 2. By comprehensively considering the user's emotional state and voice emotions, the emotional changes can be more accurately grasped, so that the intelligent voice robot can better understand and meet the needs of users, provide more accurate user demand transfer timing, and achieve a balance in the number of transfers. When the user's emotions are negative or the voice expression is difficult, timely transfer to the human operator can provide more accurate support and answers, improve user experience, and increase user satisfaction.
[0030] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The drawings are only for the purpose of illustrating specific embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.
[0032] Figure 1 This is a flow chart of a decision-making method for an intelligent voice robot to transfer a call to a human being in an embodiment of the present invention;
[0033] Figure 2 Schematic diagram of the structure of the multimodal comprehensive model in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0035] A specific embodiment of the present invention discloses a decision method for transferring a call between an intelligent voice robot and a human being, such as Figure 1As shown, the following steps are included:
[0036] S1, convert the user audio received by the intelligent voice robot into voice text and then pass it into the large language model to identify the user's intention and retrieve multiple corpus texts that are most similar to the user's intention from the knowledge base;
[0037] S2. If the highest similarity of multiple corpus texts does not exceed the threshold, the call is transferred to manual service. Otherwise, multiple corpus texts are randomly sampled and the mean similarity is calculated. When the mean similarity is equal to 0, the call is transferred to manual service. When the mean similarity is greater than 0, the MFCC features of the user audio and the word embedding vector of the speech text are passed into the multimodal comprehensive model, and the speech emotion value and emotional state value of each word in the speech text are output. According to the weight, speech emotion value, emotional state value and mean similarity of each word, the transfer index is calculated. When the transfer index is less than 0, the call is transferred to manual service.
[0038] During implementation, by combining the generative capabilities and semantic understanding of the large language model, the identified intent similarity is used to query answers, the user's emotional state and voice emotions are analyzed, and the need to transfer to manual operation is identified based on the similarity of the query results and the analysis results of emotional emotions. This improves the accuracy of intelligent voice robot decisions and user experience, ensuring that users can get more accurate support and answers when needed.
[0039] Specifically, in step S1, the domain knowledge corresponding to the robot customer service is used to enhance the training of the pre-trained large language model, such as the LLaMA large model, to obtain a trained large language model. During implementation, the questions raised by the user in audio form are converted into speech text through speech recognition technology, such as Google Cloud Speech-to-Text and IBM Watson Speech to Text. The converted speech text is passed into the large language model to identify the user's intention, that is, to obtain the embedding vector in the high-dimensional space corresponding to the user's intention.
[0040] When retrieving from the knowledge base, the same large language model is first used to obtain the embedding vectors of each corpus text in the knowledge base, and then the similarity between the embedding vector of the user's intention and the embedding vector of each corpus text in the knowledge base is calculated to obtain multiple corpus texts that are most similar to the embedding vector of the user's intention and their similarities.
[0041] Preferably, the cosine similarity is calculated as the similarity between vectors, and the top five most similar corpus texts are selected.
[0042] In step S2, whether to transfer to a human operator is determined based on whether the highest similarity among the retrieved corpus texts exceeds a threshold. Preferably, the threshold is set to 0.85.
[0043] If the highest similarity does not exceed the threshold, the call is triggered to be transferred to a manual operation; otherwise, further identification is performed from two aspects, namely (1) the similarity mean and (2) the user's voice emotion and emotional state. Next, the identification process of these two aspects is specifically described.
[0044] (1) Similarity mean
[0045] It should be noted that the similarity mean is obtained by calculating the mean of the similarities of all sampled corpus texts through the similarities of multiple corpus texts retrieved by random sampling step S1. Among them, random sampling is to generate a corresponding random number from uniform distribution uniform(0,1) for each corpus text. If the generated random number is less than the similarity of the corpus text, the similarity of the corpus text is sampled, that is, the similarity of the corpus text is retained. Otherwise, the similarity of the corpus text is discarded.
[0046] When the mean similarity is 0, it means that the similarities of all the retrieved corpus texts are discarded, which triggers manual transfer; when the mean similarity is greater than 0, the next step of recognition is performed.
[0047] (2) User’s voice emotion and emotional state
[0048] This step uses the trained multimodal comprehensive model to fuse the user audio and the speech text generated by speech recognition conversion, and obtains the speech emotion value and emotional state value of each word in the speech text.
[0049] It should be noted that the multimodal comprehensive model Figure 2 As shown, it includes a first BILSTM (Bi-directional Long Short-Term Memory) module, a second BILSTM module, a multi-head attention module, a first CNN (Convolutional Neural Network) module and a second CNN module.
[0050] Specifically, each BILSTM module consists of two unidirectional LSTMs, which are respectively referred to as forward LSTM and backward LSTM for ease of explanation. The forward LSTM is used to calculate the forward context information, and the backward LSTM is used to calculate the reverse context information. In this way, complete context state information can be provided for each word in each frame of audio and speech text.
[0051] In this embodiment, after pre-emphasis, framing and windowing of the user audio, the MFCC (Mel Frequency Cepstrum Coefficient) feature of each frame of audio is obtained, and the MFCC feature of each frame of audio is passed to the first BILSTM module, and the feature vector obtained by adding the feature vectors of the hidden layers of the first forward LSTM and the first backward LSTM is used as the audio feature vector, and the formula is expressed as follows:
[0052]
[0053] Among them, s i represents the audio feature vector of the i-th frame, i∈{1,…,N}, N represents the total number of frames of the user audio; x i and x N-i+1 Represents the MFCC features of the i-th frame user audio and the N-i+1-th frame user audio, represents the first forward LSTM, represents the first backward LSTM, represents the output vector of the hidden layer of the first forward LSTM, Represents the output vector of the hidden layer of the first backward LSTM.
[0054] The converted speech text is preprocessed, such as removing noise words and punctuation marks, and then the speech text is segmented into separate words using the word segmentation technology in natural language processing (NLP). The word embedding vector of each word in the speech text is obtained using models such as word2vec and text2vec-large-chinese, and then passed to the second BILSTM module. The feature vector obtained by adding the feature vectors of the hidden layers of the second forward LSTM and the second backward LSTM is used as the word feature vector. The formula is as follows:
[0055]
[0056] Among them, h j represents the jth word feature vector, j∈{1,…,M}, M represents the total number of words in the speech text; e j and e M-j+1 represents the i-th word embedding vector and the M-j+1-th word embedding vector, represents the second forward LSTM, represents the second backward LSTM, represents the output vector of the hidden layer of the second forward LSTM, Represents the output vector of the hidden layer of the second backward LSTM.
[0057] It should be emphasized that this embodiment reduces the dimension and redundant information of the features by adding the output vectors of the hidden layers of the forward and backward LSTMs, avoids the dimensionality disaster, and makes it easier to learn the comprehensive influence between the forward and reverse features, thereby improving the generalization ability of the model.
[0058] Since the user audio and its converted speech text have an inherent coexistence relationship in time, the multi-head attention module in this embodiment receives the audio feature vector and the word feature vector, and uses the cross-attention mechanism to learn the attention weights from these two different sequence feature vectors on the basis of the multi-head attention mechanism, capture the complex relationship between different sequences, and then weight them to obtain the speech alignment feature vector of each word, so as to learn the consistency of audio and speech text in the time domain.
[0059] Specifically, the multi-head attention model uses the cross-attention mechanism to learn the attention weights between the audio feature vector and the word feature vector, including:
[0060] For each attention head, the query matrix of each word is calculated based on the query weight and the feature vector of each frame of audio; the key matrix of each frame of audio is calculated based on the key weight and the feature vector of each word; the dot product result of the query matrix and the key matrix is normalized by the softmax function to obtain the attention weight between each word feature vector of each attention head and each frame of audio feature vector. The formula is as follows:
[0061]
[0062] in, W represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, t∈{1,…,R}, R represents the number of attention heads; Q,t and W K,t denote the query weight and key weight in the t-th attention head, respectively, and s i represents the audio feature vector of the i-th frame, h j represents the j-th word feature vector, T represents the transpose operation of the matrix, and softmax(·) represents the normalization function.
[0063] Furthermore, the audio feature vector is weighted according to the attention weight and the value weight, and the speech alignment feature vector of each word is obtained by the following formula:
[0064]
[0065] in, represents the speech-aligned feature vector of the jth word in the tth attention head, W V,t represents the value weight in the tth attention head.
[0066] It should be noted that the query weight W Q,t , key weight W K,t Sum weight W V,t It is the parameter to be optimized during training.
[0067] The feature vector h of each word in the speech text j Align feature vectors with corresponding speech Concatenate to get the total feature vector of each word The use of multi-feature fusion to obtain voice emotion and sentiment information from different angles and levels makes up for the shortcomings of the single feature and limited expression ability, and improves the speech recognition rate.
[0068] The total feature vector of each word is input into the first CNN module and the second CNN module respectively, and the speech emotion and emotional state are classified, and the speech emotion value and emotional state value of each word are output. The categories of speech emotion include: strong emotion and weak emotion; the categories of emotional state include: positive emotion and negative emotion. In other words, the first CNN module and the second CNN module are both a binary convolutional neural network, which classify the strength of emotion and the positive and negative emotion respectively.
[0069] During the training process, the first CNN module updates parameters using the gradient descent method according to the cross entropy loss function based on the sigmoid activation function. The second CNN module updates parameters using the gradient descent method according to the cross entropy loss function based on the tanh activation function. The mini-batch Adam optimization algorithm is used for parameter tuning, and the dropout regularization and early stopping methods are used to prevent data overfitting.
[0070] Finally, the first CNN module outputs the speech emotion value in the range of (0,1), and the second CNN module outputs the emotional state value in the range of (-1,1).
[0071] This embodiment takes into account that user emotions change over time. Each word in the voice text has a weight related to emotion. The user's final emotion is more related to the emotion expressed by the most recent word. Therefore, the weight of each word is calculated by the following formula:
[0072]
[0073] Among them, σ j represents the weight of the jth word.
[0074] According to the weight of each word, the voice emotion value and emotional state value of each word are weighted and summed respectively to obtain the voice emotion value and emotional state value of the entire speech text, and then the similarity mean obtained by sampling is added, and the penalty coefficient is subtracted to calculate the transfer index. The formula is as follows:
[0075]
[0076] Where r represents the transfer index, σ j represents the weight of the jth word, M represents the total number of words in the speech text, and p j and q j They represent the speech emotion value and emotional state value of the j-th word respectively, μ represents the similarity mean, and λ represents the penalty coefficient, which is preferably set to 0.5.
[0077] Finally, when the transfer index is less than 0, the transfer manual operation is triggered. Otherwise, the transfer manual operation is not triggered, and the multiple corpus texts retrieved in step S1 that are most similar to the user's intention are returned to the user. Combined with formula (6), when the user's emotion is too negative, the effect of the mean similarity is weakened, and the mean similarity must be at least higher than the penalty coefficient to not trigger the transfer manual operation; when the user's emotion is positive, as long as the mean similarity is higher than the penalty coefficient, the transfer manual operation will not be triggered.
[0078] Compared with the prior art, the decision method for transferring a call to a human provided by the present embodiment is a decision method for transferring a call to a human being that integrates a large language model, knowledge base retrieval, sentiment analysis, and voice emotion analysis technology. The method combines the generation capability and semantic understanding of the large language model, and while using the intention similarity to query the answer, comprehensively considers the user's emotional state and voice emotion, and calculates the transfer index by integrating the intention similarity with the emotional state and voice emotion value to identify whether a transfer to a human operator is required. This not only reduces the dependence on human operators, saves labor costs, and improves work efficiency; it also avoids unnecessary transfers or delays, and improves the accuracy of transfer decisions. By comprehensively considering the user's emotional state and voice emotion, the emotional changes can be more accurately grasped, which facilitates the intelligent voice robot to better understand and meet the needs of users, provide more accurate user demand transfer timing, and achieve a balance in the number of transfers. When the user is emotionally negative or has difficulty expressing voice, timely transfer to a human operator can provide more accurate support and answers, improve user experience, and increase user satisfaction.
[0079] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0080] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A decision-making method for transferring human input by an intelligent voice robot, characterized in that: The following steps are involved: The user audio received by the intelligent voice robot is converted into voice text and then passed into the large language model to identify the user's intention and retrieve multiple corpus texts that are most similar to the user's intention from the knowledge base; If the highest similarity of the multiple corpus texts does not exceed the threshold, the call is transferred to a human operator; Otherwise, randomly sample the multiple corpus texts and calculate the similarity mean. When the similarity mean is equal to 0, transfer to manual. When the similarity mean is greater than 0, pass the MFCC features of the user audio and the word embedding vector of the speech text into the multimodal comprehensive model, and output the speech emotion value and emotional state value of each word in the speech text. A transfer index is calculated according to the weight, voice emotion value, emotional state value and similarity average of each word. When the transfer index is less than 0, the call is transferred manually.
2. The decision-making method for transferring human resources by an intelligent voice robot according to claim 1 is characterized in that: The random sampling of the plurality of corpus texts and calculation of the similarity mean is to generate a corresponding random number from a uniform distribution uniform(0,1) for each corpus text, and if the generated random number is less than the similarity of the corpus text, then the similarity of the corpus text is sampled; The similarity mean is calculated by averaging the similarities of all sampled corpus texts.
3. The decision-making method for transferring from an intelligent voice robot to a human according to claim 1, characterized in that: The transfer index is calculated according to the weight, voice emotion value, emotional state value and similarity mean of each word by the following formula: Where r represents the transfer index, σ j represents the weight of the jth word, M represents the total number of words in the speech text, and p j and q j They represent the speech emotion value and emotional state value of the jth word respectively, μ represents the mean similarity, and λ represents the penalty coefficient.
4. The decision-making method for transferring from an intelligent voice robot to a human according to claim 1, characterized in that: The multimodal comprehensive model includes a first BILSTM module, a second BILSTM module, a multi-head attention module, a first CNN module and a second CNN module; the first BILSTM module is used to receive the MFCC features of the user audio and output the audio feature vector to the multi-head attention module; the second BILSTM module is used to receive the word embedding vector of the speech text and output the word feature vector to the multi-head attention module; the multi-head attention module is used to learn the attention weights between the audio feature vector and the word feature vector using the cross-attention mechanism, and weightedly obtain the speech alignment feature vector of each word; the first CNN module and the second CNN module are used to receive the feature vector after the word feature vector is spliced with the corresponding speech alignment feature vector, respectively classify the speech emotion and emotional state, and output the speech emotion value and emotional state value of each word.
5. The decision-making method for transferring from an intelligent voice robot to a human according to claim 4, characterized in that: The first BILSTM module outputs an audio feature vector obtained by adding the feature vectors of the hidden layers of the first forward LSTM and the first backward LSTM therein; the second BILSTM module outputs a word feature vector obtained by adding the feature vectors of the hidden layers of the second forward LSTM and the second backward LSTM therein.
6. The decision-making method for transferring from an intelligent voice robot to a human according to claim 4, characterized in that: The multi-head attention model uses a cross-attention mechanism to learn the attention weights between the audio feature vector and the word feature vector, including: For each attention head, the query matrix of each word is calculated according to the query weight and the feature vector of each frame of audio; the key matrix of each frame of audio is calculated according to the key weight and the feature vector of each word; the dot product result of the query matrix and the key matrix is normalized by the softmax function to obtain the attention weight between each word feature vector of each attention head and each frame of audio feature vector.
7. The decision-making method for transferring from an intelligent voice robot to a human according to claim 6, characterized in that: The attention weight between the audio feature vector and the word feature vector is calculated by the following formula: in, represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, W Q,t and W K,t denote the query weight and key weight in the t-th attention head, respectively, and s i represents the audio feature vector of the i-th frame, h j represents the j-th word feature vector, T represents the transpose operation of the matrix, and softmax(-) represents the normalization function.
8. The decision-making method for transferring from an intelligent voice robot to a human according to claim 4, characterized in that: The speech alignment feature vector of each word is obtained by weighting it according to the attention weight using the following formula: in, represents the speech-aligned feature vector of the jth word in the tth attention head, represents the attention weight between the jth word feature vector and the i-th frame audio feature vector in the t-th attention head, W V,t represents the value weight in the tth attention head, s i Represents the audio feature vector of the i-th frame.
9. The decision method for transferring a call from an intelligent voice robot to a human according to claim 4, characterized in that: After the first CNN module and the second CNN module classify the intensity of emotion and the positive and negative emotion respectively, the classification results are respectively obtained using sigmoid and tanh activation functions to obtain the speech emotion value and the emotional state value.
10. The decision-making method for transferring from an intelligent voice robot to a human according to claim 1, characterized in that: The weight of each word is calculated by the following formula: Among them, σ j represents the weight of the jth word, and M represents the total number of words in the speech text.
