Voice emotion recognition method and device, computer device and readable storage medium

Abstract

The application provides a voice emotion recognition method and device, computer equipment and readable storage medium. The method comprises the following steps: obtaining original voice to be recognized and corresponding original text; analyzing the original voice to obtain a plurality of voice features corresponding to the original voice; the voice features at least comprise a mel-frequency spectrum, a mel-frequency cepstral coefficient and a chroma vector; a voice vector corresponding to each voice feature is obtained based on a self-attention mechanism; analyzing the original text to obtain text embedding information corresponding to the original text; inputting the text embedding information into a preset text extraction model; the text extraction model analyzes the text embedding information and outputs a text feature vector corresponding to the text embedding information; generating a target emotion type corresponding to the original voice according to the text feature vector and each voice vector, and completing emotion recognition of the original voice. A multi-modal emotion recognition method is proposed, which combines the content of the text side and the voice side for emotion recognition, and improves the accuracy of emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic parsing, and particularly relates to a speech emotion recognition method and device, a computer device and a readable storage medium. BACKGROUND

[0002] Speech emotion recognition (SER) has a wide range of applications in robots, security, language translation, automatic identification, intelligent toys, lie detection, etc. Although there have been many developments in speech processing, natural emotion understanding is still a challenging task for computing systems.

[0003] Existing speech emotion recognition methods use pre-training models based on Transformer to fuse the information of text and speech modalities. However, these deep learning-based systems have always been in the role of a black box. It is impossible to know its internal mechanism. In order to study the internal mechanism, the prior art uses a method of decomposing the output prediction by tracking the contribution of all neuron modules by finding the input part responsible for a specific output. However, the above method cannot show the layer-by-layer training of the network and cannot know how the network comes to the classification conclusion during the training process. This makes the accuracy and reliability of the speech emotion recognition result not very ideal.

[0004] Therefore, there is an urgent need for a question and answer information generation method for multi-modal data to improve the performance of intelligent customer service to which the method is applied. SUMMARY

[0005] The present application provides a speech emotion recognition method, device, computer device and readable storage medium, aiming to solve the problem that existing speech emotion recognition methods use pre-training models based on Transformer to fuse the information of text and speech modalities. However, these deep learning-based systems have always been in the role of a black box. It is impossible to know its internal mechanism. In order to study the internal mechanism, the prior art uses a method of decomposing the output prediction by tracking the contribution of all neuron modules by finding the input part responsible for a specific output. However, the above method cannot show the layer-by-layer training of the network and cannot know how the network comes to the classification conclusion during the training process. This makes the accuracy and reliability of the speech emotion recognition result not very ideal.

[0006] In a first aspect, the present application provides a speech emotion recognition method, comprising:

[0007] obtaining original speech to be recognized and original text corresponding to the original speech;

[0008] parsing the original speech to obtain a plurality of speech features corresponding to the original speech; the speech features at least include a mel spectrum, a mel frequency cepstral coefficient and a chroma vector;

[0009] Based on the self-attention mechanism, the speech vector corresponding to each speech feature is obtained respectively;

[0010] Parse the original text and obtain the text embedding information corresponding to the original text;

[0011] The text embedding information is input into a preset text extraction model, which parses the text embedding information and outputs a text feature vector corresponding to the text embedding information;

[0012] The target emotion type corresponding to the original speech is generated according to the text feature vector and each speech vector, completing the emotion recognition of the original speech.

[0013] In some embodiments, the speech vector corresponding to each speech feature is obtained based on the self-attention mechanism, including: inputting each speech feature into a preset gated recurrent unit to output dependency information corresponding to the speech feature; inputting the dependency information corresponding to each speech feature into a preset first attention module, and the first attention module outputs the speech vector corresponding to each speech feature based on the self-attention mechanism.

[0014] Exemplarily, before inputting the dependency information corresponding to each speech feature into the preset first attention module, it also includes: inputting the dependency information into a hyperbolic tangent activation unit, the hyperbolic tangent activation unit parses the dependency information, and outputs the normalization information corresponding to each speech feature, so as to input the normalization information and the corresponding speech feature into the first attention module.

[0015] In some embodiments, there are multiple text embedding information; the text extraction model includes a second attention module and a hidden module; the text embedding information is input into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information, including: inputting each text embedding information into the second attention module and the hidden module in turn, obtaining a hidden vector corresponding to each text embedding information; obtaining a text feature vector based on multiple hidden vectors.

[0016] In some embodiments, the target emotion type is any one of a plurality of preset emotion types; after completing the emotion recognition of the original speech, it also includes: determining each emotion type except the target emotion type in the plurality of emotion types as a verification emotion type; obtaining the intersection information corresponding to the target emotion type and each verification emotion type respectively; if it is determined according to the intersection information that the target emotion type is different from each verification emotion type, the verification of the emotion recognition is completed.

[0017] For example, the intersection information corresponding to each verification emotion type and the target emotion type is obtained respectively, including: obtaining principal component information corresponding to the emotion type; calculating the average value information and the standard deviation information corresponding to the principal component information; obtaining the first emotion component corresponding to the emotion category according to the average value information and the standard deviation information; obtaining the second emotion component corresponding to each verification emotion type; and calculating the intersection information of the first emotion component and each second emotion component respectively.

[0018] In some embodiments, the target emotion type corresponding to the original speech is generated according to the text feature vector and each speech vector, including: full connection of the text feature vector and each speech vector to obtain a splicing vector; calculating the target emotion type according to a preset prediction function; the expression corresponding to the prediction function includes:

[0019] y e =softmax(W T E i +b);

[0020] Wherein, e is the target emotion type, y e is the prediction of the target emotion type, W is the weight matrix, b represents the bias term, E i represents the splicing vector, and softmax() is the prediction function.

[0021] In a second aspect, the present application provides a speech emotion recognition device, comprising:

[0022] A text acquisition unit is configured to acquire original speech to be recognized and original text corresponding to the original speech.

[0023] A speech analysis unit is configured to analyze the original speech to acquire a plurality of speech features corresponding to the original speech, wherein the speech features at least include a mel-frequency spectrum, a mel-frequency cepstral coefficient, and a chroma vector.

[0024] A vector acquisition unit is configured to acquire a speech vector corresponding to each speech feature based on a self-attention mechanism.

[0025] An embedding acquisition unit is configured to analyze the original text to acquire text embedding information corresponding to the original text.

[0026] A vector output unit is configured to input the text embedding information to a preset text extraction model, and the text extraction model analyzes the text embedding information to output a text feature vector corresponding to the text embedding information.

[0027] A recognition completion unit is configured to generate a target emotion type corresponding to the original speech according to the text feature vector and each speech vector, and complete emotion recognition of the original speech.

[0028] In a third aspect, the present application further provides a computer device, comprising:

[0029] a memory and a processor;

[0030] the memory is configured to store a computer program;

[0031] the processor is configured to execute the computer program and implement the steps of the voice emotion recognition method according to the first aspect.

[0032] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, causes the processor to implement the steps of the voice emotion recognition method according to the first aspect.

[0033] The voice emotion recognition method, device, computer device and readable storage medium provided by the embodiments of the present application aim to improve the accuracy and transparency of emotion recognition by combining the information of text and voice modalities and introducing more refined feature extraction and vector generation techniques, solve the problem that the existing pre-training model based on Transformer is a "black box", and overcome the limitations in the prior art that the network cannot be trained layer by layer and it is difficult to understand how the network comes to a classification conclusion.

[0034] Firstly, the method prepares for the fusion of text and voice modalities by inputting the voice data to be recognized and converting it into text by ASR (Automatic Speech Recognition) technology.

[0035] Then, a plurality of voice features corresponding to the original voice are obtained: multi-dimensional feature extraction is performed on the voice data, at least including: Mel spectrum: used to represent the energy distribution of the voice signal at different frequencies, which is a commonly used feature in voice processing. Mel frequency cepstral coefficients (MFCCs): a feature that can capture the frequency domain characteristics and time domain characteristics of the voice signal, widely used in voice recognition and emotion analysis. Chroma vector: used to describe the pitch distribution in the audio signal, which can provide additional information about the emotional state of the speaker.

[0036] Then, a voice vector corresponding to each voice feature is obtained based on a self-attention mechanism, which allows the model to dynamically focus on the more critical part of the feature when processing the voice feature, thereby generating a more representative voice vector. This step is crucial for improving the accuracy of emotion recognition.

[0037] Further, the original text is parsed to obtain text embedding information corresponding to the original text: a pre-trained language model (such as BERT) is used to convert the text into high-dimensional embedding vectors, which can capture the context and emotional tendencies of the text.

[0038] Furthermore, the text embedding information is input into a pre-set text extraction model: this model further processes the text embedding information, aiming to extract the most important features for sentiment recognition from a large amount of text information, generating a text feature vector.

[0039] Finally, the target emotion type corresponding to the original speech is generated according to the text feature vector and each speech vector: finally, the text feature vector is fused with multiple speech vectors, and the trained classification model (such as support vector machine, neural network, etc.) is used to predict the emotional type of the original speech, completing the sentiment recognition process.

[0040] Furthermore, the method has at least the following beneficial effects:

[0041] 1. Improve the accuracy of sentiment recognition: by using multiple speech features and text information, this method can capture more dimensions of emotional expression, thereby improving the accuracy of sentiment recognition.

[0042] 2. Enhance the interpretability of the model: self-attention mechanism not only helps the model focus on more critical information, but also makes it possible to understand the decision-making process of the model. By tracking the contribution of different neuron modules, researchers can more clearly understand how each feature affects the final sentiment recognition result.

[0043] 3. Support layer-by-layer training and analysis of the network: unlike traditional methods, the process generated by this method allows layer-by-layer analysis of the network behavior during training, which is of great value for optimizing the model structure and understanding the mechanism behind sentiment recognition.

[0044] 4. Improve the credibility of the system: due to the enhanced interpretability of the model, users and researchers can better understand and verify the sentiment recognition results of the system, thereby improving the overall credibility of the system.

[0045] 5. More adaptable: the design of this method takes into account the fusion of text and speech modalities, making it more adaptable and generalizable in handling cross-modal sentiment recognition tasks, and can be applied to more diverse scenarios such as online customer service, smart home, mental health assessment, etc.

[0046] In summary, the speech emotion recognition method proposed in this application, through innovative technical means, not only breaks through in improving the accuracy of sentiment recognition, but also greatly enhances the transparency and interpretability of the model, which has important significance for promoting the development of speech emotion recognition technology.

[0047] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0049] Figure 1 is a step schematic flow chart of a voice emotion recognition method provided by an embodiment of the present application;

[0050] Figure 2 is a principle schematic diagram of a voice emotion recognition method provided by an embodiment of the present application;

[0051] Figure 3 is a step schematic flow chart of a voice vector acquisition method provided by an embodiment of the present application;

[0052] Figure 4 is a structure schematic diagram of a voice emotion recognition device provided by an embodiment of the present application;

[0053] Figure 5 is a structure schematic block diagram of a computer device provided by an embodiment of the present application.

[0054] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0056] The flow charts shown in the drawings are only exemplary and are not necessarily required to include all the contents and operations / steps, and are not necessarily executed in the described order. For example, some operations / steps can be decomposed, combined or partially merged, so that the actual execution order can be changed according to the actual situation.

[0057] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms of "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. The skilled in the art can understand that the terms of "first", "second", etc. do not limit the quantity and execution order, and the terms of "first", "second", etc. also do not necessarily mean different.

[0058] It should be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clear from the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0059] It should also be understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0060] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0061] Speech emotion recognition (SER) has a wide range of applications in the fields of robots, security, language translation, automatic identification, intelligent toys, lie detection, etc. Although there have been many developments in speech processing, natural emotion understanding is still a challenging task for computing systems.

[0062] Existing speech emotion recognition methods use Transformer-based pre-trained models to fuse information of text and speech modalities. However, these deep learning-based systems have always been in the role of a black box. The internal mechanism cannot be known. In order to study the internal mechanism, the prior art uses a method of decomposing the output prediction by tracking the contribution of all neuron modules by finding the input part responsible for a specific output. However, the above method cannot show the layer-by-layer training of the network and cannot know how the network comes to the classification conclusion during the training process. This makes the accuracy and reliability of the speech emotion recognition result not very ideal.

[0063] Therefore, there is an urgent need for a question and answer information generation method for multi-modal data to improve the performance of the intelligent customer service to which the method is applied.

[0064] To solve the above problems, please refer to Figure 1 , Figure 1This is a schematic flow chart of a speech emotion recognition method provided by one embodiment of the present application. This speech emotion recognition method can be implemented by a computer device, which can be deployed on a single server or a server cluster. Alternatively, it can be deployed on a handheld terminal, laptop computer, wearable device, or robot.

[0065] It should be noted that the acquisition of any information mentioned in the provided method complies with relevant regulations and is carried out with the user's consent, and will not infringe on the user's privacy or violate relevant laws and regulations.

[0066] At the same time, the method provided by this application can be applied to scenarios such as the financial field, where users need to communicate frequently. By obtaining the emotion corresponding to the user's current voice, the next communication content or service strategy can be intelligently determined. At the same time, the method provided in other scenarios that require communication with people, such as medical and education fields, can also achieve good results. Therefore, the financial field is only a specific example of this application, and the specific application scope of the method provided by this application is not limited.

[0067] To solve the above problems, please refer to Figure 1 Specifically, Figure 1 As shown, the provided speech emotion recognition method includes steps S101 to S106. The details are as follows:

[0068] Step S101: Acquire the original speech to be recognized and the original text corresponding to the original speech.

[0069] Specifically, the method can perform real-time emotion recognition on the original speech by reading the original speech from a recording device (such as a phone, microphone) or a file, and can also obtain the original speech uploaded offline. The embodiment of the present application does not limit the method for obtaining the original speech. At the same time, to ensure the quality of emotion extraction, background noise and interference can be filtered before recognition. By extracting the original text corresponding to the original speech, such as using models such as Deep Speech, wav2vec 2.0, and Google's Speech-to-Text API. These models usually use convolutional neural networks (CNN), recurrent neural networks (RNN), or Transformer architectures, and are trained through a large number of speech and text pairs to achieve high-accuracy speech-to-text conversion. Furthermore, by ensuring the integrity and accuracy of the input data, the foundation is laid for subsequent feature extraction and emotion recognition.

[0070] When the provided method is applied in a financial scenario, in a financial call center, the system automatically receives the customer's phone recording and converts the voice into text in real time. Figure 2As shown, when the customer says, "I feel great!" or "I am very dissatisfied with the speed of this transaction", the system can accurately obtain the text content of this sentence, providing data support for further sentiment analysis.

[0071] Step S102. Analyze the original speech and obtain a plurality of speech features corresponding to the original speech; the speech features at least include mel spectrum, mel frequency cepstral coefficient and chroma vector.

[0072] Specifically, the speech signal corresponding to the original speech is converted into a spectrum graph, and then processed by a mel filter bank to generate a mel spectrum graph. The mel spectrum graph can better capture the sensitivity of the human auditory system to frequency. At the same time, by performing logarithmic processing and discrete cosine transform (DCT) on the mel spectrum, the cepstral coefficient of the speech is extracted. The cepstral coefficient can effectively capture the spectral envelope of the speech and has good robustness to background noise. The speech signal is also converted into a chroma graph, and the chroma vector is used to capture the musical emotion. Although it is not as commonly used as the previous two features in speech emotion recognition, it can provide additional auxiliary information.

[0073] Through the extraction of multiple speech features, the information in the speech can be more comprehensively captured, and the accuracy of emotion recognition can be improved. Each feature has different representation capabilities, can be complementary, enhance the robustness of the system, and reduce misjudgment.

[0074] For example, in a financial scenario, by extracting speech features such as mel spectrum and MFCC, the customer's tone and intonation can be captured. For example, when the customer's tone becomes rapid or the pitch rises, these features can help the system identify that the customer may be feeling anxious or angry.

[0075] Step S103. Obtain a speech vector corresponding to each speech feature based on a self-attention mechanism.

[0076] Specifically, the self-attention mechanism is used to process each extracted speech feature to generate a corresponding speech vector. The self-attention mechanism can dynamically focus on important parts of each feature to generate a higher-dimensional feature representation. Common self-attention mechanism models include Transformer models such as BERT, etc. By applying the self-attention mechanism to each speech feature (such as mel spectrum, MFCC, chroma vector), the corresponding speech vector is generated. These vectors can capture the complex relationships between features and provide more rich information.

[0077] Furthermore, the self-attention mechanism can capture the complex relationships between speech features to generate more rich feature representations. It can also better understand the emotional information in the speech data, thereby improving the recognition accuracy and credibility of the system.

[0078] As in the financial scenario, when processing the voice data of a customer, the self-attention mechanism can better understand the customer's tone and emotional changes. For example, if the customer's tone suddenly becomes sharp during the call, the system can capture this change through the self-attention mechanism and more accurately identify the customer's emotional state.

[0079] In some embodiments, please refer to Figure 2 and Figure 3 As shown in Figure 3 , based on the self-attention mechanism, the voice vector corresponding to each voice feature is obtained, including steps S103a to S103b.

[0080] Step S103a. Each voice feature is input into a preset gated recurrent unit to output the dependency information corresponding to the voice feature.

[0081] GRU is a variant of recurrent neural network (RNN) for processing sequence data. GRU controls the transmission of information through a gating mechanism (update gate and reset gate), effectively capturing the dependency relationship in time series. In this step, each extracted voice feature (such as mel spectrum, MFCC, chroma vector) is input into a preset GRU model. The time series data of each voice feature (for example, each frame of MFCC feature) is input. The output contains a sequence of feature vectors containing time-dependent information. These feature vector sequences not only contain the feature information of each time point, but also contain the dependency relationship between the previous and subsequent time points.

[0082] Through GRU, the dependency relationship of voice features in time can be effectively captured, improving the richness and representation ability of the features. Compared with traditional RNN, GRU reduces the gradient vanishing problem through the gating mechanism, making the model more stable when processing long sequence data.

[0083] As in the financial scenario, in the conversation between the customer and the customer service, the change in the customer's emotion can be captured through GRU. For example, if the customer's tone is relatively calm in the first half of the call, and the tone becomes rapid in the second half, GRU can effectively capture this change and help the system more accurately identify the customer's emotional state.

[0084] Step S103b. The dependency information corresponding to each voice feature is input into a preset first attention module, and the first attention module outputs the voice vector corresponding to each voice feature based on the self-attention mechanism.

[0085] Attention mechanism is a model mechanism that helps the model focus on specific parts when processing sequence data, thereby improving the representation ability and classification accuracy of the model. In this step, the dependency information corresponding to each voice feature (i.e. the output of GRU) is input into a preset first attention module.

[0086] The self-attention mechanism generates a weight matrix by calculating the correlation between each feature vector. These weights can be used for weighted summation to generate the final feature vector. Input: sequence of feature vectors output by GRU. Output: integrated speech vector corresponding to each speech feature. These vectors not only contain temporal dependency information, but also strengthen the information of key parts through the self-attention mechanism.

[0087] The self-attention mechanism can strengthen the key information in the speech features and improve the representation ability of the features. It can also dynamically adjust the focus according to the input data, making the model more flexible in handling different emotional expressions.

[0088] For example, in a financial scenario, when processing customer speech data, the first attention module can dynamically focus on the key emotional parts of the speech. For example, if a customer's sentence has a particularly strong tone, the attention mechanism can give this part a higher weight, helping the system more accurately identify the customer's emotional state. Through the self-attention mechanism, the system can better assess the intensity of the customer's emotions. For example, "very angry" and "a little angry" can be distinguished by different attention weights, providing more accurate emotional labels.

[0089] In summary, through the method of steps S103a to S103b, the customer's emotional changes can be more accurately identified. For example, the system can determine in real time whether the customer is angry or disappointed, and adjust the customer service strategy in a timely manner. According to the customer's emotional state, the system can provide personalized service recommendations. For example, if the system detects that the customer's emotions are not good, it can recommend a cool-down period of service or directly transfer to a senior customer service personnel for handling. The method can provide more accurate and flexible emotional analysis capabilities in financial scenarios, thereby improving service quality and customer satisfaction and reducing potential risks.

[0090] For example, before inputting the dependency information corresponding to each speech feature into the preset first attention module, the method further includes: inputting the dependency information into a hyperbolic tangent activation unit, the hyperbolic tangent activation unit analyzes the dependency information, and outputs normalized information corresponding to each speech feature, so as to input the normalized information and the corresponding speech feature into the first attention module.

[0091] The hyperbolic tangent activation function (tanh) is a commonly used activation function that can compress input data to the range of -1 to 1, achieving normalization processing. The dependency information feature vector output from the gated recurrent unit (GRU). The normalized dependency information feature vector.

[0092] By inputting the dependency information feature vector output by the GRU into the hyperbolic tangent activation unit, normalization processing is performed through the tanh function. After normalization, the value of the feature vector will be compressed to a range such as -1 to 1, which helps to stabilize the calculation of the subsequent attention module. The normalized feature vector can better highlight the information of the key part, reduce the influence of extreme values, and improve the representation ability of the feature.

[0093] At the same time, the normalized dependency information feature vector and the original speech feature vector are input into the first attention module. The first attention module outputs the final speech vector corresponding to each speech feature based on the self-attention mechanism.

[0094] By normalizing the dependency information feature vector through the hyperbolic tangent activation unit to a range such as -1 to 1, the influence of extreme values on the model can be reduced, and the stability of the feature vector can be improved. The normalized feature vector can prevent the problem of gradient explosion in the subsequent attention module calculation, making the model training more stable.

[0095] At the same time, normalization processing can better highlight the key part of the speech feature, reduce the interference of irrelevant features, and improve the representation ability of the feature. The attention mechanism can dynamically adjust the focus according to the normalized feature vector, further enhancing the recognition of key information. By inputting the normalized dependency information feature vector and the original speech feature vector into the attention module, the emotional information can be more comprehensively captured.

[0096] For example, in a financial system, the combination of the present example can more accurately identify the emotional changes of customers. For example, if the customer's tone is particularly strong at a certain time, normalization processing and attention mechanism can better capture this key part, thereby more accurately identifying the emotional state of the customer. According to the emotional state of the customer, the system can provide personalized service recommendations. For example, if it is detected that the customer's emotion is very low, the system can recommend psychological counseling or emotional relief services. The emotional labels generated by the system are more accurate and can better reflect the overall satisfaction of the customer. For example, the combination of speech features and text features can more accurately assess the emotional intensity of the customer.

[0097] In summary, by adding the processing step of the hyperbolic tangent activation unit, as shown in Figure 2 the input linear layer and then inputting the tanh part, the dependency information feature vector is normalized, further improving the stability and representation ability of the speech feature. Combined with the attention module of the self-attention mechanism, the speech emotion of the customer can be more accurately identified and classified. This improvement has important application value in the financial scenario and can improve the service quality, customer satisfaction, and risk management ability of financial institutions.

[0098] It should be noted that, in some embodiments, Figure 2 As shown in the figure, in the speech emotion recognition module, a 128-dimensional Mel spectrum (m), a 40-dimensional Mel frequency cepstral coefficient mfcc (f), and a 12-dimensional chroma vector can be used as input (c). Each feature is subjected to the attention mechanism to extract the part with the most important emotional information. For the i-th input sample, the language feature first passes through the GRU layer, then passes through the tanh activation function to obtain the normalized importance weight of each feature. Finally, the speech vector is calculated based on the weight, which is the weighted sum of all speech features. The vector then passes through the dense layer to obtain the speech vector C of Chroma, Mel-Spectrogram and MFCC features respectively. i 、M i and F i .

[0099] Step S104: Parse the original text and obtain text embedding information corresponding to the original text.

[0100] Specifically, by preprocessing the original text, such as word segmentation, removal of stop words, stemming, etc., the cleanliness and standardization of the text data are ensured. The preprocessed text is converted into a high-dimensional vector using a text embedding model. Common text embedding models include: Word2Vec: Capturing the relationship between words through word vectors. GloVe: Generate word vectors through global word frequency statistics. BERT: Use a bidirectional Transformer model to generate context-related word vectors to better capture emotional vocabulary and contextual information. The corresponding text embedding method can be selected according to specific needs, and the embodiments of this application are not limited to this.

[0101] Text embedding can capture the semantic information in the text and improve the accuracy of sentiment recognition. Text embedding can better understand the emotional vocabulary and contextual information in the text, such as "dissatisfaction" and "anger".

[0102] For example, in financial scenarios, the system can extract keywords and phrases such as "dissatisfied," "complaint," and "refund" from customer call texts, and convert these words into high-dimensional vectors through text embedding models to help the system more accurately identify the customer's emotional state.

[0103] Step S105: Input the text embedding information into a preset text extraction model. The text extraction model parses the text embedding information and outputs a text feature vector corresponding to the text embedding information.

[0104] Specifically, the text embedding information generated in step S104 is input into a preset text extraction model. Common text extraction models include: LSTM (Long Short-Term Memory Network): Capture long dependencies in the text through memory units. GRU (Gated Recurrent Units): Similar to LSTM, but with fewer parameters and higher computational efficiency. Transformer: Capture complex contextual information in the text through self-attention mechanism and multi-head attention mechanism. The embodiment of the present application does not limit the method of text extraction. The text embedding information is parsed by the text extraction model to generate higher-level text feature vectors. These feature vectors can capture the sentiment, semantics and structural information in the text.

[0105] Through multi-layer processing, we can better capture the deep features in the text and improve the accuracy of sentiment recognition. The text extraction model can handle complex text structures, better understand the context of sentiment words, and reduce misjudgment.

[0106] In financial scenarios, for example, text extraction models can more accurately capture the content and tone of customer complaints. For example, if a customer texts a call saying, "I'm very disappointed with your service. This transaction was processed too slowly," a text extraction model can generate a corresponding feature vector to help assess the customer's emotional state as "angry" or "dissatisfied."

[0107] Step S106: Generate the target emotion type corresponding to the original speech according to the text feature vector and each speech vector, and complete the emotion recognition of the original speech.

[0108] By fusing the text feature vector generated in step S105 and each speech vector generated in step S103. Common fusion methods include: splicing: splicing all feature vectors into a higher dimensional vector through full connection. Weighted summation: weighted summation of each feature vector according to its importance. Multimodal fusion network: use a special multimodal fusion network (such as multimodal Transformer) to perform feature fusion. Use a classifier to perform sentiment classification on the fused feature vectors. Common classifiers include: support vector machine (SVM): classification by maximizing the interval. Decision tree: classification through a tree structure. Neural network: such as multi-layer perceptron (MLP), convolutional neural network (CNN) or Transformer, etc. According to the classification results, the target emotion type corresponding to the original speech is output to complete the emotion recognition task. Common target emotion types include anger, happiness, sadness, excitement, etc.

[0109] By fusing multi-modal features, the emotional state can be more comprehensively evaluated, and the accuracy and reliability of recognition can be improved. This method is suitable for various emotion recognition tasks, such as emotion classification, emotion intensity evaluation, etc.

[0110] As in the financial scenario, in the financial call center, the emotional state of the customer can be comprehensively judged according to the voice and text features of the customer. For example, the voice features of the customer show a rapid tone, and the text features show a large number of negative words. The system can integrate these information to judge the emotional state of the customer as "angry" or "very dissatisfied", so as to provide more personalized services and solutions. Financial institutions can monitor the service quality of customer service personnel through the system and timely discover and handle the emotional problems of customers. For example, if the system equipped with the provided method detects that the service of a certain customer service personnel has caused customer dissatisfaction several times, further training or adjustment can be made to the customer service personnel.

[0111] In summary, the voice emotion recognition method can more comprehensively and accurately recognize emotional information in voice by extracting and fusing multi-modal features, combining self-attention mechanism and text extraction model. In the financial scenario, this method can help financial institutions better understand the emotional needs of customers, provide higher quality services, and improve customer satisfaction and loyalty. Through detailed steps and technical content, this method can handle various complex emotion recognition tasks and ensure efficient and accurate emotion analysis in the financial scenario.

[0112] In some embodiments, the number of text embedding information is multiple; the text extraction model includes a second attention module and a hidden module; the text embedding information is input into the preset text extraction model, and the text extraction model analyzes the text embedding information to output the text feature vector corresponding to the text embedding information, including: inputting each text embedding information into the second attention module and the hidden module in turn to obtain the hidden vector corresponding to each text embedding information; and obtaining the text feature vector according to the multiple hidden vectors.

[0113] In step S104, the original text is converted into multiple text embedding information. Each text embedding information can correspond to different sentence fragments or contexts. Assuming that the original text of the customer is: "I am very dissatisfied with the processing speed of this transaction, and your service attitude is also very poor.", the system may split this text into multiple fragments, such as "I am very dissatisfied with the processing speed of this transaction" and "your service attitude is also very poor", and generate text embedding information for each fragment respectively.

[0114] The second attention module is used to capture the key parts of each text embedding information and generate attention weights. This helps the system focus more accurately on the parts of the text that express emotions. The hidden module is used to generate a hidden vector for each text embedding information. Common hidden modules include multi-layer neural networks (such as MLP) or recurrent neural networks (such as LSTM, GRU).

[0115] Each text embedding information generated in step S104 is sequentially input into the second attention module and the hidden module to obtain the corresponding hidden vector of each text embedding information. The second attention module calculates the attention weight of each text embedding information. The hidden module generates a hidden vector for each text embedding information according to the attention weight. The multiple hidden vectors are fused to generate a final text feature vector. This step fuses the information of all text segments to generate a comprehensive text feature vector.

[0116] The fusion method can use a special multi-modal fusion network (such as multi-modal Transformer) for feature fusion. The final text feature vector is used for subsequent sentiment recognition tasks.

[0117] In summary, by increasing the processing steps of multiple text embedding information and introducing the second attention module and hidden module, the comprehensiveness and representation ability of the text feature are further improved. This improvement enables the system to more accurately identify and classify customer sentiment information, thereby providing higher quality services and more effective risk management in financial scenarios. The specific technical content and steps ensure the robustness and flexibility of the system, suitable for a variety of complex sentiment recognition tasks.

[0118] As shown in Figure 2 , the text extraction model is a BERT model with 12 Transformer modules, 12 attention heads, and 768 hidden units. For the i-th embedding E i Generate the encoded hidden vector R_i. The encoding of the special tokens [CLS:Classification] and [SEP:Separator] in the BERT model are C1 and C2.{x i,1 ,x i,2 …x i,T} are the input tokens of the i-th data point at T time steps, B i is the text feature vector extracted by the BERT module for the i-th input.

[0119] In some embodiments, the target emotion type is any one of a plurality of preset emotion types; after completing emotion recognition on the original speech, further comprising: determining each emotion type in the plurality of emotion types other than the target emotion type as a verification emotion type; respectively obtaining intersection information corresponding to the target emotion type and each verification emotion type; and if the target emotion type is different from each verification emotion type according to the intersection information, completing verification of the emotion recognition.

[0120] The target emotion type is the result output by the emotion recognition model after completing emotion recognition, and can be one of a plurality of preset emotion types, such as "angry", "unhappy", "happy", "satisfied", etc. Assuming that the preset emotion types are "angry", "unhappy", "happy", "satisfied", and "neutral", the final target emotion type output by the emotion recognition model is "angry". After completing emotion recognition, each emotion type in the plurality of preset emotion types other than the target emotion type is determined as a verification emotion type. If the target emotion type is "angry", the verification emotion types include "unhappy", "happy", "satisfied", and "neutral". The intersection information of the target emotion type and each verification emotion type is respectively obtained. The intersection information can be obtained by various methods, such as feature similarity calculation, emotion vocabulary overlap calculation, etc.

[0121] For example, the feature similarity is to calculate the similarity between the feature vectors of the target emotion type and the verification emotion type, for example, using cosine similarity or Euclidean distance. The emotion vocabulary overlap is to calculate the overlapping part of the target emotion type and the verification emotion type in the emotion vocabulary table, for example, using Jaccard similarity.

[0122] According to the obtained intersection information, it is determined whether the target emotion type is different from each verification emotion type. If the intersection information of the target emotion type and all verification emotion types indicates that they are different emotion types, the verification of the emotion recognition result is completed. For example, a similarity threshold is set, and if the similarity of the target emotion type and any verification emotion type exceeds the threshold, it is considered that the emotion recognition result may be inaccurate and needs to be further verified or adjusted. A threshold of emotion vocabulary overlap is set, and if the vocabulary overlap of the target emotion type and any verification emotion type exceeds the threshold, it is considered that the emotion recognition result may be inaccurate and needs to be further verified or adjusted.

[0123] As in the financial scenario, after identifying the customer's emotion, the recognition result is further confirmed by comparing with the intersection information of multiple verification emotion types. For example, if the target emotion type is "angry", the system will compare with "unhappy", "happy", "satisfied", "neutral" to ensure the accuracy of the recognition result. If the recognition result is found to be inaccurate, the service strategy can be adjusted in real time. For example, if the customer is mistakenly recognized as "happy" but the actual customer emotion is "unhappy", the system can timely correct and provide targeted service suggestions.

[0124] In summary, by introducing the intersection information comparison between the target emotion type and the verification emotion type, the accuracy of emotion recognition and the reliability of the system are further improved. This verification mechanism is particularly important in the financial scenario, which can help financial institutions better understand the emotional state of customers, thereby providing higher quality services and more effective risk management. The specific technical content and steps ensure the effectiveness and efficiency of the verification process, which is suitable for a variety of complex emotion recognition tasks.

[0125] For example, the intersection information corresponding to the target emotion type and each verification emotion type is obtained respectively, including: obtaining the principal component information corresponding to the emotion type; calculating the average value information and the standard deviation information corresponding to the principal component information; obtaining the first emotion component corresponding to the emotion category according to the average value information and the standard deviation information; obtaining the second emotion component corresponding to each verification emotion type; and calculating the intersection information of the first emotion component and each second emotion component respectively.

[0126] Principal component analysis (PCA) is performed on the feature vectors of the target emotion type and each verification emotion type to extract the principal component information. Principal component analysis can help reduce the feature dimension while retaining the most important information. The average value and standard deviation of the principal component information of each emotion type are calculated. The average value information reflects the main features of the emotion type, and the standard deviation information reflects the dispersion degree of the data distribution of the emotion type.

[0127] According to the average value information and the standard deviation information of the target emotion type, the first emotion component is generated. The first emotion component is a quantitative representation of the main emotional features of the target emotion type. According to the average value information and the standard deviation information of each verification emotion type, the second emotion component is generated. The second emotion component is a quantitative representation of the main emotional features of the verification emotion type. The intersection information between the first emotion component and each second emotion component is calculated. The intersection information can be used to determine whether the target emotion type and the verification emotion type are different.

[0128] The intersection information can be calculated by, for example, similarity calculation: calculating the similarity between the first emotion component and each second emotion component, such as using cosine similarity or Euclidean distance. Or statistical intersection: calculating the intersection part of the first emotion component and each second emotion component in the emotional features.

[0129] After identifying a customer's emotion, principal component analysis is used to extract key emotional features. The primary and secondary emotional components are then generated, and similarity and intersection information is calculated. For example, if the target emotion type is "anger," the system compares it with validation emotion types such as "dissatisfaction," "happiness," and "satisfaction" to ensure the accuracy of the recognition result. With more accurate emotion recognition results, the system can provide more personalized service recommendations and improve customer satisfaction. For example, if the system confirms that the customer's emotion is "anger," it can recommend that the customer service representative provide a quick solution and apologize.

[0130] By introducing principal component analysis, average and standard deviation information, and the calculation of the first and second emotion components, the accuracy and reliability of emotion recognition results are further improved. This improved method is particularly important in financial scenarios, helping financial institutions better understand their customers' emotional states, thereby providing higher-quality services and more effective risk management. The specific technical content and steps ensure the effectiveness and efficiency of the verification process, and are applicable to a variety of complex emotion recognition tasks.

[0131] It should be noted that, in some embodiments, in order to explore the prediction mechanism of the proposed model, Figure 2 The following steps are performed to analyze the last few layers of the multimodal emotion recognition module (composed of the speech emotion recognition module and the text emotion recognition module). First, for each emotion category i, we calculate the principal component (x i ,y i ,z i ). For each emotion category i and the corresponding (x i ,y i ,z i ) components, calculate the mean and Standard deviation

[0132]

[0133] where n i is the number of samples in the i-th category, and k represents the k-th data point.

[0134] Set L i (m) and R i (m) is the component of the mth emotion in the left and right extreme points of the i-th emotion category, and the formula is defined as follows:

[0135]

[0136] In addition, I i，j (m) is used to calculate the intersection between sentiment categories, which represents the degree of intersection between categories i and j:

[0137]

[0138] Then calculate the product I of all component crosses between sentiment categories i and j i,j , as shown in the formula. i,j The value of is in the range [0,1].

[0139] I i,j =I i,j (x)*I i,j (y)*I i,j (z);

[0140] Finally, define a 4×4 matrix as I. The (i, j)th element in I represents the total intersection degree I of emotion i and emotion j. i,j , I i,j The smaller the value of , the better the elements of the i-th and j-th emotions are divided into different emotion clusters. The analysis of the model output layer explains the mechanism of the model's internal output results and improves the credibility of the emotion recognition results.

[0141] In some embodiments, generating a target emotion type corresponding to the original speech based on the text feature vector and each speech vector includes: fully connecting the text feature vector and each speech vector to obtain a concatenated vector; calculating the target emotion type based on a preset prediction function; the expression corresponding to the prediction function includes:

[0142] y e =softmax(W T E i +b);

[0143] Among them, e is the target emotion type, y e is the prediction of the target emotion type, W is the weight matrix, b represents the bias term, E i represents the concatenated vector, and softmax() is the prediction function.

[0144] like Figure 2 As shown, the four vectors obtained in the speech emotion recognition stage are connected (C i ,M i ,F i ,B i ), and obtain E through the fully connected layer i . The final vector E i Pass it to the softmax layer and get the prediction result shown in the above formula, such as Figure 2 The result shown is "Happy". You can use sparse categorical cross entropy as the loss function and Adam as the optimizer. In each iteration, the model is updated based on the output of the loss function.

[0145] See also Figure 4 As shown, Figure 4 2 is a schematic diagram of the structure of a speech emotion recognition device 200 provided in an embodiment of the present application. The speech emotion recognition device 200 is used to perform the steps of the speech emotion recognition method shown in the above embodiments. The speech emotion recognition device 200 can be a single server or a server cluster, or the speech emotion recognition device 200 can be a terminal, such as a handheld terminal, a laptop computer, a wearable device, or a robot.

[0146] like Figure 4 As shown, the speech emotion recognition device 200 includes:

[0147] The text acquisition unit 201 is used to acquire the original speech to be recognized and the original text corresponding to the original speech;

[0148] The speech analysis unit 202 is used to analyze the original speech and obtain multiple speech features corresponding to the original speech; the speech features include at least Mel spectrum, Mel frequency cepstral coefficients and chrominance vector;

[0149] A vector acquisition unit 203 is configured to acquire a speech vector corresponding to each speech feature based on a self-attention mechanism;

[0150] The embedding acquisition unit 204 is used to parse the original text and obtain the text embedding information corresponding to the original text;

[0151] The vector output unit 205 is used to input the text embedding information into a preset text extraction model, and the text extraction model parses the text embedding information and outputs a text feature vector corresponding to the text embedding information;

[0152] The recognition completion unit 206 is used to generate a target emotion type corresponding to the original speech according to the text feature vector and each speech vector, and complete the emotion recognition of the original speech.

[0153] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the speech emotion recognition device and each module described above can refer to the corresponding processes in the speech emotion recognition method embodiments described in the above embodiments, and will not be repeated here.

[0154] The above-mentioned speech emotion recognition method can be implemented in the form of a computer program. The computer program can be used in Figure 4 Run on the device shown.

[0155] See also Figure 5 , Figure 51 is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.

[0156] The storage medium can store an operating device and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the speech emotion recognition methods.

[0157] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0158] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any speech emotion recognition method.

[0159] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the terminal to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0160] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0161] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0162] Obtaining the original speech to be recognized and the original text corresponding to the original speech;

[0163] Parse the original speech to obtain multiple speech features corresponding to the original speech; the speech features include at least Mel spectrum, Mel frequency cepstral coefficients and chrominance vector;

[0164] Based on the self-attention mechanism, the speech vector corresponding to each speech feature is obtained respectively;

[0165] Parse the original text and obtain the text embedding information corresponding to the original text;

[0166] The text embedding information is input into a preset text extraction model, which parses the text embedding information and outputs a text feature vector corresponding to the text embedding information;

[0167] The target emotion type corresponding to the original speech is generated according to the text feature vector and each speech vector, completing the emotion recognition of the original speech.

[0168] In some embodiments, the speech vector corresponding to each speech feature is obtained based on the self-attention mechanism, including: inputting each speech feature into a preset gated recurrent unit to output dependency information corresponding to the speech feature; inputting the dependency information corresponding to each speech feature into a preset first attention module, and the first attention module outputs the speech vector corresponding to each speech feature based on the self-attention mechanism.

[0169] Exemplarily, before inputting the dependency information corresponding to each speech feature into the preset first attention module, it also includes: inputting the dependency information into a hyperbolic tangent activation unit, the hyperbolic tangent activation unit parses the dependency information, and outputs the normalization information corresponding to each speech feature, so as to input the normalization information and the corresponding speech feature into the first attention module.

[0170] In some embodiments, there are multiple text embedding information; the text extraction model includes a second attention module and a hidden module; the text embedding information is input into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information, including: inputting each text embedding information into the second attention module and the hidden module in turn, obtaining a hidden vector corresponding to each text embedding information; obtaining a text feature vector based on multiple hidden vectors.

[0171] In some embodiments, the target emotion type is any one of a plurality of preset emotion types; after completing the emotion recognition of the original speech, it also includes: determining each emotion type except the target emotion type in the plurality of emotion types as a verification emotion type; obtaining the intersection information corresponding to the target emotion type and each verification emotion type respectively; if it is determined according to the intersection information that the target emotion type is different from each verification emotion type, the verification of the emotion recognition is completed.

[0172] Exemplarily, the intersection information corresponding to the target emotion type and each verified emotion type is obtained respectively, including: obtaining the principal component information corresponding to the emotion type; calculating the mean value information and standard deviation information corresponding to the principal component information; obtaining the first emotion component corresponding to the emotion category according to the mean value information and the standard deviation information; obtaining the second emotion component corresponding to each verified emotion type; and respectively calculating the intersection information of the first emotion component and each second emotion component.

[0173] In some embodiments, generating a target emotion type corresponding to the original speech based on the text feature vector and each speech vector includes: fully connecting the text feature vector and each speech vector to obtain a concatenated vector; calculating the target emotion type based on a preset prediction function; the expression corresponding to the prediction function includes:

[0174] y e =softmax(W T E i +b);

[0175] Among them, e is the target emotion type, y e is the prediction of the target emotion type, W is the weight matrix, b represents the bias term, E i represents the concatenated vector, and softmax() is the prediction function.

[0176] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the speech emotion recognition method as described in the first aspect above.

[0177] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.

[0178] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A speech emotion recognition method, characterized in that: include: Obtaining an original speech to be recognized and an original text corresponding to the original speech; Analyzing the original speech to obtain multiple speech features corresponding to the original speech; The speech features include at least Mel spectrum, Mel frequency cepstral coefficients and chromaticity vector; Based on the self-attention mechanism, the speech vector corresponding to each speech feature is obtained respectively, including: inputting each speech feature into a preset gated recurrent unit to output dependency information corresponding to the speech feature; inputting the dependency information corresponding to each speech feature into a preset first attention module, the first attention module outputting the speech vector corresponding to each speech feature based on the self-attention mechanism, and before inputting the dependency information corresponding to each speech feature into the preset first attention module, further comprising: inputting the dependency information into a hyperbolic tangent activation unit, the hyperbolic tangent activation unit parsing the dependency information and outputting normalized information corresponding to each speech feature, so as to input the normalized information and the corresponding speech feature into the first attention module; Parsing the original text to obtain text embedding information corresponding to the original text; Inputting the text embedding information into a preset text extraction model, the text extraction model parses the text embedding information and outputs a text feature vector corresponding to the text embedding information; A target emotion type corresponding to the original speech is generated according to the text feature vector and each of the speech vectors, thereby completing emotion recognition of the original speech.

2. The method according to claim 1, characterized in that There are multiple text embedding information; the text extraction model includes a second attention module and a hidden module; the text embedding information is input into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information, including: Input each of the text embedding information into the second attention module and the hidden module in sequence to obtain the hidden vector corresponding to each of the text embedding information; The text feature vector is obtained according to a plurality of hidden vectors.

3. The method according to claim 1, characterized in that The target emotion type is any one of a plurality of preset emotion types; after completing the emotion recognition of the original speech, the method further includes: Determining each of the plurality of emotion types except the target emotion type as a verification emotion type; Respectively obtain intersection information corresponding to the target emotion type and each of the verification emotion types; If it is determined according to the intersection information that the target emotion type is different from each of the verification emotion types, verification of the emotion recognition is completed.

4. The method according to claim 3, characterized in that The obtaining of intersection information corresponding to the target emotion type and each verification emotion type includes: Obtaining principal component information corresponding to the target emotion type; Calculate the mean value and standard deviation of the principal component information. Acquire a first emotion component corresponding to the target emotion type according to the average value information and the standard deviation information; Obtaining a second emotion component corresponding to each of the verified emotion types; The intersection information of the first emotion component and each of the second emotion components is calculated respectively.

5. The method according to claim 1, wherein Generating the target emotion type corresponding to the original speech according to the text feature vector and each of the speech vectors includes: Fully connect the text feature vector and each of the speech vectors to obtain a concatenated vector; The target emotion type is calculated according to a preset prediction function; the expression corresponding to the prediction function includes: ； in, is the target emotion type, To predict the target emotion type, is the weight matrix, represents the deviation term, represents the splicing vector, is the prediction function.

6. A question-answer information generating device, characterized in that: include: A text acquisition unit, configured to acquire an original speech to be recognized and an original text corresponding to the original speech; A speech analysis unit, configured to analyze the original speech and obtain a plurality of speech features corresponding to the original speech; the speech features at least including a mel spectrum, a mel frequency cepstral coefficient, and a chroma vector; A vector acquisition unit, configured to acquire the speech vector corresponding to each of the speech features based on a self-attention mechanism, comprising: inputting each of the speech features into a preset gated recurrent unit to output dependency information corresponding to the speech feature; inputting the dependency information corresponding to each of the speech features into a preset first attention module, the first attention module outputting the speech vector corresponding to each of the speech features based on the self-attention mechanism; and before inputting the dependency information corresponding to each of the speech features into the preset first attention module, further comprising: inputting the dependency information into a hyperbolic tangent activation unit, the hyperbolic tangent activation unit parsing the dependency information and outputting normalized information corresponding to each of the speech features, so as to input the normalized information and the corresponding speech features into the first attention module; an embedding acquisition unit, configured to parse the original text and acquire text embedding information corresponding to the original text; A vector output unit, configured to input the text embedding information into a preset text extraction model, wherein the text extraction model parses the text embedding information and outputs a text feature vector corresponding to the text embedding information; The recognition completion unit is used to generate a target emotion type corresponding to the original speech according to the text feature vector and each of the speech vectors, and complete the emotion recognition of the original speech.

7. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 5.

Images