Speech emotion recognition method and device, computer equipment and readable storage medium

Abstract

The invention provides a speech emotion recognition method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring a to-be-recognized original speech and a corresponding original text; analyzing the original voice, and obtaining a plurality of voice features corresponding to the original voice; the voice features at least comprise a Mel frequency spectrum, a Mel frequency cepstrum coefficient and a chrominance vector; a voice vector corresponding to each voice feature is acquired based on a self-attention mechanism; analyzing the original text to obtain text embedding information corresponding to the original text; inputting the text embedded information into a preset text extraction model, analyzing the text embedded information by the text extraction model, and outputting a text feature vector corresponding to the text embedded information; and generating a target emotion type corresponding to the original voice according to the text feature vector and each voice vector, and completing emotion recognition of the original voice. The multi-mode emotion recognition method is provided, emotion recognition is carried out by combining contents of a text side and a voice side, and the emotion recognition accuracy is improved.

Description

Technical Field

[0001] The present application relates to the technical field of semantic analysis, and in particular to a speech emotion recognition method, apparatus, computer equipment and readable storage medium. Background Art

[0002] Speech emotion recognition (SER) has a wide range of applications in robotics, security, language translation, automatic identification, smart toys, lie detection, etc. Despite many advances in speech processing, natural emotion understanding remains a challenging task for computing systems.

[0003] Existing speech emotion recognition methods use Transformer-based pre-trained models to fuse information from text and speech modalities. However, these deep learning-based systems have always acted as black boxes. Their internal mechanisms are unknown. In order to study the internal mechanisms, existing technologies use methods to decompose output predictions by tracking the contributions of all neuron modules by finding the input part responsible for a specific output. However, the above methods cannot show the layer-by-layer training of the network, and cannot know how the network draws classification conclusions during the training process. This makes the accuracy and credibility of speech emotion recognition results less than ideal.

[0004] Therefore, there is an urgent need for a question-and-answer information generation method for multimodal data to improve the performance of the intelligent customer service to which the method is applied. Summary of the invention

[0005] The present application provides a speech emotion recognition method, apparatus, computer equipment and readable storage medium, aiming to solve the problem that the existing speech emotion recognition method uses a Transformer-based pre-training model to fuse information from text and speech modalities. However, these deep learning-based systems have always acted as black boxes. It is impossible to know their internal mechanisms. In order to study the internal mechanisms, the prior art uses a method of decomposing output predictions by tracking the contributions of all neuron modules by finding the input part responsible for a specific output. However, the above methods cannot display the layer-by-layer training of the network, and it is impossible to know how the network draws classification conclusions during the training process. This makes the accuracy and credibility of speech emotion recognition results less than ideal.

[0006] In a first aspect, the present application provides a method for speech emotion recognition, comprising:

[0007] Obtaining the original speech to be recognized and the original text corresponding to the original speech;

[0008] Parse the original speech to obtain multiple speech features corresponding to the original speech; the speech features at least include Mel spectrum, Mel frequency cepstral coefficients and chromaticity vector;

[0009] Based on the self-attention mechanism, the speech vector corresponding to each speech feature is obtained respectively;

[0010] Parse the original text and obtain the text embedding information corresponding to the original text;

[0011] The text embedding information is input into a preset text extraction model, and the text extraction model parses the text embedding information and outputs a text feature vector corresponding to the text embedding information;

[0012] The target emotion type corresponding to the original speech is generated according to the text feature vector and each speech vector, and the emotion recognition of the original speech is completed.

[0013] In some embodiments, a speech vector corresponding to each speech feature is obtained based on a self-attention mechanism, including: inputting each speech feature into a preset gated recurrent unit to output dependency information corresponding to the speech feature; inputting the dependency information corresponding to each speech feature into a preset first attention module, and the first attention module outputs a speech vector corresponding to each speech feature based on the self-attention mechanism.

[0014] Exemplarily, before inputting the dependency information corresponding to each speech feature into a preset first attention module, it also includes: inputting the dependency information into a hyperbolic tangent activation unit, the hyperbolic tangent activation unit parses the dependency information, and outputs the normalization information corresponding to each speech feature, so as to input the normalization information and the corresponding speech feature into the first attention module.

[0015] In some embodiments, there are multiple text embedding information; the text extraction model includes a second attention module and a hidden module; the text embedding information is input into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information, including: inputting each text embedding information into the second attention module and the hidden module in turn, and obtaining a hidden vector corresponding to each text embedding information; obtaining a text feature vector based on multiple hidden vectors.

[0016] In some embodiments, the target emotion type is any one of a plurality of preset emotion types; after completing the emotion recognition of the original speech, it also includes: determining each emotion type except the target emotion type among the plurality of emotion types as a verification emotion type; respectively obtaining the intersection information corresponding to the target emotion type and each verification emotion type; if it is determined according to the intersection information that the target emotion type is different from each verification emotion type, the verification of the emotion recognition is completed.

[0017] Exemplarily, the intersection information corresponding to the target emotion type and each verified emotion type is obtained respectively, including: obtaining the principal component information corresponding to the emotion type; calculating the mean value information and standard deviation information corresponding to the principal component information; obtaining the first emotion component corresponding to the emotion category according to the mean value information and the standard deviation information; obtaining the second emotion component corresponding to each verified emotion type; and respectively calculating the intersection information of the first emotion component and each second emotion component.

[0018] In some embodiments, generating a target emotion type corresponding to the original speech according to the text feature vector and each speech vector includes: fully connecting the text feature vector and each speech vector to obtain a concatenated vector; calculating the target emotion type according to a preset prediction function; the expression corresponding to the prediction function includes:

[0019] y e =softmax(W T E i + b);

[0020] Among them, e is the target emotion type, y e is the prediction of the target sentiment type, W is the weight matrix, b represents the bias term, E i represents the concatenated vector, and softmax() is the prediction function.

[0021] In a second aspect, the present application provides a speech emotion recognition device, comprising:

[0022] A text acquisition unit, used to acquire the original speech to be recognized and the original text corresponding to the original speech;

[0023] A speech analysis unit, used to analyze the original speech and obtain multiple speech features corresponding to the original speech; the speech features at least include Mel spectrum, Mel frequency cepstral coefficients and chromaticity vector;

[0024] A vector acquisition unit, used to acquire the speech vector corresponding to each speech feature based on a self-attention mechanism;

[0025] An embedding acquisition unit, used to parse the original text and acquire text embedding information corresponding to the original text;

[0026] A vector output unit, used to input the text embedding information into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information;

[0027] The recognition completion unit is used to generate the target emotion type corresponding to the original speech according to the text feature vector and each speech vector, and complete the emotion recognition of the original speech.

[0028] In a third aspect, the present application further provides a computer device, including:

[0029] Memory and processor;

[0030] The memory is used to store computer programs;

[0031] The processor is used to execute the computer program and implement the steps of the speech emotion recognition method as described in the first aspect above when executing the computer program.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the speech emotion recognition method as described in the first aspect above.

[0033] The embodiments of the present application provide a speech emotion recognition method, apparatus, computer device and readable storage medium, which aim to improve the accuracy and transparency of emotion recognition by combining information from text and speech modalities and introducing more sophisticated feature extraction and vector generation techniques on this basis, thereby solving the problem that the existing Transformer-based pre-trained model exists as a "black box", and at the same time overcoming the limitations of the prior art that it is impossible to display the network training layer by layer and it is difficult to understand how the network specifically reaches a classification conclusion.

[0034] Firstly, the method inputs the speech data to be recognized and converts it into text by ASR (Automatic Speech Recognition) technology, so as to prepare for the subsequent fusion of text and speech modality information.

[0035] Then, multiple speech features corresponding to the original speech are obtained: multi-dimensional feature extraction is performed on the speech data, including at least: Mel spectrum: used to represent the energy distribution of speech signals at different frequencies, which is a commonly used feature in speech processing. Mel frequency cepstral coefficients (MFCCs): a feature that can capture the frequency domain characteristics and time domain characteristics of speech signals, widely used in speech recognition and sentiment analysis. Chroma vector: used to describe the distribution of pitches in audio signals, which can provide additional information about the speaker's emotional state.

[0036] Then, based on the self-attention mechanism, the speech vector corresponding to each speech feature is obtained. The self-attention mechanism allows the model to dynamically focus on the more critical parts of the features when processing speech features, thereby generating more representative speech vectors. This step is crucial to improving the accuracy of emotion recognition.

[0037] Furthermore, the original text is parsed to obtain the text embedding information corresponding to the original text: using pre-trained language models (such as BERT, etc.), the text is converted into high-dimensional embedding vectors that can capture the contextual meaning and emotional tendency of the text.

[0038] Furthermore, the text embedding information is input into a preset text extraction model: this model further processes the text embedding information, aiming to extract the most important features for sentiment recognition from a large amount of text information and generate a text feature vector.

[0039] Finally, the target emotion type corresponding to the original speech is generated based on the text feature vector and each speech vector: Finally, the text feature vector is fused with multiple speech vectors, and the emotion type of the original speech is predicted through the trained classification model (such as support vector machine, neural network, etc.) to complete the emotion recognition process.

[0040] The method provided thereby has at least the following beneficial effects:

[0041] 1. Improve the accuracy of emotion recognition: By utilizing multiple speech features and text information, this method can capture more dimensions of emotional expression, thereby improving the accuracy of emotion recognition.

[0042] 2. Enhance the interpretability of the model: The self-attention mechanism not only helps the model focus on more critical information, but also makes it possible to understand the decision-making process of the model. By tracking the contribution of different neuron modules, researchers can more clearly understand how each feature affects the final emotion recognition results.

[0043] 3. Support layer-by-layer training and analysis of the network: Unlike traditional methods, the process generated by this method allows the behavior of the network to be analyzed layer by layer during training, which is of great value for optimizing the model structure and understanding the mechanism behind emotion recognition.

[0044] 4. Improve the credibility of the system: Due to the enhanced interpretability of the model, users and researchers can better understand and verify the emotion recognition results of the system, thereby improving the overall credibility of the system.

[0045] 5. Stronger adaptability: The design of this method takes into account the fusion of text and speech modalities, which makes it more adaptable and generalizable when dealing with cross-modal emotion recognition tasks, and can be applied to more diversified scenarios, such as online customer service, smart home, mental health assessment, etc.

[0046] In summary, the speech emotion recognition method proposed in this application, through innovative technical means, has not only made a breakthrough in improving the accuracy of emotion recognition, but also greatly enhanced the transparency and interpretability of the model, which is of great significance for promoting the development of speech emotion recognition technology.

[0047] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 This is a flowchart showing the steps of a method for speech emotion recognition provided by an embodiment of the present application;

[0050] Figure 2 It is a schematic diagram of the principle of a speech emotion recognition method provided by an embodiment of the present application;

[0051] Figure 3 This is a flowchart showing the steps of a method for obtaining a speech vector provided by an embodiment of the present application;

[0052] Figure 4 is a structural diagram of a speech emotion recognition device provided in an embodiment of the present application;

[0053] Figure 5 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.

[0054] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0057] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0058] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0059] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0060] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0061] Speech emotion recognition (SER) has a wide range of applications in robotics, security, language translation, automatic identification, smart toys, lie detection, etc. Despite many advances in speech processing, natural emotion understanding remains a challenging task for computing systems.

[0062] Existing speech emotion recognition methods use Transformer-based pre-trained models to fuse information from text and speech modalities. However, these deep learning-based systems have always acted as black boxes. Their internal mechanisms are unknown. In order to study the internal mechanisms, existing technologies use methods to decompose output predictions by tracking the contributions of all neuron modules by finding the input part responsible for a specific output. However, the above methods cannot show the layer-by-layer training of the network, and cannot know how the network draws classification conclusions during the training process. This makes the accuracy and credibility of speech emotion recognition results less than ideal.

[0063] Therefore, there is an urgent need for a question-and-answer information generation method for multimodal data to improve the performance of the intelligent customer service to which the method is applied.

[0064] To solve the above problems, please refer to Figure 1 , Figure 11 is a schematic flow chart of a speech emotion recognition method provided by an embodiment of the present application. The speech emotion recognition method can be implemented by a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0065] It should be noted that the acquisition of any information mentioned in the provided method is in compliance with relevant regulations and is carried out with the user's consent, and will not infringe on the user's privacy or violate relevant laws and regulations.

[0066] At the same time, the method provided by this application can be applied to scenarios such as the financial field where users need to communicate frequently. By obtaining the emotion corresponding to the user's current voice, the next communication content or service strategy can be intelligently determined. At the same time, the method provided in other scenarios where communication with people is required, such as medical and education fields, can also achieve good results. Therefore, the financial field is only a specific example of this application, and the specific application scope of the method provided by this application is not limited.

[0067] To solve the above problems, please refer to Figure 1 Specifically, Figure 1 As shown, the provided speech emotion recognition method includes steps S101 to S106. The details are as follows:

[0068] Step S101: Acquire the original speech to be recognized and the original text corresponding to the original speech.

[0069] Specifically, the method can perform real-time emotion recognition on the original speech by reading the original speech from a recording device (such as a phone, microphone) or a file, or can obtain the original speech uploaded offline. The embodiment of the present application does not limit the method for obtaining the original speech. At the same time, to ensure the quality of emotion extraction, background noise and interference can be filtered before recognition. By extracting the original text corresponding to the original speech, such as using models such as Deep Speech, wav2vec 2.0, and Google's Speech-to-Text API. These models usually use convolutional neural networks (CNN), recurrent neural networks (RNN) or Transformer architectures, and are trained through a large number of speech and text pairs to achieve high-accuracy speech-to-text conversion. And then by ensuring the integrity and accuracy of the input data, it lays the foundation for subsequent feature extraction and emotion recognition.

[0070] When the provided method is applied in a financial scenario, in a financial call center, the system automatically receives the customer's telephone recording and converts the voice into text in real time. Figure 2As shown in the figure, when a customer says, "I feel great!" or "I am very dissatisfied with the processing speed of this transaction", the system can accurately obtain the text content of this sentence and provide data support for further sentiment analysis.

[0071] Step S102: Analyze the original speech to obtain multiple speech features corresponding to the original speech; the speech features at least include Mel spectrum, Mel frequency cepstral coefficients and chromaticity vector.

[0072] Specifically, the speech signal corresponding to the original speech is converted into a spectrogram, and then processed by a Mel filter bank to generate a Mel spectrogram. The Mel spectrogram can better capture the sensitivity of the human auditory system to frequency. At the same time, the cepstral coefficients of the speech are extracted by logarithmic processing and discrete cosine transform (DCT) of the Mel spectrum. The cepstral coefficients can effectively capture the spectral envelope of the speech and have good robustness to background noise. The speech signal is also converted into a chromaticity map, and the chromaticity vector is used to capture the musical emotion. Although it is not as commonly used as the first two features in speech emotion recognition, it can provide additional auxiliary information.

[0073] By extracting multiple speech features, we can capture the information in the speech more comprehensively and improve the accuracy of emotion recognition. Each feature has different representation capabilities and can complement each other, enhance the robustness of the system and reduce misjudgment.

[0074] For example, in financial scenarios, when extracting customer voice features, such as Mel spectrum and MFCC, the customer's tone and intonation can be captured. For example, when the customer's tone becomes urgent or the pitch increases, these features can help the system identify that the customer may be feeling anxious or angry.

[0075] Step S103. Obtain the speech vector corresponding to each speech feature based on the self-attention mechanism.

[0076] Specifically, the self-attention mechanism is used to process each extracted speech feature to generate a corresponding speech vector. The self-attention mechanism can dynamically focus on the important parts of each feature and generate a higher-dimensional feature representation. Common self-attention mechanism models include Transformer models, such as BERT. By applying the self-attention mechanism to each speech feature (such as Mel spectrum, MFCC, chrominance vector), the corresponding speech vector is generated. These vectors can capture the complex relationship between features and provide richer information.

[0077] The self-attention mechanism can capture the complex relationship between speech features and generate richer feature representations. It can also better understand the emotional information in speech data, thereby improving the recognition accuracy and credibility of the system.

[0078] For example, in financial scenarios, when processing customer voice data, the self-attention mechanism can better understand the customer's tone and emotional changes. For example, if the customer's tone suddenly becomes sharp during a call, the system can capture this change through the self-attention mechanism and more accurately identify the customer's emotional state.

[0079] In some embodiments, please also refer to Figure 2 and Figure 3 ,like Figure 3 As shown, based on the self-attention mechanism, the speech vector corresponding to each speech feature is obtained respectively, including steps S103a to S103b.

[0080] Step S103a. Input each speech feature into a preset gated recurrent unit to output dependency information corresponding to the speech feature.

[0081] GRU is a variant of recurrent neural network (RNN) used to process sequence data. GRU controls the transmission of information through gating mechanisms (update gate and reset gate), thereby effectively capturing the dependencies in the time series. In this step, each extracted speech feature (such as Mel spectrum, MFCC, chrominance vector) is input into the preset GRU model. Input the time series data of each speech feature (for example, each frame of MFCC feature). Output a sequence of feature vectors containing time dependency information. These feature vector sequences contain not only the feature information of each time point, but also the dependency between the previous and next time points.

[0082] GRU can effectively capture the temporal dependencies of speech features, improving the richness and representation of features. Compared with traditional RNN, GRU reduces the gradient vanishing problem through the gating mechanism, making the model more stable when processing long sequence data.

[0083] For example, in financial scenarios, GRU can be used to capture changes in customer emotions during a call between a customer and customer service representative. For example, if a customer's tone is relatively calm in the first half of the call, but becomes more agitated in the second half, GRU can effectively capture this change and help the system more accurately identify the customer's emotional state.

[0084] Step S103b. Input the dependency information corresponding to each speech feature into a preset first attention module, and the first attention module outputs the speech vector corresponding to each speech feature based on the self-attention mechanism.

[0085] The attention mechanism is a model mechanism that helps the model focus on specific parts when processing sequence data, thereby improving the model's representation ability and classification accuracy. In this step, the dependency information corresponding to each speech feature (i.e., the output of the GRU) is input into the preset first attention module.

[0086] The self-attention mechanism generates an attention weight matrix by calculating the correlation between each feature vector. These weights can be used for weighted summation to generate the final feature vector. Input: Feature vector sequence output by GRU. Output: Comprehensive speech vector corresponding to each speech feature. These vectors not only contain time-dependent information, but also strengthen the key part of the information through the self-attention mechanism.

[0087] The self-attention mechanism can strengthen the key information in the speech features and improve the representation ability of the features. It can also dynamically adjust the focus according to the input data, so that the model can handle different emotional expressions more flexibly.

[0088] For example, in financial scenarios, when processing customer voice data, the first attention module can dynamically focus on the key emotional parts of the voice. For example, if a customer's sentence has a particularly strong tone, the attention mechanism can give this part a higher weight, helping the system to more accurately identify the customer's emotional state. Through the self-attention mechanism, the system can better assess the intensity of the customer's emotions. For example, a customer's "very angry" and "a little angry" can be distinguished by different attention weights, providing more accurate emotional labels.

[0089] In summary, through the method of steps S103a to S103b, the emotional changes of customers can be more accurately identified. For example, the system can determine in real time whether the customer is angry or disappointed, and adjust the customer service strategy in a timely manner. According to the emotional state of the customer, the system can provide personalized service suggestions. For example, if the system detects that the customer is in a bad mood, it can recommend a cool-off period service or directly transfer it to a senior customer service staff for processing. The method can provide more accurate and flexible sentiment analysis capabilities in financial scenarios, thereby improving service quality and customer satisfaction and reducing potential risks.

[0090] Exemplarily, before inputting the dependency information corresponding to each speech feature into a preset first attention module, it also includes: inputting the dependency information into a hyperbolic tangent activation unit, the hyperbolic tangent activation unit parses the dependency information, and outputs the normalization information corresponding to each speech feature, so as to input the normalization information and the corresponding speech feature into the first attention module.

[0091] The hyperbolic tangent activation function (tanh) is a commonly used activation function that can compress the input data into the range of -1 to 1 to achieve normalization. The dependency information feature vector output from the gated recurrent unit (GRU). The normalized dependency information feature vector.

[0092] The dependency information feature vector output by GRU is input into the hyperbolic tangent activation unit and normalized by the tanh function. After normalization, the value of the feature vector will be compressed to a range of -1 to 1, which helps stabilize the calculation of the subsequent attention module. The normalized feature vector can better highlight the information of the key parts, reduce the influence of extreme values, and improve the representation ability of the features.

[0093] At the same time, the normalized dependency information feature vector and the original speech feature vector are input into the first attention module together. The first attention module outputs the final speech vector corresponding to each speech feature based on the self-attention mechanism.

[0094] By normalizing the dependent information feature vector to a range of -1 to 1 through the hyperbolic tangent activation unit, the impact of extreme values ​​on the model can be reduced and the stability of the feature vector can be improved. The normalized feature vector can prevent the gradient explosion problem in the subsequent attention module calculation, making the model training more stable.

[0095] At the same time, normalization can better highlight the key parts of speech features, reduce the interference of irrelevant features, and improve the representation ability of features. The attention mechanism can dynamically adjust the focus according to the normalized feature vector to further enhance the recognition of key information. Inputting the normalized dependent information feature vector and the original speech feature vector into the attention module together can capture emotional information more comprehensively.

[0096] In financial systems, this example can be combined to more accurately identify changes in customer emotions. For example, if a customer's tone is particularly strong over a period of time, normalization and attention mechanisms can better capture this key part, thereby more accurately identifying the customer's emotional state. Based on the customer's emotional state, the system can provide personalized service recommendations. For example, if it is detected that the customer is in a very low mood, the system can recommend psychological counseling or emotional relief services. The emotional labels generated by the system are more accurate and can better reflect the overall satisfaction of the customer. For example, the customer's voice features and text features are combined to more accurately assess the customer's emotional intensity.

[0097] In summary, by adding the processing steps of the hyperbolic tangent activation unit, we can also Figure 2 As shown in the figure, the input is input into the linear layer and then into the tanh part to normalize the dependent information feature vector, further improving the stability and representation ability of the speech feature. The attention module combined with the self-attention mechanism can more accurately identify and classify the customer's voice emotion. This improvement has important application value in financial scenarios and can improve the service quality, customer satisfaction and risk management capabilities of financial institutions.

[0098] It should be noted that, in some embodiments, Figure 2 As shown in the figure, in the speech emotion recognition module, a 128-dimensional Mel spectrum (m), a 40-dimensional Mel frequency cepstral coefficient mfcc (f), and a 12-dimensional chroma vector can be used as input (c). Each feature is subjected to the attention mechanism to extract the part with the most important emotional information. For the i-th input sample, the language feature first passes through the GRU layer, then passes through the tanh activation function to obtain the normalized importance weight of each feature. Finally, the speech vector is calculated based on the weight, which is the weighted sum of all speech features. Then the vector passes through the dense layer to obtain the speech vector C of Chroma, Mel-Spectrogram and MFCC features respectively. i 、M i and F i .

[0099] Step S104: Parse the original text and obtain text embedding information corresponding to the original text.

[0100] Specifically, by preprocessing the original text, such as word segmentation, removal of stop words, stemming, etc., ensure that the text data is clean and standardized. Use a text embedding model to convert the preprocessed text into a high-dimensional vector. Common text embedding models include: Word2Vec: Capture the relationship between words through word vectors. GloVe: Generate word vectors through global word frequency statistics. BERT: Use a bidirectional Transformer model to generate context-related word vectors to better capture emotional vocabulary and contextual information. The corresponding text embedding method can be selected according to specific needs, and the embodiments of the present application are not limited to this.

[0101] Text embedding can capture semantic information in text and improve the accuracy of sentiment recognition. Text embedding can better understand sentiment words and contextual information in text, such as "dissatisfaction" and "anger".

[0102] For example, in financial scenarios, the system can extract keywords and phrases such as "dissatisfaction", "complaint", "refund", etc. from customer call texts, and convert these words into high-dimensional vectors through text embedding models to help the system more accurately identify the customer's emotional state.

[0103] Step S105: Input the text embedding information into a preset text extraction model. The text extraction model parses the text embedding information and outputs a text feature vector corresponding to the text embedding information.

[0104] Specifically, the text embedding information generated in step S104 is input into a preset text extraction model. Common text extraction models include: LSTM (Long Short-Term Memory Network): Capture long dependencies in text through memory units. GRU (Gated Recurrent Units): Similar to LSTM, but with fewer parameters and higher computational efficiency. Transformer: Capture complex contextual information in text through self-attention mechanism and multi-head attention mechanism. The embodiment of the present application does not limit the method of text extraction. The text embedding information is parsed by the text extraction model to generate a higher-level text feature vector. These feature vectors can capture the sentiment, semantics and structural information in the text.

[0105] Through multi-layer processing, the deep features in the text can be better captured and the accuracy of sentiment recognition can be improved. The text extraction model can handle complex text structures, better understand the context of sentiment words, and reduce misjudgment.

[0106] In financial scenarios, for example, text extraction models can be used to more accurately capture the content and tone of customer complaints. For example, a customer's call text "I am very disappointed with your service. The transaction was processed too slowly" can be used to generate a corresponding feature vector through a text extraction model to help assess the customer's emotional state as "angry" or "dissatisfied."

[0107] Step S106: Generate the target emotion type corresponding to the original speech according to the text feature vector and each speech vector, and complete the emotion recognition of the original speech.

[0108] By fusing the text feature vector generated in step S105 and each speech vector generated in step S103. Common fusion methods include: splicing: splicing all feature vectors into a higher dimensional vector through full connection. Weighted summation: weighted summation of each feature vector according to its importance. Multimodal fusion network: use a special multimodal fusion network (such as multimodal Transformer) to fuse features. Use a classifier to perform sentiment classification on the fused feature vectors. Common classifiers include: support vector machine (SVM): classification by maximizing the interval. Decision tree: classification through a tree structure. Neural network: such as multi-layer perceptron (MLP), convolutional neural network (CNN) or Transformer, etc. According to the classification results, the target emotion type corresponding to the original speech is output to complete the emotion recognition task. Common target emotion types include anger, happiness, sadness, excitement, etc.

[0109] By fusing multimodal features, the emotional state can be evaluated more comprehensively, improving the accuracy and credibility of recognition. This method is applicable to a variety of emotion recognition tasks, such as emotion classification, emotion intensity assessment, etc.

[0110] For example, in financial scenarios, in financial call centers, the emotional state of customers can be comprehensively judged based on their voice and text features. For example, if the voice features of a customer show a hasty tone, and the text features show the extensive use of negative words, the system can combine this information to judge the customer's emotional state as "angry" or "very dissatisfied," thereby providing more personalized services and solutions. Financial institutions can use this system to monitor the service quality of customer service personnel and promptly identify and handle customer emotional problems. For example, if a system equipped with the provided method detects that a customer service staff's service has repeatedly caused customer dissatisfaction, the customer service staff can be further trained or adjusted.

[0111] In summary, this speech emotion recognition method can more comprehensively and accurately identify emotional information in speech by extracting and fusing multimodal features, combining the self-attention mechanism and text extraction model. In financial scenarios, this method can help financial institutions better understand customers' emotional needs, provide higher quality services, and improve customer satisfaction and loyalty. Through detailed steps and technical content, this method can cope with various complex emotion recognition tasks and ensure efficient and accurate emotion analysis in financial scenarios.

[0112] In some embodiments, there are multiple text embedding information; the text extraction model includes a second attention module and a hidden module; the text embedding information is input into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information, including: inputting each text embedding information into the second attention module and the hidden module in turn, and obtaining a hidden vector corresponding to each text embedding information; obtaining a text feature vector based on multiple hidden vectors.

[0113] In step S104, the original text is converted into multiple text embedding information. Each text embedding information may correspond to a different sentence fragment or context. Assuming that the original text of the customer is: "I am very dissatisfied with the processing speed of this transaction, and your service attitude is also very poor.", the system may split this text into multiple fragments, such as "I am very dissatisfied with the processing speed of this transaction" and "Your service attitude is also very poor", and generate text embedding information for each fragment.

[0114] The second attention module is used to capture the key parts of each text embedding information and generate attention weights. This helps the system focus more accurately on the emotional part of the text. The hidden module is used to generate hidden vectors for each text embedding information. Common hidden modules include multi-layer neural networks (such as MLP) or recurrent neural networks (such as LSTM, GRU).

[0115] Each text embedding information generated in step S104 is input into the second attention module and the hidden module in turn to obtain the hidden vector corresponding to each text embedding information. The second attention module calculates the attention weight of each text embedding information. The hidden module generates a hidden vector for each text embedding information according to the attention weight. Multiple hidden vectors are fused to generate a final text feature vector. This step fuses the information of all text fragments to generate a comprehensive text feature vector.

[0116] The fusion method can use a specialized multimodal fusion network (such as a multimodal Transformer) to perform feature fusion. The final text feature vector is used for subsequent sentiment recognition tasks.

[0117] In summary, by adding multiple processing steps for text embedding information and introducing the second attention module and hidden module, the comprehensiveness and representation ability of text features are further improved. This improvement enables the system to more accurately identify and classify customers' emotional information, thereby providing higher quality services and more effective risk management in financial scenarios. The specific technical content and steps ensure the robustness and flexibility of the system, and are suitable for a variety of complex emotion recognition tasks.

[0118] like Figure 2 As shown, the text extraction model is a BERT model, which has 12 Transformer modules, 12 attention heads, and 768 hidden units. For the i-th embedding E i Generate the encoded hidden vector R_i. The encoding of the special tags [CLS:Classification] and [SEP:Separator] in the BERT model is represented as C1 and C2. {x i,1 ,x i,2 …x i,T} is the input token of the ith data point of T time steps, B i It is the text feature vector extracted by the BERT module for the i-th input.

[0119] In some embodiments, the target emotion type is any one of a plurality of preset emotion types; after completing the emotion recognition of the original speech, it also includes: determining each emotion type except the target emotion type among the plurality of emotion types as a verification emotion type; respectively obtaining the intersection information corresponding to the target emotion type and each verification emotion type; if it is determined according to the intersection information that the target emotion type is different from each verification emotion type, the verification of the emotion recognition is completed.

[0120] The target emotion type is the result output by the emotion recognition model after completing emotion recognition, and can be one of the preset multiple emotion types, such as "anger", "dissatisfaction", "happiness", "satisfaction", etc. Assume that the preset emotion types are "anger", "dissatisfaction", "happiness", "satisfaction", and "neutral", and the final target emotion type output by the emotion recognition model is "anger". After completing emotion recognition, each emotion type other than the target emotion type in the multiple preset emotion types is determined as a verification emotion type. If the target emotion type is "anger", the verification emotion types include "dissatisfaction", "happiness", "satisfaction", and "neutral". Obtain the intersection information of the target emotion type and each verification emotion type respectively. The intersection information can be obtained by a variety of methods, such as feature similarity calculation, emotion vocabulary overlap calculation, etc.

[0121] For example, feature similarity is to calculate the similarity between the feature vectors of the target sentiment type and the verified sentiment type, for example, using cosine similarity or Euclidean distance. Emotion vocabulary overlap is to calculate the overlap between the target sentiment type and the verified sentiment type in the sentiment vocabulary, for example, using Jaccard similarity.

[0122] Based on the obtained intersection information, determine whether the target emotion type is different from each verified emotion type. If the intersection information of the target emotion type and all verified emotion types indicates that they are different emotion types, the verification of the emotion recognition result is completed. For example, if a similarity threshold is set, if the similarity between the target emotion type and any verified emotion type exceeds the threshold, it is considered that the emotion recognition result may be inaccurate and needs further verification or adjustment. Set an emotion vocabulary overlap threshold. If the vocabulary overlap between the target emotion type and any verified emotion type exceeds the threshold, it is considered that the emotion recognition result may be inaccurate and needs further verification or adjustment.

[0123] For example, in financial scenarios, after identifying customer emotions, the identification results are further confirmed by comparing them with the intersection information of multiple verified emotion types. For example, if the target emotion type is "angry", the system will compare it with "dissatisfied", "happy", "satisfied", and "neutral" to ensure the accuracy of the identification results. If the identification results are found to be inaccurate, the service strategy can be adjusted in real time. For example, if a customer is mistakenly identified as "happy", but the actual customer emotion is "dissatisfied", the system can make corrections in time and provide targeted service suggestions.

[0124] In summary, by introducing the intersection information comparison between the target emotion type and the verification emotion type, the accuracy of emotion recognition and the reliability of the system are further improved. This verification mechanism is particularly important in financial scenarios, as it can help financial institutions better understand the emotional state of their customers, thereby providing higher quality services and more effective risk management. The specific technical content and steps ensure the effectiveness and efficiency of the verification process, and are applicable to a variety of complex emotion recognition tasks.

[0125] Exemplarily, the intersection information corresponding to the target emotion type and each verified emotion type is obtained respectively, including: obtaining the principal component information corresponding to the emotion type; calculating the mean value information and standard deviation information corresponding to the principal component information; obtaining the first emotion component corresponding to the emotion category according to the mean value information and the standard deviation information; obtaining the second emotion component corresponding to each verified emotion type; and respectively calculating the intersection information of the first emotion component and each second emotion component.

[0126] Perform principal component analysis (PCA) on the feature vectors of the target sentiment type and each validation sentiment type to extract the main component information. Principal component analysis can help reduce feature dimensions while retaining the most important information. Calculate the mean and standard deviation of the main component information for each sentiment type. The mean information reflects the main characteristics of the sentiment type, and the standard deviation information reflects the degree of dispersion of the data distribution of the sentiment type.

[0127] Generate a first emotion component based on the mean value information and standard deviation information of the target emotion type. The first emotion component is a quantitative representation of the main emotion characteristics of the target emotion type. Generate a second emotion component based on the mean value information and standard deviation information of each verified emotion type. The second emotion component is a quantitative representation of the main emotion characteristics of the verified emotion type. Calculate the intersection information between the first emotion component and each second emotion component. The intersection information can be used to determine whether the target emotion type is different from the verified emotion type.

[0128] The similarity calculation may be performed by, for example, calculating the similarity between the first emotion component and each second emotion component, for example, using cosine similarity or Euclidean distance, or statistical intersection calculation by calculating the intersection of the first emotion component and each second emotion component in terms of emotional features.

[0129] For example, after identifying the customer's emotions, the main emotional features are extracted through principal component analysis, and then the first emotional component and the second emotional component are generated, and the similarity and intersection information are calculated. For example, if the target emotional type is "angry", the system will compare it with the verified emotional types such as "dissatisfied", "happy", and "satisfied" to ensure the accuracy of the recognition results. Based on more accurate emotion recognition results, the system can provide more personalized service suggestions and improve customer satisfaction. For example, if the system confirms that the customer's emotion is "angry", it can suggest that the customer service staff provide a quick solution and apologize.

[0130] By introducing principal component analysis, mean and standard deviation information, and calculation of the first and second emotion components, the accuracy and reliability of emotion recognition results are further improved. This improved method is particularly important in financial scenarios, as it can help financial institutions better understand the emotional state of their customers, thereby providing higher quality services and more effective risk management. The specific technical content and steps ensure the effectiveness and efficiency of the verification process, and are applicable to a variety of complex emotion recognition tasks.

[0131] It should be noted that, in some embodiments, in order to explore the prediction mechanism of the proposed model, Figure 2 The following steps are performed to analyze the last few layers of the multimodal emotion recognition module (composed of a speech emotion recognition module and a text emotion recognition module). First, for each emotion category i, we calculate the principal component (x i ,y i ,z i ). For each sentiment category i and the corresponding (x i ,y i ,z i ) components, calculate the mean and Standard Deviation

[0132]

[0133] where n i is the number of samples in the i-th category, and k represents the k-th data point.

[0134] Set L i (m) and R i (m) is the component of the mth emotion in the left and right extreme points of the i-th emotion category, and the formula is defined as follows:

[0135]

[0136] In addition, I i，j (m) is used to calculate the intersection between sentiment categories, which represents the degree of intersection between categories i and j:

[0137]

[0138] Then calculate the product I of all component crosses between sentiment categories i and j i,j , as shown in the formula. i,j The value of is in the range [0,1].

[0139] I i,j =I i,j (x)*I i,j (y)*I i,j (z);

[0140] Finally, define a 4×4 matrix as I. The (i,j)th element in I represents the total intersection degree of emotion i and emotion j. i,j , I i,j The smaller the value of is, the better the elements of the i-th and j-th emotions are divided into different emotion clusters. The analysis of the model output layer explains the mechanism of the model's internal output results and improves the credibility of the emotion recognition results.

[0141] In some embodiments, generating a target emotion type corresponding to the original speech according to the text feature vector and each speech vector includes: fully connecting the text feature vector and each speech vector to obtain a concatenated vector; calculating the target emotion type according to a preset prediction function; the expression corresponding to the prediction function includes:

[0142] y e =softmax(W T E i + b);

[0143] Among them, e is the target emotion type, y e is the prediction of the target sentiment type, W is the weight matrix, b represents the bias term, E i represents the concatenated vector, and softmax() is the prediction function.

[0144] like Figure 2 As shown, the four vectors obtained in the speech emotion recognition stage are connected (C i ,M i ,F i ,B i ), and obtain E through the fully connected layer i . The final vector E i Pass it to the softmax layer and get the prediction result shown in the above formula, such as Figure 2 The result shown is "Happy". Sparse categorical cross entropy can be used as the loss function and the optimizer is Adam. In each iteration, the model is updated according to the output of the loss function.

[0145] See also Figure 4 As shown, Figure 4 1 is a schematic diagram of the structure of the speech emotion recognition device 200 provided in an embodiment of the present application. The speech emotion recognition device 200 is used to execute the steps of the speech emotion recognition method shown in the above embodiments. The speech emotion recognition device 200 can be a single server or a server cluster, or the speech emotion recognition device 200 can be a terminal, which can be a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0146] like Figure 4 As shown, the speech emotion recognition device 200 includes:

[0147] The text acquisition unit 201 is used to acquire the original speech to be recognized and the original text corresponding to the original speech;

[0148] The speech analysis unit 202 is used to analyze the original speech and obtain multiple speech features corresponding to the original speech; the speech features at least include Mel spectrum, Mel frequency cepstral coefficients and chromaticity vector;

[0149] A vector acquisition unit 203 is used to respectively acquire a speech vector corresponding to each speech feature based on a self-attention mechanism;

[0150] The embedding acquisition unit 204 is used to parse the original text and acquire text embedding information corresponding to the original text;

[0151] The vector output unit 205 is used to input the text embedding information into a preset text extraction model, and the text extraction model parses the text embedding information and outputs a text feature vector corresponding to the text embedding information;

[0152] The recognition completion unit 206 is used to generate a target emotion type corresponding to the original speech according to the text feature vector and each speech vector, and complete the emotion recognition of the original speech.

[0153] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the speech emotion recognition device and each module described above can refer to the corresponding process in the speech emotion recognition method embodiments described in the above embodiments, and will not be repeated here.

[0154] The above-mentioned speech emotion recognition method can be implemented in the form of a computer program. The computer program can be used in Figure 4 Run on the device shown.

[0155] See also Figure 5 , Figure 51 is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device includes a processor, a memory and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.

[0156] The storage medium can store an operating device and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any speech emotion recognition method.

[0157] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0158] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any speech emotion recognition method.

[0159] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0160] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0161] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0162] Obtaining the original speech to be recognized and the original text corresponding to the original speech;

[0163] Parse the original speech to obtain multiple speech features corresponding to the original speech; the speech features at least include Mel spectrum, Mel frequency cepstral coefficients and chromaticity vector;

[0164] Based on the self-attention mechanism, the speech vector corresponding to each speech feature is obtained respectively;

[0165] Parse the original text and obtain the text embedding information corresponding to the original text;

[0166] The text embedding information is input into a preset text extraction model, and the text extraction model parses the text embedding information and outputs a text feature vector corresponding to the text embedding information;

[0167] The target emotion type corresponding to the original speech is generated according to the text feature vector and each speech vector, and the emotion recognition of the original speech is completed.

[0168] In some embodiments, a speech vector corresponding to each speech feature is obtained based on a self-attention mechanism, including: inputting each speech feature into a preset gated recurrent unit to output dependency information corresponding to the speech feature; inputting the dependency information corresponding to each speech feature into a preset first attention module, and the first attention module outputs a speech vector corresponding to each speech feature based on the self-attention mechanism.

[0169] Exemplarily, before inputting the dependency information corresponding to each speech feature into a preset first attention module, it also includes: inputting the dependency information into a hyperbolic tangent activation unit, the hyperbolic tangent activation unit parses the dependency information, and outputs the normalization information corresponding to each speech feature, so as to input the normalization information and the corresponding speech feature into the first attention module.

[0170] In some embodiments, there are multiple text embedding information; the text extraction model includes a second attention module and a hidden module; the text embedding information is input into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information, including: inputting each text embedding information into the second attention module and the hidden module in turn, and obtaining a hidden vector corresponding to each text embedding information; obtaining a text feature vector based on multiple hidden vectors.

[0171] In some embodiments, the target emotion type is any one of a plurality of preset emotion types; after completing the emotion recognition of the original speech, it also includes: determining each emotion type except the target emotion type among the plurality of emotion types as a verification emotion type; respectively obtaining the intersection information corresponding to the target emotion type and each verification emotion type; if it is determined according to the intersection information that the target emotion type is different from each verification emotion type, the verification of the emotion recognition is completed.

[0172] Exemplarily, the intersection information corresponding to the target emotion type and each verified emotion type is obtained respectively, including: obtaining the principal component information corresponding to the emotion type; calculating the mean value information and standard deviation information corresponding to the principal component information; obtaining the first emotion component corresponding to the emotion category according to the mean value information and the standard deviation information; obtaining the second emotion component corresponding to each verified emotion type; and respectively calculating the intersection information of the first emotion component and each second emotion component.

[0173] In some embodiments, generating a target emotion type corresponding to the original speech according to the text feature vector and each speech vector includes: fully connecting the text feature vector and each speech vector to obtain a concatenated vector; calculating the target emotion type according to a preset prediction function; the expression corresponding to the prediction function includes:

[0174] y e =softmax(W T E i + b);

[0175] Among them, e is the target emotion type, y e is the prediction of the target sentiment type, W is the weight matrix, b represents the bias term, E i represents the concatenated vector, and softmax() is the prediction function.

[0176] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the speech emotion recognition method as described in the first aspect above.

[0177] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0178] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech emotion recognition method, characterized in that: include: Acquire an original speech to be recognized and an original text corresponding to the original speech; Analyze the original speech to obtain multiple speech features corresponding to the original speech; The speech features at least include Mel spectrum, Mel frequency cepstral coefficients and chromaticity vector; Based on the self-attention mechanism, the speech vector corresponding to each of the speech features is obtained respectively; Parsing the original text to obtain text embedding information corresponding to the original text; Inputting the text embedding information into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information; A target emotion type corresponding to the original speech is generated according to the text feature vector and each of the speech vectors, thereby completing the emotion recognition of the original speech.

2. The method according to claim 1, characterized in that: The method of obtaining the speech vector corresponding to each speech feature based on the self-attention mechanism includes: Inputting each of the speech features into a preset gated recurrent unit to output dependency information corresponding to the speech feature; The dependency information corresponding to each of the speech features is input into a preset first attention module, and the first attention module outputs a speech vector corresponding to each of the speech features based on a self-attention mechanism.

3. The method according to claim 2, characterized in that Before inputting the dependency information corresponding to each of the speech features into the preset first attention module, the method further includes: The dependency information is input into a hyperbolic tangent activation unit, and the hyperbolic tangent activation unit parses the dependency information and outputs normalized information corresponding to each of the speech features, so as to input the normalized information and the corresponding speech features into the first attention module.

4. The method according to claim 1, characterized in that: The number of the text embedding information is multiple; the text extraction model includes a second attention module and a hidden module; the text embedding information is input into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information, including: Input each of the text embedding information into the second attention module and the hidden module in sequence to obtain a hidden vector corresponding to each of the text embedding information; The text feature vector is obtained according to a plurality of hidden vectors.

5. The method according to claim 1, characterized in that The target emotion type is any one of a plurality of preset emotion types; after completing the emotion recognition of the original speech, the method further includes: Determine each of the plurality of emotion types except the target emotion type as a verification emotion type; Respectively obtain intersection information corresponding to the target emotion type and each of the verification emotion types; If it is determined according to the intersection information that the target emotion type is different from each of the verification emotion types, the verification of the emotion recognition is completed.

6. The method according to claim 5, characterized in that The step of respectively obtaining intersection information corresponding to the target emotion type and each of the verification emotion types includes: Obtaining principal component information corresponding to the emotion type; Calculate the mean value information and standard deviation information corresponding to the principal component information; Acquire a first emotion component corresponding to the emotion category according to the mean value information and the standard deviation information; Obtaining a second emotion component corresponding to each of the verified emotion types; The intersection information of the first emotion component and each of the second emotion components is calculated respectively.

7. The method according to claim 1, characterized in that Generating the target emotion type corresponding to the original speech according to the text feature vector and each of the speech vectors includes: Fully connect the text feature vector and each of the speech vectors to obtain a concatenated vector; The target emotion type is calculated according to a preset prediction function; the expression corresponding to the prediction function includes: y e =softmax(W T E i +b) Among them, e is the target emotion type, y e is the prediction of the target emotion type, W is the weight matrix, b represents the bias term, E i represents the concatenated vector, and softmax() is the prediction function.

8. A question and answer information generating device, characterized in that: include: A text acquisition unit, used to acquire an original speech to be recognized and an original text corresponding to the original speech; A speech analysis unit, used for analyzing the original speech to obtain a plurality of speech features corresponding to the original speech; the speech features at least include a Mel spectrum, a Mel frequency cepstral coefficient and a chromaticity vector; A vector acquisition unit, used for respectively acquiring a speech vector corresponding to each of the speech features based on a self-attention mechanism; An embedding acquisition unit, used for parsing the original text and acquiring text embedding information corresponding to the original text; A vector output unit, used to input the text embedding information into a preset text extraction model, the text extraction model parses the text embedding information, and outputs a text feature vector corresponding to the text embedding information; The recognition completion unit is used to generate a target emotion type corresponding to the original speech according to the text feature vector and each of the speech vectors, and complete the emotion recognition of the original speech.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 7.

Images