Method, device, equipment and storage medium for emotion recognition

Abstract

Embodiments of the present application provide a kind of mood recognition method, device, equipment and storage medium, involve the field of financial technology.The method comprises: obtaining the speech feature vector and text feature vector corresponding to the speech data to be identified;First key-value feature vector and first value item feature vector corresponding to text feature vector, speech feature vector corresponding first query item feature vector are obtained to calculate first feature vector;Second query item feature vector corresponding to text feature vector, speech feature vector corresponding second key-value feature vector and second value item feature vector are obtained to calculate second feature vector;First feature vector and second feature vector are carried out mood classification processing, obtain the mood category corresponding to the speech data to be identified;With the service information corresponding to mood category is fed back to target object terminal.The device, equipment and storage medium apply the method, therefore, the precision of mood category identification can be improved to improve service experience in the embodiments of the present application.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of financial technology, and in particular to a method, apparatus, device, and storage medium for emotion recognition. Background Technology

[0002] Emotion recognition is a widely used field in the service industry, enabling better customer service by identifying customer emotional states. Taking online shopping in the fintech sector as an example, customer service communication is crucial. Accurately identifying customer emotional states during these interactions helps provide better service and resolve customer issues. However, most emotion recognition methods rely on single models, such as text recognition or speech recognition. While both speech and text can predict customer emotional fluctuations to some extent—for example, changes in tone of voice or the presence of emotional keywords in text—these can all serve as important indicators. Although some technologies fuse text and speech, most of these methods classify and process the speech and text separately before fusing the results to obtain an emotion category. Therefore, the accuracy of emotion category recognition in these technologies is not high, leading to a poor service experience. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, device, and storage medium for emotion recognition, aiming to improve the accuracy of emotion recognition and enhance the service experience.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for emotion recognition, the method comprising:

[0005] Acquire the voice data to be recognized from the target terminal;

[0006] Obtain the speech feature vector and text feature vector corresponding to the speech data to be recognized;

[0007] Obtain the first key-value feature vector and the first value item feature vector corresponding to the text feature vector, and the first query item feature vector corresponding to the speech feature vector;

[0008] The first feature vector is calculated based on the first key value feature vector, the first value item feature vector, and the first query item feature vector;

[0009] Obtain the second query item feature vector corresponding to the text feature vector, the second key value feature vector corresponding to the speech feature vector, and the second value item feature vector;

[0010] The second feature vector is calculated based on the second key value feature vector, the second value item feature vector, and the second query item feature vector;

[0011] The first feature vector and the second feature vector are subjected to emotion classification processing to obtain the emotion category corresponding to the speech data to be identified.

[0012] The service information corresponding to the emotion category will be fed back to the target terminal.

[0013] In some embodiments, obtaining the first key-value feature vector and the first value-item feature vector corresponding to the text feature vector, and the first query-item feature vector corresponding to the speech feature vector, includes:

[0014] The first key-value feature vector and the first value-term feature vector are obtained by performing matrix operations on the text feature vector based on the first weight set using a preset first cross-attention sub-model.

[0015] Based on the first weight set, matrix operations are performed on the speech feature vector to obtain the first query item feature vector;

[0016] Correspondingly, the step of calculating the first feature vector based on the first key-value feature vector, the first value item feature vector, and the first query item feature vector includes:

[0017] The first key-value feature vector, the first value item feature vector, and the first query item feature vector are input into the first cross-attention sub-model, so that the first cross-attention sub-model calculates the first feature vector based on the first attention formula.

[0018] In some embodiments, obtaining the second query term feature vector corresponding to the text feature vector, the second key-value feature vector corresponding to the speech feature vector, and the second value term feature vector includes:

[0019] The speech feature vector is subjected to matrix operations using the second weight set in the preset second cross-attention sub-model to obtain the second key-value feature vector and the first value term feature vector.

[0020] The second query item feature vector is obtained by performing matrix operations on the text feature vector using the second weight set.

[0021] Correspondingly, the step of calculating the second feature vector based on the second key-value feature vector, the second value-item feature vector, and the second query-item feature vector includes:

[0022] The second key-value feature vector, the second value-item feature vector, and the second query-item feature vector are input into the second attention formula in the second cross-attention sub-model to calculate the second feature vector.

[0023] In some embodiments, emotion classification processing is performed on the first feature vector and the second feature vector to obtain the emotion category corresponding to the speech data to be identified, including:

[0024] The third feature vector is obtained by performing residual calculation on the first feature vector and the speech feature vector;

[0025] The fourth feature vector is obtained by performing residual calculation on the second feature vector and the text feature vector;

[0026] The third feature vector and the fourth feature vector are respectively subjected to feature enhancement to obtain a fifth feature vector corresponding to the third feature vector and a sixth feature vector corresponding to the fourth feature vector;

[0027] The fourth feature vector and the sixth feature vector are fused and classified to obtain the emotion category corresponding to the speech data to be identified.

[0028] In some embodiments, feature enhancement is performed on the third feature vector and the fourth feature vector respectively to obtain a fifth feature vector corresponding to the third feature vector and a sixth feature vector corresponding to the fourth feature vector, including:

[0029] The fifth feature vector is obtained by performing attention calculation on the third feature vector through a preset first self-attention mechanism;

[0030] The fourth feature vector is obtained by performing attention calculation on the fourth feature vector through a preset second self-attention mechanism.

[0031] In some embodiments, the step of performing attention calculation on the third feature vector through a preset first self-attention mechanism to obtain the fifth feature vector includes:

[0032] The third feature vector is input into a preset first self-attention sub-model to obtain the first attention feature;

[0033] The fifth feature vector is obtained by performing residual calculation on the first attention feature;

[0034] The step of performing attention calculation on the fourth feature vector through a preset second self-attention mechanism to obtain the sixth feature vector includes:

[0035] The fourth feature vector is input into a preset second self-attention sub-model to obtain the second attention feature;

[0036] The sixth feature vector is obtained by performing residual calculation on the second attention feature.

[0037] In some embodiments, the step of fusing and classifying the fourth feature vector and the sixth feature vector to obtain the emotion category corresponding to the speech data to be identified includes:

[0038] The fourth feature vector and the sixth feature vector are input into a preset fully connected layer to calculate the fused feature vector;

[0039] The fused feature vector is classified and predicted by a preset logistic regression layer to obtain the emotion category.

[0040] To achieve the above objectives, a second aspect of this application provides an emotion recognition apparatus, the apparatus comprising:

[0041] An interaction module, which is used to acquire voice data to be recognized from the target terminal;

[0042] A feature acquisition module, wherein the acquisition module is used to acquire the speech feature vector and text feature vector corresponding to the speech data to be recognized;

[0043] The text acquisition module is used to acquire the first key-value feature vector and the first value item feature vector corresponding to the text feature vector, and the first query item feature vector corresponding to the speech feature vector;

[0044] A text processing module is configured to calculate a first feature vector based on the first key-value feature vector, the first value item feature vector, and the first query item feature vector.

[0045] The voice acquisition module is used to acquire the second query item feature vector corresponding to the text feature vector, the second key value feature vector and the second value item feature vector corresponding to the voice feature vector;

[0046] A voice processing module is used to calculate a second feature vector based on the second key-value feature vector, the second value item feature vector, and the second query item feature vector;

[0047] A determination module is used to perform emotion classification processing on the first feature vector and the second feature vector to obtain the emotion category corresponding to the speech data to be recognized.

[0048] An application module is used to feed back service information corresponding to the emotion category to the target terminal.

[0049] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0050] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0051] The emotion recognition method, apparatus, device, and storage medium proposed in this application determine a first key-value feature vector, a first value-item feature vector, and a first query-item feature vector related to the text feature vector, and a first query-item feature vector, a second key-value feature vector, and a second value-item feature vector related to the speech feature vector, respectively, through text feature vectors and speech feature vectors. This allows the first feature vector to fully learn the advantages of speech, and the second feature vector to fully learn the advantages of text. When the first and second feature vectors are fused, the advantages of the two different directions are combined, thereby improving the accuracy of emotion category recognition. At this time, corresponding service information is provided based on the recognized emotion category. Since this service information is generated in response to the emotions of the target object, the embodiments of this application can alleviate negative emotions and enhance positive emotions by improving the accuracy of emotion category recognition, thereby improving the service experience. Attached Figure Description

[0052] Figure 1 This is a flowchart of the emotion recognition method provided in the embodiments of this application;

[0053] Figure 2 This is a flowchart illustrating the residual calculation process in the emotion recognition method provided in this application embodiment;

[0054] Figure 3 This is a flowchart illustrating a specific embodiment of the emotion recognition method provided in this application.

[0055] Figure 4 This is a schematic diagram illustrating an application of the emotion recognition method provided in this application embodiment;

[0056] Figure 5 This is a schematic diagram illustrating another application of the emotion recognition method provided in the embodiments of this application;

[0057] Figure 6 This is a schematic diagram of the modules corresponding to the emotion recognition method provided in the embodiments of this application;

[0058] Figure 7This is a schematic diagram of the hardware structure corresponding to the emotion recognition method provided in the embodiments of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0062] First, let's analyze some of the terms used in this application:

[0063] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0064] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0065] Emotion recognition is a widely used field in the service industry, enabling better customer service by identifying customer emotional states. Taking online shopping in the fintech sector as an example, customer service communication is crucial. Accurately identifying customer emotional states during these interactions helps provide better service and resolve customer issues. However, most emotion recognition methods rely on single models, such as text recognition or speech recognition. While both speech and text can predict customer emotional fluctuations to some extent—for example, changes in tone of voice or the presence of emotional keywords in text—these can all serve as important indicators. Although some technologies fuse text and speech, most of these methods classify and process the speech and text separately before fusing the results to obtain an emotion category. Therefore, the accuracy of emotion category recognition in these technologies is not high, leading to a poor service experience.

[0066] Based on this, embodiments of this application provide a method, apparatus, device, and storage medium for emotion recognition, aiming to improve the accuracy of emotion category recognition, thereby alleviating negative emotions and enhancing positive emotions, and thus improving the service experience.

[0067] The emotion recognition method, apparatus, device, and storage medium provided in this application are specifically described through the following embodiments. First, the emotion recognition method in this application is described.

[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] The emotion recognition method provided in this application relates to, but is not limited to, the field of fintech. The emotion recognition method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the emotion recognition method, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] Figure 1 This is an optional flowchart of the emotion recognition method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S108.

[0073] Step S101: Obtain the voice data to be recognized from the target terminal.

[0074] It should be noted that the voice data to be recognized is the voice information input by the target object on the target object's terminal, and the voice information includes feedback, application evaluation, requests, etc.

[0075] Step S102: Obtain the speech feature vector and text feature vector corresponding to the speech data to be recognized.

[0076] It should be noted that the speech data to be recognized can be converted into text data. This application does not limit the speech-to-text conversion method, and those skilled in the art can determine the conversion method based on existing technology. The speech feature vector is the speech feature extracted from the speech data to be recognized, and the text feature vector is the text feature extracted from the text data. For example, text data is processed using a Roberta pre-trained language model to extract text features, resulting in a text feature vector; the speech data to be recognized is processed using a wav2vec pre-trained model to extract speech features, resulting in a speech feature vector. Therefore, this application does not limit the extraction methods for speech feature vectors and text feature vectors.

[0077] Step S103: Obtain the first key-value feature vector and the first value-item feature vector corresponding to the text feature vector, and the first query item feature vector corresponding to the speech feature vector.

[0078] It should be noted that the three weight matrices W can be determined by training a pre-defined first cross-attention model. Q1 W K1 W V1 The first query item feature vector, the first key value feature vector, and the first value item feature vector are determined respectively. In some other embodiments, the weights used in existing attention models for attention calculation of text features can also be directly used to determine the first query item feature vector, the first key value feature vector, and the first value item feature vector. This application does not impose limitations on this step.

[0079] Step S104: Calculate the first feature vector based on the first key value feature vector, the first value item feature vector, and the first query item feature vector.

[0080] It should be noted that the first feature vector is calculated using the first attention formula. For example, the first attention formula is: Where Q1 represents the feature vector of the first query term; K1 represents the feature vector of the first key value; and V1 represents the feature vector of the first value term. By fusing speech feature vectors into text processing, the advantages of features from both directions can be fully utilized to enhance recognition capabilities.

[0081] Step S105: Obtain the second query term feature vector corresponding to the text feature vector, the second key value feature vector corresponding to the speech feature vector, and the second value term feature vector.

[0082] It should be noted that the three weight matrices W can be determined by training a pre-defined second cross-attention model. Q2 W K2 W V2 The second query term feature vector, the second key-value feature vector, and the second value term feature vector are determined respectively. In some other embodiments, the weights used in existing attention models for attention calculation of speech features can also be directly used to determine the second query term feature vector, the second key-value feature vector, and the second value term feature vector. This application does not impose limitations on this step.

[0083] Step S106: Calculate the second feature vector based on the second key value feature vector, the second value item feature vector, and the second query item feature vector.

[0084] It should be noted that the second feature vector is calculated using the second attention formula. For example, the second attention formula is: Where Q2 represents the feature vector of the second query term; K2 represents the feature vector of the second key value; and V2 represents the feature vector of the second value term. Therefore, by fusing speech feature vectors in speech processing, the advantages of features from both directions can be fully utilized to enhance recognition capabilities.

[0085] Step S107: Perform emotion classification processing on the first feature vector and the second feature vector to obtain the emotion category corresponding to the speech data to be recognized.

[0086] It should be noted that in some embodiments, the emotion classification process includes: directly normalizing and classifying the first feature vector and the second feature vector to predict the emotion category; in other embodiments, the emotion classification process includes: performing self-attention learning on the first feature vector to enhance text features and then fusing it with the second feature vector, or performing self-attention learning on the second feature vector to enhance speech features and then fusing it with the first feature vector, or referring to... Figure 3 As shown, in some other embodiments, the emotion classification process includes: enhancing the text and speech features respectively by applying corresponding self-attention learning to the first feature vector and the second feature vector, and then performing normalized classification prediction to determine the emotion category. It should be noted that in some embodiments, reference can also be made to... Figure 2 As shown, in the emotion classification process, after attention calculation, the prediction accuracy is improved by adding a residual network, Add&Norm. Therefore, the method of emotion classification in this step is not limited; those skilled in the art can selectively set it according to the actual situation.

[0087] It should be noted that emotion classification processing refers to determining the probability that the first and second feature vectors are similar to the corresponding features of each preset emotion category, and then determining the emotion category based on the calculated probability. The preset emotion categories are broadly divided into three categories: positive emotions, negative emotions, and neutral emotions. Positive emotions indicate a positive feeling towards the service, such as high satisfaction or high interest; negative emotions indicate a negative feeling towards the service, such as low satisfaction, disinterest, or resistance; neutral emotions indicate indifference towards the service. This application does not limit the classification of emotions; those skilled in the art can further subdivide based on the above three categories or classify them according to the above categories to improve the service experience.

[0088] Step S108: Feed back the service information corresponding to the emotion category to the target terminal.

[0089] It should be noted that the service information refers to the response related to the product or service fed back by the voice data to be identified. It can be product information related to the product or a voice or text response to the user's request. This application embodiment will not elaborate on this further.

[0090] For example, in the insurance industry, if a negative emotion is identified, the current business recommendation is suspended, and routine communication is conducted to ease the emotional state before resuming business promotion, thereby reducing the probability of customer complaints. In the case of a product production planning survey, if a positive emotion is identified, the product can be determined to be popular. If the product is found to be popular, similar products can be promoted based on its characteristics, and production can be increased accordingly. This application does not impose restrictions on service information in this regard.

[0091] For example, in the case of online medical consultation in the healthcare industry, when the emotion category is determined to be negative based on the voice data to be recognized input by the patient, such as panic or fear, the doctor can first soothe the patient's emotions before providing further medical advice. This can help the doctor consult the patient for more detailed and accurate medical data and provide the patient with a better consultation experience.

[0092] For example, taking online shopping as an example, when the emotion category is determined to be negative based on the voice data to be recognized input by the consumer, such as disinterest or resistance, similar products will be avoided when recommending products, thereby assisting product sales. When the recognized emotion type is positive, such as interest or liking, similar products will be collected and recommended, thereby improving the consumer's shopping experience.

[0093] Steps S101 to S108 of this embodiment determine, respectively, a first key-value feature vector, a first value-item feature vector, and a first query-item feature vector related to the text feature vector, and a first query-item feature vector, a second key-value feature vector, and a second value-item feature vector related to the speech feature vector, using text feature vector and speech feature vector. This allows the first feature vector to fully learn the advantages of speech, and the second feature vector to fully learn the advantages of text. When the first and second feature vectors are fused, the advantages of the two different directions are combined, thereby improving the accuracy of emotion category recognition. At this time, corresponding service information is provided based on the recognized emotion category. Since this service information is generated for the emotions of the target object, this embodiment improves the accuracy of emotion category recognition, thereby alleviating negative emotions and enhancing positive emotions, thus improving the service experience.

[0094] It should be noted that the embodiments of this application adopt multimodal fusion technology, which fully integrates the features of text in emotion recognition in the processing of speech data to be recognized, and fully integrates the features of speech in emotion recognition in the processing of text data. Therefore, the embodiments of this application can make full use of the advantages of the two-way model.

[0095] It should be noted that the multimodal fusion technique uses a pure attention mechanism to calculate the first and second feature vectors, which is simple to implement and yields significant results. It also allows text and speech to fully integrate and learn each other's features based on their respective models.

[0096] In some embodiments, step S103 may include, but is not limited to, the following steps:

[0097] The first key-value feature vector and the first value-term feature vector are obtained by performing matrix operations on the text feature vector based on the first weight set using the preset first cross-attention sub-model.

[0098] The first query term feature vector is obtained by performing matrix operations on the speech feature vector based on the first weight set.

[0099] Corresponding to the sub-steps of step S103 above, step S104 may include, but is not limited to, the following steps:

[0100] The first key-value feature vector, the first value term feature vector, and the first query term feature vector are input into the first cross-attention sub-model, so that the first cross-attention sub-model calculates the first feature vector based on the first attention formula.

[0101] It should be noted that the first cross-attention sub-model is a pre-trained attention model for text processing, and the first weight set includes... The text feature vectors are respectively compared with Multiplying them yields the first key-value eigenvector and the first value-term eigenvector; The feature vector of the first query term is obtained by calculating the feature vector of the speech feature vector.

[0102] It should be noted that the formula for the first attention is:

[0103] In some embodiments, step S105 may include, but is not limited to, the following steps:

[0104] The speech feature vector is subjected to matrix operations by the second weight set in the preset second cross-attention sub-model to obtain the second key-value feature vector and the first value term feature vector.

[0105] The second query term feature vector is obtained by performing matrix operations on the text feature vector using the second weight set.

[0106] Corresponding to the sub-steps of step S105 above, step S106 may include, but is not limited to, the following steps:

[0107] The second key-value feature vector, the second value term feature vector, and the second query term feature vector are input into the second attention formula in the second cross-attention sub-model to calculate the second feature vector.

[0108] It should be noted that the second cross-attention sub-model is a pre-trained attention model for speech processing, and the second weight set includes... The speech feature vectors are respectively compared with Multiplying them yields the second key-value eigenvector and the second-value term eigenvector; The feature vector of the second query term is obtained by calculating the feature vector of the text feature vector.

[0109] It should be noted that the formula for the second attention is:

[0110] Reference Figure 2 As shown, in some embodiments, step S107 may include, but is not limited to, the following steps:

[0111] S201, perform residual calculation on the first feature vector and the speech feature vector to obtain the third feature vector.

[0112] S202, perform residual calculation on the second feature vector and the text feature vector to obtain the fourth feature vector.

[0113] S203, perform feature enhancement on the third and fourth feature vectors respectively to obtain the fifth feature vector corresponding to the third feature vector and the sixth feature vector corresponding to the fourth feature vector.

[0114] S204, the fourth feature vector and the sixth feature vector are fused and classified to obtain the emotion category corresponding to the speech data to be recognized.

[0115] It should be noted that by using residual calculation, the third and fourth feature vectors carry the original speech feature vector and text feature vector respectively, avoiding the problems of gradient vanishing and network difficulty in learning, thus improving the effect of emotion category prediction.

[0116] It should be noted that residual calculation can be performed using a trained residual network, as shown in the reference. Figure 3 As shown, the output and input of the first cross-attention sub-model are used as inputs to the residual network Add&Norm for residual calculation, resulting in the third feature vector. Similarly, the input and output of the second cross-attention sub-model are used as inputs to another residual network, Add&Norm, for residual calculation. These two residual networks can have the same network structure or different network structures. This application does not impose any limitations on this. Furthermore, this application does not specifically limit the composition of the residual networks; those skilled in the art can selectively configure them according to actual conditions, such as using Add as a residual connection and Norm employing Layer normalization.

[0117] In step S203 of some embodiments, feature enhancement is performed on the third feature vector and the fourth feature vector respectively to obtain a fifth feature vector corresponding to the third feature vector and a sixth feature vector corresponding to the fourth feature vector. This step may include, but is not limited to, the following steps:

[0118] The fifth feature vector is obtained by performing attention calculation on the third feature vector through a preset first self-attention mechanism;

[0119] The fourth feature vector is obtained by performing attention calculation on the fourth feature vector through a pre-set second self-attention mechanism.

[0120] It should be noted that by adding a self-attention mechanism, the third feature vector can be fully self-learned to improve the accuracy of emotion category prediction. For the first self-attention mechanism, Q, K, and V in the self-attention formula all originate from themselves. For example, the first self-attention mechanism uses the third feature vector and W in the first self-attention mechanism... Q W K W V Multiplying these values ​​yields the corresponding Q, K, and V. Therefore, it allows for sufficient self-learning of features.

[0121] It should be noted that, referring to Figure 3As shown, the first self-attention mechanism, the first cross-attention sub-model, the second self-attention mechanism, the second cross-attention sub-model, and the residual network can all be set as one of the sub-models in the attention model. During the training process of the attention model, the first self-attention mechanism, the first cross-attention sub-model, the second self-attention mechanism, the second cross-attention sub-model, and the residual network are trained and adjusted simultaneously until the output result of the attention model meets the expected value. In some other embodiments, the above sub-models are trained separately to reach preset convergence conditions. In this regard, the embodiments of this application do not impose too many constraints on the training process of the above sub-models.

[0122] In some embodiments, the step of performing attention calculation on the third feature vector through a preset first self-attention mechanism to obtain the fifth feature vector may include, but is not limited to, the following steps:

[0123] The third feature vector is input into the preset first self-attention sub-model to obtain the first attention feature;

[0124] The residual of the first attention feature is calculated to obtain the fifth feature vector;

[0125] In some embodiments, the step of performing attention calculation on the fourth feature vector through a preset second self-attention mechanism to obtain the sixth feature vector may include, but is not limited to, the following steps:

[0126] The fourth feature vector is input into the preset second self-attention sub-model to obtain the second attention feature;

[0127] The residual of the second attention feature is calculated to obtain the sixth feature vector.

[0128] It should be noted that performing residual calculation after performing self-attention calculation in the first self-attention sub-model can further enable the fifth feature vector to carry the third feature vector and the sixth feature vector to relax the fourth feature vector, thereby further improving the accuracy of emotion classification and recognition.

[0129] It should be noted that residual calculation can be performed using residual joins and then standardized, with layer normalization used for standardization. Residual joins involve adding the inputs and outputs of the previous model.

[0130] For example, refer to Figure 3 As shown, a residual network Add&Norm is set after the first self-attention sub-model, where the output and input of the first self-attention sub-model are both inputs to the residual network.

[0131] In step S201 of some embodiments, the fourth feature vector and the sixth feature vector are fused and classified to obtain the emotion category corresponding to the speech data to be identified, including:

[0132] The fourth and sixth feature vectors are input into a pre-defined fully connected layer to calculate the fused feature vector.

[0133] The fused feature vectors are classified and predicted by a pre-defined logistic regression layer to obtain the emotion category.

[0134] It should be noted that the logistic regression layer is a softmax layer. The softmax layer is a commonly used network layer for normalization classification processing in this field, and will not be described in detail in the embodiments of this application.

[0135] The following reference Figure 3 As shown, the flow of a specific example illustrating the emotion recognition method described in this application is as follows:

[0136] Referring to step S101, the speech data to be recognized is obtained from the target terminal, and the speech data to be recognized is converted into text data. The speech data to be recognized and the text data are then subjected to feature learning and fusion processing as follows:

[0137] Referring to S102, speech feature vectors are obtained by extracting speech features using the wav2vec pre-trained model. Text feature vectors are obtained by extracting text features using the Roberta pre-trained model.

[0138] For the speech data to be recognized, refer to sub-steps S105 and S106, step S107, and... Figure 2 The second feature vector is calculated using the second cross-attention sub-model. The second feature vector and the speech feature vector are then input into the residual network connected to the second cross-attention sub-model, respectively, and then added together before being standardized by Layer normalization to output the fourth feature vector. The fourth feature vector is then processed by the second self-attention sub-model to obtain the second attention feature. The second attention feature and the fourth feature vector are then input into the residual network connected to the second self-attention model, added together, and then standardized by Layer normalization to output the sixth feature vector.

[0139] Similarly, for text data, refer to sub-steps S103 and S104, step S107, and... Figure 2 As shown, the first feature vector is calculated through the first cross-attention sub-model. The first feature vector is then concatenated with the text feature vector using an add operation, and then standardized using layer normalization to obtain the third feature vector. This third feature vector is then processed using the first self-attention sub-model to obtain the first attention feature. Finally, the first attention feature and the third feature vector are concatenated using an add operation, and then standardized using layer normalization to obtain the fifth feature vector.

[0140] After obtaining the fifth and sixth feature vectors, a normalized classification prediction is performed using a fully connected network (FN). Specifically, the FN includes a fully connected layer and a softmax layer. The fifth and sixth feature vectors are input into the fully connected layer for normalization, and the normalized result is then input into the softmax layer for prediction to obtain the emotion category. At this point, referring to step S108, service information is fed back based on the emotion category. Since the service information itself is set for the emotion category, as the accuracy of emotion category recognition improves, the service information can help alleviate emotions and enhance positive emotions, thereby improving the service experience.

[0141] Therefore, refer to Figures 1 to 3 The illustrated embodiment employs multimodal fusion technology, fully integrating the features of text and speech in emotion recognition and leveraging the advantages of both models. Furthermore, the fusion technology utilizes a pure attention mechanism, which is simple to implement, yields significant results, and allows text and speech to fully integrate and learn each other's features based on their respective models. Secondly, self-attention processing is applied to each vector after fusion, ensuring thorough learning of its own vectors. Multiple residual network calculations are employed to increase the information carried and avoid gradient vanishing and network learning difficulties. Therefore, the embodiment of this application improves the performance of emotion recognition.

[0142] It should be noted that the emotion recognition method of this application can be applied to, for example... Figure 4The system shown includes a target terminal 100, a server 200, and a service terminal 300. The target terminal 100 is used for external interaction and to acquire external data. The server 200 is communicatively connected to the service terminal 300 and is used to execute the aforementioned emotion recognition method. The service terminal 300 is used to display the emotion recognition results and feed back service information to the server 200 for display on the target terminal 100. In some embodiments, a service plan is recommended based on the emotion recognition results. Once the service terminal 300 determines the service plan, it generates service information based on the plan and feeds it back to the target terminal 100. The target terminal 100 and the service terminal 300 are in one-to-one correspondence. The target terminal 100 and the service terminal 300 can be handheld terminals or electronic devices such as computers. The server 200 includes a database, a processor, etc.

[0143] It should be noted that the emotion recognition method of this application can also be applied to, for example, Figure 5 The system shown includes a target terminal 100 and a service analysis platform 400. The target terminal 100 is used for external interaction. The emotion recognition method runs on either the target terminal 100 or the service analysis platform 400. Multiple target terminals 100 are communicatively connected to the same service analysis platform 400. For example, when the emotion recognition method is executed on the target terminal 100, a service plan is automatically determined based on the confirmed emotion category, and service information is fed back according to the service plan. At this time, the target terminal 100 feeds back the emotion category, the data to be recognized for speech recognition, the service information, and the response result to the service information to the service analysis platform 400. The service analysis platform 400 collects data over a period of time, summarizes and analyzes it, and then readjusts the service plan and loads it onto the target terminal 100 to achieve further intelligent service.

[0144] In conclusion, by using computer programs to assist in the analysis of speech data to be recognized, the accuracy of emotion category recognition can be improved, thereby providing more accurate intelligent solutions or better services, and thus enhancing the service experience.

[0145] Please see Figure 6 This application also provides an emotion recognition device that can implement the above-described emotion recognition method. The device includes:

[0146] Interaction module 510, which is used to acquire the voice data to be recognized from the target terminal;

[0147] The feature acquisition module 520 is used to acquire the speech feature vector and text feature vector corresponding to the speech data to be recognized.

[0148] The text acquisition module 530 is used to acquire the first key value feature vector and the first value item feature vector corresponding to the text feature vector, and the first query item feature vector corresponding to the speech feature vector;

[0149] The text processing module 540 is used to calculate the first feature vector based on the first key value feature vector, the first value item feature vector, and the first query item feature vector.

[0150] The voice acquisition module 550 is used to acquire the second query item feature vector corresponding to the text feature vector, the second key value feature vector corresponding to the voice feature vector, and the second value item feature vector.

[0151] The voice processing module 560 is used to calculate the second feature vector based on the second key value feature vector, the second value item feature vector, and the second query item feature vector.

[0152] The determination module 570 is used to perform emotion classification processing on the first feature vector and the second feature vector to obtain the emotion category corresponding to the speech data to be recognized.

[0153] Application module 580 is used to feed back service information corresponding to the emotion category to the target terminal.

[0154] The specific implementation of this emotion recognition device is basically the same as the specific implementation of the emotion recognition method described above, and will not be repeated here.

[0155] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described emotion recognition method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0156] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0157] The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0158] The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called by the processor 601 to execute the emotion recognition method of the embodiments of this application.

[0159] The input / output interface 603 is used to implement information input and output;

[0160] The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0161] Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604);

[0162] The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0163] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described emotion recognition method.

[0164] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] The emotion recognition method, apparatus, electronic device, and storage medium provided in this application embodiment determine, respectively, a first key-value feature vector, a first value-item feature vector, and a first query-item feature vector related to the text feature vector, and a first query-item feature vector, a second key-value feature vector, and a second value-item feature vector related to the speech feature vector, through text feature vectors and speech feature vectors. This allows the first feature vector to fully learn the advantages of speech, and the second feature vector to fully learn the advantages of text. When the first and second feature vectors are fused, the advantages of the two different directions are combined, thereby improving the accuracy of emotion category recognition. At this time, corresponding service information is provided based on the recognized emotion category. Since the service information is generated in response to the emotions of the target object, it can alleviate negative emotions and enhance positive emotions, thereby improving the service experience.

[0166] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0167] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0170] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0171] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0173] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for emotion recognition, characterized in that, The method includes: Acquire the speech data to be recognized from the target terminal; Obtain the speech feature vector and text feature vector corresponding to the speech data to be recognized; Obtain the first key-value feature vector and the first value item feature vector corresponding to the text feature vector, and the first query item feature vector corresponding to the speech feature vector; The first feature vector is calculated based on the first key value feature vector, the first value item feature vector, and the first query item feature vector; Obtain the second query item feature vector corresponding to the text feature vector, the second key value feature vector corresponding to the speech feature vector, and the second value item feature vector; The second feature vector is calculated based on the second key value feature vector, the second value item feature vector, and the second query item feature vector; The first feature vector and the second feature vector are subjected to emotion classification processing to obtain the emotion category corresponding to the speech data to be identified. The service information corresponding to the emotion category will be fed back to the target terminal.

2. The emotion recognition method according to claim 1, characterized in that, The step of obtaining the first key-value feature vector and the first value-item feature vector corresponding to the text feature vector, and the first query-item feature vector corresponding to the speech feature vector, includes: The first key-value feature vector and the first value-term feature vector are obtained by performing matrix operations on the text feature vector based on the first weight set using a preset first cross-attention sub-model. Based on the first weight set, matrix operations are performed on the speech feature vector to obtain the first query item feature vector; Correspondingly, the step of calculating the first feature vector based on the first key-value feature vector, the first value item feature vector, and the first query item feature vector includes: The first key-value feature vector, the first value-item feature vector, and the first query-item feature vector are input into the first cross-attention sub-model, so that the first cross-attention sub-model calculates the first feature vector based on the first attention formula.

3. The emotion recognition method according to claim 2, characterized in that, The step of obtaining the second query term feature vector corresponding to the text feature vector, the second key-value feature vector corresponding to the speech feature vector, and the second value term feature vector includes: The speech feature vector is subjected to matrix operations using the second weight set in the preset second cross-attention sub-model to obtain the second key-value feature vector and the first value term feature vector. The second query item feature vector is obtained by performing matrix operations on the text feature vector using the second weight set. Correspondingly, the step of calculating the second feature vector based on the second key-value feature vector, the second value-item feature vector, and the second query-item feature vector includes: The second key-value feature vector, the second value-item feature vector, and the second query-item feature vector are input into the second attention formula in the second cross-attention sub-model to calculate the second feature vector.

4. The emotion recognition method according to claim 1, characterized in that, Emotion classification processing is performed on the first feature vector and the second feature vector to obtain the emotion category corresponding to the speech data to be recognized, including: The third feature vector is obtained by performing residual calculation on the first feature vector and the speech feature vector; The fourth feature vector is obtained by performing residual calculation on the second feature vector and the text feature vector; The third feature vector and the fourth feature vector are respectively subjected to feature enhancement to obtain a fifth feature vector corresponding to the third feature vector and a sixth feature vector corresponding to the fourth feature vector; The fourth feature vector and the sixth feature vector are fused and classified to obtain the emotion category corresponding to the speech data to be identified.

5. The emotion recognition method according to claim 4, characterized in that, The step of performing feature enhancement on the third feature vector and the fourth feature vector respectively to obtain a fifth feature vector corresponding to the third feature vector and a sixth feature vector corresponding to the fourth feature vector includes: The fifth feature vector is obtained by performing attention calculation on the third feature vector through a preset first self-attention mechanism; The fourth feature vector is obtained by performing attention calculation on the fourth feature vector through a preset second self-attention mechanism.

6. The emotion recognition method according to claim 5, characterized in that, The step of performing attention calculation on the third feature vector through a preset first self-attention mechanism to obtain the fifth feature vector includes: The third feature vector is input into a preset first self-attention sub-model to obtain the first attention feature; The fifth feature vector is obtained by performing residual calculation on the first attention feature; The step of performing attention calculation on the fourth feature vector through a preset second self-attention mechanism to obtain the sixth feature vector includes: The fourth feature vector is input into a preset second self-attention sub-model to obtain the second attention feature; The sixth feature vector is obtained by performing residual calculation on the second attention feature.

7. The emotion recognition method according to claim 4, characterized in that, The step of fusing and classifying the fourth feature vector and the sixth feature vector to obtain the emotion category corresponding to the speech data to be identified includes: The fourth feature vector and the sixth feature vector are input into a preset fully connected layer to calculate the fused feature vector; The fused feature vector is classified and predicted by a preset logistic regression layer to obtain the emotion category.

8. An emotion recognition device, characterized in that, The device includes: An interaction module, which is used to acquire voice data to be recognized from the target terminal; A feature acquisition module, wherein the acquisition module is used to acquire the speech feature vector and text feature vector corresponding to the speech data to be recognized; The text acquisition module is used to acquire the first key-value feature vector and the first value item feature vector corresponding to the text feature vector, and the first query item feature vector corresponding to the speech feature vector; A text processing module is configured to calculate a first feature vector based on the first key-value feature vector, the first value item feature vector, and the first query item feature vector. The voice acquisition module is used to acquire the second query item feature vector corresponding to the text feature vector, the second key value feature vector and the second value item feature vector corresponding to the voice feature vector; A voice processing module is used to calculate a second feature vector based on the second key-value feature vector, the second value item feature vector, and the second query item feature vector; A determination module is used to perform emotion classification processing on the first feature vector and the second feature vector to obtain the emotion category corresponding to the speech data to be recognized. An application module is used to feed back service information corresponding to the emotion category to the target terminal.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the emotion recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the emotion recognition method according to any one of claims 1 to 7.

Images