Emotion recognition method and device, electronic equipment and storage medium

Through the dual-modal fusion model of speech and text and the LSTM recurrent neural network structure, the emotional information of bank tellers in the process of financial business is identified, and the problem of poor emotional recognition effect in the existing technology is solved, and the accuracy of emotional recognition is achieved. Personalized features are retained, suitable for the real-time needs of bank service scenarios.

CN119049516APending Publication Date: 2024-11-29中国邮政储蓄银行股份有限公司
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411263010.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

It is difficult to effectively identify the emotional information of bank tellers during financial business processing, which affects the success rate and customer experience of business processing.

Method used

The pre-trained speech emotion recognition model and text emotion recognition model are adopted, combined with the LSTM recurrent neural network structure and the bimodal fusion model, and the emotional characteristics of speech and text are extracted through the bimodal fusion of speech text, and the attention mechanism is used for weighting. Integrate to identify the phonological sequence emotional information in financial business scenarios.

Benefits of technology

It realizes accurate identification of bank teller emotions, improves the accuracy of identification results and personalized feature retention, which is suitable for the real-time needs of bank branch service processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049516A_ABST
    Figure CN119049516A_ABST
Patent Text Reader

Abstract

The invention discloses an emotion recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: recognizing voice emotion information and text emotion information based on a financial business handling scene; according to the speech emotion information and the text emotion information, a bimodal fusion result is obtained through a pre-trained speech text bimodal fusion model, and the pre-trained speech text bimodal fusion model is used for respectively receiving output results of the speech emotion recognition model and the text emotion recognition model; and obtaining an emotion recognition result of the voice sequence in the financial business handling scene according to the bimodal fusion result and an LSTM recurrent neural network structure. On one hand, emotion recognition is carried out on the bank teller service process, and on the other hand, the accuracy of the recognition result is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of emotion recognition technology, and in particular to an emotion recognition method, device, electronic device, and storage medium. Background Art

[0002] Speech emotion recognition technology is a technology that recognizes the speaker's emotions through speech. The main research content of this technology includes the extraction of emotional features and the construction of a suitable emotion recognition model.

[0003] Bank tellers communicate directly with customers and mainly handle deposit, withdrawal or loan transactions. The teller's emotions can largely determine the customer's business experience during the service process, and directly or indirectly determine the success rate of the business handling. Summary of the invention

[0004] The embodiments of the present application provide an emotion recognition method, device, electronic device, and storage medium to realize emotion recognition during the bank teller service process.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an emotion recognition method, wherein the method comprises:

[0007] Based on the financial business handling scenario, recognize voice emotion information and text emotion information, wherein the voice emotion information is obtained by using a pre-trained voice emotion recognition model, and the text emotion information is obtained by using a pre-trained text emotion recognition model;

[0008] According to the speech emotion information and the text emotion information, a bimodal fusion result is obtained through a pre-trained speech-text bimodal fusion model, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively;

[0009] According to the bimodal fusion result and the LSTM recurrent neural network structure, the emotion recognition result of the speech sequence in the financial business processing scenario is obtained.

[0010] In some embodiments, obtaining the emotion recognition result of the speech sequence in the financial business handling scenario according to the bimodal fusion result and the LSTM recurrent neural network structure includes:

[0011] Introduce LSTM recurrent neural network structure between different recognition windows;

[0012] Extract paragraph sequence information according to the LSTM recurrent neural network structure, and use the output of the last neuron in the hidden layer of the LSTM as a sequence feature vector;

[0013] At the feature level, the sequence feature vector is concatenated and fused with the speech emotion feature vector extracted by the convolutional neural network CNN in the bimodal fusion result.

[0014] In some embodiments, the bimodal fusion result is obtained through a pre-trained speech-text bimodal fusion model based on the speech emotion information and the text emotion information, and the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively, including:

[0015] Use a globally unified learnable query vector to extract the emotional features of speech and text;

[0016] According to the weighted method of attention mechanism, fusion is performed at the decision level to extract global common features to obtain weighted global features;

[0017] The weighted global features are concatenated with the original speech and text features to obtain a bimodal fusion result.

[0018] In some embodiments, the bimodal fusion result is obtained through a pre-trained speech-text bimodal fusion model based on the speech emotion information and the text emotion information, and the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively, including:

[0019] The last fully connected layer of the network in the speech emotion recognition model and the last fully connected layer of the network in the text emotion recognition model are output respectively for weighted fusion.

[0020] In some embodiments, the bimodal fusion result is obtained through a pre-trained speech-text bimodal fusion model based on the speech emotion information and the text emotion information, and the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively, including:

[0021] The feature quantities of the two modalities of speech and text are transformed through the same fully connected layer to obtain the key;

[0022] The weight is obtained by calculating the inner product between the query vector and each key, and then normalized using the softmax function:

[0023] The weighted sum of the weight and the corresponding key is used to obtain the multimodal feature, and the multimodal feature is concatenated with the features of the two modalities and the multimodal feature as the final decision vector.

[0024] In some embodiments, the pre-trained speech emotion recognition model includes: a convolutional neural network (CNN).

[0025] In some embodiments, the pre-trained text emotion recognition model includes: a Bert pre-trained language model.

[0026] In a second aspect, an embodiment of the present application further provides an emotion recognition device, wherein the device comprises:

[0027] A recognition module, used to recognize voice emotion information and text emotion information based on a financial business handling scenario, wherein the voice emotion information is obtained by recognizing a pre-trained voice emotion recognition model, and the text emotion information is obtained by recognizing a pre-trained text emotion recognition model;

[0028] A fusion module is used to obtain a bimodal fusion result according to the speech emotion information and the text emotion information through a pre-trained speech-text bimodal fusion model, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively;

[0029] The result module is used to obtain the emotion recognition result of the speech sequence in the financial business processing scenario based on the bimodal fusion result and the LSTM recurrent neural network structure.

[0030] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the above method.

[0031] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0032] At least one of the above technical solutions adopted in the embodiment of the present application can achieve the following beneficial effects: based on the financial business handling scenario, voice emotion information and text emotion information are identified, the voice emotion information is obtained by a pre-trained voice emotion recognition model, and the text emotion information is obtained by a pre-trained text emotion recognition model. By identifying the voice emotion information and the text emotion information, voice and text bimodal fusion is performed later. According to the voice emotion information and the text emotion information, a bimodal fusion result is obtained through a pre-trained voice and text bimodal fusion model, and the pre-trained voice and text bimodal fusion model is used to receive the output results of the voice emotion recognition model and the text emotion recognition model respectively, which can retain the emotional personality characteristics of different people. According to the bimodal fusion result and the LSTM recurrent neural network structure, the emotion recognition result of the voice sequence in the financial business handling scenario is obtained, and the emotional trend characteristics of the detected person can be learned, and the continuity of human emotional trends can also be used to improve the accuracy of the recognition result. The above method can better identify the emotional information of bank tellers in the process of handling business. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0034] Figure 1 This is a flow chart of the emotion recognition method in the embodiment of the present application;

[0035] Figure 2 A schematic diagram of a network structure for emotion recognition of a speech sequence in an emotion recognition method in an embodiment of the present application;

[0036] Figure 3 Schematic diagram of the network structure of the speech-text bimodal fusion model in the emotion recognition method in the embodiment of the present application;

[0037] Figure 4 This is a schematic diagram of the structure of the emotion recognition device in the embodiment of the present application;

[0038] Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0040] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0041] The present application embodiment provides an emotion recognition method, such as Figure 1 As shown, a flow chart of the emotion recognition method in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S130:

[0042] Step S110, based on the financial business processing scenario, recognize voice emotion information and text emotion information, the voice emotion information is obtained by recognizing through a pre-trained voice emotion recognition model, and the text emotion information is obtained by recognizing through a pre-trained text emotion recognition model.

[0043] After extracting speech features, the extracted feature vectors are input frame by frame into the constructed speech emotion recognition model. It can be understood that the speech emotion recognition model can be trained by machine learning using a common network in related technologies. The text data of the speech is obtained through the speech recognition API, and the text data is processed by word segmentation, encoding, etc., and then input into the pre-trained text emotion recognition model for recognition.

[0044] It can be understood that the training process of the speech emotion recognition model and the text emotion recognition model can be obtained by training with a pre-collected training set.

[0045] Step S120, according to the speech emotion information and the text emotion information, a bimodal fusion result is obtained through a pre-trained speech-text bimodal fusion model, and the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively.

[0046] The outputs of the last fully connected layer of the network of the speech emotion recognition module and the last fully connected layer of the network of the text emotion recognition module are weighted fused and then input into the softmax layer for category prediction. During the training process, the gradient will update the networks of the two modules at the same time. The decision vector output by the pre-trained speech-text bimodal fusion model is spliced ​​with the original speech and text features, retaining the personalized emotional characteristics of different people. By combining the two types of features, the effect of emotion recognition is improved.

[0047] Step S130, obtaining the emotion recognition result of the speech sequence in the financial business processing scenario according to the bimodal fusion result and the LSTM recurrent neural network structure.

[0048] In view of the continuity of speech in the service process of bank branches, according to the dual-modal fusion results and the LSTM recurrent neural network structure, the paragraph sequence information is fused on the basis of the jointly trained decision-level modal fusion model, and the LSTM recurrent neural network structure is introduced. It can learn the emotional trend characteristics of the person being detected, and at the same time, it can also use the continuity of human emotional trends to improve the accuracy of the recognition results.

[0049] Through the above method, according to the speech emotion information and the text emotion information, a pre-trained speech-text bimodal fusion model is used to obtain a bimodal fusion result. The global common features of different speakers can be better extracted, and the original features of each modality spliced ​​after fusion can better retain the personalized features of each speaker, thereby making the fusion method more comprehensive.

[0050] Through the above method, based on the bimodal fusion result and the LSTM recurrent neural network structure, the emotion recognition result of the speech sequence in the financial business handling scenario is obtained. Specifically, by introducing the cross-modal attention mechanism, the weight distribution of the two modal information is automatically learned in the weighted manner, so that the contribution of each modality's decision vector to the final prediction category can be actively adjusted according to their respective characteristics. The original speech and text features are spliced ​​in the final decision vector, retaining the personalized emotional characteristics of different people. By combining the two types of features, the effect of emotion recognition is improved.

[0051] Through the above method, taking the teller's voice data as the entry point, fully considering the strong correlation and complementarity between voice and text modalities, as well as the impact of context on emotion, integrating voice information, text information, and sequence information in the emotion recognition model, it is possible to accurately identify the teller's emotions during the service process, provide data reference for the teller's service attitude evaluation, and help improve the quality of counter service.

[0052] In one embodiment of the present application, the emotion recognition result of the speech sequence in the financial business processing scenario is obtained based on the bimodal fusion result and the LSTM recurrent neural network structure, including: introducing an LSTM recurrent neural network structure between different recognition windows; extracting paragraph sequence information according to the LSTM recurrent neural network structure, and taking the output of the last neuron in the hidden layer of the LSTM as the sequence feature vector; splicing and fusing the sequence feature vector with the speech emotion feature vector extracted by the convolutional neural network CNN in the bimodal fusion result at the feature level.

[0053] like Figure 2 As shown, it is a schematic diagram of the model structure of the fusion sequence information. In view of the continuity characteristics of speech in the service process of bank branches, a speech emotion recognition model that integrates speech, text and sequence information is provided in the embodiment of the present application. The paragraph sequence information is integrated on the basis of the jointly trained decision-level modal fusion model. Specifically, the fusion sequence information model is composed of three basic model blocks: CNN, Bert and LSTM. CNN and Bert can adopt the model structure commonly used in related technologies. On this basis, the LSTM recurrent neural network structure is introduced, which can learn the emotional trend characteristics of the detected person, and can also use the continuity of human emotional trends to improve the accuracy of the recognition results.

[0054] It can be understood that the pre-trained speech-text bimodal fusion model includes CNN, BERT and LSTM recurrent neural network structures, and they are trained together during the training phase.

[0055] The LSTM model consists of a unidirectional LSTM layer with 32 neurons. Within each window, the speech features extracted by CNN are concatenated with the hidden layer of LSTM at the feature level, and then three fully connected layers (containing 512, 64, and 4 neurons respectively) are used to obtain speech emotion features containing temporal information. The fusion features are weightedly fused with the output of the fully connected layer network and the deep text emotion features extracted by Bert and three fully connected layers (containing 256, 64, and 4 neurons respectively) to obtain the speech-text multimodal emotion features of the frame. The features are output to the sofmax layer for category prediction to obtain the emotion recognition result of this window.

[0056] It should be noted that between different windows, the emotion recognition results predicted by each window need to be encoded as the input of the next LSTM cell and used as the input of the emotion recognition model of the next window.

[0057] It can be understood that the individual data of neurons in the above-mentioned fully connected layer are only examples and are not used to limit the protection scope of the present application.

[0058] It should be noted that the LSTM recurrent neural network structure is a unidirectional network, which is modeled sequentially from front to back. Unlike Transformer, which is a bidirectional network, both the previous and the following text are taken into account. The speech recognition scenario in the embodiment of the present application is a scenario that has a strong dependence on the time sequence. That is to say, when each sentence is spoken, the model only knows what has been said before, and cannot know the sentence that will appear next. Therefore, the LSTM sequence information module introduced in the example of the present application cannot be replaced by Transformer.

[0059] In addition, emotion recognition during bank counter service is a task with high real-time requirements. The LSTM-based sequence model proposed in the embodiment of the present application has better real-time performance, less computational complexity, and lower hardware requirements than Transformer, so it is more suitable for bank branch service scenarios.

[0060] In one embodiment of the present application, the bimodal fusion result is obtained based on the speech emotion information and the text emotion information through a pre-trained speech-text bimodal fusion model, and the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively, including: using a globally unified learnable query vector to extract the emotion features of speech and text; fusing at the decision level according to the weighted method of the attention mechanism, extracting global common features to obtain weighted global features; splicing the weighted global features with the original speech and text features to obtain a bimodal fusion result.

[0061] Multimodal fusion at the decision-making level is one of the current mainstream multimodal fusion methods. After making a decision based on the feature information of each modality, it uses manually set rules to fuse the decisions, which can make full use of the unique characteristics of each modality. However, deep learning can automatically learn the connotation of the weight distribution of the two modal information. Figure 3 As shown, in an embodiment of the present application, a decision-level fusion method for joint training is proposed, in which the outputs of the last fully connected layer of the network of the speech emotion recognition module and the last fully connected layer of the network of the text emotion recognition module are weightedly fused and then input into the softmax layer for category prediction. During the training process, the gradient will update the networks of the two modules at the same time.

[0062] The attention mechanism used is a technology that allows the model to focus on important information and fully learn and absorb it. The embodiments of this application summarize that the attention mechanism is introduced in the process of decision-level fusion, so that when two decision vectors (i.e., the 4-dimensional outputs of the two fully connected layers connected by CNN and Bert in the model) are weighted fused, the weight is generated based on the category, so that the model automatically learns which decision outputs of the two models should be focused on respectively, so that their contribution to the final prediction category can be actively adjusted according to their respective characteristics.

[0063] It can be understood that the softmax layer is only an example, and those skilled in the art may use other activation functions, and it is not used to limit the protection scope of the present application.

[0064] In one embodiment of the present application, the bimodal fusion result is obtained based on the speech emotion information and the text emotion information through a pre-trained speech-text bimodal fusion model, and the pre-trained speech-text bimodal fusion model is used to respectively receive the output results of the speech emotion recognition model and the text emotion recognition model, including: outputting the last fully connected layer of the network in the speech emotion recognition model and the last fully connected layer of the network in the text emotion recognition model respectively for weighted fusion.

[0065] like Figure 3 As shown in the figure, the outputs of the last fully connected layer of the network of the speech emotion recognition module and the last fully connected layer of the network of the text emotion recognition module are weighted fused and then input into the softmax layer for category prediction. During the training process, the gradient will update the networks of the two modules at the same time.

[0066] In one embodiment of the present application, the bimodal fusion result is obtained based on the speech emotion information and the text emotion information through a pre-trained speech-text bimodal fusion model, and the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively, including: transforming the feature quantities of the two modalities of speech and text through the same fully connected layer to obtain a key; calculating the inner product based on the query vector and each key to obtain the weight, and normalizing it with the softmax function: weighted summing the weight and the corresponding key to obtain a multimodal feature, and concatenating the multimodal feature with the features of each of the two modalities and the multimodal feature as the final decision vector.

[0067] Through the globally unified learnable query vector, the common emotional features of different people's speech and pronunciation can be extracted; the cross-modal attention weighting is introduced to automatically learn the weight distribution of the two modal information, so that the contribution of each modality's decision vector to the final prediction category can be actively adjusted according to their own characteristics; the final decision vector is spliced ​​with the original speech and text features, retaining the personalized emotional features of different people. By combining the two types of features, the effect of emotion recognition is improved.

[0068] Please continue to refer to Figure 3 In the implementation process, a learnable embedding can be used as a query, and the feature quantities of the two modalities of speech and text can be transformed through the same fully connected layer to obtain the key.

[0069] key audio =FullConnection(feature audio )

[0070] keytext =FullConnection(feature text )

[0071] Calculate the inner product of the query and each key to get the weight, and normalize it with the softmax function:

[0072]

[0073]

[0074] Then, the weights and corresponding keys are weighted and summed to obtain the multimodal features, which are then concatenated with the features of the two modalities as the final decision vector:

[0075] feature multi_modal =w audio ·key audio +w text ·key text

[0076] feature=concat([feature multi_modal , feature audio , feature text ])

[0077] Although there are similar methods in the multimodal fusion solution, such as artificial experience fusion and Transformer fusion, compared with artificial fusion, the method in the embodiment of the present application is more flexible and intelligent, and can automatically learn the weight distribution of the two modal information, and can selectively use voice or text data for different voice data.

[0078] In one embodiment of the present application, the pre-trained speech emotion recognition model includes: a convolutional neural network CNN.

[0079] In order to obtain short speech data, the speech waveform data is first preprocessed, including framing, windowing, pre-emphasis, resampling, breakpoint detection and other steps. Then, referring to the ComparE speech emotion feature set, the opensmile tool is used to extract the 130-dimensional frame-level vector (LLDs) of each frame of speech data, and the frame-level feature vector is processed by Splice, padding and normalization, and the extracted feature vector is input into the constructed speech emotion recognition model frame by frame.

[0080] The speech emotion recognition model consists of 4 convolution blocks and 3 fully connected layers. One of the convolution blocks consists of one convolution layer and one pooling layer. Take the first convolution block as an example. The first layer of the convolution block is a convolution layer, denoted as C1, which consists of 8 feature maps with a size of 50×130 and a convolution kernel size of 3×3. C1 has 80 trainable parameters (3×3=9 bunit parameters and one bias parameter for each filter, a total of 8 filters), with a total of 80×(50×130)=520000 connections. The second layer is a pooling layer S2, which has 8 feature maps of size 50×130, a pooling kernel size of 2×2, and a pooling method of maximum pooling. The maximum element in each region is extracted, and the size of the feature map after downsampling is 65×25. The third layer is an activation layer, and the activation function is the Relu function, which transforms each pixel value according to the Relu function value. The second convolutional layer consists of 32 feature maps, the third convolutional layer consists of 64 feature maps, and the fourth convolutional layer consists of 128 feature maps. The convolution kernel size and activation function of the pooling layer remain unchanged. Finally, three fully connected layers are connected. Each fully connected layer is connected to each neuron in the previous layer. The previous layer of the first fully connected layer has 3072 neurons, and the first fully connected layer has 512 neurons. Therefore, the first fully connected layer F1 has 3072×512 trainable parameters; the second fully connected layer F2 has 64 neurons, so F2 has 512×64 trainable parameters; the third fully connected layer F3 has 4 neurons, so F3 has 64×4 trainable parameters.

[0081] Optionally, the input ComparE speech emotion feature set of the pre-trained speech emotion recognition model can also be replaced by other feature extraction methods, which can be freely selected according to the business scenario. This feature set does not fall within the protection scope required by the present invention.

[0082] Alternatively, the pre-trained speech emotion recognition model can also use neural networks of other structures (such as DNN). However, after experiments, it was found that when the total number of parameters remains unchanged, CNN has the best effect due to its advantages such as local perception and translation invariance.

[0083] In one embodiment of the present application, the pre-trained text emotion recognition model includes: a Bert pre-trained language model.

[0084] The text data of the speech is obtained through the speech recognition API, the text data is segmented and encoded, and the word encoding and position encoding are used as the input of the constructed text emotion recognition model. The constructed text emotion recognition model uses the Bert model to extract the deep emotion features of the text. Three fully connected layers are added after the [CLS] bit output of Bert. The first layer contains 256 neurons, the second layer contains 64 neurons, and the third layer has 4 neurons. Finally, it is input to the softmax layer for category prediction.

[0085] Bert is a commonly used pre-trained language model. Pre-training is a concept of transfer learning, which uses huge data to train a model with strong generalization ability. When it needs to be used in a specific scenario, it only needs to simply modify some output layers, and then use its own data for incremental training and make a slight adjustment to the weights. Bert is implemented based on Transformer, using the Transformer Encoder structure to encode words to extract sentence information.

[0086] The text emotion recognition model can also use other more advanced Bert variant models, which may achieve better recognition accuracy, but the network computing time will also increase significantly. Considering that speech emotion recognition is a scenario with high real-time requirements, the Bert model was selected after comprehensively weighing the effect-performance.

[0087] The embodiment of the present application also provides an emotion recognition device 400, such as Figure 4 As shown, a schematic diagram of the structure of an emotion recognition device in an embodiment of the present application is provided, wherein the emotion recognition device 400 comprises at least: a recognition module 410, a fusion module 420 and a result module 430, wherein:

[0088] In one embodiment of the present application, the recognition module 410 is specifically used to: based on the financial business processing scenario, identify voice emotion information and text emotion information, the voice emotion information is obtained by recognizing a pre-trained voice emotion recognition model, and the text emotion information is obtained by recognizing a pre-trained text emotion recognition model.

[0089] After extracting speech features, the extracted feature vectors are input frame by frame into the constructed speech emotion recognition model. It can be understood that the speech emotion recognition model can be trained by machine learning using a common network in related technologies. The text data of the speech is obtained through the speech recognition API, and the text data is processed by word segmentation, encoding, etc., and then input into the pre-trained text emotion recognition model for recognition.

[0090] It can be understood that the training process of the speech emotion recognition model and the text emotion recognition model can be obtained by training with a pre-collected training set.

[0091] In one embodiment of the present application, the fusion module 420 is specifically used to: obtain a bimodal fusion result based on the speech emotion information and the text emotion information through a pre-trained speech-text bimodal fusion model, and the pre-trained speech-text bimodal fusion model is used to respectively receive the output results of the speech emotion recognition model and the text emotion recognition model.

[0092] The outputs of the last fully connected layer of the network of the speech emotion recognition module and the last fully connected layer of the network of the text emotion recognition module are weighted fused and then input into the softmax layer for category prediction. During the training process, the gradient will update the networks of the two modules at the same time. The decision vector output by the pre-trained speech-text bimodal fusion model is spliced ​​with the original speech and text features, retaining the personalized emotional characteristics of different people. By combining the two types of features, the effect of emotion recognition is improved.

[0093] In one embodiment of the present application, the result module 430 is specifically used to obtain the emotion recognition result of the speech sequence in the financial business processing scenario based on the bimodal fusion result and the LSTM recurrent neural network structure.

[0094] In view of the continuity of speech in the service process of bank branches, according to the dual-modal fusion results and the LSTM recurrent neural network structure, the paragraph sequence information is fused on the basis of the jointly trained decision-level modal fusion model, and the LSTM recurrent neural network structure is introduced. It can learn the emotional trend characteristics of the person being detected, and at the same time, it can also use the continuity of human emotional trends to improve the accuracy of the recognition results.

[0095] In one embodiment of the present application, the result module 430 is also used to

[0096] Introduce LSTM recurrent neural network structure between different recognition windows;

[0097] Extract paragraph sequence information according to the LSTM recurrent neural network structure, and use the output of the last neuron in the hidden layer of the LSTM as a sequence feature vector;

[0098] At the feature level, the sequence feature vector is concatenated and fused with the speech emotion feature vector extracted by the convolutional neural network CNN in the bimodal fusion result.

[0099] In one embodiment of the present application, the fusion module 420 is also used to

[0100] Use a globally unified learnable query vector to extract the emotional features of speech and text;

[0101] According to the weighted method of attention mechanism, fusion is performed at the decision level to extract global common features to obtain weighted global features;

[0102] The weighted global features are concatenated with the original speech and text features to obtain a bimodal fusion result.

[0103] In one embodiment of the present application, the fusion module 420 is further used to output the last fully connected layer of the network in the speech emotion recognition model and the last fully connected layer of the network in the text emotion recognition model respectively for weighted fusion.

[0104] In one embodiment of the present application, the fusion module 420 is also used to

[0105] The feature quantities of the two modalities of speech and text are transformed through the same fully connected layer to obtain the key;

[0106] The weight is obtained by calculating the inner product between the query vector and each key, and then normalized using the softmax function:

[0107] The weighted sum of the weight and the corresponding key is used to obtain the multimodal feature, and the multimodal feature is concatenated with the features of the two modalities and the multimodal feature as the final decision vector.

[0108] In one embodiment of the present application, the pre-trained speech emotion recognition model includes: a convolutional neural network CNN.

[0109] In one embodiment of the present application, the pre-trained text emotion recognition model includes: a Bert pre-trained language model.

[0110] It can be understood that the above-mentioned emotion recognition device can implement the various steps of the emotion recognition method provided in the aforementioned embodiment, and the relevant explanations about the emotion recognition method are applicable to the emotion recognition device and will not be repeated here.

[0111] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0112] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0114] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an emotion recognition device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0115] Based on the financial business handling scenario, recognize voice emotion information and text emotion information, wherein the voice emotion information is obtained by using a pre-trained voice emotion recognition model, and the text emotion information is obtained by using a pre-trained text emotion recognition model;

[0116] According to the speech emotion information and the text emotion information, a bimodal fusion result is obtained through a pre-trained speech-text bimodal fusion model, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively;

[0117] According to the bimodal fusion result and the LSTM recurrent neural network structure, the emotion recognition result of the speech sequence in the financial business processing scenario is obtained.

[0118] The above application Figure 1 The method performed by the emotion recognition device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0119] The electronic device may also perform Figure 1 The method executed by the emotion recognition device in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0120] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the emotion recognition device in the illustrated embodiment is specifically used to perform:

[0121] Based on the financial business handling scenario, recognize voice emotion information and text emotion information, wherein the voice emotion information is obtained by using a pre-trained voice emotion recognition model, and the text emotion information is obtained by using a pre-trained text emotion recognition model;

[0122] According to the speech emotion information and the text emotion information, a bimodal fusion result is obtained through a pre-trained speech-text bimodal fusion model, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively;

[0123] According to the bimodal fusion result and the LSTM recurrent neural network structure, the emotion recognition result of the speech sequence in the financial business processing scenario is obtained.

[0124] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0133] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for emotion recognition, wherein: The method comprises: Based on the financial business handling scenario, recognize voice emotion information and text emotion information, wherein the voice emotion information is obtained by using a pre-trained voice emotion recognition model, and the text emotion information is obtained by using a pre-trained text emotion recognition model; According to the speech emotion information and the text emotion information, a bimodal fusion result is obtained through a pre-trained speech-text bimodal fusion model, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively; According to the bimodal fusion result and the LSTM recurrent neural network structure, the emotion recognition result of the speech sequence in the financial business processing scenario is obtained.

2. The method of claim 1, wherein: The emotion recognition result of the speech sequence in the financial business handling scenario is obtained according to the bimodal fusion result and the LSTM recurrent neural network structure, including: Introduce LSTM recurrent neural network structure between different recognition windows; Extract paragraph sequence information according to the LSTM recurrent neural network structure, and use the output of the last neuron in the hidden layer of the LSTM as a sequence feature vector; At the feature level, the sequence feature vector is concatenated and fused with the speech emotion feature vector extracted by the convolutional neural network CNN in the bimodal fusion result.

3. The method of claim 1, wherein: The method of obtaining a bimodal fusion result through a pre-trained speech-text bimodal fusion model according to the speech emotion information and the text emotion information, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively, includes: Use a globally unified learnable query vector to extract the emotional features of speech and text; According to the weighted method of attention mechanism, fusion is performed at the decision level to extract global common features to obtain weighted global features; The weighted global features are concatenated with the original speech and text features to obtain a bimodal fusion result.

4. The method of claim 3, wherein: The method of obtaining a bimodal fusion result through a pre-trained speech-text bimodal fusion model according to the speech emotion information and the text emotion information, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively, includes: The last fully connected layer of the network in the speech emotion recognition model and the last fully connected layer of the network in the text emotion recognition model are output respectively for weighted fusion.

5. The method of claim 4, wherein: The method of obtaining a bimodal fusion result through a pre-trained speech-text bimodal fusion model according to the speech emotion information and the text emotion information, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively, includes: The feature quantities of the two modalities of speech and text are transformed through the same fully connected layer to obtain the key; The weight is obtained by calculating the inner product between the query vector and each key, and then normalized using the softmax function: The weighted sum of the weight and the corresponding key is used to obtain the multimodal feature, and the multimodal feature is concatenated with the features of the two modalities and the multimodal feature as the final decision vector.

6. The method of claim 1, wherein: The pre-trained speech emotion recognition model includes: a convolutional neural network (CNN).

7. The method of claim 1, wherein: The pre-trained text emotion recognition model includes: a Bert pre-trained language model.

8. An emotion recognition device, wherein: The device comprises: A recognition module, used to recognize voice emotion information and text emotion information based on a financial business handling scenario, wherein the voice emotion information is obtained by recognizing a pre-trained voice emotion recognition model, and the text emotion information is obtained by recognizing a pre-trained text emotion recognition model; A fusion module is used to obtain a bimodal fusion result according to the speech emotion information and the text emotion information through a pre-trained speech-text bimodal fusion model, wherein the pre-trained speech-text bimodal fusion model is used to receive the output results of the speech emotion recognition model and the text emotion recognition model respectively; The result module is used to obtain the emotion recognition result of the speech sequence in the financial business processing scenario based on the bimodal fusion result and the LSTM recurrent neural network structure.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.

Citation Information

Cited By

  • Power enterprise hidden danger risk identification system and method based on multi-modal data

    CN121327665A