Emotion recognition method, apparatus, device, and storage medium
By combining a deep neural network with a BLSTM model and an SVM classifier, and fusing frame-level and sentence-level sentiment information, the problem of neglecting temporal information and overfitting in traditional methods is solved, and high-precision sentiment recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PACHIRA TIMES (ZHUHAI HENGQIN) INFORMATION TECH CO LTD
- Filing Date
- 2022-01-14
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, traditional classifiers ignore the temporal information of audio segments, leading to errors in emotion recognition; neural network modeling units are too short, resulting in overfitting and insufficient representation of emotional information.
By combining a deep neural network with a BLSTM model and an attention layer, and using frame-level features and statistical features, an SVM classifier is employed for sentiment recognition, thereby achieving the fusion of neural networks and traditional classifiers and balancing frame-level and sentence-level sentiment information.
It improves the accuracy of emotion recognition, obtains more robust emotion recognition results, effectively prevents overfitting, and enhances the accuracy of recognition.
Smart Images
Figure CN114387996B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, apparatus, device, and storage medium. Background Technology
[0002] Emotion recognition refers to the simulation of human emotional perception by computers. It involves extracting relevant emotional features from collected audio and establishing an emotion recognition model to map acoustic features to human emotional perception, thereby enabling truly harmonious and natural human-computer interaction.
[0003] Traditional classifiers, such as SVM classifiers, model the entire audio segment as their unit, often ignoring temporal information within the segment, leading to errors or inaccuracies in emotion recognition. Neural network-based emotion recognition systems, on the other hand, use frame-level information for modeling. However, the short duration of these modeling units results in insufficient representation of emotional information and a tendency to overfit. Summary of the Invention
[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, embodiments of this invention propose an emotion recognition method, apparatus, device, and storage medium to improve the accuracy of emotion recognition.
[0005] On one hand, embodiments of the present invention provide an emotion recognition method, comprising: obtaining statistical features of the audio data based on the frame-level features; inputting the frame-level features into a deep neural network to obtain a first emotion score; wherein the deep neural network comprises, from the input side to the output side, a first DNN layer, a BLSTM model, an attention layer, a second DNN layer, and a softmax layer; the attention layer is used to adjust the weights corresponding to the outputs of the BLSTM model at different time points during training, and the attention layer performs a weighted average calculation on the outputs of the BLSTM model at different time points to obtain a first emotion representation; inputting the statistical features and the first emotion representation into an SVM classifier model to obtain a second emotion score; obtaining an emotion recognition result based on the first emotion score and the second emotion score; the emotion recognition result is used to characterize the emotional state identified from the audio data.
[0006] The emotion recognition method according to embodiments of the present invention has at least the following beneficial effects: by inputting the frame-level features corresponding to the audio data into a neural network to obtain a first emotion score, and inputting the statistical features corresponding to the audio data and the first emotion representation output by the attention layer of the neural network into an SVM classifier to obtain a second emotion score, and obtaining the final emotion recognition result based on the first emotion score and the second emotion score, the integration of the neural network and the traditional classifier is realized, the frame-level and sentence-level emotion information in the audio segment is fully balanced and utilized, the emotion recognition result with high robustness can be obtained, the accuracy of emotion recognition is improved, and overfitting is effectively prevented.
[0007] According to some embodiments of the present invention, acquiring audio data and extracting frame-level features from the audio data includes: acquiring the audio data; and extracting the frame-level features of fundamental frequency correlation, spectral correlation, and energy correlation from the audio data according to a time series.
[0008] According to some embodiments of the present invention, the step of inputting the frame-level features into a deep neural network to obtain a first emotion score includes: performing frame-by-frame processing on the frame-level features to obtain a feature sequence of the audio data; and inputting the feature sequence into the deep neural network to obtain the first emotion score.
[0009] According to some embodiments of the present invention, the step of performing frame-by-frame processing on the frame-level features to obtain the feature sequence of the audio data includes: performing frame-by-frame processing on the frame-level features according to a time series to obtain the feature sequence [x1,…,x]. t-1 ,x t ,x t+1 ,…,x T ]; where T is the total number of frames of the audio data, x t The features are those after frame stitching at time t.
[0010] According to some embodiments of the present invention, the input of the hidden layer of the BLSTM model at each time step includes: the output of the hidden layer at the previous time step, the input at the current time step, and the output of the hidden layer at the next time step.
[0011] According to some embodiments of the present invention, the step of inputting the statistical features and the first sentiment representation into an SVM classifier model to obtain a second sentiment score includes: concatenating the statistical features and the first sentiment representation to obtain a first concatenation result; and inputting the first concatenation result into the SVM classifier model to obtain the second sentiment score.
[0012] According to some embodiments of the present invention, obtaining the emotion recognition result based on the first emotion score and the second emotion score includes: multiplying the first emotion score by a first weighting coefficient to obtain a first parameter; multiplying the first emotion score by a second weighting coefficient to obtain a second parameter; wherein the first weighting coefficient is not equal to the second weighting coefficient; and summing the first parameter and the second parameter to obtain a summation result as the emotion recognition result.
[0013] On the other hand, embodiments of the present invention provide an emotion recognition device, comprising: a first module for acquiring audio data and extracting frame-level features from the audio data; a second module for acquiring statistical features of the audio data based on the frame-level features; a third module for inputting the frame-level features into a deep neural network to obtain a first emotion score; wherein the deep neural network comprises, from input to output, a first DNN layer, a BLSTM model, an attention layer, a second DNN layer, and a softmax layer; the attention layer is used to adjust the weights corresponding to the outputs of the BLSTM model at different time points during training, and the attention layer performs a weighted average calculation on the outputs of the BLSTM model at different time points to obtain a first emotion representation; a fourth module for inputting the statistical features and the first emotion representation into an SVM classifier model to obtain a second emotion score; and a fifth module for obtaining an emotion recognition result based on the first emotion score and the second emotion score; the emotion recognition result is used to characterize the emotional state identified from the audio data.
[0014] The emotion recognition device according to embodiments of the present invention has at least the following beneficial effects: by inputting the frame-level features corresponding to the audio data into a neural network to obtain a first emotion score, and inputting the statistical features corresponding to the audio data and the first emotion representation output by the attention layer of the neural network into an SVM classifier to obtain a second emotion score, and obtaining the final emotion recognition result based on the first emotion score and the second emotion score, the fusion of the neural network and the traditional classifier is realized, which can balance and utilize the frame-level and sentence-level emotion information in the audio segment, obtain a more robust emotion recognition result, improve the accuracy of emotion recognition, and effectively prevent overfitting.
[0015] On the other hand, embodiments of the present invention provide a computer device, including: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the emotion recognition method as described above.
[0016] On the other hand, there is a computer-readable storage medium storing program instructions that, when executed by a processor, implement the aforementioned emotion recognition method.
[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:
[0019] Figure 1 This is a schematic diagram of the process steps of the emotion recognition method provided in the embodiments of the present invention.
[0020] Figure 2 This is a block diagram of the system in an embodiment of the present invention and a schematic diagram of the processing flow of frame-level features of audio.
[0021] Figure 3 This is a schematic block diagram of the emotion recognition device provided in an embodiment of the present invention.
[0022] Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation
[0023] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0024] Reference Figure 1 This embodiment discloses an emotion recognition method, which involves artificial intelligence, including but not limited to the following steps S100 to S500.
[0025] Step S100: Acquire audio data and extract frame-level features from the audio data.
[0026] Specifically, for each input audio data, firstly, frame-level features related to fundamental frequency, spectrum, and energy are extracted according to the time series. That is, frame-level information for each frame is extracted in chronological order. These frame-level features will be used as input to the neural network.
[0027] Step S200: Obtain statistical features of the audio data based on frame-level features.
[0028] Specifically, such as Figure 2 As shown, for each input audio data, the statistical characteristics of the audio data are calculated based on frame-level features.
[0029] Step S300: Input the frame-level features into the deep neural network to obtain the first sentiment score.
[0030] The deep neural network, from input to output, includes: a first DNN layer, a BLSTM model, an attention layer, a second DNN layer, and a softmax layer. The attention layer is used to adjust the weights corresponding to the outputs of the BLSTM model at different time points during training. The attention layer performs a weighted average calculation on the outputs of the BLSTM model at different time points to obtain the first sentiment representation.
[0031] Specifically, in this embodiment, the features of each frame in the audio data are processed by frame stitching to obtain the feature sequence [x1,…,x]. t-1 ,x t ,x t+1 ,…,x T ], where T is the total number of frames in audio S, x t The features are those after frame stitching at time t. The method for stitching the frame features can be any known method. For example... Figure 2 As shown, this feature sequence is input into a deep neural network. By stitching frames together, the contextual information of the input features in each frame of the deep neural network model can be enhanced, thereby improving the recognition accuracy of the final emotion recognition result.
[0032] In a deep neural network, the BLSTM model's time series length is equal to the audio length T (i.e., the total number of frames). At each time step, the input to the hidden layer of this BLSTM model is... In other words, the input at each time step includes the input of the input layer at that time step, as well as the output of the hidden layer at the previous time step and the output of the hidden layer at the next time step.
[0033] During training, the attention layer adjusts the weights corresponding to the outputs of the BLSTM model at different time points. Then, a weighted average of the outputs at different time points is calculated to obtain the first sentiment representation of the audio data. Since the contribution of audio at different times to the first sentiment score may vary—for example, silent or non-emotional speech will have a relatively lower sentiment weight—the attention layer helps improve the accuracy of sentiment recognition.
[0034] like Figure 2 As shown, the first sentiment representation is input into the next layer of the deep neural network, passes through a fully connected layer of the DNN, and then undergoes softmax to obtain the first sentiment score. The first sentiment score is the sentiment score given by the deep neural network to this audio data segment.
[0035] Step S400: Input the statistical features and the first sentiment representation into the SVM classifier model to obtain the second sentiment score.
[0036] like Figure 2 As shown, the statistical features and the first sentiment representation of the audio data are concatenated and used as the feature input to the SVM classifier. After passing through the SVM classifier model, a second sentiment score is obtained. The second sentiment score is the sentiment score given to the audio data by the SVM classifier. For example, if the statistical features are AAA and the first sentiment representation is B, the concatenation result is AAAB.
[0037] Step S500: Based on the first emotion score and the second emotion score, obtain the emotion recognition result; the emotion recognition result is used to characterize the emotional state identified from the audio data.
[0038] Specifically, the emotion recognition result is obtained by using the formula: emo_score = λ1 × emo_score1 + λ2 × emo_score2. Here, emo_score represents the emotion recognition result, emo_score1 is the first emotion score, emo_score2 is the second emotion score, and λ1 and λ2 are weights. In this embodiment, the weight values λ1 and λ2 are not equal. In other embodiments of the present invention, these two weight values can also be configured to be the same.
[0039] Taking the SVM classifier as an example, Figure 2 A system block diagram of one embodiment of the present invention is shown. Figure 2 The left side of the diagram shows the processing flow of a deep neural network, and the right side shows the processing flow of an SVM classifier. A deep neural network, from input to output, includes: the first DNN layer (…). Figure 2 The DNN layer on the lower left), BLSTM model, attention layer, second DNN layer ( Figure 2The top left DNN layer). Frame-level features [x1,…,x] extracted from the audio data. t-1 ,x t ,x t+1 ,…,x T The input is fed into the first DNN layer of the deep neural network to obtain the first sentiment score.
[0040] During training, the attention layer automatically adjusts the weights w1, w2, ..., w corresponding to the output of the BLSTM model at different time points. T The first sentiment representation is obtained by weighting the output of the BLSTM model at different time points based on these weight values.
[0041] like Figure 2 As shown in ①, the output of the attention layer of this deep neural network (equivalent to the first sentiment representation) is not only input to the next layer, the second DNN layer, but also output to the SVM model for processing. This first sentiment representation is concatenated with the statistical features calculated through frame-level features of the audio, input into the SVM model, and the output is the second sentiment score.
[0042] like Figure 2 As shown in ②, the first emotion score and the second emotion score are weighted and summed, and the sum is used as the emotion recognition result.
[0043] Through such Figure 2 The feature fusion (i.e., inputting statistical features and the first sentiment representation output by the attention layer of the neural network into the SVM classifier to obtain the second sentiment score) and the result fusion (i.e., deriving the sentiment recognition result based on the first sentiment score and the second sentiment score) shown in the embodiment can balance and utilize the frame-level and sentence-level sentiment information in the audio segment, and can obtain a more robust sentiment recognition result, making the sentiment recognition result more accurate and effectively reducing the possibility of a decrease in recognition accuracy due to overfitting of the neural network.
[0044] Reference Figure 3This embodiment discloses an emotion recognition device, including but not limited to the following modules: a first module 310, a second module 320, a third module 330, a fourth module 340, and a fifth module 350. The first module is used to acquire audio data and extract frame-level features from the audio data. The second module is used to acquire statistical features of the audio data based on the frame-level features. The third module is used to input the frame-level features into a deep neural network to obtain a first emotion score. The deep neural network, from input to output, includes: a first DNN layer, a BLSTM model, an attention layer, a second DNN layer, and a softmax layer. The attention layer is used to adjust the weights corresponding to the outputs of the BLSTM model at different time points during training. The attention layer performs a weighted average calculation on the outputs of the BLSTM model at different time points to obtain a first emotion representation. The fourth module is used to input the statistical features and the first emotion representation into an SVM classifier model to obtain a second emotion score. The fifth module is used to obtain an emotion recognition result based on the first and second emotion scores. The emotion recognition result is used to represent the emotional state identified from the audio data.
[0045] Figure 3 The emotion recognition device shown is Figure 1 The emotion recognition method shown is based on the same inventive concept. Therefore, under the synergistic effect of the above modules, the emotion recognition device can balance and utilize frame-level and sentence-level emotional information in audio segments through feature fusion and result fusion, which can obtain a more robust emotion recognition result, reduce the possibility of recognition accuracy decline due to neural network overfitting, and improve the accuracy of emotion recognition.
[0046] Reference Figure 3This embodiment provides a computer device, including: at least one processor 410; at least one memory 420 for storing at least one program; when the at least one program is executed by the at least one processor 410, the at least one processor 410 implements the emotion recognition method described above. The processor 410 may also be referred to as a CPU (Central Processing Unit). The processor 410 may be an integrated circuit chip with signal processing capabilities. The processor 410 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor, or it may be any conventional processor. The memory 420 may include various components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination thereof. The memory 420 may also include (e.g., instructions (e.g., software) stored in one or more machine-readable media); these instructions implement the method of this embodiment of the invention. It is understood that, in order to avoid redundancy, any content not covered in this embodiment can be referred to the above-described emotion recognition method.
[0047] This embodiment also provides a computer-readable storage medium storing program instructions that, when executed by a processor, implement the aforementioned emotion recognition method. It is understood that, to avoid redundancy, content not covered in this embodiment can be referred to the aforementioned emotion recognition method.
[0048] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the device, can be implemented as software, firmware, hardware, and suitable combinations thereof.
[0049] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium (hereinafter referred to as storage medium), which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer-readable medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0050] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be considered within the scope of the present invention.
Claims
1. An emotion recognition method, characterized in that, include: Acquire audio data and extract frame-level features from the audio data; Based on the frame-level features, obtain the statistical features of the audio data; The frame-level features are input into a deep neural network to obtain a first sentiment score. The deep neural network, from input to output, comprises: a first DNN layer, a BLSTM model, an attention layer, a second DNN layer, and a softmax layer. During training, the attention layer adjusts the weights corresponding to the outputs of the BLSTM model at different time points. The attention layer performs a weighted average calculation on the outputs of the BLSTM model at different time points to obtain the first sentiment representation. The input to the hidden layer of the BLSTM model at each time step includes: the output of the hidden layer at the previous time step, the input of the input layer at the current time step, and the output of the hidden layer at the next time step. The statistical features and the first sentiment representation are input into the SVM classifier model to obtain the second sentiment score; wherein, the first sentiment representation is input into the next layer of the deep neural network, and after passing through the second DNN layer, softmax is performed to obtain the first sentiment score; An emotion recognition result is obtained based on the first emotion score and the second emotion score; the emotion recognition result is used to characterize the emotional state identified from the audio data.
2. The emotion recognition method according to claim 1, characterized in that, The process of acquiring audio data and extracting frame-level features from the audio data includes: Obtain the audio data; Based on the time series, the frame-level features of fundamental frequency correlation, spectral correlation, and energy correlation are extracted from the audio data.
3. The emotion recognition method according to claim 1, characterized in that, The step of inputting the frame-level features into a deep neural network to obtain a first sentiment score includes: The frame-level features are then processed by frame-by-frame stitching to obtain the feature sequence of the audio data; The feature sequence is input into the deep neural network to obtain the first sentiment score.
4. The emotion recognition method according to claim 3, characterized in that, The step of performing frame-by-frame processing on the frame-level features to obtain the feature sequence of the audio data includes: The frame-level features are processed by concatenating frames according to the time series to obtain the feature sequence [x1,…,x]. t-1 ,x t ,x t+1 ,…,x T ]; Where T is the total number of frames of the audio data, x t The features are those after frame stitching at time t.
5. The emotion recognition method according to claim 1, characterized in that, The step of inputting the statistical features and the first sentiment representation into the SVM classifier model to obtain the second sentiment score includes: The statistical features and the first sentiment representation are concatenated to obtain the first concatenation result; The first concatenation result is input into the SVM classifier model to obtain the second sentiment score.
6. The emotion recognition method according to claim 1, characterized in that, The process of obtaining the emotion recognition result based on the first emotion score and the second emotion score includes: Multiply the first emotion score by the first weighting coefficient to obtain the first parameter; The first emotional score is multiplied by the second weighting coefficient to obtain the second parameter; wherein the first weighting coefficient is not equal to the second weighting coefficient. The first parameter and the second parameter are summed to obtain the summation result as the emotion recognition result.
7. An emotion recognition device, characterized in that, include: The first module is used to acquire audio data and extract frame-level features from the audio data. The second module is used to obtain the statistical features of the audio data based on the frame-level features; The third module is used to input the frame-level features into a deep neural network to obtain a first sentiment score. The deep neural network, from input to output, comprises: a first DNN layer, a BLSTM model, an attention layer, a second DNN layer, and a softmax layer. During training, the attention layer adjusts the weights corresponding to the outputs of the BLSTM model at different time points. The attention layer performs a weighted average calculation on the outputs of the BLSTM model at different time points to obtain the first sentiment representation. The input to the hidden layer of the BLSTM model at each time step includes: the output of the hidden layer at the previous time step, the input of the input layer at the current time step, and the output of the hidden layer at the next time step. The fourth module is used to input the statistical features and the first sentiment representation into the SVM classifier model to obtain the second sentiment score; wherein, the first sentiment representation is input into the next layer of the deep neural network, and after passing through the second DNN layer, softmax is performed to obtain the first sentiment score; The fifth module is used to obtain an emotion recognition result based on the first emotion score and the second emotion score; the emotion recognition result is used to characterize the emotional state identified from the audio data.
8. A computer device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the emotion recognition method as described in any one of claims 1 to 6.
9. A computer-readable storage medium storing program instructions that, when executed by a processor, implement the emotion recognition method as described in any one of claims 1 to 6.