Voice data emotion and emotion recognition method based on CNN model
Through the emotion and emotion recognition method combined with CNN-BiLSTM neural network and CNN convolutional neural network, the problems of low accuracy of speech emotion recognition and high computational complexity are solved, and efficient emotion and emotion recognition is achieved, which is suitable for smart security, smart home and health monitoring.
Patent Information
- Application Number
- CN202510889567.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-30
- Publication Date
- 2025-08-12
AI Technical Summary
In the prior art, the speech emotion recognition accuracy is low and the operation complexity is high, making it difficult to effectively apply in real life natural scenes.
The emotion recognition model is constructed by the CNN-BiLSTM neural network model, and the emotion recognition model is established by combining the CNN convolutional neural network. The word vectors are obtained through the Glove model for feature extraction and distributed semantic feature coding, reducing the operation cost.
It significantly improves the accuracy and robustness of speech data emotion recognition, reduces the computing cost, and achieves clear detection of the speaker's own emotions and emotions.
Smart Images
Figure CN120472946A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for recognizing the emotion and mood of speech data based on a CNN model. Background Art
[0002] Understanding human speech has long been a core research issue in the fields of artificial intelligence and digital speech processing. With the rapid development of artificial intelligence and big data technologies, emotion recognition has gained widespread application in fields such as smart security, smart homes, and health monitoring. As an integral component of natural speech, emotions add depth and rich context to our communication. The same sentence can have vastly different meanings depending on the emotion it conveys. This non-verbal data is crucial for communication, profoundly influencing our interpretation of the speaker's intent and the overall message. Recognizing and understanding these emotions is crucial for effective communication in both text and spoken language. However, while modern speech algorithms are adept at processing neutral speech, recognizing and understanding the emotions embedded in speech remains a significant challenge. This difficulty stems from the complex and diverse nature of emotions and their diverse manifestations in speech.
[0003] Research on speech emotion recognition has made significant progress both domestically and internationally. However, traditional acoustic prosodic features, spectral-based correlation features, and sound quality features describe speech emotion from the perspectives of the time and frequency domains, respectively, and are unable to simultaneously reflect the time-frequency and emotional characteristics of speech signals. Static image algorithms, when applied to natural scenes, lack effective utilization of dynamic sequence information, resulting in poor algorithm robustness and a need for improvement. The current leading approach uses spectrograms as speech emotion features, simultaneously reflecting the time-frequency and emotional characteristics of speech signals, and then utilizes deep learning algorithms to extract and classify emotion features. While these algorithms have improved recognition accuracy, they also require high hardware computing performance due to the large amount of data they process and their high algorithm complexity. They are often deployed on high-performance servers, making them difficult to apply to real-life natural scenes. Summary of the Invention
[0004] The purpose of the present invention is to provide a method for voice data emotion and mood recognition based on a CNN model, which can solve the problems of low voice recognition accuracy and overly complex calculations in the prior art.
[0005] The technical solution of the present invention is:
[0006] In a first aspect, the present application provides a method for speech data emotion and mood recognition based on a CNN model, which comprises the following steps:
[0007] S1. Obtaining original speech data from the corpus and performing data preprocessing on the original speech data;
[0008] S2. Establishing a data set based on the preprocessed speech data, and performing feature extraction on the data set to obtain speech features;
[0009] S3. Build an emotion recognition model based on the CNN-BiLSTM neural network model and train the emotion recognition model based on the dataset. Input the speech features into the CNN classification model for emotion classification. At the same time, input the emotion classification results into the trained emotion recognition model for validity verification and output the emotion recognition results.
[0010] S4. Processing the speech features to obtain text features, and performing distributed semantic feature encoding on the text features to obtain a distributed semantic feature vector;
[0011] S5. Establish an emotion recognition model through CNN convolutional neural network, train the emotion recognition model according to the data set, input the distributed semantic feature vector into the trained emotion recognition model for validity verification and output the emotion recognition result;
[0012] S6. Combine the emotion recognition results and the sentiment recognition results to comprehensively analyze and obtain the emotion and sentiment recognition results of the original speech data.
[0013] Furthermore, in step S1, the data preprocessing process includes: data screening, data labeling, data simplification and data enhancement.
[0014] Furthermore, in step S2, the feature extraction process includes: using the Glove model to obtain a vector representation of the word based on an unsupervised paradigm, and obtaining a linear vector substructure by using aggregated word-word co-occurrence statistics from the corpus to capture the semantic relationship between words to obtain a feature extraction result. The calculation formula includes:
[0015]
[0016] Where, J is the result of speech feature extraction, X ij is the element matrix, f is the weight function, N is the matrix dimension, i and j represent the i-th word and the j-th word respectively, V i 、V j Both are word vectors, b i 、b j are all deviation terms, T is the transpose of the matrix, x max , α are specific parameters assigned by experience, and x is the independent variable of the function.
[0017] Furthermore, in step S3, the calculation formula of the above CNN classification model includes:
[0018] C ij =V(t(1),V(t(2))…Vt(i)),1≤i≤n
[0019] s ij =f(F×V(t(i:i+r-1))+b)
[0020] S=(s1,s2,…,s r-h+1 )
[0021] Where C ij is the sentence matrix, Vt(i) is the corresponding word vector converted by Glove, t(i) is the i-th word, n is the number of words, s ij is the local feature extracted by the CNN classification model for the jth sentence consisting of i words, f is the weight function, F represents the filter, b is the bias, r is the number of rows in the word vector, S is the local feature set, and h is the size parameter of the filter.
[0022] Furthermore, step S4 includes:
[0023] S41. Use the Librosa audio processing library in the Python ecosystem to process speech features to obtain text features.
[0024] S42. Use the Glove word embedding model based on co-occurrence matrix decomposition to perform distributed semantic feature encoding on text features through pre-trained word vector space to obtain distributed semantic feature vectors.
[0025] In a second aspect, the present application provides an electronic device, comprising:
[0026] a memory for storing one or more programs;
[0027] processor;
[0028] When the above one or more programs are executed by the above processor, a method for speech data emotion and mood recognition based on a CNN model as described in any one of the first aspects above is implemented.
[0029] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for speech data emotion and mood recognition based on a CNN model as described in any one of the first aspects above.
[0030] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:
[0031] (1) The present invention provides a method for recognizing emotion and mood from speech data based on a CNN model. The method constructs an emotion recognition model through a CNN-BiLSTM neural network model and simultaneously establishes an emotion recognition model through a CNN convolutional neural network. Combining the two models can significantly improve the accuracy of emotion recognition in speech data and enhance the accuracy and robustness of emotion recognition in an open environment, thereby clearly detecting the speaker's own emotions and moods.
[0032] (2) The present invention adopts the Glove model to obtain word vectors and uses aggregated word-word co-occurrence statistics to obtain a linear vector substructure from the corpus to capture the semantic relationship between words for feature extraction. At the same time, the present invention adopts the Glove word embedding model based on co-occurrence matrix decomposition, and performs distributed semantic feature encoding on text features through pre-trained word vector space, which can capture the semantic association and context dependency between words, provide high-quality text feature representation for subsequent natural language processing tasks (such as classification, translation, etc.), and thus reduce the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.
[0034] Figure 1 This is a step diagram of a method for speech data emotion and mood recognition based on a CNN model of the present invention;
[0035] Figure 2 This is a schematic diagram of the emotion recognition model architecture built based on the CNN-BiLSTM neural network model;
[0036] Figure 3 This is a schematic diagram of the emotion recognition model architecture based on the CNN convolutional neural network;
[0037] Figure 4 The figure is a schematic structural block diagram of an electronic device according to an embodiment of the present invention.
[0038] Icon: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION
[0039] Explanation of terms:
[0040] CNN-BiLSTM (Convolutional Neural Network-Bidirectional Long Short-Term Memory Network): A combination of two deep learning models, CNN (Convolutional Neural Network) and BiLSTM (Bidirectional Long Short-Term Memory Network). CNN is a convolutional neural network, and BiLSTM is a bidirectional long short-term memory network.
[0041] GloVe word embedding model based on co-occurrence matrix decomposition: The core of this model is to use the co-occurrence matrix between words. The co-occurrence matrix records the frequency of co-occurrence of each word pair in the corpus. The GloVe model decomposes this co-occurrence matrix to learn the vector representation of each word, so that these word vectors can capture the semantic relationship between words.
[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.
[0043] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.
[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.
[0045] It should be noted that, in this document, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprises..." do not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the elements.
[0046] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features thereof may be combined with each other.
[0047] Example 1
[0048] See also Figure 1-3 , Figure 1 FIG2 is a step diagram of a method for recognizing emotion and mood from speech data based on a CNN model provided in an embodiment of the present application. Figure 2 、 Figure 3 They are the emotion recognition model architecture diagram and the sentiment recognition model architecture diagram respectively.
[0049] In a first aspect, the present application provides a method for speech data emotion and mood recognition based on a CNN model, which comprises the following steps:
[0050] S1. Obtaining original speech data from the corpus and performing data preprocessing on the original speech data;
[0051] S2. Establishing a data set based on the preprocessed speech data, and performing feature extraction on the data set to obtain speech features;
[0052] S3. Build an emotion recognition model based on the CNN-BiLSTM neural network model and train the emotion recognition model based on the dataset. Input the speech features into the CNN classification model for emotion classification. At the same time, input the emotion classification results into the trained emotion recognition model for validity verification and output the emotion recognition results.
[0053] S4. Processing the speech features to obtain text features, and performing distributed semantic feature encoding on the text features to obtain a distributed semantic feature vector;
[0054] S5. Establish an emotion recognition model through CNN convolutional neural network, train the emotion recognition model according to the data set, input the distributed semantic feature vector into the trained emotion recognition model for validity verification and output the emotion recognition result;
[0055] S6. Combine the emotion recognition results and the sentiment recognition results to comprehensively analyze and obtain the emotion and sentiment recognition results of the original speech data.
[0056] As a preferred implementation, in step S1, the data preprocessing process includes: data screening, data labeling, data simplification and data enhancement.
[0057] Among them, data screening includes deleting punctuation marks and numbers as well as keywords. Various terms such as "in" and "this" are redundant. In addition, articles, conjunctions and prepositions are supplements to sentences and there is no supplement to the sentence, so they may also be omitted; data labeling includes tokenizing opinion data, which is broken down into tokens as words and symbols, which helps to further solve more data preprocessing technical problems, such as positive emotion labeling.
[0058] As a preferred embodiment, in step S2, the feature extraction process includes: using the Glove model to obtain a vector representation of the word based on the unsupervised paradigm, and obtaining a linear vector substructure by using aggregated word-word co-occurrence statistics from the corpus to capture the semantic relationship between words to obtain a feature extraction result. The calculation formula includes:
[0059]
[0060] Where, J is the result of speech feature extraction, X ij is the element matrix, f is the weight function, N is the matrix dimension, i and j represent the i-th word and the j-th word respectively, V i 、V j Both are word vectors, b i 、b j are all deviation terms, T is the transpose of the matrix, x max , α are specific parameters assigned by experience, and x is the independent variable of the function.
[0061] As a preferred embodiment, in step S3, the calculation formula of the CNN classification model includes:
[0062]
[0063] Where C ij is the sentence matrix, Vt(i) is the corresponding word vector converted by Glove, t(i) is the i-th word, n is the number of words, s ij is the local feature extracted by the CNN classification model for the jth sentence consisting of i words, f is the weight function, F represents the filter, b is the bias, r is the number of rows in the word vector, S is the local feature set, and h is the size parameter of the filter.
[0064] As a preferred embodiment, step S4 includes:
[0065] S41. Use the Librosa audio processing library in the Python ecosystem to process speech features to obtain text features.
[0066] S42. Use the Glove word embedding model based on co-occurrence matrix decomposition to perform distributed semantic feature encoding on text features through pre-trained word vector space to obtain distributed semantic feature vectors.
[0067] Example 2
[0068] See also Figure 4 , Figure 4 A schematic structural block diagram of an electronic device provided in an embodiment of the present application.
[0069] An electronic device includes a memory 101, a processor 102, and a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 can be used to communicate signaling or data with other node devices.
[0070] Among them, the memory 101 can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.
[0071] The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0072] It is understood that the structure shown in the figure is merely illustrative, and a method for recognizing emotion and feeling from speech data based on a CNN model may include more or fewer components than shown in the figure, or have a different configuration than shown in the figure. Each component shown in the figure may be implemented using hardware, software, or a combination thereof.
[0073] In the embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The embodiments described above are merely illustrative. For example, the flowcharts or block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the methods and computer program products according to multiple embodiments of the application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.
[0074] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0075] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0076] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.
[0077] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.
Claims
1. A method for speech data emotion and mood recognition based on a CNN model, characterized in that: The following steps are involved: S1. Obtaining original speech data from the corpus and performing data preprocessing on the original speech data; S2. Establishing a data set based on the preprocessed speech data, and performing feature extraction on the data set to obtain speech features; S3. Build an emotion recognition model based on the CNN-BiLSTM neural network model and train the emotion recognition model based on the dataset. Input the speech features into the CNN classification model for emotion classification. At the same time, input the emotion classification results into the trained emotion recognition model for validity verification and output the emotion recognition results. S4. Processing the speech features to obtain text features, and performing distributed semantic feature encoding on the text features to obtain a distributed semantic feature vector; S5. Establish an emotion recognition model through CNN convolutional neural network, train the emotion recognition model according to the data set, input the distributed semantic feature vector into the trained emotion recognition model for validity verification and output the emotion recognition result; S6. Combine the emotion recognition results and the sentiment recognition results to comprehensively analyze and obtain the emotion and sentiment recognition results of the original speech data.
2. The method for speech data emotion and mood recognition based on a CNN model according to claim 1, wherein: In step S1, the data preprocessing process includes: data screening, data labeling, data simplification and data enhancement.
3. The method for speech data emotion and mood recognition based on a CNN model according to claim 1, wherein: In step S2, the feature extraction process includes: using the Glove model to obtain a vector representation of the word based on the unsupervised paradigm, and obtaining a linear vector substructure by using aggregated word-word co-occurrence statistics from the corpus to capture the semantic relationship between words to obtain a feature extraction result. The calculation formula includes: Where, J is the result of speech feature extraction, X ij is the element matrix, f is the weight function, N is the matrix dimension, i and j represent the i-th word and the j-th word respectively, V i 、V j Both are word vectors, b i 、b j are all deviation terms, T is the transpose of the matrix, x max , α are specific parameters assigned by experience, and x is the independent variable of the function.
4. The method for speech data emotion and mood recognition based on a CNN model according to claim 1, wherein: In step S3, the calculation formula of the CNN classification model is include: C ij =V(t(1),V(t(2))…Vt(i)),1≤i≤n s ij =f(F×V(t(i:i+r-1))+b) S=(s1,s2,…,s r-h+1 ) Where C ij is the sentence matrix, Vt(i) is the corresponding word vector converted by Glove, t(i) is the i-th word, n is the number of words, s ij is the local feature extracted by the CNN classification model for the jth sentence consisting of i words, f is the weight function, F represents the filter, b is the bias, r is the number of rows in the word vector, S is the local feature set, and h is the size parameter of the filter.
5. The method for speech data emotion and mood recognition based on a CNN model according to claim 1, wherein: Step S4 includes: S41. Use the Librosa audio processing library in the Python ecosystem to process speech features to obtain text features. S42. Use the Glove word embedding model based on co-occurrence matrix decomposition to perform distributed semantic feature encoding on text features through pre-trained word vector space to obtain distributed semantic feature vectors.
6. An electronic device, characterized in that: include: a memory for storing one or more programs; processor; When the one or more programs are executed by the processor, a method for speech data emotion and mood recognition based on a CNN model as described in any one of claims 1 to 5 is implemented.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements a method for recognizing speech data emotions and feelings based on a CNN model as described in any one of claims 1 to 5.