Self-supervised comparative learning speech emotion recognition method and device and storage device
The CNN-GRU model constructed through self-supervised contrastive learning solves the problem of difficult data acquisition in traditional supervised learning, achieves efficient speech emotion recognition, and improves recognition accuracy and service quality.
Patent Information
- Application Number
- CN202410258433.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-07
- Publication Date
- 2025-09-09
AI Technical Summary
Traditional supervised learning methods require a large amount of labeled data in speech emotion recognition, which makes data acquisition difficult and the labeling cost high. Self-supervised contrastive learning can learn feature representations through the information characteristics of the data itself without explicit labels.
A self-supervised contrastive learning method was adopted to build a lightweight online speech emotion recognition model based on CNN-GRU. Contrastive Predictive Coding (CPC) was used for self-supervised pre-training. The CNN encoder parameters were frozen and connected to the GRU unit. The final output of the GRU was used for prediction to achieve speech emotion recognition.
It achieved fast and efficient model adaptation in online speech emotion recognition, improved service quality and response efficiency, and achieved a test accuracy of 96.97% and a new positive sample accuracy of 95.89%.
Smart Images
Figure BDA0004729994720000051 
Figure BDA0004729994720000053 
Figure BDA0004729994720000054
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence speech emotion recognition and its application, and particularly relates to a state perception method, device and storage medium in the field of speech emotion recognition. Background Art
[0002] Speech emotion recognition is a crucial task in a variety of fields, including human-computer interaction, social robotics, intelligent customer service, and multi-domain natural language processing applications. It enables machines to better understand and respond to human emotions, improving service quality, enhancing user experience, and strengthening emotional understanding and communication. It is crucial for achieving more intelligent and user-friendly systems and applications. Speech emotion recognition is widely used in medical assistance, advertising and market research, call centers, and customer service. Customer dissatisfaction or anger can trigger customer service representatives to provide more attentive service, improving the customer experience.
[0003] Self-supervised contrastive learning, as a cutting-edge technical approach, has a crucial technical foundation in the field of speech emotion recognition. Traditional supervised learning methods typically require large amounts of labeled data to train models, but this approach presents challenges with data acquisition and high labeling costs. Self-supervised contrastive learning, on the other hand, leverages the inherent information properties of the data, eliminating the need for explicit labels. By contrasting data samples, the model learns feature representations independently, thereby achieving emotion recognition. Summary of the Invention
[0004] The present invention provides a speech emotion recognition method, device and storage device based on self-supervised contrastive learning, aiming to provide strong support for the further development of the field of online speech emotion recognition.
[0005] The technical solution of the present invention is actually based on a self-supervised contrastive learning speech emotion recognition method, device and storage device, characterized in that the method includes:
[0006] Step 1: Collecting audio data of conversation recordings in a real communication environment, and performing data cleaning and normalization on the audio data to obtain preprocessed data;
[0007] Step 2: Build a lightweight online speech emotion recognition model based on CNN-GRU self-supervised contrastive learning;
[0008] Step 3: Input the preprocessed data into the 1D-CNN encoder and perform self-supervised pre-training of the CNN encoder using contrastive predictive coding (CPC).
[0009] Step 4: Freeze the parameters in the CNN encoder and connect it to the new GRU unit;
[0010] Step 5: Use the final output of GRU to generate the final prediction;
[0011] Step 6: Deploy the speech emotion recognition pre-training model to the target domain to obtain an online language emotion classification model.
[0012] In addition, to achieve the above-mentioned purpose, the present application also provides a method, device and storage device for speech emotion recognition based on self-supervised contrastive learning, which comprises:
[0013] Module 1, data collection and preprocessing module, collects audio files during the communication process, and performs data cleaning and normalization on the audio file data to obtain preprocessed data;
[0014] Module 2, model building module, builds a self-supervised contrastive learning speech emotion recognition model, inputs the preprocessed data into the CNN encoder, and uses CPC to perform self-supervised pre-training on the CNN encoder;
[0015] Module 3, fine-tuning and optimization module, after pre-training, freezes the CNN encoder and connects it to the new GRU unit to complete the fine-tuning of the model;
[0016] Module 4, the reasoning application module, uses the trained self-supervised contrastive learning speech emotion recognition model to perform reasoning and planning, and applies it to various downstream tasks, so that the model can quickly adapt to new tasks.
[0017] In addition, to achieve the above-mentioned purpose, the present application also provides a method, device and storage device for speech emotion recognition based on self-supervised comparative learning. The method and device for speech emotion recognition based on self-supervised comparative learning include a processor, a memory, and a speech emotion recognition model and device program based on self-supervised comparative learning stored on the memory and executable by the processor. When the speech emotion recognition model and device program based on self-supervised comparative learning are executed by the processor, the steps of the method based on self-supervised comparative learning speech emotion recognition model and device are implemented as described above.
[0018] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which is stored a speech emotion recognition model and device program based on self-supervised contrastive learning. When the speech emotion recognition model and device program based on self-supervised contrastive learning is executed by a processor, the steps of the speech emotion recognition model and device based on self-supervised contrastive learning as described above are implemented.
[0019] These and other aspects of the present application will be more clearly understood in the following description of the embodiments. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application.
[0021] In the attached figure:
[0022] Figure 1 This is a flow chart of the model and device for speech emotion recognition method based on self-supervised contrastive learning in this application.
[0023] Figure 2 This is a flow chart of the first embodiment of the method model and device method for speech emotion recognition based on self-supervised comparative learning in this application.
[0024] Figure 3 This is a schematic diagram of the device model and apparatus program based on self-supervised comparative learning for speech emotion recognition in the first embodiment of the present application.
[0025] The purpose, features and advantages of this application will be further explained with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION
[0026] To further clarify the objectives, technical solutions, and advantages of the embodiments of the present invention, this section will describe in detail the specific embodiments of the present invention in conjunction with the accompanying drawings, providing a clear and complete description of the technical solutions in the embodiments of the present invention. The embodiments described herein are some of the preferred embodiments of the present invention. The purpose of the drawings is to supplement the description in the text of the specification with graphics, enabling people to intuitively and vividly understand each technical feature and overall technical solution of the present invention. However, they should not be construed as limiting the scope of protection of the present invention.
[0027] The following will provide a clear and complete description of the concept, specific structure and technical effects of the present invention in conjunction with the embodiments and drawings to fully understand the purpose, scheme and effects of the present invention.
[0028] It should be noted that, unless otherwise specified, in the description of the present invention, the singular forms "a", "said" and "the" used are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art. The terms used in this specification are only for describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any combination of one or more related listed items. In the description of the present invention, the meaning of "several" is one or more, the meaning of "a plurality" is more than two, and the meanings of "greater than", "less than", and "exceed" are understood to exclude the number itself, and "above", "below", and "within" are understood to include the number itself.
[0029] In the description of the present invention, the consecutive numbering of the method steps is for the convenience of review and understanding. Combined with the overall technical solution of the present invention and the logical relationship between the various steps, adjusting the implementation order between the steps will not affect the technical effect achieved by the technical solution of the present invention.
[0030] In the description of the present invention, unless otherwise clearly defined, words such as “setting” should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.
[0031] In an embodiment of the present application, a speech emotion recognition model and device based on self-supervised contrastive learning are provided, wherein the speech emotion recognition model and device based on self-supervised contrastive learning can be applied to a speech emotion recognition model and device based on self-supervised contrastive learning. The speech emotion recognition model and device based on self-supervised contrastive learning can be a PC, a portable computer, a mobile terminal, or other device with display and processing functions, but is of course not limited to this.
[0032] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.
[0033] Please refer to Figure 1 and Figure 2 , Figure 1 This is a flowchart of the speech emotion recognition model and device based on self-supervised contrastive learning in this application. Figure 2 This is a flowchart of the first embodiment of the method model and device for speech emotion recognition based on self-supervised contrastive learning of this application. In the embodiment of this application, the speech emotion recognition model and device based on self-supervised contrastive learning include the following steps S10 to S60:
[0034] S10, collecting audio data of conversation recordings in a real communication environment, and performing data cleaning and normalization processing on the audio data to obtain preprocessed data;
[0035] S20. Construct a lightweight online speech emotion recognition model based on self-supervised contrastive learning using CNN-GRU.
[0036] S30, inputting the preprocessed data into the 1D-CNN encoder, and performing self-supervised pre-training on the CNN encoder using contrastive predictive coding (CPC);
[0037] S40, freeze the parameters in the CNN encoder and connect it to the new GRU unit;
[0038] S50, use the final output of GRU to generate the final prediction;
[0039] S60. Deploy the speech emotion recognition pre-training model to the target domain to obtain an online language emotion classification model.
[0040] In an embodiment of the present application, the method of collecting audio data of a conversation recording in a real communication environment and performing data cleaning and normalization on the audio data to obtain pre-processed data further includes:
[0041] Synchronously collect and upload audio files in communication in real time. The audio file data is the recording file of the online communication process; the audio file data is cleaned, including removing data that does not meet the requirements, removing abnormal and invalid data, and filling in missing data; based on audio and tags, the audio file data is classified and normalized to obtain preprocessed data.
[0042] The audio file data is classified and normalized, including: collecting audio files during the communication process, obtaining a certain number of wav files of different lengths, consisting of negative (normal) samples and positive (abnormal) samples. First, use the librosa library to read the wav file and set a fixed sampling rate. Cut the front audio where the norm is less than a certain value. Since most of the audio has a norm much larger than this value, the cutting retains the overall shape of the audio wave. Finally, the tail data is also intercepted. At the inference time, the cutting point of the data is selected. During training, the cutting point is uniformly sampled from a certain range of the cutting point. After processing, the data is divided into training data and test data.
[0043] In an embodiment of the present application, a CNN-GRU lightweight online speech emotion recognition model based on self-supervised contrastive learning is constructed, further comprising:
[0044] A self-supervised learning strategy for the model is based on contrastive predictive coding. Specifically, the CPC loss function is used for self-supervised learning of the model. To meet the real-time requirements of online detection, a lightweight CNN-GRU encoder-decoder network structure is designed.
[0045] The model consists of a 1D-CNN encoder and a GRU decoder. Raw audio is fed into the 1D-CNN, where the CNN encoder undergoes self-supervised pre-training using contrastive predictive coding (CPC) to extract audio features. These features are then fed into the GRU unit. The parameters of the CNN encoder are then frozen, and the GRU is trained with labeled data. The final GRU output is extracted and passed through a fully connected neural network to obtain the final prediction.
[0046] In an embodiment of the present application, the preprocessed data is input into a 1D-CNN encoder, and the CNN encoder is self-supervised pre-trained using contrastive predictive coding (CPC), further comprising:
[0047] The model first feeds the preprocessed audio data into a one-dimensional convolutional neural network (1D-CNN) encoder, and then trains the CNN encoder through self-supervised pre-training of contrastive predictive coding (CPC) to extract relevant features from the audio data.
[0048] In CPC pre-training, in order to make all samples in a batch have the same length, it is necessary to cut the longer samples in a batch. The audio features are extracted using the 1D-CNN encoder and the nonlinear encoder g is used to extract the audio features. enc The observations x in the time window t Mapped to a latent representation sequence z t =g enc (x t ), these sequences have a lower resolution in time. Then, an autoregressive model gar is used to combine the potential representation of the current moment with the relevant information of all previous moments to generate the contextual potential representation c t =g ar (z≤t);
[0049] Extract audio x using 1D-CNN encoder t The feature representation of the tth segment z t , and use GRU to summarize the first t segments to obtain representation c t Using c t Prediction from z t+1 to z t+12 That is, for each k=1,2,…,12, the correct z must be identified from 16z t+k ;
[0050] To reduce x t+k and c t Independence between them, that is, minimizing P(x t+k ) and P(c t ), where KL divergence is
[0051]
[0052] Modeling a density ratio through an exponential of bilinear form
[0053]
[0054] Minimize InfoNCE loss to optimize the t to z t+k predictions.
[0055]
[0056] For the CNN part, we have a CNN with a fixed downsampling rate and a certain number of output features, which is hard-coded in the CNN architecture. The CNN is connected to a GRU, each with a certain number of units. The CPC model is then trained on the unlabeled training data using the Adam optimizer, with a set learning rate and L2 regularization for a certain number of epochs.
[0057] In an embodiment of the present application, the parameters in the CNN encoder are frozen and connected to the new GRU unit, further comprising:
[0058] After CPC pre-training, freeze the CNN encoder and connect it to the new GRU unit, and use the final output of the GRU to produce the final prediction. Then, the prediction and label are fed into the binary cross entropy loss function;
[0059] After pre-training, the GRU unit is replaced with a new single-layer GRU, and the new GRU is trained on the training data using the Adam optimizer, with a set learning rate and L2 regularization. The training lasts for a certain period of time. In fine-tuning, the shorter samples in the batch are padded with 0. Assume that the length of the i-th sample is l i , set the downsampling rate of 1D-CNN to fs, extract y i The first i / fs entries are used as the feature representation of the i-th sample in the batch and are input into the neural network.
[0060] In an embodiment of the present application, the final output of the GRU is used to generate the final prediction, which also includes:
[0061] Testing the model on the test data yielded 96.97% accuracy, along with the confusion matrix After the first round of training, the misclassified test data was moved to the training data, and the model was trained for a certain number of iterations. Then, on the remaining data, 100% accuracy was achieved. Furthermore, the model was tested on the newly obtained positive samples, achieving 95.89% accuracy.
[0062] In an embodiment of the present application, deploying and applying the target speech emotion recognition model to an online language emotion classification model also includes: deploying and applying the trained target speech emotion recognition model to an online language emotion classification model, thereby achieving rapid and efficient recognition of the customer's speech emotions, which can effectively improve the quality of service and the efficiency of response.
[0063] Reference Figure 3 The present invention also proposes a method, device, and storage device for speech emotion recognition based on self-supervised contrastive learning, which are used to implement the method for speech emotion recognition based on self-supervised contrastive learning. The method, device, and storage device for speech emotion recognition based on self-supervised contrastive learning include:
[0064] The data collection and preprocessing module collects audio files during the communication process, and performs data cleaning and normalization on the audio file data to obtain preprocessed data;
[0065] The model building module builds a self-supervised contrastive learning speech emotion recognition model, inputs the preprocessed data into the CNN encoder, and uses CPC to perform self-supervised pre-training on the CNN encoder;
[0066] Fine-tuning optimization module, after pre-training, freeze the CNN encoder and connect it to the new GRU unit to complete the fine-tuning of the model;
[0067] The reasoning application module uses the trained self-supervised contrastive learning speech emotion recognition model to perform reasoning and planning, and applies it to various downstream tasks, so that the model can quickly adapt to new tasks.
[0068] In addition, the present application also provides a computer-readable storage medium. The computer-readable storage medium of the present application stores a method and device program for speech emotion recognition based on self-supervised contrastive learning, wherein when the method and device program for speech emotion recognition based on self-supervised contrastive learning is executed by a processor, the steps of the method and device program for speech emotion recognition based on self-supervised contrastive learning are implemented as described above:
[0069] Step 1: Collecting audio data of conversation recordings in a real communication environment, and performing data cleaning and normalization on the audio data to obtain preprocessed data;
[0070] Step 2: Build a lightweight online speech emotion recognition model based on CNN-GRU self-supervised contrastive learning;
[0071] Step 3: Input the preprocessed data into the 1D-CNN encoder and perform self-supervised pre-training of the CNN encoder using contrastive predictive coding (CPC).
[0072] Step 4: Freeze the parameters in the CNN encoder and connect it to the new GRU unit;
[0073] Step 5: Use the final output of GRU to generate the final prediction;
[0074] Step 6: Deploy the speech emotion recognition pre-training model to the target domain to obtain an online language emotion classification model.
[0075] Among them, the method implemented when the self-supervised comparative learning speech emotion recognition method and device program are executed can refer to the embodiment of the self-supervised comparative learning speech emotion recognition method and device program of this application, and will not be repeated here.
[0076] The present application can be used in a variety of general or special computer device environments or configurations. Furthermore, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, RAM, ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.
[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0078] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A speech emotion recognition method based on self-supervised contrastive learning, characterized by: The following steps are involved: Collecting audio data of call recordings in a real environment, and performing data cleaning and normalization on the audio data to obtain preprocessed data; Build a lightweight online speech emotion recognition model based on CNN-GRU (convolution neural network-Gated Recurrent Units) based on self-supervised contrastive learning; The preprocessed data is fed into a 1D-CNN (One-Dimensional Convolutional Neural Network) encoder, and the CNN encoder is self-supervised pre-trained using Contrastive Predictive Coding (CPC). Freeze the parameters in the CNN encoder and connect it to the new GRU unit; Use the final output of the GRU to produce the final prediction; Deploy the speech emotion recognition pre-training model to the target domain to obtain an online language emotion classification model.
2. The self-supervised contrastive learning speech emotion recognition method according to claim 1, characterized in that The collecting of audio data of call recordings in a real environment and performing data cleaning and normalization on the audio data to obtain pre-processed data includes: Synchronously collect and upload call recording audio files in real time. The audio file data is a recording file of the online communication process, which consists of normal and abnormal emotion samples. Normal samples are composed of normal emotion samples, and abnormal samples show emotions such as anxiety, urging, and dissatisfaction; Cleaning the audio file data includes removing data that does not meet the requirements, removing abnormal and invalid data, and filling in missing data; Based on the audio and the label, the audio file data is classified and normalized to obtain preprocessed data; The first audio segment is clipped within a fixed norm range. Taking into account the majority of the audio's norm characteristics, the clipping preserves the overall shape of the audio wave while truncating the tail data. During inference time, a fixed clipping point is selected. During training, the clipping points are uniformly sampled within the selected range. Finally, the dataset is divided into training, validation, and test sets.
3. The self-supervised contrastive learning speech emotion recognition method according to claim 1, characterized in that The construction of a CNN-GRU lightweight online speech emotion recognition model based on self-supervised contrastive learning includes: A self-supervised learning strategy for the model is based on contrastive predictive coding. Specifically, the CPC loss function is used for self-supervised learning of the model. To meet the real-time requirements of online detection, a lightweight CNN-GRU encoder-decoder network structure is designed. The model consists of a 1D-CNN encoder and a GRU decoder. Raw audio is fed into the 1D-CNN, where the CNN encoder is self-supervised pre-trained using contrastive predictive coding (CPC) to extract audio features. These features are then fed into the GRU unit. The parameters in the CNN encoder are then frozen, and the GRU is trained with labeled data. The final GRU output is extracted and passed through a fully connected neural network to obtain the final prediction.
4. The self-supervised contrastive learning speech emotion recognition method according to claim 1, characterized in that The pre-processed data is input into the 1D-CNN encoder, and the CNN encoder is self-supervised pre-trained using contrastive predictive coding (CPC), including: The model first feeds the preprocessed audio data into a one-dimensional convolutional neural network (1D-CNN) encoder, and then trains the CNN encoder through self-supervised pre-training of contrastive predictive coding (CPC) to extract relevant features from the audio data. In the CPC pre-training phase, the longer data in a batch of samples are first cut so that all samples in a batch have the same length. Then the 1D-CNN encoder is used to extract the audio x t The feature representation of the tth segment z t , use GRU to summarize the first t segments to obtain representation c t , using c t The data after prediction. Then in order to reduce x t+k and c t Independence between them, that is, minimizing P(x t+k ) and P(c t ), where the KL divergence is: Modeling a density ratio through an exponential of bilinear form Then reduce InfoNCE Loss (Information Noise Contrastive Estimation Loss, contrast learning loss) to the minimum to optimize the learning process from c t to z t+k predictions, For the CNN part, the CNN architecture is hard-coded and the CNN is connected to the GRU. The CPC model is trained on the training data (without labels) using the Adam optimizer.
5. The self-supervised contrastive learning speech emotion recognition method according to claim 1, characterized in that The freezing of parameters in the CNN encoder and connecting it to the new GRU unit includes: After pre-training, the GRU unit is replaced with a new single-layer GRU and trained on the training data using the Adam optimizer. During fine-tuning, shorter samples in the batch need to be padded. Based on the length of each sample and the downsampling rate of the 1D-CNN, the corresponding entry is extracted as the feature representation of the sample and input into the neural network.
6. The self-supervised contrastive learning speech emotion recognition method according to claim 1, characterized in that The final prediction is generated using the final output of the GRU, including: The final GRU output is extracted and passed through a fully connected neural network to obtain the final prediction. The model is tested on the test data. After the first round of training, the misclassified test data is moved to the training data, the model is trained, and then tested on the remaining data. Furthermore, the model is tested on the newly obtained positive examples.
7. A speech emotion recognition device based on self-supervised contrastive learning, characterized in that: The speech emotion recognition pre-training model is deployed to the target domain to obtain an online language emotion classification model for implementing the method according to any one of claims 1 to 6, wherein the speech emotion recognition device based on self-supervised contrastive learning comprises: The data collection and preprocessing module collects audio files during the communication process, and performs data cleaning and normalization on the audio file data to obtain preprocessed data; The model building module builds a self-supervised contrastive learning speech emotion recognition model, inputs the preprocessed data into the CNN encoder, and uses CPC to perform self-supervised pre-training on the CNN encoder; Fine-tuning optimization module, after pre-training, freeze the CNN encoder and connect it to the new GRU unit to complete the fine-tuning of the model; The reasoning application module uses the trained self-supervised contrastive learning speech emotion recognition model to perform reasoning and planning, and applies it to various downstream tasks, so that the model can quickly adapt to new tasks.