A speech emotion recognition method and system based on multi-task learning

Abstract

The application provides a speech emotion recognition method and system based on multi-task learning in the technical field of speech emotion recognition, which comprises the following steps: S1, constructing a speech data set after pre-processing and emotion labeling of acquired speech data; S2, creating a speech emotion recognition model based on a speech feature extraction module, a speech feature fusion module and a multi-task learning module; S3, dividing the speech data set into a training set, a test set and a verification set based on a preset ratio, training the speech emotion recognition model through the training set, testing the trained speech emotion recognition model through the test set, and verifying the tested speech emotion recognition model through the verification set; and S4, publishing the verified speech emotion recognition model through a mini program and accessing the published speech emotion recognition model through the mini program to perform speech emotion recognition. The application has the advantage of greatly improving the accuracy of speech emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech emotion recognition, and particularly discloses a speech emotion recognition method and system based on multi-task learning. BACKGROUND

[0002] With the popularity of intelligent devices and human-computer interaction interfaces, the development of speech emotion recognition technology has become particularly important because it can greatly improve the response ability of machines to human emotions, thereby improving user experience. However, due to the subjectivity and complexity of emotions and the variability of speech signals, accurately recognizing emotional states remains a highly challenging task. Traditional speech emotion recognition methods often rely on manual design and selection of features, which not only consumes time and effort, but also makes it difficult to capture subtle emotional changes in speech. Since it is impossible to fully capture the rich information in speech signals, the accuracy of speech emotion recognition is limited.

[0003] Therefore, how to provide a speech emotion recognition method and system based on multi-task learning to improve the accuracy of speech emotion recognition has become a technical problem to be solved. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a speech emotion recognition method and system based on multi-task learning to improve the accuracy of speech emotion recognition.

[0005] In a first aspect, the present application provides a speech emotion recognition method based on multi-task learning, comprising the following steps:

[0006] Step S1, a large amount of speech data containing several emotions is obtained, and after pre-processing and emotion labeling of each speech data, a speech data set is constructed;

[0007] Step S2, a speech emotion recognition model is created based on a speech feature extraction module, a speech feature fusion module and a multi-task learning module, and a loss function of the speech emotion recognition model is set;

[0008] The speech feature extraction module is constructed based on wav2vec2.0; the multi-task learning module includes an emotion recognition unit and a semantic recognition unit;

[0009] Step S3, the speech data set is divided into a training set, a test set and a validation set based on a preset ratio, the speech emotion recognition model is trained through the training set, the trained speech emotion recognition model is tested through the test set, and the tested speech emotion recognition model is verified through the validation set;

[0010] Step S4, publishing the verified voice emotion recognition model through the applet, and accessing the published voice emotion recognition model through the applet for voice emotion recognition.

[0011] Further, the step S1 is specifically:

[0012] A large amount of voice data containing several emotions is obtained, each of the voice data is converted into a wav format, each of the voice data is segmented based on a preset time threshold to obtain voice segment data, and each of the voice segment data is subjected to noise reduction processing to complete preprocessing of the voice data.

[0013] Each of the voice segment data subjected to noise reduction processing is subjected to emotion labeling to construct a voice data set.

[0014] Further, in the step S2, the voice feature extraction module is configured to generate a plurality of time step low-level feature vectors including acoustic features based on the input voice data, perform a mask operation on each of the time step low-level feature vectors at a preset probability, input each of the low-level feature vectors subjected to the mask operation into a context network, and extract a plurality of time step high-level feature vectors including semantic features.

[0015] The voice feature fusion module is configured to fuse the low-level feature vectors and the high-level feature vectors through attention weights to obtain a self-supervised feature vector.

[0016] The emotion recognition unit is configured to learn the self-supervised feature vector to perform emotion recognition, and the semantic recognition unit is configured to learn the self-supervised feature vector to perform semantic recognition.

[0017] Further, the step S3 is specifically:

[0018] The voice data set is divided into a training set, a test set, and a validation set based on a preset ratio, the voice emotion recognition model is trained through the training set until a loss value of the loss function is less than a preset loss threshold, and the voice emotion recognition model is continuously optimized in the training process at least in terms of hyperparameters including a random inactivation rate, a weight decay rate, and a learning rate.

[0019] The trained voice emotion recognition model is tested through the test set to determine whether the recognition accuracy is greater than a preset accuracy threshold, if not, the test fails, the training set is expanded for continuous training, and if yes, the test succeeds.

[0020] The tested voice emotion recognition model is verified through the validation set to determine whether the confidence is greater than a preset confidence threshold, if not, the verification fails, the training set is expanded for continuous training, and if yes, the verification succeeds.

[0021] Further, the step S4 is specifically:

[0022] The verified voice emotion recognition model is published through the applet, voice emotion recognition is performed by accessing the published voice emotion recognition model through the applet after identity authentication, an emotion recognition result is fed back, an emotion recognition log is recorded and stored, and the voice emotion recognition model is continuously optimized based on the emotion recognition log.

[0023] In a second aspect, the present application provides a voice emotion recognition system based on multi-task learning, comprising the following modules:

[0024] A voice data set construction module is configured to obtain a large amount of voice data containing several emotions, and to construct a voice data set after pre-processing and emotion labeling of each voice data.

[0025] A voice emotion recognition model creation module is configured to create a voice emotion recognition model based on a voice feature extraction module, a voice feature fusion module and a multi-task learning module, and to set a loss function of the voice emotion recognition model.

[0026] The voice feature extraction module is constructed based on wav2vec2.0; and the multi-task learning module comprises an emotion recognition unit and a semantic recognition unit.

[0027] A voice emotion recognition model training module is configured to divide the voice data set into a training set, a test set and a validation set based on a preset proportion, to train the voice emotion recognition model through the training set, to test the trained voice emotion recognition model through the test set, and to verify the tested voice emotion recognition model through the validation set.

[0028] A voice emotion recognition module is configured to publish the verified voice emotion recognition model through the applet, and to perform voice emotion recognition by accessing the published voice emotion recognition model through the applet.

[0029] Further, the voice data set construction module is specifically configured to:

[0030] A large amount of voice data containing several emotions is obtained, each voice data is converted into a wav format, each voice data is segmented based on a preset duration threshold to obtain voice segment data, and each voice segment data is denoised to complete the pre-processing of the voice data.

[0031] Each voice segment data after denoising is labeled with emotions to construct a voice data set.

[0032] Further, in the voice emotion recognition model creating module, the voice feature extraction module is configured to generate a plurality of time step low-level feature vectors including acoustic features according to input voice data, perform a mask operation on each of the time step low-level feature vectors at a preset probability, input each of the low-level feature vectors after the mask operation into a context network, and extract high-level feature vectors including semantic features.

[0033] The voice feature fusion module is configured to fuse the low-level feature vectors and the high-level feature vectors through attention weights to obtain a self-supervised feature vector.

[0034] The emotion recognition unit is configured to learn the self-supervised feature vector to perform emotion recognition, and the semantic recognition unit is configured to learn the self-supervised feature vector to perform semantic recognition.

[0035] Further, the voice emotion recognition model training module is specifically configured to:

[0036] divide the voice data set into a training set, a test set, and a validation set based on a preset ratio, train the voice emotion recognition model through the training set until a loss value of the loss function is less than a preset loss threshold, and continuously optimize hyperparameters of the voice emotion recognition model including a random inactivation rate, a weight decay rate, and a learning rate during the training process.

[0037] test the trained voice emotion recognition model through the test set, determine whether an identification accuracy is greater than a preset accuracy threshold, if not, the test fails, the training set is expanded, and the training is continued, and if yes, the test succeeds.

[0038] verify the tested voice emotion recognition model through the validation set, determine whether a confidence degree is greater than a preset confidence threshold, if not, the verification fails, the training set is expanded, and the training is continued, and if yes, the verification succeeds.

[0039] Further, the voice emotion recognition module is specifically configured to:

[0040] publish the verified voice emotion recognition model through an applet, access the published voice emotion recognition model through the applet after identity authentication to perform voice emotion recognition, feed back an emotion recognition result, record and store an emotion recognition log, and continuously optimize the voice emotion recognition model based on the emotion recognition log.

[0041] The present application has the following advantages:

[0042] 1. A method for recognizing speech emotion, comprising: obtaining a large amount of speech data containing several emotions, preprocessing and emotion labeling each speech data to construct a speech data set; then creating a speech emotion recognition model based on a speech feature extraction module, a speech feature fusion module and a multi-task learning module, setting the loss function of the speech emotion recognition model; wherein the speech feature extraction module is constructed based on wav2vec2.0, which is used to generate several time step low-level feature vectors including acoustic features according to the input speech data, to perform a mask operation on each low-level feature vector at a preset probability, and to input each low-level feature vector after the mask operation into a context network to extract high-level feature vectors including semantic features; the speech feature fusion module is used to fuse the low-level feature vectors and the high-level feature vectors through attention weights to obtain self-supervised feature vectors; the multi-task learning module includes an emotion recognition unit and a semantic recognition unit, the emotion recognition unit is used to learn the self-supervised feature vectors for emotion recognition, and the semantic recognition unit is used to learn the self-supervised feature vectors for semantic recognition; then the speech data set is divided into a training set, a test set and a validation set based on a preset proportion, the speech emotion recognition model is trained through the training set, the trained speech emotion recognition model is tested through the test set, and the tested speech emotion recognition model is verified through the validation set; finally, the verified speech emotion recognition model is published through a small program, and the published speech emotion recognition model is accessed through the small program for speech emotion recognition; that is, the speech emotion recognition model uses the self-supervised network of wav2vec2.0 based on time domain signal as the backbone network to extract more abundant and effective acoustic features and semantic features in the speech data; in an end-to-end manner, the multi-task learning module is used to simultaneously learn emotion recognition and semantic recognition to efficiently recognize speech emotion, and finally greatly improves the accuracy of speech emotion recognition.

[0043] 2. Each speech data is converted into a wav format, segmented based on a preset duration threshold to obtain speech segment data, and each speech segment data is denoised, that is, the speech data is standardized and the influence of noise is reduced, effectively improving the accuracy of speech emotion recognition.

[0044] 3. The speech emotion recognition model is published and accessed through the small program, and the speech emotion recognition model can be quickly accessed through a mobile terminal such as a mobile phone to recognize speech emotion, effectively improving the convenience of speech emotion recognition.

[0045] 4. By recording and storing emotion recognition logs, continuously optimizing the speech emotion recognition model based on the emotion recognition logs, the accuracy of speech emotion recognition can be continuously improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] The application will be further described below with reference to the accompanying drawings in conjunction with the embodiments.

[0047] Fig. 1 is a flowchart of a speech emotion recognition method based on multi-task learning of the application.

[0048] Fig. 2 is a structural schematic diagram of a speech emotion recognition system based on multi-task learning of the application. DETAILED DESCRIPTION

[0049] The technical solution in the embodiments of the application has the following general idea: the speech emotion recognition model is set to use a self-supervised network based on a time domain signal wav2vec2.0 as a backbone network to extract more abundant and effective acoustic features and semantic features in speech data; an end-to-end manner is used to simultaneously perform learning of emotion recognition and semantic recognition through a multi-task learning module to efficiently recognize speech emotion and further improve the accuracy of speech emotion recognition.

[0050] Please refer to Figs. 1-2 The preferred embodiment of the speech emotion recognition method based on multi-task learning of the application includes the following steps:

[0051] Step S1: A large amount of speech data containing several emotions is acquired, and after pre-processing and emotion labeling of each speech data, a speech data set is constructed;

[0052] Step S2: A speech emotion recognition model is created based on a speech feature extraction module, a speech feature fusion module and a multi-task learning module, and a loss function of the speech emotion recognition model is set;

[0053] The speech feature extraction module is constructed based on wav2vec2.0; the multi-task learning module includes an emotion recognition unit and a semantic recognition unit; the loss function includes a first loss sub-function for training the emotion recognition unit and a second loss sub-function for training the semantic recognition unit; when the speech emotion recognition model is trained, the loss values of the first loss sub-function and the second loss sub-function are calculated simultaneously for joint optimization;

[0054] Step S3: The speech data set is divided into a training set, a test set and a validation set based on a preset ratio, the speech emotion recognition model is trained through the training set, the trained speech emotion recognition model is tested through the test set, and the tested speech emotion recognition model is verified through the validation set;

[0055] Step S4: The verified speech emotion recognition model is published through a mini program, and the published speech emotion recognition model is accessed through the mini program for speech emotion recognition.

[0056] The voice emotion recognition model is published and accessed through the applet, and the voice emotion recognition model can be quickly accessed through a mobile terminal such as a mobile phone to perform voice emotion recognition, thereby effectively improving the convenience of voice emotion recognition.

[0057] The step S1 is specifically:

[0058] A large amount of voice data containing several emotions is obtained, each of the voice data is converted into a wav format, each of the voice data is segmented based on a preset duration threshold to obtain voice segment data, and each of the voice segment data is subjected to noise reduction processing to complete preprocessing of the voice data; the duration threshold can be 10S;

[0059] Each of the voice segment data subjected to noise reduction processing is subjected to emotion labeling to construct a voice data set.

[0060] By converting each of the voice data into a wav format, each of the voice data is segmented based on a preset duration threshold to obtain voice segment data, and each of the voice segment data is subjected to noise reduction processing, that is, the voice data is subjected to standardized processing and the influence of noise is reduced, thereby effectively improving the accuracy of voice emotion recognition.

[0061] In the step S2, the voice feature extraction module is configured to generate a plurality of time step low-level feature vectors including acoustic features based on the input voice data, perform a mask operation on each of the time step low-level feature vectors at a preset probability, input each of the low-level feature vectors subjected to the mask operation into a context network, and extract a high-level feature vector including semantic features;

[0062] The voice feature fusion module is configured to fuse the low-level feature vector and the high-level feature vector through attention weights to obtain a self-supervised feature vector;

[0063] The emotion recognition unit is configured to learn the self-supervised feature vector to perform emotion recognition, and the semantic recognition unit is configured to learn the self-supervised feature vector to perform semantic recognition.

[0064] The step S3 is specifically:

[0065] The voice data set is divided into a training set, a test set, and a validation set based on a preset ratio, the voice emotion recognition model is trained through the training set until a loss value of the loss function is less than a preset loss threshold, and the voice emotion recognition model is continuously optimized during the training process, at least including hyperparameters of a random inactivation rate, a weight decay rate, and a learning rate;

[0066] Test the trained voice emotion recognition model through the test set to determine whether the recognition accuracy is greater than a preset accuracy threshold, if not, the test fails, and the training set is expanded to continue training, if yes, the test succeeds:

[0067] Verify the tested voice emotion recognition model through the verification set to determine whether the confidence is greater than a preset confidence threshold, if not, the verification fails, and the training set is expanded to continue training, if yes, the verification succeeds.

[0068] The step S4 specifically includes:

[0069] The voice emotion recognition model after verification is published through the applet, and after identity authentication through the applet, the published voice emotion recognition model is accessed to perform voice emotion recognition, the emotion recognition result is fed back, the emotion recognition log is recorded and stored, and the voice emotion recognition model is continuously optimized based on the emotion recognition log. The emotion recognition log at least includes input voice data, emotion recognition result, recognition time, user ID and user feedback.

[0070] By recording and storing the emotion recognition log, the voice emotion recognition model is continuously optimized based on the emotion recognition log, which can continuously improve the accuracy of voice emotion recognition.

[0071] A preferred embodiment of a voice emotion recognition system based on multi-task learning includes the following modules:

[0072] A voice data set construction module is used to obtain a large amount of voice data containing several emotions, and to construct a voice data set after pre-processing and emotion labeling of each voice data.

[0073] A voice emotion recognition model creation module is used to create a voice emotion recognition model based on a voice feature extraction module, a voice feature fusion module and a multi-task learning module, and to set a loss function of the voice emotion recognition model.

[0074] The voice feature extraction module is constructed based on wav2vec2.0; the multi-task learning module includes an emotion recognition unit and a semantic recognition unit; the loss function includes a first loss sub-function for training the emotion recognition unit and a second loss sub-function for training the semantic recognition unit; when the voice emotion recognition model is trained, the loss values of the first loss sub-function and the second loss sub-function are calculated simultaneously for joint optimization.

[0075] The voice emotion recognition model training module is configured to divide the voice data set into a training set, a test set and a verification set based on a preset proportion, train a voice emotion recognition model based on the training set, test the trained voice emotion recognition model based on the test set, and verify the tested voice emotion recognition model based on the verification set.

[0076] The voice emotion recognition module is configured to publish the verified voice emotion recognition model through the applet and access the published voice emotion recognition model through the applet to perform voice emotion recognition.

[0077] The voice emotion recognition model is published and accessed through the applet, and the voice emotion recognition model can be quickly accessed through a mobile terminal such as a mobile phone to perform voice emotion recognition, thereby effectively improving the convenience of voice emotion recognition.

[0078] The voice data set construction module is specifically configured to:

[0079] A large amount of voice data containing several emotions is obtained, each voice data is converted into a wav format, each voice data is segmented based on a preset duration threshold to obtain voice segment data, and each voice segment data is subjected to noise reduction processing to complete the preprocessing of the voice data; the duration threshold can be 10S.

[0080] Each voice segment data subjected to noise reduction processing is subjected to emotion labeling to construct a voice data set.

[0081] By converting each voice data into a wav format, segmenting each voice data based on a preset duration threshold to obtain voice segment data, and subjecting each voice segment data to noise reduction processing, the voice data is standardized and the influence of noise is reduced, thereby effectively improving the accuracy of voice emotion recognition.

[0082] In the voice emotion recognition model creation module, the voice feature extraction module is configured to generate a plurality of time step low-level feature vectors including acoustic features based on input voice data, perform a mask operation on each time step low-level feature vector at a preset probability, input each low-level feature vector subjected to the mask operation into a context network, and extract a plurality of high-level feature vectors including semantic features.

[0083] The voice feature fusion module is configured to fuse the low-level feature vectors and the high-level feature vectors through attention weights to obtain a self-supervised feature vector.

[0084] The emotion recognition unit is configured to learn the self-supervised feature vector to perform emotion recognition, and the semantic recognition unit is configured to learn the self-supervised feature vector to perform semantic recognition.

[0085] The voice emotion recognition model training module is specifically used for:

[0086] The voice data set is divided into a training set, a test set and a verification set based on a preset ratio, the voice emotion recognition model is trained through the training set, until the loss value of the loss function is less than a preset loss threshold, and the voice emotion recognition model is continuously optimized in the training process, at least including hyperparameters of a random inactivation rate, a weight decay rate and a learning rate;

[0087] The trained voice emotion recognition model is tested through the test set, whether the recognition accuracy is greater than a preset accuracy threshold is judged, if not, the test fails, the training set is expanded to continue training, if yes, the test succeeds:

[0088] The tested voice emotion recognition model is verified through the verification set, whether the confidence is greater than a preset confidence threshold is judged, if not, the verification fails, the training set is expanded to continue training, if yes, the verification succeeds.

[0089] The voice emotion recognition module is specifically used for:

[0090] The voice emotion recognition model after verification is published through a small program, the voice emotion recognition model is accessed through the small program after identity authentication to perform voice emotion recognition, the emotion recognition result is fed back, the emotion recognition log is recorded and stored, and the voice emotion recognition model is continuously optimized based on the emotion recognition log. The emotion recognition log at least includes input voice data, emotion recognition result, recognition time, user ID and user feedback.

[0091] By recording and storing the emotion recognition log, the voice emotion recognition model is continuously optimized based on the emotion recognition log, and the accuracy of voice emotion recognition can be continuously improved.

[0092] In summary, the advantages of the present application are:

[0093] 1. A method for recognizing speech emotion, comprising: obtaining a large amount of speech data containing several emotions, preprocessing and emotion labeling each speech data to construct a speech data set; then creating a speech emotion recognition model based on a speech feature extraction module, a speech feature fusion module and a multi-task learning module, setting the loss function of the speech emotion recognition model; wherein the speech feature extraction module is constructed based on wav2vec2.0, which is used to generate low-level feature vectors including acoustic features for several time steps according to the input speech data, to perform a mask operation on each low-level feature vector at a preset probability, and to input each low-level feature vector after the mask operation into a context network to extract high-level feature vectors including semantic features; the speech feature fusion module is used to fuse the low-level feature vectors and the high-level feature vectors through attention weights to obtain self-supervised feature vectors; the multi-task learning module includes an emotion recognition unit and a semantic recognition unit, the emotion recognition unit is used to learn the self-supervised feature vectors for emotion recognition, and the semantic recognition unit is used to learn the self-supervised feature vectors for semantic recognition; then the speech data set is divided into a training set, a test set and a validation set based on a preset proportion, the speech emotion recognition model is trained through the training set, the trained speech emotion recognition model is tested through the test set, and the tested speech emotion recognition model is verified through the validation set; finally, the verified speech emotion recognition model is published through a small program, and the published speech emotion recognition model is accessed through the small program for speech emotion recognition; that is, the speech emotion recognition model uses a self-supervised network based on the wav2vec2.0 time domain signal as the backbone network to extract more abundant and effective acoustic features and semantic features in the speech data; in an end-to-end manner, the multi-task learning module is used to simultaneously learn emotion recognition and semantic recognition to efficiently recognize speech emotion, and finally greatly improves the accuracy of speech emotion recognition.

[0094] 2. Each speech data is converted into a wav format, segmented based on a preset duration threshold to obtain speech segment data, and each speech segment data is denoised, that is, the speech data is standardized and the influence of noise is reduced, effectively improving the accuracy of speech emotion recognition.

[0095] 3. The speech emotion recognition model is published and accessed through the small program, and the speech emotion recognition model can be quickly accessed through a mobile terminal such as a mobile phone to recognize speech emotion, effectively improving the convenience of speech emotion recognition.

[0096] 4. By recording and storing emotion recognition logs, continuously optimizing the speech emotion recognition model based on the emotion recognition logs, the accuracy of speech emotion recognition can be continuously improved.

[0097] While the foregoing description has described specific embodiments of the application, one ordinary skill in the art will appreciate that various modifications and changes can be made thereto without departing from the spirit and scope of the application, as set forth in the appended claims.

Claims

1. A speech emotion recognition method based on multi-task learning, characterized in that: Includes the following steps: Step S1: Obtain a large amount of speech data containing several emotions, and construct a speech dataset after preprocessing and emotion annotation of each speech data. Step S2: Create a speech emotion recognition model based on the speech feature extraction module, speech feature fusion module and multi-task learning module, and set the loss function of the speech emotion recognition model; The speech feature extraction module is built on wav2vec2.0; the multi-task learning module includes an emotion recognition unit and a semantic recognition unit. Step S3: Divide the speech dataset into a training set, a test set, and a validation set based on a preset ratio. Train the speech emotion recognition model using the training set, test the trained speech emotion recognition model using the test set, and validate the tested speech emotion recognition model using the validation set. Step S4: Publish the verified voice emotion recognition model through the mini-program, and access the published voice emotion recognition model through the mini-program to perform voice emotion recognition.

2. The speech emotion recognition method based on multi-task learning as described in claim 1, characterized in that: Step S1 specifically involves: Acquire a large amount of voice data containing several emotions, convert each voice data into WAV format, segment each voice data into voice segment data based on a preset duration threshold, and perform noise reduction processing on each voice segment data to complete the preprocessing of the voice data. A speech dataset is constructed by performing emotion annotation on each of the noise-reduced speech segments.

3. The speech emotion recognition method based on multi-task learning as described in claim 1, characterized in that: In step S2, the speech feature extraction module is used to generate several low-level feature vectors including acoustic features at several time steps based on the input speech data, perform a masking operation on the low-level feature vectors at each time step with a preset probability, and input the low-level feature vectors after the masking operation into the context network to extract high-level feature vectors including semantic features. The speech feature fusion module is used to fuse low-level feature vectors and high-level feature vectors through attention weights to obtain self-supervised feature vectors. The emotion recognition unit is used to learn the self-supervised feature vector for emotion recognition; the semantic recognition unit is used to learn the self-supervised feature vector for semantic recognition.

4. The speech emotion recognition method based on multi-task learning as described in claim 1, characterized in that: Step S3 specifically involves: The speech dataset is divided into a training set, a test set, and a validation set based on a preset ratio. The speech emotion recognition model is trained using the training set until the loss value of the loss function is less than a preset loss threshold. During the training process, the hyperparameters of the speech emotion recognition model, including at least the random inactivation rate, weight decay rate, and learning rate, are continuously optimized. The trained speech emotion recognition model is tested using the test set to determine if the recognition accuracy exceeds a preset accuracy threshold. If not, the test fails, and the training set is expanded for further training; if yes, the test succeeds. The tested speech emotion recognition model is validated using the validation set to determine whether the confidence level is greater than a preset confidence threshold. If not, the validation fails, and the training set is expanded for continued training; if so, the validation succeeds.

5. The speech emotion recognition method based on multi-task learning as described in claim 1, characterized in that: Step S4 specifically involves: The verified voice emotion recognition model is published through a mini-program. After identity authentication through the mini-program, users can access the published voice emotion recognition model to perform voice emotion recognition, receive feedback on the emotion recognition results, record and store the emotion recognition log, and continuously optimize the voice emotion recognition model based on the emotion recognition log.

6. A speech emotion recognition system based on multi-task learning, characterized in that: Includes the following modules: The speech dataset construction module is used to acquire a large amount of speech data containing several emotions, and to construct a speech dataset after preprocessing and emotion annotation of each speech data. The speech emotion recognition model creation module is used to create a speech emotion recognition model based on the speech feature extraction module, the speech feature fusion module, and the multi-task learning module, and to set the loss function of the speech emotion recognition model. The speech feature extraction module is built on wav2vec2.0; the multi-task learning module includes an emotion recognition unit and a semantic recognition unit. The speech emotion recognition model training module is used to divide the speech dataset into a training set, a test set, and a validation set based on a preset ratio. The speech emotion recognition model is trained using the training set, tested using the test set, and validated using the validation set. The voice emotion recognition module is used to publish the verified voice emotion recognition model through a mini-program, and to access the published voice emotion recognition model through the mini-program to perform voice emotion recognition.

7. A speech emotion recognition system based on multi-task learning as described in claim 6, characterized in that: The speech dataset construction module is specifically used for: Acquire a large amount of voice data containing several emotions, convert each voice data into WAV format, segment each voice data into voice segment data based on a preset duration threshold, and perform noise reduction processing on each voice segment data to complete the preprocessing of the voice data. A speech dataset is constructed by performing emotion annotation on each of the noise-reduced speech segments.

8. A speech emotion recognition system based on multi-task learning as described in claim 6, characterized in that: In the speech emotion recognition model creation module, the speech feature extraction module is used to generate several low-level feature vectors including acoustic features at several time steps based on the input speech data, perform a masking operation on the low-level feature vectors at each time step with a preset probability, input the low-level feature vectors after the masking operation into the context network, and extract high-level feature vectors including semantic features. The speech feature fusion module is used to fuse low-level feature vectors and high-level feature vectors through attention weights to obtain self-supervised feature vectors. The emotion recognition unit is used to learn the self-supervised feature vector for emotion recognition; the semantic recognition unit is used to learn the self-supervised feature vector for semantic recognition.

9. A speech emotion recognition system based on multi-task learning as described in claim 6, characterized in that: The speech emotion recognition model training module is specifically used for: The speech dataset is divided into a training set, a test set, and a validation set based on a preset ratio. The speech emotion recognition model is trained using the training set until the loss value of the loss function is less than a preset loss threshold. During the training process, the hyperparameters of the speech emotion recognition model, including at least the random inactivation rate, weight decay rate, and learning rate, are continuously optimized. The trained speech emotion recognition model is tested using the test set to determine if the recognition accuracy exceeds a preset accuracy threshold. If not, the test fails, and the training set is expanded for further training; if yes, the test succeeds. The tested speech emotion recognition model is validated using the validation set to determine whether the confidence level is greater than a preset confidence threshold. If not, the validation fails, and the training set is expanded for continued training; if so, the validation succeeds.

10. A speech emotion recognition system based on multi-task learning as described in claim 6, characterized in that: The voice emotion recognition module is specifically used for: The verified voice emotion recognition model is published through a mini-program. After identity authentication through the mini-program, users can access the published voice emotion recognition model to perform voice emotion recognition, receive feedback on the emotion recognition results, record and store the emotion recognition log, and continuously optimize the voice emotion recognition model based on the emotion recognition log.

Images