Speech emotion recognition method based on multi-label correction and spatio-temporal collaborative fusion

Abstract

This invention belongs to the field of artificial intelligence, and specifically relates to a speech emotion recognition method based on multi-label correction and spatiotemporal collaborative fusion. It constructs a spatiotemporal collaborative fusion network including a temporal module, a spatial module, a collaborative fusion module, and a classification module. The network is pre-trained and optimized using single-label speech. The pre-trained spatiotemporal collaborative fusion network is then used to modify the emotion labels of ambiguous speech. The ambiguous speech with corrected labels and the single-label speech are then mixed, and the spatiotemporal collaborative fusion network is retrained and optimized to perform speech emotion recognition. This invention focuses on how to successfully utilize speech samples with ambiguous labels, enabling the network to obtain emotional information from ambiguous speech, thus improving the utilization rate of existing samples when the number of training samples is scarce.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, and specifically relates to a speech emotion recognition method based on multi-label correction and spatiotemporal collaborative fusion. Background Technology

[0002] Speech is one of the primary mediums of natural human communication, conveying not only the speaker's purpose but also their emotional state. The process of using computers to identify a speaker's emotional state from speech is called speech emotion recognition. It is an important task in human-computer interaction, helping intelligent voice interaction systems understand the user's underlying intentions and bringing a better user experience to products. Examples include voice assistants in call centers, voice services in smart homes, emotion detection systems in autonomous driving, and emotion-assisted therapy in healthcare. With the increasing demand for these applications, speech emotion recognition has attracted growing attention from researchers.

[0003] Common speech emotion recognition methods are based on single-label training, meaning each speech entry corresponds to a fixed real label, representing that the speech contains only one emotion. However, in real life, most speech emotions are ambiguous, often mixing multiple emotions; for example, a speech expressing sadness may contain expressions of anger and disappointment. Furthermore, emotion experts may present different interpretations of ambiguous emotions based on their own culture and personality, meaning that emotion perception is subjective. Therefore, representing the true emotion of speech with a single label not only lacks the ability to express mixed emotions but also ignores the subjectivity of emotion experts' perception of emotions.

[0004] Recognizing these issues, a multi-label approach was proposed, using label definitions to represent the ambiguity of emotions and the subjectivity of emotion perception. Multi-label approaches include two types: one calculates the percentage of votes from emotion experts for various emotions to describe the ambiguity of emotions, but this fixed percentage does not represent the true percentage accepted by the majority; the other is not limited by percentages, but estimates the presence or absence of an emotion solely based on whether emotion experts vote for it, but still relies on the emotional perceptions assigned by some emotion experts and lacks a clear emotional bias.

[0005] Consequently, more effective training methods for handling fuzzy emotions were developed, such as joint learning, meta-learning, sentiment profiling, and multi-classifier interaction. These methods aim to combine the model's own knowledge to avoid the problem of single-label and multi-label methods relying on labels assigned by sentiment experts. However, these methods only consider speech samples with single labels, i.e., samples that can obtain consensus among most sentiment experts, without utilizing unlabeled speech samples in the dataset. However, the ambiguity of speech sentiment is mainly reflected in these unlabeled samples. Since unlabeled samples arise from the inability of sentiment experts to reach a consensus on the sentiment of the speech, this indicates that the sentiment of the sample is ambiguous, making it difficult for humans to recognize. Furthermore, not every sentence in a real-world environment carries the sentiment that is widely accepted. Therefore, these methods that do not utilize unlabeled speech samples do not fully consider speech samples with truly ambiguous sentiment. Summary of the Invention

[0006] In view of this, this invention proposes a speech emotion recognition method based on multi-label correction and spatiotemporal collaborative fusion. A spatiotemporal collaborative fusion network is constructed, comprising a temporal module, a spatial module, a collaborative fusion module, and a classification module. This network is pre-trained and optimized using single-label speech. The pre-trained spatiotemporal collaborative fusion network is then used to modify the emotion labels of ambiguous speech. The label-corrected ambiguous speech and single-label speech are then mixed, and the spatiotemporal collaborative fusion network is retrained and optimized. The optimized spatiotemporal collaborative fusion network then performs speech emotion recognition. The specific steps include:

[0007] S1. Based on the spatial characteristics of the speech spectrum and the temporal characteristics of the speech waveform, convolutional neural networks and Wav2vec models are used to extract emotional features from the spatial and temporal domains of speech, respectively. A collaborative fusion method is adopted to realize the interaction of spatiotemporal features and form a spatiotemporal collaborative fusion network.

[0008] S2. Initialize the spatiotemporal collaborative fusion network. Use the speech with a single emotion tag as the first training set to pre-train the spatiotemporal collaborative fusion network. This pre-trained network is called M. p ;

[0009] S3, The pre-trained spatiotemporal collaborative fusion network M completes the input of ambiguous speech. p The generated sentiment labels for the input samples are predicted.

[0010] S4. Combine the generated sentiment labels of ambiguous speech with the original multi-labels for label correction to obtain ambiguous speech samples with correction labels.

[0011] S5. Using speech with a single emotion label and ambiguous speech with a correction label as the second training set, retrain and optimize the spatiotemporal collaborative fusion network, and call this network M. f ;

[0012] Ambiguous speech refers to speech samples with multiple emotional labels.

[0013] Furthermore, the spatiotemporal collaborative fusion network includes:

[0014] 101. In the time domain module, the Wav2vec model is used to extract time domain emotional features from the original waveform of the speech information;

[0015] 102. In the spatial domain module, a convolutional neural network is used to process the frequency domain of the speech message to obtain the spatial domain emotional features of the speech message.

[0016] 103. In the collaborative fusion module, a fully connected layer is used to convert spatial domain sentiment features into spatial sentiment weights. The spatial sentiment weights are multiplied and fused with temporal domain sentiment features to obtain temporal sentiment features with spatial sentiment information.

[0017] 104. Input the temporal sentiment features with spatial sentiment information into the classification module to obtain the sentiment classification results.

[0018] Furthermore, the loss function for pre-training the spatiotemporal collaborative fusion network, using samples with a single sentiment label as the first training set, is expressed as:

[0019]

[0020] Where N represents the total number of samples in the sentiment dataset. Represented as a single emotion tag in speech. The pre-trained spatiotemporal collaborative fusion network M represents p The predicted output, x i This represents the i-th sample input to the network.

[0021] Furthermore, individual emotion tags in speech Represented as:

[0022]

[0023] in, The expression indicates whether the i-th sample in the sentiment dataset belongs to the j-th sentiment category, where N represents the total number of samples in the sentiment dataset, and K represents the number of sentiment categories.

[0024] Furthermore, the loss function used to train the spatiotemporal collaborative fusion network by using samples with a single sentiment label and ambiguous samples with a correction label as the second training set is expressed as follows:

[0025]

[0026] Where N represents the total number of samples in the sentiment dataset. Indicates a correction label. Network M f The predicted output, x i This represents the i-th sample input to the network.

[0027] Furthermore, the process of obtaining the corrected label includes:

[0028]

[0029] in, The label x represents the result after error correction of ambiguous samples. i M represents the i-th sample input to the network. p This represents a pre-trained spatiotemporal collaborative fusion network; The original labels for ambiguous samples; For pre-trained spatiotemporal collaborative fusion network M p The generated label, λ∈[0,1] represents the balance factor.

[0030] Furthermore, the original labels of ambiguous samples The multi-label form obtained by adding multiple one-hot vectors is represented as:

[0031]

[0032] in, Let N represent whether the i-th sample in the sentiment dataset contains the j-th sentiment category, N represents the total number of samples in the sentiment dataset, and K represents the number of sentiment categories.

[0033] Furthermore, generate tags. It can be divided into two forms: multi-tag and single-tag. The multi-tag form generates tags. For network M p For input sample x i The predicted output is expressed as:

[0034]

[0035] Generate tags in single-tag format Further research is needed on network M. p The category with the highest probability is selected from the output probability distribution vector and represented as a one-hot vector as follows:

[0036]

[0037]

[0038] in, The pre-trained spatiotemporal collaborative fusion network M representsp The probability distribution vector of the predicted output, x i This represents the i-th sample input to the network; j,k∈[1,K] are integers. express The value of the k-th sentiment category, for The value of the j-th sentiment category.

[0039] The beneficial effects of this invention are as follows:

[0040] 1) A spatiotemporal collaborative fusion network was constructed, which collaboratively fused spatial domain emotion information obtained from the speech spectrum with temporal domain emotion classification of the speech waveform, thereby improving the performance of speech emotion recognition.

[0041] 2) Provides a reference method for exploring and processing speech emotion models for labeling ambiguous speech samples, and shows that samples with ambiguous emotion annotators can still help the network build emotion cognition.

[0042] 3) An effective label correction strategy is proposed, which modifies the emotionless multi-labels by generating labels through a pre-trained spatiotemporal collaborative fusion network, so that the model training does not completely depend on the emotional cognition of a few emotion experts, and the corrected labels have a clear emotional bias through a balancing factor.

[0043] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0044] Figure 1 This describes the process of the speech emotion recognition method based on multi-label correction and spatiotemporal collaborative fusion in this invention;

[0045] Figure 2 This is a system model diagram of the spatiotemporal collaborative fusion network of the present invention;

[0046] Figure 3 This is a flowchart of the label correction strategy of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] This invention provides a speech emotion recognition method based on multi-label correction and spatiotemporal collaborative fusion. The method constructs a spatiotemporal collaborative fusion network comprising a temporal module, a spatial module, a collaborative fusion module, and a classification module. The network is pre-trained and optimized using single-label speech. The pre-trained spatiotemporal collaborative fusion network is then used to modify the emotion labels of ambiguous speech. The corrected ambiguous speech and single-label speech are then mixed, and the spatiotemporal collaborative fusion network is retrained and optimized. The optimized spatiotemporal collaborative fusion network then performs speech emotion recognition. The specific steps include:

[0049] S1. Based on the spatial characteristics of the speech spectrum and the temporal characteristics of the speech waveform, convolutional neural networks and Wav2vec models are used to extract emotional features from the spatial and temporal domains of speech, respectively. A collaborative fusion method is adopted to realize the interaction of spatiotemporal features and form a spatiotemporal collaborative fusion network.

[0050] S2. Initialize the spatiotemporal collaborative fusion network. Use the speech with a single emotion tag as the first training set to pre-train the spatiotemporal collaborative fusion network. This pre-trained network is called M. p ;

[0051] S3, The pre-trained spatiotemporal collaborative fusion network M completes the input of ambiguous speech. p The generated sentiment labels for the input samples are predicted.

[0052] S4. Combine the generated sentiment labels of ambiguous speech with the original multi-labels for label correction to obtain ambiguous speech samples with correction labels.

[0053] S5. Using speech with a single emotion label and ambiguous speech with a correction label as the second training set, retrain and optimize the spatiotemporal collaborative fusion network, and call this network M. f .

[0054] In this embodiment, during the implementation of the present invention, as follows: Figure 1 It mainly includes the following steps:

[0055] Step 1: Construct a spatiotemporal collaborative fusion network including a temporal module, a spatial module, a collaborative fusion module, and a classification module;

[0056] Step 2: Pre-train and optimize the spatiotemporal collaborative fusion network using speech with a single emotion tag;

[0057] Step 3: Modify the sentiment labels of ambiguous speech using a pre-trained spatiotemporal collaborative fusion network;

[0058] Step 4: Mix the ambiguous speech after label correction with the single-label speech, and retrain and optimize the spatiotemporal collaborative fusion network;

[0059] Step 5: Complete the optimized spatiotemporal collaborative fusion network to perform emotion recognition on speech.

[0060] The spatiotemporal coordination and fusion network of this invention includes a temporal module, a spatial module, a collaborative fusion module, and a classification module. Figure 2 This is a model diagram of the spatiotemporal collaborative fusion network system of the present invention. The following is in conjunction with... Figure 2 Explanation of each module:

[0061] 1) Time domain module.

[0062] Speech is a continuous signal of variable length with temporal characteristics, and its emotional expression can dynamically change over time. Benefiting from advances in speech recognition research, the Wav2vec model, pre-trained with a large amount of unlabeled data, can construct contextualized speech feature representations. This is because Wav2vec is based on the Transformer concept, learning from the contextual information of the current input to predict certain future sampling points. Therefore, the pre-trained Wav2vec model can obtain a contextualized potential emotional representation f from the raw waveform of the speech. w :

[0063]

[0064] in Let θ represent the i-th speech waveform input to the network. w These are a series of trainable parameters for the Wav2vec model. T represents the acquired latent sentiment representation with contextual features. t d represents the time dimension and its size depends on the length of the input speech. t Indicates the feature dimension.

[0065] Secondly, since the length of the input speech waveform is variable, the latent sentiment representation processed by the Wav2vec model still exhibits length variability in the time dimension, which is detrimental to subsequent feature fusion and classification. Therefore, Temporal Pyramid Pooling (TPP) is introduced to handle this variability. This pooling method is widely used in audio and video processing, extracting multi-level information from the time dimension and outputting a fixed-dimensional output from a variable time dimension. The fixed-dimensional output depends on the pyramid levels (PL) setting. For example, if PL = {1, 2, 3}, then performing 3 region partitions and 6 pooling operations on the feature's time axis results in a time dimension of 6. Here, we assume n pooling operations are performed, meaning the fixed time dimension is n:

[0066] f t =TPP(f w )

[0067] in This represents a fixed-size temporal sentiment feature output by the temporal pyramid pooling.

[0068] 2) Spatial domain module.

[0069] The spatial domain of the speech spectrum not only possesses time and frequency dimensions but also includes the frequency variation over time. To describe the features of each dimension of the speech spectrum spatial domain, three parallel CNN layers are used in the first layer of the spatial domain module. A CNN layer with a longer temporal kernel is used to extract the temporal information of the spectrum; a CNN layer with a longer frequency kernel is used to extract the frequency information of the spectrum; and a CNN layer with kernels of the same temporal and frequency dimensions is used to extract the temporal and frequency variation relationship of the frequency. Each CNN layer is a combination of CNN, batch normalization, ReLU activation function, and average pooling. After the above processing, the results of the three branches are concatenated to form a new feature map f. a :

[0070]

[0071] in Let conv be the i-th speech spectrum input to the network. 1a (·), conv 1b (·) and conv 1c (·) represents the convolutions that capture the spectral time, frequency, and the relationship between time and frequency, respectively, and θ 1a ,θ 1b ,θ 1c These are the trainable parameters for the CNN layer, and Concat(·) represents the connection operation.

[0072] Secondly, a five-layer CNN is used to stack the feature maps f. a The model learns more fine-grained spatial sentiment features. Furthermore, after feature extraction, the first three CNN layers all incorporate average pooling to reduce the dimensionality of the features. This aims to focus the model on specific sentiment-triggered regions and prevent overfitting due to excessive parameters. After these processing steps, the feature map f... a Processed into feature map f c :

[0073] f c =conv 5 (θ c ,f a )

[0074] Where conv 5 (·) represents the feature extraction process of a five-layer CNN, θ c It is a set of trainable parameters.

[0075] Finally, the temporal pyramid pooling method is used to fix the size of the feature map's temporal dimension. However, since the feature dimension is already condensed after dimensionality reduction through multiple layers of CNN and average pooling, using multiple layers of information extraction would introduce redundant zero-padding information. Therefore, setting PL = {1} is sufficient. This is equivalent to automatically averaging along the time axis using global average pooling (GAP), the process of which is represented as:

[0076] f s =GAP(f c )

[0077] in d s Dimensions that represent spatial features.

[0078] 3) Collaborative Integration Module.

[0079] Because the temporal domain module lacks extraction of speech frequency characteristics, and the spatial domain module lacks long-term dependencies in speech context, combining the advantages of the temporal and spatial domains is a worthwhile research topic. Inspired by the idea of ​​collaborative attention, this paper integrates spatial domain sentiment features into temporal sentiment features, achieving spatial domain features to assist temporal domain sentiment classification. First, a fully connected layer is used to integrate the spatial domain sentiment features f s Convert to spatial emotional weight f s ′:

[0080] f s ′=δ(f s W s +B s )

[0081] in n represents the temporal dimension of the temporal sentiment feature, and δ(·) represents the ReLU activation function. and These are the trainable parameters for the fully connected layer.

[0082] Secondly, the spatial sentiment weights obtained above are applied to the temporal sentiment features to obtain temporal sentiment features f with spatial sentiment information. t ′:

[0083] f t ′=f s ′·f t

[0084] in d t The dimension representing the temporal emotional characteristics that are accompanied by spatial emotional information.

[0085] 4) Classification module.

[0086] To enable the model to learn distributed feature representations more meticulously, a multi-layer fully connected layer is used to map the above sentiment features to classification outputs. This process is represented as follows:

[0087]

[0088]

[0089] in All are fully connected trainable parameters, K is the number of categories in the classification task, δ(·) represents the ReLU activation function, and * refers to a certain model. That is, this invention does not limit the classification model used, and those skilled in the art can choose the best one according to the situation.

[0090] Figure 3 The flowchart for the label correction strategy is shown below with reference to the accompanying diagram, and includes the following four steps:

[0091] Step 1: Pre-training

[0092] A spatiotemporal collaborative fusion network M with emotion cognition is generated by pre-training with single-label samples. p First, the spatiotemporal collaborative fusion network is initialized. Second, sentiment experts in the dataset assign multiple sentiment labels to each sample. Single-labeling selects the sentiment category with the most votes based on the majority principle, which fails to express the variability of sentiment and the subjectivity of sentiment cognition. Multi-labeling retains the annotation information of each sentiment expert, expressing a mixture of multiple sentiments. Based on whether the sample label has multiple sentiment annotations, all samples can be categorized into three types: samples S with only a single label. A Ambiguous but single-labeled sample S B Only ambiguous sample SC In the manual classification process, the label of a sample is determined according to the principle of majority voting. For example, if three people vote: for a sample, all three people vote for label A, then this sample has only a single label; for a sample, one person votes for label A and the others vote for label B, then label B is the single label for this sample, and labels A and B are the multiple labels for this sample. That is, the majority class is the single label, and the majority class and the minority class together constitute the multiple labels. Such a sample is ambiguous but has a single label; for a sample, all three people vote differently, then this sample has only multiple labels and belongs to the ambiguous category. Specifically, when multiple sentiment experts all generate a consistent sentiment label for a sample, this type of sample is identified as S. A When most sentiment experts produce consistent sentiment labels but a small number of inconsistent sentiment labels appear, this type of sample belongs to S. B When most sentiment experts cannot reach a consensus on sentiment labels, such samples are categorized as S. C Finally, samples with a single label, namely S, are used. A and S B The initialized spatiotemporal collaborative fusion network is pre-trained to enable it to develop emotion recognition. In a supervised K-class speech emotion classification study, single-label is a common representation of the true labels in emotion datasets. It is expressed as a one-hot vector, where the class receiving the majority of votes from the annotators is dominant and the remaining classes are set to 0. Its definition is as follows:

[0093]

[0094] in, The expression indicates whether the i-th sample in the sentiment dataset belongs to the j-th sentiment category, where N represents the total number of samples in the sentiment dataset, and K represents the number of sentiment categories.

[0095] Then, a pre-trained spatiotemporal collaborative fusion network M with emotion cognition is generated by training using traditional multi-class cross-entropy. p Its definition is as follows:

[0096]

[0097] in, The pre-trained spatiotemporal collaborative fusion network M represents p The predicted output, x i This represents the i-th sample input to the network.

[0098] Step 2: Tag Generation

[0099] A spatiotemporal collaborative fusion network with sentiment recognition is used to generate labels for samples with ambiguous labels. Since the samples used in pre-training are all single-labeled and have relatively clear sentiment, the trained spatiotemporal collaborative fusion network M... p It can learn precise sentiment knowledge. Using this network, labels are generated for ambiguous samples, enabling labeling of samples from the network's sentiment perspective, rather than relying solely on the sentiment cognition of a small number of sentiment experts. This is based on the spatiotemporal collaborative fusion network M... p The generated labels can be categorized into two types: multi-label and single-label. Specifically, the probability distribution vector output by the network can be used directly. As a generated tag, it is called a multi-tag type generated tag, and is defined as follows:

[0100]

[0101] Furthermore, for single-label type generated labels, the category with the highest probability needs to be selected from the probability distribution vector output by the model, and the generated label is used in the form of a one-hot vector, which more clearly represents the network's sentiment selection, as defined below:

[0102]

[0103]

[0104] Where j, k∈[1,K] are integers. express The value of the k-th sentiment category, for The value of the j-th sentiment category.

[0105] Step 3: Label Correction

[0106] A label correction strategy is used to combine the original labels of ambiguous samples with the labels generated by the network. The original labels provided in the dataset are multi-label forms obtained by adding multiple one-hot vectors, and their definitions are as follows:

[0107]

[0108] in Similarly, it indicates whether the i-th sample in the sentiment dataset contains the j-th sentiment category, not limited to a single category. This demonstrates the multi-label nature of the dataset. It can represent multiple emotions present in the sample, which is consistent with the phenomenon of mixed emotions in reality. However, this multi-label system still cannot represent the emotional cognition of most people, because its construction still relies entirely on the annotations of a small number of emotion experts, and the multi-label system only represents the presence of certain emotions in speech, but cannot represent the dominant emotional bias in speech.

[0109] Therefore, the network with sentiment recognition is used to modify the labels of ambiguous samples to correct multi-labeling and sentiment recognition that is not entirely dependent on the annotator, enabling the network to clearly define the sentiment bias of samples during training. Due to differences in the degree of sample ambiguity, it is uncertain whether the network's generated labels or the multi-labels provided by the dataset are better overall. Therefore, a balance factor λ is introduced to represent the relative importance of the generated labels and the original, unbiased multi-labels. The process of obtaining corrected labels through the label correction strategy is described below:

[0110]

[0111] in Indicates a correction label. For the original multi-tag, λ represents the generated labels of the network, and λ∈[0,1] represents the balancing factor, which is used to balance the relative importance of the original multi-labels and the model-generated labels.

[0112] Step 4: Retraining

[0113] The spatiotemporal collaborative fusion network M is trained by mixing single-labeled samples with ambiguous samples with corrected labels. f First, the spatiotemporal collaborative fusion network is reinitialized. Second, due to sample S... B With S C All samples exhibited sentiment ambiguity, therefore their labels were corrected, while sample S... A The sentiment label has only one type and is unambiguous, so its original label is retained. Finally, the spatiotemporal collaborative fusion network M, reconstructed using the above three types of data, is trained. f This allows the network to incorporate emotional information from ambiguous samples, thereby improving its ability to recognize speech emotions. The training process still uses the cross-entropy loss function as the objective function, defined as follows:

[0114]

[0115] in For input x i Time Network M f The predicted output vector.

[0116] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A speech emotion recognition method based on multi-label correction and spatio-temporal collaborative fusion, characterized in that, The space-time collaborative fusion network is constructed, which comprises a time domain module, a space domain module, a collaborative fusion module and a classification module. The network is pre-trained and optimized by using single-label speech. The pre-trained space-time collaborative fusion network is used to modify the emotional label of ambiguous speech. The ambiguous speech with corrected label and single-label speech are mixed, and the space-time collaborative fusion network is retrained and optimized. The optimized space-time collaborative fusion network is used for emotional recognition of speech. The process comprises the following steps: S1. According to the spatial characteristics of the speech spectrum and the time sequence characteristics of the speech waveform, the emotional features are extracted from the spatial domain and the time domain of the speech by using the convolutional neural network and the Wav2vec model, respectively. The space-time feature interaction is realized by using the collaborative fusion method to form the space-time collaborative fusion network. S2, initialize the space-time collaborative fusion network, take the voice with a single emotion label as the first training set, use it to pre-train the space-time collaborative fusion network, and call the obtained pre-training network ; S3, inputting the ambiguous voice into the pre-trained spatiotemporal collaborative fusion network , and a generation sentiment label of the input sample is predicted. S4. The generated emotional label of the ambiguous speech is combined with the original multi-label to correct the label, and the ambiguous speech sample with corrected label is obtained. S5, retrain the optimized spatiotemporal collaborative fusion network with the speech with single sentiment label and the ambiguous speech with correction label as the second training set, and call the network ; Wherein, the ambiguous speech refers to the speech sample with multiple emotional labels. The space-time collaborative fusion network comprises:

101. In the time domain module, the Wav2vec model is used to obtain the time domain emotional features from the original waveform of the speech information.

102. In the space domain module, the convolutional neural network is used to process the frequency domain of the speech message to obtain the spatial domain emotional features of the speech message.

103. In the collaborative fusion module, a full connection layer is used to convert the spatial domain emotional features into spatial emotional weights, and the spatial emotional weights are multiplied with the time domain emotional features for fusion to obtain the time emotional features with spatial emotional information.

104. The time emotional features with spatial emotional information are input into the classification module to obtain the emotional classification result.

2. The speech emotion recognition method based on multi-label correction and spatio-temporal collaborative fusion according to claim 1, characterized in that, The sample with a single emotional label is used as the first training set, and the loss function of the pre-training of the space-time collaborative fusion network is represented as: in, This represents the total number of samples in the sentiment dataset. A single emotion tag for speech. Represents a pre-trained spatiotemporal collaborative fusion network The predicted output, Indicates the input network's first... One sample.

3. The speech emotion recognition method based on multi-label correction and spatio-temporal collaborative fusion according to claim 2, characterized in that, Single emotion label for speech is represented as: wherein, represents whether the i-th sample in the emotion dataset exists the j-th emotion category, represents the total number of samples in the emotion dataset, represents the number of classified emotion categories.​​ 4. The speech emotion recognition method based on multi-label correction and spatio-temporal collaborative fusion according to claim 1, characterized in that, The sample with a single emotional label and the ambiguous sample with corrected label are used as the second training set, and the loss function used when the space-time collaborative fusion network is trained is represented as: in, This represents the total number of samples in the sentiment dataset. Indicates a correction label. Represents network The predicted output, Indicates the input network's first... One sample.

5. The speech emotion recognition method based on multi-label correction and spatio-temporal collaborative fusion according to claim 4, characterized in that, The process of obtaining the corrected label comprises: wherein, represents the label obtained after correcting the ambiguous sample, represents the i-th sample input into the network, represents the i-th sample input into the network, represents the pre-trained spatiotemporal collaborative fusion network; is the original label of the ambiguous sample; is the generated label of the pre-trained spatiotemporal collaborative fusion network is the generated label of the pre-trained spatiotemporal collaborative fusion network represents the balancing factor.

6. The speech emotion recognition method based on multi-label correction and spatio-temporal collaborative fusion according to claim 5, characterized in that, Original labels of ambiguous samples The multi-label form obtained by adding the plurality of one-hot vectors is represented as: wherein, represents whether the i-th sample in the emotion dataset exists the j-th emotion category, represents the total number of samples in the emotion dataset, represents the number of categories of emotions.​​ 7. The speech emotion recognition method based on multi-label correction and spatio-temporal collaborative fusion according to claim 5, characterized in that, Generating labels Can be divided into two forms of multi-label and single-label, multi-label form of generating labels For the network The predicted output of the input sample Is expressed as: Generated labels in single-label form Further from the network The class with the highest probability is selected from the output probability distribution vector, and is represented in the form of a one-hot vector: wherein, represents a probability distribution vector of a prediction output of the pre-trained spatio-temporal collaborative fusion network, represents an i-th sample inputted into the network; is an integer, represents a value of an i-th sentiment category of the is a value of an i-th sentiment category of the is a value of an i-th sentiment category of the​​​​​

Images