Local Attribute Learning Method and Device for Speech Emotion Recognition

Abstract

The present application discloses a local attribute learning method and device for speech emotion recognition. The method includes: obtaining general features of unsupervised speech data by using a feature extraction structure; processing the general features by using a local feature adapter to obtain local features adapted to emotional speech data; determining a corresponding attribute learning loss based on the second features; determining a plurality of probability values based on the first features and a plurality of local attributes; determining a corresponding local attribute contrast loss based on the plurality of probability values; and determining a loss function of the target model based on the attribute learning loss and the local attribute contrast loss. Based on the attribute learning loss and the local attribute loss as the loss function of the target model, the method can, in the case of insufficient speech emotion data volume, deeply model the local features extracted by the target model, accurately capture the changes in local statistical information brought about by the complexity of speech emotions, and thus improve the speech emotion recognition performance of the target model.

Description

Technical Field

[0001] The present application relates to the technical field of speech signal processing, and in particular, to a local attribute learning method and device for speech emotion recognition. Background Art

[0002] Speech Emotion Recognition (SER) refers to the use of computer technology to deeply analyze human speech and identify the emotions contained therein. This technology is of great significance in promoting the naturalness and depth of Human-Computer Interaction (HCI). Current speech emotion recognition methods usually follow traditional machine learning frameworks. For example, speech features are extracted through deep neural network architectures such as deep neural networks, convolutional neural networks, or recurrent neural networks, and emotion labels are obtained through classification.

[0003] In speech emotion recognition tasks, traditional analysis methods usually rely on segment-level emotion labels such as "happy" and "sad". However, as a complex psychological phenomenon, emotion expression is often not limited to a single, static emotion label. Recently, some self-supervised pre-training methods have been used to improve the performance of speech tasks by pre-training on a large amount of unlabeled speech data. However, current research simply globally analyzes the features extracted by the self-supervised model, unable to fully utilize the rich local information extracted by the model, and also difficult to effectively represent complex and diverse emotion information. Some existing methods introduce other information to improve the effect, such as text information or speaker information, but this is difficult to generalize. Obviously, although current practices have achieved certain results, due to emotion being a deep and complex state, there are still many problems restricting the speech emotion recognition effect of machine models. Summary of the Invention

[0004] The present application provides a local attribute learning method and device for speech emotion recognition, aiming to improve the performance of speech emotion recognition models.

[0005] To achieve the above objective, the present application provides the following technical solutions:

[0006] A local attribute learning method for speech emotion recognition, comprising:

[0007] Obtaining general features of unsupervised speech data by using a preset feature extraction structure;

[0008] Processing the general features by using a preset local feature adapter to obtain local features adapted to emotion speech data; the local features include first features and second features;

[0009] Based on the second feature, determine the corresponding attribute learning loss; the attribute learning loss is used to generate a local attribute set; the local attribute set includes multiple local attributes;

[0010] Based on the first feature and the multiple local attributes, determine multiple probability values; the probability value represents the probability that the similarity between the first feature and any one of the local attributes meets a preset threshold;

[0011] Based on the multiple probability values, determine the corresponding local attribute contrast loss; the local attribute contrast loss is used to optimize the local feature adapter so as to maximize the log-likelihood function of the first feature;

[0012] Based on the attribute learning loss and the local attribute contrast loss, determine the loss function of the target model; the target model is used to perform speech emotion recognition on the user input speech.

[0013] Optionally, using a preset local feature adapter to process the general feature to obtain a local feature adapted to the emotional speech data, including:

[0014] Input the general feature into the preset local feature adapter to obtain the first feature output by the local feature adapter;

[0015] Perform momentum update on the local feature adapter to obtain the local feature adapter after momentum update;

[0016] Input the general feature into the local feature adapter after momentum update to obtain the second feature output by the local feature adapter after momentum update.

[0017] Optionally, based on the attribute learning loss and the local attribute contrast loss, determining the loss function of the target model includes:

[0018] Based on the probability values corresponding to the first feature and each local attribute, obtain a local attribute mapping;

[0019] Use the softmax function to perform global aggregation on the local attribute mapping to obtain a segment-level attribute feature;

[0020] According to the preset speech emotion recognition task, perform attribute separation on the segment-level attribute feature to obtain a target feature; the target feature contains attributes related to the speech emotion recognition task;

[0021] Based on the target feature, determine the corresponding cross-entropy loss; the cross-entropy loss is used to calculate the classification result of the speech emotion recognition task;

[0022] Determine the loss function of the target model based on the attribute learning loss, the local attribute contrast loss, and the cross-entropy loss.

[0023] Optionally, according to a preset speech emotion recognition task, perform attribute separation on the segment-level attribute features to obtain target features, including:

[0024] Determine the corresponding Fisher information matrix according to a preset speech emotion recognition task;

[0025] Based on the Fisher information matrix, determine the importance index of each global attribute probability in the segment-level attribute features;

[0026] Determine the global attribute probabilities that meet the preset requirements based on the importance index as target features; wherein, the global attribute probabilities are updated in real time based on the training process of the target model.

[0027] Optionally, the local feature adapter includes two adapter blocks and a convolutional layer, and the adapter block includes a self-attention module and a feed-forward neural network.

[0028] Optionally, the feature extraction structure includes a waveform encoder and a BERT encoder. Use the preset feature extraction structure to obtain the general features of unsupervised speech data, including:

[0029] Obtain unsupervised speech data; the unsupervised speech data includes the speech signal input by the user;

[0030] Input the speech signal into the waveform encoder to obtain the speech features output by the waveform encoder;

[0031] Input the speech features into the BERT encoder to obtain the general features output by the BERT encoder.

[0032] Optionally, the waveform encoder includes a seven-layer one-dimensional convolutional neural network, and the BERT encoder includes 17 Transformer blocks.

[0033] A local attribute learning device for speech emotion recognition, including:

[0034] A feature extraction unit, configured to obtain the general features of unsupervised speech data by using a preset feature extraction structure;

[0035] A feature adaptation unit, configured to process the general features by using a preset local feature adapter to obtain local features adapted to emotional speech data; the local features include a first feature and a second feature;

[0036] An attribute learning unit, configured to determine a corresponding attribute learning loss based on the second feature; the attribute learning loss is used to generate a local attribute set; the local attribute set includes multiple local attributes;

[0037] A probability calculation unit, configured to determine multiple probability values based on the first feature and the multiple local attributes; the probability value represents the probability that the similarity between the first feature and any one of the local attributes meets a preset threshold;

[0038] An attribute comparison unit, configured to determine a corresponding local attribute comparison loss based on the multiple probability values; the local attribute comparison loss is used to optimize the local feature adapter so as to maximize the log-likelihood function of the first feature;

[0039] A model optimization unit, configured to determine a loss function of a target model based on the attribute learning loss and the local attribute comparison loss; the target model is used to perform speech emotion recognition on user input speech.

[0040] A storage medium, the storage medium includes a stored program, wherein the program, when run by a processor, executes the local attribute learning method for speech emotion recognition as described above.

[0041] An electronic device, including: a processor, a memory, and a bus; the processor is connected to the memory through the bus;

[0042] The memory is used to store a program, and the processor is used to run the program, wherein the program, when run by the processor, executes the local attribute learning method for speech emotion recognition as described above.

[0043] The technical solution provided by this application uses a preset feature extraction structure to obtain general features of unsupervised speech data. A preset local feature adapter is used to process the general features to obtain local features adapted to emotional speech data. Based on the second feature, a corresponding attribute learning loss is determined. Based on the first feature and multiple local attributes, multiple probability values are determined. Based on the multiple probability values, a corresponding local attribute comparison loss is determined. Based on the attribute learning loss and the local attribute comparison loss, a loss function of the target model is determined. This application uses the attribute learning loss and the local attribute loss as the loss function of the target model, and can, in the case of insufficient speech emotion data volume, capture the local attribute changes brought by special emotions by performing local feature analysis on the general features extracted by a pre-trained model (including the feature extraction structure), obtain local features adapted to the speech emotion task, thereby improving the speech emotion recognition performance of the target model. Description of the Drawings

[0044] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0045] Figure 1 It is a schematic flow chart of a local attribute learning method for speech emotion recognition provided by an embodiment of the present application;

[0046] Figure 2 It is a schematic flow chart of another local attribute learning method for speech emotion recognition provided by an embodiment of the present application;

[0047] Figure 3 It is a schematic flow chart of another local attribute learning method for speech emotion recognition provided by an embodiment of the present application;

[0048] Figure 4 It is a schematic flow chart of another local attribute learning method for speech emotion recognition provided by an embodiment of the present application;

[0049] Figure 5 It is a schematic flow chart of another local attribute learning method for speech emotion recognition provided by an embodiment of the present application;

[0050] Figure 6 It is a schematic architecture diagram of a local attribute learning device for speech emotion recognition provided by an embodiment of the present application;

[0051] Figure 7 It is a schematic diagram of a local attribute learning structure for speech emotion recognition provided by an embodiment of the present application. Detailed implementation manners

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0053] In this application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element qualified by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0054] The applicant has found that the main problems restricting the performance of the speech emotion recognition model include the following points.

[0055] (1) Insufficient data volume problem: Since emotion data must be manually labeled, which is time-consuming and laborious, the lack of sufficient data volume has always been the most important factor restricting speech emotion recognition. To this end, many studies have chosen to use speaker datasets to pre-train feature extractors and achieved positive results. However, since speaker recognition is a segment-level task, this method may cause the feature extractor to overly focus on the differences between different sentences while ignoring the expression of local attributes, thus affecting the accuracy of emotion recognition. At the same time, the proportion of speaker information in the extracted features is too high, which will also affect emotion recognition. In addition, the data volume of the speaker dataset itself is also limited. Self-supervised pre-trained language models represented by the wav2vec2 model use self-supervised learning methods to pre-train with a large amount of unlabeled speech data, bringing significant performance improvements to the speech emotion recognition task. The pre-training objectives of these models are usually designed for speech recognition tasks, and they are effective in modeling the differences in local information. However, they often only use a simple average pooling layer to process the features, ignoring the differences in local features. In addition, the feature information extracted by the self-supervised model is extremely rich and may contain a large amount of information irrelevant to emotion recognition, interfering with emotion recognition.

[0056] (2) Local variability of emotions: In people's actual communication, emotional states are dynamically changing. Using global emotion labels for annotation and training cannot accurately describe the emotional differences in local information in sentences. Since the duration of emotional expression is variable, it is difficult for existing speech emotion recognition models to find an appropriate local feature length to fully express the true emotional state, and the extracted local features and emotion categories cannot match each other.

[0057] (3) Influence of other speech attributes on emotion: The expression of emotion is easily affected by other speech attributes. There are significant differences in the expression of emotion among different speakers, phonemes, and semantic contents.

[0058] For the above discovery by the applicant, the embodiments of the present application disclose a local attribute learning method for speech emotion recognition, aiming to optimize the self-supervised learning features of the speech emotion recognition model and improve the recognition performance of the speech emotion model. Specifically, without using additional labeled speech data, in-depth modeling is performed on the local features extracted by the self-supervised model (i.e., the speech emotion recognition model), local feature analysis is performed on the general features extracted by the pre-trained model (including the feature extraction structure), the local attribute changes brought by special emotions are captured, local features suitable for the speech emotion task are obtained, and the local attributes related to the target task are separated during the emotion classification process, thereby improving the accuracy, real-time performance, and robustness of emotion speech recognition.

[0059] As Figure 1 shown, it is a schematic flowchart of a local attribute learning method for speech emotion recognition provided by the embodiments of the present application, including the following steps.

[0060] S101: Use a preset feature extraction structure to obtain the general features of unsupervised speech data.

[0061] Among them, unsupervised speech data refers to speech data without labeled tags. Unsupervised speech data can be used to train a self-supervised model. Generally speaking, unsupervised speech data can usually be the speech signal input by the user or unlabeled audio data.

[0062] Optionally, the feature extraction structure includes a waveform encoder and a BERT encoder. The waveform encoder includes a seven-layer one-dimensional convolutional neural network, and the BERT encoder includes 17 Transformer blocks.

[0063] In some examples, the waveform encoder can be used to convert the input speech signal into a speech feature sequence (including multiple speech features). The waveform encoder includes a seven-layer one-dimensional convolutional neural network. The step sizes of the seven-layer one-dimensional convolutional neural network are respectively (5, 2, 2, 2, 2, 2, 2). The convolutional kernel sizes of each layer in the seven-layer one-dimensional convolutional neural network are respectively (10, 3, 3, 3, 3, 2, 2). The number of channels in each layer of the seven-layer one-dimensional convolutional neural network is 512. The speech signal can be a 16kHz waveform signal, and the frame length of the output speech features is 25ms and the frame shift is 20ms.

[0064] In some examples, the 17 Transformer blocks included in the BERT encoder all use 16-head attention, and the feature dimension is 1024. The BERT encoder is used to process speech features and extract context information.

[0065] Optionally, for the implementation process of obtaining the general features of unsupervised speech data using a preset feature extraction structure, reference can be made to Figure 2 the steps shown and the corresponding explanatory notes.

[0066] S102: Use a preset local feature adapter to process the general features to obtain local features adapted to the emotional speech data.

[0067] Among them, the local features include the first feature and the second feature.

[0068] Optionally, the local feature adapter includes two adapter blocks and a convolutional layer. The adapter block includes a self-attention module and a feed-forward neural network.

[0069] In some examples, the local feature adapter consists of two adapter blocks and a 1*1 convolutional layer. Among them, each adapter block contains a 16-head self-attention module and a feed-forward neural network layer. The 1*1 convolutional layer is used to reduce the dimension of the speech features to avoid computational difficulties caused by too high dimensions.

[0070] Optionally, for the implementation process of obtaining the local features of the general features using a preset local feature adapter, reference can be made to Figure 3 the steps shown and the corresponding explanatory notes.

[0071] S103: Based on the second feature, determine the corresponding attribute learning loss.

[0072] Among them, the attribute learning loss is used to generate a local attribute set, and the local attribute set includes multiple local attributes.

[0073] It should be noted that in order to model the local features without using labels, a local attribute set P = [p1, p2,..., p K can be constructed, where p j ∈R C , and p j represents a certain attribute of the local information, such as phoneme, timbre, prosody, etc. This attribute is used to help analyze the local features.

[0074] In some examples, the expression of the attribute learning loss L Q can be seen in formula (1)

[0075] L Q = ||sg(x' i ) - pj ||2 (1)

[0076] In formula (1), L Q Represents the loss function of online attribute learning (i.e., attribute learning loss), x' i Represents the local feature adapter θ m The local part of the second feature of the output, that is, X'=[x' i ], p j Represents the distance from the second feature x' in the local attribute set i The nearest attribute, sg represents the stop gradient operator. In addition, the attribute learning loss is only used to update the local attribute set and is not passed to the local feature adapter θ. The local feature adapter θ m The local feature adapter θ is obtained by momentum update, ensuring that in the feature adaptation process, the local feature adapter θ m The second feature x' of the output i The change range is small, and the corresponding local properties will not change much.

[0077] S104: Determine multiple probability values based on the first feature and multiple local attributes.

[0078] The probability value represents the probability that the similarity between the first feature and any local attribute meets a preset threshold.

[0079] It should be noted that the similarity between the first feature and any local attribute can be determined by calculating the cosine similarity between the first feature and any local attribute. In addition, the probability that the similarity between the first feature and any local attribute meets the preset threshold can be understood as the normalized indicator function (Softmax) value of the cosine similarity in the local attribute set.

[0080] In some examples, the first feature x i With any local attribute p j The expression of cosine similarity between can be recorded as cosine(x i , p j ), where x i By using the attribute contrast learning method, the local feature adapter is optimized to obtain the appropriate local features. To this end, this embodiment needs to find the local feature adapter parameter θ that maximizes the log-likelihood function of the first feature as shown in formula (2).

[0081]

[0082] In formula (2), B represents the number of samples in the batch, each sample contains T local features, q θ (x i ) is the local feature adapter θ extracted by xi The likelihood function of the distribution. Next, introduce the local attribute p j , then there is formula (3).

[0083]

[0084] For formula (4), according to Jensen's inequality, formula (4) can be obtained.

[0085]

[0086] Thus, logq θ (x i )'s lower bound can be obtained, as shown in formula (5).

[0087]

[0088] In formula (5), the latter part is the distribution of the local attribute, which has nothing to do with the local feature adapter θ. Therefore, the optimization objective can be written as shown in formula (6).

[0089]

[0090] In formula (6), assume that each x i is associated with a local attribute p j , then formula (6) can be rewritten as formula (7).

[0091]

[0092] In formula (7), p(p j |x i ) is the probability value of the local attribute corresponding to x i . When calculating, set the local attribute closest to x i as p(p j |x i ) = 1, and set the remaining local attributes to 0. q θ (x i |p j ) is the distribution that this embodiment hopes the first feature to follow.

[0093] In the ideal case, the first feature follows a Gaussian distribution around the corresponding attribute. Assume that x i follows a Gaussian distribution with a variance of 0.5 around p j , then formula (8) can be obtained.

[0094]

[0095] Based on formula (8), maximizing the log-likelihood function can be simplified to formulas (9), (10), and (11).

[0096]

[0097] In formulas (9), (10), and (11), α i,j represents a probability value, that is, the Softmax value of the similarity in the local attribute set, and τ represents a temperature coefficient (set to 1).

[0098] S105: Determine the corresponding local attribute contrast loss based on multiple probability values.

[0099] Among them, the local attribute contrast loss is used to optimize the local feature adapter to maximize the log-likelihood function of the first feature.

[0100] In some examples, the local attribute contrast loss L p determined based on multiple probability values can be seen in formula (11).

[0101] It should be noted that while using the local attribute contrast loss to optimize the local feature adapter θ, the exponential moving average (EMA) method can also be used to update the local feature adapter θ m That is, the local feature adapter θ and the local feature adapter θ m can be learned and updated based on the same sample, and this update process can be seen in formula (12).

[0102] θ m ←γθ+(1 - γ)θ m (12)

[0103] In formula (12), γ represents an update coefficient (which can be set to 0.001).

[0104] S106: Determine the loss function of the target model based on the attribute learning loss and the local attribute contrast loss.

[0105] Among them, the target model is used to perform speech emotion recognition on the speech input by the user.

[0106] It should be noted that the target model can specifically be a self-supervised model composed of the local feature adapter θ and the local feature adapter θ m In addition, the emotion classification loss adopted by the target model for the classification and attribute separation parts is not related to the attribute learning loss L Q and the local attribute contrast loss L p

[0107] In some examples, the speech emotion adaptation part of the target model can use the attribute learning loss L Q ​and the local attribute contrast loss L p , the loss function L_attribute adopted by the target model can be denoted as shown in formula (13).

[0108] L_attribute = L p + λL Q (13)

[0109] In formula (13), λ represents the relative weight of the loss function (which can be set to 1).

[0110] It should be noted that, based on the attribute learning loss and the local attribute contrast loss, determining the loss function of the target model can perform emotional speech adaptation on the general local speech features extracted by the target model, effectively alleviating the problem of insufficient emotional data volume, and designing local attribute mapping to depict the complexity of emotions. At the same time, the exponential softmax mechanism is adopted to achieve an effective mapping from local attributes to global attributes.

[0111] In some examples, for a specific speech emotion recognition task, during the fine-tuning process of the target model, a task-oriented online attribute separation strategy can also be designed to select relevant local attributes for mapping, so as to reduce the impact of irrelevant information on the emotion recognition performance and improve the recognition efficiency and accuracy.

[0112] Optionally, the implementation process of selecting relevant local attributes for mapping can refer to Figure 4 the steps shown and the corresponding explanatory notes.

[0113] To make full use of the local attribute learning method to improve the performance of speech emotion recognition and to verify the effectiveness of the method proposed in the present invention, the following experiments were designed: Experiments were conducted on the IEMOCAP dataset and the Multimodal Emotion Recognition Challenge MER2023 dataset to evaluate the effectiveness of the proposed method. The IEMOCAP dataset contains approximately 12 hours of English audio-visual data. The database is divided into 5 parts, each part containing one male and one female actor. According to the recording scenario, it can be further subdivided into impromptu dialogue or scripted dialogue parts. Each utterance is annotated by multiple annotators with 8 emotion labels. To make a fair comparison with previous work, 4 emotion types (i.e., neutral, happy, angry, and sad) were selected in the experiment. The experiment adopted a 5-fold cross-validation strategy to evaluate this dataset. For each fold, the utterances of 4 parts were used for training, and the remaining parts were used for validation and testing. The MER2023 dataset provides 3373 Chinese speech data segments with a total duration of nearly 4 hours. Each audio is annotated with six categories (neutral, angry, happy, sad, worried, and surprised), and is attached with a one-dimensional value Valence to evaluate the positive and negative degree of the emotion. Following the pre-set benchmark, the dataset was randomly divided into five parts and 5-fold cross-validation was performed to evaluate the performance. All systems were implemented using the PyTorch deep learning framework. The experiment adopted the Adam optimizer and set the mini-batch size to 128. For the IEMOCAP dataset, the system was trained for 50 epochs with an initial learning rate of 0.0001. For the MER2023 dataset, the system was trained for 100 epochs with the same initial learning rate of 0.0001.

[0114] In addition, the evaluation metrics were pre-set for the experiment as follows: Weighted Accuracy (WA), Unweighted Accuracy (UA), F1-Score, Mean-Square Error (MSE). For the MER2023 dataset, the experiment used F1-Score to evaluate the six-class emotion recognition, and used MSE to evaluate the one-dimensional value Valence.

[0115] Specifically, WA considers the accuracy calculation method for each class sample quantity, and the calculation process of WA is shown in formula (14).

[0116]

[0117] In formula (14), N i represents the sample quantity of class i, A iAccuracy represents the accuracy of class i, and N represents the total number of samples. Formula (14) makes the contribution of the error of each class to the overall accuracy correspond to its proportion in the dataset.

[0118] Specifically, UA is the average precision calculated for each emotion category respectively, that is, the number of correctly predicted instances in a category divided by the total number of instances in that category, and then the accuracies of all categories are averaged. The calculation process of UA is shown in formula (15).

[0119]

[0120] In formula (15), A i represents the accuracy of class i, and N classes represents the number of classes.

[0121] Specifically, F1-Score is the harmonic mean of accuracy and recall. The calculation process of F1-Score is shown in formula (16).

[0122]

[0123] In formula (16), precision represents precision rate, and recall represents recall rate.

[0124] The experimental result table 1 shows the influence of different numbers of prototypes on the speech emotion recognition effect without using the attribute separation method shown in the present invention. In table 1, "average pooling" and "max pooling" mean that without using the attribute learning method, the average pooling or max pooling method is directly used to process local features after the BERT encoder.

[0125] Table 1

[0126]

[0127]

[0128] Based on the experimental results shown in table 1, on the IEMOCAP and MER2023 datasets, the pre-trained local attribute learning method we proposed significantly improves the performance of emotion recognition. On the IEMOCAP dataset, the WA metric is improved by about 3.31%, the UA is improved by about 8.04%, and the F1-score is improved by about 6.56%. On the MER2023 dataset, the F1-score is improved by about 4.39%.

[0129] The experimental result table 2 shows the influence of different separation ratios on the emotion recognition effect based on the use of 1024 local attributes for attribute separation.

[0130] Table 2

[0131]

[0132] The experimental results shown in Table 2 indicate that selecting a small number of attributes is significantly helpful for emotion recognition. The best emotion recognition effect is achieved when only 6%-10% of the local attributes are selected.

[0133] Regarding the performance comparison between the existing self-supervised model and the target model shown in the present invention, Table 3 is publicly available in the experiment.

[0134] Table 3

[0135]

[0136] Based on the experimental results shown in Table 3, it is indicated that the local attribute learning method shown in the present invention has made relatively obvious progress in performance compared with the existing technology.

[0137] Generally speaking, the present invention proposes a local attribute learning method based on pre-training for speech emotion recognition. The aim is to model the local features of speech signals through an unsupervised attribute contrast learning method. The network conducts contrast training on the local features by online learning an attribute set, thereby modeling the frame-level local features. In the fine-tuning stage, for different downstream tasks, the speech attributes related to the tasks are screened out for attribute separation. Experiments on the IEMOCAP and MER2023 datasets prove that this method significantly improves the effect of speech emotion recognition.

[0138] In some examples, for the role of the feature extraction structure and the local feature adapter shown in the embodiments of the present application during the pre-training process, reference can be made to Figure 7 as shown.

[0139] For the processes shown in S101-S106 above, by using the local feature adapter to obtain the corresponding attribute learning loss and local attribute loss, and based on the attribute learning loss and the local attribute loss as the loss function of the target model, it is possible to deeply model the local features extracted by the target model in the case of insufficient speech emotion data volume, accurately capture the changes in local statistical information brought about by the complexity of speech emotions, and thus improve the speech emotion recognition performance of the target model.

[0140] As Figure 2 shown, it is a schematic flow diagram of a local attribute learning method for speech emotion recognition provided by the embodiments of the present application, including the following steps.

[0141] S201: Obtain unsupervised speech data.

[0142] Among them, the unsupervised speech data includes the speech signals input by users.

[0143] S202: Input the speech signal into a waveform encoder to obtain the speech features output by the waveform encoder.

[0144] Among them, the waveform encoder can be pre-trained using a large amount of unlabeled audio data (such as Libri-Light 60k).

[0145] S203: Input the speech features into a BERT encoder to obtain the general features output by the BERT encoder.

[0146] Among them, the BERT encoder can be pre-trained using a large amount of unlabeled audio data (such as Libri-Light 60k).

[0147] The process shown in the above S201 - S203 can use the waveform encoder and the BERT encoder to obtain the corresponding general features from unsupervised speech data.

[0148] As Figure 3 shown, it is a schematic flowchart of a local attribute learning method for speech emotion recognition provided by an embodiment of the present application, including the following steps.

[0149] S301: Input the general features into a preset local feature adapter to obtain the first features output by the local feature adapter.

[0150] Among them, the local feature adapter can adjust the dimension of the general features to X = [x1, x2,..., x T , x i ∈ R C , that is to say, the first features can be denoted as X = [x1, x2,..., x T .

[0151] In some examples, the local feature adapter can be denoted as θ.

[0152] S302: Perform momentum update on the local feature adapter to obtain the local feature adapter after momentum update.

[0153] Among them, momentum update is a technique for accelerating gradient descent. By calculating the exponentially weighted average of past gradients to calculate the current update direction, momentum update can accelerate the convergence of the local feature adapter. Especially for flat regions with small gradients, momentum update can suppress high-frequency oscillations and make the gradient descent more stable.

[0154] In some examples, the local feature adapter after momentum update can be denoted as θ m . It can be understood that the network structure of the local feature adapter θ is the same as that of the local feature adapter θ m .

[0155] S303: Input the general features into the locally feature adapter after momentum update to obtain the second features output by the locally feature adapter after momentum update.

[0156] Among them, the locally feature adapter after momentum update can adjust the dimension of the general features to X’ = [x’1, x’2, …, x’ T , x’ i ∈R C , that is to say, the first features can be denoted as X’ = [x’1, x’2, …, x’ T .

[0157] The processes shown in the above S301 - S303 use the locally feature adapter and the locally feature adapter after momentum update to obtain the first features and the second features of the general features.

[0158] As Figure 4 shown, it is a schematic flowchart of a local attribute learning method for speech emotion recognition provided by an embodiment of this application, including the following steps.

[0159] S401: Obtain the local attribute mapping based on the probability values between the first features and each local attribute.

[0160] Among them, the probability value α i,j between the first features and each local attribute can be regarded as the probability that the first features x i belong to the local attribute p j . Based on all the probability values α i,j , it can be used as the mapping of the first features in the feature space constituted by the local attribute set, that is, the so-called local attribute mapping d i = [α i,1 , α i,2 , …, α[[ID=4,2]] i,k .

[0161] S402: Use the normalized exponential function to perform global aggregation on the local attribute mapping to obtain the segment-level attribute features.

[0162] Among them, the normalized exponential function is the exponential Softmax function. The exponential Softmax function uses exp(α i,j ) as the weight and assigns it to the global attribute probability y[[ID=5,2]] j . This assignment process can be seen in formula (17) shown.

[0163]

[0164] When using formula (17) to perform global aggregation on the local attribute mapping, calculate y j with respect to αi,j The gradient, and the calculation formula of this gradient can be seen in Formula (18).

[0165]

[0166] In Formula (18), since y j must be less than 1, so this gradient is always positive, that is, the final loss function will perform directional adjustment on all frames in the entire speech segment, and the adjustment amplitude will vary according to their respective attribute situations.

[0167] It should be noted that in people's actual communication, the emotional state is dynamically changing. For example, when a sentence is overall judged as "happy", the sentence will not always express "happy", but there will be some expressions of neutral or other emotions. Traditional methods often use average pooling for aggregation, which will cause the network to judge each part of the sentence as the target category. The speech emotion recognition task is essentially a detection task. For various basic attributes in it, assuming that at least one local feature in the segment-level speech contains the information of this attribute, then the entire sentence of speech should be judged as containing this attribute. Using the max pooling method can achieve this purpose, that is, taking the largest α i,j as y j , however, the max pooling method has a defect, that is, only one frame in the entire speech segment can pass the signal to the subsequent processing. If an event occurs multiple times within the speech segment, these frames are easily missed. Therefore, the exponential Softmax function is selected to alleviate this problem.

[0168] In some examples, the expression of the segment-level attribute feature can be Y = [y1, y2,..., y k .

[0169] S403: According to the preset speech emotion recognition task, perform attribute separation on the segment-level attribute feature to obtain the target feature.

[0170] Among them, the target feature contains the attributes related to the speech emotion recognition task.

[0171] Optionally, the implementation process of performing attribute separation on the segment-level attribute feature according to the preset speech emotion recognition task to obtain the target feature can be seen in Figure 5 the steps shown and the corresponding explanatory notes.

[0172] S404: Based on the target feature, determine the corresponding cross-entropy loss.

[0173] Among them, the cross-entropy loss is used to calculate the classification result of the speech emotion recognition task.

[0174] In some examples, the segment-level attribute features (i.e., target features) obtained after decomposing the attributes are input into a classifier to obtain a predicted classification result, and the cross-entropy loss L between the prediction and the true value is calculated. CE .

[0175] In a possible implementation, the sentiment classifier is composed of fully connected layers, with the input being the target features and the output being the classification result of the speech sentiment recognition task. Different speech sentiment recognition tasks use classifiers of different sizes.

[0176] S405: Determine the loss function of the target model based on the attribute learning loss, the local attribute contrast loss, and the cross-entropy loss.

[0177] Among them, based on the attribute learning loss, the local attribute contrast loss, and the cross-entropy loss, the loss function of the target model is determined. The loss function Loss of the target model can be seen as shown in formula (19).

[0178] Loss = L_attribute + λ CE ·L CE (19)

[0179] In formula (19), λ CE represents the fine-tuning part weight (which can be set to 1).

[0180] In some examples, the cross-entropy loss L CE and the training of the loss function L_attribute can be carried out separately.

[0181] The processes shown in S401 - S405 above can use local attribute mapping to obtain target features related to the speech sentiment recognition task, determine the corresponding cross-entropy loss using the target features, incorporate the cross-entropy loss into the loss function of the target model, accurately capture the changes in local statistical information brought by the complexity of speech sentiment, extract attribute features closely related to the sentiment recognition task, and map them to the corresponding global representations, thereby improving the accuracy, real-time performance, and robustness of sentiment speech recognition.

[0182] As Figure 5 shown, it is a schematic flowchart of a local attribute learning method for speech sentiment recognition provided by an embodiment of the present application, including the following steps.

[0183] S501: Determine the corresponding Fisher information matrix according to the preset speech sentiment recognition task.

[0184] Among them, the Fisher information matrix is the generalization of the Fisher information quantity from a single parameter to multiple parameters. The Fisher information quantity represents the average information quantity that a sample of a random variable can provide about the state parameter in a certain sense. The Fisher information quantity can be understood as a measure of the expected value of the information quantity that a single observation value can provide about the position parameter.

[0185] S502: Based on the Fisher information matrix, determine the importance index of each global attribute probability in the segment-level attribute features.

[0186] Among them, since the local attribute set is obtained through unsupervised learning based on local information, not all local attributes are useful for the speech emotion recognition task. Therefore, attribute decomposition is performed on the segment-level attribute features Y. Specifically, during the specific training for the speech emotion recognition task, for the Fisher information matrix, the more important the attribute (i.e., the global attribute probability) is for the speech emotion recognition task, the greater its Fisher information quantity. Calculate the importance index of all attributes for the speech emotion recognition task, and the weights of some attributes with the highest importance index can be set to 1, and the weights of the remaining attributes can be set to 0.

[0187] S503: Based on the global attribute probabilities whose importance indexes meet the preset requirements, determine them as target features.

[0188] Among them, the global attribute probabilities are updated in real time based on the training process of the target model. In addition, the preset requirements can be: the importance index is the highest, or the importance index is greater than the specified threshold.

[0189] It should be noted that for different speech emotion recognition tasks, the number of attributes required varies greatly, so different attribute weight settings can be adopted.

[0190] The above process shown in S501 - S503 can utilize the Fisher information matrix to implement attribute decomposition of segment-level attribute features to obtain target features related to the speech emotion recognition task.

[0191] As Figure 6 shown, it is a schematic architecture diagram of a local attribute learning device for speech emotion recognition provided by an embodiment of the present application, including the following units.

[0192] Feature extraction unit 100, configured to obtain general features of unsupervised speech data by using a preset feature extraction structure.

[0193] Optionally, the feature extraction structure includes a waveform encoder and a BERT encoder. The feature extraction unit 100 is specifically configured to: obtain unsupervised speech data; the unsupervised speech data includes a speech signal input by a user; input the speech signal into the waveform encoder to obtain speech features output by the waveform encoder; input the speech features into the BERT encoder to obtain general features output by the BERT encoder.

[0194] Optionally, the waveform encoder includes a seven-layer one-dimensional convolutional neural network, and the BERT encoder includes 17 Transformer blocks.

[0195] The feature adaptation unit 200 is configured to process the general features by using a preset local feature adapter to obtain local features adapted to the emotional speech data; the local features include a first feature and a second feature.

[0196] Optionally, the feature adaptation unit 200 is specifically configured to: input the general features into the preset local feature adapter to obtain the first feature output by the local feature adapter; perform momentum update on the local feature adapter to obtain the local feature adapter after momentum update; input the general features into the local feature adapter after momentum update to obtain the second feature output by the local feature adapter after momentum update.

[0197] Optionally, the local feature adapter includes two adapter blocks and a convolutional layer, and the adapter block includes a self-attention module and a feed-forward neural network.

[0198] The attribute learning unit 300 is configured to determine a corresponding attribute learning loss based on the second feature; the attribute learning loss is used to generate a local attribute set; the local attribute set includes multiple local attributes.

[0199] The probability calculation unit 400 is configured to determine multiple probability values based on the first feature and multiple local attributes; the probability value represents the probability that the similarity between the first feature and any local attribute meets a preset threshold.

[0200] The attribute comparison unit 500 is configured to determine a corresponding local attribute comparison loss based on the multiple probability values; the local attribute comparison loss is used to optimize the local feature adapter to maximize the log-likelihood function of the first feature.

[0201] The model optimization unit 600 is configured to determine a loss function of the target model based on the attribute learning loss and the local attribute comparison loss; the target model is used to perform speech emotion recognition on the speech input by the user.

[0202] Optionally, the model optimization unit 600 is specifically configured to: obtain a local attribute mapping based on the probabilities corresponding to the first feature and each local attribute; use the softmax function to perform global aggregation on the local attribute mapping to obtain segment-level attribute features; perform attribute separation on the segment-level attribute features according to a preset speech emotion recognition task to obtain target features; the target features include attributes related to the speech emotion recognition task; determine a corresponding cross-entropy loss based on the target features; the cross-entropy loss is used to calculate the classification result of the speech emotion recognition task; determine the loss function of the target model based on the attribute learning loss, the local attribute contrast loss, and the cross-entropy loss.

[0203] Optionally, the model optimization unit 600 is specifically configured to: determine a corresponding Fisher information matrix according to a preset speech emotion recognition task; determine an importance index of each global attribute probability in the segment-level attribute features based on the Fisher information matrix; determine the global attribute probabilities that meet the preset requirements as target features; wherein, the global attribute probabilities are updated in real time based on the training process of the target model.

[0204] Each of the above-described units uses the local feature adapter to obtain the corresponding attribute learning loss and the local attribute loss, and uses the attribute learning loss and the local attribute loss as the loss function of the target model. It can, in the case of insufficient speech emotion data volume, deeply model the local features extracted by the target model, accurately capture the changes in local statistical information brought about by the complexity of speech emotions, and thus improve the speech emotion recognition performance of the target model.

[0205] The present application also provides a computer-readable storage medium. The computer-readable storage medium includes a stored program, wherein the program executes the local attribute learning method for speech emotion recognition provided by the present application.

[0206] The present application also provides an electronic device, including: a processor, a memory, and a bus. The processor is connected to the memory through the bus. The memory is used to store a program, and the processor is used to run the program. When the program runs, it executes the local attribute learning method for speech emotion recognition provided by the present application.

[0207] In addition, the functions described above in the embodiments of the present application can be at least partially executed by one or more hardware logic components. For example, without limitation, the exemplary types of hardware logic components that can be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0208] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present application. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0209] The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present application.

Claims

1. A local attribute learning method for speech emotion recognition, characterized in that Including: Obtain the general features of unsupervised speech data by using a preset feature extraction structure; Process the general features by using a preset local feature adapter to obtain local features adapted to emotional speech data; the local features include a first feature and a second feature; wherein, the local feature adapter includes two adapter blocks and a convolutional layer, and the adapter block includes a self-attention module and a feed-forward neural network; the first feature is the local feature generated by using the original local feature adapter; the second feature is the local feature generated by using the local feature adapter after momentum update; Based on the second feature, determine the corresponding attribute learning loss; the attribute learning loss is used to generate a local attribute set; the local attribute set includes multiple local attributes; Based on the first feature and the multiple local attributes, determine multiple probability values; the probability value represents the probability that the similarity between the first feature and any one of the local attributes meets a preset threshold; Based on the multiple probability values, determine the corresponding local attribute contrast loss; the local attribute contrast loss is used to optimize the local feature adapter so that the log-likelihood function of the first feature is maximized; Based on the attribute learning loss and the local attribute contrast loss, determine the loss function of the target model; the target model is used to perform speech emotion recognition on the speech input by the user.

2. The method according to claim 1, characterized in that, Using a preset local feature adapter to process the general features to obtain local features adapted to emotional speech data, including: Input the general features into a preset local feature adapter to obtain the first feature output by the local feature adapter; Perform momentum update on the local feature adapter to obtain the local feature adapter after momentum update; Input the general features into the local feature adapter after momentum update to obtain the second feature output by the local feature adapter after momentum update.

3. The method according to claim 1, characterized in that, Based on the attribute learning loss and the local attribute contrast loss, determine the loss function of the target model, including: Based on the probability values corresponding to the first feature and each local attribute, obtain a local attribute mapping; Use the softmax function to globally aggregate the local attribute mapping to obtain segment-level attribute features; According to a preset speech emotion recognition task, perform attribute separation on the segment-level attribute features to obtain target features; the target features contain attributes related to the speech emotion recognition task; Based on the target features, determine the corresponding cross-entropy loss; the cross-entropy loss is used to calculate the classification result of the speech emotion recognition task; Based on the attribute learning loss, the local attribute contrast loss, and the cross-entropy loss, determine the loss function of the target model.

4. The method according to claim 3, characterized in that, According to a preset speech emotion recognition task, perform attribute separation on the segment-level attribute features to obtain target features, including: According to a preset speech emotion recognition task, determine the corresponding Fisher information matrix; Based on the Fisher information matrix, determine the importance index of the probability of each global attribute in the segment-level attribute features; Determine the global attribute probability that meets the preset requirements based on the importance index as the target feature; wherein, the global attribute probability is updated in real time based on the training process of the target model.

5. The method according to claim 1, characterized in that, The local feature adapter includes two adapter blocks and a convolutional layer, and each adapter block includes a self-attention module and a feed-forward neural network.

6. The method according to claim 1, wherein The feature extraction structure includes a waveform encoder and a BERT encoder. By using the preset feature extraction structure, general features of unsupervised speech data are obtained, including: Obtain unsupervised speech data; the unsupervised speech data includes a speech signal input by a user. Input the speech signal into the waveform encoder to obtain the speech features output by the waveform encoder. Input the speech features into the BERT encoder to obtain the general features output by the BERT encoder.

7. The method according to claim 6, characterized in that The waveform encoder includes a seven-layer one-dimensional convolutional neural network, and the BERT encoder includes 17 Transformer blocks.

8. A local attribute learning device for speech emotion recognition, characterized in that, Including: A feature extraction unit for obtaining general features of unsupervised speech data by using a preset feature extraction structure. A feature adaptation unit for processing the general features by using a preset local feature adapter to obtain local features adapted to emotional speech data; the local features include a first feature and a second feature; wherein, the local feature adapter includes two adapter blocks and a convolutional layer, and each adapter block includes a self-attention module and a feed-forward neural network; the first feature is the local feature generated by using the original local feature adapter; the second feature is the local feature generated by using the local feature adapter after momentum update. An attribute learning unit for determining a corresponding attribute learning loss based on the second feature; the attribute learning loss is used to generate a local attribute set; the local attribute set includes multiple local attributes. A probability calculation unit for determining multiple probability values based on the first feature and the multiple local attributes; the probability value represents the probability that the similarity between the first feature and any one of the local attributes meets a preset threshold. An attribute comparison unit for determining a corresponding local attribute comparison loss based on the multiple probability values; the local attribute comparison loss is used to optimize the local feature adapter so as to maximize the log-likelihood function of the first feature. A model optimization unit for determining a loss function of the target model based on the attribute learning loss and the local attribute comparison loss; the target model is used for performing speech emotion recognition on a speech input by a user.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein the program, when run by a processor, executes the local attribute learning method for speech emotion recognition according to any one of claims 1-7.

10. An electronic device, characterized in that, Including: A processor, a memory and a bus; The processor is connected to the memory through the bus; The memory is used for storing a program, and the processor is used for running the program, wherein the program, when run by the processor, executes the local attribute learning method for speech emotion recognition according to any one of claims 1-7.

Images