Local attribute learning method and device for speech emotion recognition

Abstract

The invention discloses a local attribute learning method and device for speech emotion recognition, and the method comprises the steps: obtaining the universal features of unsupervised speech data through a feature extraction structure; processing the general features by using a local feature adapter to obtain local features adaptive to the emotional voice data; based on the second feature, determining a corresponding attribute learning loss; determining a plurality of probability values based on the first feature and the plurality of local attributes; determining corresponding local attribute comparison loss based on the plurality of probability values; and determining a loss function of the target model based on the attribute learning loss and the local attribute comparison loss. According to the method, based on attribute learning loss and local attribute loss as loss functions of a target model, in-depth modeling can be carried out on local features extracted from the target model under the condition of insufficient voice emotion data volume, local statistical information changes caused by voice emotion complexity are accurately captured, and the accuracy of voice emotion data is improved. Therefore, the speech emotion recognition performance of the target model is improved.

Description

Technical Field

[0001] The present application relates to the field of speech signal processing technology, and in particular to a local attribute learning method and device for speech emotion recognition. Background Art

[0002] Speech Emotion Recognition (SER) refers to the use of computer technology to deeply analyze human speech and identify the emotions contained in it. This technology is of great significance in promoting the naturalness and depth of human-computer interaction (HCI). Current language emotion recognition methods usually follow the traditional machine learning framework, such as extracting speech features through deep neural network architectures such as deep neural networks, convolutional neural networks, or recurrent neural networks, and classifying them to obtain emotion labels.

[0003] In speech emotion recognition tasks, traditional analysis methods usually rely on segment-level emotion labels such as "happy" and "sad". However, emotion, as a complex psychological phenomenon, is often not limited to a single, static emotion label. Recently, some methods based on self-supervised pre-training have improved the performance of speech tasks by pre-training on a large amount of unlabeled speech data. However, current research only conducts a simple global analysis of the features extracted by the self-supervised model, which cannot fully utilize the rich local information extracted by the model, and it is also difficult to effectively represent complex and diverse emotional information. Some existing methods will introduce other information to improve the effect, such as text information or speaker information, but this is difficult to generalize. Obviously, although the current approach has achieved certain results, since emotion is a deep and complex state, there are still many problems that limit the speech emotion recognition effect of machine models. Summary of the invention

[0004] The present application provides a local attribute learning method and device for speech emotion recognition, aiming to improve the performance of the speech emotion recognition model.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] A local attribute learning method for speech emotion recognition, comprising:

[0007] Using the preset feature extraction structure, the common features of unsupervised speech data are obtained;

[0008] Using a preset local feature adapter, the general feature is processed to obtain a local feature adapted to the emotional speech data; the local feature includes a first feature and a second feature;

[0009] Based on the second feature, determining a corresponding attribute learning loss; the attribute learning loss is used to generate a local attribute set; the local attribute set includes a plurality of local attributes;

[0010] Based on the first feature and the plurality of local attributes, a plurality of probability values ​​are determined; the probability values ​​represent the probability that the similarity between the first feature and any of the local attributes meets a preset threshold;

[0011] Based on the plurality of probability values, a corresponding local attribute contrast loss is determined; the local attribute contrast loss is used to optimize the local feature adapter so as to maximize the log-likelihood function of the first feature;

[0012] Based on the attribute learning loss and the local attribute contrast loss, a loss function of a target model is determined; the target model is used to perform speech emotion recognition on speech input by a user.

[0013] Optionally, the general features are processed using a preset local feature adapter to obtain local features adapted to the emotional speech data, including:

[0014] Inputting the general feature into a preset local feature adapter to obtain a first feature output by the local feature adapter;

[0015] Performing momentum update on the local feature adapter to obtain a momentum-updated local feature adapter;

[0016] The general feature is input into the momentum-updated local feature adapter to obtain a second feature output by the momentum-updated local feature adapter.

[0017] Optionally, determining a loss function of a target model based on the attribute learning loss and the local attribute contrast loss includes:

[0018] Obtaining a local attribute mapping based on a probability value corresponding to the first feature and each of the local attributes;

[0019] Using a normalized exponential function, globally aggregating the local attribute maps to obtain segment-level attribute features;

[0020] According to a preset speech emotion recognition task, the segment-level attribute features are attribute-separated to obtain target features; the target features include attributes related to the speech emotion recognition task;

[0021] Based on the target feature, determining a corresponding cross entropy loss; the cross entropy loss is used to calculate the classification result of the speech emotion recognition task;

[0022] A loss function of a target model is determined based on the attribute learning loss, the local attribute contrast loss, and the cross entropy loss.

[0023] Optionally, according to a preset speech emotion recognition task, the segment-level attribute features are attribute-separated to obtain target features, including:

[0024] According to the preset speech emotion recognition task, determine the corresponding Fisher information matrix;

[0025] Based on the Fisher information matrix, determining the importance index of each global attribute probability in the segment-level attribute features;

[0026] A global attribute probability that meets preset requirements based on the importance index is determined as a target feature; wherein the global attribute probability is updated in real time based on the training process of the target model.

[0027] Optionally, the local feature adapter includes two adapter blocks and a convolutional layer, and the adapter block includes a self-attention module and a feedforward neural network.

[0028] Optionally, the feature extraction structure includes a waveform encoder and a BERT encoder, and a preset feature extraction structure is used to obtain common features of unsupervised speech data, including:

[0029] Obtaining unsupervised speech data; the unsupervised speech data includes a speech signal input by a user;

[0030] Inputting the speech signal into the waveform encoder to obtain speech features output by the waveform encoder;

[0031] The speech features are input into the BERT encoder to obtain the general features output by the BERT encoder.

[0032] Optionally, the waveform encoder includes a seven-layer one-dimensional convolutional neural network, and the BERT encoder includes 17 Transformer blocks.

[0033] A local attribute learning device for speech emotion recognition, comprising:

[0034] A feature extraction unit, used to obtain common features of unsupervised speech data using a preset feature extraction structure;

[0035] A feature adaptation unit, used to process the general feature using a preset local feature adapter to obtain a local feature adapted to the emotional speech data; the local feature includes a first feature and a second feature;

[0036] An attribute learning unit, used to determine a corresponding attribute learning loss based on the second feature; the attribute learning loss is used to generate a local attribute set; the local attribute set includes a plurality of local attributes;

[0037] A probability calculation unit, configured to determine a plurality of probability values ​​based on the first feature and the plurality of local attributes; the probability values ​​representing the probability that the similarity between the first feature and any of the local attributes meets a preset threshold;

[0038] An attribute comparison unit, configured to determine a corresponding local attribute comparison loss based on the plurality of probability values; the local attribute comparison loss is used to optimize the local feature adapter so as to maximize a log-likelihood function of the first feature;

[0039] A model optimization unit is used to determine the loss function of a target model based on the attribute learning loss and the local attribute contrast loss; the target model is used to perform speech emotion recognition on the speech input by the user.

[0040] A storage medium includes a stored program, wherein the program is executed by a processor to execute the local attribute learning method for speech emotion recognition.

[0041] An electronic device, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0042] The memory is used to store programs, and the processor is used to run programs, wherein the program executes the local attribute learning method for speech emotion recognition when the processor runs the program.

[0043] The technical solution provided by the present application utilizes a preset feature extraction structure to obtain universal features of unsupervised speech data. The universal features are processed using a preset local feature adapter to obtain local features adapted to emotional speech data. Based on the second feature, a corresponding attribute learning loss is determined. Based on the first feature and multiple local attributes, multiple probability values ​​are determined. Based on the multiple probability values, a corresponding local attribute contrast loss is determined. Based on the attribute learning loss and the local attribute contrast loss, a loss function of the target model is determined. The present application uses attribute learning loss and local attribute loss as the loss function of the target model. When the amount of speech emotion data is insufficient, the application can capture the local attribute changes brought about by special emotions by performing local feature analysis on the universal features extracted by the pre-training model (including the feature extraction structure), and obtain local features adapted to the speech emotion task, thereby improving the speech emotion recognition performance of the target model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A flowchart of a local attribute learning method for speech emotion recognition provided in an embodiment of the present application;

[0046] Figure 2 A flowchart of another local attribute learning method for speech emotion recognition provided in an embodiment of the present application;

[0047] Figure 3 A flowchart of another local attribute learning method for speech emotion recognition provided in an embodiment of the present application;

[0048] Figure 4 A flowchart of another local attribute learning method for speech emotion recognition provided in an embodiment of the present application;

[0049] Figure 5 A flowchart of another local attribute learning method for speech emotion recognition provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the architecture of a local attribute learning device for speech emotion recognition provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of a local attribute learning structure for speech emotion recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0054] The applicant has found that the main problems that limit the effectiveness of speech emotion recognition models include the following points.

[0055] (1) Insufficient data: Since emotional data must be manually labeled, which is time-consuming and laborious, the lack of sufficient data has always been the most important factor limiting speech emotion recognition. For this reason, many studies have chosen to use speaker datasets to pre-train feature extractors and have achieved positive results. However, since speaker recognition is a segment-level task, this method may cause the feature extractor to pay too much attention to the differences between different sentences and ignore the expression of local attributes, which in turn affects the accuracy of emotion recognition. At the same time, the proportion of speaker information in the extracted features is too high, which will also affect emotion recognition. In addition, the amount of data in the speaker dataset itself is also limited. Self-supervised pre-trained language models represented by the wav2vec2 model use self-supervised learning methods and use massive unlabeled speech data for pre-training, which has brought significant performance improvements to speech emotion recognition tasks. The pre-training objectives of these models are usually designed for speech recognition tasks, and the difference modeling effect of local information is significant, but they often only use a simple average pooling layer to process the features, ignoring the differences in local features. In addition, the feature information extracted by the self-supervised model is extremely rich and may contain a large amount of information irrelevant to emotion recognition, which interferes with emotion recognition.

[0056] (2) Local variability of emotions: In actual communication, emotional states change dynamically. Using global emotion tags for annotation and training cannot accurately describe the emotional differences of local information in sentences. Since the duration of emotional expression is variable, it is difficult for existing speech emotion recognition models to find the appropriate local feature length to fully express the true emotional state, and the extracted local features and emotion categories cannot match each other.

[0057] (3) The influence of other speech attributes on emotions: The expression of emotions is easily affected by other speech attributes. There will be great differences in the expression of emotions among different speakers, phonemes, and semantic contents.

[0058] In response to the above-mentioned applicant's discovery, an embodiment of the present application discloses a local attribute learning method for speech emotion recognition, which aims to optimize the self-supervised learning features of the speech emotion recognition model and improve the recognition performance of the speech emotion model. Specifically, without using additional labeled speech data, the local features extracted by the self-supervised model (i.e., the speech emotion recognition model) are deeply modeled, and the common features extracted by the pre-trained model (including the feature extraction structure) are subjected to local feature analysis to capture the local attribute changes brought about by special emotions, obtain local features adapted to the speech emotion task, and separate the local attributes related to the target task in the emotion classification process, thereby improving the accuracy, real-time and robustness of emotional speech recognition.

[0059] like Figure 1 As shown, a flow chart of a local attribute learning method for speech emotion recognition provided in an embodiment of the present application includes the following steps.

[0060] S101: Using a preset feature extraction structure, obtain common features of unsupervised speech data.

[0061] Among them, unsupervised speech data refers to speech data without labels. Unsupervised speech data can be used to train self-supervised models. Generally speaking, unsupervised speech data can usually be speech signals input by users, or unlabeled audio data.

[0062] Optionally, the feature extraction structure includes a waveform encoder and a BERT encoder, the waveform encoder includes a seven-layer one-dimensional convolutional neural network, and the BERT encoder includes 17 Transformer blocks.

[0063] In some examples, a waveform encoder can be used to convert an input speech signal into a speech feature sequence (including multiple speech features), the waveform encoder includes a seven-layer one-dimensional convolutional neural network, the step sizes of the seven-layer one-dimensional convolutional neural network are (5, 2, 2, 2, 2, 2, 2), the convolution kernel sizes of each layer in the seven-layer one-dimensional convolutional neural network are (10, 3, 3, 3, 3, 2, 2), the number of channels of each layer in the seven-layer one-dimensional convolutional neural network is 512, the speech signal can be a 16kHz waveform signal, and the output speech feature has a frame length of 25ms and a frame shift of 20ms.

[0064] In some examples, the 17 Transformer blocks contained in the BERT encoder all use 16-head attention and the feature dimension is 1024. The BERT encoder is used to process speech features and extract contextual information.

[0065] Optionally, the implementation process of obtaining the common features of unsupervised speech data using a preset feature extraction structure can be found in Figure 2 The steps are shown and the corresponding explanations.

[0066] S102: Using a preset local feature adapter, the general features are processed to obtain local features adapted to the emotional speech data.

[0067] The local features include a first feature and a second feature.

[0068] Optionally, the local feature adapter includes two adapter blocks and a convolutional layer, and the adapter block includes a self-attention module and a feed-forward neural network.

[0069] In some examples, the local feature adapter consists of two adapter blocks and a 1*1 convolutional layer. Each adapter block contains a 16-head self-attention module and a feedforward neural network layer. The 1*1 convolutional layer is used to reduce the dimension of the speech features to avoid the computational difficulties caused by too high a dimension.

[0070] Optionally, the implementation process of obtaining local features of general features by using a preset local feature adapter can be found in Figure 3 The steps are shown and the corresponding explanations.

[0071] S103: Based on the second feature, determine the corresponding attribute learning loss.

[0072] The attribute learning loss is used to generate a local attribute set, which includes multiple local attributes.

[0073] It should be noted that in order to model local features without using labels, a local attribute set P = [p 1 , p 2 ,…,p K ], p j ∈R C , p j An attribute that represents local information, such as phonemes, timbre, rhythm, etc. This attribute is used to help analyze local features.

[0074] In some examples, the attribute learning loss L Q The expression of can be found in formula (1)

[0075] L Q=||sg(x' i )-p j || 2 (1)

[0076] In formula (1), L Q represents the loss function of online attribute learning (i.e., attribute learning loss), x' i Represents the local feature adapter θ m The local part of the second feature of the output, that is, X' = [x' i ], p j Represents the distance from the second feature x' in the local attribute set i The nearest attribute, sg represents the stop gradient operator. In addition, the attribute learning loss is only used to update the local attribute set and is not passed to the local feature adapter θ. The local feature adapter θ m The local feature adapter θ is obtained by momentum update to ensure that during the feature adaptation process, the local feature adapter θ m The second feature x' of the output i The change is small, and the corresponding local properties will not change much.

[0077] S104: Determine multiple probability values ​​based on the first feature and multiple local attributes.

[0078] The probability value represents the probability that the similarity between the first feature and any local attribute meets a preset threshold.

[0079] It should be noted that the similarity between the first feature and any local attribute can be determined by calculating the cosine similarity between the first feature and any local attribute. In addition, the probability that the similarity between the first feature and any local attribute meets a preset threshold can be understood as the normalized indicator function (Softmax) value of the cosine similarity in the local attribute set.

[0080] In some examples, the first feature x i With any local attribute p j The expression of cosine similarity between can be recorded as cosine(x i , p j ), where x i By using the attribute contrast learning method, the local feature adapter is optimized to obtain a suitable local feature. To this end, this embodiment needs to find a local feature adapter parameter θ that maximizes the log-likelihood function of the first feature as shown in formula (2).

[0081]

[0082] In formula (2), B represents the number of samples in the batch, each sample contains T local features, qθ (x i ) is the local feature adapter θ extracted by x i Next, we introduce the local attribute p j , then we have formula (3).

[0083]

[0084] For formula (4), according to the Jünsen inequality, we can get formula (4).

[0085]

[0086] So we can get logq θ (x i ), as shown in formula (5).

[0087]

[0088] In formula (5), the second half is the distribution of local attributes, which is independent of the local feature adapter θ. Therefore, the optimization objective can be written as shown in formula (6).

[0089]

[0090] In formula (6), it is assumed that each x i Both have a local property p j Then formula (6) can be rewritten as formula (7).

[0091]

[0092] In formula (7), p(p j |x i ) is x i The probability value corresponding to the local attribute will be calculated closest to x i The local property is set to p(p j |x i )=1, and the other local attributes are set to 0. θ (x i |p j ) is the distribution that this embodiment hopes to make the first feature obey.

[0093] Ideally, the first feature follows a Gaussian distribution around the corresponding attribute. Assume x i In p j The surrounding area follows a Gaussian distribution with a variance of 0.5, and we can get formula (8).

[0094]

[0095] Based on formula (8), the log-likelihood function that needs to be maximized can be simplified to formulas (9), (10) and (11).

[0096]

[0097] In formulas (9), (10) and (11), α i,j represents the probability value, that is, the Softmax value of the similarity in the local attribute set, and τ represents the temperature coefficient (set to 1).

[0098] S105: Determine a corresponding local attribute contrast loss based on the multiple probability values.

[0099] Among them, the local attribute contrast loss is used to optimize the local feature adapter to maximize the log-likelihood function of the first feature.

[0100] In some examples, based on multiple probability values, the determined local attribute contrast loss L p , as shown in formula (11).

[0101] It should be noted that while using the local attribute contrast loss to optimize the local feature adapter θ, the exponential moving average (EMA) method can also be used to optimize the local feature adapter θ. m Update, that is, local feature adapter θ and local feature adapter θ m Learning updates can be performed based on the same sample, and the update process can be seen in formula (12).

[0102] θ m ←γθ+(1-γ)θ m (12)

[0103] In formula (12), γ represents the update coefficient (which can be set to 0.001).

[0104] S106: Determine a loss function of the target model based on the attribute learning loss and the local attribute contrast loss.

[0105] Among them, the target model is used to perform speech emotion recognition on the speech input by the user.

[0106] It should be noted that the target model can be specifically a local feature adapter θ and a local feature adapter θ m In addition, the target model uses sentiment classification loss for classification and attribute separation, and attribute learning loss L Q And the local attribute contrast loss L p Not relevant.

[0107] In some examples, the speech emotion adaptation part of the target model can use the attribute learning loss L Q And the local attribute contrast loss L p , then the loss function L_attribute adopted by the target model can be expressed as shown in formula (13).

[0108] L_attribute = L p +λL Q (13)

[0109] In formula (13), λ represents the relative weight of the loss function (which can be set to 1).

[0110] It should be noted that by determining the loss function of the target model based on attribute learning loss and local attribute contrast loss, the universal local speech features extracted by the target model can be adapted to emotional speech, which effectively alleviates the problem of insufficient emotional data. Local attribute mapping is designed to characterize the complexity of emotions, and an exponential softmax mechanism is used to achieve effective mapping of local attributes to global attributes.

[0111] In some examples, for specific speech emotion recognition tasks, a task-oriented online attribute separation strategy can be designed during the target model fine-tuning process to select relevant local attributes for mapping, thereby reducing the impact of irrelevant information on emotion recognition performance and improving recognition efficiency and accuracy.

[0112] Optionally, the implementation process of selecting relevant local attributes for mapping can be seen in Figure 4 The steps are shown and the corresponding explanations.

[0113] In order to make full use of the local attribute learning method to improve the performance of speech emotion recognition, and to verify the effectiveness of the method proposed in this invention, the following experiments were designed: Experiments were conducted on the IEMOCAP dataset and the multimodal emotion recognition challenge MER2023 dataset to evaluate the effectiveness of the proposed method. The IEMOCAP dataset contains about 12 hours of English audio-visual data. The database is divided into 5 parts, each containing a male and a female actor. According to the recording scene, it can be further subdivided into improvisational dialogue or script dialogue parts. Each utterance is annotated with 8 emotion tags by multiple annotators. In order to make a fair comparison with previous work, 4 emotion types (i.e., neutral, happy, angry, and sad) were selected in the experiment. The experiment adopted a 5-fold cross-validation strategy to evaluate the dataset, with 4 parts of sentences used for training in each fold, and the remaining parts used for validation and testing. The MER2023 dataset provides 3373 Chinese speech data with a total duration of nearly 4 hours. Each audio clip is annotated with six categories (neutral, angry, happy, sad, worried, and surprised) and is accompanied by a one-dimensional numerical valence to assess the degree of positivity of the emotion. Following the pre-set benchmark, the dataset was randomly divided into five parts, and a 5-fold cross-validation was performed to evaluate the performance. All systems were implemented using the PyTorch deep learning framework. The experiment used the Adam optimizer and set the mini-batch size to 128. For the IEMOCAP dataset, the system was trained for 50 cycles with an initial learning rate of 0.0001. For the MER2023 dataset, the system was trained for 100 cycles with an initial learning rate of 0.0001.

[0114] In addition, the evaluation indicators pre-set for the experiment are: Weighted Accuracy (WA), Unweighted Accuracy (UA), F1-Score (F1-Score), Mean Square Error (MSE). For the MER2023 dataset, the experiment used F1-Score to evaluate the six-category emotion recognition, and used MSE to evaluate the one-dimensional numerical Valence.

[0115] Specifically, WA considers the accuracy calculation method of the number of samples in each category, and the WA calculation process is shown in formula (14).

[0116]

[0117] In formula (14), N i represents the number of samples of category i, A iRepresents the accuracy of category i, N represents the total number of samples, and formula (14) makes the contribution of each category's error to the overall accuracy correspond to its proportion in the data set.

[0118] Specifically, UA calculates the average accuracy for each emotion category, that is, the number of correctly predicted instances in a category divided by the total number of instances in the category, and then averages the accuracy of all categories. The UA calculation process is shown in formula (15).

[0119]

[0120] In formula (15), A i represents the accuracy of category i, N classes Represents the number of categories.

[0121] Specifically, F1-Score is the harmonic mean of precision and recall. The calculation process of F1-Score is shown in formula (16).

[0122]

[0123] In formula (16), precision represents the accuracy and recall represents the recall.

[0124] The experimental results shown in Table 1 show the effect of different numbers of prototypes on speech emotion recognition when the attribute separation method shown in the present invention is not used. In Table 1, "average pooling" and "maximum pooling" indicate that the attribute learning method is not used, and the average pooling or maximum pooling method is directly used to process local features after the BERT encoder.

[0125] Table 1

[0126]

[0127]

[0128] The experimental results shown in Table 1 show that the pre-trained local attribute learning method proposed by us significantly improves the performance of emotion recognition on the IEMOCAP and MER2023 datasets. On the IEMOCAP dataset, the WA index is improved by about 3.31%, the UA is improved by about 8.04%, and the F1-score is improved by about 6.56%. On the MER2023 dataset, the F1-score is improved by about 4.39%.

[0129] Table 2 shows the experimental results of attribute separation based on 1024 local attributes and the influence of different separation ratios on the effect of emotion recognition.

[0130] Table 2

[0131]

[0132] The experimental results shown in Table 2 show that selecting a small number of attributes is significantly helpful for emotion recognition. The best effect of emotion recognition is achieved when only 6%-10% of local attributes are selected.

[0133] Table 3 is disclosed for the performance comparison between the existing self-supervised model and the target model shown in the present invention.

[0134] Table 3

[0135]

[0136] The experimental results shown in Table 3 indicate that the local attribute learning method shown in the present invention has made significant progress in performance compared to the prior art.

[0137] In general, the present invention proposes a local attribute learning method based on pre-training for speech emotion recognition. It aims to model the local features of speech signals through an unsupervised attribute contrast learning method. The network performs contrast training on local features by learning a set of attributes online, thereby modeling frame-level local features. In the fine-tuning stage, for different downstream tasks, task-related speech attributes are screened out and attribute separation is performed. Experiments on the IEMOCAP and MER2023 datasets have shown that this method significantly improves the effect of speech emotion recognition.

[0138] In some examples, the role of the feature extraction structure and the local feature adapter shown in the embodiments of the present application in the pre-training process can be seen in Figure 7 shown.

[0139] The process shown in the above S101-S106 utilizes the local feature adapter to obtain the corresponding attribute learning loss and local attribute loss, and uses the attribute learning loss and local attribute loss as the loss function of the target model. When the amount of speech emotion data is insufficient, the local features extracted by the target model can be deeply modeled to accurately capture the changes in local statistical information brought about by the complexity of speech emotion, thereby improving the speech emotion recognition performance of the target model.

[0140] like Figure 2 As shown, a flow chart of a local attribute learning method for speech emotion recognition provided in an embodiment of the present application includes the following steps.

[0141] S201: Obtain unsupervised speech data.

[0142] Among them, the unsupervised speech data includes the speech signal input by the user.

[0143] S202: Input the speech signal into a waveform encoder to obtain speech features output by the waveform encoder.

[0144] Among them, the waveform encoder can be pre-trained using a large amount of unlabeled audio data (such as Libri-Light 60k).

[0145] S203: Input the speech features into the BERT encoder to obtain the general features output by the BERT encoder.

[0146] Among them, the BERT encoder can be pre-trained using a large amount of unlabeled audio data (such as Libri-Light 60k).

[0147] The process shown in S201-S203 above can utilize the waveform encoder and the BERT encoder to obtain corresponding universal features from the unsupervised speech data.

[0148] like Figure 3 As shown, a flow chart of a local attribute learning method for speech emotion recognition provided in an embodiment of the present application includes the following steps.

[0149] S301: Input the general feature into a preset local feature adapter to obtain a first feature output by the local feature adapter.

[0150] Among them, the local feature adapter can adjust the dimension of the general feature to X = [x 1 , x 2 , …, x T ], x i ∈R C , that is, the first feature can be recorded as X = [x 1 , x 2 , …, x T ].

[0151] In some examples, the local feature adapter may be denoted as θ.

[0152] S302: Perform momentum update on the local feature adapter to obtain a momentum-updated local feature adapter.

[0153] Among them, momentum update is a technique used to accelerate gradient descent. It calculates the current update direction by accumulating the exponentially weighted average of past gradients. Momentum update can speed up the convergence of local feature adapters, especially for flat areas with small gradients. Momentum update can suppress high-frequency oscillations and make gradient descent smoother.

[0154] In some examples, the local feature adapter after momentum update can be denoted as θ mIt can be understood that the network structure of the local feature adapter θ is similar to the local feature adapter θ m The network structure is the same.

[0155] S303: Input the general feature into the momentum-updated local feature adapter to obtain a second feature output by the momentum-updated local feature adapter.

[0156] Among them, the local feature adapter after momentum update can adjust the dimension of the general feature to X'=[x' 1 , x' 2 , …, x' T ],x' i ∈R C , that is, the first feature can be recorded as X' = [x' 1 , x' 2 , …, x' T ].

[0157] The process shown in S301-S303 above uses the local feature adapter and the local feature adapter after momentum update to obtain the first feature and the second feature of the general feature.

[0158] like Figure 4 As shown, a flow chart of a local attribute learning method for speech emotion recognition provided in an embodiment of the present application includes the following steps.

[0159] S401: Obtain a local attribute mapping based on a probability value between the first feature and each local attribute.

[0160] Among them, the probability value α between the first feature and each local attribute i,j , which can be regarded as the first feature x i Belongs to the local attribute p j The probability of, based on all probability values ​​α i,j , which can be used as the mapping of the first feature in the feature space composed of the local attribute set, namely the so-called local attribute mapping d i =[α i,1 ,α i,2 ,…,α i,k ].

[0161] S402: Using a normalized exponential function, globally aggregate the local attribute mapping to obtain segment-level attribute features.

[0162] Among them, the normalized exponential function is the exponential Softmax function, which converts exp(α i,j ) as a weight and assigned to the global attribute probability y j , the allocation process can be seen in formula (17).

[0163]

[0164] When global aggregation is performed on the local attribute mapping using formula (17), y is calculated j Relative to α i,j The calculation formula of the gradient can be seen in formula (18).

[0165]

[0166] In formula (18), because y j It must be less than 1, so the gradient is always positive, that is, the final loss function will make directional adjustments to all frames in the entire speech, and the magnitude of the adjustment will vary according to their respective attributes.

[0167] It should be noted that in people's actual communication, emotional states change dynamically. For example, when a sentence is judged as "happy" as a whole, the speech will not always express "happiness", but will partially express neutrality or other emotions. Traditional methods often use average pooling for aggregation, which will cause the network to judge every part of the sentence as the target category. The speech emotion recognition task is essentially a detection task. For various basic attributes, assuming that at least one local feature in the segment-level speech contains information about the attribute, the entire sentence should be judged to contain the attribute. This goal can be achieved using the maximum pooling method, that is, the maximum α i,j Take y j However, the maximum pooling method has a flaw, that is, only one frame in the entire speech segment can pass the signal to the subsequent processing. If an event occurs multiple times in the speech segment, it is easy to miss these frames. Therefore, the exponential Softmax function is selected to alleviate this problem.

[0168] In some examples, the expression of the segment-level attribute feature may be Y=[y 1 ,y 2 ,…,y k ].

[0169] S403: According to a preset speech emotion recognition task, segment-level attribute features are separated to obtain target features.

[0170] Among them, the target features contain attributes related to the speech emotion recognition task.

[0171] Optionally, according to the preset speech emotion recognition task, the segment-level attribute features are separated to obtain the target feature. For details, see Figure 5 The steps shown are accompanied by corresponding explanations.

[0172] S404: Determine the corresponding cross entropy loss based on the target feature.

[0173] Among them, the cross entropy loss is used to calculate the classification results of the speech emotion recognition task.

[0174] In some examples, the segment-level attribute features (i.e., target features) after attribute decomposition are input into the classifier to obtain the predicted classification results, and the cross entropy loss L between the predicted and true values ​​is calculated. CE .

[0175] In a possible implementation, the emotion classifier is composed of a fully connected layer, the input is the target feature, and the output is the classification result of the speech emotion recognition task. Different speech emotion recognition tasks use classifiers of different sizes.

[0176] S405: Determine a loss function of the target model based on attribute learning loss, local attribute contrast loss, and cross entropy loss.

[0177] Among them, based on attribute learning loss, local attribute contrast loss and cross entropy loss, the loss function of the target model is determined. The loss function Loss of the target model can be shown in formula (19).

[0178] Loss = L_attribute + λ CE ·L CE (19)

[0179] In formula (19), λ CE Represents the weight of the fine-tuned part (can be set to 1).

[0180] In some examples, the cross entropy loss L CE The training of the loss function L_attribute can be performed separately.

[0181] The process shown in S401-S405 above can utilize local attribute mapping to obtain target features related to the speech emotion recognition task, utilize the target features to determine the corresponding cross entropy loss, and incorporate the cross entropy loss into the loss function of the target model. This can accurately capture the changes in local statistical information brought about by the complexity of speech emotions, and extract attribute features closely related to the emotion recognition task, and map them to the corresponding global representation, thereby improving the accuracy, real-time and robustness of emotional speech recognition.

[0182] like Figure 5 As shown, a flow chart of a local attribute learning method for speech emotion recognition provided in an embodiment of the present application includes the following steps.

[0183] S501: Determine a corresponding Fisher information matrix according to a preset speech emotion recognition task.

[0184] The Fisher information matrix is ​​a generalization of the Fisher information from a single parameter to multiple parameters. The Fisher information represents the average amount of information that a sample of a random variable can provide about the state parameter in a certain sense. The Fisher information can be understood as a measure of the expected value of the amount of information about the location parameter that a single observation can provide.

[0185] S502: Determine the importance index of each global attribute probability in the segment-level attribute feature based on the Fisher information matrix.

[0186] Among them, since the local attribute set is obtained based on unsupervised learning of local information, not all local attributes are useful for the speech emotion recognition task. Therefore, the segment-level attribute feature Y is attributed. Specifically, in the specific training of the speech emotion recognition task, for the Fisher information matrix, the more important the attribute (i.e., the global attribute probability) is to the speech emotion recognition task, the greater its Fisher information. The importance index of all attributes to the speech emotion recognition task is calculated, and the weights of some attributes with the highest importance index can be set to 1, and the weights of the remaining attributes can be set to 0.

[0187] S503: Determine the global attribute probability that meets the preset requirements based on the importance index as the target feature.

[0188] Among them, the global attribute probability is updated in real time based on the training process of the target model. In addition, the preset requirements can be: the importance index is the highest, or the importance index is greater than the specified threshold.

[0189] It should be noted that for different speech emotion recognition tasks, the number of attributes required varies greatly, so different attribute weight settings can be adopted.

[0190] The process shown in S501-S503 above can use the Fisher information matrix to achieve attribute decomposition of segment-level attribute features to obtain target features related to the speech emotion recognition task.

[0191] like Figure 6 As shown, it is a schematic diagram of the architecture of a local attribute learning device for speech emotion recognition provided in an embodiment of the present application, including the units shown below.

[0192] The feature extraction unit 100 is used to obtain the general features of unsupervised speech data using a preset feature extraction structure.

[0193] Optionally, the feature extraction structure includes a waveform encoder and a BERT encoder, and the feature extraction unit 100 is specifically used to: obtain unsupervised speech data; the unsupervised speech data includes a speech signal input by a user; input the speech signal into the waveform encoder to obtain speech features output by the waveform encoder; input the speech features into the BERT encoder to obtain general features output by the BERT encoder.

[0194] Optionally, the waveform encoder includes seven layers of one-dimensional convolutional neural networks and the BERT encoder includes 17 Transformer blocks.

[0195] The feature adaptation unit 200 is used to process the general features using a preset local feature adapter to obtain local features adapted to the emotional speech data; the local features include a first feature and a second feature.

[0196] Optionally, the feature adaptation unit 200 is specifically used to: input the general feature into a preset local feature adapter to obtain a first feature output by the local feature adapter; perform momentum update on the local feature adapter to obtain a local feature adapter after the momentum update; input the general feature into the local feature adapter after the momentum update to obtain a second feature output by the local feature adapter after the momentum update.

[0197] Optionally, the local feature adapter includes two adapter blocks and a convolutional layer, and the adapter block includes a self-attention module and a feed-forward neural network.

[0198] The attribute learning unit 300 is used to determine the corresponding attribute learning loss based on the second feature; the attribute learning loss is used to generate a local attribute set; the local attribute set includes multiple local attributes.

[0199] The probability calculation unit 400 is used to determine multiple probability values ​​based on the first feature and multiple local attributes; the probability value represents the probability that the similarity between the first feature and any local attribute meets a preset threshold.

[0200] The attribute comparison unit 500 is used to determine the corresponding local attribute comparison loss based on multiple probability values; the local attribute comparison loss is used to optimize the local feature adapter to maximize the log-likelihood function of the first feature.

[0201] The model optimization unit 600 is used to determine the loss function of the target model based on the attribute learning loss and the local attribute contrast loss; the target model is used to perform speech emotion recognition on the speech input by the user.

[0202] Optionally, the model optimization unit 600 is specifically used to: obtain a local attribute mapping based on the probability value corresponding to the first feature and each local attribute; use a normalized exponential function to globally aggregate the local attribute mapping to obtain segment-level attribute features; according to a preset speech emotion recognition task, perform attribute separation on the segment-level attribute features to obtain target features; the target features include attributes related to the speech emotion recognition task; based on the target features, determine the corresponding cross entropy loss; the cross entropy loss is used to calculate the classification result of the speech emotion recognition task; based on the attribute learning loss, the local attribute contrast loss and the cross entropy loss, determine the loss function of the target model.

[0203] Optionally, the model optimization unit 600 is specifically used to: determine the corresponding Fisher information matrix according to a preset speech emotion recognition task; determine the importance index of each global attribute probability in the segment-level attribute feature based on the Fisher information matrix; determine the global attribute probability that meets preset requirements based on the importance index as the target feature; wherein the global attribute probability is updated in real time based on the training process of the target model.

[0204] Each unit shown above uses a local feature adapter to obtain the corresponding attribute learning loss and local attribute loss. Based on the attribute learning loss and local attribute loss as the loss function of the target model, when the amount of speech emotion data is insufficient, the local features extracted by the target model can be deeply modeled to accurately capture the changes in local statistical information brought about by the complexity of speech emotions, thereby improving the speech emotion recognition performance of the target model.

[0205] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the local attribute learning method for speech emotion recognition provided by the present application.

[0206] The present application also provides an electronic device, including: a processor, a memory and a bus. The processor and the memory are connected via the bus, the memory is used to store programs, and the processor is used to run the programs, wherein the local attribute learning method for speech emotion recognition provided by the present application is executed when the programs are run.

[0207] In addition, the functions described above in the embodiments of the present application may be at least partially performed by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0208] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0209] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A local attribute learning method for speech emotion recognition, characterized in that: include: Using the preset feature extraction structure, the common features of unsupervised speech data are obtained; Using a preset local feature adapter, the general feature is processed to obtain a local feature adapted to the emotional speech data; the local feature includes a first feature and a second feature; Based on the second feature, determining a corresponding attribute learning loss; the attribute learning loss is used to generate a local attribute set; the local attribute set includes a plurality of local attributes; Based on the first feature and the plurality of local attributes, a plurality of probability values ​​are determined; the probability values ​​represent the probability that the similarity between the first feature and any of the local attributes meets a preset threshold; Based on the plurality of probability values, a corresponding local attribute contrast loss is determined; the local attribute contrast loss is used to optimize the local feature adapter so as to maximize the log-likelihood function of the first feature; Based on the attribute learning loss and the local attribute contrast loss, a loss function of a target model is determined; the target model is used to perform speech emotion recognition on speech input by a user.

2. The method according to claim 1, characterized in that The general features are processed by using a preset local feature adapter to obtain local features adapted to the emotional speech data, including: Inputting the general feature into a preset local feature adapter to obtain a first feature output by the local feature adapter; Performing momentum update on the local feature adapter to obtain a momentum-updated local feature adapter; The general feature is input into the momentum-updated local feature adapter to obtain a second feature output by the momentum-updated local feature adapter.

3. The method according to claim 1, characterized in that Based on the attribute learning loss and the local attribute contrast loss, determining a loss function of a target model includes: Obtaining a local attribute mapping based on a probability value corresponding to the first feature and each of the local attributes; Using a normalized exponential function, globally aggregating the local attribute maps to obtain segment-level attribute features; According to a preset speech emotion recognition task, the segment-level attribute features are attribute-separated to obtain target features; the target features include attributes related to the speech emotion recognition task; Based on the target feature, determining a corresponding cross entropy loss; the cross entropy loss is used to calculate the classification result of the speech emotion recognition task; A loss function of a target model is determined based on the attribute learning loss, the local attribute contrast loss, and the cross entropy loss.

4. The method according to claim 3, characterized in that According to the preset speech emotion recognition task, the segment-level attribute features are attribute-separated to obtain target features, including: According to the preset speech emotion recognition task, determine the corresponding Fisher information matrix; Based on the Fisher information matrix, determining the importance index of each global attribute probability in the segment-level attribute features; A global attribute probability that meets preset requirements based on the importance index is determined as a target feature; wherein the global attribute probability is updated in real time based on the training process of the target model.

5. The method according to claim 1, characterized in that: The local feature adapter includes two adapter blocks and a convolutional layer, and the adapter block includes a self-attention module and a feedforward neural network.

6. The method according to claim 1, characterized in that The feature extraction structure includes a waveform encoder and a BERT encoder, and uses a preset feature extraction structure to obtain common features of unsupervised speech data, including: Obtaining unsupervised speech data; the unsupervised speech data includes a speech signal input by a user; Inputting the speech signal into the waveform encoder to obtain speech features output by the waveform encoder; The speech features are input into the BERT encoder to obtain the general features output by the BERT encoder.

7. The method according to claim 6, characterized in that The waveform encoder includes a seven-layer one-dimensional convolutional neural network, and the BERT encoder includes 17 Transformer blocks.

8. A local attribute learning device for speech emotion recognition, characterized in that: include: A feature extraction unit, used to obtain common features of unsupervised speech data using a preset feature extraction structure; A feature adaptation unit, used to process the general feature using a preset local feature adapter to obtain a local feature adapted to the emotional speech data; the local feature includes a first feature and a second feature; An attribute learning unit, used to determine a corresponding attribute learning loss based on the second feature; the attribute learning loss is used to generate a local attribute set; the local attribute set includes a plurality of local attributes; A probability calculation unit, configured to determine a plurality of probability values ​​based on the first feature and the plurality of local attributes; the probability values ​​representing the probability that the similarity between the first feature and any of the local attributes meets a preset threshold; An attribute comparison unit, used for determining a corresponding local attribute comparison loss based on the plurality of probability values; The local attribute contrast loss is used to optimize the local feature adapter so as to maximize the log-likelihood function of the first feature; A model optimization unit is used to determine the loss function of a target model based on the attribute learning loss and the local attribute contrast loss; the target model is used to perform speech emotion recognition on the speech input by the user.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program, when executed by a processor, executes the local attribute learning method for speech emotion recognition according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program, when run by the processor, executes the local attribute learning method for speech emotion recognition described in any one of claims 1-7.

Images