Voice emotion recognition method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a speech emotion recognition method and device, an electronic device and a storage medium. The method comprises: obtaining a pre-trained speech emotion recognition model, wherein the speech emotion recognition model comprises a first module and a second module; obtaining a first speech sample set corresponding to a target user, wherein the first speech sample set comprises a plurality of first speech samples input by the target user and a plurality of emotion classification labels corresponding to the plurality of first speech samples; for any first speech sample in the first speech sample set, obtaining a speech feature corresponding to the first speech sample; inputting the speech feature corresponding to the first speech sample into the second module to obtain an emotion classification prediction result corresponding to the first speech sample; and updating parameters of the second module according to the emotion classification prediction result corresponding to the first speech sample and the emotion classification label corresponding to the first speech sample.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a voice emotion recognition method, a voice emotion recognition device, an electronic device, and a storage medium. Background Technology

[0002] Voice is the primary medium for communication in daily life, conveying not only thoughts but also the speaker's emotional state. Through voice emotion recognition, a user's mental health can be assessed, and appropriate feedback and responses can be provided based on the user's emotional changes in different scenarios (e.g., digital humans, chatbot customer service). Voice emotion recognition also helps parents monitor indicators such as their children's mental health. Therefore, accurate voice emotion recognition is of great significance. Summary of the Invention

[0003] This disclosure provides a speech emotion recognition technology solution.

[0004] According to one aspect of this disclosure, a method for training a speech emotion recognition model is provided, comprising:

[0005] A pre-trained speech emotion recognition model is obtained, wherein the speech emotion recognition model includes a first module and a second module;

[0006] Obtain a first voice sample set corresponding to the target user, wherein the first voice sample set includes multiple first voice samples input by the target user, and multiple emotion classification labels corresponding to the multiple first voice samples;

[0007] For any first speech sample in the first speech sample set, obtain the speech features corresponding to the first speech sample;

[0008] Input the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample;

[0009] The parameters of the second module are updated based on the emotion classification prediction result and the emotion classification label corresponding to the first speech sample.

[0010] In one possible implementation,

[0011] The method further includes: obtaining a trained speaker recognition model;

[0012] The step of obtaining the speech features corresponding to any first speech sample in the first speech sample set includes: obtaining the speech features corresponding to any first speech sample in the first speech sample set through the first module and / or the speaker recognition model.

[0013] In one possible implementation, obtaining the pre-trained speech emotion recognition model includes:

[0014] Obtain a second speech sample set, wherein the second speech sample set includes multiple second speech samples corresponding to multiple speakers, and multiple emotion classification labels corresponding to the multiple second speech samples;

[0015] The speech emotion recognition model is trained using the second speech sample set until the first preset training condition is met.

[0016] For any second speech sample in the second speech sample set, the speech features corresponding to the second speech sample are obtained through the first module and / or the speaker recognition model;

[0017] Input the speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample;

[0018] The parameters of the second module are updated based on the sentiment classification prediction result corresponding to the second speech sample and the sentiment classification label corresponding to the second speech sample.

[0019] In one possible implementation,

[0020] For any second speech sample in the second speech sample set, obtaining the speech features corresponding to the second speech sample through the first module and / or the speaker recognition model includes: for any second speech sample in the second speech sample set, obtaining the first speech feature corresponding to the second speech sample through the first module, and obtaining the second speech feature corresponding to the second speech sample through the speaker recognition model; obtaining the third speech feature corresponding to the second speech sample based on the first speech feature and the second speech feature corresponding to the second speech sample.

[0021] The step of inputting the speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample includes: inputting the third speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample.

[0022] In one possible implementation, the speaker recognition model is a lightweight model.

[0023] In one possible implementation, the speaker recognition model is deployed at least on the user terminal corresponding to the target user;

[0024] The step of obtaining the speech features corresponding to the first speech sample through the first module and / or the speaker recognition model includes: obtaining the second speech features corresponding to the first speech sample through the speaker recognition model deployed on the user terminal corresponding to the target user;

[0025] The step of inputting the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample includes: inputting the second speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0026] In one possible implementation, obtaining the second speech feature corresponding to the first speech sample through a speaker recognition model deployed on the user terminal corresponding to the target user includes:

[0027] In response to the target user's request to retain the first speech sample set on the user terminal, the second speech feature corresponding to the first speech sample is obtained by using a speaker recognition model deployed on the user terminal corresponding to the target user.

[0028] In one possible implementation,

[0029] The step of obtaining the speech features corresponding to the first speech sample through the first module and / or the speaker recognition model includes: in response to the target user's consent to upload the first speech sample set, obtaining the first speech features corresponding to the first speech sample through the first module;

[0030] The step of inputting the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample includes: inputting the first speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0031] According to one aspect of this disclosure, a voice emotion recognition method is provided, comprising:

[0032] The training method for the speech emotion recognition model is used to obtain the trained speech emotion recognition model;

[0033] The speech to be processed is input into the trained speech emotion recognition model, and the emotion classification label corresponding to the speech to be processed is obtained through the trained speech emotion recognition model.

[0034] According to one aspect of this disclosure, a training apparatus for a speech emotion recognition model is provided, comprising:

[0035] The first acquisition module is used to acquire a pre-trained speech emotion recognition model, wherein the speech emotion recognition model includes a first module and a second module;

[0036] The first acquisition module is used to acquire a first voice sample set corresponding to the target user, wherein the first voice sample set includes multiple first voice samples input by the target user, and multiple emotion classification labels corresponding to the multiple first voice samples;

[0037] The second obtaining module is used to obtain the speech features corresponding to any first speech sample in the first speech sample set.

[0038] The third acquisition module is used to input the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0039] The update module is used to update the parameters of the second module based on the sentiment classification prediction result and the sentiment classification label corresponding to the first speech sample.

[0040] In one possible implementation,

[0041] The device further includes: obtaining a trained speaker recognition model;

[0042] The second obtaining module is used to: for any first speech sample in the first speech sample set, obtain the speech features corresponding to the first speech sample through the first module and / or the speaker recognition model.

[0043] In one possible implementation, the first obtaining module is used to:

[0044] Obtain a second speech sample set, wherein the second speech sample set includes multiple second speech samples corresponding to multiple speakers, and multiple emotion classification labels corresponding to the multiple second speech samples;

[0045] The speech emotion recognition model is trained using the second speech sample set until the first preset training condition is met.

[0046] For any second speech sample in the second speech sample set, the speech features corresponding to the second speech sample are obtained through the first module and / or the speaker recognition model;

[0047] Input the speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample;

[0048] The parameters of the second module are updated based on the sentiment classification prediction result corresponding to the second speech sample and the sentiment classification label corresponding to the second speech sample.

[0049] In one possible implementation, the first obtaining module is used to:

[0050] For any second speech sample in the second speech sample set, the first speech feature corresponding to the second speech sample is obtained through the first module, and the second speech feature corresponding to the second speech sample is obtained through the speaker recognition model; based on the first speech feature corresponding to the second speech sample and the second speech feature corresponding to the second speech sample, the third speech feature corresponding to the second speech sample is obtained.

[0051] The third speech feature corresponding to the second speech sample is input into the second module to obtain the emotion classification prediction result corresponding to the second speech sample.

[0052] In one possible implementation, the speaker recognition model is a lightweight model.

[0053] In one possible implementation, the speaker recognition model is deployed at least on the user terminal corresponding to the target user;

[0054] The second obtaining module is used to: obtain the second speech feature corresponding to the first speech sample by using a speaker recognition model deployed on the user terminal corresponding to the target user;

[0055] The third obtaining module is used to: input the second speech feature corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0056] In one possible implementation, the second obtaining module is used to:

[0057] In response to the target user's request to retain the first speech sample set on the user terminal, the second speech feature corresponding to the first speech sample is obtained by using a speaker recognition model deployed on the user terminal corresponding to the target user.

[0058] In one possible implementation,

[0059] The second obtaining module is used to: in response to the target user's consent to upload the first voice sample set, obtain the first voice feature corresponding to the first voice sample through the first module;

[0060] The third obtaining module is used to: input the first speech feature corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0061] According to one aspect of this disclosure, a voice emotion recognition device includes:

[0062] The second acquisition module is used to acquire the voice emotion recognition model trained by the training device of the voice emotion recognition model;

[0063] The speech emotion recognition module is used to input the speech to be processed into the trained speech emotion recognition model, and obtain the emotion classification label corresponding to the speech to be processed through the trained speech emotion recognition model.

[0064] According to one aspect of this disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the executable instructions stored in the memory to perform the method described above.

[0065] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0066] According to one aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device performs the above-described method.

[0067] In this embodiment, a pre-trained speech emotion recognition model is obtained, comprising a first module and a second module. The model acquires a first speech sample set corresponding to the target user. For any first speech sample in the first speech sample set, speech features corresponding to the first speech sample are obtained. These speech features are input into the second module to obtain an emotion classification prediction result for the first speech sample. Based on the emotion classification prediction result and the emotion classification label, the parameters of the second module are updated. This constrains the pre-trained speech emotion recognition model using a first speech sample set strongly correlated with the target user, allowing for fine-tuning of the model. This enables rapid adaptation to the target user's emotional characteristics. Since the speech emotion recognition model for the target user is fine-tuned based on the pre-trained model, it can quickly adapt to different users. Because it eliminates the need to train a separate speech emotion recognition model for each user from scratch, the training cost for speech emotion recognition models for different users is reduced.

[0068] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0069] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0071] Figure 1 A flowchart illustrating the training method of the speech emotion recognition model provided in the embodiments of this disclosure is shown.

[0072] Figure 2 A schematic diagram of the speech emotion recognition model provided in an embodiment of this disclosure is shown.

[0073] Figure 3 The diagram illustrates a method for training a speech emotion recognition model provided in this embodiment of the present disclosure, in which, after training the speech emotion recognition model to a first preset training condition, a speaker recognition model is introduced to continue training the speech emotion recognition model.

[0074] Figure 4 An exemplary flowchart illustrating the training and deployment of a speech emotion recognition model provided in embodiments of this disclosure is shown.

[0075] Figure 5 A block diagram of a training apparatus for a speech emotion recognition model provided in an embodiment of this disclosure is shown.

[0076] Figure 6 A block diagram of an electronic device 1900 provided in an embodiment of this disclosure is shown. Detailed Implementation

[0077] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0078] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0079] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0080] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0081] Because voice emotion data is difficult to label, voice emotion recognition models tend to favor the emotions of speakers in the training set. This means that if a different speaker is used for voice emotion recognition testing, misidentification often occurs. Furthermore, voice emotion recognition is subjective; different users may interpret the same sentence differently—some may perceive it as emotionally charged, while others may perceive it as calm. Therefore, customizing appropriate voice emotion recognition models for different speakers is of great importance.

[0082] In speech emotion recognition systems based on relevant technologies, the following two methods are mainly used to achieve speaker adaptation:

[0083] The first approach involves pre-acquiring each speaker's voiceprint using a voiceprint model during the initial training of the speech emotion recognition model, and then training different speech emotion recognition models for different speakers. In other words, a separate speech emotion recognition model is trained from scratch for each speaker. The drawback of this approach is that it requires simultaneous modeling of voiceprint features and emotion classification during the initial training phase, making it difficult to iterate quickly for different users. That is, it cannot quickly adapt the trained speech emotion recognition model to different users, and the cost of training different speech emotion recognition models for different speakers is relatively high.

[0084] The second approach involves constraining different speakers during the training process of the speech emotion recognition model, allowing the model to learn emotion recognition features independent of the speaker. While this effectively improves the robustness of the speech emotion recognition system to different speakers, it cannot quickly adapt to a particular user. For example, if a user's emotional expression is not very pronounced, the emotion recognition may not be accurate.

[0085] This disclosure provides a method for training a speech emotion recognition model. By obtaining a pre-trained speech emotion recognition model, which includes a first module and a second module, the method involves acquiring a first speech sample set corresponding to a target user. For any first speech sample in the first speech sample set, the method obtains the speech features corresponding to the first speech sample. These speech features are then input into the second module to obtain an emotion classification prediction result for the first speech sample. Based on the emotion classification prediction result and the emotion classification label of the first speech sample, the parameters of the second module are updated. This method constrains and fine-tunes the pre-trained speech emotion recognition model using a first speech sample set strongly correlated with the target user, enabling rapid adaptation to the target user's emotional characteristics. Since the speech emotion recognition model for the target user is fine-tuned based on the pre-trained model, it can quickly adapt to different users. Because it eliminates the need to train a separate speech emotion recognition model for each user from scratch, the training cost for speech emotion recognition models for different users is reduced.

[0086] The training method of the speech emotion recognition model provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0087] Figure 1A flowchart illustrating a training method for a speech emotion recognition model provided in an embodiment of this disclosure is shown. In one possible implementation, the execution entity of the speech emotion recognition model training method can be a speech emotion recognition model training device. For example, the speech emotion recognition model training method can be executed by a terminal device, a server, or other electronic devices. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device, etc. In some possible implementations, the speech emotion recognition model training method can be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the training method of the speech emotion recognition model includes steps S11 to S15.

[0088] In step S11, a pre-trained speech emotion recognition model is obtained, wherein the speech emotion recognition model includes a first module and a second module.

[0089] In step S12, a first voice sample set corresponding to the target user is obtained, wherein the first voice sample set includes multiple first voice samples input by the target user, and multiple emotion classification labels corresponding to the multiple first voice samples.

[0090] In step S13, for any first speech sample in the first speech sample set, the speech features corresponding to the first speech sample are obtained.

[0091] In step S14, the speech features corresponding to the first speech sample are input into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0092] In step S15, the parameters of the second module are updated based on the emotion classification prediction result and the emotion classification label corresponding to the first speech sample.

[0093] In this embodiment of the disclosure, the pre-trained speech emotion recognition model can be a general speech emotion recognition model. For any target user, the pre-trained speech emotion recognition model can be fine-tuned to obtain a speech emotion recognition model specific to that target user.

[0094] The speech emotion recognition model can employ network structures such as CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory), and Transformer, without limitation. The speech emotion recognition model comprises a first module and a second module. In one possible implementation, the first module may include at least one sub-module, and the second module may include at least one sub-module, with the network structures of the sub-modules in the first and second modules potentially being identical. The number of sub-modules in the first and second modules may be the same or different. In another possible implementation, the network structures of the sub-modules in the first and second modules may also differ.

[0095] In one example, the speech emotion recognition model can be denoted as B1, the first module in the speech emotion recognition model can be denoted as M1, the second module in the speech emotion recognition model can be denoted as M2, and the speech features output by the first module can be denoted as N. The speech features output by the first module can be high-dimensional features.

[0096] Figure 2 A schematic diagram of the speech emotion recognition model provided in an embodiment of this disclosure is shown. Figure 2 As shown, the speech emotion recognition model includes a first module M1 and a second module M2. The speech emotion recognition model can take audio of a person speaking as input. The first module M1 outputs representation N, and the second module M2 outputs the emotion classification prediction result. Specifically, during training, the first module M1 models the speech to a high-dimensional representation N, while the second module M2 models the high-dimensional representation N to a specific emotion classification (e.g., happy, angry, calm).

[0097] In one possible implementation, the method further includes: obtaining a trained speaker recognition model; the step of obtaining the speech features corresponding to any first speech sample in the first speech sample set includes: obtaining the speech features corresponding to any first speech sample in the first speech sample set through the first module and / or the speaker recognition model.

[0098] In this implementation, the speaker recognition model can be used to output the speaker's corresponding speech features (e.g., voiceprint features). In one example, the speaker recognition model can be denoted as M3, and the speech features output by the speaker recognition model can be denoted as S. As an example of this implementation, the speech features output by the speaker recognition model have the same dimension as the speech features output by the first module in the speech emotion recognition model.

[0099] The speaker recognition model can be a neural network-based model or a Markov sequence-based model, etc., and there are no restrictions here.

[0100] As an example of this implementation, for any first speech sample in the first speech sample set, the speech features corresponding to the first speech sample can be obtained through the first module. Fine-tuning the second module based on the speech features obtained from the first module helps improve the fine-tuning effect of the speech emotion recognition model and enhances the accuracy of speech emotion recognition.

[0101] As another example of this implementation, for any first speech sample in the first speech sample set, the speech features corresponding to the first speech sample can be obtained through the speaker recognition model. The speaker recognition model can be deployed on the user terminal, thereby enabling the acquisition of speech features corresponding to speech samples in the first speech sample set without uploading the speech samples in the first speech sample set to the cloud, thus allowing for fine-tuning of the second model. According to this example, it is possible to determine whether to upload the first speech sample set to the cloud based on user needs, thereby meeting the user's data transmission requirements.

[0102] As another example of this implementation, for any first speech sample in the first speech sample set, the speech features corresponding to the first speech sample can be obtained through the first module and the speaker recognition model. By fine-tuning the second module based on the speech features corresponding to the first speech sample obtained by the first module and the speech features corresponding to the first speech sample obtained by the speaker recognition model, the fine-tuning effect of the speech emotion recognition model can be further improved.

[0103] In one possible implementation, a third speech sample set can be used to train the speaker recognition model until a second preset training condition is met. The third speech sample set may include multiple third speech samples corresponding to multiple speakers, and multiple speaker labels corresponding to the multiple third speech samples. The second preset training condition may be any one of the following: a preset convergence condition, a preset number of iterations, a preset number of epochs, etc.

[0104] In one possible implementation, obtaining the pre-trained speech emotion recognition model includes: acquiring a second speech sample set, wherein the second speech sample set includes multiple second speech samples corresponding to multiple speakers, and multiple emotion classification labels corresponding to the multiple second speech samples; training the speech emotion recognition model using the second speech sample set until a first preset training condition is met; for any second speech sample in the second speech sample set, obtaining the speech features corresponding to the second speech sample through the first module and / or the speaker recognition model; inputting the speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample; and updating the parameters of the second module according to the emotion classification prediction result corresponding to the second speech sample and the emotion classification labels corresponding to the second speech sample.

[0105] In this implementation, the second speech sample set can represent the speech sample set used to train a general speech emotion recognition model. The second speech sample can represent a speech sample within the second speech sample set. The second speech sample set can include a large number of second speech samples, and can include second speech samples corresponding to multiple speakers; for example, the second speech sample set can include second speech samples corresponding to a large number of speakers. That is, different speech samples in the second speech sample set can include the speech of different speakers, thereby improving the generality of the trained speech emotion recognition model.

[0106] In this implementation, the speech emotion recognition model can be trained using a second speech sample set until a first preset training condition is met. This first preset training condition can be any one of the following: a preset convergence condition, a preset number of iterations, or a preset number of periods.

[0107] In one example, a speech emotion recognition model trained to meet the first preset training conditions can be called a basic speech emotion recognition model.

[0108] In this implementation, after the speech emotion recognition model is trained to meet the first preset training conditions (i.e., after the basic speech emotion recognition model is trained), a speaker recognition model is introduced. The speaker recognition model can be used to extract relevant features of the speaker (e.g., voiceprint features).

[0109] In this implementation, a second speech sample set is acquired, comprising multiple second speech samples corresponding to multiple speakers and multiple emotion classification labels corresponding to the multiple second speech samples. The speech emotion recognition model is trained using the second speech sample set until a first preset training condition is met. For any second speech sample in the second speech sample set, the speech features corresponding to the second speech sample are obtained through the first module and / or the speaker recognition model. The speech features corresponding to the second speech sample are input into the second module to obtain the emotion classification prediction result corresponding to the second speech sample. The parameters of the second module are updated based on the emotion classification prediction result corresponding to the second speech sample and the emotion classification label corresponding to the second speech sample. Thus, after the speech emotion recognition model is trained to meet the first preset training condition, a speaker recognition model is introduced, thereby injecting relevant features of the speaker (e.g., voiceprint features) into the training of the speech emotion recognition model, enabling the speech emotion recognition model to learn the ability to model speaker features (e.g., voiceprint features) into emotion classification.

[0110] As an example of this implementation, obtaining the speech features corresponding to any second speech sample in the second speech sample set through the first module and / or the speaker recognition model includes: obtaining a first speech feature corresponding to the second speech sample through the first module and obtaining a second speech feature corresponding to the second speech sample through the speaker recognition model; obtaining a third speech feature corresponding to the second speech sample based on the first speech feature and the second speech feature corresponding to the second speech sample; and inputting the speech feature corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample includes: inputting the third speech feature corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample.

[0111] In one example, obtaining the third speech feature corresponding to the second speech sample based on the first speech feature and the second speech feature corresponding to the second speech sample includes: performing average pooling on the first speech feature and the second speech feature corresponding to the second speech sample to obtain the third speech feature corresponding to the second speech sample. For example, if the first speech feature corresponding to the second speech sample is denoted as N and the second speech feature corresponding to the second speech sample is denoted as S, then by performing average pooling on the first speech feature N and the second speech feature S, the third speech feature corresponding to the second speech sample can be obtained as (N+S) / 2. In this example, the first speech feature, the second speech feature, and the third speech feature corresponding to the second speech sample have the same dimension, and the value of any dimension of the third speech feature corresponding to the second speech sample is equal to the average value of the corresponding dimension of the first speech feature and the second speech feature.

[0112] In another example, obtaining the third speech feature corresponding to the second speech sample based on the first speech feature and the second speech feature corresponding to the second speech sample includes: performing max pooling on the first speech feature and the second speech feature corresponding to the second speech sample to obtain the third speech feature corresponding to the second speech sample. In this example, the first speech feature, the second speech feature, and the third speech feature corresponding to the second speech sample have the same dimension, and the value of any dimension of the third speech feature corresponding to the second speech sample is the maximum value of the corresponding dimension of the first speech feature and the second speech feature corresponding to the second speech sample.

[0113] In another example, obtaining the third speech feature corresponding to the second speech sample based on the first speech feature and the second speech feature corresponding to the second speech sample includes: performing minimum pooling on the first speech feature and the second speech feature corresponding to the second speech sample to obtain the third speech feature corresponding to the second speech sample. In this example, the first speech feature, the second speech feature, and the third speech feature corresponding to the second speech sample have the same dimension, and the value of any dimension of the third speech feature corresponding to the second speech sample is the minimum value of the corresponding dimension of the first speech feature and the second speech feature corresponding to the second speech sample.

[0114] In this example, for any second speech sample in the second speech sample set, the first module obtains the first speech feature corresponding to the second speech sample, the speaker recognition model obtains the second speech feature corresponding to the second speech sample, and based on the first and second speech features corresponding to the second speech sample, the third speech feature corresponding to the second speech sample is obtained. The third speech feature corresponding to the second speech sample is then input into the second module to obtain the emotion classification prediction result corresponding to the second speech sample. This allows for data augmentation and expansion of the limited second speech sample set, thereby improving the training effect of the speech emotion recognition model.

[0115] As an example of this implementation, for any second speech sample in the second speech sample set, a first speech feature corresponding to the second speech sample can be obtained through the first module of the speech emotion recognition model, and a second speech feature corresponding to the second speech sample can be obtained through the speaker recognition model. A third speech feature corresponding to the second speech sample can be obtained based on the first and second speech features. The first, second, or third speech features corresponding to the second speech sample can be randomly input into the second module of the speech emotion recognition model to obtain the emotion classification prediction result corresponding to the second speech sample.

[0116] Figure 3 This diagram illustrates a method for training a speech emotion recognition model according to an embodiment of the present disclosure. After training the speech emotion recognition model to a first preset training condition, a speaker recognition model is introduced to continue training the speech emotion recognition model. Figure 3 As shown, the second speech sample can be extracted through the first module M1 in the speech emotion recognition model (i.e., Figure 3 The first speech feature N is obtained from the audio sample, and the second speech feature S is extracted from the second speech sample using the speaker recognition model M3. The first speech feature N and the second speech feature S can be averaged to obtain the third speech feature (N+S) / 2. The first speech feature N, the second speech feature S, or the third speech feature (N+S) / 2 can be randomly input into the second module of the speech emotion recognition model to obtain the emotion classification prediction result corresponding to the second speech sample (i.e.,...). Figure 3 The second module is trained by analyzing the emotional expression in the voice.

[0117] In another possible implementation, obtaining the pre-trained speech emotion recognition model includes: acquiring a second speech sample set, wherein the second speech sample set includes multiple second speech samples corresponding to multiple speakers, and multiple emotion classification labels corresponding to the multiple second speech samples; for any second speech sample in the second speech sample set, obtaining the speech features corresponding to the second speech sample through the first module and / or the speaker recognition model; inputting the speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample; and updating the parameters of the second module based on the emotion classification prediction result corresponding to the second speech sample and the emotion classification label corresponding to the second speech sample. In this implementation, the speaker recognition model can be directly introduced when pre-training the speech emotion recognition model, without waiting for the speech emotion recognition model to be trained to meet the first preset training conditions before introducing the speaker recognition model.

[0118] In this embodiment of the disclosure, the first voice sample may represent a voice sample input by the target user, and the first voice sample set may represent a voice sample set composed of multiple first voice samples. In this embodiment of the disclosure, the first voice sample set may include a small number of first voice samples input by the target user, that is, the first voice sample set may include only a small number of voice samples input by a single user. The first voice sample set may include first voice samples corresponding to at least one of a plurality of preset emotion categories.

[0119] In one possible implementation, the first speech sample set may include first speech samples corresponding to various preset emotion categories. For example, the preset emotion categories include happiness, anger, and calmness, and the first speech sample set includes first speech samples corresponding to happiness, anger, and calmness.

[0120] In another possible implementation, the first voice sample set may include first voice samples corresponding to some of a set of preset emotion categories. For example, the preset emotion categories include happiness, anger, and calmness, and the first voice sample set includes first voice samples corresponding to happiness and first voice samples corresponding to anger.

[0121] In one possible implementation, the speaker recognition model is a lightweight model.

[0122] For example, speaker recognition models can be based on Markov sequences.

[0123] In this implementation, by employing a lightweight speaker recognition model, the deployment cost of the speaker recognition model on the user terminal can be reduced.

[0124] As an example of this implementation, the speaker recognition model is deployed at least on the user terminal corresponding to the target user; obtaining the speech features corresponding to the first speech sample through the first module and / or the speaker recognition model includes: obtaining the second speech features corresponding to the first speech sample through the speaker recognition model deployed on the user terminal corresponding to the target user; inputting the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample includes: inputting the second speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0125] In this example, by deploying a speaker recognition model on the user terminal corresponding to the target user, the second speech feature corresponding to the first speech sample is obtained, and the second speech feature corresponding to the first speech sample is input into the second module to obtain the emotion classification prediction result corresponding to the first speech sample. Thus, it is not necessary to upload the first speech sample in the first speech sample set to the server, thereby enabling the speech emotion recognition model to adapt to the speaker (i.e. the target user) while meeting the data transmission needs of the target user.

[0126] It should be noted that all data in this disclosure embodiment is used with the user's authorization.

[0127] In one example, obtaining the second speech feature corresponding to the first speech sample by means of a speaker recognition model deployed on the user terminal corresponding to the target user includes: in response to the target user's request to retain the first speech sample set on the user terminal, obtaining the second speech feature corresponding to the first speech sample by means of a speaker recognition model deployed on the user terminal corresponding to the target user.

[0128] In this example, the decision to upload the first set of voice samples can be made based on the target user's choice, thus balancing user needs with the training effect of the voice emotion recognition model.

[0129] In one possible implementation, the same speaker recognition model can be deployed in the cloud and on the user terminal. For example, a Markov sequence-based model can be deployed in both the cloud and the user terminal.

[0130] In another possible implementation, different speaker recognition models can be deployed in the cloud and on the user terminal. For example, a neural network-based model can be deployed in the cloud, while a Markov sequence-based model can be deployed on the user terminal.

[0131] In one possible implementation, obtaining the speech features corresponding to the first speech sample through the first module and / or the speaker recognition model includes: in response to the target user agreeing to upload the first speech sample set, obtaining the first speech features corresponding to the first speech sample through the first module; and inputting the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample includes: inputting the first speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0132] In this implementation, the second module can be fine-tuned based on the first speech features corresponding to the first speech sample obtained by the first module and the second speech features corresponding to the first speech sample obtained by the speaker recognition model, thereby improving the fine-tuning effect of the speech emotion recognition model.

[0133] In one possible implementation, if the target user agrees to upload a first set of speech samples, for any first speech sample in the first set, a first speech feature corresponding to the first speech sample can be obtained through a first module in the speech emotion recognition model, and a second speech feature corresponding to the first speech sample can be obtained through a speaker recognition model. A third speech feature corresponding to the first speech sample can be obtained based on the first and second speech features. The first, second, or third speech features corresponding to the first speech sample can be randomly input into a second module of the speech emotion recognition model to obtain the emotion classification prediction result corresponding to the first speech sample.

[0134] In one example, obtaining the third speech feature corresponding to the first speech sample based on the first speech feature and the second speech feature corresponding to the first speech sample includes: performing average pooling on the first speech feature and the second speech feature corresponding to the first speech sample to obtain the third speech feature corresponding to the first speech sample. For example, if the first speech feature corresponding to the first speech sample is denoted as N and the second speech feature corresponding to the first speech sample is denoted as S, then by performing average pooling on the first speech feature N and the second speech feature S, the third speech feature corresponding to the first speech sample can be obtained as (N+S) / 2. In this example, the first speech feature, the second speech feature, and the third speech feature corresponding to the first speech sample have the same dimension, and the value of any dimension of the third speech feature corresponding to the first speech sample is equal to the average value of the corresponding dimension of the first speech feature and the second speech feature.

[0135] In another example, obtaining the third speech feature corresponding to the first speech sample based on the first speech feature and the second speech feature corresponding to the first speech sample includes: performing max pooling on the first speech feature and the second speech feature corresponding to the first speech sample to obtain the third speech feature corresponding to the first speech sample. In this example, the first speech feature, the second speech feature, and the third speech feature corresponding to the first speech sample have the same dimension, and the value of any dimension of the third speech feature corresponding to the first speech sample is the maximum value of the corresponding dimension of the first speech feature and the second speech feature corresponding to the first speech sample.

[0136] In another example, obtaining the third speech feature corresponding to the first speech sample based on the first speech feature and the second speech feature corresponding to the first speech sample includes: performing minimum pooling on the first speech feature and the second speech feature corresponding to the first speech sample to obtain the third speech feature corresponding to the first speech sample. In this example, the first speech feature, the second speech feature, and the third speech feature corresponding to the first speech sample have the same dimension, and the value of any dimension of the third speech feature corresponding to the first speech sample is the minimum value of the corresponding dimension of the first speech feature and the second speech feature corresponding to the first speech sample.

[0137] In this embodiment, by inputting the speech features corresponding to the first speech sample into the second module, a sentiment classification prediction result corresponding to the first speech sample can be obtained. Based on the sentiment classification prediction result and the sentiment classification label corresponding to the first speech sample, the value of the loss function corresponding to the first speech sample can be determined. Based on the value of the loss function corresponding to the first speech sample, the parameters of the second module can be updated.

[0138] The training method for the speech emotion recognition model provided in this disclosure can be applied to the fields of artificial intelligence, speech emotion recognition, voiceprint, etc., and is not limited thereto.

[0139] The training method of the speech emotion recognition model provided in this embodiment is illustrated below through a specific application scenario.

[0140] In this application scenario, a first voice sample set, a second voice sample set, and a third voice sample set can be obtained.

[0141] The second speech sample set may include multiple second speech samples corresponding to multiple speakers, and multiple emotion classification labels corresponding one-to-one with the multiple second speech samples. The second speech sample set can be used to train the speech emotion recognition model until a first preset training condition is met. The speech emotion recognition model trained to meet the first preset training condition can be called the basic speech emotion recognition model. This speech emotion recognition model can be denoted as B1.

[0142] The third speech sample set may include multiple third speech samples corresponding to multiple speakers, and multiple speaker labels corresponding one-to-one with the multiple third speech samples. The third speech sample set can be used to train a speaker recognition model until a second preset training condition is met. The speaker recognition model can be denoted as M3.

[0143] After obtaining the basic speech emotion recognition model and speaker recognition model, the second module of the speech emotion recognition model can be trained based on the second speech sample set. The speech emotion recognition model includes a first module and a second module, where the first module can be denoted as M1 and the second module as M2.

[0144] For any second speech sample in the second speech sample set, the first speech feature N corresponding to the second speech sample can be obtained through the first module of the speech emotion recognition model, and the second speech feature S corresponding to the second speech sample can be obtained through the speaker recognition model. The average of the first speech feature N and the second speech feature S corresponding to the second speech sample can be calculated to obtain the third speech feature (N+S) / 2 corresponding to the second speech sample. The first speech feature N, the second speech feature S, or the third speech feature (N+S) / 2 corresponding to the second speech sample can be randomly input into the second module of the speech emotion recognition model to obtain the emotion classification prediction result corresponding to the second speech sample. The parameters of the second module can be updated based on the emotion classification prediction result and the emotion classification label corresponding to the second speech sample. By introducing the speaker recognition model after the speech emotion recognition model has been trained to meet the first preset training conditions, the speaker's voiceprint features are injected into the training of the speech emotion recognition model, thereby enabling the speech emotion recognition model to learn the ability to model the speaker's voiceprint features into emotion classification. Among them, the speaker recognition model is introduced to train the speech emotion recognition model, which can be trained to meet the third preset training condition to obtain a pre-trained speech emotion recognition model (i.e., a general speech emotion recognition model).

[0145] The first speech sample set may include multiple first speech samples input by the target user, and multiple emotion classification labels corresponding one-to-one with the multiple first speech samples. For any first speech sample in the first speech sample set, the speech features corresponding to the first speech sample can be obtained through the first module in the pre-trained speech emotion recognition model and / or the trained speaker recognition model.

[0146] For example, if the target user does not agree to upload the first voice sample set, a speaker recognition model deployed on the user terminal corresponding to the target user can be used to obtain the second voice feature corresponding to the first voice sample, and the second voice feature corresponding to the first voice sample can be input into the second module to obtain the emotion classification prediction result corresponding to the first voice sample.

[0147] For example, if the target user agrees to upload a first set of voice samples, for any first voice sample in the first set, the first voice feature corresponding to the first voice sample can be obtained through the first module of the voice emotion recognition model, and the second voice feature corresponding to the first voice sample can be obtained through the speaker recognition model. A third voice feature corresponding to the first voice sample can be obtained based on the first and second voice features. The first, second, or third voice features corresponding to the first voice sample can be randomly input into the second module of the voice emotion recognition model to obtain the emotion classification prediction result corresponding to the first voice sample.

[0148] Based on the sentiment classification prediction result and the sentiment classification label corresponding to the first speech sample, the value of the loss function corresponding to the first speech sample can be determined. The parameters of the second module can then be updated based on the value of the loss function corresponding to the first speech sample.

[0149] This disclosure also provides a speech emotion recognition method, comprising: acquiring a speech emotion recognition model trained by the training method of the speech emotion recognition model; inputting the speech to be processed into the trained speech emotion recognition model, and obtaining the emotion classification label corresponding to the speech to be processed through the trained speech emotion recognition model.

[0150] Figure 4 An exemplary flowchart illustrating the training and deployment of a speech emotion recognition model provided in embodiments of this disclosure is shown. Figure 4 As shown, a speech emotion model (i.e., a speech emotion recognition model) can be trained using speech emotion data (e.g., a second speech sample set) until the first preset training condition is met. After the speech emotion model is trained to meet the first preset training condition, emotion information can be extracted through the speech emotion model (e.g., extracting the first speech feature through the first module), and voiceprint information can be extracted through the speaker recognition model (e.g., extracting the second speech feature through the speaker recognition model). The emotion dataset can be expanded based on the emotion information extracted by the speech emotion model and the voiceprint information extracted by the speaker recognition model. For example, the first speech feature N and the second speech feature S of the same speech sample can be averaged to obtain the third speech feature (N+S) / 2 of that speech sample. The speech emotion model can be fine-tuned based on the first, second, and third speech features of a large number of speech samples until the third preset training condition is met. After fine-tuning the speech emotion model to meet the third preset training condition, the deployment phase can begin. The speech emotion model can be fine-tuned using actual user data to obtain a speaker-adaptive speech emotion model.

[0151] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0152] In addition, this disclosure also provides a training device for a speech emotion recognition model, a speech emotion recognition device, an electronic device, a computer-readable storage medium, and a computer program product. All of the above can be used to implement any of the speech emotion recognition model training methods or speech emotion recognition methods provided in this disclosure. The corresponding technical solutions and technical effects can be found in the relevant descriptions in the method section, and will not be repeated here.

[0153] Figure 5 A block diagram of a training apparatus for a speech emotion recognition model provided in an embodiment of this disclosure is shown. Figure 5 As shown, the training device for the speech emotion recognition model includes:

[0154] The first acquisition module 51 is used to acquire a pre-trained speech emotion recognition model, wherein the speech emotion recognition model includes a first module and a second module;

[0155] The first acquisition module 52 is used to acquire a first voice sample set corresponding to the target user, wherein the first voice sample set includes multiple first voice samples input by the target user, and multiple emotion classification labels corresponding to the multiple first voice samples;

[0156] The second obtaining module 53 is used to obtain the speech features corresponding to any first speech sample in the first speech sample set.

[0157] The third acquisition module 54 is used to input the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0158] The update module 55 is used to update the parameters of the second module based on the emotion classification prediction result and the emotion classification label corresponding to the first speech sample.

[0159] In one possible implementation,

[0160] The device further includes: obtaining a trained speaker recognition model;

[0161] The second obtaining module 53 is used to: for any first speech sample in the first speech sample set, obtain the speech features corresponding to the first speech sample through the first module and / or the speaker recognition model.

[0162] In one possible implementation, the first obtaining module 51 is used to:

[0163] Obtain a second speech sample set, wherein the second speech sample set includes multiple second speech samples corresponding to multiple speakers, and multiple emotion classification labels corresponding to the multiple second speech samples;

[0164] The speech emotion recognition model is trained using the second speech sample set until the first preset training condition is met.

[0165] For any second speech sample in the second speech sample set, the speech features corresponding to the second speech sample are obtained through the first module and / or the speaker recognition model;

[0166] Input the speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample;

[0167] The parameters of the second module are updated based on the sentiment classification prediction result corresponding to the second speech sample and the sentiment classification label corresponding to the second speech sample.

[0168] In one possible implementation, the first obtaining module 51 is used to:

[0169] For any second speech sample in the second speech sample set, the first speech feature corresponding to the second speech sample is obtained through the first module, and the second speech feature corresponding to the second speech sample is obtained through the speaker recognition model; based on the first speech feature corresponding to the second speech sample and the second speech feature corresponding to the second speech sample, the third speech feature corresponding to the second speech sample is obtained.

[0170] The third speech feature corresponding to the second speech sample is input into the second module to obtain the emotion classification prediction result corresponding to the second speech sample.

[0171] In one possible implementation, the speaker recognition model is a lightweight model.

[0172] In one possible implementation, the speaker recognition model is deployed at least on the user terminal corresponding to the target user;

[0173] The second obtaining module 53 is used to: obtain the second speech feature corresponding to the first speech sample by using a speaker recognition model deployed on the user terminal corresponding to the target user;

[0174] The third obtaining module 54 is used to: input the second speech feature corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0175] In one possible implementation, the second obtaining module 53 is used for:

[0176] In response to the target user's request to retain the first speech sample set on the user terminal, the second speech feature corresponding to the first speech sample is obtained by using a speaker recognition model deployed on the user terminal corresponding to the target user.

[0177] In one possible implementation,

[0178] The second obtaining module 53 is configured to: in response to the target user's consent to upload the first voice sample set, obtain the first voice feature corresponding to the first voice sample through the first module;

[0179] The third obtaining module 54 is used to: input the first speech feature corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

[0180] This disclosure also provides a voice emotion recognition device, the voice emotion recognition device comprising:

[0181] The second acquisition module is used to acquire the voice emotion recognition model trained by the training device of the voice emotion recognition model;

[0182] The speech emotion recognition module is used to input the speech to be processed into the trained speech emotion recognition model, and obtain the emotion classification label corresponding to the speech to be processed through the trained speech emotion recognition model.

[0183] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation and technical effects can be referred to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0184] This disclosure also provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium.

[0185] This disclosure also proposes a computer program including computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device executes the above-described method.

[0186] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, the processor in the electronic device executes the above-described method.

[0187] This disclosure also provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the executable instructions stored in the memory to perform the above-described method.

[0188] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0189] Figure 6 A block diagram of an electronic device 1900 provided according to an embodiment of this disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal. (Refer to...) Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0190] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (MacOS X) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0191] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0192] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0193] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0194] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0195] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0196] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0197] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0198] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0200] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0201] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0202] If the technical solution of this disclosure involves personal information, the product applying the technical solution of this disclosure has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this disclosure involves sensitive personal information, the product applying the technical solution of this disclosure has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to indicate that the user has entered the scope of personal information collection and that personal information will be collected. If the user voluntarily enters the collection scope, it is deemed to have consented to the collection of their personal information; or on the personal information processing device, with clear signs / information informing the user of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0203] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a speech emotion recognition model, characterized in that, include: A pre-trained speech emotion recognition model is obtained, wherein the speech emotion recognition model includes a first module and a second module, and the speech emotion recognition model for any target user is obtained by fine-tuning the pre-trained speech emotion recognition model; Obtain a first voice sample set corresponding to the target user, wherein the first voice sample set includes multiple first voice samples input by a single target user, and multiple emotion classification labels corresponding to the multiple first voice samples; For any first speech sample in the first speech sample set, obtain the speech features corresponding to the first speech sample; Input the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample; The parameters of the second module are updated based on the emotion classification prediction result and the emotion classification label corresponding to the first speech sample.

2. The method according to claim 1, characterized in that, The method further includes: obtaining a trained speaker recognition model; The step of obtaining the speech features corresponding to any first speech sample in the first speech sample set includes: obtaining the speech features corresponding to any first speech sample in the first speech sample set through the first module and / or the speaker recognition model.

3. The method according to claim 1 or 2, characterized in that, The process of obtaining the pre-trained speech emotion recognition model includes: Obtain a second speech sample set, wherein the second speech sample set includes multiple second speech samples corresponding to multiple speakers, and multiple emotion classification labels corresponding to the multiple second speech samples; The speech emotion recognition model is trained using the second speech sample set until the first preset training condition is met. For any second speech sample in the second speech sample set, the speech features corresponding to the second speech sample are obtained through the first module and / or the speaker recognition model; Input the speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample; The parameters of the second module are updated based on the sentiment classification prediction result corresponding to the second speech sample and the sentiment classification label corresponding to the second speech sample.

4. The method according to claim 3, characterized in that, For any second speech sample in the second speech sample set, obtaining the speech features corresponding to the second speech sample through the first module and / or the speaker recognition model includes: for any second speech sample in the second speech sample set, obtaining the first speech feature corresponding to the second speech sample through the first module, and obtaining the second speech feature corresponding to the second speech sample through the speaker recognition model; obtaining the third speech feature corresponding to the second speech sample based on the first speech feature and the second speech feature corresponding to the second speech sample. The step of inputting the speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample includes: inputting the third speech features corresponding to the second speech sample into the second module to obtain the emotion classification prediction result corresponding to the second speech sample.

5. The method according to claim 2, characterized in that, The speaker recognition model is a lightweight model.

6. The method according to claim 5, characterized in that, The speaker recognition model is deployed at least on the user terminal corresponding to the target user; The step of obtaining the speech features corresponding to the first speech sample through the first module and / or the speaker recognition model includes: obtaining the second speech features corresponding to the first speech sample through the speaker recognition model deployed on the user terminal corresponding to the target user; The step of inputting the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample includes: inputting the second speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

7. The method according to claim 6, characterized in that, The step of obtaining the second speech feature corresponding to the first speech sample through the speaker recognition model deployed on the user terminal corresponding to the target user includes: In response to the target user's request to retain the first speech sample set on the user terminal, the second speech feature corresponding to the first speech sample is obtained by using a speaker recognition model deployed on the user terminal corresponding to the target user.

8. The method according to claim 2, characterized in that, The step of obtaining the speech features corresponding to the first speech sample through the first module and / or the speaker recognition model includes: in response to the target user's consent to upload the first speech sample set, obtaining the first speech features corresponding to the first speech sample through the first module; The step of inputting the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample includes: inputting the first speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample.

9. A voice emotion recognition method, characterized in that, include: Obtain the voice emotion recognition model trained by the training method of any one of claims 1 to 8; The speech to be processed is input into the trained speech emotion recognition model, and the emotion classification label corresponding to the speech to be processed is obtained through the trained speech emotion recognition model.

10. A training device for a speech emotion recognition model, characterized in that, include: The first acquisition module is used to acquire a pre-trained speech emotion recognition model, wherein the speech emotion recognition model includes a first module and a second module, and the speech emotion recognition model for any target user is obtained by fine-tuning the pre-trained speech emotion recognition model; The first acquisition module is used to acquire a first voice sample set corresponding to a target user, wherein the first voice sample set includes multiple first voice samples input by a single target user, and multiple emotion classification labels corresponding to the multiple first voice samples; The second obtaining module is used to obtain the speech features corresponding to any first speech sample in the first speech sample set. The third acquisition module is used to input the speech features corresponding to the first speech sample into the second module to obtain the emotion classification prediction result corresponding to the first speech sample. The update module is used to update the parameters of the second module based on the sentiment classification prediction result and the sentiment classification label corresponding to the first speech sample.

11. A voice emotion recognition device, characterized in that, include: The second acquisition module is used to acquire the voice emotion recognition model trained by the training device of the voice emotion recognition model as described in claim 10. The speech emotion recognition module is used to input the speech to be processed into the trained speech emotion recognition model, and obtain the emotion classification label corresponding to the speech to be processed through the trained speech emotion recognition model.

12. An electronic device, characterized in that, include: One or more processors; Memory used to store executable instructions; The one or more processors are configured to invoke executable instructions stored in the memory to perform the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 9.

Images