Emotion recognition model determination method, emotion recognition method, and electronic device

Abstract

Provided in the present application are an emotion recognition model determination method, an emotion recognition method, an electronic device, and a computer storage medium. The emotion recognition model determination method comprises: acquiring a pre-trained large speech model; acquiring training data having frame-level emotion recognition labels; and performing fine-tuning training on the large speech model on the basis of the training data, so as to obtain an emotion recognition model. The present application can improve the emotion recognition accuracy of a trained emotion recognition model.

Description

Emotion recognition model determination method, emotion recognition method, and electronic device

[0001] The present application claims priority to the Chinese Patent Application No. 202410835406.5, filed on June 25, 2024, and entitled "Emotion recognition model determination method, emotion recognition method, and electronic device", which is incorporated by reference in its entirety.

TECHNICAL FIELD

[0002] The present application relates to the technical field of emotion recognition, in particular to an emotion recognition model determination method, an emotion recognition method, an electronic device, and a computer storage medium.

BACKGROUND

[0003] With the continuous development of science and technology, it has a deeper research significance to realize automatic recognition of voice data by using computer technology. Emotion recognition of voice data has become one of the research fields that researchers in the field pay more and more attention to. However, the accuracy of emotion recognition of voice data by traditional emotion recognition models is low, and how to improve the accuracy of emotion recognition models is still a problem to be solved by researchers in the field.

SUMMARY

[0004] The present application provides an emotion recognition model determination method, an emotion recognition method, an electronic device, and a computer storage medium, which can improve the emotion recognition accuracy of the trained emotion recognition model.

[0005] To solve the above technical problems, the present application provides an emotion recognition model determination method, which comprises: obtaining a pre-trained voice large model; obtaining training data with frame-level emotion recognition labels; and fine-tuning the voice large model based on the training data to obtain an emotion recognition model.

[0006] The step of obtaining training data with frame-level emotion recognition labels comprises: determining emotion recognition labels based on the application scenario of the emotion recognition model; and obtaining training data with frame-level emotion recognition labels.

[0007] The emotion recognition labels include calm and not calm.

[0008] Each training data comprises at least a plurality of voice units and an emotion recognition label corresponding to each voice unit.

[0009] The step of training the voice large model based on the training data to obtain an emotion recognition model comprises: inputting the training data into the voice large model to perform emotion recognition task training on the voice large model to obtain an emotion recognition model.

[0010] The step of training the speech large model based on the training data to obtain the emotion recognition model comprises: simultaneously performing an emotion recognition task and an auxiliary recognition task based on the training data to complete multi-task training and obtain the emotion recognition model.

[0011] The auxiliary recognition task comprises at least one of a speech recognition task and a speaker recognition task.

[0012] The step of training the speech large model based on the training data to obtain the emotion recognition model comprises: updating parameters of a convolutional neural network layer of the speech large model using the training data to obtain the emotion recognition model.

[0013] To solve the above technical problems, the present application further provides an emotion recognition method, comprising: obtaining an emotion recognition model by the emotion recognition model determination method; performing emotion recognition on speech data using the emotion recognition model; and outputting an emotion recognition label and an emotion recognition curve corresponding to the speech data.

[0014] To solve the above technical problems, the present application further provides an electronic device, comprising a memory and a processor, the memory storing computer program instructions, and the processor executing the computer program instructions to implement the emotion recognition model determination method and / or the emotion recognition method.

[0015] To solve the above technical problems, the present application further provides a computer storage medium, wherein program instructions are stored thereon, and the program instructions are executed by a processor to implement the emotion recognition model determination method and / or the emotion recognition method.

[0016] To solve the above technical problems, the present application further provides a computer program product, comprising computer program instructions, and the computer program instructions enable a computer to implement the emotion recognition model determination method and / or the emotion recognition method.

[0017] To solve the above technical problems, the present application further provides an emotion recognition model determination device, wherein the emotion recognition model determination device comprises a model obtaining module, a training data obtaining module and a model training module, the model obtaining module is configured to obtain a pre-trained speech large model, the training data obtaining module is configured to obtain training data with frame-level emotion recognition labels, and the model training module is configured to fine-tune the speech large model based on the training data to obtain an emotion recognition model.

[0018] The beneficial effects of the present application are: the present application can train a pre-trained voice large model using training data with frame-level emotion recognition labels to obtain an emotion recognition model, which can improve the training efficiency of the emotion recognition model, and the training data is training data with frame-level emotion recognition labels, which can make the granularity of the label marking of the training data accurate to the frame level, thereby improving the training accuracy of the voice large model, and further improving the emotion recognition accuracy of the trained emotion recognition model. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Among them:

[0021] FIG. 1 is a flowchart of a first embodiment of the emotion recognition model confirmation method of the present application;

[0022] FIG. 2 is a partial structure diagram of the training framework of one embodiment of the fine-tuning training of the present application;

[0023] FIG. 3 is a flowchart of a second embodiment of the emotion recognition model confirmation method of the present application;

[0024] FIG. 4 is a flowchart of a first embodiment of the emotion recognition method of the present application;

[0025] FIG. 5 is a schematic diagram of one embodiment of the emotion recognition curve and voice data of the present application;

[0026] FIG. 6 is a partial structure diagram of the training framework of the multi-task training of the present application;

[0027] FIG. 7 is a structure diagram of one embodiment of the computer storage medium of the present application.

DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures, techniques, etc. are presented for the purpose of explanation, not for the purpose of limitation, so that the embodiments of the present application can be thoroughly understood. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0029] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. It should be understood that when used in the specification and the appended claims, the term "include" indicates the presence of described features, integers, steps, operations, elements, and / or components, but does not exclude one or more other features, integers, steps, operations, elements, components, and / or sets thereof. It should also be understood that the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in the specification and the appended claims of this application, the singular forms "a", "an" and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should be further understood that the term "and / or" used in the specification and the appended claims of this application means any combination of one or more of the associated listed items and all possible combinations thereof.

[0030] As used in the specification and the appended claims of this application, the term "if" can be interpreted as "when" or "upon" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0031] It should be noted that when an element is fixed to another element, it includes fixing the element directly to the other element, or fixing the element to the other element through at least one other element in the middle. When one element is connected to another element, it includes connecting the element directly to the other element, or connecting the element to the other element through at least one other element in the middle.

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0033] The present application first proposes an emotion recognition model determination method, as shown in FIG. 1, which is a flowchart of the first embodiment of the emotion recognition model determination method of the present application. The emotion recognition model determination method specifically includes steps S11-S13.

[0034] Step S11: Obtain a pre-trained speech large model.

[0035] The pre-trained speech large model, such as WavLM, Hubert, Wav2Vec model, Wav2Vec2.0 model, etc., is a speech large model pre-trained based on large-scale data. The pre-trained speech large model processes speech data and is usually used for tasks such as automatic speech recognition, speech synthesis, sentiment analysis, etc. Such a model can convert speech signals into text or other forms of representation. The pre-trained speech large model usually contains multiple components, such as feature extraction, sequence modeling, and decoder, etc. The pre-trained speech large model is mainly applied to the field related to speech. The pre-trained speech large model can be fine-tuned for specific tasks to apply a speech large model that has been pre-trained on a large dataset to a new related task.

[0036] Step S12: Obtain training data with frame-level emotion recognition labels.

[0037] Specifically, the training data with frame-level emotion recognition labels means that the training data is labeled to a fine-grained level of frame level. For example, in a training data, each frame corresponds to an emotion recognition label. For another example, when labeling data to obtain training data, the data is labeled at time points, and the labeling points are accurate to the millisecond level, so that the training data with frame-level emotion recognition labels is obtained.

[0038] Optionally, each training data includes at least a plurality of speech units and an emotion recognition label corresponding to each speech unit.

[0039] In an application scenario, each speech unit includes 1 frame, 3 frames, 5 frames, 15 frames, or 20 frames, etc., each frame includes 1 ms, 2 ms, 5 ms, 15 ms, or 20 ms, etc., an emotion recognition label can be set for each speech unit, and each training data includes at least a plurality of speech units. This setting facilitates the enrichment of the data content of the training data and improves the subsequent training effect.

[0040] Optionally, in an application scenario, when obtaining the training data, the customer service reply audio in the actual business scenario can be used as the candidate training data, the candidate training data is preliminarily screened, the candidate training data with too short duration and / or too high speech recognition error rate is deleted, and the remaining candidate training data meeting the standard is labeled with frame-level emotion recognition labels to obtain the training data with frame-level emotion recognition labels.

[0041] Optionally, step S12 can also be implemented through step S12A and step S12B.

[0042] Step S12A: Determine the emotion recognition label based on the application scenario of the emotion recognition model.

[0043] In different application scenarios, the classification of emotions is different, and therefore the emotion recognition labels are different. For example, in the application scenario of customer service voice quality inspection, the emotion recognition labels can be determined as calm and uncalm; for example, in the application scenario of tourist evaluation, the emotion recognition labels can be determined based on specific activity content and evaluation dimensions, such as determining the emotion recognition labels as excited and calm, or other label types, etc. Based on the application scenario to determine the classification method of emotions, and then determine the emotion recognition labels, the scene applicability of the trained emotion recognition model can be improved, and the use experience can be improved.

[0044] In different application scenarios, the degree of refinement of emotion recognition is also different, and therefore the emotion recognition labels are different. For example, in the application scenario of customer service voice quality inspection, the emotion recognition labels can be set as two categories, such as calm and uncalm, to better meet the use requirements of the customer service voice quality inspection scene, and to reduce the complexity of data processing in the model training process; for example, in the application scenario of patient emotion recognition, the emotions can be classified more finely based on the recognition requirements, such as setting three or more emotion recognition labels, such as emotion recognition labels including calm, anger, anxiety, sadness, and happiness, for example, emotion recognition labels including anger, disgust, fear, happiness, sadness, and surprise. Based on the application scenario to determine the degree of refinement of emotion recognition, and then determine the emotion recognition labels, the scene applicability of the trained emotion recognition model can be improved, and the use experience can be improved.

[0045] Step S12B: Obtain training data with frame-level emotion recognition labels.

[0046] After the emotion recognition labels are determined, the training data can be further labeled based on the emotion recognition labels, and then training data with frame-level emotion recognition labels can be obtained.

[0047] Optionally, the emotion recognition labels include calm and uncalm.

[0048] In the traditional customer service automated voice quality inspection method, the emotion recognition labels in the emotion recognition module are usually derived from common emotion classification theories, such as emotion recognition labels including anger, disgust, fear, happiness, sadness, etc. However, this classification method uses conventional emotion classification methods, but does not make targeted adjustments for the customer service voice quality inspection scene, so in the customer service voice quality inspection scene, this classification method not only increases the complexity of data processing and emotion recognition in this scene, but also because the similar emotion recognition labels are less different, the evaluation results of different evaluation subjects (such as evaluators, etc.) for the same voice data to be inspected are prone to evaluation errors, so the evaluation consistency of the traditional method is poor.

[0049] Therefore, the emotion recognition label can be limited to calm and uncalm to better meet the use requirements of the customer service voice quality inspection scene, which can not only reduce the complexity of data processing and emotion recognition and improve the emotion recognition efficiency, but also make the processed data clearer and more intuitive and improve the evaluation consistency of the customer service voice quality inspection evaluation method.

[0050] Alternatively, the training data with the frame-level emotion recognition label can be manually labeled, or the labeling can be realized by other models, without limitation. In an application scenario, in a piece of voice data, for example, the period from the 5.661st second to the 9.833rd second is labeled as calm, and for example, the period from the 34.823rd second to the 40.723rd second is labeled as uncalm.

[0051] Step S13: Fine-tuning training of the voice large model based on the training data to obtain an emotion recognition model.

[0052] The pre-trained voice large model is trained by using the training data with the frame-level emotion recognition label, that is, the voice large model is fine-tuned by using the training data with the frame-level emotion recognition label to obtain an emotion recognition model.

[0053] Fine-tuning training is a machine learning technique that can apply a model that has been pre-trained on a large dataset to a new related task. In the fine-tuning process, most of the parameters of the pre-trained model remain unchanged, and only part of the model (usually the top layer) is updated and adjusted according to the data of the new task. For example, in practice, fine-tuning training usually includes the following steps: selecting a pre-trained model, freezing most of the layers of the pre-trained model, unlocking the last few layers or the output layer for training, and updating the relevant parameters of the unlocked layer to optimize the performance of the model on the new task. Fine-tuning training enables the model to learn from the knowledge learned in the pre-training phase and the specific training data to improve the training efficiency.

[0054] The embodiment can train the pre-trained voice large model by using the training data with the frame-level emotion recognition label to obtain an emotion recognition model, which can improve the training efficiency of the emotion recognition model, and the training data is the training data with the frame-level emotion recognition label, which can make the granularity of the label labeling of the training data accurate to the frame level, thereby improving the training accuracy of the voice large model and further improving the emotion recognition accuracy of the trained emotion recognition model.

[0055] Alternatively, step S13 can also be realized by inputting the training data into the voice large model to train the voice large model for an emotion recognition task to obtain an emotion recognition model.

[0056] The training data is input into the speech large model, and only the emotion recognition task is trained, so as to improve the training efficiency.

[0057] Optionally, the step S13 can also be implemented by the step of simultaneously performing the emotion recognition task and the auxiliary recognition task on the speech large model based on the training data, so as to complete the multi-task training and obtain the emotion recognition model.

[0058] The multi-task training can simultaneously optimize multiple targets in different tasks using the shared backbone network in the model, and the auxiliary information and cross regularization from different tasks can improve the model training effect.

[0059] The multi-task training can solve multiple tasks by sharing the parameters or feature representations of the model. For example, the model can share some common underlying features when processing different tasks, thereby reducing the need for a large amount of labeled data, improving the utilization efficiency of data, and reducing the consumption of training resources and time. The tasks in the training can enhance the learning effect by sharing data with other tasks, thereby helping to improve the performance of the model on all tasks. The model can capture more general information in the process of learning multiple tasks, which helps to improve the generalization ability of the model on new tasks, thereby improving the training effect of the model. Therefore, the multi-task training can improve the training efficiency and training effect of the model.

[0060] Optionally, the auxiliary recognition task includes at least one of a speech recognition task and a speaker recognition task.

[0061] In one application scenario, the emotion recognition task and the speech recognition task are simultaneously performed on the speech large model based on the training data, so as to complete the multi-task training and obtain the emotion recognition model. In another application scenario, the emotion recognition task and the speaker recognition task are simultaneously performed on the speech large model based on the training data, so as to complete the multi-task training and obtain the emotion recognition model. In one application scenario, as shown in FIG. 6, FIG. 6 is a partial structure diagram of the training framework of the multi-task training of the present application. The emotion recognition task, the speech recognition task, and the speaker recognition task are simultaneously performed on the speech large model based on the training data, so as to complete the multi-task training and obtain the emotion recognition model.

[0062] Optionally, in one application scenario, in the step of simultaneously performing the emotion recognition task, the speech recognition task, and the speaker recognition task on the speech large model based on the training data, so as to complete the multi-task training and obtain the emotion recognition model, the overall loss function L can be composed of three parts, and specifically satisfies the following formula 1-1:

[0063] L = a * LCE1 + β * LCTC + (1-a-β) * LCE2 …… 1-1

[0064] LCE1 is a loss function of the emotion recognition task, LCTC is a loss function of the speech recognition task, LCE2 is a loss function of the speaker recognition task, and a and β are two hyperparameters, and a, β, 1-a-β are all non-negative numbers. Optionally, the hyperparameters a and β can be determined by a grid search method.

[0065] a, β, 1-a-β are all non-negative numbers, which can reduce the search range of the model in the multi-task training process and improve the training efficiency of multi-task training.

[0066] a, β, 1-a-β are all non-negative numbers, which can reduce the search range of the model in the multi-task training process and improve the training efficiency of multi-task training.

[0067] Optionally, the speaker recognition task includes a speaker gender recognition task and / or a speaker age recognition task, and the specific implementation is not limited.

[0068] Optionally, the emotion recognition model obtained by training in the embodiment can only perform the emotion recognition task and not perform the speech recognition task and the auxiliary recognition task when performing emotion recognition, so as to improve the emotion recognition efficiency.

[0069] Optionally, step S13 can also be implemented by the step of updating the parameters of the convolutional neural network layer of the speech large model by using the training data to obtain the emotion recognition model.

[0070] Updating the parameters of the convolutional neural network layer of the speech large model means fine-tuning the convolutional neural network (CNN) layer of the speech large model. In one application scenario, the speech large model is trained for the emotion recognition task based on the training data with frame-level emotion recognition labels, the parameters of the convolutional neural network layer (i.e., the CNN layer) of the speech large model are updated to realize the training of the speech large model, and then the emotion recognition model is obtained. In another application scenario, the speech large model is trained for multi-task based on the training data, the parameters of the convolutional neural network layer (i.e., the CNN layer) of the speech large model are updated to realize the training of the speech large model, and then the emotion recognition model is obtained.

[0071] For example, as shown in FIG. 2, FIG. 2 is a partial structure diagram of a training framework of an embodiment of fine-tuning training of the present application. The training data is input into the speech large model (for example, the Wav2vec2.0 model), and the convolutional neural network layer is the trainable part; the embedding layer is followed by the classification layer; the frame-level prediction results made by the speech large model (for example, the Wav2vec2.0 model) can be obtained in the classification layer, the classification loss calculation is performed on the obtained prediction results and the frame-level emotion recognition labels in the training data to obtain the frame-level classification error, and then the convolutional neural network layer can be trained based on the classification loss calculation result, that is, the parameters of the convolutional neural network layer in FIG. 2 are updated based on the classification loss calculation result. The advantage of selecting the convolutional neural network layer for training is that local feature focusing can be achieved, and only a small amount of parameters need to be added in the training process, which can improve the training efficiency.

[0072] The convolutional neural network layer can process the original speech waveform data, extract local speech features (such as frequency features, etc.) through convolution operation, and convert the original speech signal into a series of compact feature representations.

[0073] The convolutional neural network layer of the pre-trained speech large model has learned general speech features on a large amount of unlabeled speech data. Fine-tuning training of the convolutional neural network layer can make the feature extraction more suitable for the needs of specific tasks, for example, it can better capture the details of the speech signal closely related to the downstream task (such as the emotion recognition task), so that more discriminative features can be extracted; the fine-tuning training stage only updates the parameters of the convolutional neural network layer, which can reduce the risk of overfitting of the model on limited labeled data; and updating the parameters of the convolutional neural network layer usually only requires less computing resources and time. Therefore, the above setting can improve the training efficiency and training effect, and thus improve the emotion recognition accuracy of the emotion recognition model obtained by training.

[0074] In other embodiments, other layers of the speech large model can also be trained to update the parameters, which are not limited in particular.

[0075] In an application scenario, step S13 can also be implemented through steps A1-A2.

[0076] Step A1: Extracting semantic expression and acoustic expression of the training data.

[0077] Step A2: Training the speech large model based on the semantic expression and the acoustic expression to obtain the emotion recognition model.

[0078] In an application scenario, the emotion recognition label is calm and uncalm. Obtain customer service reply audio in an actual business production environment as candidate training data, and preliminarily screen the candidate training data, delete the candidate training data with too short time length and / or too high speech recognition error rate, and then label the remaining candidate training data meeting the standard with frame-level emotion recognition labels (for example, label the start and end time when the emotion in the candidate training data is uncalm) to obtain training data with frame-level emotion recognition labels; use the Wav2vec2.0 model to extract semantic expression and acoustic expression of the training data, control the Wav2vec2.0 model to first perform multi-task training (for example, emotion recognition task, speech recognition task, and speaker recognition task) based on the semantic expression and the acoustic expression, and the total loss function L in the multi-task training satisfies formula 1-1; further, control the Wav2vec2.0 model to perform emotion recognition task training to update the parameters of the convolutional neural network layer (i.e., the CNN layer) of the Wav2vec2.0 model, to complete fine-tuning training and obtain an emotion recognition model.

[0079] Different from pure text information, the semantic expression refers to the meaning and information content conveyed through language, text, symbols, tone, or other communication methods. It focuses on the internal meaning of the information, that is, the ideas, feelings, intentions, or knowledge that the speaker or author wants to convey. The semantic expression and the acoustic expression can contain more rich emotion information, and extracting the semantic expression and the acoustic expression for task training can improve the accuracy of model training and improve the emotion recognition accuracy of the emotion recognition model obtained by training.

[0080] The multi-task training can refer to FIG. 6. The speech data is input into a speech large model (for example, Wav2Vec), the model extracts local speech features using a convolutional neural network layer (i.e., a CNN layer), and the local speech features are respectively input into an emotion recognition task (for example, a frame-level emotion recognition task) and a speech recognition task (i.e., an ASR task) and a speaker recognition task for task training, and then relevant calculations of respective corresponding loss functions are performed. The speaker recognition task can first perform pooling processing on the local speech features to obtain a sentence-level speaker recognition label (for example, the sentence-level recognition label in FIG. 6), and then perform relevant calculations of the corresponding loss function.

[0081] The application further provides an emotion recognition model determination method, as shown in FIG. 3, which is a flowchart of a second embodiment of the emotion recognition model determination method of the application. The emotion recognition model determination method specifically includes steps S31-S34.

[0082] Step S31: Obtain a pre-trained speech large model.

[0083] The specific implementation of step S31 can refer to step S11, which will not be described here.

[0084] Step S32: Obtain training data with frame-level emotion recognition labels.

[0085] The specific implementation of step S32 can refer to step S12, which will not be repeated here.

[0086] Step S33: Perform emotion recognition task and auxiliary recognition task on the speech large model based on the training data to complete multi-task training and obtain a second training model.

[0087] The specific implementation of multi-task training based on the training data can refer to the above embodiments, which will not be repeated here. In this embodiment, the model obtained by performing emotion recognition task and auxiliary recognition task on the speech large model based on the training data to complete multi-task training is taken as the second training model.

[0088] Step S34: Input the training data into the second training model to perform emotion recognition training on the second training model to obtain an emotion recognition model.

[0089] In this embodiment, the speech large model is first trained by at least emotion recognition task based on the training data, and the second training model obtained by training is further trained by emotion recognition task, which can further improve the training effect of the model. After improving the processing capability of the model for multiple tasks in multi-task training, the model is specifically trained by emotion recognition task, which can improve the accuracy of emotion recognition of the emotion recognition model obtained by training.

[0090] In other embodiments, the emotion recognition model determination method can be similarly improved, which will not be repeated here.

[0091] The present application further proposes an emotion recognition method, as shown in FIG. 4, which is a flowchart of the first embodiment of the emotion recognition method of the present application. The emotion recognition method specifically includes steps S41-S43.

[0092] Step S41: Obtain the emotion recognition model by the above emotion recognition model determination method.

[0093] Step S42: Perform emotion recognition on the speech data by using the emotion recognition model.

[0094] Step S43: Output the emotion recognition label and the emotion recognition curve corresponding to the speech data.

[0095] Optionally, the emotion recognition curve can be a curve of the probability value of a certain emotion recognition label changing with time axis, for example, when the emotion recognition label includes unrest, the emotion recognition curve corresponding to a certain speech data can include a curve of the probability value of the unrest label changing with time.

[0096] In an application scenario, voice data is input into the emotion recognition model, the emotion recognition model is used to process the voice data, and a frame-level emotion recognition label corresponding to the voice data is output.

[0097] Since the emotion recognition model is obtained by training a pre-trained voice large model using training data with frame-level emotion recognition labels, the emotion recognition task using the emotion recognition model can effectively improve the accuracy of emotion recognition, and the output emotion recognition curve can improve the visualization effect and improve the user experience of using the emotion recognition model.

[0098] Optionally, step S43 can also be implemented by outputting the frame-level emotion recognition label and the frame-level emotion recognition curve corresponding to the voice data. This method can effectively improve the granularity of the output result, facilitate further pooling processing of the output result, facilitate obtaining sentence-level, section-level, and other level emotion recognition results, and improve the flexibility of the emotion recognition model.

[0099] Optionally, as shown in FIG. 5, FIG. 5 is a schematic diagram of an embodiment of an emotion recognition curve and voice data according to the present application. In FIG. 5, the horizontal axis is the time axis, and the vertical axis corresponding to the emotion recognition curve is the probability value of the emotion label being not calm.

[0100] Optionally, step S43 can also be implemented by steps B1-B3:

[0101] Step B1: outputting an emotion recognition label probability score corresponding to a voice unit using a softmax function.

[0102] Specifically, the voice data includes at least one voice unit, each voice unit can include 1 frame, 3 frames, 5 frames, 15 frames, or 20 frames, etc., and each frame can include 1 ms, 2 ms, 5 ms, 15 ms, or 20 ms, etc. The softmax function is used to output an emotion recognition label probability score corresponding to a voice unit.

[0103] Step B2: obtaining an emotion recognition label corresponding to the voice unit based on a preset condition and the emotion recognition label probability score.

[0104] For example, in an application scenario, the probability of a certain voice unit being not calm is 80%, and the preset condition is that the probability of being not calm is greater than 50% to determine that it is not calm. Therefore, the emotion recognition label corresponding to the voice unit is not calm.

[0105] Step B3: obtaining an emotion recognition curve corresponding to the voice data based on a preset condition and the emotion recognition label probability score.

[0106] The order of steps B3 and B2 is not limited.

[0107] The emotion recognition curve is a curve of a certain emotion recognition label changing over time. For example, in an application scenario, each speech unit is 10 ms, and the emotion recognition model can output the probability score of the unrest corresponding to the speech unit. It can be known that the probability score of the unrest will change along with the time axis of the speech data, and thus the emotion recognition curve corresponding to the speech data can be obtained.

[0108] In an application scenario, the number of unrest labels appearing in a certain conversation, a certain day, or a certain month of a certain customer service personnel can also be counted by using the output emotion recognition label corresponding to the speech data, and thus the service quality of the customer service personnel can be evaluated by using the information.

[0109] The present application further proposes an emotion recognition model determination device, wherein the emotion recognition model determination device comprises a model acquisition module, a training data acquisition module, and a model training module. The model acquisition module is configured to acquire a pre-trained speech large model. The training data acquisition module is configured to acquire training data with frame-level emotion recognition labels. The model training module is configured to fine-tune the speech large model based on the training data to obtain an emotion recognition model.

[0110] The emotion recognition model determination device can also use the emotion recognition model determination method described in any of the above embodiments to determine the emotion recognition model.

[0111] The present application further proposes an emotion recognition device, which comprises an acquisition unit, an identification unit, and an output unit. The acquisition unit is configured to acquire an emotion recognition model by using the emotion recognition model determination method described in any of the above embodiments. For example, the emotion recognition model determination device described above can be used as the acquisition unit of the present embodiment. The identification unit is configured to use the emotion recognition model to perform emotion recognition on speech data. The output unit is configured to output emotion recognition labels and emotion recognition curves corresponding to the speech data.

[0112] The present application further proposes an electronic device, which comprises a memory and a processor. The memory stores computer program instructions. The processor executes the computer program instructions to implement the emotion recognition model determination method and / or the emotion recognition method described above.

[0113] The specific implementation and working principle of the emotion recognition model determination method can be referred to the above embodiments, which will not be described here.

[0114] The specific implementation and working principle of the emotion recognition method can be referred to the above embodiments, which will not be described here.

[0115] The application further provides a computer storage medium, as shown in FIG. 7, which is a structural schematic diagram of an embodiment of the computer storage medium of the application. The computer storage medium stores program instructions 61, which are executed by a processor to implement the emotion recognition model determination method and / or the emotion recognition method.

[0116] The program instructions 61 can form a program file and be stored in the storage medium in the form of a software product, so that an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor executes all or part of the steps of the various embodiments of the application. The storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage media that can store program codes, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0117] The computer storage medium 60 of the embodiment can be, but is not limited to, a U disk, an SD card, a PD optical drive, a mobile hard disk, a large-capacity floppy disk drive, a flash memory, a multimedia memory card, a server, etc.

[0118] In one embodiment, a computer program product or computer program is provided, which includes computer instructions stored in a computer storage medium. A processor of an electronic device reads the computer instructions from the computer storage medium, and the processor executes the computer instructions to enable the electronic device to perform the steps in the above-mentioned method embodiments.

[0119] In addition, when the above-mentioned functions are implemented in the form of software functions and sold or used as independent products, they can be stored in a mobile terminal readable storage medium. That is, the application also provides a storage device storing program data, which can be executed to implement the above-mentioned method embodiments. The storage device can be, for example, a U disk, an optical disk, a server, etc. That is, the application can be embodied in the form of a software product, which includes a plurality of instructions for enabling an intelligent terminal to execute all or part of the steps of the method described in the various embodiments.

[0120] Compared with the prior art, the application can train a pre-trained voice large model using training data with frame-level emotion recognition labels to obtain an emotion recognition model, which can improve the training efficiency of the emotion recognition model. The training data is training data with frame-level emotion recognition labels, which can make the granularity of the label marking of the training data accurate to the frame level, thereby improving the training accuracy of the voice large model and further improving the emotion recognition accuracy of the trained emotion recognition model.

[0121] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other manners. For example, the division of the apparatus embodiments is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, and there can be electric, mechanical or other forms.

[0122] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structures or equivalent flow variations of the present application, or direct or indirect applications in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for determining an emotion recognition model, wherein, The emotion recognition model determination method includes: Obtain a pre-trained large speech model; Obtain training data with frame-level emotion recognition labels; The speech model is fine-tuned based on the training data to obtain an emotion recognition model.

2. The emotion recognition model determination method according to claim 1, wherein, The step of obtaining training data with frame-level emotion recognition labels includes: Determining emotion recognition labels based on application scenarios of emotion recognition models; Obtain the training data with the emotion recognition labels at the frame level.

3. The emotion recognition model determination method according to claim 1 or 2, wherein, The emotion recognition labels include calm and uneasy.

4. The emotion recognition model determination method according to any one of claims 1 to 3, wherein, Each training data set includes at least multiple speech units and an emotion recognition label corresponding to each speech unit.

5. The emotion recognition model determination method according to any one of claims 1 to 4, wherein, The step of training the large speech model based on the training data to obtain an emotion recognition model includes: The training data is input into the speech model to train the speech model for an emotion recognition task, thereby obtaining an emotion recognition model.

6. The emotion recognition model determination method according to any one of claims 1 to 4, wherein, The step of training the large speech model based on the training data to obtain an emotion recognition model includes: Based on the training data, the speech model is simultaneously subjected to emotion recognition and auxiliary recognition tasks to complete multi-task training and obtain the emotion recognition model.

7. The emotion recognition model determination method according to claim 6, wherein, The auxiliary recognition task includes at least one of speech recognition task and speaker recognition task.

8. The emotion recognition model determination method according to any one of claims 1 to 4, wherein, The step of training the large speech model based on the training data to obtain the emotion recognition model includes: The parameters of the convolutional neural network layer of the large speech model are updated using the training data to obtain an emotion recognition model.

9. An emotion recognition method, wherein, include: An emotion recognition model is obtained by the emotion recognition model determination method according to any one of claims 1 to 8; The emotion recognition model is used to perform emotion recognition on the speech data; Output the emotion recognition label and the emotion recognition curve corresponding to the voice data.

10. An electronic device, wherein, The method includes a memory and a processor, the memory storing computer program instructions, and the processor executing the computer program instructions to implement the emotion recognition model determination method according to any one of claims 1 to 8 and / or the emotion recognition method according to claim 9.

11. A computer storage medium, wherein, It stores program instructions that are executed by a processor to implement the emotion recognition model determination method according to any one of claims 1 to 8 and / or the emotion recognition method according to claim 9.

12. A computer program product, wherein, It includes computer program instructions that cause a computer to implement the emotion recognition model determination method according to any one of claims 1 to 8 and / or the emotion recognition method according to claim 9.

13. An emotion recognition model determination device, wherein, The emotion recognition model determination device includes: A model acquisition module, which is used to acquire a pre-trained large speech model; The training data acquisition module is used to acquire training data with frame-level emotion recognition labels. The model training module is used to fine-tune the large speech model based on the training data to obtain an emotion recognition model.

Images