A speech emotion recognition method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of voice processing, in particular to a voice emotion recognition method and device, electronic equipment and a storage medium. The application inputs the obtained target voice to be recognized and target text corresponding to the target voice into a trained voice emotion calculation model, so that the main emotion, the accompanying emotion and the emotion intensity corresponding to the target voice can be obtained; wherein the voice emotion calculation model is obtained by training a plurality of sample voices, and sample main emotion labels, sample accompanying emotion labels and sample emotion intensity labels corresponding to each sample voice. In this way, the fine-grained emotion containing the main emotion and the accompanying emotion and the emotion intensity can be recognized from the target voice, and the emotion recognition effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech emotion recognition method, device, electronic device and storage medium. Background Art

[0002] Speech emotion recognition is a technology that identifies a user's emotional state from their speech. It is used to perceive and transmit emotions between people and between people and machines. It has urgent application needs in areas such as human-computer interaction, health monitoring, and virtual reality, and is an important branch of the field of artificial intelligence.

[0003] Currently, several technologies are available for speech emotion computing. However, in reality, emotional expression in speech is extremely complex, especially in film, television, animation, and gaming. Even a single sentence often contains multiple emotional transitions. Current mainstream speech emotion data annotation solutions and speech emotion computing technologies often overlook the emotional shifts within a sentence. As a result, these annotated data can only handle coarse-grained emotions and are difficult to apply to more nuanced emotion perception scenarios. Consequently, they cannot achieve effective emotion recognition results. Summary of the Invention

[0004] In view of this, the embodiments of the present application at least provide a speech emotion recognition method, device, electronic device and storage medium, which can identify fine-grained emotions and emotion intensity including main emotions and accompanying emotions from the target speech, thereby improving the emotion recognition effect.

[0005] This application mainly includes the following aspects:

[0006] In the first aspect, an embodiment of the present application provides a speech emotion recognition method, which includes: obtaining a target speech to be recognized, and determining a target text corresponding to the target speech; inputting the target speech and the target text into a trained speech emotion calculation model together to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech; wherein the speech emotion calculation model is trained using multiple sample speech, and a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to each sample speech.

[0007] In a possible embodiment, for any of the sample speech, a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to the sample speech are generated according to the following steps: the sample speech is recognized to obtain the sample text corresponding to the sample speech and the timestamps of each sub-speech sentence; based on the sample text corresponding to the sample speech and the timestamps corresponding to each sub-speech sentence, a visual initial annotation information file is generated; based on the sample speech, a predefined main emotion set, an accompanying emotion set and preset annotation rules, the initial annotation information file is annotated to generate a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to the sample speech.

[0008] In the second aspect, an embodiment of the present application also provides a speech emotion recognition device, which includes: an acquisition module for acquiring a target speech to be recognized and determining a target text corresponding to the target speech; an input module for inputting the target speech and the target text into a trained speech emotion calculation model to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech; wherein the speech emotion calculation model is trained using multiple sample speech and a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to each sample speech.

[0009] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the speech emotion recognition method described in the first aspect or any possible implementation of the first aspect.

[0010] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the speech emotion recognition method described in the first aspect or any possible implementation of the first aspect are executed.

[0011] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the speech emotion recognition method described in the above-mentioned first aspect or any possible implementation of the first aspect.

[0012] The speech emotion recognition method, device, electronic device and storage medium provided in the embodiments of the present application use a speech emotion calculation model obtained by training multiple sample speech and the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to each sample speech to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech. Compared with the existing technology that often ignores the emotional changes in a paragraph, resulting in the data labeled in this way can only process coarse-grained emotions and is difficult to apply to more delicate emotion perception scenarios. Therefore, it cannot achieve good emotion recognition effect. The present application can identify fine-grained emotions and emotion intensity including main emotion and accompanying emotion from the target speech, which can improve the emotion recognition effect.

[0013] Furthermore, the speech emotion recognition method provided in the embodiment of the present application, when training the initial multimodal speech emotion model, divides the obtained multiple sample speech into a first sample group and a second sample group, and masks the sample speech features or sample text features corresponding to each second speech in the second sample group. Further, the unmasked first sample group and the masked second sample group are used to train the initial multimodal speech emotion model to generate the speech emotion calculation model. In this way, by adopting a random masking strategy during the training process to dynamically erase text features or speech features, this strategy can effectively prevent the model from being biased to a single modal feature and promote the model to utilize dual-modal features to achieve better emotion calculation effects.

[0014] Furthermore, in the speech emotion recognition method provided in an embodiment of the present application, the speech emotion calculation model includes two attention networks. Text emotion features can be obtained through the text timing features and the first attention network, and speech emotion features can be obtained through the speech timing features and the second attention network. In this way, the corresponding attention networks can extract the most emotion-related features from the input speech features and text features, respectively, allowing the model to fully utilize the dual-modal emotion features for more accurate emotion prediction.

[0015] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flow chart of a method for speech emotion recognition provided by an embodiment of the present application is shown;

[0018] Figure 2 It shows a flow chart for labeling the real labels of the sample speech provided by the embodiment of the present application;

[0019] Figure 3 A schematic diagram of a marking completion page provided in an embodiment of the present application is shown;

[0020] Figure 4 A flowchart of another speech emotion recognition method provided by an embodiment of the present application is shown;

[0021] Figure 5 One of the functional module diagrams of a speech emotion recognition device provided in an embodiment of the present application is shown;

[0022] Figure 6 The second functional module diagram of a speech emotion recognition device provided in an embodiment of the present application is shown;

[0023] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0025] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0026] In order to enable those skilled in the art to use the contents of this application, the following implementation methods are provided in combination with the specific application scenario "speech processing". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0027] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario requiring speech processing. The embodiments of the present application are not limited to specific application scenarios. Any scheme using the speech emotion recognition method and device provided by the embodiments of the present application is within the scope of protection of this application.

[0028] It is worth noting that before this application was filed, some technologies had already been used for speech emotion computing. However, in reality, the emotional expression of speech is very complex, especially in the speech of movies, animation, games, etc. Even a single sentence often contains multiple emotional transitions. Based on the current mainstream speech emotion data annotation scheme and speech emotion computing technology, the emotional changes in a paragraph are often ignored. As a result, the data annotated in this way can only process coarse-grained emotions and is difficult to apply to more delicate emotion perception scenarios. Therefore, it cannot achieve good emotion recognition results.

[0029] In response to the above problems, the embodiment of the present application inputs the acquired target speech to be recognized and the target text corresponding to the target speech into a trained speech emotion calculation model, thereby obtaining the main emotion, accompanying emotion, and emotion intensity corresponding to the target speech; wherein the speech emotion calculation model is trained using multiple sample speech and the sample main emotion label, sample accompanying emotion label, and sample emotion intensity label corresponding to each sample speech. In this way, fine-grained emotions including the main emotion and accompanying emotion and emotion intensity can be identified from the target speech, which can improve the emotion recognition effect.

[0030] To facilitate understanding of the present application, the technical solutions provided in the present application are described in detail below in conjunction with specific embodiments.

[0031] Figure 1 This is a flow chart of a method for speech emotion recognition provided by an embodiment of the present application. Figure 1 As shown, the speech emotion recognition method provided in the embodiment of the present application includes the following steps:

[0032] S101: Acquire a target speech to be recognized, and determine a target text corresponding to the target speech.

[0033] S102: Input the target speech and the target text into a trained speech emotion calculation model to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech.

[0034] The voice emotion calculation model is obtained by training a plurality of sample voices and sample main emotion labels, sample accompanying emotion labels and sample emotion intensity labels corresponding to each sample voice.

[0035] It can be understood that voice emotion recognition can be applied to many fields, such as human-computer interaction, health monitoring, virtual reality and the like. For different fields, the requirement for the delicacy of voice emotion recognition will also be different. Based on the current mainstream voice emotion data labeling scheme and voice emotion calculation technology, only basic voice emotion labels are defined, and more delicate voice emotion characteristics are ignored. The recognition effect of processing coarse-grained emotion is still acceptable, so the data labeled based on this labeling scheme is limited in application scenarios, and it is also difficult to be applied to more delicate emotion perception scenarios. In actual situations, as the subject of perception, emotion often requires higher requirements for more delicate emotion perception ability.

[0036] In actual situations, the expression of voice emotion is very complex, especially the voice in films, animations and games. Even in a sentence, it often contains the conversion of multiple emotions. Therefore, how to apply voice emotion recognition to more delicate emotion perception scenarios to achieve better emotion recognition effect is a technical problem to be solved at present.

[0037] To solve the above problems, the commonly thought solution is to subdivide the coarse-grained emotion, that is, to divide each type of coarse-grained emotion into multiple fine-grained emotions. The coarse-grained emotion categories include positive emotion and negative emotion, and the fine-grained emotion is specifically divided under the coarse-grained emotion, such as happy, optimistic and happy, etc. corresponding to the fine-grained emotion of positive emotion. However, this solution is only based on the subdivision of a certain type of coarse-grained emotion, which can only show the fine-grained of positive emotion or negative emotion, but cannot reflect the change of emotion and the complexity of emotion, that is, it cannot achieve the comprehensiveness of emotion delicacy.

[0038] Based on this, the inventors continued to conduct further research on speech emotion recognition, and found that, for example, in the synthesis of character expression animations in game plots, that is, when the game plot voice data has been generated, how to use the voice data to synthesize character expressions, the more fine-grained the speech emotion recognition, the more vivid the character expressions can be synthesized based on the recognized emotions, that is, the finer the perception of speech emotions, the better. For example, if the main emotion is "happy" and the accompanying emotion is "approval", when synthesizing expressions, it is more vivid to focus on the "happy" state and modulate the "approval" expression on this basis. In some human-computer interaction scenarios, the feedback logic that the machine can make is completely different when only the basic emotion (also known as the main emotion) is recognized, and the experience given to the user is also completely different when the accompanying emotion in the main emotion state is also recognized. Based on the above findings, the inventors proposed a set of solutions for training a speech emotion computing model through the main emotion label, the accompanying emotion label and the emotion intensity label, and using the speech emotion computing model to realize the recognition of fine-grained emotion and emotion intensity of the target speech, which can significantly improve the delicacy of emotion recognition.

[0039] It should be noted that emotion can also be understood as mood. Primary emotion refers to the primary emotion expressed by a person, i.e., the fundamental or basic emotion. Primary emotions primarily include feelings such as joy, anger, sorrow, happiness, surprise, fear, and thought. Primary emotions include, for example, neutrality, happiness, anger, surprise, fear, and sadness. Primary emotions can occur independently. Accompanying emotions are usually other emotions that accompany the primary emotion and generally cannot occur independently. Accompanying emotions include doubt, disbelief, eagerness, excitement, determination, seriousness, sarcasm, sneer, arrogance, disdain, contemplation, nostalgia, drunkenness, aftertaste, helplessness, indulgence, arrogance, shyness, guilty conscience, pride, admiration, recognition, and expectation. The relationship between primary and accompanying emotions is generally not one of subordination, correspondence, or causality, but rather one of accompaniment. The "accompaniment relationship" here refers to the co-occurrence between the primary and accompanying emotions. That is, the relationship between the primary and accompanying emotions is non-causal and uncertain, but there is a dependency relationship between the two. Any accompanying emotion must be accompanied by at least one primary emotion and cannot appear alone. The same accompanying emotion can form an accompanying relationship with one or more primary emotions, and different accompanying emotions can form accompanying relationships with the same primary emotion. For example, if the primary emotions are "happy" or "angry," and the accompanying emotions are "excited," "shy," and "disappointed," the fine-grained emotions that a person may experience might be "happy with excitement," "happy with shyness," "angry with excitement," or "angry with disappointment."

[0040] The following is a detailed description of the above-mentioned solution (this application) proposed by the inventor. This application selects a multimodal model as a model for speech emotion recognition. Here, the multimodal model is more advanced because the multimodal model can accept a variety of different input methods (such as language, image, voice, video), and in some cases produce different output modes. For this application, the input of the multimodal model is speech and text. In this way, the two characteristics of text and speech can be fully learned, thereby improving the accuracy of speech emotion recognition.

[0041] For this application, we first obtain the target speech to be recognized, and then determine the target text corresponding to the target speech. Specifically, we can use a speech recognition tool to recognize the target speech and obtain the target text. Here, the speech recognition tool is a tool that converts speech into text, such as an automatic speech recognition tool (ASR).

[0042] It should be noted that a speech emotion computing model can be obtained by training the selected multimodal model, i.e., the initial multimodal speech emotion model. Then, the target speech and target text are input into the trained speech emotion computing model to obtain the main emotion, accompanying emotion, and emotion intensity corresponding to the target speech. In the model training stage, a large number of sample speech can be obtained, and labels can be added to each sample speech to obtain the sample main emotion label, sample accompanying emotion label, and sample emotion intensity label corresponding to each sample speech. Then, these sample speech and the sample main emotion label, sample accompanying emotion label, and sample emotion intensity label corresponding to each sample speech are used to train the initial multimodal speech emotion model to obtain the speech emotion computing model.

[0043] It should also be noted that even the same emotion can convey different feelings to people at different emotional intensities. Therefore, the intensity of speech emotion also plays an important role in various scenarios. To this end, this application uses a multi-task approach. The speech emotion calculation model also predicts the intensity of emotion, providing information on the intensity of emotion for downstream tasks. In other words, in addition to being able to perform more fine-grained emotion recognition, this application also adds the recognition of emotion intensity.

[0044] In addition, the recognition result for the target speech in this application can be a sentence-level recognition result, that is, if a target speech includes multiple sub-speech sentences, the main emotion, accompanying emotion and emotion intensity corresponding to the target speech are essentially the label sequence of the main emotion, the label sequence of the accompanying emotion and the label sequence of the emotion intensity, and each label sequence contains the labels corresponding to each sub-speech sentence of the target speech.

[0045] In an embodiment of the present application, by inputting the acquired target speech to be recognized and the target text corresponding to the target speech into a trained speech emotion calculation model, the main emotion, accompanying emotion, and emotion intensity corresponding to the target speech can be obtained; wherein the speech emotion calculation model is trained using multiple sample speech and the sample main emotion label, sample accompanying emotion label, and sample emotion intensity label corresponding to each sample speech. In this way, fine-grained emotion and emotion intensity including the main emotion and accompanying emotion can be identified from the target speech, which can improve the emotion recognition effect.

[0046] It is understandable that before training the initial multimodal speech emotion model, it is necessary to first obtain training data for training the initial multimodal speech emotion model. For this application, the training data includes a large number of sample speech and the real label corresponding to each sample speech. Here, the real label includes the sample main emotion label, the sample accompanying emotion label and the sample emotion intensity label. The real labels can be annotated one by one by the annotator, or they can be annotated by the machine according to preset labeling rules.

[0047] Next, for any sample speech from a large number of sample speech, the process of marking the true label of the sample speech is described, that is, the implementation process of obtaining the true label corresponding to the sample speech is described. Figure 2 The flowchart of labeling the real labels of the sample speech provided by the embodiment of the present application is shown as follows. Figure 2 As shown, the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to the sample speech are generated according to the following steps:

[0048] S201: Recognize the sample speech to obtain a sample text corresponding to the sample speech and a timestamp of each sub-speech sentence.

[0049] In a specific implementation, the sample speech can be first obtained, and then converted into sample text using a speech recognition tool. The sample speech is then divided into sub-speech sentences, and the timestamps of each sub-speech sentence are recorded. Here, the present application can significantly improve the annotation efficiency by introducing speech recognition tools such as ASR tools to extract timestamps and text from the sample speech to be annotated.

[0050] Furthermore, the implementation process of determining the timestamps of each sub-speech sentence corresponding to the sample speech in S201 is described below, including the following steps: identifying at least one silent segment from the sample speech, and determining the silence duration corresponding to each silent segment; segmenting the sample speech according to the silent segments whose silence duration is greater than or equal to the preset duration, and obtaining the sub-speech sentences corresponding to the sample speech and the timestamp of each sub-speech sentence.

[0051] Here, the preset duration is determined based on the duration of the segmentation between each sub-speech sentence commonly used by the crowd, including but not limited to the average value, median value, etc., for example, 150ms.

[0052] S202: Generate a visualized initial annotation information file based on the sample text corresponding to the sample speech and the timestamps corresponding to each sub-speech sentence.

[0053] In a specific implementation, the sample text corresponding to the sample speech and the timestamps corresponding to each sub-speech sentence can be saved to generate a visual initial annotation information file.

[0054] S203: Based on the sample speech, the predefined main emotion set, the accompanying emotion set and the preset annotation rules, the initial annotation information file is annotated to generate a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to the sample speech.

[0055] Here, the present application proposes to divide emotions into main emotions and accompanying emotions, and adds annotation of the emotional intensity of speech. Specifically, the main emotion set, accompanying emotion set and preset annotation rules can be pre-defined. The main emotion set is such as {neutral, happy, pleasant, angry, worried, surprised, fearful, sad}, and the accompanying emotion set is such as {doubt, disbelief, eager, excited, determined, serious, sarcastic, sneer, arrogant, disdainful, thoughtful, nostalgic, drunken, aftertaste, helpless, doting, arrogant, shy, guilty, proud, admiration, recognition, expectation}. The preset annotation rules can be to use characters, numbers, IDs, etc. instead of emotions in a specific emotion set as emotion labels. For example, character a2 is used to represent "happy" and character b1 is used to represent "doubt". In addition, the emotion intensity can be defined as 1 to 10. The emotion set definition method of the present application is more reasonable and accurate. The annotation data based on this fine-grained speech emotion definition can provide more fine-grained emotion support for downstream tasks.

[0056] In this way, the labeler or machine can directly use the sample speech, the pre-defined main emotion set, the accompanying emotion set and the preset labeling rules to label the visual initial labeling information file to obtain the sample main emotion label, the sample accompanying emotion label and the sample emotion intensity label corresponding to the sample speech. Figure 3 The diagram of the marking completion page provided in the embodiment of the present application is shown as follows: Figure 3As shown, the first grid shows the speech data of the sample speech, the second grid shows the frequency data of the sample speech, the third grid shows the sample main emotion label, sample accompanying emotion label, and sample emotion intensity label of each sub-speech sentence of the sample speech, and the fourth grid shows the sample text corresponding to the sample speech.

[0057] Specifically, the speech file of the sample speech and the initial annotation information file corresponding to the sample speech (the original file that has not been annotated) can be opened through the annotation tool. The way of opening the file for visual display through the annotation tool can improve the annotation efficiency. Furthermore, multiple annotators or machines can annotate on the visual initial annotation file in accordance with the annotation rules. In the annotation process, this application breaks the form of annotating with a single sentence as the annotation unit, that is, this application does not play the sub-speech sentence one by one, but first plays the entire sample speech, and then annotates each sub-speech sentence of the sample speech after listening to the entire sample speech. In this way, emotional perception can be performed from the overall to the local, and accurate emotions can be perceived more easily, making the annotation more reasonable and accurate.

[0058] Here, annotation tools such as Praat phonetics software, originally named doing phonetics by computer, usually referred to as Praat, is a cross-platform multi-functional phonetics professional software, mainly used for analysis, annotation, processing and synthesis of digital speech signals, and generating various speech graphs and text reports.

[0059] Furthermore, if the labeling is performed by labelers, the same sample speech can be labeled by multiple labelers or multiple different machines, which can further improve the accuracy of the labeling. Specifically, in step S203, the initial labeling information file is labeled based on the sample speech, the predefined main emotion set, the accompanying emotion set and the preset labeling rules, and the sample main emotion label, the sample accompanying emotion label and the sample emotion intensity label corresponding to the sample speech are generated, including: obtaining multiple labeling results obtained by labeling the initial labeling information file based on the sample speech, the predefined main emotion set, the accompanying emotion set and the preset labeling rules; screening out the target result with a consistency greater than a preset threshold from the multiple labeling results, and determining the sample main emotion label, the sample accompanying emotion label and the sample emotion intensity label corresponding to the sample speech based on the target result.

[0060] This approach, where multiple annotators or machines perform the labeling, selects the labels whose consistency exceeds a preset threshold as the final labels. This approach allows different annotators or machines to produce more consistent results, demonstrating the superiority of this labeling solution.

[0061] Furthermore, the implementation process of training the initial multimodal speech emotion model and generating the speech emotion computing model is described, including the following steps: obtaining the multiple sample speech and dividing the multiple sample speech into a first sample group and a second sample group; for any second speech in the second sample group, masking the sample speech features or sample text features corresponding to the second speech; using the first sample group, the masked second sample group, and the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to each sample speech to train the initial multimodal speech emotion model to generate the speech emotion computing model.

[0062] Here, considering that in the process of emotion recognition using voice and text dual modalities, due to the asymmetry of voice and text in transmitting emotional information, modal bias often occurs in the actual training process of the algorithm model, that is, it tends to learn more of one modality. In response to the above situation, this application adopts a random masking strategy during the model training process to dynamically erase the text features or voice features of a part of the sample speech. This strategy can effectively prevent the model from being biased to a single modal feature, promote the model to utilize dual-modal features, and allow the model to strengthen the information perception of the two modalities to achieve better emotion calculation effects, thereby obtaining more accurate recognition effects. Specifically, a large number of sample texts for model training are first obtained, and these sample texts are divided into two groups, a first sample group and a second sample group. No additional processing is performed on the first sample group, and a random masking strategy is implemented for the second sample group. That is, for any second speech in the second sample group, one of the speech features and sample text features corresponding to the second speech is masked. Further, the unmasked first sample group, the masked second sample group, and the true labels corresponding to each sample speech are used to train the initial multimodal speech emotion model to generate a speech emotion computing model. The speech emotion computing model trained in the above manner can fully learn the bimodal features. Therefore, when the speech emotion computing model trained in this manner is used to recognize the target speech, the emotion recognition accuracy can be improved.

[0063] Further, it is found through a large amount of training that the model is more inclined to obtain the sentiment label from the text features, that is, the model will be more inclined to use the text features for subsequent sentiment recognition. However, in fact, the speech features are more important for speech emotion recognition, because the same sentence expressed with different emotions has different emotions. Based on this, in order to further solve the modal bias problem, the present application can make the model fully utilize the features of the two modalities by masking the text features in a higher proportion than the speech features, so as to achieve better emotion calculation effect. Specifically, for any second speech in the second sample group, the sample speech features or sample text features corresponding to the second speech are masked, including: masking the sample speech features of the second speech with a first masking probability, and masking the sample text features of the second speech with a second masking probability; wherein the first masking probability is less than the second masking probability.

[0064] In an example, a set of values all of which are 1 is generated, and the mask has a 50% probability of remaining unchanged (using all features for subsequent emotion recognition), a 35% probability of changing the mask value of the index position corresponding to the text features to 0 (using only speech features), and a 15% probability of setting the mask value of the index corresponding to the speech features to 0 (using only text features).

[0065] Here, in the model training process, a series of feature extraction processes are performed on the sample speech and the sample text corresponding to the sample speech. The mask in the present application is not limited to being performed at any feature extraction link. Taking the text features as an example, the mask can be performed on the text timing features, text emotion features, etc. as long as the mask can be implemented on the sample text.

[0066] The application process of the speech emotion calculation model will be described below. The speech emotion calculation model includes a text feature extraction network, a first attention network, a speech feature extraction network, a second attention network, and an emotion classification network. Figure 4 A flowchart of another speech emotion recognition method provided by an embodiment of the present application is shown in FIG. 6. Figure 4 As shown in FIG. 6, the speech emotion recognition method includes the following steps.

[0067] S401: obtaining a target speech to be recognized, and determining a target text corresponding to the target speech.

[0068] S402: inputting the target text into the text feature extraction network to obtain text timing features, and inputting the target speech into the speech feature extraction network to obtain speech timing features.

[0069] Specifically, the target text is input into the text feature extraction network to obtain text timing features h 1:n ∈Rn×D , n is the number of subtexts, and D is the feature dimension. Here, the text feature extraction network can be any network that can extract temporal features of text, such as the BERT network (Bidirectional Encoder Representations from Transformer). The BERT network is a bidirectional encoder representation based on Transformer, that is, a bidirectional Transformer encoder. It is a pre-trained language representation model that can generate deep bidirectional language representations.

[0070] Specifically, for speech feature extraction, Mel spectrum features, MFCC, FBANK, etc. can be used to initially extract features. Then, for sub-speech of preset duration, spectrum features, such as Mel puzzle features, are extracted as input to the speech feature extraction network Ea. The speech feature extraction network obtains the encoded speech time sequence features a 1:t ∈R t×D , t is the number of sub-speech, and D is the feature dimension. The speech feature extraction network can be any network that can extract speech temporal features. The speech feature extraction network, such as a speech encoder, can adopt a transformer network structure.

[0071] S403: Obtaining text emotion features based on the text timing features and the first attention network, and obtaining speech emotion features based on the speech timing features and the second attention network.

[0072] After research, the inventors found that if an attention network is introduced during feature extraction to perform further feature extraction, the model can pay more attention to features in speech and text that are more relevant to emotions. The inventors further found that assigning independent attention networks to text and speech respectively, compared with fusing text features and speech features and then assigning attention networks, the speech emotion calculation model obtained has higher accuracy in speech emotion recognition. The reason is that the attention network pays more attention to one aspect of features (text or speech) than to two aspects of features. In this way, the model can fully utilize the dual-modal emotional features for more accurate emotion recognition.

[0073] Specifically, the text time series features output from the text feature extraction network are processed by the first attention network to obtain text emotion features, which can further characterize the emotion words involved in the text. The speech time series features output from the speech feature extraction network are processed by the second attention network to obtain speech emotion features, which can further characterize the emotion words involved in the speech. Here, the present application can extract the most emotion-related features from the input speech features and text features respectively through the corresponding attention networks, so that the model can make full use of the dual-modal emotion features to make more accurate emotion predictions.

[0074] The specific implementation process of obtaining text emotion features and speech emotion features is described below. That is, in step S403, obtaining text emotion features based on the text timing features and the first attention network, and obtaining speech emotion features based on the speech timing features and the second attention network include:

[0075] S4031: Input the text timing features into the first attention network to obtain a text attention vector, and input the speech timing features into the second attention network to obtain a speech attention vector; each element in the text attention vector is used to represent the importance of the corresponding frame text in expressing emotions in the current context; each element in the speech attention vector is used to represent the importance of the corresponding frame speech in expressing emotions in the current context.

[0076] Specifically, the text temporal feature h 1:n Input the first attention network to get the text attention vector m 1:n , text attention vector m 1:n It includes n elements, the text attention vector m 1:n Each element is used to represent the importance of the corresponding frame text in expressing emotions in the current context. Each element can be represented by a number between 0 and 1. The larger the value of the number, the more important the feature corresponding to the element is in speech emotion recognition. It can also be understood as a weight.

[0077] In addition, the speech time sequence feature a 1:t Input the second attention network to obtain the speech attention vector m 1:t , speech attention vector m 1:t It includes t elements, the speech attention vector m 1:t Each element is used to represent the importance of the corresponding frame speech in expressing emotions in the current context. Each element can be represented by a number between 0 and 1. The larger the value of the number, the more important the feature corresponding to the element is in speech emotion recognition. It can also be understood as a weight.

[0078] Furthermore, the first attention network includes a first fully connected layer, a second fully connected layer and an activation layer; the step S4031 of inputting the text temporal features into the first attention network to obtain a text attention vector includes: inputting the text temporal features into the first fully connected layer of the first attention network to obtain a first text dimension feature; inputting the first text dimension feature into the second fully connected layer of the first attention network to obtain a second text dimension vector; and inputting the second text dimension feature into the activation layer of the first attention network to obtain a text attention vector.

[0079] Here, after obtaining the text temporal features, the first attention network is used to obtain the self-attention of each frame feature. The specific implementation mechanism is to use two layers of fully connected layers to change the feature dimension of each frame to 1, and then use the activation function (Sigmoid) to limit the value to 0-1. This value determines the importance of each frame of text in expressing emotions in the current context, and then obtain the text attention vector m 1:n .

[0080] Furthermore, the second attention network includes a first fully connected layer, a second fully connected layer and an activation layer; the step S4031 of inputting the speech timing feature into the second attention network to obtain a speech attention vector includes: inputting the speech timing feature into the first fully connected layer of the second attention network to obtain a first speech dimension feature; inputting the first speech dimension feature into the second fully connected layer of the second attention network to obtain a second speech dimension vector; and inputting the second speech dimension feature into the activation layer of the second attention network to obtain a speech attention vector.

[0081] Here, after obtaining the speech time series features, the second attention network is used to obtain the self-attention of each frame feature. The specific implementation mechanism is to use two layers of fully connected layers to change the feature dimension of each frame to 1, and then use the activation function to limit the value to 0-1. This value determines the importance of each frame of speech in expressing emotions in the current context, and then obtain the speech attention vector m 1:t .

[0082] S4032: Determine the text emotion feature based on the text attention vector and the text timing feature, and determine the speech emotion feature based on the speech attention vector and the speech timing feature.

[0083] Specifically, the text attention vector m 1:n and text temporal features h 1:n Perform dot product operation to obtain text sentiment features, which are further refined features related to emotions. 1:t and speech temporal features a 1:tThe dot product operation is performed to obtain a speech emotion feature, which is a further refined feature related to emotion. In this way, the speech emotion feature and the text emotion feature with emotion representation can be obtained.

[0084] S404: obtaining a main emotion, a companion emotion and an emotion intensity corresponding to the target speech based on the text emotion feature, the speech emotion feature and the emotion classification network.

[0085] Here, the speech emotion calculation model in the application is a multi-task model, and the speech emotion calculation model includes an emotion classification network. By inputting the text emotion feature and the speech emotion feature into the emotion classification network, the main emotion, the companion emotion and the emotion intensity corresponding to the target speech can be identified.

[0086] Further, different network layers are used to implement emotion recognition and intensity recognition, and the emotion classification network includes an emotion recognition layer and an intensity recognition layer. The step S404 of obtaining the main emotion, the companion emotion and the emotion intensity corresponding to the target speech based on the text emotion feature, the speech emotion feature and the emotion classification network includes: splicing the text emotion feature and the speech emotion feature to obtain a spliced emotion feature; inputting the spliced emotion feature into the emotion recognition layer to obtain the main emotion and the companion emotion corresponding to each sub-speech sentence of the target speech, and inputting the spliced emotion feature into the intensity recognition layer to obtain the emotion intensity corresponding to each sub-speech sentence of the target speech. Thus, the identification of the main emotion, the companion emotion and the emotion intensity of the target speech is completed, the fine-grained emotion recognition of the target speech can be realized, the emotion intensity can be identified, and the emotion recognition effect can be improved.

[0087] It should be noted that the training data is obtained based on the above-mentioned label annotation method, and then the multi-task fine-grained speech emotion calculation model is learned through the training data. The speech emotion calculation model takes the input of two modalities of speech and text, predicts the main emotion and the companion emotion, and simultaneously predicts the emotion intensity of each sub-speech sentence in the target speech. In order to fully utilize the advantages of the respective modalities of speech and text, two self-attention networks are used to obtain the most relevant emotion features in the speech features and the text features, and a random mask strategy is introduced in the training process to enable the model to strengthen the information perception of the two modalities, thereby obtaining more accurate recognition effect.

[0088] Based on the same application concept, the application embodiment also provides a speech emotion recognition device corresponding to the speech emotion recognition method provided in the above-mentioned embodiment. Since the principle of solving problems in the device in the application embodiment is similar to that of the speech emotion recognition method in the above-mentioned embodiment of the application, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0089] like Figure 5 、 Figure 6 As shown, Figure 5 This is one of the functional module diagrams of a speech emotion recognition device 500 provided in an embodiment of the present application. Figure 6 This is a second functional module diagram of a speech emotion recognition device 500 provided in an embodiment of the present application.

[0090] like Figure 5 As shown, the speech emotion recognition device 500 includes: an acquisition module 510, used to acquire the target speech to be recognized and determine the target text corresponding to the target speech; an input module 520, used to input the target speech and the target text into a trained speech emotion calculation model to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech; wherein, the speech emotion calculation model is trained using multiple sample speech, and the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to each sample speech.

[0091] In some possible embodiments, such as Figure 6 As shown, the speech emotion recognition device 500 also includes a sample label generation module 530; the sample label generation module 530 is used to generate a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to any of the sample speech according to the following steps: recognizing the sample speech to obtain the sample text corresponding to the sample speech and the timestamps of each sub-speech sentence; generating a visual initial annotation information file based on the sample text corresponding to the sample speech and the timestamps corresponding to each sub-speech sentence; annotating the initial annotation information file based on the sample speech, a predefined main emotion set, an accompanying emotion set and a preset annotation rule to generate a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to the sample speech.

[0092] In some possible embodiments, such as Figure 6 As shown, the sample label generation module 530 is specifically used to determine the timestamps of each sub-speech sentence corresponding to the sample speech according to the following steps: identifying at least one silent segment from the sample speech, and determining the silence duration corresponding to each silent segment; segmenting the sample speech according to the silent segments with a silence duration greater than or equal to a preset duration, and obtaining the sub-speech sentences corresponding to the sample speech and the timestamp of each sub-speech sentence.

[0093] In some possible embodiments, such as Figure 6As shown, the sample label generation module 530 is specifically used to: obtain multiple annotation results obtained by annotating the initial annotation information file based on the sample speech, a pre-defined main emotion set, an accompanying emotion set and a preset annotation rule; filter out target results with a consistency degree greater than a preset threshold from the multiple annotation results, and determine the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to the sample speech based on the target result.

[0094] In some possible embodiments, such as Figure 6 As shown, the speech emotion recognition device 500 also includes a model training module 540; the model training module 540 is used to generate the speech emotion calculation model according to the following steps: obtaining the multiple sample speech, and dividing the multiple sample speech into a first sample group and a second sample group; for any second speech in the second sample group, masking the sample speech features or sample text features corresponding to the second speech; using the first sample group, the masked second sample group, and the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to each sample speech to train the initial multimodal speech emotion model to generate the speech emotion calculation model.

[0095] In some possible embodiments, such as Figure 6 As shown, the model training module 540 is specifically used to mask the sample speech features or sample text features corresponding to the second speech according to the following steps: masking the sample speech features of the second speech with a first masking probability, and masking the sample text features of the second speech with a second masking probability; wherein the first masking probability is less than the second masking probability.

[0096] In some possible embodiments, such as Figure 6 As shown, the speech emotion calculation model includes a text feature extraction network, a first attention network, a speech feature extraction network, a second attention network, and a sentiment classification network; the input module 520 includes: a first extraction unit 521, which is used to input the target text into the text feature extraction network to obtain text timing features, and input the target speech into the speech feature extraction network to obtain speech timing features; a second extraction unit 522, which is used to obtain text emotion features based on the text timing features and the first attention network, and to obtain speech emotion features based on the speech timing features and the second attention network; a determination unit 523, which is used to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech based on the text emotion features, the speech emotion features and the sentiment classification network.

[0097] In some possible embodiments, such as Figure 6As shown, the second extraction unit 522 is specifically used to: input the text timing feature into the first attention network to obtain a text attention vector, and input the speech timing feature into the second attention network to obtain a speech attention vector; each element in the text attention vector is used to represent the importance of the corresponding frame text in expressing emotions in the current context; each element in the speech attention vector is used to represent the importance of the corresponding frame speech in expressing emotions in the current context; determine the text emotion feature based on the text attention vector and the text timing feature, and determine the speech emotion feature based on the speech attention vector and the speech timing feature.

[0098] In some possible embodiments, such as Figure 6 As shown, the first attention network and the second attention network both include a first fully connected layer, a second fully connected layer and an activation layer; the second extraction unit 522 is specifically used to obtain a text attention vector and a speech attention vector according to the following steps: inputting the text timing feature into the first fully connected layer of the first attention network to obtain a first text dimension feature, and inputting the speech timing feature into the first fully connected layer of the second attention network to obtain a first speech dimension feature; inputting the first text dimension feature into the second fully connected layer of the first attention network to obtain a second text dimension vector, and inputting the first speech dimension feature into the second fully connected layer of the second attention network to obtain a second speech dimension vector; inputting the second text dimension feature into the activation layer of the first attention network to obtain a text attention vector, and inputting the second speech dimension feature into the activation layer of the second attention network to obtain a speech attention vector.

[0099] In some possible embodiments, such as Figure 6 As shown, the emotion classification network includes an emotion recognition layer and an intensity recognition layer; the determination unit 523 is used to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech according to the following steps: splicing the text emotion feature and the speech emotion feature to obtain a spliced ​​emotion feature; inputting the spliced ​​emotion feature into the emotion recognition layer to obtain the main emotion and accompanying emotion corresponding to each sub-speech sentence of the target speech, and inputting the spliced ​​emotion feature into the intensity recognition layer to obtain the emotion intensity corresponding to each sub-speech sentence of the target speech.

[0100] In an embodiment of the present application, the target speech to be recognized is acquired through the acquisition module 510, and the target text corresponding to the target speech is determined; the acquired target speech to be recognized and the target text corresponding to the target speech are input into the trained speech emotion calculation model through the input module 520, and the main emotion, accompanying emotion and emotion intensity corresponding to the target speech can be obtained; wherein, the speech emotion calculation model is trained using multiple sample speech, and the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to each sample speech. In this way, fine-grained emotions including the main emotion and accompanying emotion and the emotion intensity can be identified from the target speech, which can improve the emotion recognition effect.

[0101] Based on the same application concept, see Figure 7 As shown, it is a structural diagram of an electronic device 700 provided in an embodiment of the present application, including: a processor 710, a memory 720 and a bus 730, wherein the memory 720 stores machine-readable instructions executable by the processor 710. When the electronic device 700 is running, the processor 710 and the memory 720 communicate with each other through the bus 730, and the machine-readable instructions are executed by the processor 710 when running to perform the steps of the speech emotion recognition method as described in any of the above embodiments.

[0102] Specifically, when the machine-readable instructions are executed by the processor 710, the following processing can be performed: obtaining the target speech to be recognized and determining the target text corresponding to the target speech; inputting the target speech and the target text into the trained speech emotion calculation model together to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech.

[0103] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: the sample speech is recognized to obtain the sample text corresponding to the sample speech and the timestamps of each sub-speech sentence; based on the sample text corresponding to the sample speech and the timestamps corresponding to each sub-speech sentence, a visual initial annotation information file is generated; based on the sample speech, a predefined main emotion set, an accompanying emotion set and preset annotation rules, the initial annotation information file is annotated to generate a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to the sample speech.

[0104] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: identifying at least one silent segment from the sample speech and determining the silence duration corresponding to each silent segment; segmenting the sample speech according to the silent segments whose silence duration is greater than or equal to the preset duration, and obtaining the sub-speech sentences corresponding to the sample speech and the timestamp of each sub-speech sentence.

[0105] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: obtaining multiple annotation results obtained by annotating the initial annotation information file based on the sample speech, a predefined main emotion set, an accompanying emotion set and preset annotation rules; filtering out target results with a consistency degree greater than a preset threshold from the multiple annotation results, and determining the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to the sample speech based on the target result.

[0106] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: obtaining the multiple sample speech, and dividing the multiple sample speech into a first sample group and a second sample group; for any second speech in the second sample group, masking the sample speech features or sample text features corresponding to the second speech; using the first sample group, the masked second sample group, and the sample main emotion label, sample accompanying emotion label and sample emotion intensity label corresponding to each sample speech to train the initial multimodal speech emotion model to generate the speech emotion calculation model.

[0107] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: masking the sample speech features of the second speech with a first masking probability, and masking the sample text features of the second speech with a second masking probability; wherein, the first masking probability is less than the second masking probability.

[0108] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: the target text is input into the text feature extraction network to obtain text timing features, and the target speech is input into the speech feature extraction network to obtain speech timing features; text emotion features are obtained based on the text timing features and the first attention network, and speech emotion features are obtained based on the speech timing features and the second attention network; based on the text emotion features, the speech emotion features and the emotion classification network, the main emotion, accompanying emotion and emotion intensity corresponding to the target speech are obtained.

[0109] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: the text timing features are input into the first attention network to obtain a text attention vector, and the speech timing features are input into the second attention network to obtain a speech attention vector; each element in the text attention vector is used to represent the importance of the corresponding frame text in expressing emotions in the current context; each element in the speech attention vector is used to represent the importance of the corresponding frame speech in expressing emotions in the current context; the text emotion features are determined based on the text attention vector and the text timing features, and the speech emotion features are determined based on the speech attention vector and the speech timing features.

[0110] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: inputting the text timing feature into the first fully connected layer of the first attention network to obtain a first text dimension feature, and inputting the speech timing feature into the first fully connected layer of the second attention network to obtain a first speech dimension feature; inputting the first text dimension feature into the second fully connected layer of the first attention network to obtain a second text dimension vector, and inputting the first speech dimension feature into the second fully connected layer of the second attention network to obtain a second speech dimension vector; inputting the second text dimension feature into the activation layer of the first attention network to obtain a text attention vector, and inputting the second speech dimension feature into the activation layer of the second attention network to obtain a speech attention vector.

[0111] Furthermore, when the machine-readable instructions are executed by the processor 710, the following processing can also be performed: the text emotion feature and the speech emotion feature are spliced ​​together to obtain a spliced ​​emotion feature; the spliced ​​emotion feature is input into the emotion recognition layer to obtain the main emotion and accompanying emotion corresponding to each sub-speech sentence of the target speech, and the spliced ​​emotion feature is input into the intensity recognition layer to obtain the emotion intensity corresponding to each sub-speech sentence of the target speech.

[0112] In an embodiment of the present application, by inputting the acquired target speech to be recognized and the target text corresponding to the target speech into a trained speech emotion calculation model, the main emotion, accompanying emotion, and emotion intensity corresponding to the target speech can be obtained; wherein the speech emotion calculation model is trained using multiple sample speech and the sample main emotion label, sample accompanying emotion label, and sample emotion intensity label corresponding to each sample speech. In this way, fine-grained emotion and emotion intensity including the main emotion and accompanying emotion can be identified from the target speech, which can improve the emotion recognition effect.

[0113] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the speech emotion recognition method provided in the above embodiment are executed.

[0114] Specifically, when the computer program is run by the processor, it performs at least the following steps: obtaining the target speech to be recognized and determining the target text corresponding to the target speech; inputting the target speech and the target text into the trained speech emotion calculation model together to obtain the main emotion, accompanying emotion and emotion intensity corresponding to the target speech.

[0115] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned speech emotion recognition method, and can identify fine-grained emotions and emotion intensity including main emotions and accompanying emotions from the target speech, which can improve the emotion recognition effect.

[0116] Based on the same application concept, an embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the speech emotion recognition method provided in the above embodiment.

[0117] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0118] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0119] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A speech emotion recognition method, characterized in that: The speech emotion recognition method comprises: Acquire a target speech to be recognized and determine a target text corresponding to the target speech; Inputting the target speech and the target text together into a trained speech emotion calculation model to obtain the main emotion, accompanying emotion, and emotion intensity corresponding to the target speech; the accompanying emotion appears along with at least one main emotion, and there is an accompanying relationship between the main emotion and the accompanying emotion; the accompanying relationship refers to the accompanying relationship between the main emotion and the accompanying emotion; The speech emotion calculation model is trained using a plurality of sample speech and a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to each sample speech.

2. The speech emotion recognition method according to claim 1, wherein For any of the sample speech, generate a sample main emotion label, a sample accompanying emotion label, and a sample emotion intensity label corresponding to the sample speech according to the following steps: Recognize the sample speech to obtain a sample text corresponding to the sample speech and a timestamp of each sub-speech sentence; Generate a visual initial annotation information file based on the sample text corresponding to the sample speech and the timestamps corresponding to each sub-speech sentence; Based on the sample speech, the predefined main emotion set, the accompanying emotion set and the preset annotation rules, the initial annotation information file is annotated to generate a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to the sample speech.

3. The speech emotion recognition method according to claim 2, characterized in that Determine the timestamps of each sub-speech sentence corresponding to the sample speech according to the following steps: Identifying at least one silent segment from the sample speech, and determining a silence duration corresponding to each silent segment; The sample speech is segmented according to the silence segments whose silence duration is greater than or equal to the preset duration, and each sub-speech sentence corresponding to the sample speech and the timestamp of each sub-speech sentence are obtained.

4. The speech emotion recognition method according to claim 2, characterized in that The method of labeling the initial labeling information file based on the sample speech, the predefined main emotion set, the predefined accompanying emotion set, and the preset labeling rules to generate a sample main emotion label, a sample accompanying emotion label, and a sample emotion intensity label corresponding to the sample speech includes: Acquire multiple annotation results obtained by annotating the initial annotation information file based on the sample speech, a predefined main emotion set, a predefined accompanying emotion set, and a preset annotation rule; A target result having a consistency greater than a preset threshold is screened out from the multiple annotation results, and a sample main emotion label, a sample accompanying emotion label, and a sample emotion intensity label corresponding to the sample speech are determined based on the target result.

5. The speech emotion recognition method according to claim 1, wherein Generate the speech emotion computing model according to the following steps: Acquire the plurality of sample voices, and divide the plurality of sample voices into a first sample group and a second sample group; For any second speech in the second sample group, masking a sample speech feature or a sample text feature corresponding to the second speech; The initial multimodal speech emotion model is trained using the first sample group, the masked second sample group, and the sample main emotion label, sample accompanying emotion label, and sample emotion intensity label corresponding to each sample speech to generate the speech emotion calculation model.

6. The speech emotion recognition method according to claim 5, characterized in that The masking of the sample speech feature or sample text feature corresponding to any second speech in the second sample group includes: Masking the sample speech features of the second speech with a first masking probability, and masking the sample text features of the second speech with a second masking probability; The first mask probability is smaller than the second mask probability.

7. The speech emotion recognition method according to claim 1, characterized in that The speech emotion calculation model includes a text feature extraction network, a first attention network, a speech feature extraction network, a second attention network, and an emotion classification network; the target speech and the target text are input into the trained speech emotion calculation model together to obtain the main emotion, accompanying emotion, and emotion intensity corresponding to the target speech, including: Inputting the target text into the text feature extraction network to obtain text time sequence features, and inputting the target speech into the speech feature extraction network to obtain speech time sequence features; Obtaining text emotion features based on the text timing features and the first attention network, and obtaining speech emotion features based on the speech timing features and the second attention network; Based on the text emotion feature, the speech emotion feature and the emotion classification network, the main emotion, accompanying emotion and emotion intensity corresponding to the target speech are obtained.

8. The speech emotion recognition method according to claim 7, characterized in that The obtaining of text emotion features based on the text timing features and the first attention network, and the obtaining of speech emotion features based on the speech timing features and the second attention network, include: Inputting the text time sequence features into the first attention network to obtain a text attention vector, and inputting the speech time sequence features into the second attention network to obtain a speech attention vector; each element in the text attention vector is used to represent the importance of the corresponding frame text in expressing emotion in the current context; each element in the speech attention vector is used to represent the importance of the corresponding frame speech in expressing emotion in the current context; The text emotion feature is determined according to the text attention vector and the text timing feature, and the speech emotion feature is determined according to the speech attention vector and the speech timing feature.

9. The speech emotion recognition method according to claim 8, characterized in that The first attention network and the second attention network both include a first fully connected layer, a second fully connected layer, and an activation layer; the step of inputting the text temporal features into the first attention network to obtain a text attention vector, and inputting the speech temporal features into the second attention network to obtain a speech attention vector, comprises: Inputting the text time sequence feature into the first fully connected layer of the first attention network to obtain a first text dimension feature, and inputting the speech time sequence feature into the first fully connected layer of the second attention network to obtain a first speech dimension feature; Inputting the first text dimension feature into the second fully connected layer of the first attention network to obtain a second text dimension vector, and inputting the first speech dimension feature into the second fully connected layer of the second attention network to obtain a second speech dimension vector; The second text dimension feature is input into the activation layer of the first attention network to obtain a text attention vector, and the second speech dimension feature is input into the activation layer of the second attention network to obtain a speech attention vector.

10. The speech emotion recognition method according to claim 7, characterized in that: The emotion classification network includes an emotion recognition layer and an intensity recognition layer; obtaining the main emotion, accompanying emotion, and emotion intensity corresponding to the target speech based on the text emotion features, the speech emotion features, and the emotion classification network includes: Splicing the text emotion feature and the speech emotion feature to obtain a spliced ​​emotion feature; The spliced ​​emotional features are input into the emotional recognition layer to obtain the main emotions and accompanying emotions corresponding to each sub-speech sentence of the target speech, and the spliced ​​emotional features are input into the intensity recognition layer to obtain the emotional intensity corresponding to each sub-speech sentence of the target speech.

11. A speech emotion recognition device, characterized in that: The speech emotion recognition device comprises: An acquisition module is used to acquire a target speech to be recognized and determine a target text corresponding to the target speech; An input module is configured to input the target speech and the target text into a trained speech emotion calculation model to obtain a main emotion, an accompanying emotion, and an emotion intensity corresponding to the target speech; the accompanying emotion appears along with at least one main emotion, and there is an accompanying relationship between the main emotion and the accompanying emotion; the accompanying relationship refers to a relationship of accompanying occurrence between the main emotion and the accompanying emotion; The speech emotion calculation model is trained using a plurality of sample speech and a sample main emotion label, a sample accompanying emotion label and a sample emotion intensity label corresponding to each sample speech.

12. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the speech emotion recognition method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the speech emotion recognition method according to any one of claims 1 to 10 are executed.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the speech emotion recognition method according to any one of claims 1 to 10 are implemented.

Images