Voice emotion recognition methods, devices, electronic devices and storage media

Abstract

This application provides a voice emotion recognition method, apparatus, electronic device, and storage medium. The method includes: acquiring multiple voice segments corresponding to voice data to be processed; determining the voice emotion change timing points corresponding to the multiple voice segments; determining non-silent voice segments from the multiple voice segments based on the voice emotion change timing points; inputting the non-silent voice segments into a pre-constructed emotion decoder to determine emotion label features; and performing multi-label recognition and classification on the emotion label features to obtain an emotion recognition result; wherein the emotion recognition result includes: a first emotion recognition result and a second emotion recognition result with different emotion types.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech emotion recognition method, device, electronic device, and storage medium. Background Technology

[0002] With the development of natural language processing technology, artificial intelligence and human-computer interaction technologies are widely used in many fields. Speech recognition refers to the high technology that enables machines to convert speech signals into corresponding text or commands through recognition and understanding. Speech recognition is often used to replace or simplify people's work in some work scenarios, or to conduct better and more convenient human-computer interaction operations, such as voice assistants and intelligent customer service. However, existing speech recognition technologies can usually only convert speech signals into corresponding text or commands, and cannot identify the emotions accompanying the speech signals. Human emotions are variable; even the same sentence can express vastly different meanings under different emotional states. Therefore, recognizing the emotions accompanying speech signals is crucial. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a speech emotion recognition method, device, electronic device and storage medium to solve the problem of recognizing the emotions accompanying speech.

[0004] To achieve the above objectives, this application provides a speech emotion recognition method, comprising:

[0005] Acquire multiple speech segments corresponding to the speech data to be processed;

[0006] Determine the timing points of voice emotion changes corresponding to the multiple voice segments;

[0007] Based on the timing of the changes in voice emotion, non-silent voice segments are determined from the plurality of voice segments;

[0008] The non-silent speech segment is input into a pre-built emotion decoder to determine emotion label features;

[0009] The emotion label features are subjected to multi-label recognition and classification to obtain emotion recognition results; wherein, the emotion recognition results include: a first emotion recognition result and a second emotion recognition result with different emotion types.

[0010] For the same purpose, this application provides a voice emotion recognition device, comprising:

[0011] The preprocessing module is configured to acquire multiple speech segments corresponding to the speech data to be processed;

[0012] The first determining module is configured to determine the timing points of voice emotion changes corresponding to the plurality of voice segments;

[0013] The second determining module is configured to determine non-silent speech segments from the plurality of speech segments based on the timing points of the speech emotion changes.

[0014] The third determining module is configured to input the non-silent speech segment into a pre-built emotion decoder to determine emotion label features;

[0015] The recognition module is configured to perform multi-label recognition and classification on the emotion label features to obtain emotion recognition results; wherein, the emotion recognition results include: a first emotion recognition result and a second emotion recognition result with different emotion types.

[0016] For the same purpose, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the speech emotion recognition method as described above.

[0017] For the same purpose, this application provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the speech emotion recognition method as described above.

[0018] As described above, the speech emotion recognition method, apparatus, electronic device, and storage medium provided in this application acquire multiple speech segments corresponding to the speech data to be processed, determine the speech emotion change timing points corresponding to the multiple speech segments, determine non-silent speech segments from the multiple speech segments based on the speech emotion change timing points, further input the non-silent speech segments into a pre-constructed emotion decoder to determine emotion label features, and finally perform multi-label recognition and classification on the emotion label features to obtain emotion recognition results. The emotion recognition results include a first emotion recognition result and a second emotion recognition result with different emotion types. This application generates more accurate emotion label features when performing emotion recognition on speech data, can simultaneously identify different emotions carried by non-silent speech segments in the speech data, and classify the emotion label features to obtain recognition results of different emotions carried by non-silent speech segments, thereby improving the accuracy of speech emotion recognition. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the speech emotion recognition method provided in the embodiments of this application.

[0021] Figure 2 This is an exemplary flowchart illustrating the speech emotion recognition method provided in an embodiment of this application.

[0022] Figure 3 This is a schematic diagram of a voice clip provided in an embodiment of this application.

[0023] Figure 4 This is a table comparing the main emotion tags and main emotion descriptions provided in the embodiments of this application.

[0024] Figure 5 This is a table of accompanying emotion labels and descriptions provided in the embodiments of this application.

[0025] Figure 6 This is a schematic diagram of a voice emotion recognition device provided in an embodiment of this application.

[0026] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in this embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0028] It should be noted that, unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the term encompasses the elements or objects listed following the term and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0029] As described in the background section, due to the unique nature of speech emotion expression, traditional acoustic features are weaker in their ability to express emotions compared to deep learning features obtained from models pre-trained on large-scale data. Furthermore, due to the scarcity of high-quality speech emotion datasets, the generalization performance of the learned models is poor. On the other hand, most of the training data used by current mainstream pre-trained models consists of emotionless speech, resulting in the learned features not being able to fully express emotional information, thus limiting the improvement in speech emotion recognition performance based on these emotionless speech features.

[0030] In developing this application, the applicant discovered that current mainstream algorithms for speech emotion recognition perform speech recognition on a unit-by-unit basis, regardless of the length of the input speech segment. However, in reality, even within the same speech segment, different segments often exhibit emotional shifts, and current methods cannot adaptively segment different emotional segments. Furthermore, most current speech emotion recognition methods only perform single-label speech emotion recognition. On one hand, these methods output limited labels, often only including basic emotions such as joy, anger, sorrow, and happiness; on the other hand, due to the complexity of emotions, a small segment often contains multiple emotions, such as a question in a happy state versus a question in an angry state, and existing emotion recognition methods cannot handle this problem well.

[0031] To address the aforementioned issues, this application proposes a voice emotion recognition method, apparatus, electronic device, and storage medium.

[0032] The technical solution of this application will be further described in detail below through specific embodiments.

[0033] refer to Figure 1 This diagram illustrates an application scenario of the voice emotion recognition method provided in this application. The application scenario includes a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, server 102, and data storage system 103 can all be connected via wired or wireless communication networks. The terminal device 101 includes, but is not limited to, desktop computers, mobile phones, mobile computers, tablets, media players, smart wearable devices, personal digital assistants (PDAs), or other electronic devices capable of performing the aforementioned functions. The server 102 and data storage system 103 can both be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0034] Server 102 is used to provide voice emotion recognition services to users of terminal device 101. Terminal device 101 has a client installed that communicates with server 102. Users can input a voice data to be processed through the client. The client sends the voice data to be processed to server 102. Server 102 inputs the voice data to be processed into a pre-trained voice feature extraction model to extract the voice temporal features of the voice data. Based on the voice temporal features, the voice data is segmented to determine non-silent voice segments. The non-silent voice segments are input into a pre-built emotion decoder to determine emotion label features. Then, a classifier performs multi-label classification on the emotion label features to obtain the emotion recognition result. The emotion recognition result is sent to the client. The client displays the emotion recognition result to the user to realize the emotion recognition of voice data.

[0035] The data storage system 103 stores a large amount of training data, including pre-acquired emotional speech datasets. These emotional speech datasets provide better representation of the emotional information in the speech data. The sources of the training data include, but are not limited to, existing databases, data crawled from the internet, or data uploaded by users when using the client. The pre-trained speech feature extraction model uses a Transformer encoder. Because attention is calculated between each frame, the features of each frame also contain global feature information. When the accuracy of the speech feature extraction model reaches a certain requirement, the server 102 can provide users with speech temporal feature extraction services based on the speech feature extraction model. Simultaneously, the server 102 can continuously optimize the speech feature extraction model based on newly added training data.

[0036] The following is combined Figure 1 The application scenarios described above illustrate the speech emotion recognition method according to exemplary embodiments of this application. It should be noted that the above application scenarios are merely shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application can be applied to any applicable scenario.

[0037] refer to Figure 2 This is an exemplary flowchart illustrating the speech emotion recognition method provided in the embodiments of this application.

[0038] Step S201: Obtain multiple speech segments corresponding to the speech data to be processed.

[0039] In practical implementation, voice data refers to voice data that requires emotion recognition. Due to differences in people's speaking habits and breathing needs, a single voice data segment may include multiple voice fragments, which may include silent and non-silent voice segments. Depending on the application scenario, the data source for voice data may differ. For example, in a smart voice customer service application scenario, voice data could be the user's voice data received by the system.

[0040] Furthermore, the obtained speech data is input into a pre-trained speech feature extraction model to extract the speech temporal features of the speech data;

[0041] The speech data is segmented based on the speech temporal features to determine multiple speech segments.

[0042] In this embodiment of the application, the speech feature extraction model can be constructed in the following manner;

[0043] First, an emotional speech training set is obtained, which includes several emotional speech segments with emotion labels. The emotion labels are used to represent the emotion type corresponding to the emotional speech segments. Since most of the training data used in the existing technology are emotionless speech segments, the speech features obtained by the model training cannot fully express emotional information, resulting in limited improvement in speech emotion recognition based on these features. Therefore, in the process of constructing the speech feature extraction model in this application, a large number of emotional speech segments with emotion labels are used as the emotional speech training set, which can enable the speech temporal features of the speech data input to the model to represent emotional information to the greatest extent.

[0044] Furthermore, based on the emotional speech training set obtained above, the attention-based neural network model is trained to obtain a speech feature extraction model. The attention-based neural network can perform attention calculations between each frame, so the obtained speech temporal features not only represent the local feature information of the speech segment, but also contain the global feature information of the input speech data.

[0045] In practical implementation, the temporal features of the speech are input into a speech segmentation network, which employs a Transformer encoder network. Existing technologies mostly use LSTM neural network models for speech segmentation training, characterized by iterative recognition, training one speech segment after another. This iterative recognition method is time-consuming and inefficient. In contrast, this application uses a Transformer encoder network, enabling parallel recognition of speech segments. All speech segments in the speech data are recognized simultaneously, significantly accelerating computational efficiency. This model uses positional embedding to understand the order of speech segments and employs sub-attention mechanisms and fully connected layers for computation, simultaneously considering both local and global features of the speech segments, thus improving the efficiency of speech segment recognition.

[0046] Furthermore, by inputting the speech temporal features into the speech segmentation network, new speech temporal features corresponding to the speech segments can be obtained. Speech data can then be segmented based on these new speech temporal features to determine multiple speech segments.

[0047] Step S202: Determine the timing points of voice emotion changes corresponding to the multiple voice segments.

[0048] Voice emotion change timing points can be used to characterize the timing points corresponding to changes in emotion within a user's voice content. In one optional embodiment, voice emotion change timing points can include: voice emotion start point, voice emotion end point, and voice emotion inflection point. The aforementioned new voice timing features are input into a pre-constructed voice emotion marker model to determine the voice emotion start point, voice emotion end point, and voice emotion inflection point between adjacent voice segments in the multiple voice segments. The voice emotion start point can be understood as the node where a silent context transitions to a non-silent context. For example, when voice data recording begins, no user speaks; after a while, the user speaks, and the node where the user speaks is the voice emotion start point. The voice emotion end point can be understood as the node where a non-silent context transitions to a silent context. For example, during voice data recording, the user speaks continuously; after the user finishes speaking a sentence, they stop speaking, and the context becomes silent. The node after the last syllable of the user's speech ends is the voice emotion end point. A voice emotion inflection point can be understood as the node where a non-silent context carrying one emotion changes to a non-silent context carrying another emotion. For example, during the recording of voice data, the user is constantly speaking, but before a certain point in the process of saying this sentence, the user's emotion is happy. At the next moment, the user asks a question based on the happy emotion, so the voice segment after that point will carry the emotion of questioning. Therefore, the node where the user's emotion begins to change during speaking is the voice emotion inflection point.

[0049] It should be noted that because users think while speaking, pauses may occur even while expressing the same sentence. These pauses may make it difficult to accurately distinguish whether the current point is the end of a speech emotion or a turning point in the speech emotion. Therefore, this could lead to confusion between subsequent speech emotion turning points and end points, resulting in temporal errors in the speech data. To address this issue, the temporal interval between speech emotion ending points and turning points can be limited, but is not limited to, setting a pre-defined time threshold. For example, if two adjacent index labels are determined to be in the order of speech emotion ending point and speech emotion beginning point, further judgment is needed. The relationship between the temporal interval between the end point and the beginning point of voice emotion and the threshold of this time period is as follows: If the temporal interval between the end point and the beginning point of voice emotion is greater than the threshold of this time period, it can be determined that the operation of determining the temporal order of two adjacent voice emotion change time points as the end point and the beginning point of voice emotion is correct. If the temporal interval between the end point and the beginning point of voice emotion is less than the threshold of this time period, it can be determined that the operation of determining the temporal order of two adjacent voice emotion change time points as the end point and the beginning point of voice emotion is incorrect. It is necessary to further correct the voice emotion beginning point to the voice emotion turning point to ensure the continuity of emotion recognition in the voice segment.

[0050] Step S203: Based on the timing of the changes in voice emotion, determine the non-silent voice segments from the multiple voice segments.

[0051] As an optional embodiment, non-silent speech segments can be determined from the plurality of speech segments based on the temporal relationship between the timing points of the speech emotion changes;

[0052] Specifically, in multiple speech segments, the timing points of speech emotion changes between adjacent speech segments are assigned corresponding index labels. The index labels can be used to indicate the type of timing points of speech emotion changes, and then the non-silent speech segments can be determined based on the timing relationship between the index labels.

[0053] As an optional embodiment, the timing points of voice emotion change include: voice emotion start point, voice emotion end point, and voice emotion turning point; the index labels include: first index label, second index label, and third index label.

[0054] Furthermore, the starting point of voice emotion corresponds to the first index label; the ending point of voice emotion corresponds to the second index label; and the turning point of voice emotion corresponds to the third index label.

[0055] Furthermore, non-silent speech segments in speech data can be determined based on the temporal relationship between adjacent index tags using any of the following methods:

[0056] refer to Figure 3 This is a schematic diagram of a voice segment provided in an embodiment of this application.

[0057] As an optional embodiment, in response to adjacent index tags being all third index tags, the speech segment between adjacent index tags is determined to be a non-silent speech segment. Specifically, speech segment 1, speech segment 2, and speech segment 3 can be a continuously completed sentence provided by the user, except that the user expresses the sentence with two emotional shifts. Using the method described in this embodiment, speech segment 2 can be determined to be a non-silent speech segment.

[0058] As an optional embodiment, in response to adjacent index tags including a first index tag and a second index tag, and the timing of the first index tag preceding the timing of the second index tag, the speech segment between adjacent index tags is determined to be a non-silent speech segment. Specifically, speech segment 4 may be a silent speech segment when the user is not speaking, and speech segment 5 is a continuously completed sentence expressed by the user, which can also be understood as a speech segment spoken in one breath without emotional change. Using the method described in this embodiment, speech segment 5 can be determined to be a non-silent speech segment.

[0059] As an optional embodiment, in response to adjacent index tags including a first index tag and a third index tag, and the timing of the first index tag preceding the timing of the third index tag, the speech segment between adjacent index tags is determined to be a non-silent speech segment. Specifically, speech segment 6 may be a silent speech segment when the user is not speaking, and speech segments 7 and 8 may be a continuously completed sentence provided by the user, except that the user expresses the sentence with an emotional shift. Using the method described in this embodiment, speech segment 7 can be determined to be a non-silent speech segment before the shift point in the sentence.

[0060] As an optional embodiment, in response to adjacent index tags including a second index tag and a third index tag, and the timing of the third index tag preceding that of the second index tag, the speech segment between adjacent index tags is determined to be a non-silent speech segment. Specifically, speech segments 7 and 8 can be a continuously completed sentence provided by the user, except that the user expresses the sentence with an emotional shift. Using the method described in this embodiment, speech segment 8 can be determined to be a non-silent speech segment after the emotional shift in the sentence. Similarly, speech segment 3 can be determined to be a non-silent speech segment after the emotional shift in the speech.

[0061] As an optional embodiment, in response to adjacent index tags including a second index tag and a first index tag, and the timing of the second index tag being earlier than that of the first index tag, the speech segment between adjacent index tags is determined as a silent speech segment.

[0062] Step S204: Input the non-silent speech segment into a pre-built emotion decoder to determine emotion label features.

[0063] As an optional embodiment, for the non-silent speech segment determined in step S203, it is necessary to calculate the corresponding non-silent segment attention mask. Specifically, an initial attention mask of length T is initialized, and the value includes multiple flag bits. The vector value between every two flag bits is 0. Further, the initial attention mask and the flag bits corresponding to the index tags are encoded according to the index tags of the non-silent segments to determine the non-silent segment attention mask. For example, assuming there is a speech data of 30 frames, the flag bits of the index tags output by the speech emotion tag model are as follows: the third index tag flag bit in frame 3, the second index tag flag bit in frame 7, the third index tag flag bit in frame 11, the third index tag flag bit in frame 15, the first index tag flag bit in frame 20, and the second index tag flag bit in frame 25. The non-silent segments are: Segment 1: frames 3 to 7, Segment 2: frames 11 to 15, Segment 3: frames 15 to 20, and Segment 4: frames 20 to 25. For each segment, a vector of length 30 is obtained, and the flag value of the corresponding segment index label is set to 1, serving as the non-silent segment attention mask for the subsequent decoder network.

[0064] Furthermore, to obtain the first emotion query code and the second emotion query code, the first emotion query code can be used as the primary emotion query code for the speech data, and the second emotion query code can be used as the accompanying emotion query code for the speech data. Alternatively, the first emotion query code can be used as the accompanying emotion query code for the speech data, and the second emotion query code can be used as the primary emotion query code for the speech data. Different query codes are mainly used to determine different types of speech emotion features. Considering the correlation between the primary emotion and the accompanying emotion, for example, a non-silent speech segment is a worried inquiry, which may be accompanied by a large amount of the primary emotion of worry and a small amount of the accompanying emotion of inquiry; but a non-silent speech segment is a happy coquettishness, which may be accompanied by a large amount of the accompanying emotion of coquettishness and a small amount of the primary emotion of happiness. Therefore, the primary emotion query code and the accompanying emotion query code are concatenated into a single query vector, rather than querying twice.

[0065] It should be noted that since a non-silent speech segment can have multiple primary emotions and multiple accompanying emotions, the query vector can also be formed by concatenating multiple primary emotion query codes and multiple accompanying emotion query codes.

[0066] Furthermore, the speech temporal features, non-silent segment attention mask, and query vector are input into the emotion decoder to determine emotion label features.

[0067] As an optional embodiment, the attention mask of the non-silent segment is queried according to the query vector and the speech temporal features to obtain the emotion recognition result. The emotion recognition result may include a first emotion recognition result and a second emotion recognition result. Since the query vector is concatenated from two different emotion query codes, the emotion types indicated by the first emotion recognition result and the second emotion recognition result obtained by the query are different from each other. An emotion recognition result indicating the main emotion and an emotion recognition result indicating the accompanying emotion can be obtained.

[0068] Step S205: Perform multi-label recognition and classification on the emotion label features to obtain emotion recognition results; wherein, the emotion recognition results include: a first emotion recognition result and a second emotion recognition result with different emotion types.

[0069] Figure 4 This is a table comparing the main emotion tags and main emotion descriptions provided in the embodiments of this application.

[0070] Figure 5 This is a table of accompanying emotion labels and descriptions provided in the embodiments of this application.

[0071] As an optional embodiment, the first emotion recognition result is input into a first emotion classifier to obtain at least one first emotion label; the second emotion recognition result is input into a second emotion labeler to obtain at least one second emotion label. The emotion labels are divided into primary emotion labels and accompanying emotion labels. Each primary emotion label can indicate multiple primary emotion descriptions, and each accompanying emotion label can indicate multiple accompanying emotion descriptions.

[0072] It should be noted that the voice data involved in this application is voice data provided by users under normal emotional states, that is, there will be no unreasonable situations such as being happy or feeling disgusted.

[0073] As described above, the speech emotion recognition method, apparatus, electronic device, and storage medium provided in this application acquire multiple speech segments corresponding to the speech data to be processed, determine the speech emotion change timing points corresponding to the multiple speech segments, determine non-silent speech segments from the multiple speech segments based on the speech emotion change timing points, further input the non-silent speech segments into a pre-constructed emotion decoder to determine emotion label features, and finally perform multi-label recognition and classification on the emotion label features to obtain emotion recognition results. The emotion recognition results include a first emotion recognition result and a second emotion recognition result with different emotion types. This application generates more accurate emotion label features when performing emotion recognition on speech data, can simultaneously identify different emotions carried by non-silent speech segments in the speech data, and classify the emotion label features to obtain recognition results of different emotions carried by non-silent speech segments, thereby improving the accuracy of speech emotion recognition.

[0074] It should be noted that the method in this application embodiment can be executed by a single device, such as a computer or server. The method in this application embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this application embodiment, and the multiple devices will interact with each other to complete the aforementioned voice emotion recognition method.

[0075] It should be noted that the above description describes specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] Based on the same concept, this application also provides a voice emotion recognition device.

[0077] refer to Figure 6 This is a schematic diagram of a voice emotion recognition device provided in an embodiment of this application.

[0078] The preprocessing module 601 is configured to acquire multiple speech segments corresponding to the speech data to be processed;

[0079] The first determining module 602 is configured to determine the timing points of voice emotion changes corresponding to the plurality of voice segments;

[0080] The second determining module 603 is configured to determine non-silent speech segments from the plurality of speech segments based on the timing points of the speech emotion changes.

[0081] The third determining module 604 is configured to input the non-silent speech segment into a pre-built emotion decoder to determine emotion label features;

[0082] The recognition module 605 is configured to perform multi-label recognition and classification on the emotion label features to obtain an emotion recognition result; wherein the emotion recognition result includes: a first emotion recognition result and a second emotion recognition result with different emotion types.

[0083] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0084] The apparatus described above is used to implement the corresponding speech emotion recognition method in the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0085] Based on the same concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the voice emotion recognition method described in any of the above embodiments.

[0086] Figure 7 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 710, a memory 720, an input / output interface 730, a communication interface 740, and a bus 750. The processor 710, memory 720, input / output interface 730, and communication interface 740 are interconnected internally via the bus 750.

[0087] The processor 710 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0088] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0089] The input / output interface 730 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0090] The communication interface 740 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, BlueRay).

[0091] Bus 750 includes a pathway for transmitting information between various components of the device, such as processor 710, memory 1020, input / output interface 730, and communication interface 740.

[0092] It should be noted that although the above-described device only shows the processor 710, memory 720, input / output interface 730, communication interface 740, and bus 750, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0093] The electronic devices described above are used to implement the corresponding voice emotion recognition methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0094] Based on the same concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the speech emotion recognition method as described in any of the above embodiments.

[0095] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0096] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the voice emotion recognition method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0097] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0098] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0099] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0100] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A voice emotion recognition method, characterized in that, include: Acquire multiple speech segments corresponding to the speech data to be processed; wherein, the speech data includes speech temporal features; Determine the timing points of voice emotion changes corresponding to the multiple voice segments; Based on the timing of the changes in voice emotion, non-silent voice segments are determined from the plurality of voice segments; Obtain an initial attention mask; the initial attention mask includes: multiple flag bits; The initial attention mask and the flag bits corresponding to the index tags are encoded according to the index tags of the non-silent speech segments to determine the non-silent segment attention mask; Obtain the first emotion query code and the second emotion query code, and determine the query vector based on the first emotion query code and the second emotion query code; The speech temporal features, the non-silent segment attention mask, and the query vector are input into the emotion decoder to determine the emotion label features; The emotion label features are subjected to multi-label recognition and classification to obtain emotion recognition results; wherein, the emotion recognition results include: a first emotion recognition result and a second emotion recognition result with different emotion types.

2. The method according to claim 1, characterized in that, The acquisition of multiple speech segments corresponding to the speech data to be processed includes: The speech data to be processed is input into a pre-trained speech feature extraction model to extract the speech temporal features of the speech data; The speech data is segmented based on the speech temporal features to determine multiple speech segments.

3. The method according to claim 2, characterized in that, This also includes constructing the pre-trained speech feature extraction model in the following ways: Obtain an emotional speech training set; wherein, the emotional speech training set includes several emotional speech segments with emotional label features; the emotional label features are used to characterize the emotion type corresponding to the emotional speech segment; The speech feature extraction model is obtained by training an attention-based neural network model based on the emotional speech training set.

4. The method according to claim 2, characterized in that, Determining the timing points of voice emotion changes corresponding to the plurality of voice segments includes: The speech temporal features are input into a pre-constructed speech emotion labeling model to determine the temporal points of speech emotion changes between adjacent speech segments in the multiple speech segments; The step of determining non-silent speech segments from the plurality of speech segments based on the temporal points of speech emotion changes corresponding to the plurality of speech segments includes: Based on the temporal relationship between the time points of the voice emotion change, non-silent voice segments are determined from the multiple voice segments.

5. The method according to claim 4, characterized in that, The timing points of voice emotion changes between adjacent voice segments are assigned corresponding index labels; The step of determining non-silent speech segments from the plurality of speech segments based on the temporal relationship between the time points of the speech emotion changes includes: The non-silent speech segment is determined based on the temporal relationship between the index tags.

6. The method according to claim 5, characterized in that, The timing points of voice emotion change include: voice emotion start point, voice emotion end point, and voice emotion turning point; the index labels include: first index label, second index label, and third index label; The starting point of the voice emotion corresponds to the first index tag; The end point of the voice emotion corresponds to the second index label; The emotional turning point in the voice corresponds to the third index label.

7. The method according to claim 6, characterized in that, Determining the non-silent speech segment based on the temporal relationship between the index tags includes: In response to the fact that both adjacent index tags are the third index tag, the speech segment between adjacent index tags is determined as the non-silent speech segment; or, In response to adjacent index tags including the first index tag and the second index tag, and the timing of the first index tag being earlier than the timing of the second index tag, the speech segment between adjacent index tags is determined as the non-silent speech segment; or, In response to adjacent index tags including the first index tag and the third index tag, and the timing of the first index tag being earlier than that of the third index tag, the speech segment between adjacent index tags is determined as the non-silent speech segment. or, In response to adjacent index tags including the second index tag and the third index tag, and the timing of the third index tag being earlier than that of the second index tag, the speech segment between adjacent index tags is determined as the non-silent speech segment.

8. The method according to claim 1, characterized in that, The step of inputting the speech temporal features, the non-silent segment attention mask, and the query vector into the emotion decoder to determine the emotion label features further includes: The first emotion recognition result and the second emotion recognition result are obtained by querying the attention mask of the non-silent segment based on the query vector and the speech temporal features.

9. The method according to claim 8, characterized in that, The process of querying the attention mask of the non-silent segment based on the query vector and the speech temporal features to obtain the first emotion recognition result and the second emotion recognition result further includes: The first emotion recognition result is input into the first emotion labeler to obtain at least one first emotion label; The second emotion recognition result is input into the second emotion labeler to obtain at least one second emotion label.

10. A voice emotion recognition device, characterized in that, include: The preprocessing module is configured to acquire multiple speech segments corresponding to the speech data to be processed; wherein, the speech data includes speech temporal features; The first determining module is configured to determine the timing points of voice emotion changes corresponding to the plurality of voice segments; The second determining module is configured to determine non-silent speech segments from the plurality of speech segments based on the timing points of the speech emotion changes. The third determining module is configured to acquire an initial attention mask; the initial attention mask includes: multiple flag bits; encoding the initial attention mask and the flag bits corresponding to the index tags according to the index tags of the non-silent speech segments to determine the non-silent segment attention mask; acquiring a first emotion query code and a second emotion query code, determining a query vector according to the first emotion query code and the second emotion query code; and inputting the speech temporal features, the non-silent segment attention mask, and the query vector into the emotion decoder to determine the emotion label features; The recognition module is configured to perform multi-label recognition and classification on the emotion label features to obtain emotion recognition results; wherein, the emotion recognition results include: a first emotion recognition result and a second emotion recognition result with different emotion types.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 9.

Images