A speech emotion recognition method based on spectrum segmentation and deep learning

Abstract

The present invention proposes a speech emotion recognition method based on spectrum segmentation and deep learning, which relates to the field of speech recognition technology, including: S1, obtaining an ESD emotion data set, and reading the data set to obtain audio data; S2, performing data preprocessing on the audio data to obtain a grayscale spectrogram; S3, performing spectrum segmentation on the grayscale spectrogram according to different frequency bands of the grayscale spectrogram to obtain a segmentation result; S4, selecting different networks based on the segmentation result to perform emotion classification to obtain a classification result; S5, outputting the classification result to obtain emotion recognition data; the present invention effectively improves the utilization rate of features, thereby using speech data to predict and classify the speaker's emotions, reducing the complexity of the network, improving the accuracy of speech emotion recognition, and greatly improving the efficiency and accuracy of information processing.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech emotion recognition method based on spectrum segmentation and deep learning. Background Art

[0002] With the rapid development of human-computer interaction technology, emotion recognition has become a hot research topic in the AI ​​field in recent years. This research focuses on identifying and understanding emotions and feelings in contextual human-computer interaction, helping robots better understand conversational intent and underlying semantics, thereby improving the quality of service in multimedia communications and enhancing user experience. In the field of speech emotion recognition in human-computer interaction, researchers often use spectrograms as acoustic features to identify speech emotions using convolutional neural networks. Spectrograms are a comprehensive representation of speech signals in the time and frequency domains and contain rich information. However, convolutional neural networks process spectrograms locally, failing to fully utilize this information, resulting in low model recognition accuracy.

[0003] Traditional speech emotion recognition algorithms consist of feature extraction and classifiers. Their features include spectral-related features, super-sound quality features, and voice quality features. Traditional machine learning methods are used to classify speech emotions. Due to the complex intrinsic factors of speech itself, hand-crafted features are currently difficult to describe more refined feature representations. At the same time, the features extracted by this method do not take into account the relationship between speech signals in the time domain, so the accuracy is low.

[0004] Based on this, a speech emotion recognition method based on spectrum segmentation and deep learning is proposed to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a speech emotion recognition method based on spectrum segmentation and deep learning, which can improve the recognition accuracy of speech emotion recognition, that is, improve the accuracy of speech recognition.

[0006] The technical solution of the present invention is:

[0007] In a first aspect, the present application provides a method for speech emotion recognition based on spectrum segmentation and deep learning, which comprises the following steps:

[0008] S1. Obtain the ESD emotion dataset and read the dataset to obtain audio data;

[0009] S2. performing data preprocessing on the audio data to obtain a grayscale spectrogram;

[0010] S3, performing spectrum segmentation on the grayscale spectrogram according to different frequency bands of the grayscale spectrogram to obtain a segmentation result;

[0011] S4. Select different networks based on the segmentation results to perform sentiment classification to obtain classification results;

[0012] S5. Output the classification results to obtain emotion recognition data.

[0013] Furthermore, the above step S2 includes:

[0014] S21, reading audio data and converting the audio data into one-dimensional data;

[0015] S22, performing frame division and windowing processing on the one-dimensional data to convert it into two-dimensional data;

[0016] S23, performing a fast Fourier transform on the two-dimensional data to obtain a short-time Fourier spectrum;

[0017] S24, performing a multiplication operation on the short-time Fourier spectrum to obtain a short-time power spectrum;

[0018] S25. Convert the short-time power spectrum value into a two-dimensional image composed of gray levels to obtain a grayscale spectrogram.

[0019] Furthermore, the above step S3 includes:

[0020] S31, binarizing the grayscale spectrogram to obtain a binarized grayscale spectrogram;

[0021] S32. Calculating based on the binarized grayscale spectrogram to obtain an average energy of the grayscale spectrogram;

[0022] S33, plotting the average energy of the grayscale spectrogram to obtain an envelope of the average energy;

[0023] S34, performing differentiation on the envelope to obtain a differential curve;

[0024] S35. Obtaining the zero point of the first-order differential envelope according to the differential curve;

[0025] S36. Selecting the zero point of the first-order differential envelope as a dividing point to perform spectrum segmentation on the envelope to obtain a segmentation result.

[0026] Furthermore, the above step S31 includes:

[0027] S311, reading the grayscale spectrogram to obtain matrix data of the grayscale spectrogram;

[0028] S312: Setting a division threshold, binarizing the grayscale spectrogram based on the matrix data and the pixel level to obtain a binarized grayscale spectrogram.

[0029] Furthermore, the calculation process of obtaining the average energy of the grayscale spectrogram based on the binarized grayscale spectrogram in step S32 includes:

[0030] The binary grayscale spectrogram is converted into a matrix form, and the energy value of each small frequency band is obtained row by row, and saved in order from low frequency to high frequency. The above operation is repeated until all the data in the dataset are processed. Then the energy values ​​of all the binary grayscale spectrograms are summed and their average is calculated to obtain the average energy of the grayscale spectrogram.

[0031] Furthermore, the process of plotting the average energy of the grayscale spectrogram to obtain the envelope of the average energy in step S33 includes:

[0032] S331, plotting based on the average energy of the grayscale spectrogram to obtain an average energy curve;

[0033] S332, obtaining the maximum point and the minimum point of the average energy curve to obtain the horizontal and vertical coordinates corresponding to the maximum point and the minimum point of the average energy curve;

[0034] S333. Draw a plot based on the horizontal and vertical coordinates corresponding to the maximum and minimum points of the average energy curve to obtain an envelope curve of the average energy.

[0035] Furthermore, the above step S4 includes:

[0036] S41, sending the segmentation results to different networks respectively to obtain output results of different networks;

[0037] S42, aggregating output results of different networks to obtain an aggregated result;

[0038] S43. Perform sentiment classification on the aggregation results to obtain the final classification results.

[0039] Furthermore, the network used in the above step S41 is a ResNet34 network, and a lightweight attention mechanism CBAM is added to the above ResNet34 network.

[0040] In a second aspect, the present application provides an electronic device, characterized by comprising:

[0041] a memory for storing one or more programs;

[0042] processor;

[0043] When the one or more programs are executed by the processor, a speech emotion recognition method based on spectrum segmentation and deep learning as described in any one of the first aspects is implemented.

[0044] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a speech emotion recognition method based on spectrum segmentation and deep learning as described in any one of the first aspects above.

[0045] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0046] (1) The present invention provides a speech emotion recognition method based on spectrum segmentation and deep learning. By plotting the envelope of the frequency and energy of the speech signal, it is found that its energy is mainly concentrated near three resonance peaks, which reflect different characteristics. The spectrum graph is segmented in this way, which can ultimately effectively improve the utilization rate of features, thereby using speech data to predict and classify the speaker's emotions;

[0047] (2) The present invention converts the audio into a spectrogram by framing and windowing it, and then divides the spectrum according to different frequency bands. At the same time, to prevent the loss of edge features, the spectrograms overlap with each other for a part, and then send them to three networks respectively. The three networks extract features of different frequency bands, thereby reducing the complexity of the network and improving the accuracy of speech emotion recognition;

[0048] (3) The present invention improves the expressiveness of key point features by adding a lightweight attention mechanism (CBAM) to the network, while suppressing unimportant features (ambient audio, noise), and using limited attention resources to quickly filter out high-value information from a large amount of information, which can greatly improve the efficiency and accuracy of information processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flowchart of a method for speech emotion recognition based on spectrum segmentation and deep learning according to Example 1 of the present invention;

[0051] Figure 2 This is a schematic structural block diagram of an electronic device according to embodiment 2 of the present invention.

[0052] Icon: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0055] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0056] It should be noted that, in this document, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprises..." do not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the elements.

[0057] It should also be noted that, in the description of this application, unless otherwise expressly specified or limited, the terms "disposed" and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections, or electrical connections; direct connections, indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0058] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features thereof may be combined with each other.

[0059] Example 1

[0060] See also Figure 1 , Figure 1 Shown is a flowchart of a speech emotion recognition method based on spectrum segmentation and deep learning provided in Example 1 of the present application.

[0061] In a first aspect, the present application provides a method for speech emotion recognition based on spectrum segmentation and deep learning, which comprises the following steps:

[0062] S1. Obtain the ESD emotion dataset and read the dataset to obtain audio data;

[0063] S2. performing data preprocessing on the audio data to obtain a grayscale spectrogram;

[0064] S3, performing spectrum segmentation on the grayscale spectrogram according to different frequency bands of the grayscale spectrogram to obtain a segmentation result;

[0065] S4. Select different networks based on the segmentation results to perform sentiment classification to obtain classification results;

[0066] S5. Output the classification results to obtain emotion recognition data.

[0067] The present invention discloses a speech emotion classification method based on spectrum graph segmentation, which mainly obtains an ESD emotion dataset, performs framing, windowing, fast Fourier transform and other preprocessing on the speech signal, and then converts it into a grayscale spectrogram. Then, by extracting the energy envelope of the spectrogram, the spectrogram is divided into three frequency bands. The three parts of the spectrogram are then input into three different ResNet networks to extract features, and finally the final recognition result is obtained, thereby effectively improving the utilization rate of features and using speech data to predict and classify the speaker's emotions.

[0068] As a preferred embodiment, step S2 includes:

[0069] S21, reading audio data and converting the audio data into one-dimensional data;

[0070] S22, performing frame division and windowing processing on the one-dimensional data to convert it into two-dimensional data;

[0071] S23, performing a fast Fourier transform on the two-dimensional data to obtain a short-time Fourier spectrum;

[0072] S24, performing a multiplication operation on the short-time Fourier spectrum to obtain a short-time power spectrum;

[0073] S25. Convert the short-time power spectrum value into a two-dimensional image composed of gray levels to obtain a grayscale spectrogram.

[0074] As a preferred embodiment, step S3 includes:

[0075] S31, binarizing the grayscale spectrogram to obtain a binarized grayscale spectrogram;

[0076] S32. Calculating based on the binarized grayscale spectrogram to obtain an average energy of the grayscale spectrogram;

[0077] S33, plotting the average energy of the grayscale spectrogram to obtain an envelope of the average energy;

[0078] S34, performing differentiation on the envelope to obtain a differential curve;

[0079] S35. Obtaining the zero point of the first-order differential envelope according to the differential curve;

[0080] S36. Selecting the zero point of the first-order differential envelope as a dividing point to perform spectrum segmentation on the envelope to obtain a segmentation result.

[0081] In order to prevent the loss of edge features, the upper and lower bounds of each part of the spectrum overlap with each other.

[0082] As a preferred implementation, step S31 includes:

[0083] S311, reading the grayscale spectrogram to obtain matrix data of the grayscale spectrogram;

[0084] S312: Setting a division threshold, binarizing the grayscale spectrogram based on the matrix data and the pixel level to obtain a binarized grayscale spectrogram.

[0085] As a preferred embodiment, the process of calculating the average energy of the grayscale spectrogram based on the binarized grayscale spectrogram in step S32 includes:

[0086] The binary grayscale spectrogram is converted into a matrix form, and the energy value of each small frequency band is obtained row by row, and saved in order from low frequency to high frequency. The operation is repeated until all the data in the dataset are processed. Then the energy values ​​of all the binary grayscale spectrograms are summed and their average is calculated to obtain the average energy of the grayscale spectrogram.

[0087] It should be noted that the average energy of the grayscale spectrogram is the energy data of the current emotion. By plotting the energy data of the current emotion, a frequency-average energy curve can be obtained. By repeating the above steps, the energy data of all emotions and the frequency-average energy curves of all emotions can be obtained.

[0088] As a preferred embodiment, the process of plotting the average energy of the grayscale spectrogram to obtain the envelope of the average energy in step S33 includes:

[0089] S331, plotting based on the average energy of the grayscale spectrogram to obtain an average energy curve;

[0090] S332, obtaining the maximum point and the minimum point of the average energy curve to obtain the horizontal and vertical coordinates corresponding to the maximum point and the minimum point of the average energy curve;

[0091] S333. Draw a plot based on the horizontal and vertical coordinates corresponding to the maximum and minimum points of the average energy curve to obtain an envelope curve of the average energy.

[0092] As a preferred embodiment, step S4 includes:

[0093] S41, sending the segmentation results to different networks respectively to obtain output results of different networks;

[0094] S42, aggregating output results of different networks to obtain an aggregated result;

[0095] S43. Perform sentiment classification on the aggregation results to obtain the final classification results.

[0096] It should be noted that, in this embodiment, the audio is framed and windowed to be converted into a spectrogram, and then its spectrum is segmented according to different frequency bands, and then sent to three networks respectively. The three networks extract features of different frequency bands respectively, thereby reducing the complexity of the network and improving the accuracy of speech emotion recognition.

[0097] As a preferred implementation, the network used in step S41 is a ResNet34 network, to which a lightweight attention mechanism CBAM is added.

[0098] Therefore, by adding a lightweight attention mechanism CBAM to the network, the expressiveness of key point features is improved, while unimportant features (ambient audio, noise) are suppressed. High-value information can be quickly filtered out from a large amount of information using limited attention resources, which can greatly improve the efficiency and accuracy of information processing.

[0099] Example 2

[0100] See also Figure 2 , Figure 2 Shown is a schematic structural block diagram of an electronic device provided in Example 2 of the present application.

[0101] An electronic device includes a memory 101, a processor 102, and a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0102] Among them, the memory 101 can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0103] The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0104] It is understood that the structure shown in the figure is for illustration only. The method for speech emotion recognition based on spectrum segmentation and deep learning may include more or fewer components than shown in the figure, or have a different configuration than shown in the figure. Each component shown in the figure can be implemented using hardware, software, or a combination thereof.

[0105] In the embodiments provided in this application, it should be understood that the disclosed system or method can also be implemented in other ways. The embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0107] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0108] In summary, the embodiment of the present application provides a method for speech emotion recognition based on spectrum segmentation and deep learning. By drawing the envelope of the frequency and energy of the speech signal, the spectrum graph is segmented, which effectively improves the utilization rate of features, thereby using speech data to predict and classify the speaker's emotions; by framing and windowing the audio into a spectrogram, and then performing spectrum segmentation according to different frequency bands, at the same time, to prevent the loss of edge features, the spectrograms overlap with each other in part, and then are sent to three networks respectively. The three networks extract features of different frequency bands respectively, thereby reducing the complexity of the network and improving the accuracy of speech emotion recognition; by adding a lightweight attention mechanism CBAM to the network, the expressiveness of key point features is improved, and at the same time, unimportant features (ambient audio, noise) are suppressed, and high-value information is quickly screened out from a large amount of information with limited attention resources, which can greatly improve the efficiency and accuracy of information processing.

[0109] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0110] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A speech emotion recognition method based on spectrum segmentation and deep learning, characterized in that: The following steps are involved: S1. Obtain the ESD emotion dataset and read the dataset to obtain audio data; S2. Preprocess the audio data to obtain a grayscale spectrogram, specifically: S21, reading audio data and converting the audio data into one-dimensional data; S22, performing frame division and windowing processing on the one-dimensional data to convert it into two-dimensional data; S23, performing a fast Fourier transform on the two-dimensional data to obtain a short-time Fourier spectrum; S24, performing a multiplication operation on the short-time Fourier spectrum to obtain a short-time power spectrum; S25, converting the value of the short-time power spectrum into a two-dimensional image composed of gray levels to obtain a grayscale spectrogram; S3. Perform spectrum segmentation on the grayscale spectrogram according to different frequency bands of the grayscale spectrogram to obtain a segmentation result, specifically: S31, binarizing the grayscale spectrogram to obtain a binarized grayscale spectrogram; S32. Calculating based on the binarized grayscale spectrogram to obtain an average energy of the grayscale spectrogram; S33, plotting the average energy of the grayscale spectrogram to obtain an envelope of the average energy; S34, performing differentiation on the envelope to obtain a differential curve; S35. Obtaining the zero point of the first-order differential envelope according to the differential curve; S36, selecting the zero point of the first-order difference envelope as a dividing point to perform spectrum segmentation on the envelope to obtain a segmentation result; S4. Select different networks based on the segmentation results to perform sentiment classification to obtain classification results; S5. Output the classification results to obtain emotion recognition data.

2. The speech emotion recognition method based on spectrum segmentation and deep learning according to claim 1, characterized in that: The step S31 includes: S311, reading the grayscale spectrogram to obtain matrix data of the grayscale spectrogram; S312: Setting a division threshold, binarizing the grayscale spectrogram based on the matrix data and the pixel level to obtain a binarized grayscale spectrogram.

3. The speech emotion recognition method based on spectrum segmentation and deep learning according to claim 1, wherein: The process of calculating the average energy of the grayscale spectrogram according to the binarized grayscale spectrogram in step S32 includes: The binary grayscale spectrogram is converted into a matrix form, and the energy value of each small frequency band is obtained row by row, and saved in order from low frequency to high frequency. The above operation is repeated until all the data in the dataset are processed. Then the energy values ​​of all the binary grayscale spectrograms are summed and their average is calculated to obtain the average energy of the grayscale spectrogram.

4. The speech emotion recognition method based on spectrum segmentation and deep learning according to claim 1, wherein: The process of plotting the average energy of the grayscale spectrogram to obtain the envelope of the average energy in step S33 includes: S331, plotting based on the average energy of the grayscale spectrogram to obtain an average energy curve; S332, obtaining the maximum point and the minimum point of the average energy curve to obtain the horizontal and vertical coordinates corresponding to the maximum point and the minimum point of the average energy curve; S333. Draw a plot based on the horizontal and vertical coordinates corresponding to the maximum and minimum points of the average energy curve to obtain an envelope curve of the average energy.

5. The method for speech emotion recognition based on spectrum segmentation and deep learning according to claim 4, wherein: The step S4 comprises: S41, sending the segmentation results to different networks respectively to obtain output results of different networks; S42, aggregating output results of different networks to obtain an aggregated result; S43. Perform sentiment classification on the aggregation results to obtain the final classification results.

6. The method for speech emotion recognition based on spectrum segmentation and deep learning according to claim 5, wherein: The network used in step S41 is a ResNet34 network, to which a lightweight attention mechanism CBAM is added.

7. An electronic device, characterized in that: include: a memory for storing one or more programs; processor; When the one or more programs are executed by the processor, a speech emotion recognition method based on spectrum segmentation and deep learning as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for speech emotion recognition based on spectrum segmentation and deep learning as described in any one of claims 1 to 6 is implemented.

Images