A speech emotion recognition method based on multi-source features and application

Abstract

The application discloses a speech emotion recognition method based on multi-source features, comprising the following steps: collecting an open-source speech recognition dataset and an environmental sound recognition dataset, respectively training a speech recognition network model and an environmental sound recognition network model, and collecting an open-source emotional speech dataset for audio enhancement; extracting MFCC features from the enhanced emotional speech audio and inputting the features into an inflation causal interaction subnetwork to obtain high-order MFCC features; inputting the enhanced emotional speech audio into the speech recognition network model and the environmental sound recognition network model to obtain high-order text features and high-order environmental features, then inputting the features into a time sequence decomposition causal convolution block after being fused with the high-order MFCC features to generate high-order time sequence causal composite features, and outputting the features to a classification network layer through a full connection layer to recognize speech emotion. The method can solve the problem that traditional speech emotion recognition technology is only based on acoustic features of speech, depends on standard clean speech signals, and has low speech emotion recognition accuracy and cannot accurately recognize speech emotion in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a speech emotion recognition method based on multi-source features, a speech emotion recognition device based on multi-source features, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of human-computer interaction technology, voice interaction applications are becoming increasingly widespread, and voice emotion recognition technology is also receiving more and more attention.

[0003] In voice interaction, recognizing a user's emotional state can enable more intelligent human-computer interaction. However, traditional voice emotion recognition technology is mainly based on the acoustic features of speech, neglecting the important role of textual semantic information in emotion recognition. Furthermore, existing technologies largely rely on standard, clean speech signals, over-emphasizing the audio characteristics of the speech itself, without considering the impact of environmental sounds on the expression of emotions in real-world scenarios. As a result, many existing voice emotion recognition methods have low accuracy, cannot accurately identify voice emotions in complex environments, and perform poorly in enhancing emotional speech recognition, making it difficult to meet the needs of practical interactive applications. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of existing technologies, this invention proposes a speech emotion recognition method and application based on multi-source features. It aims to solve the problem that traditional speech emotion recognition technology is based only on the acoustic features of speech and relies on standard clean speech signals, resulting in low accuracy of speech emotion recognition and inability to accurately recognize speech emotions in complex environments.

[0005] On one hand, this invention proposes a speech emotion recognition method based on multi-source features, comprising: collecting open-source speech recognition datasets and environmental sound recognition datasets and pre-training them to obtain speech recognition network models and environmental sound recognition network models respectively; collecting open-source emotion speech datasets and performing audio enhancement processing to obtain enhanced emotion speech audio; extracting MFCC features from the enhanced emotion speech audio and inputting them into an inflated causal interaction subnetwork to obtain higher-order MFCC features; inputting the enhanced emotion speech audio into the pre-trained speech recognition network model and environmental sound recognition network model to obtain higher-order text features and higher-order environmental features respectively, and then fusing them with the higher-order MFCC features; inputting the fused features into a temporal decomposition causal convolution block to generate higher-order temporal causal composite features, and outputting them to a classification network layer through a fully connected layer to classify and recognize speech emotions.

[0006] In one embodiment of the present invention, the process of collecting an open-source emotional speech dataset and performing audio enhancement processing to obtain enhanced emotional speech audio includes: adding white noise to the collected open-source emotional speech dataset to achieve a preset signal-to-noise ratio, temporal random jitter, and frequency domain jitter, thereby obtaining the enhanced emotional speech audio.

[0007] In one embodiment of the present invention, the environmental sound recognition network model consists of a stack of four layers of bidirectional gated recurrent unit memory networks with dimensions of 1024, and two fully connected layers with dimensions of 1024 and 7 respectively.

[0008] In one embodiment of the present invention, the step of extracting MFCC features from the enhanced emotional speech audio and inputting them into a dilated causal interaction subnetwork to obtain higher-order MFCC features includes: dividing the enhanced emotional speech audio into frames, applying a Hamming window to perform a Fourier transform on each frame signal to obtain a spectral representation, then filtering the spectrum using a Mel filter bank and taking the logarithm, and performing a discrete cosine transform to obtain the MFCC features; inputting the MFCC features into a dilated causal interaction subnetwork to obtain higher-order MFCC features, wherein the dilated causal interaction subnetwork is a convolutional network composed of dilated convolution and causal convolution structures.

[0009] In one embodiment of the present invention, the dilated causal interaction subnetwork consists of two layers of dilated causal networks. The input sequence of the dilated causal network first captures contextual information in different ranges through a dilated convolution kernel with a dilation rate of 4, then inputs a causal convolution padding on the right to keep the order of the input sequence unchanged, and finally performs a batch normalization operation.

[0010] In one embodiment of the present invention, the step of inputting the enhanced emotional speech audio into the pre-trained speech recognition network model and the environmental sound recognition network model to obtain high-order text features and high-order environmental features respectively, and then fusing them with the high-order MFCC features, includes: inputting the enhanced emotional speech audio into the pre-trained speech recognition network model to extract the features of the layer preceding the last layer of the model, i.e., the high-order text features; inputting the enhanced emotional speech audio into the pre-trained environmental sound recognition network model to extract the features of the layer preceding the last layer of the model, i.e., the high-order environmental features; and performing maximum value fusion of the high-order MFCC features, the high-order text features, and the high-order environmental features to obtain the maximum value fused features.

[0011] In one embodiment of the present invention, the step of inputting the fused features into a temporal decomposition causal convolution block to generate a high-order temporal causal composite feature includes: decomposing the maximum fused feature into two sub-time series, an odd sequence and an even sequence, wherein the odd sequence contains elements with an odd number of time steps and the even sequence contains elements with an even number of time steps; mapping the odd sequence and the even sequence to two hidden states using causal convolution, and multiplying each of them element-wise with the hidden state of another sequence; performing a residual concatenation between the result of the element-wise multiplication and the original sequence to obtain an output odd sequence and an output even sequence; and superimposing the output odd sequence and the output even sequence to obtain the high-order temporal causal composite feature.

[0012] On the other hand, this invention also proposes a speech emotion recognition device based on multi-source features, comprising: a model pre-training module, used to collect open-source speech recognition datasets and environmental sound recognition datasets and train pre-trained speech recognition network models and environmental sound recognition network models respectively, and to collect open-source emotion speech datasets for audio enhancement processing to obtain enhanced emotion speech audio; a high-order feature acquisition module, used to extract MFCC from the enhanced emotion speech audio, input it into the dilated causal interaction subnetwork to obtain high-order MFCC features; a high-order feature fusion module, used to input the enhanced emotion speech audio into the pre-trained speech recognition network model and the environmental sound recognition network model to obtain high-order text features and high-order environmental features respectively, and then fuse them with the high-order MFCC features; and a speech emotion recognition module, used to input the fused features into a temporal decomposition causal convolution block to generate high-order temporal causal composite features, and output them to a classification network layer through a fully connected layer to classify and recognize speech emotions.

[0013] In another aspect, embodiments of the present invention also provide an electronic device, comprising: a memory and one or more processors connected to the memory, the memory storing a computer program, and the processors being configured to execute the computer program to implement the steps of the method described in any of the foregoing embodiments.

[0014] In another aspect, embodiments of the present invention also provide a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the method described in any of the foregoing embodiments.

[0015] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1) The speech emotion recognition method based on multi-source features proposed in this invention establishes an end-to-end framework for task learning and higher-order learning based on multi-source features, and integrates complementary training of speech, text and environment information, so that the speech emotion recognition and classification can adapt to complex environments and has high robustness. Compared with the traditional method of speech emotion recognition based on a single acoustic signal, it can significantly improve the recognition accuracy. 2) By adding white noise to the collected open-source emotional speech dataset to achieve the preset signal-to-noise ratio, temporal random jitter, and frequency domain jitter, enhanced emotional speech audio can be obtained. Without changing the audio content, the model's ability to learn temporal information can be enhanced by disrupting the continuity of the original speech in the temporal domain, thus improving the robustness to temporal jitter and enhancing the model's generalization ability. 3) The dilated causal interaction subnetwork for high-order feature training consists of two layers of dilated causal networks. The input sequence of the dilated causal network first captures contextual information in different ranges through dilated convolutional kernels with a dilation rate of 4, and then enters a causal convolutional layer that only fills the right side to ensure that the sequence order remains unchanged. Finally, batch normalization is performed. This expands the receptive field and maintains temporal consistency, achieving efficient sequence modeling. 4) By fusing high-order text features, high-order environment features, and high-order MFCC features at their maximum values, the most significant features among different considerations can be highlighted, maximizing the effectiveness of multi-source feature fusion and resulting in more accurate feature recognition. 5) By decomposing the maximum value fusion feature into two sub-time series, odd and even, and passing the odd and even sequences through independent causal convolutional layers, richer feature learning is achieved, which ultimately effectively improves the accuracy of speech emotion recognition and classification. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a speech emotion recognition method based on multi-source features provided in an embodiment of the present invention; Figure 2 A schematic diagram of the network architecture for a speech emotion recognition method based on multi-source features provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a speech emotion recognition device based on multi-source features provided in an embodiment of the present invention; Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] In one embodiment, such as Figure 1 As shown, the first embodiment of the present invention provides a speech emotion recognition method based on multi-source features, including, for example: Step S1, collecting open-source speech recognition datasets and environmental sound recognition datasets and pre-training them to obtain speech recognition network models and environmental sound recognition network models respectively, and collecting open-source emotion speech datasets for audio enhancement processing to obtain enhanced emotion speech audio; Step S2, extracting MFCC features from the enhanced emotion speech audio and inputting them into an inflated causal interaction subnetwork to obtain higher-order MFCC features; Step S3, inputting the enhanced emotion speech audio into the pre-trained speech recognition network model and environmental sound recognition network model to obtain higher-order text features and higher-order environmental features respectively, and then fusing them with the higher-order MFCC features; Step S4, inputting the fused features into a temporal decomposition causal convolution block to generate higher-order temporal causal composite features, and outputting them to a classification network layer through a fully connected layer to classify and recognize speech emotions.

[0020] Step S1 specifically includes the following sub-steps: Step S11: Collect an open-source speech recognition dataset and train the Whisper model: By searching the internet for open-source speech recognition datasets, such as Speechocean, Primewords Chinese Corpus Set 1, and MagicData-RAMC, a Whisper speech recognition network model was trained to obtain a pre-trained speech recognition network model. .

[0021] Step S12: Collect an open-source environmental sound recognition dataset: By searching the internet for open-source environmental sound recognition datasets, such as UrbanSound8K, ESC-50, and SINS, we can identify environmental sound recognition data. .

[0022] Step S13: Construct an environmental sound recognition network model and train it using environmental sound data. The data obtained in step S12 Train the environmental sound classification and recognition network model to obtain the pre-trained environmental sound recognition network model. .

[0023] Step S14: Collect emotional speech data and perform audio enhancement to obtain enhanced emotional speech audio: We collect open-source emotional speech datasets from the internet and obtain enhanced emotional speech audio through enhancement operations. The enhancement operation refers to amplifying the energy of each waveform point in the audio signal and enhancing the waveform points in time using a time-dependent linear function, and then inputting the features into the dynamic feature extraction subnetwork.

[0024] In one implementation, for example, the acquired open-source emotional speech dataset is processed by adding white noise to achieve a preset signal-to-noise ratio, temporal random jitter, and frequency jitter to obtain the enhanced emotional speech audio. Specifically, for example, a target signal-to-noise ratio (SNR) is first set, and then the speech signal is calculated. The energy generates a... Equal-length random white noise is generated, and its energy is calculated. Based on the signal energy, noise energy, and target signal-to-noise ratio (SNR), a scaling factor can be calculated to proportionally scale the noise to meet the SNR requirement. This scaling factor is used to adjust the noise amplitude, and then the scaled noise is directly superimposed onto the speech signal. The above steps involve adding noise to the signal to obtain a noisy speech signal with a signal-to-noise ratio (SNR). .

[0025] Furthermore, for the input noisy speech signal Dithering in the time domain is used to enhance the data. Specifically, a random time-domain offset is generated, and then the speech audio samples are shifted in the time domain according to this random offset. This enhances the model's ability to learn temporal information by disrupting the continuity of the original speech in the time domain without changing the audio content, thus improving robustness to time-domain dithering and enhancing the model's generalization ability. Time-domain dithering is used in conjunction with frequency-domain dithering to obtain enhanced audio data. .

[0026] Combination Figure 2 As shown, in one embodiment, the environmental sound recognition network model is composed of, for example, four layers of bidirectional gated recurrent unit memory networks with dimensions of 1024 stacked together, and two fully connected layers with dimensions of 1024 and 7 respectively.

[0027] Furthermore, step S2 specifically includes, for example, the following sub-steps: Step S21: Preprocess the emotional audio data to obtain Mel-frequency cepstral coefficient features (MFCC): Enhanced emotional voice audio The signal is segmented into frames, and a Hamming window is applied to each frame. A Fourier transform is then performed to obtain the spectral representation. The spectrum is then filtered using a Mel filter bank, and the logarithm is taken. Finally, a discrete cosine transform is performed to extract the Mel cepstral coefficient features, thus obtaining the Mel cepstral coefficient features of speech emotion. .

[0028] Step S22: Input the Mel-Cepstral Coefficient Features into the Dilated Causal Interaction Subnetwork to obtain the Higher-Order Mel-Cepstral Coefficient Features: The result obtained in step S21 The input is fed into the dilated causal interaction subnetwork to obtain higher-order Mel-frequency cepstral coefficient features. The dilated causal interaction subnetwork is a convolutional network composed of dilated convolution and causal convolution structures.

[0029] Furthermore, the dilated causal interaction subnetwork consists of two layers of dilated causal networks. Specifically, the input sequence of the dilated causal network first captures contextual information within different ranges through a dilated convolutional kernel with a dilation rate of 4, then enters a causal convolutional layer that only fills the right side to ensure that the sequence order remains unchanged, and finally performs batch normalization. This expands the receptive field while maintaining temporal consistency, achieving efficient sequence modeling.

[0030] Furthermore, step S3 specifically includes, for example, the following sub-steps: Step S31, enhance emotional voice audio Input into the pre-trained speech recognition network model This paper proposes the features of the layer preceding the last layer of the network in this model, namely, high-order text features. ; Step S32 will enhance the emotional voice audio. Input into the pre-trained ambient sound network model This paper proposes the features of the layer preceding the last layer of the network in this model, namely, the high-order environmental features. ; Step S33: Integrate higher-order Mel-frequency cepstral coefficient features, higher-order text features, and higher-order environmental features: The higher-order Mel-frequency cepstral coefficient features obtained in step S31 The high-order text features obtained in step S32 and the high-order environmental features obtained in step S33 Maximal fusion can highlight the most prominent features among different modes, maximize the utility of each mode, and obtain the maximal fusion feature. The specific expression is: .

[0031] In step S4, the maximum value fusion feature is... The input is fed into a temporal decomposition causal convolutional block to generate higher-order composite features. The input to the temporal decomposition causal convolutional block is the maximum value fused time series. ,in, arrive The input elements are arranged in chronological order, and the maximum value is merged into the time series. The sequence was decomposed into two sub-time series, odd sequence , Include Elements with odd time steps and even sequences , Include Elements with even time steps.

[0032] By decomposing the time series as described above, we can focus on different feature subspaces. Using causal convolution CConv... and The features are mapped to two hidden states respectively. Causal convolution only depends on the current and previous elements, maintaining the temporal consistency of the sequence features. Then, the transformed features are multiplied element-wise with the hidden states of the other sequence. The result of the element-wise multiplication is residually connected to the original sequence. Odd and even sequences are passed through independent causal convolutional layers, which can learn richer features. The calculation formula is shown below: in, To output an odd sequence, Given an odd sequence as input, For residual connections, C This represents causal convolution, and ⊙ represents element-wise multiplication. Given an even sequence as input, To output an even sequence; Then output the odd sequence and output even sequence Superposition yields higher-order composite features. The specific expression is: .

[0033] Furthermore, higher-order composite features The input is fed into two fully connected network layers, and the output is processed through a Softmax network layer for speech emotion classification.

[0034] In summary, the speech emotion recognition method based on multi-source features proposed in the first embodiment of this invention establishes an end-to-end framework for task learning and higher-order learning using multi-source features. It integrates complementary training using information from speech, text, and environment, enabling speech emotion recognition and classification to adapt to complex environments and exhibit high robustness. Compared to traditional methods that rely on a single acoustic signal for speech emotion recognition, it significantly improves recognition accuracy. By adding white noise to the collected open-source emotion speech dataset to achieve a preset signal-to-noise ratio, temporal random jitter, and frequency domain jitter, enhanced emotion speech audio is obtained. This enhances the model's ability to learn temporal information and improves robustness to temporal jitter without altering the audio content, by disrupting the temporal continuity of the original speech. It also enhances the model's generalization ability. The method employs dilated causal crosstalk for higher-order feature training. The mutual sub-network consists of two layers of dilated causal networks. The input sequence of the dilated causal network first passes through a dilated convolutional kernel with a dilation rate of 4 to capture contextual information within different ranges. Then, it enters a causal convolutional layer that only fills the right side to ensure that the sequence order remains unchanged. Finally, a batch normalization operation is performed. This expands the receptive field while maintaining temporal consistency, achieving efficient sequence modeling. By fusing high-order text features, high-order environmental features, and high-order MFCC features with maximum values, the most significant features among different considerations can be highlighted, maximizing the effectiveness of multi-source feature fusion and resulting in more accurate feature recognition. By decomposing the maximum value fused features into two sub-time series, odd and even, which are then passed through independent causal convolutional layers, richer feature learning is achieved, ultimately effectively improving the accuracy of speech emotion recognition and classification.

[0035] In addition, such as Figure 3 As shown, the second embodiment of the present invention proposes a speech emotion recognition device 20 based on multi-source features, which includes, for example, a model pre-training module 201, a high-order feature acquisition module 202, a high-order feature fusion module 203, and a speech emotion recognition module 204.

[0036] The model pre-training module 201 collects open-source speech recognition datasets and environmental sound recognition datasets and trains them to obtain pre-trained speech recognition network models and environmental sound recognition network models, respectively. It also collects open-source emotion speech datasets and performs audio enhancement processing to obtain enhanced emotion speech audio. The high-order feature acquisition module 202 extracts MFCCs from the enhanced emotion speech audio and inputs them into the dilated causal interaction subnetwork to obtain high-order MFCC features. The high-order feature fusion module 203 inputs the enhanced emotion speech audio into the pre-trained speech recognition network model and the environmental sound recognition network model to obtain high-order text features and high-order environmental features, which are then fused with the high-order MFCC features. The speech emotion recognition module 204 inputs the fused features into a temporal decomposition causal convolutional block to generate high-order temporal causal composite features, and outputs them through a fully connected layer to a classification network layer to classify and recognize the speech emotion.

[0037] It is worth mentioning that the speech emotion recognition method based on multi-source features disclosed in the second embodiment of the present invention is the same as that described in the first embodiment above, and therefore will not be described in detail here. Optionally, each component and the other operations or functions described above are for implementing the method described in the first embodiment, and the beneficial effects of this embodiment are the same as those of the first embodiment above. For the sake of brevity, they will not be repeated here.

[0038] like Figure 4 As shown, the third embodiment of the present invention also provides an electronic device, which includes: a memory and one or more processors connected to the memory, the memory storing a computer program, and the processor executing the computer program to implement the steps of the speech emotion recognition method based on multi-source features described in the first embodiment. The electronic device provided in this embodiment has the same beneficial effects as the speech emotion recognition method based on multi-source features provided in the first embodiment.

[0039] like Figure 5 As shown, the fourth embodiment of the present invention also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps of the speech emotion recognition method based on multi-source features described in the first embodiment. The computer-readable storage medium provided in this embodiment has the same beneficial effects as the speech emotion recognition method based on multi-source features provided in the first embodiment.

[0040] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0041] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0042] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0043] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0044] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0045] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0046] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech emotion recognition method based on multi-source features, characterized in that, include: We collected open-source speech recognition datasets and environmental sound recognition datasets and pre-trained them to obtain speech recognition network models and environmental sound recognition network models, respectively. We also collected open-source emotion speech datasets and performed audio enhancement processing to obtain enhanced emotion speech audio. MFCC features are extracted from the enhanced emotion speech audio and input into the dilated causal interaction subnetwork to obtain higher-order MFCC features; The enhanced emotion speech audio is input into the pre-trained speech recognition network model and the environmental sound recognition network model to obtain high-order text features and high-order environmental features, which are then fused with the high-order MFCC features. The fused features are input into a temporal decomposition causal convolutional block to generate high-order temporal causal composite features, which are then output through a fully connected layer to a classification network layer for speech emotion recognition; including: The higher-order MFCC features, the higher-order text features, and the higher-order environment features are fused by maximizing the maximum value to obtain the maximum value fused feature; The maximum value fusion feature is decomposed into two sub-time series: an odd sequence and an even sequence. The odd sequence contains elements with an odd number of time steps, and the even sequence contains elements with an even number of time steps. Causal convolution is used to map the odd sequence and the even sequence to two hidden states respectively, and each is multiplied element-wise with the hidden state of another sequence. The result of the element-wise multiplication is residually concatenated with the original sequence to obtain the output odd sequence and the output even sequence. The higher-order temporal causal composite feature is obtained by superimposing the output odd sequence and the output even sequence.

2. The speech emotion recognition method based on multi-source features according to claim 1, characterized in that, The collected open-source emotional speech dataset is subjected to audio enhancement processing to obtain enhanced emotional speech audio, including: The collected open-source emotional speech dataset is processed by adding white noise to achieve a preset signal-to-noise ratio, temporal random jitter, and frequency domain jitter to obtain the enhanced emotional speech audio.

3. The speech emotion recognition method based on multi-source features according to claim 1, characterized in that, The environmental sound recognition network model consists of a stack of four bidirectional gated recurrent unit memory networks with a dimension of 1024, and two fully connected layers with dimensions of 1024 and 7 respectively.

4. The speech emotion recognition method based on multi-source features according to claim 1, characterized in that, The step of extracting MFCC features from the enhanced emotion speech audio and inputting them into the dilated causal interaction subnetwork to obtain high-order MFCC features includes: The enhanced emotion speech audio is divided into frames, and a Hamming window is applied to the Fourier transform of each frame signal to obtain the spectrum representation. Then, the spectrum is filtered by the Mel filter bank and the logarithm is taken. After performing discrete cosine transform, the MFCC feature is obtained. The MFCC features are input into the dilated causal interaction subnetwork to obtain higher-order MFCC features. The dilated causal interaction subnetwork is a convolutional network composed of dilated convolution and causal convolution structures.

5. The speech emotion recognition method based on multi-source features according to claim 1, characterized in that, The dilated causal interaction subnetwork consists of two layers of dilated causal networks. The input sequence of the dilated causal network first captures contextual information within different ranges through a dilated convolution kernel with a dilation rate of 4, then inputs a causal convolution padding on the right to keep the order of the input sequence unchanged, and finally performs a batch normalization operation.

6. The speech emotion recognition method based on multi-source features according to claim 1, characterized in that, The process of inputting the enhanced emotion-enhanced speech audio into the pre-trained speech recognition network model and the environmental sound recognition network model, obtaining high-order text features and high-order environmental features respectively, and then fusing them with the high-order MFCC features includes: The enhanced emotional speech audio is input into the pre-trained speech recognition network model, and the features of the layer preceding the last layer of the network are extracted, namely the high-order text features. The enhanced emotional speech audio is input into the pre-trained environmental sound recognition network model, and the features of the layer preceding the last layer of the network are extracted, namely the high-order environmental features.

7. A speech emotion recognition device based on multi-source features, characterized in that, include: The model pre-training module is used to collect open-source speech recognition datasets and environmental sound recognition datasets and train them to obtain pre-trained speech recognition network models and environmental sound recognition network models, respectively. It also collects open-source emotion speech datasets and performs audio enhancement processing to obtain enhanced emotion speech audio. The high-order feature acquisition module is used to extract MFCC from the enhanced emotion speech audio, input it into the dilated causal interaction subnetwork, and obtain high-order MFCC features. The high-order feature fusion module is used to input the enhanced emotion speech audio into the pre-trained speech recognition network model and the environmental sound recognition network model to obtain high-order text features and high-order environmental features respectively, and then fuse them with the high-order MFCC features. The speech emotion recognition module is used to input the fused features into a temporal decomposition causal convolutional block to generate high-order temporal causal composite features, and then output them to a classification network layer through a fully connected layer to classify and recognize speech emotions; wherein, the generation process of the high-order temporal causal composite features is as follows: The higher-order MFCC features, the higher-order text features, and the higher-order environment features are fused by maximizing the maximum value to obtain the maximum value fused feature; The maximum value fusion feature is decomposed into two sub-time series: an odd sequence and an even sequence. The odd sequence contains elements with an odd number of time steps, and the even sequence contains elements with an even number of time steps. Causal convolution is used to map the odd sequence and the even sequence to two hidden states respectively, and each is multiplied element-wise with the hidden state of another sequence. The result of the element-wise multiplication is residually concatenated with the original sequence to obtain the output odd sequence and the output even sequence. The higher-order temporal causal composite feature is obtained by superimposing the output odd sequence and the output even sequence.

8. An electronic device, characterized in that, include: A memory and one or more processors connected to the memory, the memory storing a computer program, the processors being configured to execute the computer program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the method according to any one of claims 1-6.

Images