Speech emotion recognition method based on hollow depth separable group convolution lightweight network
By using a lightweight, depth-separable group convolutional network with dilatancy, and employing convolutional operations of dimensionality reduction, splitting, filtering, transformation, and merging, the high complexity of existing speech emotion recognition algorithms is addressed, resulting in more efficient speech emotion recognition performance and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2022-12-27
- Publication Date
- 2026-05-29
AI Technical Summary
Existing speech emotion recognition algorithms are highly complex, resulting in large computational loads and significant errors. Traditional networks are not suitable for existing models, and GAP technology disrupts network layout and increases the number of parameters.
A lightweight network based on dilated depth separable group convolution is adopted. The feature map is transformed by pointwise group convolutional layers through dimensionality reduction and decomposition, combined with dilated depth separable convolutional layers. The feature map is filtered by a similarity judgment module, and hierarchical feature fusion and downsampling learning are performed to generate the final output feature for sentiment prediction.
While reducing computational load, it improves the accuracy and performance of speech emotion recognition, reduces the computational load of unnecessary feature maps, and provides better spatial information encoding capabilities.
Smart Images
Figure CN115985345B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing, deep learning, and speech emotion recognition, and particularly to a speech emotion recognition method based on a lightweight, dilated, depth-separable group convolutional network. Background Technology
[0002] Today, with the development of technology, human-computer interaction systems are gradually entering our lives. People are increasingly accustomed to using voice commands to control smart devices in their homes, answer calls from robots, and converse with service robots to obtain information. However, the same sentence can express different meanings under different emotions, and speech recognition technology alone cannot achieve a high-quality human-computer interaction experience. Therefore, enabling machines to understand the emotions people express when they speak is of great significance. Emotion recognition technology, as one of the key technologies in human-computer interaction systems, can accurately identify emotions and help machines better understand the user's intentions, thereby improving the quality of human-computer interaction. Currently, emotions can be identified in many ways, such as by analyzing tone attributes, facial expressions, and body gestures. As the primary medium of communication, speech contains rich emotional information, making it the best choice for emotion recognition. With the rapid development of voice-based human-computer interaction applications (including intelligent service robots, automated call centers, and distance education), speech emotion recognition (SER) technology has attracted increasing interest from researchers over the past two decades.
[0003] The significance of exploring speech emotion recognition technology lies in the following aspects: First, from a theoretical perspective, it explores a more complete theoretical system, providing theoretical guidance for creating a more accurate and stable speech emotion recognition system. Second, it explores robust and accurate algorithms to overcome many challenges in speech emotion recognition, making intelligent service robots, automated call centers, and distance education more intelligent and efficient, while also promoting the development of research related to computer speech emotion understanding. Finally, in the process of speech emotion recognition, it verifies and improves relevant machine learning theories, seeking a suitable solution from practical applications to address the more abstract scientific research problems involved in the research process.
[0004] One challenge in speech emotion recognition is the abundance of redundant information in audio signals, leading to errors in emotion feature recognition and weight allocation, thus affecting the final accuracy. A second challenge is that most recent research on speech emotion recognition algorithms has focused on improving performance. Analysis of existing research reveals that current research has achieved a relatively high level of accuracy in speech emotion recognition. To further improve performance, most researchers have delved deeper into convolutional algorithms, resulting in high complexity in existing speech emotion recognition models such as ResNet and DenseNet. A common method to reduce algorithm complexity is Global Average Pooling (GAP). GAP significantly reduces computational cost, but it directly discards fully connected layers (FC), disrupting the original network layout. Furthermore, GAP requires sufficient channel size (e.g., 1024 or 2048) and large feature maps (e.g., 7 × 7) in the final transformation layer to ensure good performance. Finally, such a large channel size will significantly increase the parameters of the last classification layer, so GAP technology is not well-suited for existing traditional networks. Summary of the Invention
[0005] To address the aforementioned problems in the prior art, this invention proposes a speech emotion recognition method based on a lightweight, depth-separable group convolutional network, comprising the following steps:
[0006] S1: Short-term Fourier transform is used to perform time-frequency conversion on the audio signal to obtain its input feature map;
[0007] S2: The input feature map is reduced in dimensionality and decomposed by a pointwise group convolutional layer to obtain several low-dimensional feature maps;
[0008] S3: The low-dimensional feature map is transformed into several high-dimensional feature maps of the same dimension as the input feature map by a dilatancy-separable convolutional layer;
[0009] S4: Use a similarity determination module to determine whether the high-dimensional feature map conforms to the emotion classification standard;
[0010] S5: A hierarchical feature fusion technique is used to concatenate several groups of low-dimensional and high-dimensional feature maps to generate an output feature map;
[0011] S6: The spatiotemporal characteristics of the audio signal are learned by downsampling. At the same time, the input feature map is passed through the average pooling layer to achieve dimension matching. Finally, the pooled input feature map is concatenated and superimposed with the output feature map of the hierarchical fusion output to obtain the final output feature.
[0012] S7: Input the final output features into the linear layer. The function performs emotion prediction to obtain the emotion recognition result of the audio signal. The beneficial effects of this invention are:
[0013] To address the high complexity of existing speech emotion recognition algorithms, this invention proposes a lightweight, low-power deep dilated separable convolutional neural network. The convolution operation employs dimensionality reduction, splitting, filtering, transformation, and merging. Extended separable convolutions in the group and depth directions are used to learn representations from a larger effective receptive field. A decision module filters feature maps, dilating unnecessary ones. Without reducing discrimination accuracy, it better encodes the spatial information of the spectrogram, providing better performance and significantly reducing computational cost. Attached Figure Description
[0014] Figure 1 This is the overall flowchart of the present invention;
[0015] Figure 2 This is a schematic diagram of the group convolution (P=3) used in this invention;
[0016] Figure 3 This is a diagram of the depthwise convolutional structure used in this invention;
[0017] Figure 4 This is a block diagram of the speech emotion recognition method based on a lightweight, hole-depth separable group convolutional network according to the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] Figure 1 This is a general flowchart of the present invention, as shown below. Figure 1 As shown, this invention proposes a speech emotion recognition method based on a lightweight, depth-separable group convolutional network, comprising the following steps:
[0020] S1: Short-term Fourier transform is used to perform time-frequency conversion on the audio signal to obtain its input feature map;
[0021] S2: The input feature map is reduced in dimensionality and decomposed by a pointwise group convolutional layer to obtain several low-dimensional feature maps;
[0022] S3: The low-dimensional feature map is transformed into several high-dimensional feature maps of the same dimension as the input feature map by a dilatancy-separable convolutional layer;
[0023] S4: Use a similarity determination module to determine whether the high-dimensional feature map conforms to the emotion classification standard;
[0024] S5: A hierarchical feature fusion technique is used to concatenate several groups of low-dimensional and high-dimensional feature maps to generate an output feature map;
[0025] S6: The spatiotemporal characteristics of the audio signal are learned by downsampling. At the same time, the input feature map is passed through the average pooling layer to achieve dimension matching. Finally, the pooled input feature map is concatenated and superimposed with the output feature map of the hierarchical fusion output to obtain the final output feature.
[0026] S7: Input the final output features into the linear layer. The function performs emotion prediction to obtain the emotion recognition result of the audio signal.
[0027] in, This represents the normalized exponential function.
[0028] In this embodiment, a Hamming window function and a short-term Fourier transform are used to perform time-frequency conversion on the input audio signal, and the original input feature map is obtained from the audio signal through time-frequency conversion. The process includes:
[0029] S11: Using the Hamming window function to segment the original audio can solve the spectral leakage and aliasing phenomena that occur during the Fourier transform. The calculation formula is as follows:
[0030]
[0031] Where N is the length of the signal intercepted by the Hamming window;
[0032] S12: Perform a short-term Fourier transform on the segmented audio signal to obtain the low-level feature map of the original audio. Where W, H, and M represent the input feature maps, respectively. Width, height, and dimensions.
[0033] The short-term Fourier transform is used to obtain the frequency domain representation of the audio signal, and its calculation formula is as follows:
[0034]
[0035]
[0036] in, A function representing the time-domain characteristics of an audio signal. A frequency domain characteristic function representing an audio signal.
[0037] This embodiment uses pointwise group convolutional layers to process the original input feature map after time-frequency transformation. Preprocessing is performed using pointwise group convolutional layers on the original input feature map. Dimensionality reduction and decomposition are performed to obtain several low-dimensional feature maps. ,like Figure 3 As shown, Figure 3 This is a schematic diagram of a depthwise convolutional structure. The specific steps are shown below:
[0038] A pointwise group convolutional layer consists of P sets of convolutional kernels and activation layers. Given P, it utilizes P convolutional kernels... For the original feature map Dimensionality is reduced by performing pointwise group convolution. After the convolution operation, the feature maps are activated by an activation function through an activation layer to obtain P low-dimensional d-dimensional feature maps. Where P is an introduced hyperparameter, its function is to uniformly reduce the dimensionality of the feature mapping, and N is the dimension of the final output feature map. For example... Figure 2 As shown, Figure 2 This is a schematic diagram of a group convolution with hyperparameter P=3, where the activation function is shown below:
[0039]
[0040] This embodiment uses a deep-dilated separable convolutional layer to process low-dimensional feature maps. This is transformed into several feature maps of the same dimension as the original input feature map. The specific steps are as follows:
[0041] S31: Convert P low-dimensional convolutional images Each branch uses a dilated convolution kernel simultaneously. The void ratio r is Its receptive field increases to .
[0042] S32: Low-dimensional feature map After dilated convolutional kernels, depthwise separable convolutional layers are then used to process the low-dimensional feature maps. Convert to high-dimensional feature map .
[0043] The similarity determination module is used to determine whether the feature map meets the emotion classification criteria.
[0044] S41: Input P feature maps into the similarity determination module, and calculate the peak signal-to-noise ratio between each feature map. The unit is usually dB, and its calculation formula is:
[0045]
[0046] Where MSE is the size of two feature maps I and J. The mean square error (MSE) is the maximum possible value of an image pixel. The MSE calculation formula is:
[0047]
[0048] S42: Calculate the average peak signal-to-noise ratio (PSNR) of each feature map with the other P-1 feature maps. We set the PSNR between the I-th feature map and the J-th feature map as PSNR. (I,J) The formula for calculating the mean peak signal-to-noise ratio of the I-th feature map is:
[0049]
[0050] When the mean peak signal-to-noise ratio of the I-th high-dimensional feature map is less than a preset threshold, it indicates that the I-th high-dimensional feature map differs from others. If the high-dimensional feature maps have low similarity, they are converted into holed feature maps and input into the subsequent hierarchical feature fusion stage.
[0051] When the mean peak signal-to-noise ratio of the I-th high-dimensional feature map is greater than or equal to a preset threshold, it indicates that the I-th high-dimensional feature map differs from others. The high-dimensional feature maps have high similarity and meet the emotion classification criteria, so they are input into the subsequent hierarchical feature fusion stage.
[0052] The preset threshold in this embodiment is 30, and specifically includes:
[0053] when If the similarity between the I-th feature map and other feature maps is low, it is converted into a dilated feature map and input into the subsequent hierarchical feature fusion stage to reduce the amount of convolution computation.
[0054] when If the result is positive, it indicates that the I-th feature map has a high similarity to other feature maps and meets the emotion classification criteria. It is then input into the subsequent hierarchical feature fusion stage.
[0055] The method employs a hierarchical feature fusion technique to concatenate several sets of feature maps to generate an output feature map. Specifically, this involves extracting features from each set of images. This is combined into an output feature that is more discriminative than the input features. The calculation formula is as follows:
[0056]
[0057] in, This is the output feature map we need. These are the feature maps that the front-end network needs to fuse. It is a transformation function that upsamples or downsamples each input feature map before connection. For feature fusion function, It is a function that generates hierarchical features. It is a method function for predicting target inspection from the input feature map.
[0058] In speech emotion recognition analysis, we need to consider not only deep convolution features, but also the spatiotemporal features of the audio signal and the pooling features of the original spectrogram. The spatiotemporal features and pooling features also contain key information that has an important impact on emotion recognition.
[0059] Specifically, the spatiotemporal features of the audio signal are learned through downsampling, and the input feature map is passed through an average pooling layer to achieve dimension matching and obtain the original spectrogram pooled features. The specific operating steps include:
[0060] S61: Transfer the original input feature map A convolutional kernel with a stride of 2 Perform average pooling to obtain the pooling features of the original spectrogram. ;
[0061] S62: Process the original input feature map The output feature map is downsampled to have the same dimensions as the above output feature map, and then input into the first standard convolutional kernel. To learn the spatial features of spectrograms Then input it into a pointwise convolution kernel. To learn linear combination features between various features .
[0062] Specifically, the original spectrogram pooling features and hierarchical fusion features are concatenated and superimposed, and then fused with the downsampled spatiotemporal features to obtain the final output features. :
[0063] S63: To achieve dimensionality matching, the output feature map is first... Input to size Features are obtained from pointwise group convolution kernels ;
[0064] S64: Pool the features of the original spectrogram With features By performing concatenation and superposition, local features are obtained. The calculation formula is as follows:
[0065]
[0066] in This indicates a series addition operation;
[0067] S65: Local features The final output feature is obtained by adding the downsampled spatiotemporal features. The calculation formula is as follows:
[0068]
[0069] in This indicates an addition operation;
[0070] The obtained final output feature representation First, it undergoes a linear transformation through a linear layer, then... The function performs emotion classification prediction, resulting in a speech emotion recognition model, which is represented as follows:
[0071] Will As Sentiment classification is performed on the function's input:
[0072]
[0073]
[0074] in, The function is a normalized exponential function. for The weight matrix of the layer, for Layer bias terms, This represents all real labels, where c is the sentiment category value. To predict the probability of a class, Represents the true label of the predicted class. This represents the total value of the emotional categories.
[0075] It is understood that, in this embodiment of the invention, the speech emotion recognition model based on a dilated depth separable group convolutional lightweight network consists of a pointwise group convolutional layer, a dilated depth separable convolutional layer, a similarity determination module, a hierarchical feature fusion layer, a downsampling learning layer, and a pooling layer. Therefore, this embodiment requires training and optimization of the speech emotion recognition model composed of the above modules and network layers.
[0076] A speech emotion recognition model based on a lightweight dilated depth separable group convolutional network is trained and optimized. Specifically, the Adam optimizer is used to train the speech emotion recognition model based on the lightweight dilated depth separable group convolutional network. During the optimization process, cross-entropy is used as the loss function, and other factors are introduced. The regularization term prevents the model from overfitting, and the loss function is expressed as:
[0077]
[0078] in, This represents the difference between the model's predicted sentiment value and the actual sentiment value; the smaller the value, the closer they are. θ represents the weight matrix of a set of linear layers, and c represents the number of sentiment categories. This indicates the predicted sentiment value. Represents the true sentiment value. for Regularization parameters, This represents the parameter set of the model.
[0079] Figure 4 This is a block diagram of the speech emotion recognition method based on a lightweight, depth-separable group convolutional network according to the present invention, as shown below. Figure 4 As shown, in this embodiment of the invention, after obtaining the audio signal spectrogram, it is necessary to perform average pooling on it and time-frequency conversion using short-term Fourier transform to obtain its input feature map. Then, a pointwise group convolutional layer of size (M, d, 1) is used to reduce the dimensionality and decompose the input feature map to obtain several low-dimensional feature maps. The low-dimensional feature maps are then converted into several high-dimensional feature maps of the same dimension as the input feature map through a deep-dipping separable convolutional layer. A similarity determination module is used to determine whether the high-dimensional feature map meets the emotion classification criteria. A hierarchical feature fusion technique is used to concatenate several groups of low-dimensional and high-dimensional feature maps to generate an output feature map. Downsampling is used to learn the spatiotemporal characteristics of the audio signal, and the input feature map is simultaneously passed through an average pooling layer to achieve dimensionality matching. Finally, the pooled input feature map and the hierarchically fused output feature map are concatenated and superimposed to obtain the final output high-dimensional feature representation. The final output high-dimensional feature map representation is used for emotion prediction.
[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A speech emotion recognition method based on a lightweight, depth-separable group convolutional network, characterized in that, Includes the following steps: S1: Short-term Fourier transform is used to perform time-frequency conversion on the audio signal to obtain its input feature map; S2: The input feature map is reduced in dimensionality and decomposed by a pointwise group convolutional layer to obtain several low-dimensional feature maps; S3: The low-dimensional feature map is transformed into several high-dimensional feature maps of the same dimension as the input feature map by a dilatancy-separable convolutional layer; S4: Use a similarity determination module to determine whether the high-dimensional feature map conforms to the emotion classification standard; S5: A hierarchical feature fusion technique is used to concatenate several groups of low-dimensional and high-dimensional feature maps to generate an output feature map; A hierarchical feature fusion technique is used to concatenate several sets of low-dimensional and high-dimensional feature maps to generate an output feature map. Specifically, this involves extracting features from each set of images. This is combined into an output feature that is more discriminative than the input features. The calculation formula is as follows: in, The desired output feature map is... These are the feature maps that the front-end network needs to fuse. It is a transformation function that upsamples or downsamples each input feature map before connection. For feature fusion function; S6: The spatiotemporal characteristics of the audio signal are learned by downsampling. At the same time, the input feature map is passed through the average pooling layer to achieve dimension matching. Finally, the pooled input feature map is concatenated and superimposed with the output feature map of the hierarchical fusion output to obtain the final output feature. The spatiotemporal features of the audio signal are learned by downsampling, and the input feature map is passed through an average pooling layer to achieve dimension matching and obtain the original spectrogram pooled features. Specifically, it includes: S61: Transfer the original input feature map After passing through a convolution kernel with a stride of 2 Perform average pooling to obtain the pooling features of the original spectrogram. W, H, and M represent the input feature maps, respectively. Width, height, and dimensions; S62: Process the original input feature map The output feature map is downsampled to have the same dimensions as the above output feature map, and then input into the first standard convolutional kernel. To learn the spatial features of spectrograms Then input it into a pointwise convolution kernel. To learn linear combination features between various features N represents the dimension of the final output feature map; The original spectrogram pooling features and hierarchical fusion features are concatenated and superimposed, and then fused with the downsampled spatiotemporal features to obtain the final output features. : S63: Output feature map Input to size Features are obtained from pointwise group convolution kernels ; S64: Pool the features of the original spectrogram With features By performing concatenation and superposition, local features are obtained. ; S65: Local features The final output feature is obtained by adding it to the downsampled spatiotemporal features. ; Where add represents the addition operation; S7: Input the final output features into the linear layer. The function performs emotion prediction to obtain the emotion recognition result of the audio signal.
2. The speech emotion recognition method based on a lightweight, depth-separable group convolutional network according to claim 1, characterized in that, The input feature map is obtained from the audio signal through time-frequency conversion. The process includes: S11: Use the Hamming window function to segment the original audio signal; S12: Perform a short-term Fourier transform on the segmented audio signal to obtain the input feature map of the original audio signal. Where W, H, and M represent the input feature maps, respectively. Width, height, and dimensions.
3. The speech emotion recognition method based on a lightweight, depth-separable group convolutional network according to claim 1, characterized in that, Pointwise group convolutional layers are used to reduce the dimensionality and decompose the input feature map, resulting in several low-dimensional feature maps. The process includes: based on a given P, using P convolutional kernels For the input feature map Perform pointwise group convolution to reduce its dimensionality, resulting in P low-dimensional d-dimensional feature maps. Where P is an introduced hyperparameter, its function is to uniformly reduce the dimensionality of the feature mapping, and N is the dimension of the final output feature map.
4. The speech emotion recognition method based on a lightweight, depth-separable group convolutional network according to claim 1, characterized in that, The low-dimensional feature map is transformed into several high-dimensional feature maps of the same dimension as the original input feature map by using a deep dilated separable convolutional layer. include: S31: Convert P low-dimensional feature maps Each branch uses a dilated convolution kernel simultaneously. The void ratio r is Its receptive field increases to d represents the dimension of the low-dimensional feature map; P represents the given parameters; S32: Low-dimensional feature map After dilated convolutional kernels, depthwise separable convolutional layers are then used to process the low-dimensional feature maps. Convert to high-dimensional feature map .
5. The speech emotion recognition method based on a lightweight, depth-separable group convolutional network according to claim 1, characterized in that, The similarity determination module is used to determine whether the high-dimensional feature map meets the emotion classification criteria, including: S41: Input P high-dimensional feature maps into the similarity determination module and calculate the peak signal-to-noise ratio between each high-dimensional feature map; P represents the given parameters; S42: Using the peak signal-to-noise ratio (PSNR) between the I-th and J-th high-dimensional feature maps, calculate the PNR values of each high-dimensional feature map with the other high-dimensional feature maps. The mean peak signal-to-noise ratio of each high-dimensional feature map; When the mean peak signal-to-noise ratio of the I-th high-dimensional feature map is less than a preset threshold, it indicates that the I-th high-dimensional feature map differs from others. If the high-dimensional feature maps have low similarity, they are converted into holed feature maps and input into the subsequent hierarchical feature fusion stage. When the mean peak signal-to-noise ratio of the I-th high-dimensional feature map is greater than or equal to a preset threshold, it indicates that the I-th high-dimensional feature map differs from others. The high-dimensional feature maps have high similarity and meet the emotion classification criteria, so they are input into the subsequent hierarchical feature fusion stage.
6. The speech emotion recognition method based on a lightweight, depth-separable group convolutional network according to claim 1, characterized in that, The method further includes training and optimizing a speech emotion recognition model based on a dilated depth separable group convolutional lightweight network. Specifically, the Adam optimizer is used to train the speech emotion recognition model based on the dilated depth separable group convolutional lightweight network. During the optimization process, cross-entropy is used as the loss function, and [the following is also mentioned:] The regularization term prevents the model from overfitting, and the loss function is expressed as: Where L(θ) represents the difference between the model's predicted sentiment value and the actual sentiment value, with a smaller value indicating a closer match; θ represents the weight matrix of a set of linear layers; and c represents the number of sentiment categories. This indicates the predicted sentiment value. Represents the true sentiment value. for Regularization parameters, This represents the parameter set of the model.