Multi-scale speech emotion feature extraction method, device, equipment, storage medium and computer program product

Abstract

The application provides a multi-scale speech emotion feature extraction method, device, equipment, storage medium and computer program product. The method comprises: performing preprocessing on audio data of speech through an audio data preprocessing module, and obtaining a mel spectrum of the audio data; generating an F0 feature vector according to the mel spectrum through a multi-time scale F0 module in a pitch module; obtaining a multi-dimensional pitch feature vector according to the mel spectrum through a pitch module in the pitch module; extracting an energy feature vector according to an energy spectrum of the mel spectrum through an energy module; obtaining a rhythm feature vector according to the mel spectrum through a rhythm module; and splicing and fusing the F0 feature vector, the pitch feature vector, the energy feature vector and the rhythm feature vector through an emotion feature fusion module to generate an emotion feature vector. The application significantly improves the naturalness and representativeness of emotion feature extraction.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of speech processing technology, and more specifically, to a method, apparatus, device, storage medium and computer program product for multi-scale speech emotion feature extraction. Background Technology

[0002] With the continuous development of automotive intelligence, voice interaction will become one of the main methods of human-computer interaction, and voice interaction technology will have a wide range of applications in the automotive industry in the future. Currently, voice interaction quality is becoming more natural and fluent, which makes users more eager for this type of interaction. Current voice interaction technology has made great progress in terms of high definition and intelligibility, while also surging the demand for generating emotionally rich voices.

[0003] Emotion, as a crucial component of human communication, can be conveyed through various prosodic features, such as tone, intensity, and speech rate. It plays a vital role in the semantic and pragmatic aspects of spoken language. Machine-synthesized speech that is closer to human voice quality and more natural in expression will increase user engagement.

[0004] Emotional speech conversion focuses on transforming speech from source emotional discourse to target emotional discourse, thus it can become a key supporting technology for human-computer interaction applications. However, emotional speech conversion still faces some challenges. Researchers hope to incorporate richer emotional elements, and the quality of speech emotional feature extraction directly affects the result of target emotional discourse conversion.

[0005] Currently, known emotional speech conversion technologies have gone through several stages, evolving from initial statistical methods to later neural network architectures, and recently, research based on Generative Adversarial Networks (GANs) has emerged. These methods primarily focus on frame-by-frame spectral and prosodic feature mapping when processing emotional speech conversion, attempting to model parallel data and extract fundamental frequency (F0) features. However, these methods are limited by the complexity of emotional information, which is difficult to accurately quantify and represent. Furthermore, due to the close dependency between spectrum and prosody, studying them in isolation can lead to mismatched conversion results. Current emotional speech conversion methods face challenges in expressing emotions, failing to capture and highlight rich emotional features naturally. In fact, the essence of emotion is suprasegmental and hierarchical, a characteristic reflected in both the spectrum and prosody of speech. Therefore, simply converting spectral features frame-by-frame is insufficient; a richer approach is needed to handle emotional features at different levels and dimensions.

[0006] Statistical methods and neural network architectures primarily focus on frame-by-frame analysis, studying speech spectral feature mapping, modeling parallel data, and extracting fundamental frequency features. Generative adversarial networks (GANs) improve generated speech quality through adversarial interaction between generators and discriminators. Modeling Mel-spectral parameters and prosodic parameters addresses oversmoothing issues, resulting in better prosody and richer emotional expression. Training with non-parallel data yields excellent results. Carl Robinson's order-to-order model learns to predict the F0 features of the corresponding emotion to the target emotion, without considering the actual F0 features of neutral speech. Furthermore, the duration of F0 features and fundamental frequency contours are modeled and processed separately during training and prediction. The proposed order-to-order model jointly models the duration of both F0 features and F0 contours, allowing the output F0 contour to be duration-adjusted based on the source input signal and its linguistic context, thus achieving effective speech emotion conversion.

[0007] Current network architectures primarily model emotional features based on fundamental frequency features, Mel-spectral parameters, and prosodic parameters. Some methods have also proposed transform F0 features and fundamental frequency contour models. However, these methods only focus on specific features or a few types of features, failing to delve into the fine-grained characteristics of emotional changes in input speech. Emotional expression involves a rich variety of feature variations; describing emotional features solely on a single dimension is clearly insufficient to comprehensively capture the complexity of emotional information.

[0008] Current frame-level conversion methods at the spectral and prosodic levels can achieve emotional speech conversion while preserving language content and speaker identity. However, these methods have limitations in capturing emotional expression. Later, sequence-to-sequence neural network architectures were developed, which can control the duration of the converted sentences. However, these models are also frame-level and fail to fully highlight the richness of emotion. Since emotion is inherently suprasegmental and hierarchical, this characteristic is reflected in both the spectral and prosodic levels. Therefore, performing emotional feature conversion solely at the frame level is insufficient.

[0009] In terms of conversion effects, most solutions perform feature mapping from neutral speech sequences to target sequences of a single emotion (such as anger, sadness, happiness, or fear), which limits the practical application of the conversion effects. Another important drawback is that these methods require a large amount of emotional speech data for training, which is impractical for real-life scenarios.

[0010] When using GAN models, such as CycleGAN for simultaneous spectral and F0 conversion, while this method is quite successful in single-speaker tasks, it has unavoidable limitations when dealing with multiple speakers. Later, emotion conversion models based on Variational Autoencoder Wasserstein Generative Adversarial Network (VAW-GAN) were adopted, allowing the network to learn independent emotion patterns between different speakers. Even though the above methods can successfully perform emotion conversion while preserving the language content and speaker identity, the converted speech length is the same as the input speech length, which does not conform to the expression habits of people in real-world interactive scenarios. Summary of the Invention

[0011] To address the aforementioned problems in the prior art, in a first aspect, embodiments of the present invention provide a multi-scale speech emotion feature extraction method, the method comprising: preprocessing audio data of speech using an audio data preprocessing module to obtain a Mel spectrum of the audio data; extracting a one-dimensional F0 fundamental frequency feature from the Mel spectrum using a multi-time-scale F0 module in a pitch module, decomposing the one-dimensional F0 fundamental frequency feature into components of multiple time scales, and generating an F0 feature vector based on the components of the multiple time scales; and extracting a one-dimensional pitch contour feature from the Mel spectrum using a pitch module in a pitch module, and then transmitting the feature via a convolutional neural network and a bidirectional long-range... The short-term memory network obtains an initial pitch feature vector based on the one-dimensional pitch contour features, and then downsamples the initial pitch feature vector to obtain a multi-dimensional pitch feature vector. The energy module squares the Mel spectrum to obtain an energy spectrum, and then extracts energy feature vectors step-by-step through multiple convolutional neural network layers. The rhythm module obtains a rhythm feature vector based on the Mel spectrum through multiple convolutional neural network layers and a fully connected layer. Finally, the emotion feature fusion module concatenates and fuses the F0 feature vector, the pitch feature vector, the energy feature vector, and the rhythm feature vector to generate an emotion feature vector.

[0012] In some implementations, decomposing the one-dimensional F0 fundamental frequency feature into components of multiple time scales includes: using continuous wavelet transform to decompose the one-dimensional F0 fundamental frequency feature into components of multiple time scales.

[0013] In some embodiments, the method further includes: after extracting one-dimensional pitch contour features from the Mel spectrum using the pitch module, normalizing the one-dimensional pitch contour features to remove individual speech difference information in the one-dimensional pitch contour features, and inputting the normalized one-dimensional pitch contour features into a convolutional neural network and a bidirectional long short-term memory network.

[0014] In some implementations, a one-dimensional pitch contour feature is extracted from the Mel spectrum using a pitch module in the pitch module. An initial pitch feature vector is then obtained based on the one-dimensional pitch contour feature via a convolutional neural network and a bidirectional long short-term memory network. The process of downsampling this initial pitch feature vector to obtain a multi-dimensional pitch feature vector includes: obtaining the one-dimensional pitch contour feature using a vocoder based on the Mel spectrum; performing batch normalization on the one-dimensional pitch contour feature to generate a normalized one-dimensional pitch contour feature; projecting the normalized one-dimensional pitch contour feature onto the column space of a weight matrix via a fully connected layer; and sequentially processing the features in the column space of the weight matrix through an activation function, a one-dimensional convolutional neural network, another activation function, random deactivation, a bidirectional long short-term memory network, and downsampling to obtain the multi-dimensional pitch feature vector.

[0015] In some implementations, the method further includes: generating duration sequence features by assigning different weights to input vectors at different time steps through a rhythm module via an attention layer set after multiple convolutional neural network layers, and inputting the duration sequence features into a fully connected layer.

[0016] In some implementations, obtaining the rhythm feature vector from the Mel spectrum via a rhythm module, through multiple convolutional neural network layers and fully connected layers, includes: processing the Mel spectrum sequentially through a two-dimensional convolutional neural network layer, an activation function, a max pooling layer, another two-dimensional convolutional neural network layer, and another max pooling layer to obtain an initial rhythm feature vector, wherein the two two-dimensional convolutional neural network layers have different input dimensions and / or the number of convolutional layer channels; inputting the initial rhythm feature vector into a bidirectional long short-term memory network to extract an emotional speech context feature vector with temporal information; inputting the emotional speech context feature vector with temporal information into the attention layer to generate the duration sequence feature; and processing the duration sequence feature sequentially through two fully connected layers to obtain the rhythm feature vector.

[0017] In some implementations, the audio data preprocessing module performs preprocessing on the audio data of the speech, including: pre-emphasizing the audio data to obtain pre-emphasized audio data; performing frame-by-frame processing on the pre-emphasized audio data to obtain frame-by-frame audio data; performing a frame-by-frame windowing operation on the frame-by-frame audio data to obtain windowed audio data; performing a discrete Fourier transform on the windowed audio data to obtain the frequency domain signal of the audio data; performing a modulus-taking operation on the frequency domain signal to obtain the amplitude spectrum of the audio data; and obtaining the Mel spectrum of the audio data based on the amplitude spectrum using a Mel filter bank.

[0018] In some implementations, the stepwise extraction of energy feature vectors based on the energy spectrum via multiple convolutional neural network layers includes: inputting the energy spectrum into a first convolutional processing module to obtain a first energy feature, wherein the first convolutional processing module includes multiple layers, each including a two-dimensional convolutional neural network, an activation function, and a max-pooling layer; inputting the first energy feature into a second convolutional processing module to obtain a second energy feature, wherein the second convolutional processing module includes multiple layers, each including a two-dimensional convolutional neural network, an activation function, and a max-pooling layer; and inputting the second energy feature into a third convolutional processing module to obtain the energy feature vector, wherein the third convolutional processing module includes multiple layers, each including a two-dimensional convolutional neural network, an activation function, and a max-pooling layer. The two-dimensional convolutional neural networks in the first, second, and third convolutional processing modules have different input dimensions and / or the number of convolutional layer channels.

[0019] In some implementations, the emotional feature fusion module concatenates and fuses the F0 feature vector, the pitch feature vector, the energy feature vector, and the rhythm feature vector to generate an emotional feature vector. This includes: concatenating the F0 feature vector, the pitch feature vector, the energy feature vector, and the rhythm feature vector to generate an emotional feature vector matrix; and then processing the emotional feature vector matrix sequentially through a fully connected layer, an activation function, a two-dimensional convolutional neural network, an activation function, batch normalization, an activation function, random deactivation, and a two-dimensional convolutional neural network to obtain the emotional feature vector.

[0020] In a second aspect, embodiments of the present invention provide a multi-scale speech emotion feature extraction device, the device comprising an audio data preprocessing module, a pitch module, an energy module, a rhythm module and an emotion feature fusion module, wherein the pitch module comprises a multi-timescale F0 module and a pitch module. The audio data preprocessing module is configured to perform preprocessing on the audio data to obtain the Mel spectrum of the audio data; the multi-timescale F0 module is configured to extract a one-dimensional F0 fundamental frequency feature from the Mel spectrum, decompose the one-dimensional F0 fundamental frequency feature into multiple time-scale components, and generate an F0 feature vector based on the multiple time-scale components; the pitch module is configured to extract a one-dimensional pitch contour feature from the Mel spectrum, obtain an initial pitch feature vector based on the one-dimensional pitch contour feature through a convolutional neural network and a bidirectional long short-term memory network, and then downsample the initial pitch feature vector to obtain a multi-dimensional pitch feature vector; the energy module is configured to perform a square operation on the Mel spectrum to obtain an energy spectrum, and gradually extract an energy feature vector based on the energy spectrum through multiple convolutional neural network layers; the rhythm module is configured to obtain a rhythm feature vector based on the Mel spectrum through multiple convolutional neural network layers and a fully connected layer; and the emotion feature fusion module is configured to concatenate and fuse the F0 feature vector, the pitch feature vector, the energy feature vector, and the rhythm feature vector to generate an emotion feature vector.

[0021] In a third aspect, embodiments of the present invention provide a multi-scale speech emotion feature extraction device, the device including a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the multi-scale speech emotion feature extraction method described in any of the above embodiments.

[0022] In a fourth aspect, embodiments of the present invention provide a storage medium storing computer-readable instructions that, when executed by a processor, perform the multi-scale speech emotion feature extraction method described in any of the above embodiments.

[0023] In a fifth aspect, embodiments of the present invention provide a computer program product comprising computer-readable instructions that, when executed by a processor, perform the steps of the multi-scale speech emotion feature extraction method described in any of the above embodiments.

[0024] The embodiments of the present invention propose a multi-scale speech emotion feature extraction scheme. This scheme takes into account the complexity, suprasegmental and hierarchical nature of emotion features, and integrates rich forms of emotion feature expression. It can extract emotion features with richer layers and more comprehensive expression, providing a solid foundation for the subsequent generation of more emotionally expressive and realistic emotional speech.

[0025] The multi-scale speech emotion feature extraction scheme proposed in this invention, for the first time, extracts and fuses four emotion-related features—F0, pitch, energy, and rhythm—from three dimensions: prosody, rhythm, and energy, forming a unified emotion feature vector. By concatenating features from different dimensions and modeling them with a deep learning network, multi-level and multi-timescale emotion feature representation is achieved. This fusion method not only comprehensively captures emotional information in speech but also significantly improves the naturalness and representativeness of emotional expression, laying a solid technical foundation for subsequent emotional speech conversion and recognition. Attached Figure Description

[0026] The above and other objects, features, and advantages of embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0027] Figure 1 A flowchart of a multi-scale speech emotion feature extraction method according to an embodiment of the present invention is shown;

[0028] Figure 2 A schematic diagram of a multi-scale speech emotion feature extraction network according to an embodiment of the present invention is shown;

[0029] Figure 3 A schematic block diagram illustrating an example of an audio data preprocessing procedure according to an embodiment of the present invention is shown.

[0030] Figure 4 A schematic block diagram illustrating an example of a pitch module processing procedure according to an embodiment of the present invention is shown;

[0031] Figure 5 A schematic block diagram illustrating an example of an energy module processing procedure according to an embodiment of the present invention is shown;

[0032] Figure 6 A schematic block diagram illustrating an example of a rhythm module processing procedure according to an embodiment of the present invention is shown.

[0033] Figure 7 A schematic block diagram illustrating an example of the processing procedure of the emotion feature fusion module according to an embodiment of the present invention is shown.

[0034] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0035] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way.

[0036] In one aspect, embodiments of the present invention provide a method for multi-scale speech emotion feature extraction. (See reference) Figures 1-2 ,in Figure 1 A flowchart of a multi-scale speech emotion feature extraction method according to an embodiment of the present invention is shown. Figure 2 A schematic diagram of a multi-scale speech emotion feature extraction network according to an embodiment of the present invention is shown. The functional architecture of the multi-scale speech emotion feature extraction method proposed in the embodiment of the present invention can be represented as a multi-scale speech emotion feature extraction network, or a multi-scale speech emotion feature extraction model. The multi-scale speech emotion feature extraction network consists of three parts, as follows: Figure 2 As shown, it includes a pitch module, an energy module, and a rhythm module. The pitch module contains a multi-timescale F0 module and a pitch module.

[0037] like Figure 1 As shown, the method includes steps S101-S106.

[0038] In step S101, the audio data preprocessing module performs preprocessing on the audio data of the speech to obtain the Mel spectrum of the audio data.

[0039] In step S102, the one-dimensional F0 fundamental frequency feature is extracted from the Mel spectrum through the multi-time-scale F0 module in the pitch module, and the one-dimensional F0 fundamental frequency feature is decomposed into components of multiple time scales. The F0 feature vector is generated based on the components of multiple time scales.

[0040] In step S103, a one-dimensional pitch contour feature is extracted from the Mel spectrum through the pitch module in the pitch module. The initial pitch feature vector is obtained based on the one-dimensional pitch contour feature through a convolutional neural network and a bidirectional long short-term memory network. Then, the initial pitch feature vector is downsampled to obtain a multi-dimensional pitch feature vector.

[0041] In step S104, the energy spectrum is obtained by squaring the Mel spectrum through the energy module, and the energy feature vector is extracted step by step based on the energy spectrum through multiple convolutional neural network layers.

[0042] In step S105, the rhythm feature vector is obtained from the Mel spectrum through the rhythm module via multiple convolutional neural network layers and fully connected layers.

[0043] In step S106, the emotional feature fusion module splices and fuses the F0 feature vector, pitch feature vector, energy feature vector and rhythm feature vector to generate an emotional feature vector.

[0044] It is important to note that the processing steps S102-S105 are independent of each other. Although the steps are numbered in the flowchart, this is only for differentiation between them and does not represent the execution order of the steps. Steps S102-S105 can be executed in any order or simultaneously in parallel.

[0045] The present invention proposes a multi-scale speech emotion feature extraction network and a feature extraction method for the network. This scheme takes into account the complexity, suprasegmental and hierarchical nature of emotion features, and integrates rich forms of emotion feature expression. It can extract emotion features with richer layers and more comprehensive expression, providing a solid foundation for the subsequent generation of more emotionally expressive and realistic emotional speech.

[0046] The multi-scale speech emotion feature extraction network comprises a pitch module, an energy module, and a rhythm module. The pitch module includes a multi-timescale F0 module and a pitch module. F0 is an important prosodic component that describes the intonation at different durations from syllable to utterance. Pitch features describe the highness or lowness of the sound, determined by the fundamental frequency of the sound; pitch is a sparsely discretized fundamental frequency. Energy represents the loudness or softness of the emotional speech sound. Rhythm represents the speed at which the speaker pronounces each syllable and is related to duration. This invention, for the first time, extracts emotion features from three dimensions—prosodic, rhythmic, and energy—comprising four features, and employs different deep learning network frameworks to perform multi-level, multi-timescale extraction of emotion features, ensuring that the extracted emotion features are representative and more natural in resembling real emotion.

[0047] As one embodiment of the present invention, the preprocessing of the audio data of the speech by the audio data preprocessing module in step S101 may include: performing pre-emphasis on the audio data to obtain pre-emphasized audio data; performing frame-by-frame processing on the pre-emphasized audio data to obtain frame-by-frame audio data; performing a frame-by-frame windowing operation on the frame-by-frame audio data to obtain windowed audio data; performing a discrete Fourier transform on the windowed audio data to obtain the frequency domain signal of the audio data; performing a modulus-taking operation on the frequency domain signal to obtain the amplitude spectrum of the audio data; and obtaining the Mel spectrum of the audio data based on the amplitude spectrum using a Mel filter bank.

[0048] refer to Figure 3 The diagram illustrates a schematic block diagram of an example audio data preprocessing procedure according to an embodiment of the present invention. First, the audio signal is pre-emphasized, then processed in frames, and windowed using a Hamming window. Subsequently, a Fourier transform is performed on each frame to obtain a short-time amplitude spectrum. Finally, the short-time amplitude spectrum is passed through a Mel frequency filter bank to obtain an 80-dimensional Mel spectrum, which serves as a feature vector. The specific steps are as follows:

[0049] Step 1.1 To balance the spectrum and avoid numerical problems in Fourier transform operations, pre-emphasize the signal with α=0.97:

[0050] y(t) = x(t) - αx(t-1) Equation (1)

[0051] where x(t), the speech sampling point at time t, is the original signal, y(t) is the pre-emphasized signal, and α is the pre-emphasis coefficient.

[0052] Step 1.2 The sampling frequency of the audio is 44.1 KHz, the audio data is subjected to framing processing, the frame length M is set to 512 sampling points, and the frame shift J is M / 2, that is, 256 sampling points.

[0053] x k (m) = x[kJ: kJ+M-1], 1<m<M, k∈[0, Z] Equation (2)

[0054] where k is the frame index value, Z is the number of frames, represents the speech vector of the collected audio at the k-th frame.

[0055] Step 1.3 Windowing

[0056] Step 1.3.1 Select the window function - Hamming window

[0057]

[0058] where w(l win ) represents the Hamming window function, a is a coefficient, L win is the window length. As an example, a can be set to 0.54, and this equation can be simplified as:

[0059]

[0060] Step 1.3.2 Perform frame-by-frame windowing operation on the framed audio data obtained in Step 1.2, as shown in the following Equation (5):

[0061] y k (m) = x k (m)w(l win ) Equation (5)

[0062] where x k (m) represents the speech data to be windowed, represents the k-th frame data in the time domain of the windowed speech data.

[0063] Step 1.4 Convert the framed and windowed audio data from the time domain to the frequency domain through discrete Fourier transform, to obtain the frequency domain signal S of the audio data k(l fft ):

[0064]

[0065] in, Let be a complex number, representing the data after FFT transformation, with the real part representing the l-th frequency band in the m-th frequency band of the k-th frame. fft The amplitude of each frequency component, where the imaginary part represents the l-th frequency component in the m-th frequency band of the k-th frame. fft The phase of each frequency component, L fft is the length of the Fast Fourier Transform.

[0066] Step 1.5 Perform a modulus-taking operation on the frequency domain signal obtained in Step 1.4 to obtain the amplitude spectrum U of the audio data. k :

[0067] U k =|S k (l fft )| Formula (7)

[0068] Step 1.6: Obtain the output of each Mel filter based on the amplitude spectrum. For the input amplitude spectrum U, the output Y of the m-th Mel filter... m It can be represented as:

[0069] Y m =∑ f H m (f)·|U| 2 Formula (8)

[0070]

[0071] Where H m (f) is the frequency response function of the m-th Mel filter, f m-1 f m f m+1 These are the three key frequency points (lower bound, center, and upper bound) of the m-th filter, which are typically uniformly distributed on the Mel scale. m Y is the output value of the m-th filter, representing the energy within that frequency range. Finally, the output values ​​Y of all m filters... m This constitutes the Mel spectrum Y, namely:

[0072] Y = [Y1, Y2, ..., Y] m ]. Formula (10)

[0073] As one embodiment of the present invention, decomposing the one-dimensional F0 fundamental frequency feature into multiple time-scale components in step S102 may include using a continuous wavelet transform (CWT) to decompose the one-dimensional F0 fundamental frequency feature into multiple time-scale components. This embodiment innovatively introduces a continuous wavelet transform to decompose the one-dimensional F0 feature into 10 components at different time scales. The low-time-scale components can capture short-term subtle changes in speech, while the high-time-scale components focus on long-term trend features. This multi-scale analysis method refines the extraction of F0 features, enhancing the hierarchy of emotional features and improving the comprehensiveness of their expression, providing richer and more detailed information support for emotion recognition, and breaking through the limitations of traditional single-time-scale feature extraction.

[0074] The following description uses only examples to illustrate the process of step S102.

[0075] Because the voiceless portion of the F0 fundamental frequency feature is discontinuous and inherently layered, the multi-timescale F0 module, as a multi-scale modeling method, first extracts the one-dimensional F0 fundamental frequency feature using a WORLD vocoder. Then, it uses continuous wavelet transform to decompose the one-dimensional F0 feature into 10 time scales. The lower dimension captures short-term changes, while the higher dimension captures long-term changes. In this way, emotional features can be extracted from F0 at both the detailed prosodic level and the overall speech level. The detailed steps are as follows:

[0076] Step 2.1 Input the Mel spectrum generated in step 1.6 into the one-dimensional F0 fundamental frequency feature of the WORLD vocoder.

[0077] Step 2.2 Transform the one-dimensional F0 fundamental frequency feature from Step 2.1 using the CWT algorithm. Assuming the given F0 fundamental frequency feature is a bounded continuous signal k0, its CWT algorithm is expressed as W(k0)(τ,t):

[0078]

[0079] Where W(k0)(τ, t) represents the wavelet transform result, represents the wavelet coefficients on the scaling parameter τ and translation parameter t, and ψ is the Mexican hat wavelet function. The original signal k0 can be recovered from the wavelet representation W(k0) through inverse transform, and the recovery function is:

[0080]

[0081] In this step, the analysis is performed at 10 discrete scales, each octave apart, with the decomposition function as follows:

[0082] W i (k0)(t)=Wi (k0)(2 i+1 τ0,t)(i+2.5) -5 / 2 Formula (13)

[0083] The approximation of k0 obtained by the inverse transform is:

[0084]

[0085] Where i = 1….10, and τ is 5ms.

[0086] The 10-dimensional CWT components obtained from step 2.2 are embedded as F0 feature vectors.

[0087] As an embodiment of the present invention, in step S103, after extracting one-dimensional pitch contour features from the Mel spectrum through the pitch module, the one-dimensional pitch contour features are normalized to remove individual speech difference information in the one-dimensional pitch contour features. The normalized one-dimensional pitch contour features are then input into the convolutional neural network and the bidirectional long short-term memory network.

[0088] This implementation proposes a pitch contour normalization process to remove interference from speaker identity information in emotion feature extraction. The normalized pitch features retain only emotion-related information, avoiding emotional expression biases caused by individual differences (such as gender or voice tone). This improvement significantly enhances the universality of pitch features and the accuracy of emotion modeling, providing key technical support for achieving more natural emotional speech conversion.

[0089] As one embodiment of the present invention, step S103 may include: obtaining one-dimensional pitch contour features based on the Mel spectrum using a vocoder; performing normalization processing on the one-dimensional pitch contour features through batch normalization to generate normalized one-dimensional pitch contour features; projecting the normalized one-dimensional pitch contour features onto the column space of the weight matrix via a fully connected layer; and sequentially processing the features in the column space of the weight matrix through an activation function, a one-dimensional convolutional neural network, an activation function, random deactivation, a bidirectional long short-term memory network, and downsampling to obtain a multi-dimensional pitch feature vector.

[0090] Pitch is the human perception of sound frequency. The higher the frequency, the higher the sound. Pitch plays a very important role in emotion and is defined as the high or low pitch of each syllable. Pitch information is contained in the pitch profile, but the pitch profile is also intertwined with other information. The pitch range can reflect specific speaker identity information, such as female speakers often having a high pitch range. Therefore, in this embodiment, the pitch module normalizes the pitch profile extracted by the vocoder, at which point the pitch profile contains only pitch information. The normalized pitch profile is then input into the pitch extraction neural network, and finally, a multi-dimensional (e.g., 10-dimensional) pitch feature vector embedding is obtained through downsampling.

[0091] refer to Figure 4 It shows a schematic block diagram of an example of a pitch module processing procedure according to an embodiment of the present invention.

[0092] Step 3.1 Input the Mel spectrum generated in step 1.6 into the one-dimensional pitch profile feature of the WORLD vocoder.

[0093] Step 3.2 uses Batch Normalization (BN) to normalize the output of Step 3.1, thereby accelerating network convergence and removing speaker identity information.

[0094] Step 3.3 feeds the features extracted in Step 3.2 into the Fully Connected Layer (FC), projecting the input features from their original space onto the column space of the weight matrix. This layer contains R nodes, for example, R is 64.

[0095] Step 3.4 Activate using the Rectified Linear Unit (ReLU) function.

[0096] Step 3.5 feeds the feature vector after activation into a one-dimensional convolutional neural network with a kernel size of 1×3 and 32 output channels.

[0097] Step 3.6 Activate using the ReLU function.

[0098] Step 3.7 To prevent overfitting, a random dropout operation is used with a ratio of 0.5.

[0099] Step 3.8 uses a bidirectional long short-term memory (Bi-LSTM) network. In order to provide each node of the input sequence of the output layer with complete past and future context information, the hidden layer has 2×128 hidden nodes.

[0100] Step 3.9: Downsample the feature vector obtained in Step 3.8 to obtain a 10-dimensional pitch feature vector embedding.

[0101] As an embodiment of the present invention, step S104, which involves progressively extracting energy feature vectors based on the energy spectrum through multiple convolutional neural network layers, may include: inputting the energy spectrum into a first convolutional processing module to obtain a first energy feature, wherein the first convolutional processing module includes multiple layers, each including a two-dimensional convolutional neural network, an activation function, and a max-pooling layer; inputting the first energy feature into a second convolutional processing module to obtain a second energy feature, wherein the second convolutional processing module includes multiple layers, each including a two-dimensional convolutional neural network, an activation function, and a max-pooling layer; and inputting the second energy feature into a third convolutional processing module to obtain an energy feature vector, wherein the third convolutional processing module includes multiple layers, each including a two-dimensional convolutional neural network, an activation function, and a max-pooling layer, wherein the two-dimensional convolutional neural networks in the first, second, and third convolutional processing modules have different input dimensions and / or the number of convolutional layer channels.

[0102] refer to Figure 5 The diagram illustrates a schematic block diagram of an example of an energy module processing procedure according to an embodiment of the present invention. As a specific example, the energy module processing procedure may include the following steps.

[0103] Step 4.1 Squaring the Mel spectrum from Step 1.6 yields the energy spectrum.

[0104] Step 4.2 First, the energy features in the energy spectrum are learned through the first to third layers of the convolutional neural network, each of which is a two-dimensional CNN. The input dimension is 80, the convolutional kernel is 3×3, and the number of channels in the convolutional layer is 32. Then, the ReLU function is used for activation. The pooling layer adopts maximum pooling (MaxPooling) with a size of 2×2. This process is repeated three times.

[0105] Step 4.3 uses convolutional neural networks (CNNs) with layers 4-6, each a two-dimensional CNN, to learn energy features. The input dimension is 32, the number of channels in each convolutional layer is 64, and the convolutional kernel is 3×3. Then, ReLU activation is used, and the pooling layer uses MaxPooling with a size of 2×2. This process is repeated three times.

[0106] Step 4.4 uses convolutional neural networks (CNNs) with layers 7-9, each a two-dimensional CNN, to learn energy features. The input dimension is 64, the number of channels in each convolutional layer is 128, and the convolutional kernel is 3×3. Then, ReLU activation is used, and the pooling layer uses MaxPooling with a size of 2×2. This process is repeated three times to output the energy feature vector embedding, which has a size of 30×128.

[0107] As an embodiment of the present invention, in step S105, the timing module can generate duration sequence features by assigning different weights to the input vectors of different time steps through the attention layer set after multiple convolutional neural network layers, and input the duration sequence features into the fully connected layer.

[0108] In the rhythm module, this implementation innovatively introduces an attention mechanism, generating duration sequence features by assigning different weights to input vectors at different time steps. This design enables the model to automatically focus on key time steps of emotional expression, thereby more accurately capturing rhythmic changes in speech. Compared to traditional methods, the attention mechanism significantly improves the expressive power of rhythmic features, providing more expressive temporal information support for emotional speech conversion.

[0109] As one embodiment of the present invention, step S105 may include: processing the Mel spectrum sequentially through a two-dimensional convolutional neural network layer, an activation function, a max pooling layer, another two-dimensional convolutional neural network layer, and another max pooling layer to obtain an initial rhythm feature vector, wherein the two two-dimensional convolutional neural network layers have different input dimensions and / or the number of convolutional layer channels; inputting the initial rhythm feature vector into a bidirectional long short-term memory network to extract an emotional speech context feature vector with temporal information; inputting the emotional speech context feature vector with temporal information into an attention layer to generate a duration sequence feature; and processing the duration sequence feature sequentially through two fully connected layers to obtain a rhythm feature vector.

[0110] refer to Figure 6 The diagram illustrates a schematic block diagram of an example of a rhythm module processing procedure according to an embodiment of the present invention. As an example only, the rhythm module processing procedure may include the following steps.

[0111] Step 5.1 Input the Mel spectrum generated in Step 1.6 into the 2D CNN with an input dimension of 80, a convolutional layer with 32 channels, a convolutional kernel of 3×3, and activate it using the ReLU function.

[0112] Step 5.2 Use the MaxPooling layer for pooling, with a size of 2×2 and a stride of 2.

[0113] Step 5.3 Input the features obtained in 5.2 into a two-dimensional CNN. The number of channels in the convolutional layer is 64, the convolutional kernel is 3×3, and the ReLU function is used for activation.

[0114] Step 5.4 Use the MaxPooling layer for pooling, with a size of 2×2 and a stride of 2.

[0115] Step 5.5 inputs the feature vector obtained in step 5.4 into Bi-LSTM(128) to extract the emotional speech context feature vector with temporal information to control the duration. The Hyperbolic Tangent Function (Tanh) activation function is used.

[0116] Step 5.6 Input the features extracted in Step 5.5 into the attention layer, assign the learned feature weights to the input vector in the next time step, and generate time sequence features with a feature size of 128×128.

[0117] Step 5.7 Input the feature vector obtained in step 5.6 into the fully connected layer. The input size is 128×128 and the output size is 64×128.

[0118] Step 5.8 Input the feature vector obtained in step 5.7 into the fully connected layer. The input size is 64×128 and the output size is 30×128.

[0119] As one embodiment of the present invention, step S106 may include: concatenating the F0 feature vector, pitch feature vector, energy feature vector and rhythm feature vector to generate an emotion feature vector matrix; and processing the emotion feature vector matrix sequentially through a fully connected layer, an activation function, a two-dimensional convolutional neural network, an activation function, batch normalization, an activation function, random deactivation, and a two-dimensional convolutional neural network to obtain the emotion feature vector.

[0120] refer to Figure 7 The diagram illustrates a schematic block diagram of an example of the processing procedure of an emotion feature fusion module according to an embodiment of the present invention. As an example only, the processing procedure of the emotion feature fusion module may include the following steps.

[0121] This process concatenates the four features extracted above to obtain the emotional feature vector embedding.

[0122] Step 6.1 Concatenate the F0 feature vector, pitch feature vector, energy feature vector, and rhythm feature vector to obtain a 30×276 emotion feature vector matrix.

[0123] Step 6.2 Feed the sentiment feature vector matrix into the fully connected layer and activate it using the ReLU function.

[0124] Step 6.3 Input the feature vector into the two-dimensional convolutional neural network. The number of channels in the convolutional layer is 128, the convolutional kernel is 3×3, and the ReLU function is used for activation.

[0125] Step 6.4 uses batch normalization to normalize the output, which speeds up network convergence and prevents gradient vanishing. The Tanh activation function is used to make convergence faster.

[0126] Step 6.5 To prevent overfitting, use the Dropout operation with a scale of 0.5.

[0127] Step 6.6 Input the feature vector into a two-dimensional convolutional neural network. The number of channels in the convolutional layer is 80, the convolutional kernel is 3×3, and the output is an emotion feature vector of size 30×80.

[0128] It should be noted that the formulas, ratios, number of channels, kernel size, size of fully connected layers, size of max pooling layers, dimensionality, etc. mentioned in the exemplary descriptions of the various processes above are all examples only, and the scope of the present invention is not limited thereto.

[0129] In the specific example above, the ReLU activation function after the convolutional neural network layer is a non-linear activation function. Alternatively, the activation function used can be replaced with a logistic function (sigmoid) or Tanh. Activation functions can introduce non-linear transformations, enabling the neural network to capture and represent more complex data patterns and relationships. ReLU is computationally very efficient because its output is the input itself when the input is positive, requiring no complex calculations, thus accelerating the training process of the neural network. Compared to sigmoid and Tanh, ReLU does not saturate on the positive half-axis, avoiding the gradient vanishing problem. Using sigmoid and Tanh functions involves exponential operations during computation, and the derivative calculation during backpropagation to find the error gradient involves division, resulting in relatively high computational costs. ReLU's derivative is simpler. This helps propagate gradients in deep networks, allowing deeper networks to learn features better.

[0130] The multi-scale speech emotion feature extraction method proposed in this invention, for the first time, extracts and fuses four emotion-related features—F0, pitch, energy, and rhythm—from three dimensions: prosody, rhythm, and energy, forming a unified emotion feature vector. By concatenating features from different dimensions and modeling them with a deep learning network, multi-level and multi-timescale emotion feature representation is achieved. This fusion method not only comprehensively captures emotional information in speech but also significantly improves the naturalness and representativeness of emotional expression, laying a solid technical foundation for subsequent emotional speech conversion and recognition.

[0131] The multi-scale speech emotion feature extraction network (multi-scale speech emotion feature extraction model) proposed in the embodiments of the present invention requires a model training process beforehand. The following is an illustrative description of the model training process.

[0132] During the model training phase, the raw audio data is first preprocessed. This process includes several steps: pre-emphasizing the audio signal to balance the spectrum and avoid numerical problems in the Fourier transform; processing the audio into frames, setting fixed frame lengths and frame shifts for subsequent analysis; windowing the framed audio signal using a Hamming window function to reduce spectral leakage; converting the time-domain signal to a frequency-domain signal using a Fast Fourier Transform and calculating the short-time amplitude spectrum; finally, extracting the 80-dimensional Mel spectrum using a Mel filter bank.

[0133] Next, various emotion-related features are extracted from the preprocessed data, including F0 fundamental frequency, pitch, energy, and rhythm. These features are modeled using different deep learning network frameworks. For example, F0 features are decomposed into multi-timescale representations using continuous wavelet transform; pitch features are extracted by inputting normalized data into a convolutional neural network and a bidirectional long short-term memory network; energy features are extracted stepwise using a multi-layer convolutional neural network; and rhythm features capture temporal information using a two-dimensional convolutional neural network and an attention mechanism. All of these feature extractions and modeling rely on a labeled emotion speech dataset.

[0134] During model training, labeled datasets are used to train models such as convolutional neural networks and bidirectional long short-term memory networks. The goal of training is to optimize the loss function (such as cross-entropy loss or mean squared error), and to continuously adjust the model parameters through the backpropagation algorithm, enabling the model to accurately extract sentiment features from the input data and complete classification or conversion tasks. Furthermore, to prevent overfitting, regularization techniques such as Dropout can be employed.

[0135] The prediction process of the trained model can realize a multi-scale speech emotion feature extraction method.

[0136] In the prediction phase, the new audio signal is first subjected to the same preprocessing operations as in the training phase. For example, the input audio signal undergoes pre-emphasis, framing, windowing, FFT, and other steps to generate a Mel spectrum, providing a foundation for subsequent feature extraction.

[0137] Subsequently, the trained model is used to extract features such as F0, pitch, energy, and rhythm from the audio signal. The extraction process for these features is consistent with the training phase, but at this point the model has completed parameter optimization and can directly output high-quality feature representations. For example, F0 features are decomposed into multi-timescale representations using CWT, pitch features are extracted using normalization and deep learning networks, energy features are extracted using multi-layer convolutional networks, and rhythm features are captured using CNNs and attention mechanisms.

[0138] Finally, the extracted features are concatenated and fused to generate the final emotion feature vector. This feature vector integrates information from multiple dimensions such as prosody, rhythm, and energy, comprehensively reflecting the emotional characteristics of speech. The generated emotion feature vector can be used in various application scenarios such as emotion recognition and emotional speech conversion, providing solid technical support for practical applications.

[0139] The multi-scale speech emotion feature extraction method proposed in the embodiments of the present invention is the first to extract emotion features from three dimensions of emotion description: prosody, rhythm, and energy, including four features. It uses different deep learning network frameworks to extract emotion features at multiple levels and time scales, ensuring that the extracted emotion features are representative and more natural in resemblance to real emotions. It requires less data and is convenient for practical applications.

[0140] On the other hand, embodiments of the present invention propose a multi-scale speech emotion feature extraction device, which includes an audio data preprocessing module, a pitch module, an energy module, a rhythm module and an emotion feature fusion module, wherein the pitch module includes a multi-timescale F0 module and a pitch module.

[0141] The audio data preprocessing module can be configured to perform preprocessing on audio data and obtain the Mel spectrum of the audio data.

[0142] The multi-timescale F0 module can be configured to extract one-dimensional F0 fundamental frequency features from the Mel spectrum, decompose the one-dimensional F0 fundamental frequency features into components of multiple time scales, and generate F0 feature vectors based on the components of multiple time scales.

[0143] The pitch module can be configured to extract one-dimensional pitch contour features from the Mel spectrum. It then obtains an initial pitch feature vector based on the one-dimensional pitch contour features through a convolutional neural network and a bidirectional long short-term memory network. Finally, it downsamples the initial pitch feature vector to obtain a multi-dimensional pitch feature vector.

[0144] The energy module can be configured to perform a square operation on the Mel spectrum to obtain the energy spectrum, and then extract the energy feature vector step by step from the energy spectrum through multiple convolutional neural network layers.

[0145] The rhythm module can be configured to obtain rhythm feature vectors from the Mel spectrum via multiple convolutional neural network layers and fully connected layers.

[0146] The emotion feature fusion module can be configured to concatenate and fuse F0 feature vectors, pitch feature vectors, energy feature vectors, and rhythm feature vectors to generate emotion feature vectors.

[0147] It should be noted that the functions of each module in the multi-scale speech emotion feature extraction device proposed in the embodiments of the present invention correspond one-to-one with each step of the multi-scale speech emotion feature extraction method described above. For specific implementation methods, examples and beneficial effects, please refer to the description of the method above.

[0148] The emotional feature vector extracted through the embodiments of this invention is a highly abstract feature representation that integrates various emotion-related information such as F0, pitch, energy, and rhythm. The feature vector's dimension (e.g., 30×276), after extraction and fusion by a deep learning network, can reflect multi-layered emotional information in speech. Emotional feature vectors have many potential applications in the automotive field, bringing a more intelligent, humanized, and personalized interactive experience, and improving the comfort and satisfaction of drivers and passengers. This invention can be applied in the following aspects:

[0149] 1. Emotional speech conversion

[0150] Emotional speech conversion technology utilizes extracted emotional feature vectors to convert neutral speech into speech expressing a target emotion (such as anger, sadness, or happiness). In in-vehicle intelligent assistant systems, this technology can dynamically adjust the voice interaction style based on the driver's or passenger's emotional state, resulting in a more natural and human-like interactive experience. For example, when the system detects that the driver is tense or anxious, it can generate a gentle and soothing voice response; while when the driver is in a low mood, it can switch to a more energetic and encouraging voice style. This emotion-vector-based speech conversion not only enhances the immersiveness of human-computer interaction but also alleviates negative emotions in the driver to some extent, further improving driving safety and comfort.

[0151] 2. Emotion Recognition

[0152] Emotion recognition technology, through the analysis of emotional feature vectors, can effectively identify the types of emotions in speech (such as anger, happiness, sadness, etc.) and achieve real-time emotional state analysis in mental health monitoring systems. In practical applications, this technology can be used to assess users' mental health, especially to monitor drivers' emotional fluctuations under long-distance driving or high-pressure work environments. By capturing emotional changes in speech signals in real time, the system can issue timely warnings and suggest users rest or take other emotion regulation measures. Furthermore, emotion recognition technology can be combined with other physiological signals (such as heart rate, skin conductance, etc.) to construct multimodal emotion monitoring models, providing more comprehensive data support for mental health interventions.

[0153] 3. Personalized services

[0154] Personalized services based on emotion feature vectors can dynamically adjust the in-car environment (such as seat position, audio volume, and fragrance concentration) according to the user's current emotional state, thereby providing a highly personalized driving experience. For example, when the system detects that the driver is fatigued, it can create a relaxing atmosphere by adjusting the seat angle and playing soothing music; while when the driver is in a good mood, it can enhance the audio effects and release refreshing scents to improve the overall experience. This emotion-driven personalized service can not only significantly improve driver comfort but also indirectly improve driving safety and reduce driving errors caused by emotional distress. Furthermore, the application of this technology can be extended to autonomous driving scenarios, providing passengers with a more emotionally resonant riding experience.

[0155] 4. Entertainment Recommendations

[0156] The application of sentiment vectors in entertainment recommendation primarily manifests in recommending music, advertisements, or playlists that match a user's current emotional state. For example, when the system detects a user's low mood, it can automatically play soft, soothing music to help them relax; conversely, when a user is in a high mood, it can recommend upbeat, energetic songs to further enhance their enjoyment. Furthermore, advertising recommendation systems incorporating sentiment vectors can accurately push product or service information that aligns with a user's current mood, thereby increasing ad acceptance and conversion rates. This sentiment-based entertainment recommendation mechanism not only enriches the user's in-car entertainment experience but also enhances user interaction and engagement with the in-vehicle system through intelligent content selection, further promoting the development of personalized services.

[0157] In another aspect, embodiments of the present invention propose a multi-scale speech emotion feature extraction device, the device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, implements the multi-scale speech emotion feature extraction method described in any of the above embodiments.

[0158] In another aspect, embodiments of the present invention provide a storage medium storing computer-readable instructions that, when executed by a processor, perform the multi-scale speech emotion feature extraction method described in any of the above embodiments.

[0159] In another aspect, embodiments of the present invention provide a computer program product including computer-readable instructions that, when executed by a processor, perform the steps of the multi-scale speech emotion feature extraction method described in any of the above embodiments.

[0160] The emotion feature vector extracted through the embodiments of this invention, as an abstract representation capable of capturing and expressing emotional information in speech, has broad application potential. The following are possible uses of emotion feature vectors in multiple fields:

[0161] 1. Entertainment and Media Industry

[0162] (1) Film and television production and dubbing

[0163] In movies, TV series, or radio programs, emotional feature vectors can be used to adjust a character's emotional expression, making the voice acting more closely resemble the character's emotional state. For example, in post-production, by analyzing an actor's vocal emotional feature vectors, the emotional resonance of the dialogue can be enhanced, or more realistic voices can be generated in different emotional scenes.

[0164] (2) Game Development

[0165] In games, character dialogue can achieve dynamic emotional changes through emotional feature vectors, thereby enhancing player immersion. For example, NPCs (non-player characters) can adjust their voice emotions in real time based on player behavior, making interactions more natural and vivid.

[0166] (3) Social media and communication applications

[0167] In social media platforms and instant messaging tools, emotion vectors can be used to enrich the expressiveness of voice messages. For example, voice messages sent by users can be automatically adjusted to a tone that better matches their emotional state, making communication more vivid and engaging.

[0168] 2. Virtual Assistants and Customer Service

[0169] (1) Intelligent Customer Service System

[0170] Virtual assistants and chatbots can better understand users' emotional states through emotional feature vectors and communicate in a more natural way. For example, when a user is detected to be depressed, the system can switch to a soft and soothing voice response; while when a user is emotionally agitated, it can provide more comforting verbal support.

[0171] (2) Telephone customer service and online customer service

[0172] Customer service systems can use emotional feature vectors to analyze customers' emotional states and adjust their response strategies accordingly. For example, when a customer shows anger or dissatisfaction, the system can prioritize transferring them to a senior customer service representative or automatically adjust its voice style to soothe the customer's emotions.

[0173] 3. Education sector

[0174] (1) Personalized teaching experience

[0175] Emotional feature vectors can be used to generate emotionally expressive teaching content, thereby increasing student engagement. For example, a virtual teacher can adjust the tone of their voice based on the student's learning state to motivate the student to continue learning or provide comforting verbal support when the student is feeling stressed.

[0176] (2) Emotional exchange in distance education

[0177] In distance learning, affective feature vectors can enhance emotional communication between teachers and students. For example, the system can analyze the emotional features in a teacher's voice and transmit them to students, thereby creating a more intimate learning environment.

[0178] (3) Special Needs Education

[0179] For students with special needs (such as children with autism), affective feature vectors can help generate more understandable and acceptable voice interaction content, thereby supporting their emotional management and learning process.

[0180] 4. Medical and Mental Health

[0181] (1) Psychotherapy and speech therapy

[0182] Emotional feature vectors can be used to generate speech samples for specific emotional scenarios, helping patients adjust their emotional state during psychotherapy. For example, the system can simulate calm, encouraging, or comforting speech to assist patients in emotional management.

[0183] (2) Mental health monitoring

[0184] In mental health monitoring systems, emotional feature vectors can be used to analyze users' voice data in real time and assess their emotional fluctuations. For example, by monitoring patients' voice emotional characteristics over a long period, psychological problems such as depression and anxiety can be detected at an early stage.

[0185] (3) Psychological research

[0186] Emotional feature vectors can serve as a research tool, helping researchers better understand how emotions are expressed in speech and their impact on human behavior. For example, by analyzing the emotional features of speech from different cultural backgrounds, cross-cultural differences in emotional expression can be revealed.

[0187] 5. Smart Home and Internet of Things

[0188] (1) Smart home devices

[0189] In smart home systems, emotional feature vectors can be used to adjust the way devices interact. For example, when a user is feeling down, a smart speaker can play soothing music; while when the user is feeling up, it can play energetic audio content.

[0190] (2) Personalized service recommendations

[0191] Internet of Things (IoT) devices can analyze a user's emotional state using emotional feature vectors, thereby providing personalized services. For example, a smart refrigerator can recommend recipes that match a user's current mood.

[0192] The core value of emotion feature vectors lies in their ability to capture and express emotional information in speech, making them widely applicable in scenarios requiring emotional interaction, emotion analysis, or emotion generation. Whether it's improving user experience, optimizing human-computer interaction, or advancing scientific research, emotion feature vectors will become a crucial technological foundation for the development of a future intelligent society.

[0193] The foregoing description of embodiments of the invention has been given for illustrative purposes and is not exhaustive, nor is it intended to limit the invention to the exact forms disclosed. Those skilled in the art will understand that various changes can be made without departing from the scope of the invention, and elements therein can be substituted with equivalents. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of the invention without departing from the basic scope of the invention. Therefore, the invention is not intended to be limited to the specific embodiments disclosed as the best mode contemplated for carrying out the invention; the invention will include all embodiments falling within the scope of the appended claims.

Claims

1. A method for extracting multi-scale speech emotion features, characterized in that, The method includes: The audio data preprocessing module performs preprocessing on the audio data of the speech to obtain the Mel spectrum of the audio data; The multi-timescale F0 module in the pitch module extracts a one-dimensional F0 fundamental frequency feature from the Mel spectrum, decomposes the one-dimensional F0 fundamental frequency feature into components of multiple time scales, and generates an F0 feature vector based on the components of multiple time scales. The pitch module in the pitch module extracts one-dimensional pitch contour features from the Mel spectrum. Then, the initial pitch feature vector is obtained based on the one-dimensional pitch contour features through a convolutional neural network and a bidirectional long short-term memory network. Finally, the initial pitch feature vector is downsampled to obtain a multi-dimensional pitch feature vector. The energy spectrum is obtained by squaring the Mel spectrum through the energy module, and the energy feature vector is extracted step by step through multiple convolutional neural network layers based on the energy spectrum. The rhythm module obtains rhythm feature vectors based on the Mel spectrum through multiple convolutional neural network layers and fully connected layers. The emotional feature fusion module splices and fuses the F0 feature vector, the pitch feature vector, the energy feature vector, and the rhythm feature vector to generate an emotional feature vector.

2. The method according to claim 1, characterized in that, Decomposing the one-dimensional F0 fundamental frequency feature into components at multiple time scales includes: The one-dimensional F0 fundamental frequency feature is decomposed into components of multiple time scales using continuous wavelet transform.

3. The method according to claim 1, characterized in that, The method further includes: After extracting one-dimensional pitch contour features from the Mel spectrum using the pitch module, the one-dimensional pitch contour features are normalized to remove individual speech differences. The normalized one-dimensional pitch contour features are then input into a convolutional neural network and a bidirectional long short-term memory network.

4. The method according to claim 3, characterized in that, The pitch module in the pitch module extracts one-dimensional pitch contour features from the Mel spectrum. Then, a convolutional neural network and a bidirectional long short-term memory network are used to obtain an initial pitch feature vector based on the one-dimensional pitch contour features. Finally, the initial pitch feature vector is downsampled to obtain a multi-dimensional pitch feature vector, including: The one-dimensional pitch contour feature is obtained by a vocoder based on the Mel spectrum; Batch standardization is used to normalize the one-dimensional pitch contour features to generate normalized one-dimensional pitch contour features. The normalized one-dimensional pitch contour features are projected onto the column space of the weight matrix via a fully connected layer. The features in the column space of the weight matrix are sequentially processed through an activation function, a one-dimensional convolutional neural network, another activation function, random deactivation, a bidirectional long short-term memory network, and downsampling to obtain the multidimensional pitch feature vector.

5. The method according to claim 1, characterized in that, The method further includes: The rhythm module, via an attention layer set after multiple convolutional neural network layers, generates duration sequence features by assigning different weights to input vectors at different time steps, and then inputs the duration sequence features into a fully connected layer.

6. The method according to claim 5, characterized in that, The rhythm module, through multiple convolutional neural network layers and fully connected layers, obtains rhythm feature vectors based on the Mel spectrum, including: The Mel spectrum is processed sequentially through a two-dimensional convolutional neural network layer, an activation function, a max pooling layer, another two-dimensional convolutional neural network layer, and another max pooling layer to obtain an initial rhythm feature vector. The two two-dimensional convolutional neural network layers have different input dimensions and / or the number of convolutional layer channels. The initial rhythm feature vector is input into a bidirectional long short-term memory network to extract emotional speech context feature vectors with temporal information. The emotional speech context feature vector with temporal information is input into the attention layer to generate the duration sequence feature; The duration sequence features are processed sequentially through two fully connected layers to obtain the rhythm feature vector.

7. The method according to claim 1, characterized in that, The audio data preprocessing module performs preprocessing on the speech audio data, including: Perform pre-emphasis on the audio data to obtain pre-emphasized audio data; The pre-emphasized audio data is processed into frames to obtain framed audio data; Windowing is performed frame by frame on the segmented audio data to obtain windowed audio data; Perform a discrete Fourier transform on the windowed audio data to obtain the frequency domain signal of the audio data; The frequency domain signal is subjected to a modulus-taking operation to obtain the amplitude spectrum of the audio data; The Mel spectrum of the audio data is obtained from the amplitude spectrum via a Mel filter bank.

8. The method according to claim 1, characterized in that, The energy feature vector is extracted stepwise from the energy spectrum through multiple convolutional neural network layers, including: The energy spectrum is input into the first convolutional processing module to obtain the first energy feature. The first convolutional processing module includes multiple layers, each of which includes a two-dimensional convolutional neural network, an activation function, and a max pooling layer. The first energy feature is input into the second convolutional processing module to obtain the second energy feature. The second convolutional processing module includes multiple layers, each of which includes a two-dimensional convolutional neural network, an activation function, and a max pooling layer. The second energy feature is input into the third convolutional processing module to obtain the energy feature vector. The third convolutional processing module includes multiple layers, each containing a two-dimensional convolutional neural network, an activation function, and a max-pooling layer. The two-dimensional convolutional neural networks in the first, second, and third convolutional processing modules have different input dimensions and / or the number of convolutional layer channels.

9. The method according to claim 1, characterized in that, The emotional feature fusion module concatenates and fuses the F0 feature vector, the pitch feature vector, the energy feature vector, and the rhythm feature vector to generate an emotional feature vector, including: The F0 feature vector, the pitch feature vector, the energy feature vector, and the rhythm feature vector are concatenated to generate an emotion feature vector matrix; The emotional feature vector matrix is ​​processed sequentially through a fully connected layer, an activation function, a two-dimensional convolutional neural network, an activation function, batch normalization, an activation function, random deactivation, and a two-dimensional convolutional neural network to obtain the emotional feature vector.

10. A multi-scale speech emotion feature extraction device, characterized in that, The device includes an audio data preprocessing module, a pitch module, an energy module, a rhythm module, and an emotional feature fusion module. The pitch module includes a multi-timescale F0 module and a pitch module. The audio data preprocessing module is configured to perform preprocessing on the audio data and obtain the Mel spectrum of the audio data; The multi-timescale F0 module is configured to extract one-dimensional F0 fundamental frequency features from the Mel spectrum, decompose the one-dimensional F0 fundamental frequency features into components of multiple time scales, and generate F0 feature vectors based on the components of the multiple time scales. The pitch module is configured to extract one-dimensional pitch contour features from the Mel spectrum, obtain an initial pitch feature vector based on the one-dimensional pitch contour features via a convolutional neural network and a bidirectional long short-term memory network, and then downsample the initial pitch feature vector to obtain a multi-dimensional pitch feature vector. The energy module is configured to perform a square operation on the Mel spectrum to obtain an energy spectrum, and then extract energy feature vectors step by step from the energy spectrum through multiple convolutional neural network layers. The rhythm module is configured to obtain a rhythm feature vector based on the Mel spectrum via multiple convolutional neural network layers and fully connected layers. The emotional feature fusion module is configured to concatenate and fuse the F0 feature vector, the pitch feature vector, the energy feature vector, and the rhythm feature vector to generate an emotional feature vector.

11. A multi-scale speech emotion feature extraction device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method of any one of claims 1-9.

12. A storage medium storing computer-readable instructions that, when executed by a processor, perform the method according to any one of claims 1-9.

13. A computer program product comprising computer-readable instructions, characterized in that, When the instructions are executed by the processor, the steps of the method according to any one of claims 1-9 are performed.

Images