Multi-task speech recognition method and device for MFCC features based on fractional Fourier transform

Abstract

The invention relates to a multi-task speech recognition method based on MFCC features of fractional Fourier transform. The multi-task speech recognition method comprises the following steps: collecting speech signals; preprocessing the voice signal; performing fractional order Fourier transform on each frame of voice signal after preprocessing; extracting a power spectrum of fractional Fourier transform, and enabling the power spectrum to pass through a Mel filter bank to obtain Mel frequency characteristics; obtaining MFCC features according to the Mel frequency features; and performing emotion recognition classification and voiceprint recognition classification on the MFCC features through a convolutional neural network, and outputting a recognition result. According to the method, the dynamic characteristics of the voice signals can be effectively captured through the time-frequency domain flexible adjustment capability of fractional Fourier transform, and compared with a traditional method, the method has higher recognition accuracy and anti-noise performance in a complex environment. The method is suitable for scenes of voice interaction, emotion analysis, identity authentication and the like, and has relatively high practical value and application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech signal recognition and deep learning, and in particular relates to a multi-task speech recognition method and device based on MFCC features of fractional Fourier transform. Background Art

[0002] Speech signal processing technology is one of the important research directions in the field of artificial intelligence and is widely used in scenarios such as intelligent voice assistants, sentiment analysis, and identity authentication. As the core tasks of speech signal processing, emotion recognition and voiceprint recognition have received increasing attention in recent years. Emotion recognition aims to analyze and judge the speaker's emotional state from the speech, while voiceprint recognition identifies the speaker's identity through individual differences in speech signals. These two types of tasks have different requirements for feature extraction and require a high level of expression of the dynamic characteristics and personalized features of speech signals. However, the existing technology still has many deficiencies in feature extraction and multi-task processing.

[0003] Existing speech signal processing technologies usually use short-time Fourier transform (STFT) to extract the spectral features of speech signals and generate Mel-frequency cepstral coefficients (MFCC) as feature input. Although this method is simple to calculate, it has significant limitations. First, the short-time Fourier transform is based on a fixed time window and cannot capture the rapidly changing dynamic characteristics of speech signals, resulting in insufficient feature expression capabilities. Second, MFCC features are highly sensitive to noise and are easily interfered with in complex backgrounds. In addition, the feature requirements of emotion recognition and voiceprint recognition are different. Traditional methods are difficult to meet the accuracy requirements of these two types of tasks at the same time, and lack efficient support for multiple tasks.

[0004] In recent years, with the rapid development of deep learning, models such as convolutional neural networks (CNN) have been widely used in speech signal processing. Deep learning technology can automatically extract features from speech signals and significantly improve the recognition performance of single tasks. However, the actual effect of these methods depends to a large extent on the quality of input features, and traditional MFCC features are still a bottleneck. At the same time, most existing deep learning models focus on a single task, lack an effective collaborative framework for the joint processing of emotion recognition and voiceprint recognition, and cannot achieve efficient multi-task learning.

[0005] As a generalized form of Fourier transform, fractional Fourier transform (FrFT) can flexibly switch the characteristics of time domain and frequency domain by adjusting the transform order, which is particularly suitable for the analysis of non-stationary signals (such as speech signals). Compared with the traditional Fourier transform, fractional Fourier transform has significant advantages: first, it can capture the dynamic change characteristics in speech signals, and second, it shows higher robustness in complex noise environments. In addition, fractional Fourier transform can extract more comprehensive signal features, provide high-quality input for emotion recognition and voiceprint recognition, and lay the foundation for multi-task collaborative processing.

[0006] To sum up, the existing technology mainly has the following problems: first, the traditional MFCC feature extraction method cannot accurately capture the dynamic characteristics of speech signals and lacks noise resistance; second, the existing speech signal recognition technology lacks a multi-task optimization framework for emotion recognition and voiceprint recognition; third, the existing Fourier transform method has insufficient ability to model the non-stationary characteristics of speech signals, which affects the effect of feature extraction. Summary of the invention

[0007] The object of the present invention is to provide a multi-task speech recognition method and device based on MFCC features of fractional Fourier transform, which can better separate and represent speech signal components of different frequencies, thereby improving the accuracy of classification and recognition;

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A multi-task speech recognition method based on MFCC features of fractional Fourier transform, comprising the following steps:

[0010] Collecting voice signals;

[0011] Preprocessing the speech signal;

[0012] Performing fractional Fourier transform on each frame of preprocessed speech signal;

[0013] Extract the power spectrum of the fractional Fourier transform, pass the power spectrum through the Mel filter bank, and obtain the Mel frequency feature;

[0014] According to the Mel frequency features, MFCC features are obtained;

[0015] The MFCC features are classified for emotion recognition and voiceprint recognition through convolutional neural networks, and the recognition results are output.

[0016] Preferably, the pretreatment comprises the following steps:

[0017] The speech signal is sampled, pre-emphasized, framed and Hamming windowed.

[0018] Preferably, the fractional Fourier transform is expressed as:

[0019]

[0020] Among them, K P (u,t)=A α exp[jπ(u 2 Cotα-2utcscα+t 2 cotα)] is the kernel function, α=pπ / 2, p is 0.4~0.6.

[0021] Preferably, p is 0.5.

[0022] Preferably, the extracting the power spectrum of the fractional Fourier transform comprises the following steps:

[0023] Modulo the result of fractional Fourier transform, and convert the complex output signal into real form;

[0024] The power spectrum of each frame signal is generated by squaring the modulus value.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the multi-task speech recognition method based on MFCC features of fractional Fourier transform described above are performed.

[0026] According to another aspect of the present invention, there is also provided a multi-task speech recognition device based on MFCC features of fractional Fourier transform, comprising:

[0027] Memory, for storing software applications,

[0028] The processor is used to execute the software application program, and each program of the software application program correspondingly executes the steps in the multi-task speech recognition method based on MFCC of fractional Fourier transform described above.

[0029] The present invention introduces fractional Fourier transform, which has flexible adjustment capability in time and frequency domain, to accurately capture the dynamic change characteristics of speech signals, and is more suitable for processing non-stationary signals than traditional Fourier transform. This provides richer time and frequency information for MFCC feature extraction.

[0030] The signal after fractional Fourier transform combined with MFCC features retains the key information in the time domain and frequency domain, significantly improving the expressiveness of speech emotion features and voiceprint features. In multi-task scenarios (emotion recognition and voiceprint recognition), the performance is improved. The output features of fractional Fourier transform are suitable for multi-task collaborative recognition, and can support the two major tasks of emotion classification and voiceprint recognition under a unified model, avoiding the model limitations of a single task.

[0031] The present invention can effectively capture the dynamic characteristics of speech signals and has higher recognition accuracy and noise resistance in complex environments than traditional methods. This method is suitable for scenarios such as speech interaction, emotion analysis, and identity authentication.

[0032] The present invention significantly improves the expressiveness and robustness of feature extraction by introducing fractional Fourier transform, while achieving higher computational efficiency and classification accuracy in a multi-task processing framework.

[0033] The present invention realizes the recognition of different emotion types and the recognition of different character voiceprints by extracting MFCC features of speech signals by fractional Fourier transform and classifying them by convolutional neural network, wherein the speech emotion recognition rate can reach more than 83%, and the speech voiceprint recognition rate can reach more than 80%. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the process of the present invention;

[0035] Figure 2 This is a flow chart of the present invention using a convolutional neural network to perform speech emotion recognition;

[0036] Figure 3 This is a flow chart of the present invention using a convolutional neural network to perform speech voiceprint recognition;

[0037] Figure 4 Schematic diagram of speech emotion recognition at p=0.5;

[0038] Figure 5 This is a schematic diagram of voiceprint recognition for multi-speaker CASIA dataset when p is 0.5;

[0039] Figure 6 Schematic diagram of voiceprint recognition on the CASIA dataset when p is 0.5. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0041] like Figure 1As shown, the present invention provides a multi-task speech recognition method based on MFCC features of fractional Fourier transform, comprising the following steps:

[0042] S1, collect voice signals;

[0043] This embodiment takes the CASIA database as an example, and then defines a list "emotions" containing 6 basic emotions.

[0044] In this embodiment, in addition to selecting the CASIA database for voice and voiceprint recognition, the voices of different characters are collected to generate a multi-speaker CASIA data set, which is numbered.

[0045] S2, preprocessing the speech signal;

[0046] Specifically, the preprocessing includes the following steps: sampling, pre-emphasis, framing and Hamming window processing of the speech signal.

[0047] More specifically, the sampling rate is 16000. During the framing process of the speech signal, the length of each frame is 256 samples, and the frame shift is 128 samples. This framing process helps to convert long audio signals into short signal segments, which is convenient for feature extraction and model training. During the framing process, if the last sample is less than 256 samples, it will be skipped. This is to ensure that each segment has a consistent length and avoid errors caused by different lengths in subsequent processing.

[0048] S3, performing fractional Fourier transform on each frame of the preprocessed speech signal;

[0049] Specifically, the fractional Fourier transform is expressed as:

[0050]

[0051] Among them, K P (u,t)=A α exp[jπ(u 2 Cotα-2utcscα+t 2 cotα)] is the kernel function, α=pπ / 2, p is 0.4-0.6, preferably, p=0.5, wherein the result of the fractional Fourier transform is in complex form, and its real part and imaginary part respectively reflect different frequency components of the signal.

[0052] S4, extracting the power spectrum of the fractional Fourier transform, passing the power spectrum through a Mel filter bank, and obtaining a Mel frequency feature;

[0053] Specifically, the power spectrum of each frame is calculated by taking the modulus and squaring the fractional Fourier transform result.

[0054] The purpose of modulo is to convert the complex Fourier transform result into real number form. The calculation formula is:

[0055]

[0056] The square operation is used to generate a power spectrum to represent the energy distribution of different frequency components. Then, the power spectrum is passed through the Mel filter bank to obtain the Mel frequency feature. The corresponding relationship between the Mel frequency feature and the frequency f is:

[0057]

[0058] Among them, the design of the Mel filter adopts a triangular filter, each filter has the characteristic of smooth transition on the frequency axis, which helps to capture the spectrum details of the speech signal;

[0059] S5. Obtain MFCC feature data according to the Mel frequency domain features;

[0060] Specifically, the logarithm of the energy of the Mel spectrum is taken;

[0061] L[m]=log(E[m])

[0062] Where E[m] is the energy after processing by the Mel filter bank;

[0063] Perform discrete cosine transform (DCT) on the logarithmic Mel spectrum to obtain MFCC features;

[0064]

[0065] Where M is the number of Mel filters and n is the dimension of MFCC.

[0066] In this embodiment, 40 MFCC features are extracted for each sample;

[0067] Calculate the maximum number of frames in all MFCC features and pad each MFCC feature to make them of the same length.

[0068] S6, performing emotion recognition classification and voiceprint recognition classification on the MFCC feature data through a convolutional neural network, and outputting the recognition results;

[0069] Specifically, the convolutional neural network includes:

[0070] At least three convolutional layers, with a kernel size of 3×3, 32, 64, and 128 kernels, respectively, and ReLU as the activation function, which introduces nonlinear characteristics to help the model learn complex features;

[0071] A maximum pooling layer is added after each convolutional layer to reduce the spatial dimension of the feature map. The pooling window size is 2×22, which means taking the maximum value in each 2x2 area. This helps to reduce the amount of calculation, extract the main features, and prevent overfitting.

[0072] Flattening layer and fully connected layer, flatten the multi-dimensional feature map into a one-dimensional vector, so that it can be input into the fully connected layer, so that each input node is connected to each output node

[0073] An output layer uses a softmax activation function to output the probability distribution of emotion categories and voiceprint categories for multi-class classification. The output value of each node represents the probability that the input sample belongs to the corresponding emotion category.

[0074] Finally, the predicted emotion category is output according to the maximum value of the corresponding emotion category probability.

[0075] More specifically, in this embodiment, during the emotion recognition process, Figure 2 As shown in Figure 2, testing with a ratio of 4:1 between the training set and the test set can yield better recognition results. When p is 0.5, the average recognition rate of emotion categories is 83.3%. The recognition results of different speech emotion categories are shown in Figure 2. Figure 4 shown.

[0076] The steps for voiceprint recognition and classification using convolutional neural networks are the same as those for speech emotion recognition and classification.

[0077] In the voiceprint recognition process, according to the training set: Figure 3 As shown in the figure, the test set is tested with a ratio of 4:1, which can obtain better recognition results. When p is 0.5, Figure 5 and Figure 6 As shown, the average recognition rate of speech voiceprint categories in the multi-speaker CASIA dataset is 80%, and the recognition rate of speech voiceprint categories in the CASIA dataset is over 90%.

[0078] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the multi-task speech recognition method based on MFCC features of fractional Fourier transform described above are performed.

[0079] According to another aspect of the present invention, there is also provided a multi-task speech recognition device based on MFCC features of fractional Fourier transform, comprising:

[0080] Memory, for storing software applications,

[0081] The processor is used to execute the software application program, and each program of the software application program correspondingly executes the steps in the multi-task speech recognition method based on MFCC features of fractional Fourier transform described above.

Claims

1. A multi-task speech recognition method based on MFCC features of fractional Fourier transform, comprising the following steps: Collecting voice signals; Preprocessing the speech signal; Performing fractional Fourier transform on each frame of preprocessed speech signal; Extract the power spectrum of the fractional Fourier transform, pass the power spectrum through the Mel filter bank, and obtain the Mel frequency feature; According to the Mel frequency features, MFCC features are obtained; The MFCC features are classified for emotion recognition and voiceprint recognition through convolutional neural networks, and the recognition results are output.

2. The identification method according to claim 1, wherein the preprocessing comprises the following steps: The speech signal is sampled, pre-emphasized, framed and Hamming windowed.

3. The identification method according to claim 1, characterized in that: The order p of the fractional Fourier transform is 0.4-0.

6.

4. The identification method according to claim 3, characterized in that: The p is 0.

5.

5. The identification method according to claim 1, characterized in that: The power spectrum of the fractional Fourier transform is extracted, comprising the following steps: Modulo the result of fractional Fourier transform, and convert the complex output signal into real form; The power spectrum of each frame signal is generated by squaring the modulus value.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-task speech recognition method based on MFCC features of fractional Fourier transform according to any one of claims 1 to 5 are performed.

7. A multi-task speech recognition device based on fractional Fourier transform (MFCC) features, characterized in that: include: Memory, for storing software applications, A processor is used to execute the software application, each program of the software application correspondingly executes the steps in the multi-task speech recognition method based on MFCC features of fractional Fourier transform according to any one of claims 1 to 5.

Images