Short voice voiceprint extraction method based on TFCA-Res2Net

Abstract

The invention discloses a short voice voiceprint extraction method based on TFCA-Res2Net, and the method comprises the steps: carrying out the preprocessing of an original voice signal, and carrying out the short-time Fourier transformation of the audio of the original voice signal after preprocessing, and generating a corresponding short-time Fourier spectrogram; training Res2Net according to the obtained short-time Fourier spectrogram, and introducing a TFCA attention mechanism to obtain a short-speech speaker recognition network; in the Res2Net training process, taking a cross entropy loss function as a training objective function, and obtaining a final network model by updating parameters to minimize loss; and extracting audio features by using the trained network model, performing Euclidean distance calculation on the audio features and feature vectors in a database, comparing the Euclidean distance with a threshold value, and finally identifying a speaker. Compared with a traditional short voice voiceprint extraction method, under the conditions that the voice length is short and the noise environment is complex, the verification accuracy is remarkably improved, the verification effect is more reliable, and higher performance and better robustness are shown.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a short speech voiceprint extraction method based on TFCA-Res2Net. Background Art

[0002] Speaker recognition is a biometric authentication technology that realizes speaker identification by analyzing the speaker's voiceprint features. It has been widely used in security, access control, evidence collection, telephone banking and other fields. Therefore, speaker recognition technology has very broad application prospects and research value. Usually, in order to accurately identify the speaker, the speaker recognition system requires a large amount of medium- and long-term voice data (more than five seconds) to train the speaker model. However, in a real environment, it is extremely difficult to obtain a large amount of such voice data. The audio data usually obtained by the present invention is fragmented and short in duration. This seriously limits the application prospects of the speaker recognition system. On the other hand, with the widespread use of voice interaction devices, the present invention needs to perform fast and accurate speaker recognition. Therefore, the research on short speech speaker recognition is very important, but the present invention finds that current research rarely pays attention to short speech speaker recognition. Summary of the invention

[0003] The purpose of the present invention is to provide a short speech voiceprint extraction method based on TFCA-Res2Net. The method can significantly improve the verification accuracy when the sound length is short and the noise environment is complex, and the verification effect is more reliable, showing higher performance and better robustness.

[0004] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0005] The present invention comprises the following steps:

[0006] S1: Pre-emphasize, frame and window the original speech signal, and transform the pre-processed original speech signal audio into a corresponding short-time Fourier spectrogram through short-time Fourier transform; because short speech requires a more refined spectrogram, and the commonly used Mel harms the accuracy of the spectrogram. In addition, the present invention also studies the window length of the short-time Fourier and finds out the window length suitable for short speech;

[0007] S2: Train Res2Net based on the short-time Fourier spectrogram obtained in step S1, and introduce the TFCA attention mechanism to obtain a short speech speaker recognition network;

[0008] S3: During the training of Res2Net, the cross entropy loss function is used as the training objective function, and the final network model is obtained by minimizing the loss by updating the parameters;

[0009] S4: Use the network model trained in step S3 to extract audio features, calculate the Euclidean distance with the feature vector in the database, and compare it with the threshold to finally identify the speaker.

[0010] The beneficial effects of the present invention are:

[0011] The present invention is a short speech voiceprint extraction method based on TFCA-Res2Net. Compared with the prior art, the present invention shows higher performance and better robustness in short speech scenarios. In particular, when the sound length is short and the noise environment is complex, the verification accuracy of the system is significantly improved, and the verification effect is more reliable.

[0012] 1. The original audio is transformed by short-time Fourier transform to generate the corresponding spectrogram. Because short speech requires a more refined spectrogram, and the commonly used Mel transform damages the accuracy of the spectrogram. In addition, the present invention also studies the window length of the short-time Fourier transform and finds out the window length suitable for short speech.

[0013] 2. Res2Net network structure to increase discriminative features and improve model robustness, thereby more effectively utilizing the limited acoustic features in short speech clips.

[0014] 3. TFCA attention module. This module can effectively learn the relationship between time domain, frequency domain and channel, and enhance the global feature extraction capability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall structure diagram of the method of the present invention;

[0016] Figure 2 This is the network structure diagram of TFCA-Res2Net;

[0017] Figure 3 This is the structure diagram of the small convolution block of Res2Net block;

[0018] Figure 4 This is the TFCA attention structure diagram;

[0019] Figure 5 It is a vector dimension reduction diagram of the result of the embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0021] like Figure 1-2 As shown: The present invention comprises the following steps:

[0022] S1, pre-emphasize, frame and window pre-process the original speech signal, and transform the pre-processed speech signal through the original audio through short-time Fourier transform to generate the corresponding short-time Fourier spectrogram. Because short speech requires a more sophisticated spectrogram, and the commonly used Mel harms the accuracy of the spectrogram. In addition, the present invention also studies the window length of the short-time Fourier and finds out the window length suitable for short speech. The specific steps are as follows:

[0023] S1.1: Use a high-pass filter as shown below for pre-emphasis:

[0024] H(Z)=1-μz -1

[0025] Wherein, H(Z) is the transfer function in the z domain, z represents the coordinate value in the z domain, and μ represents the pre-emphasis coefficient, which ranges from 0.9 to 1.

[0026] In the time domain, the signal x[n] is operated as follows:

[0027] y[n]=x[n]-μx[n-1]

[0028] Wherein, x[n] represents the nth sampling point of the original speech signal, x[n-1] represents the n-1th sampling point of the original speech signal. μ represents the pre-emphasis coefficient, ranging from 0.9 to 1, and the value of the present invention is 0.95. y[n] represents the nth sampling point of the original speech signal after emphasis.

[0029] S1.2: Since the speech information is a non-stationary signal, it is necessary to frame the output y[n] after weighting. In the present invention, a Hamming window is used for windowing, and the Hamming window is shown as follows:

[0030]

[0031] y(n) represents the speech signal after frame division, w(n,a) represents the window function of the Hamming window, where a takes a value of 0.46, n=0, 1, ..., N-1, N is the frame length, and the speech signal after windowing is: s(n)=y(n)×w(n);

[0032] S1.3: After frame division and windowing, the Fourier transform is used to obtain the spectrum of the speech signal in the frequency domain, as shown in the following formula:

[0033]

[0034] Where X(m,n) is the complex spectrum value of frequency m at time point n, x(n) is the input signal, w(k) is the window function, usually Hanning window function or Gauss window function, and K is the size of the window. The selection of window length is very important. If the window is too narrow, the signal in the window is too short, resulting in inaccurate frequency analysis and poor frequency resolution of the spectrogram. If the window is too wide, the information in the time domain is not fine enough and the time domain resolution of the spectrogram is low. The present invention also studies the window length of the short-time Fourier transform and finds out the size of the window suitable for short speech. In the present invention, the window size is set to 50ms.

[0035] S2. The short-time Fourier spectrogram obtained in step S1 is used to train Res2Net, and the TFCA attention mechanism is introduced to obtain a short speech speaker recognition network, which specifically includes:

[0036] A single 7×7 convolution layer, four cascaded large convolution blocks and an output layer; the four cascaded large convolution blocks contain 3, 4, 6, and 3 small convolution blocks respectively; the channel dimensions of the four cascaded large convolution blocks are 16, 32, 64, and 128 respectively; the output layer contains an average pooling module, a fully connected layer, and a Softmax layer.

[0037] According to the short speech voiceprint recognition method based on TFCA-Res2Net of the present invention, further, the small convolution block includes a Res2Net block and a TFCA attention. The Res2Net block is as follows Figure 3 As shown, specifically including:

[0038] S2.1: Use 1×1 convolutional layer to expand the channel dimension;

[0039] S2.2: Secondly, use batch normalization layer to normalize the features;

[0040] S2.3: Use the ReLu function for nonlinear activation to obtain feature x;

[0041] S2.4: Divide the activated feature x into four subsets x1, x2, x3, x4 along the channel dimension. The number of channels in each feature subset is one quarter of x.

[0042] S2.5: Perform 3×3 convolution on each feature subset;

[0043] S2.6: Use batch normalization layers to normalize features.

[0044] S2.7: Use the ReLu function for nonlinear activation to obtain features y1, y2, y3, y4;

[0045] S2.8: Concatenate features y1, y2, y3, y4 along the channel dimension to obtain feature y;

[0046] S2.9: Convolve y using a 1×1 convolutional layer;

[0047] S2.10: Normalize the features y using a batch normalization layer.

[0048] According to the short speech voiceprint recognition method based on TFCA-Res2Net of the present invention, further, the small convolution block includes a Res2Net block and a TFCA attention. TFCA attention is as follows Figure 4 As shown, specifically including:

[0049] S2.11: Calculate channel-frequency attention M for feature y CF (y);

[0050] S2.12: Calculate channel-time attention M for feature y CT (y);

[0051] S2.13: Channel-frequency attention M CF (y) and channel-temporal attention M CT (y) Multiply point by point to get the channel-frequency-time attention M CTF (y).

[0052] S2.14: Channel-frequency-time attention M CTF Multiply (y) by y point by point to get y'.

[0053] S3. During the model training process, the cross entropy loss function is used as the training objective function. The final network model is obtained by minimizing the loss by updating the parameters. The specific steps are as follows:

[0054] The cross entropy loss function is defined as follows:

[0055]

[0056] Among them, m represents the number of samples, y' i represents the true value of the i-th sample, y i represents the predicted output value of the i-th sample, and L represents the loss value.

[0057] S4. Use the model trained in step S3 to extract audio features, calculate the Euclidean distance with the feature vector in the database, and compare it with the threshold to finally identify the speaker, as follows:

[0058] The Euclidean distance between the utterance-level features and the feature vectors already in the speaker database is calculated using the following formula:

[0059]

[0060] In the formula, x i is the extracted utterance-level feature, y i is the target in the library, L is the length of the speech feature vector. The speaker is determined based on the size of d and the threshold.

[0061] The dataset used in this method is the Voxceleb dataset, which has been widely used internationally in recent years. Voxceleb is a large noisy speech dataset collected "in the wild" and does not rely on text. The training set Voxceleb2 dataset contains 1,092,009 voices from 5,994 people. The test set is the test set of Voxceleb1, which contains 4,874 voices from 40 people, and uses the test list provided by Voxceleb for a total of 37,720 test pairs.

[0062] The test results after using this method have improved the accuracy compared with the ThinResNet34 and Res2Net models.

[0063]

[0064] Table 1

[0065] Finally, the present invention conducts an ablation experiment on the transformed window length. The present invention controls the network architecture and the attention module unchanged, and only changes the transformed window length to find the most suitable window length for the short speech speaker recognition task. The test results are shown in Table 2. It can be seen that the window length of 50ms is undoubtedly the most suitable window length for the short speech speaker recognition task.

[0066]

[0067]

[0068] Table 2

[0069] In order to better demonstrate TFCA-Res2Net's ability to learn short speech features, the present invention uses the t-SNE visualization method to visualize the TFCA-Res2Net short language speaker system. 15 different speaker samples are selected from the Voxceleb-1 test set and divided into different colors. Ten audios are randomly selected from each speaker sample, and then 10 test clips of 1 second in length are randomly extracted from each audio. The visualization results of TFCA-Res2Net are shown in Figure 1. Figure 5As shown in the figure, the visualization results of TFCA-Res2Net not only have smaller inter-class distances and larger inter-class distances, but also have basically no classification errors and conflicts. This shows that the TFCA-Res2Net we proposed is effective and has stronger feature extraction capabilities for short languages.

[0070] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0071] In the short speech scenario, due to factors such as the short length of the speech segment, the limited number of features and noise interference, the performance of the existing speaker verification system has declined significantly. In view of the problems existing in the current short speech speaker verification system, a short speech voiceprint extraction method based on TFCA-Res2Net is proposed. The Res2Net network structure is used to increase the discriminant features and improve the robustness of the model, so as to more effectively utilize the limited acoustic features in the short speech segment. The TFCA attention module is designed. This module can effectively learn the relationship between the time domain, frequency domain and channel, and enhance the global feature extraction ability of the network. At the same time, the present invention studies the optimal window length of the short-time Fourier spectrogram in short speech. The original audio is transformed by short-time Fourier transform to generate the corresponding spectrogram. Because short speech requires a more refined spectrogram, the commonly used Mel damages the accuracy of the spectrogram. In addition, the present invention also studies the window length of the short-time Fourier and finds out the window length suitable for short speech. Experimental results show that the proposed short speech speaker verification system based on TFCA-Res2Net shows higher performance and better robustness in short speech scenarios compared with traditional systems. Especially in the case of short voice length and complex noise environment, the verification accuracy of the system is significantly improved, and the verification effect is more reliable.

[0072] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A short speech voiceprint extraction method based on TFCA-Res2Net, characterized in that: The following steps are involved: S1: preprocessing the original speech signal, and generating a corresponding short-time Fourier spectrogram by short-time Fourier transforming the preprocessed original speech signal audio; S2: Train Res2Net based on the short-time Fourier spectrogram obtained in step S1, and introduce the TFCA attention mechanism to obtain a short speech speaker recognition network; S3: During the training of Res2Net, the cross entropy loss function is used as the training objective function, and the final network model is obtained by minimizing the loss by updating the parameters; S4: Use the network model trained in step S3 to extract audio features, calculate the Euclidean distance with the feature vector in the database, and compare it with the threshold to finally identify the speaker.

2. The short speech voiceprint extraction method based on TFCA-Res2Net according to claim 1, characterized in that: The preprocessing in step S1 includes pre-emphasis, frame division, and windowing preprocessing to obtain a short-time Fourier spectrum. The specific steps are as follows: S1.1: Use a high-pass filter as shown below for pre-emphasis: H(Z)=1-μz -1 Wherein, H(Z) is the transfer function in the z domain, z represents the coordinate value in the z domain, and μ represents the pre-emphasis coefficient; In the time domain, the signal x[n] is operated as follows: y[n]=x[n]-μx[n-1] Wherein, x[n] represents the nth sampling point of the original speech signal, x[n-1] represents the n-1th sampling point of the original speech signal; μ represents the pre-emphasis coefficient; S1.2: Frame the output y[n] after emphasis; use a Hamming window for windowing, and the Hamming window is shown in the following formula: Wherein, y(n) represents the speech signal after frame division, w(n,a) represents the window function of the Hamming window, N is the frame length, and the speech signal after windowing is: s(n)=y(n)×w(n); S1.3: After frame division and windowing, the Fourier transform is used to obtain the spectrum of the speech signal in the frequency domain, as shown in the following formula: Where X(m,n) is the complex spectrum value of frequency m at time point n, x(n) is the input signal, w(k) is the window function, and K is the size of the window.

3. The short speech voiceprint extraction method based on TFCA-Res2Net according to claim 2, characterized in that: The step S2 specifically includes: a single 7×7 convolution layer, four cascaded large convolution blocks and an output layer; the four cascaded large convolution blocks contain 3, 4, 6, and 3 small convolution blocks respectively; the channel dimensions of the four cascaded large convolution blocks are 16, 32, 64, and 128 respectively; the output layer contains an average pooling module, a fully connected layer, and a Softmax layer.

4. The short speech voiceprint extraction method based on TFCA-Res2Net according to claim 3 is characterized in that: The small convolution block includes a Res2Net block and a TFCA attention block; the Res2Net block specifically includes: S2.1: Use 1×1 convolutional layer to expand the channel dimension; S2.2: Secondly, use batch normalization layer to normalize the features; S2.3: Use the ReLu function for nonlinear activation to obtain feature x; S2.4: Divide the activated feature x into four subsets x1, x2, x3, x4 along the channel dimension. The number of channels in each feature subset is one quarter of x. S2.5: Perform 3×3 convolution on each feature subset; S2.6: Use batch normalization layers to normalize features. S2.7: Use the ReLu function for nonlinear activation to obtain features y1, y2, y3, y4; S2.8: Concatenate features y1, y2, y3, y4 along the channel dimension to obtain feature y; S2.9: Convolve y using a 1×1 convolutional layer; S2.10: Normalize the features y using a batch normalization layer.

5. The short speech voiceprint extraction method based on TFCA-Res2Net according to claim 4, characterized in that: The TFCA attention specifically includes: S2.11: Calculate channel-frequency attention M for feature y CF (y); S2.12: Calculate channel-time attention M for feature y CT (y); S2.13: Channel-frequency attention M CF (y) and channel-temporal attention M CT (y) Multiply point by point to get the channel-frequency-time attention M CTF (y); S2.14: Channel-frequency-time attention M CTF (y) is multiplied point by point with y to get y ’ .

6. The short speech voiceprint extraction method based on TFCA-Res2Net according to claim 5, characterized in that: In the process of Res2Net training, step S3 adopts the cross entropy loss function training function to obtain the final trained network model, which specifically includes: The cross entropy loss function is defined as follows: Among them, m represents the number of samples, y ’ i represents the true value of the i-th sample, y i represents the predicted output value of the i-th sample, and L represents the loss value.

7. The short speech voiceprint extraction method based on TFCA-Res2Net according to claim 6, characterized in that: The step S4 is specifically: calculating the Euclidean distance between the utterance-level feature and the feature vector already in the speaker database, and the calculation formula is as follows: In the formula, x i is the extracted utterance-level feature, y i is the target already in the library, L is the length of the feature vector; The speaker is determined based on the size of d and the threshold.

Images