Fractional domain feature extraction method and device for deep learning speaker verification, and medium

Abstract

The invention belongs to the technical field of voiceprint recognition, and particularly relates to a fractional domain feature extraction method and device for deep learning speaker verification and a medium. Comprising the following steps: preprocessing a voice signal; respectively using modulation factors of different orders extracted from fractional order Fourier transform for the selected window function to obtain a fractional order window function; performing short-time Fourier transform on the voice signals to obtain a power spectrum spectrogram; filtering the obtained power spectrum spectrogram through a Mel filter bank and taking logarithms to obtain M-dimensional filter bank characteristics; for the Fbank features, uniformly distributing learnable parameters with an initial value of 1 as weights, and using a softmax activation function to limit the sum of the weights to be unchanged; the weighted Fbank features are connected in series and spliced into a multi-window fractional order Fbank feature in the time dimension according to the order from high to low, and the multi-window fractional order Fbank feature is input into a deep learning model for training; according to the method, the importance of different order features is highlighted, and the accuracy of a deep learning speaker verification system can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of voiceprint recognition, and in particular relates to a fractional domain feature extraction method, device and medium for deep learning speaker verification. Background Art

[0002] Speaker verification belongs to the field of voiceprint recognition, that is, given two audios, determine whether they are spoken by the same person. Traditional speaker verification systems mainly rely on extracting features from speech signals to represent the identity information of the speaker. Common feature extraction methods include filter bank features (Fbank) and Mel-frequency cepstral coefficients (MFCC). Fbank features simulate the auditory characteristics of the human ear and use a set of filters to analyze the spectral energy distribution of speech signals. MFCC is further processed based on Fbank, using discrete cosine transform (DCT) to reduce the correlation between features. Although these methods have proven their effectiveness in practice, they still have some limitations, such as difficulty in fully capturing subtle differences and dynamic features in speech signals.

[0003] In recent years, deep learning technology has made significant progress in the field of speech processing. Models such as convolutional neural networks (CNN) and time-delay neural networks (TDNN) are widely used in speaker verification tasks. CNN can effectively learn the time-frequency structure of speech signals through its local connection and weight sharing characteristics. TDNN better captures the dynamic characteristics of speech by considering the temporal dependency of input features. These deep learning models have improved the performance of the system to a certain extent, but they still face the challenge of how to more effectively extract and utilize the rich information in speech signals. Summary of the invention

[0004] The purpose of the present invention is to provide a fractional domain feature extraction method for deep learning speaker verification, which can utilize multi-order fractional window functions to provide more flexible and comprehensive information than traditional features, and provide richer signal representation in the time-frequency plane. Combining this fractional domain feature with the existing deep learning model has the potential to reveal subtle features of speech signals that are difficult to capture by traditional methods.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A score domain feature extraction method for deep learning speaker verification, comprising the following steps:

[0007] S1, pre-emphasize the high frequency part of the speech signal;

[0008] Specifically, a pre-emphasis coefficient of 0.97 is selected for pre-emphasis as follows:

[0009] x(t)=x0(t)-0.97*x0(t-1)

[0010] Where x0(t) is the original speech signal, and x(t) is the pre-emphasized speech signal;

[0011] S2, using N modulation factors of different orders extracted from fractional Fourier transform to obtain a fractional window function for the selected window function;

[0012] Specifically, the fractional Fourier transform is expressed as:

[0013]

[0014] Kernel function K p (t,u) can be expressed as

[0015]

[0016] in Rotation angle of the time-frequency surface

[0017] Transform the kernel function formula and use the window function w(t) to add a window, and get

[0018]

[0019] The fractional-order window function is composed of the original window function and the modulation factor Combined, it is expressed as:

[0020]

[0021] Specifically, w(t) is a Hamming window expressed as:

[0022]

[0023] Window function length N=400;

[0024] The orders are p1=1, p2=0.8, p3=0.6, p4=0.4, p5=0.2;

[0025] S3, using 5 fractional-order window functions of different orders to obtain a power spectrum spectrogram for each speech signal;

[0026] Extract the M-dimensional filter bank (Fbank) features from the obtained power spectrum spectrogram;

[0027] The orders of the five fractional-order window functions are p1=1, p2=0.8, p3=0.6, p4=0.4, and p5=0.2;

[0028] Specifically, the step of extracting the power spectrum spectrum diagram includes the following steps:

[0029] Framing: divide the input speech signal into overlapping segments, the length of which is consistent with the length of the window function;

[0030] Windowing: Each frame of speech signal is windowed using 5 fractional-order window functions of different orders;

[0031] Perform a Fast Fourier Transform (FFT) on each frame to convert the time domain signal to the frequency domain; calculate the power spectrum; take the square of the modulus of the FFT result and divide it by the signal length for normalization;

[0032] Generate a spectrogram: Arrange the spectrum of each frame in time order to form a two-dimensional spectrogram;

[0033] The extracted power spectral density is expressed as:

[0034]

[0035] Each speech signal is windowed using 5 fractional-order window functions of different orders;

[0036] S4, for the power spectrum spectrogram, use the M-Vermeer filter bank and take the logarithm to extract the filter bank (Fbank) features;

[0037] Specifically, an M-dimensional Fbank feature is extracted from each power spectrum spectrogram, that is, the number of speech signals × N M-dimensional Fbank features are established;

[0038] The Fbank feature is expressed as:

[0039]

[0040] Among them, H l [k] is an 80-dimensional Mel filter bank, l = 1, 2, ..., 80 represents the Mel filter index, m = 1, 2, ..., T represents the index of the time dimension, R is the frame shift, and ∈ is a very small positive number to avoid zero or negative values ​​when taking the logarithm

[0041] Extracting Fbank features based on N power spectrum spectrograms of each speech signal includes the following steps:

[0042] Apply a Mel filter bank H to the power spectrum spectrogram l [k]; These filters are evenly distributed on the Mel frequency scale, simulating the sensitivity of the human ear to different frequencies; l = 1, 2, ..., L represents the Mel filter index;

[0043] S5, weight the N Fbank features separately, and use the softmax activation function to limit the sum of weights to remain unchanged;

[0044] Specifically, the N Fbank features are weighted respectively, and the softmax activation function is used to limit the sum of the weights to remain unchanged, including the following steps:

[0045] Perform zero mean processing on each set of features; that is, subtract the mean of each set of features so that the mean of the features is 0 and the distribution center of the features is aligned to the zero point;

[0046] Initialize weight parameters; initialize original weight parameters for N Fbank features respectively;

[0047] A total of 5 original weight parameters w1, w2, w3, w4, w5 are initialized, and the initial values ​​are all set to 1.0;

[0048] Use the softmax activation function to limit the weights: After stacking the original weight parameters into a vector, use the softmax function to normalize the vector and multiply it by the original weight number to ensure that the total weight is constant; multiply it by 5 to ensure that the total weight is constant at 5; the softmax function is expressed as:

[0049]

[0050] S6. Concatenate the N weighted Fbank features in series in the time dimension from high to low order to form a multi-window fractional-order Fbank feature, and use the feature to input the deep learning model for training;

[0051] Specifically, the N weighted Fbank features are concatenated in series in the time dimension from high to low order to form a multi-window fractional-order Fbank feature, which is expressed as:

[0052] C[l,m]=5·Softmax[w1·F1[l,m]; w2·F2[l,m]; w3·F3[l,m]; w4·F4[l,m]; w5·F5[l,m]]

[0053] Specifically, using the multi-window fractional-order Fbank feature input deep learning model for training includes the following steps:

[0054] Selected model: The TDNN model and the ECAPA-TDNN model are selected and trained with the traditional Fbank features and the multi-window fractional-order Fbank features respectively;

[0055] Set the training hyperparameters: the length of the input speech signal segment is 200ms, the maximum number of training rounds (epoch) is max_epoch=100, and the batch size is batch_size=64; use the Adam optimizer, weight decay is weight_decay=1e-4, and the learning rate decay strategy is preheated cosine annealing;

[0056] Select training set data: Select Voxceleb1 speech dataset as the training set;

[0057] Input features for training: Set a Boolean flag freeze_weights with an initial value of False to control whether the weights can be updated, and initialize a fixed weight vector fixed_weights: Decide whether to update the weights based on the current training epoch. If the current round is less than or equal to the preset threshold 10, the weights are allowed to be updated, otherwise, the weights are frozen. Update: Update the calculated weight values ​​to fixed_weights. When the weights are frozen, use fixed_weights directly as the current weights: Allow weights to be adjusted freely in the early stages of training, and fix the weights in the later stages of training.

[0058] The present invention utilizes fractional-order window functions of N different orders to introduce fractional-order information, can extract multi-angle time-frequency domain features, and utilizes learnable parameter weights to adaptively determine the importance of different segment features, which can conveniently and more effectively perform speaker verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the flow chart of the feature extraction method of the present invention;

[0060] Figure 2 Schematic diagram of cosine annealing for learning rate warmup. DETAILED DESCRIPTION

[0061] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0062] like Figure 1 As shown, the score domain feature extraction method for deep learning speaker verification described in this embodiment includes the following steps:

[0063] S1, preprocessing the speech signal;

[0064] Specifically, in step S1, the preprocessing of the speech signal is pre-emphasis. Pre-emphasis is a technology in audio signal processing, which is mainly used to enhance the energy of the high-frequency part. It can improve the overall spectrum balance of the signal and improve the effect of subsequent feature extraction.

[0065] More specifically, in this embodiment, a pre-emphasis coefficient of 0.97 is preferably used to perform pre-emphasis as follows:

[0066] x(t)=x0(t)-0.97*x0(t-1)

[0067] Where x0(t) is the original speech signal and x(t) is the pre-emphasized speech signal.

[0068] S2, using N modulation factors of different orders extracted from fractional Fourier transform to obtain a fractional window function for the selected window function;

[0069] Specifically, the fractional Fourier transform is expressed as:

[0070]

[0071] Kernel function K p (t,u) can be expressed as

[0072]

[0073] in Rotation angle of the time-frequency surface

[0074] Transform the kernel function formula and use the window function w(t) to add a window, and get

[0075]

[0076] The fractional-order window function is composed of the original window function and the modulation factor Combined, it is expressed as:

[0077]

[0078] More specifically, in this embodiment, w(t) is preferably expressed as a Hamming window:

[0079]

[0080] The preferred window function length N=400.

[0081] The preferred orders are p1=1, p2=0.8, p3=0.6, p4=0.4, and p5=0.2.

[0082] S3, using N fractional-order window functions of different orders to obtain a power spectrum spectrogram for each speech signal;

[0083] Specifically, the step of extracting the power spectrum spectrum diagram includes the following steps:

[0084] Frame division. The input speech signal is divided into overlapping segments, and the segment length is consistent with the window function length. More specifically, in this example, the preferred window function length is 400 samples, the frame shift is 160, and the sample rate is 16kHz, then each frame is about 25ms (400 / 16000=0.025s), and the frame shift, i.e. the sliding step of the segmentation window, is 10ms (160 / 16000=0.010s).

[0085] Windowing: Each frame of speech signal is preferably windowed using fractional-order window functions of five different orders.

[0086] Perform a Fast Fourier Transform (FFT). Perform FFT on each frame to convert the time domain signal into the frequency domain. In this embodiment, the preferred FFT point number is 512, which means that each frame of data will undergo a 512-point Fourier transform.

[0087] Compute the power spectrum. Take the square of the magnitude of the FFT result and divide by the signal length to normalize.

[0088] Generate a spectrogram. Arrange the spectrum of each frame in time order to form a two-dimensional spectrogram.

[0089] The extracted power spectral density is expressed as:

[0090]

[0091] Each speech signal is preferably windowed using five fractional-order window functions of different orders, that is, in this embodiment, five power spectrum spectrograms are extracted from each speech signal.

[0092] S4, for the power spectrum spectrogram, use the M-Vermeer filter bank and take the logarithm to extract the filter bank (Fbank) features;

[0093] Specifically, an M-dimensional Fbank feature is extracted from each power spectrum spectrogram, that is, the number of speech signals × N M-dimensional Fbank features are established. In this embodiment, the Fbank feature is expressed as:

[0094]

[0095] Among them, H l [k] is an 80-dimensional Mel filter bank, l = 1, 2, ..., 80 represents the Mel filter index, m = 1, 2, ..., T represents the index of the time dimension, R is the frame shift, and ∈ is a very small positive number to avoid zero or negative values ​​when taking the logarithm

[0096] More specifically, the extracting of Fbank features according to N power spectrum spectrograms of each speech signal comprises the following steps:

[0097] Apply a Mel filter bank H to the power spectrum spectrogram l [k]. These filters are evenly distributed on the Mel frequency scale, simulating the sensitivity of the human ear to different frequencies. l = 1, 2, ..., L represents the Mel filter index. In this embodiment, L = 80 is preferably used to obtain 80-dimensional features, and the Mel filter bank frequency range is preferably 20 Hz to 7600 Hz.

[0098] A very small positive number ∈ is added to the result to apply logarithmic compression, which helps to compress the dynamic range and make the features closer to human auditory perception. In this embodiment, ∈=1e-6 is preferred.

[0099] S5, weight the N Fbank features separately, and use the softmax activation function to limit the sum of weights to remain unchanged;

[0100] Specifically, the N Fbank features are weighted respectively, and the softmax activation function is used to limit the sum of the weights to remain unchanged, including the following steps:

[0101] Each set of features is processed by zero mean, that is, the mean of each set of features is subtracted so that the mean of the features is 0 and the distribution center of the features is aligned to the zero point.

[0102] Initialize weight parameters. Initialize original weight parameters for N Fbank features respectively. More specifically, in this embodiment, 5 original weight parameters w1, w2, w3, w4, w5 are initialized, and the initial values ​​are all set to 1.0.

[0103] The softmax activation function is used to constrain the weights. After the original weight parameters are stacked into a vector, the vector is normalized using the softmax function and multiplied by the original weight number to ensure that the sum of the weights is constant. More specifically, in this embodiment, it is multiplied by 5 to ensure that the sum of the weights is constant at 5. The softmax function is expressed as:

[0104]

[0105] S6. N weighted Fbank features are concatenated in time dimension from high to low order to form a multi-window fractional-order Fbank feature, and this feature is used as input into the deep learning model for training.

[0106] Specifically, the N weighted Fbank features are concatenated in series in the time dimension from high to low order to form a multi-window fractional-order Fbank feature, which is expressed as:

[0107] C[l,m]=5·Softmax[w1·F1[l,m]; w2·F2[l,m]; w3·F3[l,m]; w4·F4[l,m]; w5·F5[l,m]]

[0108] Specifically, using the multi-window fractional-order Fbank feature input deep learning model for training includes the following steps:

[0109] Selecting a model. In this embodiment, a TDNN model and an ECAPA-TDNN model are selected for training with traditional Fbank features and the multi-window fractional-order Fbank features, respectively.

[0110] Set training hyperparameters. In this embodiment, the length of the input speech signal segment is 200ms, the maximum number of training rounds (epoch) is max_epoch=100, and the batch size is batch_size=64. The Adam optimizer is used, the weight decay is weight_decay=1e-4, and the learning rate decay strategy is preheated cosine annealing. More specifically, in the first 10 epochs, the learning rate is gradually preheated from 1e-4 to 1e-3, and in 10 to 100 epochs, the learning rate gradually decays from 1e-3 to 1e-6 according to the cosine function curve. The learning rate change process is as follows: Figure 2 shown.

[0111] Select training set data. In this embodiment, the Voxceleb1 speech data set is selected as the training set.

[0112] Input features for training. More specifically, in this embodiment, a Boolean flag freeze_weights is set with an initial value of False, which is used to control whether the weights can be updated, and initialize a fixed weight vector fixed_weights. Whether to update the weights is determined based on the current training epoch. If the current round is less than or equal to the preset threshold 10, the weight update is allowed, otherwise, the weight update is frozen. The calculated weight value is updated to fixed_weights. When the weight is frozen, fixed_weights is directly used as the current weight. The weights are allowed to be adjusted freely in the early stage of training, and fixed in the later stage of training, thereby maintaining the stability of the model while making full use of the advantages of features of different orders.

[0113] The model trained after epoch 100 is selected to perform speaker verification task test on the Voxceleb1-O test set. The results are shown in Table 1.

[0114] Table 1 Comparison of the proposed feature and traditional Fbank features on the Voxceleb1 dataset

[0115] Model feature Parameter quantity EER(%) <![CDATA[minDCF 0.05 ]]> TDNN Fbank 4.2M 3.45 0.2306 TDNN The present invention 4.2M 3.31 0.2226 ECAPA-TDNN Fbank 6.2M 2.54 0.1715 ECAPA-TDNN The present invention 6.2M 2.39 0.1593

[0116] The parameter unit is million (M), the equal error rate (EER) is the probability when the false acceptance rate is equal to the false rejection rate, and the minimum detection cost function (minDCF) is an indicator to measure the performance of the system in actual application scenarios, taking into account the cost of errors and the prior probability of errors. Different from EER, minDCF measures the cost of errors by setting a specific cost function. More specifically, in this embodiment, p target =0.05,C FA =C Miss =1, where p target is the prior probability of a positive example, C FA is the risk factor of false acceptance of samples, C Miss is the risk factor for false rejection of the sample.

[0117] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned method for extracting fractional domain features for deep learning speaker verification are performed.

Claims

1. A fractional domain feature extraction method for deep learning speaker verification, characterized in that: The following steps are involved: S1, pre-emphasize the high frequency part of the speech signal; Specifically, a pre-emphasis coefficient of 0.97 is selected for pre-emphasis as follows: x(t)=x0(t)-0.97*x0(t-1) Where x0(t) is the original speech signal, and x(t) is the pre-emphasized speech signal; S2, using N modulation factors of different orders extracted from fractional Fourier transform to obtain a fractional window function for the selected window function; Specifically, the fractional Fourier transform is expressed as: Kernel function K p (t,u) can be expressed as in Rotation angle of the time-frequency surface Transform the kernel function formula and use the window function w(t) to add a window, and get The fractional-order window function is composed of the original window function and the modulation factor Combined, it is expressed as: Specifically, w(t) is a Hamming window expressed as: Window function length N=400; The orders are p1=1, p2=0.8, p3=0.6, p4=0.4, p5=0.2; S3, using 5 fractional-order window functions of different orders to obtain a power spectrum spectrogram for each speech signal; Extract the M-dimensional filter bank (Fbank) features from the obtained power spectrum spectrogram; The orders of the five fractional-order window functions are p1=1, p2=0.8, p3=0.6, p4=0.4, and p5=0.2; Specifically, the step of extracting the power spectrum spectrum diagram includes the following steps: Framing: divide the input speech signal into overlapping segments, the length of which is consistent with the length of the window function; Windowing: Each frame of speech signal is windowed using 5 fractional-order window functions of different orders; Perform a Fast Fourier Transform (FFT) on each frame to convert the time domain signal to the frequency domain; calculate the power spectrum; take the square of the modulus of the FFT result and divide it by the signal length for normalization; Generate a spectrogram: Arrange the spectrum of each frame in time order to form a two-dimensional spectrogram; The extracted power spectral density is expressed as: Each speech signal is windowed using 5 fractional-order window functions of different orders; S4, for the power spectrum spectrogram, use the M-Vermeer filter bank and take the logarithm to extract the filter bank (Fbank) features; Specifically, an M-dimensional Fbank feature is extracted from each power spectrum spectrogram, that is, the number of speech signals × N M-dimensional Fbank features are established; The Fbank feature is expressed as: Among them, H l [k] is an 80-dimensional Mel filter bank, l = 1, 2, ..., 80 represents the Mel filter index, m = 1, 2, ..., T represents the index of the time dimension, R is the frame shift, A very small positive number used to avoid zero or negative values ​​when taking logarithms Extracting Fbank features based on N power spectrum spectrograms of each speech signal includes the following steps: Apply a Mel filter bank H to the power spectrum spectrogram l [k]; these filters are evenly distributed on the Mel frequency scale, simulating the sensitivity of the human ear to different frequencies; l = 1, 2, ..., L represents the Mel filter index; S5, weight the N Fbank features separately, and use the softmax activation function to limit the sum of weights to remain unchanged; Specifically, the N Fbank features are weighted respectively, and the softmax activation function is used to limit the sum of the weights to remain unchanged, including the following steps: Perform zero mean processing on each set of features; that is, subtract the mean of each set of features so that the mean of the features is 0 and the distribution center of the features is aligned to the zero point; Initialize weight parameters; initialize original weight parameters for N Fbank features respectively; A total of 5 original weight parameters w1, w2, w3, w4, w5 are initialized, and the initial values ​​are all set to 1.0; Use the softmax activation function to limit the weights: After stacking the original weight parameters into a vector, use the softmax function to normalize the vector and multiply it by the original weight number to ensure that the total weight is constant; multiply it by 5 to ensure that the total weight is constant at 5; the softmax function is expressed as: S6. Concatenate the N weighted Fbank features in series in the time dimension from high to low order to form a multi-window fractional-order Fbank feature, and use the feature to input the deep learning model for training; Specifically, the N weighted Fbank features are concatenated in series in the time dimension from high to low order to form a multi-window fractional-order Fbank feature, which is expressed as: C[l,m]=5·Softmax[w1·F1[l,m]; w2·F2[l,m]; w3·F3[l,m]; w4·F4[l,m]; w5·F5[l,m]] Specifically, using the multi-window fractional-order Fbank feature input deep learning model for training includes the following steps: Selected model: The TDNN model and the ECAPA-TDNN model are selected and trained with the traditional Fbank features and the multi-window fractional-order Fbank features respectively; Set the training hyperparameters: the length of the input speech signal segment is 200ms, the maximum number of training rounds (epoch) is max_epoch=100, and the batch size is batch_size=64; use the Adam optimizer, weight decay is weight_decay=1e-4, and the learning rate decay strategy is preheated cosine annealing; Select training set data: Select Voxceleb1 speech dataset as the training set; Input features for training: Set a Boolean flag freeze_weights with an initial value of False to control whether the weights can be updated, and initialize a fixed weight vector fixed_weights: Decide whether to update the weights based on the current training epoch. If the current round is less than or equal to the preset threshold 10, the weights are allowed to be updated, otherwise, the weights are frozen. Update: Update the calculated weight values ​​to fixed_weights. When the weights are frozen, use fixed_weights directly as the current weights: Allow weights to be adjusted freely in the early stages of training, and fix the weights in the later stages of training.

2. A processing device, characterized in that: include: at least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor is enabled to implement the method of claim 1.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method of claim 1 can be implemented.

Images