Speaker confirmation method based on SASFV aggregation model

Abstract

The invention discloses a speaker confirmation method based on an SASFV aggregation model, and relates to the field of speech recognition, and the method comprises the steps: extracting a logarithmic Mel spectrogram through short-time Fourier transform and Mel filtering, generating a frame-level feature through an ERes2Net network, and carrying out the recognition of the frame-level feature; and an SASFV aggregation model is introduced, and a Fisher Vector variable, a self-attention mechanism and a statistical method are combined to generate speaker-level features with a fixed length, and finally, the identity of a speaker is judged through a cosine distance. According to the invention, the problem that features cannot be effectively represented and aggregated in a short voice task in the prior art is solved, and the accuracy, robustness and performance of a speaker recognition system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a speaker verification method based on a SASFV aggregation model. Background Art

[0002] Speaker recognition technology is an important technique for determining the speaker's identity by analyzing audio signals. It has broad application prospects in fields such as security and intelligent voice interaction. With the development of deep neural networks, speaker verification systems based on deep learning have gradually become mainstream. However, in practical applications, the length of the input audio is usually variable, and deep learning models usually require fixed-length features for training and verification. Therefore, how to effectively aggregate variable-length frame-level features into fixed-length sentence-level features has become a key technical link. Existing aggregation models perform well when processing long speech, but in short speech scenarios, due to limited speaker identity information, traditional models have difficulty extracting salient features and effectively aggregating them, resulting in reduced recognition accuracy. This problem is particularly prominent in application scenarios such as mobile devices and voice commands, because the audio in these scenarios is usually short and noisy, making speaker verification for short speech a technical challenge that needs to be solved urgently.

[0003] In the existing technology, commonly used aggregation methods include SSDP (self-attentive standard deviation pooling), MTSP (multiple time-scale statistics pooling), MRMHA (multiple-resolution multiple-head attention), GhostVLAD and NetFV. Although these methods have improved the performance of speaker verification systems to a certain extent, they still have shortcomings. For example, NetFV, as an aggregation method based on Fisher Vector, can better represent features, but it has low efficiency in utilizing limited information in short speech scenarios, making it difficult to achieve efficient feature aggregation. In addition, traditional aggregation models often ignore the contextual relationship between frame-level features and the modeling of short-term and long-term feature changes, resulting in insufficient robustness and generalization capabilities. Therefore, how to design a method that can efficiently aggregate frame-level features in short speech scenarios and improve the performance of speaker verification systems has become a key research direction.

[0004] In response to the above problems, the present invention proposes a speaker verification method and system based on the SASFV (Self-Attention Statistical Fisher Vector) aggregation model. This method proposes a more efficient and robust aggregation method by combining Fisher Vector variables, self-attention mechanism and statistical methods. Specifically, Fisher Vector can better characterize features and aggregate based on gradients. The self-attention mechanism reflects the overall structure and contextual relationships while minimizing the loss of key information during the aggregation process, while the statistical method can simultaneously model short-term and long-term feature changes, thereby enhancing the robustness of the features. By introducing the SASFV aggregation model, the present invention solves the problem that traditional models cannot effectively characterize and aggregate based on limited information in short audio, significantly improves the accuracy and robustness of the speaker recognition system, and exhibits higher performance and better adaptability. Summary of the Invention

[0005] This paper addresses the existing inability to effectively represent and aggregate features based on limited information in short speech speaker verification tasks. By combining Fisher Vector variables, a self-attention mechanism, and statistical methods, this method proposes a more efficient and robust aggregation framework to generate highly discriminative speaker-level features. Ultimately, speaker identification is achieved through cosine distance calculation.

[0006] The present invention provides a speaker verification method based on the SASFV aggregation model, the overall solution of which includes the following steps: S1: Extract frame-level features from the original audio signal, specifically: First, the input original audio signal is subjected to a short-time Fourier transform to obtain a linear spectrogram; the short-time Fourier transform to obtain a linear spectrogram is expressed as follows: In the formula is the signal at time t, is the window function, w is a continuous function of angular frequency, and n is a discrete function of time.

[0007] Then, the linear spectrogram is processed by a Mel filter to obtain a logarithmic Mel spectrogram; the linear spectrogram is processed by a Mel filter to obtain a logarithmic Mel spectrogram specifically as follows: First, the short-time Fourier transform result is obtained , calculate the power spectrum, and its formula is expressed as: in Represents the amplitude value of the complex spectrum; Secondly, the power spectrum Mapping to the Mel frequency scale using a set of triangular filters Perform weighted summation, and the formula is expressed as: in represents the energy of the kth Mel-band, is the kth filter in the Mel filter bank; Finally, we convert the energy to a logarithmic scale and take the logarithm of the energy of the Mel band to obtain a log-Mel spectrogram: in is a minimum value.

[0008] Finally, the log-mel spectrogram is input into the ERes2Net network to extract frame-level features.

[0009] S2: Aggregating the frame-level features using the SASFV aggregation model to obtain aggregated frame-level features; aggregating the frame-level features using the SASFV aggregation model specifically includes the following steps: S21: Calculate based on the one-dimensional Gaussian distribution probability density function, its probability density Expressed as: in represents the parameter set of the kth Gaussian model in the Gaussian mixture model, where are the mixing weights, mean vectors and covariance matrices respectively, represents the standard deviation vector; S22: According to The posterior probability Expressed as: make ,and , then the posterior probability Rewritten as: Among them, the posterior probability It is The probability density of a Gaussian distribution is divided by the sum of the probability densities of all Gaussian distributions. And c represents a Gaussian distribution among the K distributions. It represents the weight of the cth distribution, represents the bias of the c-th distribution.

[0010] S23: In the gradient calculation process, the self-attention mechanism is combined with the statistical method to generate the mean vector of the k-th Gaussian component in SASFV and standard deviation vector The calculation is as follows: in are the query, key, and value of the mean vector mean respectively; and are the weights of the linear transformation respectively; represents the key obtained by statistical methods; The dimension representing the hidden features; are the standard deviations of the query, key, and value of the standard deviation vector std respectively; is the weight of the linear transformation, ⊙ is the element-by-element multiplication; and yes and The matrix transposed form of .

[0011] S24: According to the mean vector , standard deviation vector and the posterior probability Get the gradient vector of the SASFV mean vector and the gradient vector of the SASFV standard deviation vector , as follows: S25: All SASFV The gradients of the components are concatenated to form the gradient vector , which is expressed as follows: S26: Average the gradients of all features to get SASFV : in, is the gradient vector.

[0012] S3: The aggregated frame-level features are obtained and the cross entropy loss function is used as the training objective function. The network model is obtained by updating the parameters to minimize the loss. The cross entropy loss function is defined as follows: Where m represents the number of samples, represents the true value of the i-th sample, represents the predicted output value of the i-th sample, and L represents the loss value.

[0013] S4: Use the trained network model to extract audio features and calculate the cosine distance between the feature vectors in the database, and compare them with the threshold to finally identify the speaker. First, the cosine distance between the speaker-level features and the feature vectors already in the speaker database is calculated using the following formula: in, is the extracted speaker-level feature, For targets already in the library; Then, according to d The speaker is determined by the size of the threshold.

[0014] The beneficial effects of the present invention are: The SASFV aggregation model of this invention combines Fisher Vector variables, a self-attention mechanism, and statistical methods to address the inability of traditional aggregation models (such as NetFV and GhostVLAD) to effectively represent and aggregate features in short speech. Experimental results demonstrate that the proposed model demonstrates higher accuracy and robustness in short speech scenarios, significantly outperforming existing methods. DET curve analysis further validates the superior performance of the SASFV aggregation model in areas with low false positive rates.

[0015] In summary, by introducing the SASFV aggregation model, the present invention realizes the effective representation and aggregation of frame-level features in the short speech speaker verification task, significantly improving the accuracy, robustness and performance of the speaker recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall structure diagram of the method of the present invention; Figure 2 is the DET curve of the aggregation model. DETAILED DESCRIPTION

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0018] like Figure 1 As shown, the present invention provides a speaker verification method and system based on the SASFV aggregation model, comprising the following steps: S1. Extract frame-level features from the original audio signal. Figure 1 As shown in the feature extraction module in , the linear spectrogram obtained by short-time Fourier transform of the original audio is expressed as follows: In the formula is the signal at time t, is the window function, w is a continuous function of angular frequency, and n is a discrete function of time.

[0019] S2. The linear spectrogram is passed through a Mel filter to obtain a logarithmic Mel spectrogram. The specific steps include: (1) The short-time Fourier transform result , calculate the power spectrum, and its formula is expressed as: in Represents the amplitude value of the complex spectrum; (2) The power spectrum Mapping to the Mel frequency scale using a set of triangular filters Perform weighted summation, and the formula is expressed as: in represents the energy of the kth Mel-band, is the kth filter in the Mel filter bank; (3) Convert the energy to a logarithmic scale, take the logarithm of the energy of the Mel frequency band, and obtain the logarithmic Mel spectrum: in is a minimum value.

[0020] S3. Put the logarithmic Mel spectrogram into the ERes2Net network to extract frame-level features, where the ERes2Net network structure is shown in Table 1 and trained to obtain frame-level features; Figure 1 The training module consists of step S3.

[0021] Table 1 S4. According to the frame-level features, the SASFV aggregation model is introduced to obtain the aggregated frame-level features as follows Figure 1 As shown in the aggregation module in Figure 1 The speaker recognition module in the example is composed of step S4; the specific steps include: (1) calculating according to the one-dimensional Gaussian distribution probability density function, whose probability density Expressed as: in represents the parameter set of the kth Gaussian model in the Gaussian mixture model, where are the mixing weights, mean vectors and covariance matrices respectively, represents the standard deviation vector.

[0022] (2) According to The posterior probability Expressed as: ,and , then the posterior probability Rewritten as: (3) In the gradient calculation process, the self-attention mechanism is combined with the statistical method to generate the mean vector of the k-th Gaussian component in SASFV and standard deviation vector The calculation is as follows: in are the query, key, and value of the mean vector mean, respectively. and are the weights of the linear transformation respectively. Represents a statistically derived key. Represents the dimensionality of the latent features. are the standard deviations of the query, key, and value of the standard deviation vector std, respectively. is the weight of the linear transformation, ⊙ is the element-by-element multiplication; (4) According to the mean vector , standard deviation vector and the posterior probability Get the gradient vector of the SASFV mean vector and the gradient vector of the SASFV standard deviation vector , as follows: (5) All SASFV The gradients of the components are concatenated to form the gradient vector , which is expressed as follows: (6) Average the gradients of all features to obtain SASFV : S5. Based on the aggregated frame-level features, the AM-softmax cross entropy function is used as the training objective function, and the loss is minimized by updating the parameters to obtain the final network model as follows: Figure 1 As shown in the training module, it specifically includes: The cross entropy loss function is defined as follows: Where m represents the number of samples, represents the true value of the i-th sample, represents the predicted output value of the i-th sample, and L represents the loss value.

[0023] S6, use the model trained in step S5 to extract audio features, calculate the cosine distance with the feature vector in the database, and compare it with the threshold to finally identify the speaker, such as Figure 1 As shown in the speaker recognition module, it specifically includes: (1) The cosine distance between the speaker-level feature and the feature vectors in the speaker database is calculated using the following formula: in, is the extracted speaker-level feature, For targets already in the library; (2) According to d The speaker is determined by the size of the threshold.

[0024] This method is trained using the VoxCeleb2 dataset, which contains audio and video from YouTube. Therefore, it contains real-world noise such as laughter, conversation, channel artifacts, music, and other sounds. The test set includes four datasets: Vox1O, Vox1E, Vox1H, and SITW. The Vox1O / E / H datasets are derived from the publicly available VoxCeleb1 dataset, which, like VoxCeleb2, is also sourced from YouTube.

[0025] The network model selected ERes2Net for performance testing. Full-length audio was tested using Vox1O, Vox1E, Vox1H, and SITW. The DET curves of different aggregation models in ERes2Net on Vox1O were plotted, as shown below: Figure 2 As shown. After the model was trained, the SSDP aggregation model, MTSP aggregation model, MRMHA aggregation model, GhostVLAD aggregation model, NetFV aggregation model, and SASFV aggregation model were used to conduct experimental tests on full-length audio on four validation sets, and the results are shown in Table 2. The test results after using this method are more accurate than those using the NetFV aggregation model. Then, we tested short speeches of different lengths on Vox1O again, and the results are shown in Table 3. The test results after using this method are better than all tested models in the short speech test. The present invention solves the problem that the existing technology cannot effectively represent and aggregate features in short speech tasks, and significantly improves the accuracy, robustness and performance of the speaker recognition system.

[0026] Table 2 Table 3 The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0027] The technical solutions of the present application are not limited to the above specific embodiments, and any technical variations made according to the technical solutions of the present application fall within the scope of protection of the present application.

Claims

1. A speaker verification method based on the SASFV aggregation model, characterized in that: The following steps are involved: S1: Extract frame-level features from the original audio signal; S2: Aggregate the frame-level features using the SASFV aggregation model to obtain aggregated frame-level features; S3: Get the aggregated frame-level features and use the cross entropy loss function as the training objective function. Minimize the loss by updating the parameters to obtain the network model. S4: Use the trained network model to extract audio features and calculate the cosine distance with the feature vector in the database, and compare it with the threshold to finally identify the speaker.

2. The speaker verification method based on the SASFV aggregation model according to claim 1, characterized in that: The frame-level features extracted from the original audio signal in step S1 are specifically: First, the input original audio signal is short-time Fourier transformed to obtain a linear spectrogram; Then, the linear spectrogram is processed by a Mel filter to obtain a logarithmic Mel spectrogram; Finally, the log-mel spectrogram is input into the ERes2Net network to extract frame-level features.

3. The speaker verification method based on the SASFV aggregation model according to claim 2, characterized in that: The short-time Fourier transform is used to obtain a linear spectrum, which is expressed as follows: In the formula is the signal at time t, is the window function, w is a continuous function of angular frequency, and n is a discrete function of time.

4. The speaker verification method based on the SASFV aggregation model according to claim 3, characterized in that: The linear spectrogram is processed by a Mel filter to obtain a logarithmic Mel spectrogram: First, the short-time Fourier transform result is obtained , calculate the power spectrum, and its formula is expressed as: in Represents the amplitude value of the complex spectrum; Secondly, the power spectrum Mapping to the Mel frequency scale using a set of triangular filters Perform weighted summation, and the formula is expressed as: in represents the energy of the kth Mel-band, is the kth filter in the Mel filter bank; Finally, we convert the energy to a logarithmic scale and take the logarithm of the energy of the Mel band to obtain a log-Mel spectrogram: in is a minimum value.

5. The speaker verification method based on the SASFV aggregation model according to claim 1, characterized in that: The step S2 of aggregating the frame-level features using the SASFV aggregation model specifically includes the following steps: S21: Calculate based on the one-dimensional Gaussian distribution probability density function, its probability density Expressed as: in , represents the parameter set of the kth Gaussian model in the Gaussian mixture model, where 、 、 are the mixing weights, mean vectors and covariance matrices respectively, represents the standard deviation vector; S22: According to The posterior probability Expressed as: make , ,and , then the posterior probability Rewritten as: S23: In the gradient calculation process, the self-attention mechanism is combined with the statistical method to generate the mean vector of the k-th Gaussian component in SASFV and the standard deviation vector The calculation is as follows: in 、 、 are the query, key, and value of the mean vector mean respectively; , and are the weights of the linear transformation respectively; represents the key obtained by statistical methods; The dimension representing the hidden features; 、 、 are the standard deviations of the query, key, and value of the standard deviation vector std respectively; 、 、 is the weight of the linear transformation, ⊙ is the element-by-element multiplication; S24: According to the mean vector , standard deviation vector and the posterior probability Get the gradient vector of the SASFV mean vector and the gradient vector of the SASFV standard deviation vector , as follows: S25: All SASFV 、 The gradients of the components are concatenated to form the gradient vector , which is expressed as follows: S26: Average the gradients of all features to get SASFV : in, Gradient vector.

6. The speaker verification method based on the SASFV aggregation model according to claim 1, characterized in that: The cross entropy loss function in step S3 is defined as follows: Where m represents the number of samples, represents the true value of the i-th sample, represents the predicted output value of the i-th sample, and L represents the loss value.

7. The speaker verification method based on the SASFV aggregation model according to claim 1, characterized in that: The step S4 is specifically as follows: First, the cosine distance between the speaker-level features and the feature vectors already in the speaker database is calculated using the following formula: in, is the extracted speaker-level feature, For targets already in the library; Then, according to d The speaker is determined by the size of the threshold.

Images