A speaker verification method based on SASFV aggregation model

Abstract

The application discloses a speaker verification method based on a SASFV aggregation model, and relates to the field of speech recognition.The method extracts a log Mel spectrogram through short-time Fourier transform and Mel filtering, generates frame-level features by using an ERes2Net network, introduces a SASFV aggregation model to generate fixed-length speaker-level features in combination with a Fisher Vector variable, a self-attention mechanism and a statistical method, and finally determines the identity of a speaker by using a cosine distance.The application solves the problem that the prior art cannot effectively represent and aggregate features in a short speech task, and significantly improves the accuracy, robustness and performance of a speaker recognition system.

Description

Technical Field

[0001] This invention relates to the field of speech recognition, and more particularly to a speaker identification method based on the SASFV aggregation model. Background Technology

[0002] Speaker recognition technology is an important technique for determining speaker identity by analyzing audio signals, and it has broad application prospects in fields such as security and intelligent voice interaction. With the development of deep neural networks, deep learning-based speaker verification systems have gradually become mainstream. However, in practical applications, the length of the input audio is usually variable, while deep learning models typically require fixed-length features for training and validation. Therefore, effectively aggregating variable-length frame-level features into fixed-length sentence-level features has become a key technical challenge. Existing aggregation models perform well when processing long speech, but in short speech scenarios, due to limited speaker identity information, traditional models struggle to extract significant features and perform effective aggregation, leading to decreased recognition accuracy. This problem is particularly prominent in applications such as mobile devices and voice commands, as the audio in these scenarios is usually short and subject to significant noise interference, making short speech speaker verification a pressing technical challenge.

[0003] In existing technologies, commonly used aggregation methods include SSDP (self-attentive standard deviation pooling), MTSP (multiple time-scale statistics pooling), MRMHA (multiple-resolution multiple-head attention), GhostVLAD, and NetFV. While these methods have improved the performance of speaker verification systems to some extent, they still have shortcomings. For example, NetFV, as an aggregation method based on Fisher Vectors, can represent features well, but its efficiency in utilizing limited information in short speech scenarios is low, making it difficult to achieve efficient feature aggregation. Furthermore, traditional aggregation models often neglect the contextual relationships between frame-level features and the modeling of short-term and long-term feature changes, resulting in insufficient robustness and generalization ability. Therefore, designing a method that can efficiently aggregate frame-level features in short speech scenarios and improve the performance of speaker verification systems has become a key research focus.

[0004] To address the aforementioned issues, this invention proposes a speaker identification method and system based on the SASFV (Self-Attention Statistical Fisher Vector) aggregation model. This method combines Fisher Vector variables, a self-attention mechanism, and statistical methods to offer a more efficient and robust aggregation approach. Specifically, Fisher Vector can better represent features and aggregate based on gradients; the self-attention mechanism minimizes the loss of key information during aggregation while reflecting the overall structure and contextual relationships; and the statistical method can simultaneously model short-term and long-term feature changes, thereby enhancing feature robustness. By introducing the SASFV aggregation model, this invention solves the problem that traditional models cannot effectively represent and aggregate based on limited information in short audio clips, significantly improving the accuracy and robustness of the speaker recognition system, demonstrating higher performance and better adaptability. Summary of the Invention

[0005] This invention addresses the problem that existing technologies cannot effectively represent and aggregate features based on limited information in short speech speaker identification tasks. It proposes a speaker identification method based on the SASFV aggregation model. This method combines Fisher Vector variables, a self-attention mechanism, and statistical methods to propose a more efficient and robust aggregation framework, thereby generating highly discriminative speaker-level features and ultimately achieving speaker identification through cosine distance calculation.

[0006] This invention provides a speaker identification method based on the SASFV aggregation model, the overall scheme of which includes the following steps:

[0007] S1: Extract frame-level features from the original audio signal, specifically:

[0008] First, a short-time Fourier transform is performed on the input raw audio signal to obtain a linear spectrum; the formula for obtaining the linear spectrum by the short-time Fourier transform is as follows:

[0009]

[0010] In the formula The signal at time t, Let w be a window function, w be a continuous function of angular frequency, and n be a discrete function of time.

[0011] Then, the linear spectrum is processed by a Mel filter to obtain a log-Mel spectrum; specifically, the linear spectrum is processed by a Mel filter to obtain a log-Mel spectrum as follows:

[0012] First, the obtained short-time Fourier transform results The power spectrum is calculated using the following formula:

[0013]

[0014] in Represents the amplitude value of a complex number spectrum;

[0015] Secondly, the power spectrum Mapped to the Mel frequency scale, using a set of triangular filters The weighted summation is expressed by the following formula:

[0016]

[0017] in This represents the energy of the k-th Mel band. It is the kth filter in the Mel filter bank;

[0018] Finally, the energy is converted to a logarithmic scale, and the logarithm of the energy in the Mel band is taken to obtain the log-Mel spectrum:

[0019]

[0020] in It is a minimum value.

[0021] Finally, the log-Mel spectrum is input into the ERes2Net network to extract frame-level features.

[0022] S2: Aggregate the frame-level features using the SASFV aggregation model to obtain aggregated frame-level features; the aggregation of the frame-level features using the SASFV aggregation model specifically includes the following steps:

[0023] S21: The probability density is calculated based on the one-dimensional Gaussian distribution probability density function. Represented as:

[0024]

[0025] in Let represent the parameter set of the k-th Gaussian model in the Gaussian mixture model, where These are the mixed weights, mean vector, and covariance matrix, respectively. Represents the standard deviation vector;

[0026] S22: According to posterior probability Represented as:

[0027]

[0028] make ,and Then the posterior probability Rewritten as:

[0029]

[0030] Among them, the posterior probability It is the first The probability density of a Gaussian distribution is divided by the sum of the probability densities of all Gaussian distributions. Here, c represents a single Gaussian distribution among the K distributions. Then it represents the weight of the c-th distribution. This represents the bias of the c-th distribution.

[0031] S23: In the gradient calculation process, the self-attention mechanism is combined with statistical methods to generate the mean vector of the k-th Gaussian component in SASFV. and standard deviation vector The calculation is as follows:

[0032]

[0033] in These represent the query, key, and value of the mean vector, respectively. and These are the weights of the linear transformation; This represents the key obtained through statistical methods; The dimension of the hidden feature; These represent the standard deviations of the query, key, and value in the standard deviation vector std, respectively. represents the weights of the linear transformation, and ⊙ represents element-wise multiplication; and yes and The matrix transpose.

[0034] S24: Based on the mean vector Standard deviation vector and posterior probability Obtain the gradient vector of the SASFV mean vector and the gradient vector of the SASFV standard deviation vector The details are as follows:

[0035]

[0036] S25: All SASFV The gradients of the components are connected to form a gradient vector. , means as follows:

[0037]

[0038] S26: Average the gradients of all features to obtain the SASFV. :

[0039]

[0040] in, This is the gradient vector.

[0041] S3: The aggregated frame-level features are used as the training objective function through the cross-entropy loss function. The network model is obtained by minimizing the loss through parameter updates. The cross-entropy loss function is defined as follows:

[0042]

[0043] Where m represents the sample size. This represents the true value of the i-th sample. Let L represent the predicted output value of the i-th sample, and L represent the loss value.

[0044] S4: Use the trained network model to extract audio features and calculate the cosine distance between these features and the feature vectors in the database. Then compare this cosine distance with a threshold to ultimately identify the speaker. Specifically:

[0045] First, the cosine distance between the speaker-level features and the existing feature vectors in the speaker database is calculated using the following formula:

[0046]

[0047] in, For the extracted speaker-level features, For targets already existing in the library;

[0048] Then, according to d The speaker is determined by the threshold value.

[0049] The beneficial effects of this invention are:

[0050] This invention's SASFV aggregation model combines Fisher Vector variables, a self-attention mechanism, and statistical methods, addressing the problem that traditional aggregation models (such as NetFV and GhostVLAD) cannot effectively represent and aggregate features in short speech. Experimental results show that this invention exhibits higher accuracy and robustness in short speech scenarios, significantly outperforming existing methods. DET curve analysis further validates the superior performance of the SASFV aggregation model in regions with low false alarm rates.

[0051] In summary, by introducing the SASFV aggregation model, this invention achieves effective representation and aggregation of frame-level features in short speech speaker identification tasks, significantly improving the accuracy, robustness, and performance of the speaker recognition system. Attached Figure Description

[0052] Figure 1 This is a diagram illustrating the overall structure of the method of this invention;

[0053] Figure 2 This is the DET curve for the aggregation model. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0055] like Figure 1 As shown, this invention provides a speaker identification method and system based on the SASFV aggregation model, comprising the following steps:

[0056] S1. Extract frame-level features from the original audio signal, such as... Figure 1 As shown in the feature extraction module, the linear spectrum obtained by performing a short-time Fourier transform on the original audio is expressed by the following formula:

[0057]

[0058] In the formula The signal at time t, Let w be a window function, w be a continuous function of angular frequency, and n be a discrete function of time.

[0059] S2. Obtain the log-Mel spectrum from the linear spectrum using a Mel filter. Specific steps include:

[0060] (1) The result of the short-time Fourier transform The power spectrum is calculated using the following formula:

[0061]

[0062] in Represents the amplitude value of a complex number spectrum;

[0063] (2) Power spectrum Mapped to the Mel frequency scale, using a set of triangular filters The weighted summation is expressed by the following formula:

[0064]

[0065] in This represents the energy of the k-th Mel band. It is the kth filter in the Mel filter bank;

[0066] (3) Convert the energy to a logarithmic scale, take the logarithm of the energy in the Mel band, and obtain the log-Mel spectrum:

[0067]

[0068] in It is a minimum value.

[0069] S3. Input the log-Mel spectrum into the ERes2Net network to extract frame-level features. The ERes2Net network structure is shown in Table 1. Train the network to obtain frame-level features. Figure 1 The training module consists of step S3.

[0070] Table 1

[0071]

[0072] S4. Based on the frame-level features, the SASFV aggregation model is introduced to obtain the aggregated frame-level features as follows: Figure 1 As shown in the aggregation module, Figure 1 The speaker recognition module consists of step S4; the specific steps include: (1) calculating the probability density based on the one-dimensional Gaussian distribution probability density function. Represented as:

[0073]

[0074] in Let represent the parameter set of the k-th Gaussian model in the Gaussian mixture model, where These are the mixed weights, mean vector, and covariance matrix, respectively. This represents the standard deviation vector.

[0075] (2) According to posterior probability Represented as:

[0076]

[0077] ,and Then the posterior probability Rewritten as:

[0078]

[0079] (3) In the gradient calculation process, the self-attention mechanism is combined with statistical methods to generate the mean vector of the k-th Gaussian component in SASFV. and standard deviation vector The calculation is as follows:

[0080]

[0081] in These represent the query, key, and value of the mean vector, respectively. and These are the weights of the linear transformation. This represents the key obtained through statistical methods. The dimension of the hidden feature. These represent the standard deviations of the query, key, and value in the standard deviation vector std, respectively. represents the weights of the linear transformation, and ⊙ represents element-wise multiplication;

[0082] (4) Based on the mean vector Standard deviation vector and posterior probability Obtain the gradient vector of the SASFV mean vector and the gradient vector of the SASFV standard deviation vector The details are as follows:

[0083]

[0084] (5) All SASFV The gradients of the components are connected to form a gradient vector. , means as follows:

[0085]

[0086] (6) Averaging the gradients of all features yields the SASFV. :

[0087]

[0088] S5. Based on the aggregated frame-level features, using the AM-softmax cross-entropy function as the training objective function, and minimizing the loss by updating the parameters, the final network model is obtained as follows: Figure 1 The training module shown in the document specifically includes:

[0089] The cross-entropy loss function is defined as follows:

[0090]

[0091] Where m represents the sample size. This represents the true value of the i-th sample. Let L represent the predicted output value of the i-th sample, and L represent the loss value.

[0092] S6. Using the model trained in step S5, extract audio features, calculate the cosine distance between the features and the model vectors in the database, and compare the result with a threshold to ultimately identify the speaker. Figure 1 The speaker recognition module, as shown in the diagram, specifically includes:

[0093] (1) The cosine distance between the speaker-level features and the existing feature vectors in the speaker database is calculated using the following formula:

[0094]

[0095] in, For the extracted speaker-level features, For targets already existing in the library;

[0096] (2) According to d The speaker is determined by the threshold value.

[0097] This method is trained using the training set from the VoxCeleb2 dataset, whose audio and video are sourced from YouTube. Therefore, it includes real-world noise such as laughter, conversation, channel effects, music, and other sounds. The test set includes four datasets: Vox1O, Vox1E, Vox1H, and SITW. The Vox1O / E / H datasets are derived from the publicly available VoxCeleb1 dataset, which, like VoxCeleb2, is also sourced from YouTube.

[0098] The ERes2Net network model was selected for performance testing. Full-length audio tests were performed using Vox1O, Vox1E, Vox1H, and SITW, and the DET curves of different aggregation models in ERes2Net on Vox1O were plotted, as shown below. Figure 2 As shown in Table 2, after the model was trained, full-length audio samples were tested on four validation sets using the SSDP, MTSP, MRMHA, GhostVLAD, NetFV, and SASFV aggregation models. The results show that the accuracy of the test results using this method is improved compared to the NetFV aggregation model. Then, we tested short speech samples of different lengths on Vox10 again, and the results are shown in Table 3. The test results using this method are superior to all other models in the short speech test. This invention solves the problem that existing technologies cannot effectively represent and aggregate features in short speech tasks, significantly improving the accuracy, robustness, and performance of speaker recognition systems.

[0099] Table 2

[0100]

[0101] Table 3

[0102]

[0103] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0104] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A speaker identification method based on the SASFV aggregation model, characterized in that, Includes the following steps: S1: Extract frame-level features from the raw audio signal; S2: Aggregate the frame-level features using the SASFV aggregation model to obtain aggregated frame-level features; S21: The probability density is calculated based on the one-dimensional Gaussian distribution probability density function. Represented as: in The parameter set of the k-th Gaussian model in a Gaussian mixture model ,in , , These are the mixed weights, mean vector, and covariance matrix, respectively. Represents the standard deviation vector; S22: According to posterior probability Represented as: make , ,and Then the posterior probability Rewritten as: S23: In the gradient calculation process, the self-attention mechanism is combined with statistical methods to generate the mean vector of the k-th Gaussian component in SASFV. and standard deviation vector The calculation is as follows: in , , These represent the query, key, and value of the mean vector, respectively. , and These are the weights of the linear transformation; This represents the key obtained through statistical methods; The dimension of the hidden feature; , , These represent the standard deviations of the query, key, and value in the standard deviation vector std, respectively. , , represents the weights of the linear transformation, and ⊙ represents element-wise multiplication; S24: Based on the mean vector Standard deviation vector and posterior probability Obtain the gradient vector of the SASFV mean vector and the gradient vector of the SASFV standard deviation vector The details are as follows: S25: All SASFV , The gradients of the components are connected to form a gradient vector. , means as follows: S26: Average the gradients of all features to obtain the SASFV. : in, Gradient vector; S3: Obtain the aggregated frame-level features, use the cross-entropy loss function as the training objective function, and minimize the loss by updating the parameters to obtain the network model; S4: Use the trained network model to extract audio features and calculate the cosine distance with the feature vectors in the database, and compare it with the threshold to finally identify the speaker.

2. The speaker identification method based on the SASFV aggregation model according to claim 1, characterized in that: The specific steps in step S1 of extracting frame-level features from the original audio signal are as follows: First, a short-time Fourier transform is performed on the input raw audio signal to obtain a linear spectrum; Then, the linear spectrum is processed by a Mel filter to obtain a log-Mel spectrum; Finally, the log-Mel spectrum is input into the ERes2Net network to extract frame-level features.

3. The speaker identification method based on the SASFV aggregation model according to claim 2, characterized in that: The short-time Fourier transform is used to obtain a linear spectrum, and its formula is expressed as: In the formula The signal at time t, Let w be a window function, w be a continuous function of angular frequency, and n be a discrete function of time.

4. The speaker identification method based on the SASFV aggregation model according to claim 3, characterized in that: The linear spectrum is processed by a Mel filter to obtain a log-Mel spectrum, specifically as follows: First, the obtained short-time Fourier transform results The power spectrum is calculated using the following formula: in Represents the amplitude value of a complex number spectrum; Secondly, the power spectrum Mapped to the Mel frequency scale, using a set of triangular filters The weighted summation is expressed by the following formula: in This represents the energy of the k-th Mel band. It is the kth filter in the Mel filter bank; Finally, the energy is converted to a logarithmic scale, and the logarithm of the energy in the Mel band is taken to obtain the log-Mel spectrum: in It is a minimum value.

5. The speaker identification method based on the SASFV aggregation model according to claim 1, characterized in that: The cross-entropy loss function in step S3 is defined as follows: Where m represents the sample size. This represents the true value of the i-th sample. Let L represent the predicted output value of the i-th sample, and L represent the loss value.

6. The speaker identification method based on the SASFV aggregation model according to claim 1, characterized in that: Step S4 specifically involves: First, the cosine distance between the speaker-level features and the existing feature vectors in the speaker database is calculated using the following formula: in, For the extracted speaker-level features, For targets already existing in the library; Then, according to d The speaker is determined by the threshold value.

Images