Voiceprint recognition method based on double-feature branch structure

Abstract

The invention relates to the technical field of deep learning voiceprint recognition, in particular to a voiceprint recognition method based on a double-feature branch structure. The voiceprint recognition method comprises the steps that Mel cepstrum features and wavelet transform features are extracted from original voice signals respectively, and two branches are formed; respectively inputting the features into a self-attention network and a convolutional TDNN network to carry out multi-scale feature modeling, and fusing two paths of outputs; further calculating multi-level discrimination loss for the fused voiceprint representation so as to enhance the speaker discrimination degree in a noisy or mismatched environment; and performing decoding or up-sampling operation on the fused output and taking the fused output as the input of next-stage processing, and finally generating multi-resolution and more robust voiceprint features through a cascade codec structure. The method aims to overcome the defects of single path feature extraction in a complex environment, and the capture and recognition capability of multi-resolution speech features is remarkably improved by combining the advantages of self-attention and convolution TDNN.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning voiceprint recognition, and particularly to a voiceprint recognition method based on a dual-feature branch structure. Background Art

[0002] Voiceprint recognition has experienced a rapid development from statistical methods based on Gaussian mixture models and i-vectors to end-to-end solutions such as x-vectors that combine deep neural networks. Current research mainly focuses on better feature extraction and modeling strategies, such as combining attention mechanisms and fusing multi-scale features, to achieve more robust speech identity recognition in complex environments.

[0003] Current methods mostly extract Mel cepstrum or other single features through a single path, ignoring the multi-scale information of speech signals; in deep models, there is also insufficient combination of global and local feature information. In summary, existing voiceprint recognition methods have problems of poor feature extraction effect for low-quality speech signals and easy loss of voiceprint feature information. Summary of the Invention

[0004] The purpose of the present invention is to provide a voiceprint recognition method based on a dual-feature branch structure, which solves the problems of poor feature extraction effect for low-quality speech signals and easy loss of voiceprint feature information in existing voiceprint recognition methods.

[0005] To achieve the above purpose, the present invention provides a voiceprint recognition method based on a dual-feature branch structure, and the voiceprint recognition method based on a dual-feature branch structure includes the following steps:

[0006] Preprocess the original speech signal, and extract Mel cepstrum features and wavelet transform features respectively to form two feature branches;

[0007] Input the Mel cepstrum features into the self-attention module for sequence modeling of multi-scale time-frequency features;

[0008] Input the wavelet transform features into the convolutional TDNN network for time-delay convolution processing of segmented or framed wavelet coefficients;

[0009] Fuse the outputs of the above two branches to obtain a fused voiceprint feature representation;

[0010] Input the fused voiceprint feature representation into the SE module and the average statistical pooling layer and connect the linear layer, and finally use the AAM-Softmax loss for speaker classification or similarity measurement.

[0011] Among them, in the step of "preprocessing the original speech signal, extracting Mel cepstrum features and wavelet transform features respectively, and forming two feature branches", the extraction of the Mel cepstrum features satisfies the following formula:

[0012]

[0013] where X[k] is the Fourier transform of the signal, and B m [k] is the mask of the Mel frequency cepstrum, which can be calculated by the following formula:

[0014]

[0015] where K is the number of frequency domain sampling points and M is the number of Mel filters.

[0016] Among them, in the step of "preprocessing the original speech signal, extracting Mel cepstrum features and wavelet transform features respectively, and forming two feature branches", the extraction of the wavelet transform features satisfies the following formula:

[0017]

[0018] where x(t) is the input signal, ψ is the mother wavelet function, a is the scale factor, and b is the translation factor.

[0019] Among them, the specific content of the step of "inputting the Mel cepstrum features into the self-attention module for sequence modeling of multi-scale time-frequency features" includes:

[0020] Linearly map to obtain Q, K, and V. Multiply the Mel cepstrum feature matrix X by the learnable parameter matrices W h (Q) , W h (K) and W h (V) :

[0021]

[0022] Dot-product Q h with K h and divide by for normalization, then use softmax to obtain the attention weight matrix, and finally multiply by V h :

[0023]

[0024] Concatenate the outputs of all attention heads (h = 1,..., H) and multiply by the learnable output mapping W O :

[0025] MultiHead(X) = [head1|head2|…|head H ·W O 。

[0026] Among them, in the step of "inputting the wavelet transform features into the convolutional TDNN network and performing time-delay convolution processing on the segmented or framed wavelet coefficients", the convolutional TDNN network performs time-delay convolution processing on the segmented or framed wavelet transform features, and captures local features through layer normalization and convolutional spatial gating units and then projects them back to the original dimension.

[0027] Among them, when performing the step of "feature fusion on the outputs of the above two branches", optionally introduce a covariance transformation or channel normalization mechanism to align the feature distributions, and restore them to the fused feature map through a linear mapping, expressed as:

[0028] Y = ω1·Y mfcc + ω2·Y wav 。

[0029] Among them, in the step of "inputting the fused speaker feature representation into the SE module and the average statistical pooling layer and connecting the linear layer", the SE module performs channel weighting on the fused feature map, the average statistical pooling layer aggregates the global temporal features, and the connected linear layer outputs the speaker embedding. The operation process of the average statistical pooling layer is expressed as:

[0030] Calculate the soft attention mechanism related to the channel, the self-attention weight α t,c represents the importance of each frame of a given channel, calculated from the activation output h of each frame:

[0031]

[0032] For the weighted mean of a given speech for each channel component in is:

[0033]

[0034] For the weighted mean of a given speech for each channel component in is:

[0035]

[0036] Among them, in the step of "finally using the AAM-Softmax loss for speaker classification or similarity measurement",

[0037] the AAM-Softmax loss function can be calculated by the following formula:

[0038]

[0039] Among them, for all categories j≠y i , z j = s·cos(θ j ), an additional angle m is added to the original cosine θ for the correct category.

[0040] Among them, the voiceprint recognition method based on the dual-feature branch structure further includes a voiceprint database update step, and the voiceprint database update step specifically refers to updating the corresponding matching voiceprint data in the database with the model output to ensure its timeliness, which is represented by the following formula:

[0041] speaker i = α·speaker i +(1 - α)·Y

[0042] Among them, i is the recognition result, and α is a hyperparameter.

[0043] A voiceprint recognition method based on a dual-feature branch structure of the present invention uses the dual-feature branch structure to independently extract and model Mel cepstrum hypotheses and wavelet multi-scale time-frequency information at the front end, and can better retain key acoustic features in low-quality or noisy environments; combines the respective advantages of self-attention and convolutional TDNN to refine the fusion of local and global features at the same time; through parallel or cascaded operations on multi-level feature streams, effectively reduces the risk of loss of voiceprint information in a single feature channel or fixed resolution, and helps to improve the discrimination and robustness of the deep model for different speakers' voices; further combines the SE module and the average statistical pooling layer to aggregate global temporal information, and uses the AAM-Softmax loss for more stringent similarity constraints to achieve accurate recognition of different voiceprint features. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 is a flowchart of the steps of the voiceprint recognition method based on the dual-feature branch structure provided by the present invention.

[0046] Figure 2 is a schematic diagram of the dual-feature branch model structure provided by the present invention.

[0047] Figure 3It is a schematic diagram of the voiceprint feature SE module provided by the present invention.

[0048] Figure 4 It is a schematic diagram of the voiceprint database update operation provided by the present invention. Detailed implementation manners

[0049] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as a limitation to the present invention.

[0050] Please refer to Figures 1 to 4 , the present invention provides a voiceprint recognition method based on a dual-feature branch structure. The voiceprint recognition method based on the dual-feature branch structure specifically includes the following steps:

[0051] Step S1, perform preprocessing on the original signal such as frame segmentation, windowing, and denoising.

[0052] Step S2, extract the preprocessed speech signal into two types of features respectively to form a multi-level cascade input; among them, one branch is the Mel cepstrum feature, and the extraction process of the Mel cepstrum feature is represented by the following formula:

[0053]

[0054] Among them, X[k] is the Fourier transform of the signal, B m [k] is the mask of the Mel frequency cepstrum, which can be calculated by the following formula, K is the number of frequency domain sampling points, and M is the number of Mel filters. The Mel cepstrum feature of each frame can be obtained through this process.

[0055]

[0056] Another branch is the wavelet decomposition feature, and the extraction process of the wavelet transform feature is represented by the following formula:

[0057]

[0058] Among them, x(t) is the input signal, ψ is the mother wavelet function, a is the scale factor, and b is the translation factor. Through the transformation under different a and b, a multi-scale representation of the signal in the time-frequency domain can be obtained.

[0059] Optionally, use a cascade coding network that can support the increase of the number of channels to perform convolution or attention modeling on different features, and save the pooling position index during downsampling so as to retain key acoustic structure information in subsequent operations.

[0060] Specifically, a voiceprint recognition method based on a dual-feature branch structure may further include extracting and downsampling multi-scale features. The input layer of each stage of the encoder is padded by boundary reflection, and a sliding window convolution with a stride of 1 is used for feature activation; the output layer combines group-separable convolution to achieve an increase in the number of feature channels.

[0061] Step S3, after processing each scale of the two-way features through the corresponding network module, fusing to generate a joint voiceprint feature representation; as Figure 2 shown, one of the branches is that a self-attention network performs sequence context modeling on the Mel cepstrum features. The specific calculation process of the multi-head attention module can be divided into the following steps (taking the h-th attention head as an example):

[0062] Linearly map to obtain Q, K, and V. Multiply the Mel cepstrum feature matrix X by the learnable parameter matrices W h (Q) , W h (K) and W h (V) :

[0063]

[0064] Multiply Q h by K h for dot product and divide by for normalization, then use softmax to obtain the attention weight matrix, and finally multiply by V h :

[0065]

[0066] Concatenate the outputs of all attention heads (h = 1, …, H) and multiply by the learnable output mapping W O :

[0067] MultiHead(X) = [head1|head2|…|head H · W O .

[0068] Through the parallel and concatenation of the multiple heads of the above-mentioned multi-head attention, the dependencies of the Mel cepstrum features in different angles or subspaces can be comprehensively modeled.

[0069] Another branch is a convolutional TDNN network that performs time-delay convolution processing on the wavelet features. After the layer normalization operation, the input feature channels are projected to a higher dimension and activated. Subsequently, a convolutional spatial gating unit is used to capture local information. This module first divides the features into two segments, then applies layer normalization and depth convolution to one segment, and finally performs element-wise multiplication of the two branches. Finally, the output is projected back to the original dimension.

[0070] Finally, the output Y of the multi-head self-attention module mfcc and the output Y of the convolutional TDNN network wav are fused, and the fusion process can be formally expressed as follows:

[0071] Y = ω1·Y mfcc + ω2·Y wav .

[0072] Optionally, a fusion module based on covariance or channel normalization is introduced. First, the feature distributions of different branches are differentially aligned, and then restored to a fused feature map.

[0073] Step S4, further input the fused feature map into the SE module and the average statistical pooling layer, and connect the linear layer; as Figure 3 shown, first send the fused feature map into the SE module to highlight the key channels and suppress redundant information, then aggregate the global features through the average statistical pooling layer, and finally input the linear layer for embedding mapping or classification output.

[0074] Express the operation process of the average statistical pooling layer as:

[0075] Calculate the channel-related soft attention mechanism, and the self-attention weight α t,c represents the importance of each frame of a given channel, and is calculated from the activation output h t of each frame:

[0076]

[0077] For each channel component in the weighted mean of a given speech is estimated as:

[0078]

[0079] For each channel component in the weighted mean of a given speech is estimated as:

[0080]

[0081] Specifically, a voiceprint recognition method based on a dual-feature branch structure may further include the multi-level use of covariance matrices: during the fusion process, the differential features of the wavelet branch and the Mel cepstrum branch are first adjusted by normalization, and then feature mixing is performed through covariance matrices or attention matrices to remove redundant information and highlight complementary features.

[0082] Step S5: Calculate multi-level recognition or classification losses for the fused feature maps to enable the fused feature maps to maintain the recognition of the speaker's voiceprint. The AAM-Softmax or other loss functions based on metric learning can be selected to perform discriminative training on the multi-scale embedded features, strengthen the model's ability to distinguish different speaker features, and reduce the influence of noise and distortion.

[0083] The AAM-Softmax loss function can be calculated by the following formula:

[0084]

[0085] Among them, for all classes j≠y i , z j = s·cos(θ j ), an additional angle m is added to the original cosine θ for the correct class.

[0086] Step S6: Match the classification result with the voiceprint data in the voiceprint database. After successful matching, as Figure 4 shown, the following formula is used to update the corresponding data to ensure the timeliness of the voiceprint recognition system:

[0087] speaker i = α·speaker i + (1 - α)·Y

[0088] Where i is the recognition result, α is a hyperparameter that can be set manually or calculated by methods such as cosine similarity. Generally, this value should preferably exceed 0.9 to ensure the accuracy and security of the voiceprint database.

[0089] In one embodiment, a voiceprint recognition method based on a dual-feature branch structure may further include the calculation process of a covariance transformation network: first, normalize the Mel cepstrum and wavelet features so that their distributions meet similar statistical requirements, then introduce a covariance matrix to achieve feature mixing and restoration, and through re-adding with the original mean information, obtain a more robust voiceprint feature representation after fusion.

[0090] In summary, the voiceprint recognition method based on the dual-feature branch structure provided by the present technical solution uses the dual-feature branch structure to independently extract and model Mel cepstrum hypotheses and wavelet multi-scale time-frequency information at the front end, and can better retain key acoustic features in low-quality or noisy environments; combines the respective advantages of self-attention and convolutional TDNN to refine the fusion of local and global features simultaneously; through parallel or cascaded operations on multi-level feature streams, effectively reduces the risk of loss of voiceprint information in a single feature channel or at a fixed resolution, which helps to improve the discrimination and robustness of the deep model for different speakers' voices; further combines the SE module and the average statistical pooling layer to aggregate global temporal information, and uses the AAM-Softmax loss for more stringent similarity constraints to achieve accurate recognition of different voiceprint features.

[0091] The above-disclosed is only a preferred embodiment of the present invention, and of course, it cannot be used to limit the scope of the rights of the present invention. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A voiceprint recognition method based on a dual-feature branch structure, characterized in that It includes the following steps: Preprocess the original speech signal, and extract Mel cepstrum features and wavelet transform features respectively to form two feature branches; Input the Mel cepstrum features into the self-attention module for sequence modeling of multi-scale time-frequency features; Input the wavelet transform features into the convolutional TDNN network to perform time-delay convolution processing on the segmented or framed wavelet coefficients; Fuse the outputs of the above two branches to obtain the fused speaker feature representation; Input the fused speaker feature representation into the SE module and the average statistical pooling layer and connect the linear layer, and finally use the AAM-Softmax loss for speaker classification or similarity measurement.

2. The speaker recognition method based on the dual-feature branch structure according to claim 1, wherein In the step of "preprocessing the original speech signal, and extracting Mel cepstrum features and wavelet transform features respectively to form two feature branches", the extraction of the Mel cepstrum features satisfies the following formula: where X[k] is the Fourier transform of the signal, and B m [k] is the mask of the mel-frequency cepstrum, which can be calculated by the following formula: where K is the number of frequency domain sampling points and M is the number of Mel filters.

3. The speaker recognition method based on the dual-feature branch structure according to claim 2, wherein In the step of "preprocessing the original speech signal, and extracting Mel cepstrum features and wavelet transform features respectively to form two feature branches", the extraction of the wavelet transform features satisfies the following formula: where x(t) is the input signal, ψ is the mother wavelet function, a is the scale factor, and b is the translation factor.

4. The speaker recognition method based on the dual-feature branch structure according to claim 3, wherein The specific content of the step of "inputting the Mel cepstrum features into the self-attention module for sequence modeling of multi-scale time-frequency features" includes: The linear mapping obtains Q, K, and V by multiplying the Mel cepstrum feature matrix X by the learnable parameter matrices W h (Q) , W h (K) and W h (V) : Dot product Q h with K h and divide by to normalize, then use softmax to obtain the attention weight matrix, and finally multiply by V h : Concatenate the outputs of all attention heads (h = 1, …, H) and multiply by the learnable output projection W O : MultiHead(X) = [head1|head2|…|head H ·W O .

5. The speaker recognition method based on the dual-feature branch structure according to claim 4, wherein In the step of "inputting the wavelet transform features into the convolutional TDNN network to perform time-delay convolution processing on the segmented or framed wavelet coefficients", the convolutional TDNN network performs time-delay convolution processing on the wavelet transform features after segmentation or framing, and captures local features through layer normalization and convolutional spatial gating units and then projects back to the original dimension.

6. The speaker recognition method based on the dual-feature branch structure according to claim 5, wherein When performing feature fusion on the outputs of the above two branches, optionally introduce a covariance transformation or channel normalization mechanism to align the feature distributions, and restore them to the fused feature map through linear mapping, expressed as: Y = ω1·Y mfcc + ω2·Y wav .

7. The speaker recognition method based on the dual-feature branch structure according to claim 6, wherein In the step of "inputting the fused speaker feature representation into the SE module and the average statistical pooling layer and connecting the linear layer", the SE module weights the channels of the fused feature map, the average statistical pooling layer aggregates the global temporal features, connects the linear layer to output the speaker embedding, and the operation process of the average statistical pooling layer is expressed as: Soft attention mechanism related to the calculation channel, self-attention weight α t,c Indicates the importance of each frame of a given channel, which is determined by the activation output h of each frame t Calculated as follows: Weighted mean for a given voice For each channel component in It is as follows: Weighted mean of a given voice For each channel component in it is:

8. The voiceprint recognition method based on a dual-feature branch structure according to claim 7, wherein In the step of "finally using the AAM-Softmax loss for speaker classification or similarity measurement", The AAM-Softmax loss function can be calculated by the following formula: where, for all classes j ≠ y i , z j = s·cos(θ j ), adding an additional angle m to the original cosine θ for the correct class, 9. The voiceprint recognition method based on a dual-feature branch structure according to claim 8, wherein the voiceprint recognition method based on a dual-feature branch structure further includes a voiceprint database update step, and the voiceprint database update step specifically refers to updating the corresponding matching voiceprint data in the database with the model output to ensure its timeliness, which is represented by the following formula: speaker i = α · speaker i + (1 - α) · Y where i is the recognition result and α is a hyperparameter.

Images