A speaker recognition method based on reparameterization cross parallel convolutional neural network

Abstract

The application relates to a speaker recognition method based on a reparameterization cross-parallel convolutional neural network, and belongs to the field of speech recognition, and comprises the following steps: S1, constructing a structure reparameterization convolutional network RepCNN; S2, building a cross-parallel convolutional neural network based on the RepCNN; S3, extracting two features with different frequency domain resolutions from speech frames of a speaker by adopting two mel filter banks with different numbers of triangular filters; S4, taking the two features as inputs of the cross-parallel convolutional neural network, outputting enhanced features and S5, performing feature fusion on the two features, and outputting a speaker deep feature vector. The application reduces the dependence of the model on computing resources, improves the operation rate and accuracy of the model, and realizes the lightweight of the network model.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition and relates to a speaker recognition method based on a reparameterized cross-parallel convolutional neural network. Background Technology

[0002] Voice is an inherent personal characteristic. Voice data collection is convenient and inexpensive, and it has relatively low user intrusion risk, making it widely used in biometric identification. Speaker recognition is a biometric identification technology that identifies a speaker based on their personal information within the voice signal. It possesses properties such as universality, uniqueness, high applicability, and low deception. With the rapid development of mobile communication and the internet, the vast amount of voice communication data provides excellent conditions for the research and application of speaker recognition technology.

[0003] Commonly used acoustic features in speaker recognition include Mel-frequency cepstral coefficients (MFCC), gamma-pass frequency cepstral coefficients (GFCC), and Mel filter bank features (MFBF). Typically, combining different types of acoustic features from a speaker's speech as fusion features can improve the performance of speaker recognition systems. However, the implementation of speaker recognition systems still faces challenges. Input features do not fully utilize the frequency domain resolution differences of speech signals, resulting in insufficient information extraction. The direct concatenation of fusion features does not adequately address important speaker information. Classical convolutional neural network models offer high parallelism and flexibility in their single-path structure, but their large number of parameters and high computational cost make them less effective for speaker recognition tasks. Therefore, speaker recognition based on lightweight fusion features remains a challenge. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a speaker recognition method based on a reparameterized cross-parallel convolutional neural network, which reduces the model's dependence on computing resources, improves the model's operation speed and accuracy, and achieves lightweight network model.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A speaker recognition method based on a reparameterized cross-parallel convolutional neural network includes the following steps:

[0007] S1: Construct the RepCNN network with reparameterized structure;

[0008] S2: Building a cross-parallel convolutional neural network based on RepCNN;

[0009] S3: Using a Mel filter bank with two different numbers of triangular filters, features with two different frequency domain resolutions are extracted from the speaker's speech frames.

[0010] S4: with As input to a cross-parallel convolutional neural network, the output enhances features. and

[0011] S5: Features and Perform feature fusion to output the speaker's deep feature vector.

[0012] Furthermore, the RepCNN reparameterized convolutional network adds parallel identity mapping branches and residual branches to the 3x1 convolutional layers during model training, and employs multi-branch fusion technology during the model inference stage to transform the network into a single-path structure; specifically including:

[0013] When the number of input and output channels, width, and height are the same, the output is represented as:

[0014]

[0015] When the restructured convolutional network is RepCNN1, i = 3; when the restructured convolutional network is RepCNN2, i = 2. This indicates that the input channel is C. in The output channel is C. out The convolutional layer has a 3×1 kernel. This indicates a convolutional layer with a 1×3 kernel. To represent a branch with a 1×1 kernel, use μ. (3) , σ (3) γ (3) ,β (3) μ represents the parameters of the post-batch normalized layer of a 3×1 convolutional layer. (2) , σ (2) γ (2) ,β (2) μ represents the parameters of the batch normalized layer after a 1×3 convolutional layer. (1) , σ (1) γ (1) ,β (1) μ represents the parameters of the batch normalized layer after a 1×1 convolutional layer. (0) , σ (0) γ (0) ,β (0) The parameters of the batch normalization layer after the identity mapping layer are represented by M, where M represents the input and M' represents the output of RepCNN.

[0016] Convolutional layers do not use bias by default. The calculation formula for batch normalization layer fusion of the i-th channel of the input audio features is as follows:

[0017]

[0018] In a batch normalized layer, μ represents the mean, and σ 2 Let μ and σ represent the variance. 2 Calculated during the training process, γ represents the learned scaling factor, β represents the bias, and γ and β are learned through training. ε is a constant to prevent the denominator from being zero. The new weights and biases of the convolutional layer after fusion are:

[0019]

[0020] Furthermore, in step S3, the Mel filter bank consists of M triangular filters, and its frequency domain response is described as follows:

[0021]

[0022] Where M is the number of triangular filters in the Mel filter bank, k is the frequency value; f(m-1) and f(m+1) are the lower cutoff frequency and upper cutoff frequency of the m-th triangular filter, respectively;

[0023] The Mel filter bank extracts the M-dimensional power spectrum from the speech frame, then calculates the logarithm of the M-dimensional power spectrum to obtain the M-dimensional features. Two MFBF features are extracted from the same person's speech as input to the network. a M b express, M1 and M2 are the dimensions of the input features, and T is the number of input speech frames.

[0024] Furthermore, in step S4, As input to the cross-parallel convolutional neural network, k represents the k-th parallel convolutional layer, and the network consists of n parallel convolutional layers. and They represent and The length, and express and The width;

[0025] The cross-parallel convolutional neural network includes parallel layers, each of which includes two complementary feature extraction modules and two structurally reparameterized convolutional networks, RepCNN1 and RepCNN2.

[0026] The complementary feature extraction module consists of two reparameterized convolutional networks, RepCNN1 and RepCNN2, and their sigmoid activation functions, as well as an averaging module.

[0027] RepCNN1 in parallel layers is used to extract input features. The important information hidden within is expressed in the following expression:

[0028]

[0029] in This is the output of RepCNN layer 1. This represents the parameters in RepCNN1 layer;

[0030] Similarly, RepCNN2 is used to extract input features. Important information hidden within;

[0031] The complementary feature extraction module is used to obtain speaker input features. The complementary properties of these features are utilized in RepCNN1 and RepCNN2, both employing the sigmoid activation function to output feature matrices. The expression is:

[0032]

[0033] Where δ() represents the sigmoid activation function. and These represent the parameters of the convolutional layers in RepCNN2 and RepCNN1, respectively.

[0034] Then, the complementary feature extraction module, which incorporates complementary information from the speaker, outputs the result. To enhance speaker features

[0035]

[0036] in This represents the output of the cross-parallel convolutional neural network to the left of the k-th parallel convolutional layer. This indicates element-wise multiplication;

[0037] The output of the cross-parallel convolutional neural network to the right of the k-th parallel convolutional layer The formula is:

[0038]

[0039] in and These represent the parameters in the RepCNN, the right-hand cross-parallel convolutional neural network in the k-th parallel layer.

[0040] Furthermore, the final formula for the fusion feature is expressed as:

[0041]

[0042] Where K represents the last Kth parallel layer. and These represent the output feature vectors of the left and right intersecting parallel convolutional neural networks, respectively.

[0043] The beneficial effects of this invention are as follows: This invention uses Mel filter banks with different numbers of triangular filters to extract Mel filter bank features with different frequency domain resolutions from the speaker's speech frames. These features have complementary characteristics. Then, a cross-parallel convolutional neural network is used to capture complementary information and extract deep features. By connecting these deep features used for speaker recognition, fused features are obtained for speaker recognition. Furthermore, a multi-branch fusion technique is adopted. This invention reduces the model's dependence on computing resources, improves the model's operation speed and accuracy, and achieves lightweighting of the network model.

[0044] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0046] Figure 1 This is a schematic diagram of the speaker feature fusion module structure;

[0047] Figure 2 This is a schematic diagram of a parallel layer structure;

[0048] Figure 3 This is a schematic diagram of the RepCNN structure, which is a parameterization module for structural punching. Detailed Implementation

[0049] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0050] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0051] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0052] Please see Figures 1-3 To address the needs for complementary and lightweight speaker recognition models, this paper studies a speaker recognition method based on a reparameterized feature fusion convolutional neural network. This network mainly consists of two modules: a structure-reparameterized convolutional module and a speaker feature fusion module. Specifically, during model training, the structure-reparameterized convolutional module adds parallel identity mapping branches and residual branches to the 3x1 convolutional layers. Then, during the model inference stage, a multi-branch fusion technique is used to convert the network into a single-path structure. The speaker feature fusion module extracts Mel filter bank features with different frequency domain resolutions from speech frames as input features. Using a parallel cross-connection structure, it achieves complementary feature information at different frequency domain resolutions, concatenating the deep speaker features to obtain the final speaker feature vector.

[0053] The structured reparameterized convolutional module adds parallel identity mapping branches and residual branches to the 3x1 convolutional layers during model training. Then, during model inference, it employs multi-branch fusion technology to transform the network into a single-path structure. This module serves as the basic network module for speaker feature fusion, such as... Figure 2 As shown in RepCNN1 and RepCNN2, it can effectively improve model performance.

[0054] Structure reparameterization: using This indicates that the input channel is C. in The output channel is C. out The convolutional layer has a 3×1 kernel. This indicates a convolutional layer with a 1×3 kernel. To represent a branch with a 1×1 kernel, use μ.(3) , σ (3) γ (3) ,β (3) μ represents the parameters of the post-batch normalized layer of a 3×1 convolutional layer. (2) , σ (2) γ (2) ,β (2) μ represents the parameters of the batch normalized layer after a 1×3 convolutional layer. (1) , σ (1) γ (1) ,β (1) μ represents the parameters of the batch normalized layer after a 1×1 convolutional layer. (0) , σ (0) γ (0) ,β (0) Let M represent the parameters of the batch normalization layer after the identity mapping layer, and M' represent the input and the output of the reparameterized convolutional module RepCNN. When the number of input and output channels, width, and height are the same, the output can be represented as:

[0055]

[0056] When the structure-reparameterized convolutional network is RepCNN1, i = 3; when the structure-reparameterized convolutional network is RepCNN2, i = 2.

[0057] Convolutional layer and batch normalization layer fusion: Convolutional layers do not use bias by default. The calculation formula for batch normalization layer fusion of the i-th channel of the input audio features is as follows:

[0058]

[0059] In a batch normalized layer, μ represents the mean, and σ 2 Let μ and σ represent the variance. 2 γ, representing the learned scaling factor, and β, representing the bias, are calculated during the training process. Both γ and β are learned through training. To prevent the denominator from being zero, ε is set to a very small constant. The new weights and biases of the fused convolutional layer are:

[0060]

[0061] The speaker feature fusion module is based on a reparameterized convolution module and builds a parallel cross network structure. It takes Mel filter bank features with different frequency domain resolutions as input, captures the complementary characteristics of different frequency domain features, and fuses them using a parallel cross network. This module can effectively extract key speaker information and finally output the speaker's deep feature vector.

[0062] Multi-frequency domain feature extraction: Mel filter banks with different numbers of triangular filters are used to extract features with two different frequency domain resolutions from the speaker's speech frames. As the number of triangular filters in the Mel filter bank increases, the bandwidth of the triangular filters decreases, the frequency domain resolution of the Mel filter bank increases, and the Mel filter bank can obtain more frequency domain details.

[0063] The Mel filter bank consists of M triangular filters, and its frequency domain response is described as follows:

[0064]

[0065] Where M is the number of triangular filters in the Mel filter bank, commonly 13, 26, and 40, and k is the frequency value. f(m-1) and f(m+1) are the lower and upper cutoff frequencies of the m-th triangular filter, respectively. The Mel filter bank can extract M-dimensional power spectra from speech frames. Then, by calculating the logarithm of the M-dimensional power spectrum of the speech frame, M-dimensional features are obtained. Two MFBF features are extracted from the speech of the same person as the input of the network, using M... a M b express, M1 and M2 are the dimensions of the input features, and T is the number of input speech frames.

[0066] Feature fusion: As input to the cross-parallel convolutional neural network, k represents the k-th parallel convolutional layer, and the network consists of n parallel convolutional layers. and They represent and The length, and express and The width is [not specified]. The parallel layer contains two symmetrical, cross-parallel convolutional neural networks, each containing a reparameterized convolutional module (RepCNN) and a complementary feature extraction module. RepCNN1 is used to extract input features. The important information hidden within is expressed in the following expression:

[0067]

[0068] in This is the output of RepCNN layer 1. This represents the parameters in RepCNN1 layer. The cross-parallel convolutional neural network is used to extract speaker input features. The complementary properties of these features are utilized in RepCNN1 and RepCNN2, both employing the sigmoid activation function to output feature matrices. The expression is:

[0069]

[0070] Where δ() represents the sigmoid activation function. and These represent the parameters of the convolutional layers in RepCNN2 and RepCNN1, respectively. Then, the output is processed through a cross-convolutional neural network that incorporates complementary speaker information. To enhance speaker features

[0071]

[0072] in This represents the output of the cross-parallel convolutional neural network to the left of the k-th parallel convolutional layer. This indicates element-wise multiplication. Because the two intersecting parallel convolutional neural networks are symmetrical,

[0073] The output of the cross-parallel convolutional neural network to the right of the k-th parallel convolutional layer The formula is:

[0074]

[0075] in and ...

[0076]

[0077] Where K represents the last Kth parallel layer. and These represent the output feature vectors of the left and right intersecting parallel convolutional neural networks, respectively.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A speaker recognition method based on a reparameterized cross-parallel convolutional neural network, characterized in that: Includes the following steps: S1: Construct the RepCNN network with reparameterized structure; S2: Building a cross-parallel convolutional neural network based on RepCNN; S3: Using a Mel filter bank with two different numbers of triangular filters, features with two different frequency domain resolutions are extracted from the speaker's speech frames. , In step S3, the Mel filter bank consists of M triangular filters, and its frequency domain response is described as follows: Where M is the number of triangular filters in the Mel filter bank, and k is the frequency value; and These are the lower cutoff frequency and upper cutoff frequency of the m-th triangular filter, respectively; The Mel filter bank extracts the M-dimensional power spectrum from the speech frame, then calculates the logarithm of the M-dimensional power spectrum to obtain the M-dimensional features. Two MFBF features are extracted from the same person's speech and used as the network input. , express, , , and Where T is the dimension of the input features, and T is the number of input speech frames; S4: with , As input to a cross-parallel convolutional neural network, the output enhances features. and In step S4, , As input to the cross-parallel convolutional neural network, k represents the k-th parallel convolutional layer, and the network consists of n parallel convolutional layers. , , and They represent and The length, and express and The width; The cross-parallel convolutional neural network includes parallel layers, each of which includes two complementary feature extraction modules and two structurally reparameterized convolutional networks, RepCNN1 and RepCNN2. The complementary feature extraction module consists of two reparameterized convolutional networks, RepCNN1 and RepCNN2, and their sigmoid activation functions, as well as an averaging module. RepCNN1 in parallel layers is used to extract input features. The important information hidden within is expressed in the following expression: in This is the output of RepCNN layer 1. , This represents the parameters in RepCNN1 layer; Similarly, RepCNN2 is used to extract input features. Important information hidden within; The complementary feature extraction module is used to obtain speaker input features. , The complementary properties of these features are utilized in RepCNN1 and RepCNN2, both employing the sigmoid activation function to output feature matrices. The expression is: in This represents the sigmoid activation function. and These represent the parameters of the convolutional layers in RepCNN2 and RepCNN1, respectively. Then, the complementary feature extraction module, which incorporates complementary information from the speaker, outputs the result. To enhance speaker features : in This represents the output of the cross-parallel convolutional neural network to the left of the k-th parallel convolutional layer. , This indicates element-wise multiplication; The output of the cross-parallel convolutional neural network to the right of the k-th parallel convolutional layer The formula is: in , and These represent the parameters in the RepCNN, the right-hand cross-parallel convolutional neural network in the k-th parallel layer; S5: Features and Perform feature fusion to output the speaker's deep feature vector.

2. The speaker recognition method based on a reparameterized cross-parallel convolutional neural network according to claim 1, characterized in that: The Reparameterized Convolutional Network (RepCNN) adds parallel identity mapping branches and residual branches to the 3x1 convolutional layers during model training, and employs multi-branch fusion techniques during model inference to transform the network into a single-path structure; specifically including: When the number of input and output channels, width, and height are the same, the output is represented as: When the structure-reparameterized convolutional network is RepCNN1 When the reparameterized convolutional network is RepCNN2 , Indicates the input channel is The output channel is The convolutional layer has a 3×1 kernel. This indicates a convolutional layer with a 1×3 kernel. To represent a branch with a 1×1 kernel, use This represents the parameters of the post-batch normalized layer of a 3×1 convolutional layer. This represents the parameters of the post-batch normalized layer of a 1×3 convolutional layer. This represents the parameters of the post-batch normalized layer of a 1×1 convolutional layer. The parameters of the batch normalization layer after the identity mapping layer are represented by... Indicates input, This represents the output of RepCNN. , ; Convolutional layers do not use bias by default. The calculation formula for batch normalization layer fusion of the i-th channel of the input audio features is as follows: In the batch normalization layer, This represents the mean. Represents variance. and Calculated through the training process. This represents the learned scaling factor. Indicates deviation, and Acquired through training To prevent the denominator from being zero, the new weights and biases of the convolutional layer after fusion are: ， 。 3. The speaker recognition method based on a reparameterized cross-parallel convolutional neural network according to claim 1, characterized in that: The final formula for the fusion feature is expressed as: Where K represents the last Kth parallel layer. and These represent the output feature vectors of the left and right intersecting parallel convolutional neural networks, respectively.

Images