A voiceprint recognition method and system based on maximum pooling self-attention mechanism

Abstract

The present invention discloses a voiceprint recognition method and system based on a maximum pooling self-attention mechanism. The method of the present invention comprises the following steps: extracting an audio feature sequence; downsampling the audio feature sequence using a convolutional neural network; training a Transformer encoder based on a maximum pooling self-attention mechanism; regularizing the output of the Transformer encoder; extracting the speaker's voiceprint encoding using a statistical attention aggregation method; and minimizing the training loss using an angular prototype loss function. Compared to traditional Transformer voiceprint recognition, the present invention improves the Transformer's self-attention mechanism, introduces a maximum pooling operation in the similarity matrix, and enhances the self-attention mechanism's ability to model local information. The present invention is more suitable for voiceprint recognition than the original Transformer and has a higher accuracy rate.

Description

Technical Field

[0001] The present invention relates to the field of voiceprint recognition, and in particular to a voiceprint recognition method and system based on a maximum pooling self-attention mechanism. Background Art

[0002] Due to its powerful ability to model long sequences, the Transformer has become a dominant force in many natural language processing (NLP) tasks. As a core module of the Transformer, the self-attention mechanism was originally proposed to model the global information of text sequences. In the original self-attention mechanism, every score in the similarity matrix is considered, which helps to extract global information. However, the acoustic feature sequences that distinguish different speakers rely primarily on local information, making the Transformer uncompetitive in speaker recognition tasks.

[0003] As an indispensable component of convolutional neural network (CNN), the maximum pooling operation plays an important role in expanding the receptive field and reducing calculations, and can extract local information well. Summary of the Invention

[0004] To address the above problems, the present invention provides a voiceprint recognition method and system based on the maximum pooling self-attention mechanism, which introduces the maximum pooling operation into the self-attention mechanism and applies a restricted similarity score selection mode to the self-attention mechanism to improve the accuracy of Transformer in voiceprint recognition tasks.

[0005] To this end, the present invention adopts a technical solution as follows: a voiceprint recognition method based on the maximum pooling self-attention mechanism, which includes the following steps:

[0006] S1: Input audio A, extract audio feature sequence, and obtain sequence T;

[0007] S2: Use a convolutional neural network to downsample the sequence T obtained in step S1 to obtain a downsampled sequence S;

[0008] S3: Input the sequence S obtained in step S2 into the Transformer encoder based on the maximum pooling self-attention mechanism for training to obtain the output sequence O;

[0009] S4: Regularize the output sequence O obtained in step S3 to obtain the regularized sequence L;

[0010] S5: Extract the speaker's voiceprint code e from the sequence L obtained in step S4 using the statistical attention aggregation method;

[0011] S6: Use the angle prototype loss function to optimize the loss during training to minimize it.

[0012] Furthermore, the process of downsampling using a convolutional neural network in step S2 is as follows:

[0013] S2-1: Input the sequence obtained in step S1 Among them, B is the batch size, F is the sequence length, and D is the feature dimension. The sequence T is downsampled twice using a 2D convolution kernel of size 3×3 and stride 2, and activated using the ReLU activation function to obtain the sequence Where C is the number of channels, F′ is the length of the sequence after downsampling, and D′ is the feature dimension after downsampling. The process is as follows:

[0014] T′=ReLU(Conv2d(T))

[0015] S2-2: Input the sequence T′ obtained in step S2-1, and downsample the sequence T′ again using a 2D convolution kernel of size 3×3 and stride 2, and activate it using the ReLU activation function to obtain the sequence Where F″ is the length of the sequence after further downsampling, and D″ is the feature dimension after further downsampling. The process is as follows:

[0016] T″=ReLU(Conv2d(T′))

[0017] S2-3: Perform dimensionality reduction on the sequence T″ obtained in step S2-2, and finally obtain the sequence

[0018] Furthermore, the self-attention mechanism process when using the Transformer encoder for training in step S3 is as follows:

[0019] S3-1: Project the sequence S obtained in step S2 to obtain the query set Q, key-value set K, and value set V. The process is as follows:

[0020] Q=SW Q

[0021] K=SW K

[0022] V=SW V

[0023] Among them, W Q 、W K 、W V are the first learnable parameter, the second learnable parameter, and the third learnable parameter respectively;

[0024] S3-2: Calculate the similarity matrix Λ using the query set Q and the key-value set K. The calculation process is as follows:

[0025]

[0026] Where T represents transpose; d k Indicates the dimension size of K;

[0027] S3-3: Introduce the maximum pooling operation with a convolution kernel size of 2×2 to the matrix Λ obtained in step S3-2, and obtain the position information set P = {p1, p2, ..., p N}, where N is the number of location information, and the calculation process is as follows:

[0028] P=maxp(Λ)

[0029] Among them, the maxp() function returns the position information of the element with the largest similarity score in each convolution kernel;

[0030] S3-4: Generate a bias matrix M based on the position information set P obtained in step S3-3. M is defined as follows:

[0031]

[0032] S3-5: Calculate the final result, i.e., the output sequence O, based on the value set V obtained in step S3-1, the similarity matrix Λ obtained in step S3-2, and the bias matrix M obtained in step S3-4. The calculation process is as follows:

[0033] Attention(Q,K,V)=softmax(Λ+M)V

[0034] Among them, softmax() is an exponential normalization function.

[0035] Furthermore, the calculation process of regularization in step S4 is as follows:

[0036] The output sequence O obtained in step S3 is {o1, o2, ..., o x}, x is the length of sequence O, and after regularizing sequence O, we get sequence L = {l1,l2,…,l x}, the calculation process is as follows:

[0037]

[0038] Among them, ||o i ||2 means finding vector o i The 2-norm of .

[0039] Furthermore, the process of extracting the voiceprint code e in step S5 is as follows:

[0040] S5-1: The sequence L obtained in step S4 is L = {l1, l2, ..., l x}, calculate each frame l through the ReLU activation function t The scalar score s t , the calculation process is as follows:

[0041] s t =ReLU(l t )

[0042] S5-2: Calculate each scalar score s t Weight w t , the calculation process is as follows:

[0043]

[0044] Where exp(*) represents the exponential function;

[0045] S5-3: The weight w obtained in step S5-2 t The sequence L obtained in step S4 is L={l1,l2,…,l x}Weighted summation is performed to obtain the average frame vector μ. The calculation process is as follows:

[0046]

[0047] S5-4: Based on the average frame vector μ obtained in step S5-3, calculate the standard deviation of each frame from the average frame vector μ to obtain the standard deviation vector σ. The calculation process is as follows:

[0048]

[0049] Where, represents the Hadamard product;

[0050] S5-5: Concatenate the average frame vector μ obtained in step S5-3 and the standard deviation vector σ obtained in step S5-4 to obtain the final voiceprint code e.

[0051] Furthermore, the training loss calculation process in step S6 is as follows:

[0052] S6-1: Arrange the speaker's voiceprint code e obtained in step S5 into a voiceprint code set E = (e 1,1 ,e 1,2 ,…,e 1,M ,…,e H,M ), where H represents the number of speakers and M represents the number of voiceprint codes for each speaker. Take the first M-1 voiceprint codes of each speaker and calculate the center point c of the voiceprint code feature of each speaker. The calculation process is as follows:

[0053]

[0054] Among them, c irepresents the feature center of the i-th speaker, e i,j Represents the jth voiceprint code of the i-th speaker;

[0055] S6-2: Based on the center point c of the voiceprint encoding feature of each speaker obtained in step S6-1, calculate the similarity matrix G composed of the cosine similarities between the Mth voiceprint encoding of the current speaker and the center point c of the voiceprint encoding feature of all speakers. The calculation process is as follows:

[0056] G i,j =ω*cos(e i,M ,c j )+b

[0057] Among them, ω and b are the fourth and fifth learnable parameters respectively, cos(e i,M ,c j ) represents the vector e i,M with c j The cosine similarity between G i,j Represents the cosine similarity of the feature centers of the i-th speaker and the j-th speaker;

[0058] S6-3: Calculate the training loss based on the similarity matrix G obtained in step S6-2 The calculation process is as follows:

[0059]

[0060] Another technical solution adopted by the present invention is: a voiceprint recognition system based on the maximum pooling self-attention mechanism, which includes:

[0061] Audio feature sequence extraction unit: input audio A, extract audio feature sequence, and obtain sequence T;

[0062] Downsampled sequence acquisition unit: uses a convolutional neural network to downsample the sequence T obtained by the audio feature sequence extraction unit to obtain a downsampled sequence S;

[0063] Output sequence acquisition unit: The sequence S obtained by the downsampling sequence acquisition unit is input into the Transformer encoder based on the maximum pooling self-attention mechanism for training to obtain the output sequence O;

[0064] Regularization unit: Regularizes the output sequence O obtained by the output sequence acquisition unit to obtain the regularized sequence L;

[0065] Voiceprint code extraction unit: uses the statistical attention aggregation method to extract the speaker's voiceprint code e from the sequence L obtained by the regularization unit;

[0066] Minimization unit: Use the angular prototype loss function to optimize the loss during training to minimize it.

[0067] The beneficial effects of the present invention are as follows: compared with the traditional Transformer for voiceprint recognition, the present invention improves the self-attention mechanism of the Transformer, introduces the maximum pooling operation in the similarity matrix, and improves the self-attention mechanism's modeling ability for local information; applies the restricted similarity score selection mode to the self-attention mechanism to improve the accuracy of the Transformer in voiceprint recognition tasks; the present invention is more suitable for voiceprint recognition than the original Transformer and has a higher accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 This is a flow chart of a voiceprint recognition method based on the maximum pooling self-attention mechanism of the present invention;

[0070] Figure 2 Schematic diagram of the position-based self-attention mechanism adopted in the present invention;

[0071] Figure 3 Schematic diagram of the content-based self-attention mechanism adopted by the present invention;

[0072] Figure 4 Schematic diagram of the self-attention mechanism based on maximum pooling adopted in the present invention;

[0073] Figure 5 This is a schematic diagram of the architecture of a voiceprint recognition system based on the maximum pooling self-attention mechanism of the present invention;

[0074] Figure 6 Schematic diagram of the architecture of the post-downsampling sequence acquisition unit of the present invention;

[0075] Figure 7 Schematic diagram of the output sequence acquisition unit of the present invention;

[0076] Figure 8 Schematic diagram of the architecture of the voiceprint encoding and extraction unit of the present invention;

[0077] Figure 9 It is a schematic diagram of the architecture of the minimized unit of the present invention. DETAILED DESCRIPTION

[0078] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0079] Example 1

[0080] This embodiment is a voiceprint recognition method based on the maximum pooling self-attention mechanism, which includes the following steps:

[0081] S1: Input audio A, extract the audio feature sequence, and obtain sequence T.

[0082] S2: Use a convolutional neural network (CNN) to downsample the sequence T obtained in step S1 to obtain a downsampled sequence S.

[0083] Specifically, the downsampling process using CNN in step S2 is as follows:

[0084] S2-1: Input the sequence obtained in step S1 Where B is the batch size, F is the sequence length, and D is the feature dimension. The sequence T is downsampled twice using a 3×3 two-dimensional convolution kernel with a step size of 2, and activated using the ReLU activation function to obtain the sequence Where C is the number of channels, F′ is the length of the sequence after downsampling, and D′ is the feature dimension after downsampling. The process is as follows:

[0085] T′=ReLU(Conv2d(T))

[0086] S2-2: Input the sequence T′ obtained in step S2-1, and downsample the sequence T′ again using a 2D convolution kernel of size 3×3 and stride 2, and activate it using the ReLU activation function to obtain the sequence Where F″ is the length of the sequence after further downsampling, and D″ is the feature dimension after further downsampling. The process is as follows:

[0087] T″=ReLU(Conv2d(T′))

[0088] S2-3: Perform dimensionality reduction on the sequence T″ obtained in step S2-2, and finally obtain the sequence

[0089] S3: Input the sequence S obtained in step S2 into the Transformer encoder based on the maximum pooling self-attention mechanism for training to obtain the output sequence O.

[0090] Specifically, the self-attention mechanism process when step S3 uses the Transformer encoder for training is as follows:

[0091] S3-1: Project the sequence S obtained in step S2 to obtain the query set Q, key-value set K, and value set V. The calculation process is as follows:

[0092] Q=SW Q

[0093] K=SW K

[0094] V=SW V

[0095] Among them, W Q 、W K 、W V are the first learnable parameter, the second learnable parameter, and the third learnable parameter respectively;

[0096] S3-2: Calculate the similarity matrix Λ using the query set Q and the key-value set K. The calculation process is as follows:

[0097]

[0098] Where T represents transpose; d k Indicates the dimension size of K;

[0099] S3-3: Introduce the max-pooling operation with a convolution kernel size of 2×2 to the matrix Λ obtained in step S3-2, and obtain the position information set P = {p1, p2, ..., p N}, where N is the number of location information, and the calculation process is as follows:

[0100] P=maxp(Λ)

[0101] Among them, the maxp() function returns the position information of the element with the largest similarity score in each convolution kernel;

[0102] S3-4: Generate a bias matrix M using the similarity matrix Λ obtained in step S3-2 and the position information set P obtained in step S3-3. M is defined as follows:

[0103]

[0104] S3-5: Calculate the final result based on the value set V obtained in step S3-1, the similarity matrix Λ obtained in step S3-2, and the bias matrix M obtained in step S3-4. The calculation process is as follows:

[0105] Attention(Q,K,V)=softmax(Λ+M)V

[0106] Among them, softmax() is an exponential normalization function.

[0107] S4: Regularize the output sequence O obtained in step S3 to obtain a regularized sequence L.

[0108] Specifically, the calculation process of regularization in step S4 is as follows:

[0109] The output O obtained in step S3 is {o1, o2, ..., o x}, x is the length of sequence O, and after regularizing sequence O, we get sequence L = {l1,l2,…,l x}, the calculation process is as follows:

[0110]

[0111] Among them, ||X||2 means finding the 2-norm of vector X.

[0112] S5: Use the statistical attention aggregation method to extract the speaker voiceprint code e from the sequence L obtained in step S4.

[0113] Specifically, the process of extracting the voiceprint code e in step S5 is as follows:

[0114] S5-1: The sequence L obtained in step S4 is L = {l1, l2, ..., l x}, calculate each frame l through the ReLU activation function t The scalar score s t , the calculation process is as follows:

[0115] s t =ReLU(l t )

[0116] S5-2: Calculate each scalar score s t Weight w t , the calculation process is as follows:

[0117]

[0118] Where exp(*) represents the exponential function.

[0119] S5-3: The weight w obtained in step S5-2 t The sequence L obtained in step S4 is L={l1,l2,…,l x}Weighted summation is performed to obtain the average frame vector μ. The calculation process is as follows:

[0120]

[0121] S5-4: Based on the average frame vector μ obtained in step S5-3, calculate the standard deviation of each frame from the average frame vector μ to obtain the standard deviation vector σ. The calculation process is as follows:

[0122]

[0123] Where, represents the Hadamard product;

[0124] S5-5: Concatenate the average frame vector μ obtained in step S5-3 and the standard deviation vector σ obtained in step S5-4 to obtain the final voiceprint code e.

[0125] S6: Use the angle prototype loss function to optimize the loss during training to minimize it.

[0126] Specifically, the training loss calculation process in step S6 is as follows:

[0127] S6-1: Arrange the speaker's voiceprint code e obtained in step S5 into a voiceprint code set E = (e 1,1 ,e 1,2 ,…,e 1,M ,…,e H,M ), where H represents the number of speakers and M represents the number of voiceprint codes for each speaker. Take the first M-1 voiceprint codes of each speaker and calculate the center point c of the voiceprint code feature of each speaker. The calculation process is as follows:

[0128]

[0129] Among them, c i represents the feature center of the i-th speaker, e i,j Represents the j-th voiceprint code of the i-th speaker.

[0130] S6-2: Based on the center point c of the voiceprint encoding feature of each speaker obtained in step S6-1, calculate the similarity matrix G composed of the cosine similarities between the Mth voiceprint encoding of the current speaker and the center point c of the voiceprint encoding feature of all speakers. The calculation process is as follows:

[0131] G i,j =ω*cos(e i,M ,c j )+b

[0132] Among them, ω and b are the fourth and fifth learnable parameters respectively, cos(u,v) represents the cosine similarity between vectors u and v, G i,j Represents the cosine similarity between the feature centers of the i-th speaker and the j-th speaker.

[0133] S6-3: Calculate the training loss based on the similarity matrix G obtained in step S6-2 The calculation process is as follows:

[0134]

[0135] Application Examples

[0136] The present invention uses the AISHELL-1 and train-clean-360 LibriSpeech datasets. For LibriSpeech, the test-clean dataset was used for testing. After removing audio less than 2 seconds, the AISHELL-1 dataset contains 141,684 audio clips from 400 speakers, and the LibriSpeech dataset contains 103,798 audio clips from 2097 speakers for training and 2584 audio clips from 87 speakers for testing. The AISHELL-1 dataset is divided into training and test sets in a ratio of 9:1. The input acoustic features are extracted from 80-dimensional Fbank features using a 25 millisecond window and a 10 millisecond offset.

[0137] In order to increase the reliability of the experiment, the position-based and content-based similarity selection methods are introduced in the self-attention mechanism as a control based on the maximum pooling method during the implementation process, such as Figure 2-4 shown. Figure 2-4 The gray filled squares represent the selected similarity scores. At the same time, the average value and embedding learnable code extraction methods are introduced in the voiceprint code extraction process as a comparison with the statistical attention aggregation method.

[0138] The present invention uses the Adam optimizer with β1=0.9, β1=0.999, weight decay 0.1, and learning rate 0.001. The batch size is 128, and each speaker has 8 audios. Strictly following the original Transformer settings, the input dimension is 512, the number of encoder layers is 6, 8 encoder attention heads, and dropout of 0.1. The present invention uses the equal error rate (EER) and the minimum detection cost function P target =0.01 (MinDCF0.01) as the test index. ω and b are initially set to 10 and -5. The test results are shown in Table 1.

[0139] The voiceprint extraction method plays a bridging role between frame-level features and audio-level features. The present invention selects the methods of averaging, embedding learnable vectors and statistical attention aggregation to aggregate frame-level features into speaker features. As shown in Table 1, the statistical attention aggregation method outperforms the other two methods on both datasets, whether it is the original or restricted self-attention mechanism. Although averaging is simple to implement and has almost no cost, it regards each frame-level feature as the same weight, which may be inappropriate. The present invention further adds a learnable vector to the input of the Transformer, but it is not competitive with the statistical attention aggregation method. The statistical attention aggregation method calculates the weight of each frame-level vector through the self-attention mechanism. Compared with the averaging method, the model of the present invention improves the EER on AISHELL-1 and MinDCF on LibriSpeech by at least 11.4% and 8.5% respectively.

[0140] This embodiment compares restricted self-attention (i.e., position-based, content-based, and max-pooling-based) with ordinary self-attention mechanisms, and the results are shown in Table 1. On the one hand, it can be seen that restricting self-attention to local information can greatly improve the performance of the voiceprint system, and all three methods introduced in this embodiment are better than the original Transformer. On the other hand, among the restricted self-attention, the self-attention mechanism based on max-pooling achieves the best performance, with an EER of 2.47% and a MinDCF of 0.2198 on AISHELL-1, and an EER of 1.64% and a MinDCF of 0.2668 on the LibriSpeech dataset. The position-based mechanism focuses information on position information and ignores content information, while the content-based one does the opposite. Within a limited knowledge scope, introducing the max-pooling operation into the self-attention mechanism can help the Transformer combine position information and content information to model local information, further improving performance.

[0141] Table 1 Test results

[0142]

[0143]

[0144] Example 2

[0145] This embodiment is a voiceprint recognition system based on the maximum pooling self-attention mechanism, which consists of an audio feature sequence extraction unit, a downsampling sequence acquisition unit, an output sequence acquisition unit, a regularization unit, a voiceprint encoding extraction unit, and a minimization unit. Figure 5 shown.

[0146] The audio feature sequence extraction unit: inputs audio A, extracts the audio feature sequence, and obtains sequence T.

[0147] Downsampled sequence acquisition unit: Use a convolutional neural network to downsample the sequence T obtained by the audio feature sequence extraction unit to obtain a downsampled sequence S.

[0148] The down-sampling sequence acquisition unit includes the following subunits: Figure 6 As shown:

[0149] First downsampling subunit: input the sequence obtained by the audio feature sequence extraction unit Among them, B is the batch size, F is the sequence length, and D is the feature dimension. The sequence T is downsampled twice using a 2D convolution kernel of size 3×3 and stride 2, and activated using the ReLU activation function to obtain the sequence Where C is the number of channels, F′ is the length of the sequence after downsampling, and D′ is the feature dimension after downsampling. The process is as follows:

[0150] T′=ReLU(Conv2d(T))

[0151] Downsampling subunit again: Input the sequence T′ obtained by the first downsampling subunit, and downsample the sequence T′ again using a two-dimensional convolution kernel of size 3×3 and step size 2, and activate it using the ReLU activation function to obtain the sequence Where F″ is the length of the sequence after further downsampling, and D″ is the feature dimension after further downsampling. The process is as follows:

[0152] T″=ReLU(Conv2d(T′))

[0153] Dimensionality reduction operation subunit: perform dimensionality reduction operation on the sequence T″ obtained by the downsampling subunit again, and finally obtain the sequence

[0154] Output sequence acquisition unit: The sequence S obtained by the downsampling sequence acquisition unit is input into the Transformer encoder based on the maximum pooling self-attention mechanism for training to obtain the output sequence O.

[0155] The output sequence acquisition unit includes the following subunits: Figure 7 As shown:

[0156] Projection subunit: The query set Q, key-value set K, and value set V are obtained by projecting the sequence S obtained by the downsampling sequence acquisition unit. The process is as follows:

[0157] Q=SW Q

[0158] K=SWK

[0159] V=SW V

[0160] Among them, W Q 、W K 、W V are the first learnable parameter, the second learnable parameter, and the third learnable parameter respectively;

[0161] Similarity matrix calculation subunit 1: The similarity matrix Λ is calculated by the query set Q and the key value set K. The calculation process is as follows:

[0162]

[0163] Where T represents transpose; d k Indicates the dimension size of K;

[0164] Maximum pooling operation subunit: The matrix Λ obtained by the first subunit of similarity matrix calculation introduces the maximum pooling operation with a convolution kernel size of 2×2, and obtains the position information set P={p1,p2,…,p N}, where N is the number of location information, and the calculation process is as follows:

[0165] P=maxp(Λ)

[0166] Among them, the maxp() function returns the position information of the element with the largest similarity score in each convolution kernel;

[0167] Bias matrix generation subunit: The position information set P obtained by the maximum pooling operation subunit generates the bias matrix M, which is defined as follows:

[0168]

[0169] Final result calculation subunit: Calculate the final result, i.e., the output sequence O, based on the value set V obtained by the projection subunit, the similarity matrix Λ obtained by the first similarity matrix calculation subunit, and the bias matrix M obtained by the bias matrix generation subunit. The calculation process is as follows:

[0170] Attention(Q,K,V)=softmax(Λ+M)V

[0171] Among them, softmax() is an exponential normalization function.

[0172] Regularization unit: Regularizes the output sequence O obtained by the output sequence acquisition unit to obtain a regularized sequence L.

[0173] The calculation process for regularization is as follows:

[0174] The output sequence O obtained by the output sequence acquisition unit is O={o1,o2,…,o x}, x is the length of sequence O, and after regularizing sequence O, we get sequence L = {l1,l2,…,l x}, the calculation process is as follows:

[0175]

[0176] Among them, ||o i ||2 means finding vector o i The 2-norm of .

[0177] Voiceprint code extraction unit: Use the statistical attention aggregation method to extract the speaker's voiceprint code e from the sequence L obtained by the regularization unit.

[0178] The voiceprint encoding extraction unit includes the following subunits: Figure 8 As shown:

[0179] Scalar score calculation subunit: The sequence L obtained by the regularization unit is L = {l1,l2,…,l x}, calculate each frame l through the ReLU activation function t The scalar score s t , the calculation process is as follows:

[0180] s t =ReU(l t )

[0181] Scalar score weight calculation subunit: calculate each scalar score s t Weight w t , the calculation process is as follows:

[0182]

[0183] Where exp(*) represents the exponential function;

[0184] Average frame vector calculation subunit: weight w obtained by the scalar score weight calculation subunit t The sequence L obtained by the regularization unit is L = {l1,l2,…,l x}Weighted summation is performed to obtain the average frame vector μ. The calculation process is as follows:

[0185]

[0186] Standard deviation vector calculation subunit: Based on the average frame vector μ obtained by the average frame vector calculation subunit, calculate the standard deviation of each frame and the average frame vector μ to obtain the standard deviation vector σ. The calculation process is as follows:

[0187]

[0188] Where, ⊙ represents the Hadamard product;

[0189] Final voiceprint code acquisition subunit: concatenates the average frame vector μ obtained by the average frame vector calculation subunit and the standard deviation vector σ obtained by the standard deviation vector calculation subunit to obtain the final voiceprint code e.

[0190] Minimization unit: Use the angular prototype loss function to optimize the loss during training to minimize it.

[0191] The minimization unit includes the following subunits, such as Figure 9 As shown:

[0192] Feature center point calculation subunit: Arrange the speaker's voiceprint code e obtained by the voiceprint code extraction unit into a voiceprint code set E = (e 1,1 ,e 1,2 ,…,e 1,M ,…,e H,M ), where H represents the number of speakers and M represents the number of voiceprint codes for each speaker. Take the first M-1 voiceprint codes of each speaker and calculate the center point c of the voiceprint code feature of each speaker. The calculation process is as follows:

[0193]

[0194] Among them, c i represents the feature center of the i-th speaker, e i,j Represents the jth voiceprint code of the i-th speaker;

[0195] The second sub-unit of similarity matrix calculation: Based on the voiceprint encoding feature center point c of each speaker obtained by the feature center point calculation sub-unit, the similarity matrix G composed of the cosine similarity between the current speaker's M-th voiceprint encoding and the voiceprint encoding feature center point c of all speakers is calculated. The calculation process is as follows:

[0196] G i,j =ω*cos(e i,M ,c j )+b

[0197] Among them, ω and b are the fourth and fifth learnable parameters respectively, cos(e i,M ,c j ) represents the vector e i,M with c j The cosine similarity between G i,j Represents the cosine similarity of the feature centers of the i-th speaker and the j-th speaker;

[0198] Training loss calculation subunit: Calculate the similarity matrix G obtained by the second subunit based on the similarity matrix and calculate the training loss The calculation process is as follows:

[0199]

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voiceprint recognition method based on the maximum pooling self-attention mechanism, characterized in that: Including steps: S1: Input audio A, extract audio feature sequence, and obtain sequence T; S2: Use a convolutional neural network to downsample the sequence T obtained in step S1 to obtain a downsampled sequence S; S3: Input the sequence S obtained in step S2 into the Transformer encoder based on the maximum pooling self-attention mechanism for training to obtain the output sequence O; S4: Regularize the output sequence O obtained in step S3 to obtain the regularized sequence L; S5: Extract the speaker's voiceprint code e from the sequence L obtained in step S4 using the statistical attention aggregation method; S6: Arrange the speaker voiceprint codes e obtained in step S5 into a voiceprint code set, and calculate the center point of each speaker's voiceprint code feature; based on the center point of each speaker's voiceprint code feature, calculate a similarity matrix composed of the cosine similarity between the current speaker's M-th voiceprint code and the center points of all speakers' voiceprint code features; and optimize the loss during training using the angle prototype loss function based on the similarity matrix to minimize it; The self-attention mechanism process when step S3 uses the Transformer encoder for training is as follows: S3-1: Project the sequence S obtained in step S2 to obtain the query set Q, key-value set K, and value set V. The process is as follows: Q=SW Q K=SW K V=SW V Among them, W Q 、W K 、W V are the first learnable parameter, the second learnable parameter, and the third learnable parameter respectively; S3-2: Calculate the similarity matrix Λ using the query set Q and the key-value set K. The calculation process is as follows: Where T represents transpose; d k Indicates the dimension size of K; S3-3: Introduce the maximum pooling operation with a convolution kernel size of 2×2 to the matrix Λ obtained in step S3-2, and obtain the position information set P = {p1, p2, ..., p N }, where N is the number of location information, and the calculation process is as follows: P=maxp(Λ) Among them, the maxp() function returns the position information of the element with the largest similarity score in each convolution kernel; S3-4: Generate a bias matrix M based on the position information set P obtained in step S3-3. M is defined as follows: S3-5: Calculate the final result, which is the output sequence O, based on the value set V obtained in step S3-1, the similarity matrix Λ obtained in step S3-2, and the bias matrix M obtained in step S3-4. The calculation process is as follows: Attention(Q,K,V)=softmax(Λ+M)V Among them, softmax() is an exponential normalization function.

2. The voiceprint recognition method based on the maximum pooling self-attention mechanism according to claim 1 is characterized in that: The process of downsampling using a convolutional neural network in step S2 is as follows: S2-1: Input the sequence obtained in step S1 Among them, B is the batch size, F is the sequence length, and D is the feature dimension. The sequence T is downsampled twice using a 2D convolution kernel of size 3×3 and stride 2, and activated using the ReLU activation function to obtain the sequence Where C is the number of channels, F′ is the length of the sequence after downsampling, and D′ is the feature dimension after downsampling. The process is as follows: T′=ReLU(Conv2d(T)) S2-2: Input the sequence T′ obtained in step S2-1, and downsample the sequence T′ again using a 2D convolution kernel of size 3×3 and stride 2, and activate it using the ReLU activation function to obtain the sequence Where F″ is the length of the sequence after further downsampling, and D″ is the feature dimension after further downsampling. The process is as follows: T″=ReLU(Conv2d(T′)) S2-3: Perform dimensionality reduction on the sequence T″ obtained in step S2-2, and finally obtain the sequence 3. The voiceprint recognition method based on the maximum pooling self-attention mechanism according to claim 1 is characterized in that: The calculation process of regularization in step S4 is as follows: The output sequence O obtained in step S3 is {o1, o2, ..., o x }, x is the length of sequence O, and after regularizing sequence O, we get sequence L = {l1,l2,…,l x }, the calculation process is as follows: Among them, ‖o i ‖2 means finding vector o i The 2-norm of .

4. The voiceprint recognition method based on the maximum pooling self-attention mechanism according to claim 3 is characterized in that: The process of extracting the voiceprint code e in step S5 is as follows: S5-1: The sequence L obtained in step S4 is L = {l1, l2, ..., l x }, calculate each frame l through the ReLU activation function t The scalar score s t , the calculation process is as follows: s t =ReLU(l t ) S5-2: Calculate each scalar score s t Weight w t , the calculation process is as follows: Where exp(*) represents the exponential function; S5-3: The weight w obtained in step S5-2 t The sequence L obtained in step S4 is L={l1,l2,…,l x }Weighted summation is performed to obtain the average frame vector μ. The calculation process is as follows: S5-4: Based on the average frame vector μ obtained in step S5-3, calculate the standard deviation of each frame from the average frame vector μ to obtain the standard deviation vector σ. The calculation process is as follows: Where, ⊙ represents the Hadamard product; S5-5: Concatenate the average frame vector μ obtained in step S5-3 and the standard deviation vector σ obtained in step S5-4 to obtain the final voiceprint code e.

5. The voiceprint recognition method based on the maximum pooling self-attention mechanism according to claim 1 is characterized in that: Step S6 calculates the training loss process as follows: S6-1: Arrange the speaker's voiceprint code e obtained in step S5 into a voiceprint code set e=(e 1,1 ,e 1,2 ,…,e 1,M ,…,e H,M ), where H represents the number of speakers and M represents the number of voiceprint codes for each speaker. Take the first M-1 voiceprint codes of each speaker and calculate the center point c of the voiceprint code feature of each speaker. The calculation process is as follows: Among them, c i represents the feature center of the i-th speaker, e i,j Represents the jth voiceprint code of the i-th speaker; S6-2: Based on the center point c of the voiceprint encoding feature of each speaker obtained in step S6-1, calculate the similarity matrix G composed of the cosine similarities between the Mth voiceprint encoding of the current speaker and the center point c of the voiceprint encoding feature of all speakers. The calculation process is as follows: G i,j =ω*cos(e i,M ,c j )+b Among them, ω and b are the fourth and fifth learnable parameters respectively, cos(e i,M ,c j ) represents the vector e i,M with c j The cosine similarity between G i,j Represents the cosine similarity of the feature centers of the i-th speaker and the j-th speaker; S6-3: Calculate the training loss based on the similarity matrix G obtained in step S6-2 The calculation process is as follows:

6. A voiceprint recognition system based on the maximum pooling self-attention mechanism, characterized in that: include: Audio feature sequence extraction unit: input audio A, extract audio feature sequence, and obtain sequence T; Downsampled sequence acquisition unit: uses a convolutional neural network to downsample the sequence T obtained by the audio feature sequence extraction unit to obtain a downsampled sequence S; Output sequence acquisition unit: The sequence S obtained by the downsampling sequence acquisition unit is input into the Transformer encoder based on the maximum pooling self-attention mechanism for training to obtain the output sequence O; Regularization unit: Regularizes the output sequence O obtained by the output sequence acquisition unit to obtain the regularized sequence L; Voiceprint code extraction unit: uses the statistical attention aggregation method to extract the speaker's voiceprint code e from the sequence L obtained by the regularization unit; Minimization unit: The speaker's voiceprint code e obtained by the voiceprint code extraction unit is sorted into a voiceprint code set, and the center point of each speaker's voiceprint code feature is calculated; based on the center point of each speaker's voiceprint code feature, a similarity matrix is calculated, which is composed of the cosine similarity between the current speaker's M-th voiceprint code and the center points of the voiceprint code features of all speakers; based on the similarity matrix, the angle prototype loss function is used to optimize the loss during the training process to minimize it; The output sequence acquisition unit includes the following subunits: Projection subunit: The query set Q, key-value set K, and value set V are obtained by projecting the sequence S obtained by the downsampling sequence acquisition unit. The process is as follows: Q=SW Q K=SW K V=SW V Among them, W Q 、W K 、W V are the first learnable parameter, the second learnable parameter, and the third learnable parameter respectively; Similarity matrix calculation subunit 1: The similarity matrix Λ is calculated by the query set Q and the key value set K. The calculation process is as follows: Where T represents transpose; d k Indicates the dimension size of K; Maximum pooling operation subunit: The matrix Λ obtained by the first subunit of similarity matrix calculation introduces the maximum pooling operation with a convolution kernel size of 2×2, and obtains the position information set P={p1,p2,…,p N }, where N is the number of location information, and the calculation process is as follows: P=maxp(Λ) Among them, the maxp() function returns the position information of the element with the largest similarity score in each convolution kernel; Bias matrix generation subunit: The position information set P obtained by the maximum pooling operation subunit generates the bias matrix M, which is defined as follows: Final result calculation subunit: Calculate the final result, i.e., the output sequence O, based on the value set V obtained by the projection subunit, the similarity matrix Λ obtained by the first similarity matrix calculation subunit, and the bias matrix M obtained by the bias matrix generation subunit. The calculation process is as follows: Attention(Q,K,V)=softmax(Λ+M)V Among them, softmax() is an exponential normalization function.

7. A voiceprint recognition system based on maximum pooling self-attention mechanism according to claim 6, characterized in that: The down-sampled sequence acquisition unit includes the following subunits: First downsampling subunit: input the sequence obtained by the audio feature sequence extraction unit Among them, B is the batch size, F is the sequence length, and D is the feature dimension. The sequence T is downsampled twice using a 2D convolution kernel of size 3×3 and stride 2, and activated using the ReLU activation function to obtain the sequence Where C is the number of channels, F′ is the length of the sequence after downsampling, and D′ is the feature dimension after downsampling. The process is as follows: T′=ReLU(Conv2d(T)) Downsampling subunit again: Input the sequence T′ obtained by the first downsampling subunit, and downsample the sequence T′ again using a two-dimensional convolution kernel of size 3×3 and step size 2, and activate it using the ReLU activation function to obtain the sequence Where F″ is the length of the sequence after further downsampling, and D″ is the feature dimension after further downsampling. The process is as follows: T″=ReLU(Conv2d(T′)) Dimensionality reduction operation subunit: perform dimensionality reduction operation on the sequence T″ obtained by the downsampling subunit again, and finally obtain the sequence 8. The voiceprint recognition system based on the maximum pooling self-attention mechanism according to claim 6 is characterized in that: The voiceprint code extraction unit includes the following subunits: Scalar score calculation subunit: The sequence L obtained by the regularization unit is L = {l1,l2,…,l x }, calculate each frame l through the ReLU activation function t The scalar score s t , the calculation process is as follows: s t =ReLU(l t ) Scalar score weight calculation subunit: calculate each scalar score s t Weight w t , the calculation process is as follows: Where exp(*) represents the exponential function; Average frame vector calculation subunit: weight w obtained by the scalar score weight calculation subunit t The sequence L obtained by the regularization unit is L = {l1,l2,…,l x }Weighted summation is performed to obtain the average frame vector μ. The calculation process is as follows: Standard deviation vector calculation subunit: Based on the average frame vector μ obtained by the average frame vector calculation subunit, calculate the standard deviation of each frame and the average frame vector μ to obtain the standard deviation vector σ. The calculation process is as follows: Where, ⊙ represents the Hadamard product; Final voiceprint code acquisition subunit: concatenates the average frame vector μ obtained by the average frame vector calculation subunit and the standard deviation vector σ obtained by the standard deviation vector calculation subunit to obtain the final voiceprint code e.

9. The voiceprint recognition system based on the maximum pooling self-attention mechanism according to claim 6 is characterized in that: The minimization unit includes the following subunits: Feature center point calculation subunit: Arrange the speaker's voiceprint code e obtained by the voiceprint code extraction unit into a voiceprint code set E = (e 1,1 ,e 1,2 ,…,e 1,M ,…,e H,M ), where H represents the number of speakers and M represents the number of voiceprint codes for each speaker. Take the first M-1 voiceprint codes of each speaker and calculate the center point c of the voiceprint code feature of each speaker. The calculation process is as follows: Among them, c i represents the feature center of the i-th speaker, e i,j Represents the jth voiceprint code of the i-th speaker; The second sub-unit of similarity matrix calculation: Based on the voiceprint encoding feature center point c of each speaker obtained by the feature center point calculation sub-unit, the similarity matrix G composed of the cosine similarity between the current speaker's M-th voiceprint encoding and the voiceprint encoding feature center point c of all speakers is calculated. The calculation process is as follows: G i,j =ω*cos(e i,M ,c j )+b Among them, ω and b are the fourth and fifth learnable parameters respectively, cos(e i,M ,c j ) represents the vector e i,M with c j The cosine similarity between G i,j Represents the cosine similarity of the feature centers of the i-th speaker and the j-th speaker; Training loss calculation subunit: Calculate the similarity matrix G obtained by the second subunit based on the similarity matrix and calculate the training loss The calculation process is as follows:

Images