Speaker recognition feature extraction method and system based on self-supervised multi-task learning

Abstract

The application discloses a speaker recognition feature extraction method and system based on self-supervised multi-task learning, and belongs to the technical field of biometric identification. In order to solve the problem that the cost is high in the process of preparing a data set with labels, and the problem that it is difficult to determine specific speaker information without supervision information in the case that speech signals contain various complex information and have a special hierarchical structure of information, a multi-task learning encoder is used to extract a feature sequence, and the feature sequence is used for speaker recognition. In the process of training the encoder, the feature sequence corresponding to the audio in the training set is simultaneously sent to multiple processing tasks, the multiple processing tasks include an original waveform task, a logarithmic power spectrum task, a filter bank feature task and a mel frequency cepstral coefficient task, and a local information LIM identity prediction task and a global information GIM identity prediction task; and all task loss averages are used for back propagation to complete the training.

Description

Technical Field

[0001] This invention belongs to the field of biometrics technology, specifically relating to a speaker recognition feature extraction and recognition method and system. Background Technology

[0002] Speaker recognition is a biometric technology based on speech signals. It identifies the speaker by analyzing the vocal features within the speech signal. Speaker recognition technology has wide applications in security authentication, speech recognition, and speech synthesis. In security authentication, speaker recognition technology can be used for identity verification, access control, and telephone banking. It effectively prevents identity impersonation and information leakage. In speech recognition, speaker recognition technology can be used for personalized settings of speech recognition systems. By identifying the speaker, the system can be customized according to different speaker habits, thereby improving recognition accuracy. In speech synthesis, speaker recognition technology can be used for personalized settings of speech synthesis systems. By identifying the speaker, the system can be customized according to different speaker habits, thereby improving the naturalness and fluency of the synthesized speech.

[0003] The basic principle of speaker recognition is to identify the speaker by analyzing the sound features in the speech signal. Sound features include speech frequency, speech rate, pitch, volume, and rhythm. These sound features are unique to each individual and can therefore be used to identify the speaker. The typical recognition process mainly includes feature extraction and pattern matching. Feature extraction refers to extracting sound features related to the speaker's identity from the speech signal. Pattern matching refers to comparing the extracted sound features with known speaker models to determine the speaker's identity.

[0004] The development of speaker recognition technology can be divided into three stages: template matching, statistical modeling, and deep learning. Template matching involves comparing speech signals with known speaker templates to determine the speaker's identity. The drawback of this method is the need for a large amount of template data and high-quality speech signals. Statistical modeling involves building a speaker model by statistically analyzing the sound features in the speech signal. The advantage of this method is that it has lower requirements for speech signal quality but requires a large amount of training data. Deep learning utilizes deep neural networks to build speaker models. The advantage of this method is that it can automatically extract features from the speech signal, thereby improving recognition accuracy. However, speech signals contain various complex information and possess a unique hierarchical structure (sample → sentence → word → semantic content). This includes relevant information about the speaker at different time scales. Furthermore, different speech datasets exhibit considerable variability due to different language and acoustic environments or recording settings. Therefore, without any supervised guidance, it is difficult to infer the relevant underlying structure. However, training with deep learning requires a large amount of labeled datasets. Creating such labeled datasets typically requires a significant amount of time and effort, and accessing such annotated corpora is often expensive, leading to a surge in demand for technologies that enable research to extract speaker information from unlabeled data. Summary of the Invention

[0005] This invention addresses the high cost of creating labeled datasets and the difficulty in identifying specific speaker information without supervisory cues due to the complex and hierarchical nature of speech signals.

[0006] A speaker recognition feature extraction method based on self-supervised multi-task learning is proposed. For the acquired speaker audio, a multi-task learning encoder is used to extract a feature sequence U, which is then used for speaker recognition. The process of extracting the feature sequence U using the multi-task learning encoder includes the following steps:

[0007] Step 100: The audio is sent to the first feature extraction network module, and after processing by the first feature extraction network module, the speech feature sequence Z is obtained;

[0008] Step 200: Input the speech feature representation Z obtained in step 100 into the second feature extraction network module to obtain the speech feature sequence L; and input L into steps 300 and 400 simultaneously.

[0009] Step 300: The speech feature representation L obtained in step 200 is sent to the Transformer module, and after processing, the speech representation T is obtained;

[0010] Step 400: After fusing the speech feature sequence L and the T obtained by the Transformer module, the two sequences are fed into the fully connected layer and the normalization layer to obtain the feature sequence U.

[0011] The multi-task learning encoder is pre-trained, and the training process includes the following steps:

[0012] The original audio of the speaker is collected and data augmentation is performed on the original speech. A training set is constructed using the data augmented speech and the original speech. For the audio in the training set, the feature sequence U is first processed through steps 100 to 400. Then, the obtained feature sequence U is simultaneously sent to multiple processing tasks, including regression tasks and classification tasks. The regression tasks include the original waveform task, log power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task. The classification tasks include the Local Information Model (LIM) identity prediction task and the Global Information Model (GIM) identity prediction task.

[0013] The original waveform task is: based on the feature sequence U, predict the reconstructed waveform using an autoencoder; and obtain the loss between the reconstructed waveform and the original speech signal waveform.

[0014] The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task are as follows: For the data-enhanced speech and the original speech, based on the feature sequence U, the corresponding logarithmic power spectrum, filter bank features, and Mel frequency cepstral coefficients are obtained respectively; The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task each obtain the Mel frequency cepstral coefficient loss of the data-enhanced speech and the original speech.

[0015] Local Information Modeling (LIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: an anchor point and a positive sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. n It is the feature sequence U corresponding to the positive sample obtained based on the sampling strategy, and x a and x n As input to the classifier corresponding to the LIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained;

[0016] Global Information Modeling (GIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: the anchor point and the negative sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. p It is the average value of the feature sequence U corresponding to all samples in the 1-second speech block where the negative sample is obtained based on the sampling strategy. a and x pAs input to the classifier corresponding to the GIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained;

[0017] The sampling strategy mentioned above refers to the following: samples from the same sentence as the anchor point are positive samples, and samples from different sentences are negative samples;

[0018] The average of all loss values ​​is obtained based on the loss values ​​corresponding to multiple processing tasks, and used for backpropagation to obtain a multi-task learning encoder based on the training set.

[0019] Furthermore, the loss of the original waveform task is calculated using the Mean Absolute Error (MAE) function, as shown in the following formula:

[0020]

[0021] Where, f(u) i ) represents the waveform of the predicted speech signal obtained from the task, y i Represents the waveform of the original speech signal;

[0022] For the three regression tasks—log power spectrum, filter bank characteristics, and Mel frequency cepstral coefficients—the mean square error (MSE) function is used to calculate the loss value, and the formula is as follows:

[0023]

[0024] Logarithmic power spectrum task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U1 and U2, respectively. Obtain the corresponding logarithmic power spectra based on U1 and U2, denoted as g. 1i (u i ) and y 1i Filter bank feature task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U3 and U4, respectively. Based on U3 and U4, obtain the corresponding filter bank features, denoted as g. 2i (u i ) and y 2i Mel-frequency cepstral coefficients task: The feature sequences U corresponding to the data-enhanced speech and the original speech are denoted as U5 and U6, respectively. Based on U5 and U6, the corresponding Mel-frequency cepstral coefficients are obtained, denoted as g. 3i (u i ) and y 3i When the regression tasks are different, g(u i ) respectively correspond to g1(u i g2(u) i ), g3(u i ), y bi Corresponding to y 1i y 2i y 3i ;

[0025] The Local Information (LIM) identity prediction task and the Global Information (GIM) identity prediction task use the binary cross-entropy loss function to calculate the loss value, and the formula is as follows:

[0026]

[0027] in, The probabilistic labels y obtained from identity prediction in the Local Information Modeling (LIM) and Global Information Modeling (GIM) identity prediction tasks are given. i Label the category; if it is a positive sample y i =1, if it is a negative sample y i =0.

[0028] Furthermore, the first feature extraction network module is SincNet.

[0029] Furthermore, the second feature extraction network module is SCG_Res2Net50; SCG_Res2Net50 is a network obtained by applying the SCG mechanism to the output feature map of each residual block of the original Res2Net50.

[0030] Furthermore, the SCG mechanism is applied to the output feature map of each residual block in the original Res2Net50 as follows:

[0031] A channel gating unit is introduced in each channel of the feature map. Each channel gating unit consists of a sigmoid activation function, and its input is the feature map of the corresponding channel. The feature map is weighted at the channel level by performing element-wise multiplication on the output of the activation function.

[0032] A speaker recognition feature extraction system based on self-supervised multi-task learning includes:

[0033] Audio signal acquisition unit: used to acquire the speaker's audio;

[0034] Speech signal recognition feature extraction unit: For the collected speaker audio, a multi-task learning encoder is used to extract the feature sequence U;

[0035] The process of extracting the feature sequence U using a multi-task learning encoder includes the following steps:

[0036] Step 100: The audio is sent to the first feature extraction network module, and after processing by the first feature extraction network module, the speech feature sequence Z is obtained;

[0037] Step 200: Input the speech feature representation Z obtained in step 100 into the second feature extraction network module to obtain the speech feature sequence L; and input L into steps 300 and 400 simultaneously.

[0038] Step 300: The speech feature representation L obtained in step 200 is sent to the Transformer module, and after processing, the speech representation T is obtained;

[0039] Step 400: After fusing the speech feature sequence L and the T obtained by the Transformer module, the two sequences are fed into the fully connected layer and the normalization layer to obtain the feature sequence U.

[0040] The multi-task learning encoder is pre-trained, and the training process includes the following steps:

[0041] The original audio of the speaker is collected and data augmentation is performed on the original speech. A training set is constructed using the data augmented speech and the original speech. For the audio in the training set, the feature sequence U is first processed through steps 100 to 400. Then, the obtained feature sequence U is simultaneously sent to multiple processing tasks, including regression tasks and classification tasks. The regression tasks include the original waveform task, log power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task. The classification tasks include the Local Information Model (LIM) identity prediction task and the Global Information Model (GIM) identity prediction task.

[0042] The original waveform task is: based on the feature sequence U, predict the reconstructed waveform using an autoencoder; and obtain the loss between the reconstructed waveform and the original speech signal waveform.

[0043] The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task are as follows: For the data-enhanced speech and the original speech, based on the feature sequence U, the corresponding logarithmic power spectrum, filter bank features, and Mel frequency cepstral coefficients are obtained respectively; The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task each obtain the Mel frequency cepstral coefficient loss of the data-enhanced speech and the original speech.

[0044] Local Information Modeling (LIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: an anchor point and a positive sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. n It is the feature sequence U corresponding to the positive sample obtained based on the sampling strategy, and x a and x n As input to the classifier corresponding to the LIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained;

[0045] Global Information Modeling (GIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: the anchor point and the negative sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. p It is the average value of the feature sequence U corresponding to all samples in the 1-second speech block where the negative sample is obtained based on the sampling strategy. a and x p As input to the classifier corresponding to the GIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained;

[0046] The sampling strategy mentioned above refers to the following: samples from the same sentence as the anchor point are positive samples, and samples from different sentences are negative samples;

[0047] The average of all loss values ​​is obtained based on the loss values ​​corresponding to multiple processing tasks, and used for backpropagation to obtain a multi-task learning encoder based on the training set.

[0048] Furthermore, the loss of the original waveform task is calculated using the Mean Absolute Error (MAE) function, as shown in the following formula:

[0049]

[0050] Where, f(u) i ) represents the waveform of the predicted speech signal obtained from the task, y i Represents the waveform of the original speech signal;

[0051] For the three regression tasks—log power spectrum, filter bank characteristics, and Mel frequency cepstral coefficients—the mean square error (MSE) function is used to calculate the loss value, and the formula is as follows:

[0052]

[0053] Logarithmic power spectrum task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U1 and U2, respectively. Obtain the corresponding logarithmic power spectra based on U1 and U2, denoted as g. 1i (u i ) and y 1i Filter bank feature task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U3 and U4, respectively. Based on U3 and U4, obtain the corresponding filter bank features, denoted as g. 2i (u i ) and y 2i Mel-frequency cepstral coefficients task: The feature sequences U corresponding to the data-enhanced speech and the original speech are denoted as U5 and U6, respectively. Based on U5 and U6, the corresponding Mel-frequency cepstral coefficients are obtained, denoted as g. 3i (u i ) and y 3iWhen the regression tasks are different, g(u i ) respectively correspond to g1(u i g2(u) i ), g3(u i ), y bi Corresponding to y 1i y 2i y 3i ;

[0054] The Local Information (LIM) identity prediction task and the Global Information (GIM) identity prediction task use the binary cross-entropy loss function to calculate the loss value, and the formula is as follows:

[0055]

[0056] in, The probabilistic labels y obtained from identity prediction in the Local Information Modeling (LIM) and Global Information Modeling (GIM) identity prediction tasks are given. i Label the category; if it is a positive sample y i =1, if it is a negative sample y i =0.

[0057] Furthermore, the first feature extraction network module is SincNet.

[0058] Furthermore, the second feature extraction network module is SCG_Res2Net50; SCG_Res2Net50 is a network obtained by applying the SCG mechanism to the output feature map of each residual block of the original Res2Net50.

[0059] Furthermore, the SCG mechanism is applied to the output feature map of each residual block in the original Res2Net50 as follows:

[0060] A channel gating unit is introduced in each channel of the feature map. Each channel gating unit consists of a sigmoid activation function, and its input is the feature map of the corresponding channel. The feature map is weighted at the channel level by performing element-wise multiplication on the output of the activation function.

[0061] Beneficial effects:

[0062] To address the high cost of creating labeled datasets, this invention employs self-supervised multi-task learning using an unlabeled dataset. This approach not only overcomes the high cost of labeled datasets but also extracts features with generality and versatility. Furthermore, to address the challenge of capturing complex and hierarchical speech information through a single self-supervised task, this invention utilizes multi-task learning to help the model learn more features and knowledge, thereby improving model performance and generalization ability. In this invention, each self-supervised task introduces different perspectives or soft constraints to the learned representations. Simultaneously, this invention can learn cross-task consensus and impose constraints on the learned representations, further enhancing model performance and generalization ability. Attached Figure Description

[0063] Figure 1 This is a flowchart of the present invention.

[0064] Figure 2 This is a diagram of the model architecture.

[0065] Figure 3 To compare the performance of this invention with that of other methods such as Disent, CDDL, GCL, i-vector, SiMCLR, and MoCo on the voxceleb1 dataset, the performance was evaluated using equal error rate (EER).

[0066] Figure 4 This is a line graph showing the loss reduction of the present invention. Detailed Implementation

[0067] Specific implementation method one: Combining Figures 1 to 2 This implementation method is described below.

[0068] This embodiment is a speaker recognition feature extraction method based on self-supervised multi-task learning, which includes the following steps:

[0069] Step 1: First, divide the input speech into 1-second speech blocks, discarding those less than 1 second, and record each sample in the speech block as the original speech.

[0070] The original speech is augmented with data. A training set is constructed using the augmented speech and the original speech; the dataset is unlabeled. Then, the augmented speech and the original speech are fed into the first feature extraction network module. In some embodiments, the SincNet language model module is used. After processing by the SincNet module, the speech feature sequence Z = {z1, z2, ..., z} is obtained. n}

[0071] Step 2: Feed the speech feature representation Z obtained in Step 1 into the second feature extraction network module. In some embodiments, the SCG_Res2Net50 module is used to obtain the speech feature sequence L = {l1, l2, ..., l n}; and simultaneously send L into steps 3 and 4.

[0072] SCG_Res2Net50 is an improvement upon Res2Net50, a convolutional neural network with multi-scale feature representation capabilities. The modification in SCG_Res2Net50 is the application of the SCG mechanism to the output feature map of each residual block in the original Res2Net50. This mechanism dynamically gates the spatial location of each channel, enhancing or suppressing channel activation values. This attention mechanism adaptively adjusts channel importance, further improving feature expressiveness and discriminative power. Its core idea is to introduce a channel gating unit on each channel of the feature map to adaptively adjust channel importance. Each channel gating unit consists of a sigmoid activation function, whose input is the feature map of the corresponding channel. By performing element-wise multiplication on the output of this activation function, channel-level weighting can be applied to the feature map. Therefore, SCG can adaptively adjust the importance of each channel in the acoustic features, improving the performance of speech recognition and speaker identification.

[0073] Step 3: Input the speech feature representation L obtained in Step 2 into the Transformer module. After processing, the speech representation T = {t1, t2, ..., t} is obtained. n}

[0074] Step 4: After fusing the speech feature sequence L corresponding to the jump edge and T obtained by the Transformer module, the two sequences are fed into a fully connected layer and a normalization layer to obtain a feature sequence U = {u1, u2, ..., u...} n}, u1 is a 256-dimensional feature.

[0075] Step 5: The feature sequence U obtained in Step 4 is simultaneously fed into the six tasks. These six tasks will each calculate six loss values. This method will use the average of these six loss values ​​to guide the optimization of the parameters in the encoder section. These six tasks include four regression tasks: original waveform (Wave), log power spectrum (LPC), filter bank features (FBANK), and Mel frequency cepstral coefficients (MFCC); and two classification tasks: local information (LIM) identity prediction and global information (GIM) identity prediction. Each task will output a loss value. The average of the losses from all tasks is calculated and used for backpropagation to train the encoder.

[0076] Original waveform task: Based on the feature sequence U, predict the reconstructed waveform using an autoencoder;

[0077] The loss is calculated using the Mean Absolute Error (MAE) function, and the formula is as follows:

[0078]

[0079] Where, f(u) i ) represents the waveform of the predicted speech signal obtained from the task, y ai This represents the original speech signal waveform corresponding to the predicted speech signal waveform.

[0080] Logarithmic power spectrum task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U1 and U2, respectively. Obtain the corresponding logarithmic power spectra based on U1 and U2, denoted as g. 1i (u i ) and y 1i ;

[0081] Filter bank feature task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U3 and U4, respectively. Based on U3 and U4, obtain the corresponding filter bank features, denoted as g. 2i (u i ) and y 2i ;

[0082] Mel-frequency cepstral coefficient task: The feature sequences U corresponding to the data-enhanced speech and the original speech are denoted as U5 and U6, respectively. Based on U5 and U6, the corresponding Mel-frequency cepstral coefficients are obtained, denoted as g. 3i (u i ) and y 3i ;

[0083] For the three regression tasks—logarithmic power spectrum, filter bank characteristics, and Mel frequency cepstral coefficients—the mean square error (MSE) function is used to calculate the loss value, and the formula is as follows:

[0084]

[0085] Among them, when the regression tasks are different, g(u i ) respectively correspond to g1(u i g2(u) i ), g3(u i ), y bi Corresponding to y 1i y 2i y 3i .

[0086] Local Information (LIM) identity prediction and Global Information (GIM) identity prediction are as follows:

[0087] During the construction of the training set, it is necessary to ensure that the number of speakers is much greater than the number of speech samples collected by each speaker. For example, if each speaker collects 20 speech samples and the number of speakers is 50,000, then the total number of speech samples is 1 million. Samples belonging to the same sentence as the anchor point are positive samples (belonging to the same speaker), and samples belonging to different sentences are negative samples (i.e., the probability of belonging to the same speaker is extremely low, and they are considered to belong to different speakers).

[0088] Local Information Modeling (LIM) Identity Prediction: Based on a sampling strategy (i.e., samples from the same sentence as the anchor point are positive samples, and samples from different sentences are negative samples), two points are randomly sampled: the anchor point and the positive sample. a It is the feature sequence U, x corresponding to the anchor points in a random sentence. n It is the feature sequence U corresponding to the positive sample obtained based on the sampling strategy, and x a and x n As input to the classifier corresponding to the LIM task, identity prediction yields a probability label (this probability represents the probability that the input sample and the anchor point belong to the same speaker), which is the value in the loss function formula. In the loss function formula, y i Value: If it is a positive sample y i =1.

[0089] Global Information Modeling (GIM) Identity Prediction: Based on a sampling strategy, two points are randomly sampled: the anchor point and the negative sample. a It is the feature sequence U, x corresponding to the anchor points in a random sentence. p It is the average value of the feature sequence U corresponding to all samples in the 1-second speech block where the negative sample is obtained based on the sampling strategy. a and x p As input to the classifier corresponding to the GIM task, identity prediction yields a probability label, which is the value in the loss function formula. In the loss function formula, y i Value: If it is a negative sample y i =0;

[0090] For the two classification tasks, Local Information Modeling (LIM) identity prediction and Global Information Modeling (GIM) identity prediction, the binary cross-entropy loss function (BCE) is used to calculate the loss value, and the formula is as follows:

[0091]

[0092] Step 6: Repeat steps 1 to 5 until the maximum number of iterations is reached, yielding the best-performing model. Detailed process:

[0093] The speech features Z = {z1, z2, ..., z} obtained in step 1 are... n The data is sent to the SCG_Res2Net50 module for further feature extraction and analysis. The resulting speech features are L = {l1, l2, ..., l...} n The obtained speech features L = {l1, l2, ..., l} n The input consists of two branches. One branch is sent directly to the Transformer module for further analysis, while the other branch sends the obtained features to the fully connected layer via a skip edge, so that the final representation is the sum of the features extracted by the Transformer module and the SCG_Res2Net50 module, respectively.

[0094] The obtained speech feature sequence U is fed into a small feedforward neural network to calculate the loss value for the task. Each task outputs a loss value, and the average of the above losses is used for backpropagation.

[0095] In fact, the training process of the above six tasks is to train the network model of steps 1 to 4. In actual use, when performing speaker recognition, the original audio of the speaker is first collected, and then processed through steps 1 to 4 to obtain the feature sequence U. Then, the feature sequence U is input into any classifier to determine the corresponding speaker.

[0096] In the method proposed in this invention, data augmentation (noise addition) is first performed on the data to improve the robustness of the model. Then, an encoder and six different self-supervised tasks are used for processing, and the average loss of the six tasks ultimately guides the optimization of the encoder parameters. The encoder consists of a SincNet module, an SCG_Res2Net50 module, a Transformer module, fully connected layers, and normalization layers. The six tasks are: original waveform (Wave), logarithmic power spectrum (LPC), filter bank features (FBANK), and Mel-frequency cepstral coefficients (MFCC), local information LIM identity prediction, and global information GIM identity prediction. The SCG_Res2Net50 used in this invention excels at extracting speaker features and can extract subtle differences and speaker features; while the Transformer can capture semantic information when processing speech sequences, capturing long-range dependencies in the speech sequence. By combining the two, semantic and speaker information can be comprehensively utilized to improve the accuracy of speaker recognition. Furthermore, both have efficient parallel computing capabilities; combining them can fully utilize the advantages of parallel computing and improve the efficiency of training and inference. The speech features extracted using this method can be directly fed into models such as Fast ResNet-34 for speaker recognition. The dataset used in this method is unlabeled, which not only solves the problem of high costs associated with labeled datasets, but also ensures that the extracted features are general and universal. Specific Implementation Method Two:

[0098] This embodiment is a speaker recognition feature extraction system based on self-supervised multi-task learning. It is a computer program product or running program system corresponding to the speaker recognition feature extraction method based on self-supervised multi-task learning described in Specific Embodiment 1. Those skilled in the art should understand that the computer program product or running program system can take the form of a completely software embodiment or an embodiment combining software and hardware aspects. More specifically, the speaker recognition feature extraction system based on self-supervised multi-task learning described in this embodiment includes:

[0099] Audio signal acquisition unit: used to acquire the speaker's audio;

[0100] Speech signal recognition feature extraction unit: For the acquired speaker audio, a multi-task learning encoder is used to extract the feature sequence U; the extracted feature sequence U is used for speaker recognition. When performing speaker recognition, the feature sequence U is input into the classifier to identify and determine the speaker.

[0101] The process of extracting the feature sequence U using a multi-task learning encoder includes the following steps:

[0102] Step 100: The audio is sent to the first feature extraction network module, and after processing by the first feature extraction network module, the speech feature sequence Z is obtained; in some embodiments, the first feature extraction network module is SincNet.

[0103] Step 200: The speech feature representation Z obtained in step 100 is fed into the second feature extraction network module to obtain the speech feature sequence L; and L is simultaneously fed into steps 300 and 400; in some embodiments, the second feature extraction network module is SCG_Res2Net50; SCG_Res2Net50 is a network obtained by applying the SCG mechanism to the output feature map of each residual block of the original Res2Net50. The process of applying the SCG mechanism to the output feature map of each residual block of the original Res2Net50 is as follows:

[0104] A channel gating unit is introduced in each channel of the feature map. Each channel gating unit consists of a sigmoid activation function, and its input is the feature map of the corresponding channel. The feature map is weighted at the channel level by performing element-wise multiplication on the output of the activation function.

[0105] Step 300: The speech feature representation L obtained in step 200 is sent to the Transformer module, and after processing, the speech representation T is obtained;

[0106] Step 400: After fusing the speech feature sequence L and the T obtained by the Transformer module, the two sequences are fed into the fully connected layer and the normalization layer to obtain the feature sequence U.

[0107] The multi-task learning encoder is pre-trained, and the training process includes the following steps:

[0108] The original audio of the speaker is collected and data augmentation is performed on the original speech. A training set is constructed using the data augmented speech and the original speech. For the audio in the training set, the feature sequence U is first processed through steps 100 to 400. Then, the obtained feature sequence U is simultaneously sent to multiple processing tasks, including regression tasks and classification tasks. The regression tasks include the original waveform task, log power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task. The classification tasks include the Local Information Model (LIM) identity prediction task and the Global Information Model (GIM) identity prediction task.

[0109] The original waveform task is: based on the feature sequence U, predict the reconstructed waveform using an autoencoder; and obtain the loss between the reconstructed waveform and the original speech signal waveform.

[0110] The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task are as follows: For the data-enhanced speech and the original speech, based on the feature sequence U, the corresponding logarithmic power spectrum, filter bank features, and Mel frequency cepstral coefficients are obtained respectively; The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task each obtain the Mel frequency cepstral coefficient loss of the data-enhanced speech and the original speech.

[0111] Local Information Modeling (LIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: an anchor point and a positive sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. n It is the feature sequence U corresponding to the positive sample obtained based on the sampling strategy, and x a and x n As input to the classifier corresponding to the LIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained;

[0112] Global Information Modeling (GIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: the anchor point and the negative sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. p It is the average value of the feature sequence U corresponding to all samples in the 1-second speech block where the negative sample is obtained based on the sampling strategy. a and x p As input to the classifier corresponding to the GIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained;

[0113] The sampling strategy mentioned above refers to the following: samples from the same sentence as the anchor point are positive samples, and samples from different sentences are negative samples;

[0114] The average of all loss values ​​is obtained based on the loss values ​​corresponding to multiple processing tasks, and used for backpropagation to obtain a multi-task learning encoder based on the training set.

[0115] The loss of the original waveform task is calculated using the Mean Absolute Error (MAE) function, and the formula is as follows:

[0116]

[0117] Where, f(u) i ) represents the waveform of the predicted speech signal obtained from the task, y i Represents the waveform of the original speech signal;

[0118] For the three regression tasks—log power spectrum, filter bank characteristics, and Mel frequency cepstral coefficients—the mean square error (MSE) function is used to calculate the loss value, and the formula is as follows:

[0119]

[0120] Logarithmic power spectrum task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U1 and U2, respectively. Obtain the corresponding logarithmic power spectra based on U1 and U2, denoted as g. 1i (u i ) and y 1i Filter bank feature task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U3 and U4, respectively. Based on U3 and U4, obtain the corresponding filter bank features, denoted as g. 2i (u i ) and y 2i Mel-frequency cepstral coefficients task: The feature sequences U corresponding to the data-enhanced speech and the original speech are denoted as U5 and U6, respectively. Based on U5 and U6, the corresponding Mel-frequency cepstral coefficients are obtained, denoted as g. 3i (u i ) and y 3i When the regression tasks are different, g(u i ) respectively correspond to g1(u i g2(u) i ), g3(u i ), y bi Corresponding to y 1i y 2i y 3i ;

[0121] The Local Information (LIM) identity prediction task and the Global Information (GIM) identity prediction task use the binary cross-entropy loss function to calculate the loss value, and the formula is as follows:

[0122]

[0123] in, The probabilistic labels y obtained from identity prediction in the Local Information Modeling (LIM) and Global Information Modeling (GIM) identity prediction tasks are given. i Label the category; if it is a positive sample y i =1, if it is a negative sample y i =0.

[0124] Example:

[0125] This invention uses the speech dataset from voxceleb1 for performance verification. The performance evaluation metric is the equal error rate (EER), with a lower value indicating better performance.

[0126] The method of this invention achieves a minimum average loss of 3.9828 and a minimum equal error rate of 9.85% for speaker recognition tasks; this represents a significant performance improvement compared to Disent, CDDL, GCL, i-vector, SiMCLR, and MoCo on the voxceleb1 dataset. Figure 3 As shown, under the same conditions, the lowest error rate of the present invention is 9.85%. Compared with Disent, CDDL, GCL, i-vector, SiMCLR, and MoCo, the error rate of the present invention for speaker recognition tasks is reduced by 55.40%, 43.77%, 35.45%, 35.53%, 45.7%, and 34.81%, respectively. Figure 4 Used to measure the experimental performance of a model from another perspective, such as Figure 4 As shown, the lowest average loss of this invention is 3.9828. This invention uses backpropagation joint training to optimize the various parameters of the model, so that the parameters are optimized in a way that compromises the average loss of the six tasks. Therefore, the reduction in loss also reflects the continuous improvement of the model's performance.

[0127] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A speaker recognition feature extraction method based on self-supervised multi-task learning, wherein a feature sequence U is extracted from the acquired speaker audio using a multi-task learning encoder, and the feature sequence U is used for speaker recognition; characterized in that, The process of extracting the feature sequence U using a multi-task learning encoder includes the following steps: Step 100: The audio is sent to the first feature extraction network module, and after processing by the first feature extraction network module, the speech feature sequence Z is obtained; Step 200: Input the speech feature representation Z obtained in step 100 into the second feature extraction network module to obtain the speech feature sequence L; and input L into steps 300 and 400 simultaneously. Step 300: The speech feature representation L obtained in step 200 is sent to the Transformer module, and after processing, the speech representation T is obtained; Step 400: After fusing the speech feature sequence L and the T obtained by the Transformer module, the two sequences are fed into the fully connected layer and the normalization layer to obtain the feature sequence U. The multi-task learning encoder is pre-trained, and the training process includes the following steps: The original audio of the speaker is collected and data augmentation is performed on the original speech. A training set is constructed using the data augmented speech and the original speech. For the audio in the training set, the feature sequence U is first processed through steps 100 to 400. Then, the obtained feature sequence U is simultaneously sent to multiple processing tasks, including regression tasks and classification tasks. The regression tasks include the original waveform task, log power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task. The classification tasks include the Local Information Model (LIM) identity prediction task and the Global Information Model (GIM) identity prediction task. The original waveform task is: based on the feature sequence U, predict the reconstructed waveform using an autoencoder; and obtain the loss between the reconstructed waveform and the original speech signal waveform. The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task are as follows: For data-enhanced speech and original speech, based on the feature sequence U, the corresponding logarithmic power spectrum, filter bank features, and Mel frequency cepstral coefficients are obtained respectively; The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task each obtain the Mel frequency cepstral coefficient loss of the data-enhanced speech and the original speech. Local Information Modeling (LIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: an anchor point and a positive sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. n It is the feature sequence U corresponding to the positive samples obtained based on the sampling strategy, and x a and x n As input to the classifier corresponding to the LIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained; Global Information Modeling (GIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: the anchor point and the negative sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. p It is the average value of the feature sequence U corresponding to all samples in the 1-second speech block where the negative sample is obtained based on the sampling strategy. a and x p As input to the classifier corresponding to the GIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained; The sampling strategy mentioned above refers to the following: samples from the same sentence as the anchor point are positive samples, and samples from different sentences are negative samples; The average of all loss values ​​is obtained based on the loss values ​​corresponding to multiple processing tasks, and used for backpropagation to obtain a multi-task learning encoder based on the training set.

2. The speaker recognition feature extraction method based on self-supervised multi-task learning according to claim 1, characterized in that, The loss of the original waveform task is calculated using the Mean Absolute Error (MAE) function, and the formula is as follows: Where, f(u) i ) represents the waveform of the predicted speech signal obtained from the task, y i Represents the waveform of the original speech signal; For the three regression tasks—log power spectrum, filter bank characteristics, and Mel frequency cepstral coefficients—the mean square error (MSE) function is used to calculate the loss value, and the formula is as follows: Logarithmic power spectrum task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U1 and U2, respectively. Obtain the corresponding logarithmic power spectra based on U1 and U2, denoted as g. 1i (u i ) and y 1i Filter bank feature task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U3 and U4, respectively. Based on U3 and U4, obtain the corresponding filter bank features, denoted as g. 2i (u i ) and y 2i Mel-frequency cepstral coefficients task: The feature sequences U corresponding to the data-enhanced speech and the original speech are denoted as U5 and U6, respectively. Based on U5 and U6, the corresponding Mel-frequency cepstral coefficients are obtained, denoted as g. 3i (u i ) and y 3i When the regression tasks are different, g(u i ) respectively correspond to g1(u i g2(u) i ), g3(u i ), y bi Corresponding to y 1i y 2i y 3i ; The Local Information (LIM) identity prediction task and the Global Information (GIM) identity prediction task use the binary cross-entropy loss function to calculate the loss value, and the formula is as follows: in, The probabilistic labels y obtained from identity prediction in the Local Information Modeling (LIM) and Global Information Modeling (GIM) identity prediction tasks are given. i Label the category; if it is a positive sample y i =1, if it is a negative sample y i =0.

3. A speaker recognition feature extraction method based on self-supervised multi-task learning according to claim 1 or 2, characterized in that, The first feature extraction network module is SincNet.

4. The speaker recognition feature extraction method based on self-supervised multi-task learning according to claim 3, characterized in that, The second feature extraction network module is SCG_Res2Net50; SCG_Res2Net50 is a network obtained by applying the SCG mechanism to the output feature map of each residual block of the original Res2Net50.

5. The speaker recognition feature extraction method based on self-supervised multi-task learning according to claim 4, characterized in that, The process of applying the SCG mechanism to the output feature map of each residual block in the original Res2Net50 is as follows: A channel gating unit is introduced in each channel of the feature map. Each channel gating unit consists of a sigmoid activation function, and its input is the feature map of the corresponding channel. The feature map is weighted at the channel level by performing element-wise multiplication on the output of the activation function.

6. A speaker recognition feature extraction system based on self-supervised multi-task learning, characterized in that, include: Audio signal acquisition unit: used to acquire the speaker's audio; Speech signal recognition feature extraction unit: For the collected speaker audio, a multi-task learning encoder is used to extract the feature sequence U; The process of extracting the feature sequence U using a multi-task learning encoder includes the following steps: Step 100: The audio is sent to the first feature extraction network module, and after processing by the first feature extraction network module, the speech feature sequence Z is obtained; Step 200: Input the speech feature representation Z obtained in step 100 into the second feature extraction network module to obtain the speech feature sequence L; and input L into steps 300 and 400 simultaneously. Step 300: The speech feature representation L obtained in step 200 is sent to the Transformer module, and after processing, the speech representation T is obtained; Step 400: After fusing the speech feature sequence L and the T obtained by the Transformer module, the two sequences are fed into the fully connected layer and the normalization layer to obtain the feature sequence U. The multi-task learning encoder is pre-trained, and the training process includes the following steps: The original audio of the speaker is collected and data augmentation is performed on the original speech. A training set is constructed using the data augmented speech and the original speech. For the audio in the training set, the feature sequence U is first processed through steps 100 to 400. Then, the obtained feature sequence U is simultaneously sent to multiple processing tasks, including regression tasks and classification tasks. The regression tasks include the original waveform task, log power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task. The classification tasks include the Local Information Model (LIM) identity prediction task and the Global Information Model (GIM) identity prediction task. The original waveform task is: based on the feature sequence U, predict the reconstructed waveform using an autoencoder; and obtain the loss between the reconstructed waveform and the original speech signal waveform. The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task are as follows: For data-enhanced speech and original speech, based on the feature sequence U, the corresponding logarithmic power spectrum, filter bank features, and Mel frequency cepstral coefficients are obtained respectively; The logarithmic power spectrum task, filter bank feature task, and Mel frequency cepstral coefficient task each obtain the Mel frequency cepstral coefficient loss of the data-enhanced speech and the original speech. Local Information Modeling (LIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: an anchor point and a positive sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. n It is the feature sequence U corresponding to the positive samples obtained based on the sampling strategy, and x a and x n As input to the classifier corresponding to the LIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained; Global Information Modeling (GIM) Identity Prediction Task: Based on a sampling strategy, two points are randomly sampled: the anchor point and the negative sample. Let x... a It is the feature sequence U, x corresponding to the anchor points in a random sentence. p It is the average value of the feature sequence U corresponding to all samples in the 1-second speech block where the negative sample is obtained based on the sampling strategy. a and x p As input to the classifier corresponding to the GIM task, identity prediction is performed to obtain probability labels, and then the classification loss is obtained; The sampling strategy mentioned above refers to the following: samples from the same sentence as the anchor point are positive samples, and samples from different sentences are negative samples; The average of all loss values ​​is obtained based on the loss values ​​corresponding to multiple processing tasks, and used for backpropagation to obtain a multi-task learning encoder based on the training set.

7. The speaker recognition feature extraction system based on self-supervised multi-task learning according to claim 6, characterized in that, The loss of the original waveform task is calculated using the Mean Absolute Error (MAE) function, and the formula is as follows: Where, f(u) i ) represents the waveform of the predicted speech signal obtained from the task, y i Represents the waveform of the original speech signal; For the three regression tasks—log power spectrum, filter bank characteristics, and Mel frequency cepstral coefficients—the mean square error (MSE) function is used to calculate the loss value, and the formula is as follows: Logarithmic power spectrum task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U1 and U2, respectively. Obtain the corresponding logarithmic power spectra based on U1 and U2, denoted as g. 1i (u i ) and y 1i Filter bank feature task: Denote the feature sequences U corresponding to the data-enhanced speech and the original speech as U3 and U4, respectively. Based on U3 and U4, obtain the corresponding filter bank features, denoted as g. 2i (u i ) and y 2i Mel-frequency cepstral coefficients task: The feature sequences U corresponding to the data-enhanced speech and the original speech are denoted as U5 and U6, respectively. Based on U5 and U6, the corresponding Mel-frequency cepstral coefficients are obtained, denoted as g. 3i (u i ) and y 3i When the regression tasks are different, g(u i ) respectively correspond to g1(u i g2(u) i ), g3(u i ), y bi Corresponding to y 1i y 2i y 3i ; The Local Information (LIM) identity prediction task and the Global Information (GIM) identity prediction task use the binary cross-entropy loss function to calculate the loss value, and the formula is as follows: in, The probabilistic labels y obtained from identity prediction in the Local Information Modeling (LIM) and Global Information Modeling (GIM) identity prediction tasks are given. i Label the category; if it is a positive sample y i =1, if it is a negative sample y i =0.

8. A speaker recognition feature extraction system based on self-supervised multi-task learning according to claim 6 or 7, characterized in that, The first feature extraction network module is SincNet.

9. A speaker recognition feature extraction system based on self-supervised multi-task learning according to claim 8, characterized in that, The second feature extraction network module is SCG_Res2Net50; SCG_Res2Net50 is a network obtained by applying the SCG mechanism to the output feature map of each residual block of the original Res2Net50.

10. A speaker recognition feature extraction system based on self-supervised multi-task learning according to claim 9, characterized in that, The process of applying the SCG mechanism to the output feature map of each residual block in the original Res2Net50 is as follows: A channel gating unit is introduced in each channel of the feature map. Each channel gating unit consists of a sigmoid activation function, and its input is the feature map of the corresponding channel. The feature map is weighted at the channel level by performing element-wise multiplication on the output of the activation function.

Images