A speaker recognition method, system and storage medium based on adaptive class boundary interval

Abstract

The present invention discloses a speaker recognition method, system and storage medium based on adaptive class boundary interval, which relates to the technical field of speaker recognition. The method is to solve the problem that the existing method assumes that there is no difference between different speakers, and the way of adjusting the class boundary interval is fixed and consistent, which leads to the inability to distinguish speakers with small differences. The method comprises: step 1, obtaining a speaker audio data set, and adding a label to the added audio data; step 2, preprocessing the audio data, and extracting FBank features from the preprocessed audio data as acoustic features; step 3, building a speaker recognition model based on deep learning, dividing the speaker audio samples into correct classification and incorrect classification, taking the probability that the sample belongs to a certain speaker as prior information, adaptively adjusting the class boundary interval, and building a loss function to train the model; step 4, inputting the audio data of the speaker to be tested into the trained speaker recognition model to determine whether it is the audio data of the same person.

Description

Technical Field

[0001] The present invention relates to the technical field of speaker recognition, and in particular to a speaker recognition method, system and storage medium based on adaptive class boundary interval. Background Art

[0002] Speech is one of the important carriers rich in human identity information. The technology of using speech for identity recognition is called speaker recognition, also known as voiceprint recognition. It is a type of biometric authentication technology. It is a technology that automatically recognizes the identity of a speaker based on the speech parameters in the speech waveform that reflect the speaker's physiological and behavioral characteristics. Speaker recognition has broad application prospects in the fields of personalized human-computer interaction, military struggle, information security and multimedia entertainment. For example, the identity of a visitor can be automatically determined by the speaker's voiceprint, thereby providing personalized and personal services. Although the research on voiceprint recognition has been going on for half a century, the existing voiceprint recognition model still has many difficulties and is far from meeting the society's requirements for its practical application. With the widespread application of deep learning, mainstream speaker recognition models are usually built on deep learning neural networks, and the quality of the speaker characteristics learned by the model plays an important role in the recognition task.

[0003] Fundamentally speaking, the more compact the speaker features are within a class and the more distinguishable they are between classes, the more separable they are. Traditional Softmax lacks the ability to enable the model to learn speaker features that are more compact within a class and distinguishable between classes. Therefore, researchers resorted to methods based on class boundary intervals, replacing the inner product in the traditional Softmax method with angular distance or cosine distance, while adding boundary intervals to it, thereby improving the features' compactness within a class and separability between classes.

[0004] However, existing speaker recognition methods based on class boundary intervals have some problems. They assume that there is no difference between different speakers and the way to adjust the class boundary intervals is fixed and consistent. This unreasonable assumption makes the system unable to distinguish speakers with smaller differences, which restricts the performance of the speaker recognition system. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] The existing method assumes that there is no difference between different speakers, and the way of adjusting the class boundary interval is fixed and consistent, resulting in the inability to distinguish speakers with smaller differences.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] The present invention provides a speaker recognition method based on adaptive class boundary interval, the method comprising the following steps:

[0009] Step 1: Get the speaker audio data set and add labels to the speaker audio data;

[0010] Step 2: preprocess the audio data, and extract FBank features from the preprocessed audio data as acoustic features;

[0011] Step 3: construct a speaker recognition model based on deep learning, classify speaker audio samples into correct classification and incorrect classification, use the probability that the sample belongs to a certain speaker as prior information, adaptively adjust the class boundary interval, and construct a loss function to train the speaker recognition model;

[0012] Step 4: Input the audio data of the speaker to be tested into the trained speaker recognition model to determine whether it is the audio data of the same person.

[0013] Furthermore, the pre-processing of the audio data in step 2 includes: pre-emphasis, frame division and windowing processing.

[0014] Further, the step 2 of extracting FBank features from the preprocessed audio data as acoustic features comprises the following steps:

[0015] Step 21, performing a fast Fourier transform (FFT) on each preprocessed audio frame to obtain a frequency spectrum, and taking a modulus of the frequency spectrum to obtain a power spectrum of each audio frame;

[0016] Step 22: Use a series of Mel filter banks to perform frequency conversion processing on the power spectrum obtained in step 21 to obtain Mel frequencies, and perform logarithm operation on the obtained Mel frequencies to obtain FBank features, and use the FBank features as the acoustic features of the audio data.

[0017] Furthermore, the loss function in step 3 is a cross loss function ACBS-Softmax, and its construction method is:

[0018] Step 31: Use indicator function and Divide speaker features into correct classification and misclassification:

[0019]

[0020]

[0021] In the formula, the speaker feature x n The acoustic features of FBank representing audio sample n are input into the model to obtain deep features, and j represents x n The category to which the speaker belongs, k represents other speaker categories, k≠j, and Respectively represent xn Cosine similarity with speaker categories j and k, x n When it can be correctly classified as the real speaker j, that is, hour, Compared to the real speaker j, x n When it is misclassified as speaker k, that is, hour,

[0022] Step 32: According to the indicator function Taking the probability that the correctly classified speaker features belong to the real speaker as prior information, the class boundary interval between the correct speaker and other speakers is adaptively adjusted:

[0023]

[0024]

[0025] Where s,u are hyper parameters, is a reweighting function that can be used to n The probability of being classified as the real speaker j Adaptively expand the class boundary interval between the real speaker and other speakers based on prior information;

[0026] Step 33: According to the indicator function The probability of misclassified speaker features being classified as the wrong speaker is used as prior information to adaptively adjust the class boundary interval between the wrong speaker and other speakers:

[0027]

[0028]

[0029] Where s, t are hyper parameters, is a reweighting function that can be used to n The probability of misclassification to speaker k Adaptively adjust the class boundary interval between speaker k and other speakers based on prior information;

[0030] Step 34: Construct the cross loss function ACBS-Softmax as:

[0031]

[0032] Where N is the total number of audio samples.

[0033] Furthermore, the construction of a speaker recognition model based on a deep learning network in step 3 specifically involves constructing a speaker recognition model based on ECAPA-TDNN.

[0034] Furthermore, the ECAPA-TDNN-based speaker recognition model includes:

[0035] The first layer is the convolutional layer;

[0036] The second to fourth layers are SE-Res2Block modules;

[0037] The fifth layer concats the outputs from the second to fourth layers together;

[0038] The sixth layer is the convolutional layer;

[0039] The seventh layer is the statistical pooling layer;

[0040] The eighth layer is the fully connected layer.

[0041] A speaker recognition system based on adaptive class boundary intervals, the system has a program module corresponding to the steps of any one of the above technical solutions, and executes the steps of the above speaker recognition method based on adaptive class boundary intervals when running.

[0042] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of the speaker recognition method based on adaptive class boundary interval described in any one of the above technical solutions when called by a processor.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention discloses a speaker recognition method, system and storage medium based on adaptive class boundary interval. The speaker recognition method based on adaptive class boundary interval divides speaker audio samples into two categories: correctly classified and incorrectly classified. The probability that the audio sample belongs to a certain speaker is used as prior information to adaptively adjust the class boundary interval, and is committed to making the correctly classified features more compact and the incorrectly classified features closer to the correctly classified features, thereby improving the intra-class compactness and inter-class separability of the features, and enhancing the model's recognition ability for speakers with different differences. Compared with the traditional speaker recognition method based on class boundary interval, the method of the present invention is more accurate in recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a speaker recognition method based on adaptive class boundary interval in an embodiment of the present invention;

[0046] Figure 2 It is a functional block diagram of a speaker recognition model based on ECAPA-TDNN in an embodiment of the present invention;

[0047] Figure 3It is a structural diagram of the ECAPA-TDNN deep learning network in an embodiment of the present invention;

[0048] Figure 4 Flow chart of the ACBS-Softmax method in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In the description of the present invention, it should be noted that the terms "first", "second", and "third" mentioned in the embodiments of the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0051] Specific implementation plan 1: Figure 1 As shown, the present invention provides a speaker recognition method based on adaptive class boundary interval, a speaker recognition method based on adaptive class boundary interval, characterized in that: the specific process of the method is:

[0052] Step 1: Obtain a speaker audio data set, number the speakers, determine the speaker number to which the audio data belongs, and use this as the label of the audio data;

[0053] Step 2: preprocess the audio data, and extract FBank features from the preprocessed audio data as acoustic features;

[0054] Step 3: construct a speaker recognition model based on deep learning, classify speaker audio samples into correct classification and incorrect classification, use the probability that the sample belongs to a certain speaker as prior information, adaptively adjust the class boundary interval, and construct a loss function to train the speaker recognition model;

[0055] Step 4: Input the audio data of the speaker to be tested into the trained speaker recognition model to determine whether it is the audio data of the same person.

[0056] Specific implementation scheme 2: The pre-processing of the audio data described in step 2 includes: pre-emphasis, framing and windowing. The rest of this implementation scheme is the same as the specific implementation scheme 1.

[0057] Specific implementation scheme three: Extracting FBank features from preprocessed audio data as acoustic features as described in step 2 includes the following steps:

[0058] Step 21, performing a fast Fourier transform (FFT) on each preprocessed audio frame to obtain a frequency spectrum, and taking a modulus of the frequency spectrum to obtain a power spectrum of each audio frame;

[0059] Step 22: Use a series of Mel filter banks to perform frequency conversion processing on the power spectrum obtained in step 21 to obtain Mel frequencies, and perform logarithmic operation on the obtained Mel frequencies to obtain FBank features, and use the FBank features as the acoustic features of the audio data. The rest of this implementation is the same as the specific implementation plan one.

[0060] Specific implementation plan 4: Figure 4 As shown, the loss function in step 3 is the cross loss function ACBS-Softmax, and its construction method is:

[0061] Step 31: Use indicator function and Divide speaker features into correct classification and misclassification:

[0062]

[0063]

[0064] In the formula, the speaker feature x n The acoustic features of FBank representing audio sample n are input into the model to obtain deep features, and j represents x n The category to which the speaker belongs, k represents other speaker categories, k≠j, and Respectively represent x n Cosine similarity with speaker categories j and k; x n When it can be correctly classified as the real speaker j, that is, hour, Compared to the real speaker j, x n When it is misclassified as speaker k, that is, hour,

[0065] Step 32: According to the indicator function Taking the probability that the correctly classified speaker features belong to the real speaker as prior information, the class boundary interval between the correct speaker and other speakers is adaptively adjusted:

[0066]

[0067]

[0068] Where s,u are hyper parameters, is a reweighting function that can be used to nThe probability of being classified as the real speaker j Adaptively expand the class boundary interval between the real speaker and other speakers based on prior information;

[0069] Step 33: According to the indicator function The probability of misclassified speaker features being classified as the wrong speaker is used as prior information to adaptively adjust the class boundary interval between the wrong speaker and other speakers:

[0070]

[0071]

[0072] Where s, t are hyper parameters, is a reweighting function that can be used to n Probability of misclassifying to a speaker Adaptively adjust the class boundary interval between a speaker and other speakers based on prior information;

[0073] Step 34: Construct the cross loss function ACBS-Softmax as:

[0074]

[0075] Wherein, N is the total number of audio samples. The rest of this implementation is the same as the first specific implementation.

[0076] Specific implementation plan five: Figure 2 As shown, the construction of the speaker recognition model based on the deep learning network in step 3 is specifically to construct a speaker recognition model based on ECAPA-TDNN. The rest of this implementation is the same as the specific implementation plan 1.

[0077] Specific implementation plan six: Figure 3 As shown, the ECAPA-TDNN-based speaker recognition model includes:

[0078] The first layer is the convolutional layer;

[0079] The second to fourth layers are SE-Res2Block modules;

[0080] The fifth layer concats the outputs from the second to fourth layers together;

[0081] The sixth layer is the convolutional layer;

[0082] The seventh layer is the statistical pooling layer;

[0083] The eighth layer is the fully connected layer.

[0084] The rest of this implementation plan is the same as the specific implementation plan five.

[0085] Step 3 specifically includes the following process:

[0086] Step 3-1: Construct a training set based on the audio data labels and FBank acoustic features;

[0087] Step 3-2: construct a speaker recognition model based on ECAPA-TDNN, input the labels of the audio data in the training set and the FBank acoustic features into the speaker recognition model based on ECAPA-TDNN to obtain deep features;

[0088] Step 3-3: Classify the speaker audio samples into correct classification and incorrect classification, use the probability that the sample belongs to a certain speaker as prior information, adaptively adjust the class boundary interval, construct the cross loss function ACBS-Softmax, and continuously update the deep learning network parameters to minimize the loss function;

[0089] Step 3-4: Use the Adam algorithm for optimization and continuously update the deep learning network parameters to minimize the loss function to obtain the trained speaker recognition model.

[0090] Specific implementation scheme seven: A speaker recognition system based on adaptive class boundary interval, the system has a program module corresponding to the steps of any one of the above specific implementation schemes, and executes the steps of the above speaker recognition method based on adaptive class boundary interval during operation.

[0091] Specific implementation scheme eight: A computer-readable storage medium, the computer-readable storage medium storing a computer program, the computer program being configured to implement the steps of the speaker recognition method based on adaptive class boundary interval described in any one of the above-mentioned specific implementation schemes when called by a processor.

[0092] Embodiment 1:

[0093] In order to verify the effect of the method of the present invention, the method of the present invention is described by using the following examples.

[0094] Based on the existing VoxCeleb1 dataset, a comparative experiment was conducted to compare the method of the present invention with multiple existing methods. The VoxCeleb1 dataset contains more than 100,000 audio clips of 1,251 speakers extracted from YouTube videos. The speakers have different ages, nationalities, and occupations. The number of men and women in the dataset is almost balanced, with 55% of the speakers being male, and the speakers have different accents, occupations, and ages. As shown in Table 1, the VoxCeleb1 dataset is divided into a training set and a test set, where all the audio clips of the 40 speakers with ids between id10270 and id10309 are divided into the test set, so the speakers in the training set and the test set are disjoint to increase the difficulty of recognition.

[0095] Table 1

[0096]

[0097] In the training phase, the VoxCeleb1 training set is preprocessed. Each audio segment in the VoxCeleb1 training set is randomly cut into 2-second audio segments, and the audio segments are randomly noised using the RIRS dataset and the MUSAN dataset. In the testing phase, the VoxCeleb1 test set is also preprocessed. Each audio segment in the VoxCeleb1 test set is randomly cut into 2-second audio segments, but no noise is added. The experimental parameter settings are shown in Table 2.

[0098] Table 2

[0099]

[0100] Table 3 shows the performance of each speaker recognition model on the VoxCeleb1 test set. ACBS-Speaker represents the method of the present invention, A-Speaker represents the combination of A-Softmax and ECAPA-TDNN, AM-Speaker represents the combination of AM-Softmax and ECAPA-TDNN, AAM-Speaker represents the combination of AAM-Softmax and ECAPA-TDNN, and MV-Speaker represents the combination of MV-Softmax and ECAPA-TDNN.

[0101] Table 3

[0102]

[0103]

[0104] From the experimental results in Table 3, it can be seen that the performance of the method of the present invention is greatly improved compared with the existing A-Speaker method. Compared with the AM-Speaker method, the EER is reduced by 8.1%; compared with the AAM-Speaker method, the EER is reduced by 4.8%; compared with the MV-Speaker method, the EER is reduced by 1.8%, which fully verifies that the method of the present invention divides the speaker samples into two categories, namely, correctly classified and incorrectly classified, takes the probability that the sample belongs to a certain speaker as prior information, and adaptively adjusts the class boundary interval, so that the model can learn the features of stronger intra-class compactness and inter-class separability, and the model has a stronger recognition ability for speakers with different differences.

[0105] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A speaker recognition method based on adaptive class boundary interval, characterized by: The method comprises the following steps: Step 1: Get the speaker audio data set and add labels to the speaker audio data; Step 2: preprocess the audio data, and extract FBank features from the preprocessed audio data as acoustic features; Step 3: construct a speaker recognition model based on deep learning, classify speaker audio samples into correct classification and incorrect classification, use the probability that the sample belongs to a certain speaker as prior information, adaptively adjust the class boundary interval, and construct a loss function to train the speaker recognition model; Step 4: Input the audio data of the speaker to be tested into the trained speaker recognition model to determine whether it is the audio data of the same person; The loss function in step 3 is the cross loss function ACBS-Softmax, which is constructed as follows: Step 31: Use indicator function and Divide speaker features into correct classification and misclassification: In the formula, the speaker feature x n The acoustic features of FBank representing audio sample n are input into the model to obtain deep features, and j represents x n The category to which the speaker belongs, k represents other speaker categories, k≠j, and Respectively represent x n Cosine similarity with speaker categories j and k, x n When it can be correctly classified as the real speaker j, that is, hour, Compared to the real speaker j, x n When it is misclassified as speaker k, that is, hour, Step 32: According to the indicator function Taking the probability that the correctly classified speaker features belong to the real speaker as prior information, the class boundary interval between the correct speaker and other speakers is adaptively adjusted: In the formula, s, u are hyper parameters, is a reweighting function that can be used to n The probability of being classified as the real speaker j Adaptively expand the class boundary interval between the real speaker and other speakers based on prior information; Step 33: According to the indicator function The probability of misclassified speaker features being classified as the wrong speaker is used as prior information to adaptively adjust the class boundary interval between the wrong speaker and other speakers: Where s, t, are hyperparameters, is a reweighting function that can be used to classify the probability of a feature being misclassified as a speaker. Adaptively adjust the class boundary interval between a speaker and other speakers based on prior information; Step 34: Construct the cross loss function ACBS-Softmax as: Where, is the total number of audio samples.

2. The speaker recognition method based on adaptive class boundary interval according to claim 1, characterized in that: The pre-processing of the audio data described in step 2 includes: pre-emphasis, frame division and windowing processing.

3. The speaker recognition method based on adaptive class boundary interval according to claim 1, characterized in that: Extract the FBank features from the preprocessed audio data as acoustic features as described in step 2. The steps include: Step 21, performing a fast Fourier transform (FFT) on each preprocessed audio frame to obtain a frequency spectrum, and taking a modulus of the frequency spectrum to obtain a power spectrum of each audio frame; Step 22: Use a series of Mel filter banks to perform frequency conversion processing on the power spectrum obtained in step 21 to obtain Mel frequencies, and perform logarithm operation on the obtained Mel frequencies to obtain FBank features, and use the FBank features as the acoustic features of the audio data.

4. The speaker recognition method based on adaptive class boundary interval according to claim 1, characterized in that: The step 3 of constructing a speaker recognition model based on a deep learning network is specifically to construct a speaker recognition model based on ECAPA-TDNN.

5. The speaker recognition method based on adaptive class boundary interval according to claim 4, characterized in that: The ECAPA-TDNN-based speaker recognition model includes: The first layer is the convolutional layer; The second to fourth layers are SE-Res2Block modules; The fifth layer concats the outputs from the second to fourth layers together; The sixth layer is the convolutional layer; The seventh layer is the statistical pooling layer; The eighth layer is the fully connected layer.

6. A speaker recognition system based on adaptive class boundary interval, characterized in that: The system has a program module corresponding to the steps of any one of claims 1 to 5, and executes the steps of the speaker recognition method based on adaptive class boundary interval when running.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps in the speaker recognition method based on adaptive class boundary interval according to any one of claims 1 to 5 when called by a processor.

Images