Speaker recognition method and apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose a speaker recognition method, comprising: acquiring audio data output by a speaker to be recognized; performing frame division and feature extraction on the audio data by using a trained speaker recognition model to obtain a feature vector of the audio data; and performing optimization and dimension reduction on the feature vector by using the speaker recognition model to obtain an identity vector of the speaker to be recognized; wherein the identity vector is used to identify the identity of the speaker to be recognized; thus, feature extraction on the audio data to be recognized and identification of the identity of the speaker to be recognized are realized in an end-to-end manner, priori information is avoided to fix the frequency resolution of different frequency bands, and the robustness of the recognition system is improved. The embodiments of the present application also simultaneously provide a speaker recognition device, equipment and storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic devices, and relates to, but is not limited to, a speaker recognition method and device, equipment, and a storage medium. BACKGROUND

[0002] Speaker recognition (SRE) is a technology for automatically identifying the identity of a speaker according to information in a speech signal. According to different tasks, speaker recognition can be divided into speaker identification, speaker verification, speaker tracking, and the like. The speaker recognition technology is widely used in different fields such as the military, government, and finance because of its characteristics of easy-to-implement human-computer interaction, remote identity verification, and attack prevention.

[0003] In an existing speaker recognition system, the model training part based on an X-vector embedding layer mainly includes the following parts: a feature extraction part: short-time Fourier transform (STFT) is used to perform corresponding processing on the time-domain waveform of audio to obtain Mel frequency cepstral coefficients (MFCC), filter bank (FBANK), pitch (PITCH), linear prediction cepstral coefficients (LPCC), constant-Q transform (CQT), and the like, and the obtained features will be used as the input of a neural network model; a model network part: the network structure of the model mainly includes a frame-level layer, a pooling layer, and a segment-level layer. In testing, the extracted frequency domain features are used as the input of the neural network model, and a one-hot vector represented by the identity information of a speaker is output.

[0004] In the current method, the frequency domain features are obtained by processing based on short-time Fourier transform, and the time-domain window length of the short-time Fourier transform is fixed, so the corresponding frequency domain resolution is also fixed. There is also a frequency domain transform method based on constant-Q transform in the current method, which uses a window of indefinite length, and according to the characteristics that the energy of low frequencies is high and the energy of high frequencies is low, the frequency spectrum after constant-Q transform has a relatively high frequency resolution in low frequencies and a relatively high time resolution in high frequencies. However, this method is also based on certain prior information, and the determination of the sampling rate determines that there is a consistent processing method in different frequency bands. SUMMARY

[0005] The embodiment of the present application provides a speaker recognition method and device, equipment and storage medium, and at least solves the problem that the frequency resolution of different frequency bands is fixed by using prior information, and the speaker recognition system is poor in robustness.

[0006] The technical scheme of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, the embodiment of the present application provides a speaker recognition method, and the method comprises the following steps:

[0008] obtaining audio data output by a speaker to be recognized;

[0009] performing frame division and feature extraction on the audio data by using a trained speaker recognition model, to obtain a feature vector of the audio data;

[0010] performing optimization and dimension reduction on the feature vector by using the speaker recognition model, to obtain an identity vector of the speaker to be recognized; wherein the identity vector is used for discriminating the identity of the speaker to be recognized.

[0011] In a second aspect, the embodiment of the present application provides a speaker recognition device, comprising a sample acquisition module, a feature extraction module and an optimization and dimension reduction module, wherein:

[0012] The sample acquisition module is configured to acquire audio data output by a speaker to be recognized.

[0013] The feature extraction module is configured to perform frame division and feature extraction on the audio data by using a trained speaker recognition model, to obtain a feature vector of the audio data.

[0014] The optimization and dimension reduction module is configured to perform optimization and dimension reduction on the feature vector by using the speaker recognition model, to obtain an identity vector of the speaker to be recognized; wherein the identity vector is used for discriminating the identity of the speaker to be recognized.

[0015] In a third aspect, the embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor implements the steps in the above speaker recognition method when executing the program.

[0016] In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above speaker recognition method.

[0017] The technical scheme provided by the embodiment of the present application has at least the following beneficial effects:

[0018] In the embodiment of the present application, first, audio data output by a to-be-identified speaker is acquired; then, a trained speaker identification model is used to frame and extract features of the audio data, to obtain a feature vector of the audio data; finally, the speaker identification model is used to optimize and reduce dimensions of the feature vector, to obtain an identity vector of the to-be-identified speaker; wherein the identity vector is used to identify the identity of the to-be-identified speaker; in this way, the pre-trained speaker identification model is used to complete framing and feature extraction of the original audio data, and further optimize and reduce dimensions of the extracted feature vector, to obtain an identity vector representing the identity of the to-be-identified speaker. In this way, feature extraction of the to-be-identified audio data and identification of the identity of the to-be-identified speaker are realized in an end-to-end manner, avoiding using short-time Fourier transform to process the original audio data to obtain frequency domain features, thereby solving the problem that prior information is used to fix the frequency resolution of different frequency bands, and making the recognition system less robust. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A flowchart of a speaker identification method provided by an embodiment of the present application;

[0021] Figure 2 A flowchart of a speaker identification method provided by an embodiment of the present application;

[0022] Figure 3 A flowchart of a speaker identification method provided by an embodiment of the present application;

[0023] Figure 4 A flowchart of a speaker identification method provided by an embodiment of the present application;

[0024] Figure 5A A model structure diagram of a speaker identification method provided by an embodiment of the present application;

[0025] Figure 5B A system block diagram of a speaker identification method provided by an embodiment of the present application;

[0026] Figure 5C A flowchart of a pre-training method based on data enhancement provided by an embodiment of the present application;

[0027] Figure 5D A schematic diagram of a pre-training process provided by an embodiment of the present application;

[0028] Figure 6 A component structure schematic diagram of a speaker recognition device provided by an embodiment of the present application is shown in FIG. 1.

[0029] Figure 7 A hardware entity schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application but not all the embodiments. The following embodiments are used to describe the present application but not to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0031] In the following description, “some embodiments” are related to a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0032] It should be noted that the terms “first\second\third” involved in the embodiments of the present application are only to distinguish similar objects and do not represent a specific order of the objects. Understandably, “first\second\third” can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0033] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the embodiments of the present application belong. It should also be understood that terms such as those defined in a general dictionary should be understood as having a meaning consistent with that in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0034] Speech is the most direct and convenient way for human beings to communicate. It has attracted the attention of various research institutions due to its unique advantages such as convenience, economy, accuracy, etc. The research on speech signal processing is of great significance to promote human-computer interaction and the development of artificial intelligence. Therefore, the related fields of speech signal processing, such as speech recognition, speech coding, speech synthesis, speaker recognition, etc. are receiving more and more attention and theoretical research.

[0035] Speaker recognition, also known as voiceprint recognition, is a technology that aims to authenticate the identity of each speaker based on their unique pronunciation. Each speaker's voice has unique personal characteristics, which are due to the differences in their natural vocal organs and the unique voice they have developed through environmental factors. Due to these differences, voice has become a biometric characteristic for recognition, and speaker recognition has gradually developed its own relatively complete recognition system.

[0036] Speaker recognition and speech recognition technologies are similar in that they both extract certain feature parameters from the original speech signal, establish corresponding reference templates or models, and then perform recognition according to certain decision rules. In speech recognition, the differences in speech between different people are normalized as much as possible; in speaker recognition, the semantic information in the speech signal is averaged to extract the personal factors of the speaker contained in the speech signal, and the feature differences between different people are emphasized.

[0037] A speaker recognition system includes a pre-processing part, a feature extraction part, a model training and matching calculation part. The pre-processing part is the pre-processing and feature extraction of the speech signal, i.e., extracting parameters that can represent the characteristics of the speaker; the model sequence part includes the establishment of the speaker model and the training of the model parameters; the matching calculation part performs matching calculation between the test speech and the speaker model. It can be seen that the key technologies of speaker recognition include feature parameter extraction algorithm, model selection, and model matching algorithm, which directly determine the performance of the recognition system.

[0038] Speaker recognition models are divided into generative models and discriminative models. The generative model learns the features of each class, i.e., multiple models, and maps the recognition data to each model to determine which class the recognition data belongs to; the discriminative model learns the classification surface, which can be used to distinguish which class different data belongs to. These two models, represented by the i-vector based on the global difference model (Total Variability Modeling, TVM) and the x-vector based on the time-delay neural network (Time-delay Deep Neural Network, TDNN), are the two most widely used vector models.

[0039] The most advanced speaker recognition system is currently based on the X-vector embedding framework. X-vector is the current mainstream model framework in the field of voiceprint recognition. Due to the statistical pooling layer in its network, X-vector can accept inputs of any length and convert them into fixed-length feature representations; in addition, data augmentation strategies including noise and reverberation are introduced in the training, making the model more robust to noise and reverberation interference.

[0040] The X vector includes a multi-layer frame-level TDNN, a statistical pooling layer, two sentence-level fully connected layers, and a normalization layer (softmax), and a loss function is a cross-entropy error (CE). The total input of the TDNN is a voice, and the TDNN takes a fixed number of frames each time. Then, the pooling layer accumulates the output vectors of each TDNN, calculates a mean and a standard deviation as the output of the pooling layer, that is, the TDNN outputs of all frames of the input audio sequence are averaged and the standard deviation is calculated, and then the two are spliced to output a sentence-level feature expression. After the pooling layer, two fully connected layers are followed by a normalization layer. The number of neurons in the output of the normalization layer is consistent with the number of speakers in the training set, and it can be seen that the output is a posterior probability.

[0041] TDNN: From the structure, each layer of the TDNN is still a deep neural network (DNN), but the input of each layer is spliced by historical, current and future features, thereby introducing time sequence information; the advantage of the TDNN structure is that it can be trained in parallel relative to a long short-term memory (LSTM), and the time sequence context information is increased relative to the DNN.

[0042] The speaker recognition system mainly includes two stages: a training stage and a recognition stage. In the training stage, according to the training corpus of each speaker in the speaker set, the feature extraction is performed, and the templates or models of the speakers are established. In the recognition stage, the voice spoken by the speaker to be recognized is also subjected to feature extraction, and is compared with the templates or models generated during the training of the system. In speaker recognition, the speaker corresponding to the model with the maximum predicted voice similarity is taken as the recognition result; in speaker verification, whether the similarity between the test voice and the model of the claimed speaker is greater than a certain threshold is judged to make a judgment.

[0043] Embodiments of the present application provide a speaker recognition method applied to an electronic device. The electronic device includes but is not limited to a mobile phone, a notebook computer, a tablet computer, a palm computer, a multimedia device, a streaming media device, a mobile Internet device, a wearable device, or other types of electronic devices. The functions implemented by the method can be realized by calling program codes in the processor of the electronic device, and of course the program codes can be saved in a computer storage medium. Therefore, the electronic device at least includes a processor and a storage medium. The processor can be used for processing the process of speaker recognition, and the storage can be used for storing the data required in the process of speaker recognition and the generated data.

[0044] Figure 1A flowchart of a speaker recognition method provided by an embodiment of the present application is shown in FIG. 1, which includes at least the following steps: Figure 1

[0045] In step S110, audio data output by a speaker to be recognized is acquired.

[0046] Here, real-time audio data output by at least one speaker to be recognized through an acoustic path can be collected, or at least one piece of stored audio data can be acquired from a stored corpus; the audio collection can be collected through recording.

[0047] In step S120, a trained speaker recognition model is used to frame and extract features from the audio data, to obtain a feature vector of the audio data.

[0048] Here, since the speech is produced by the movement of the vocal organs, the speech signal is sequentially changed, and thus the acquired audio data needs to be divided into 20 to 30 milliseconds (ms) of signals, and in each segment, it is assumed that the signal is continuously stable, and each segment is called a frame.

[0049] The speech feature extraction is to extract the speech features and vocal tract characteristics of the speaker. The difference between different speakers mainly lies in the difference in their short-time speech spectrum. The current speaker system converts the time-domain signal to the frequency domain, and uses the short-time Fourier transform, which has the characteristic that all frequencies share the same frequency resolution because the time-domain window is fixed. At the same time, the short-time spectrum features extracted by this method mainly extract mainstream feature parameters such as MFCC and LPCC, which are mainly single features, and the information representing the personality of the speaker is insufficient, which affects the recognition accuracy. The neural network feature extraction method of the present application can learn more information, and the finally extracted features are more abstract and have better speaker representation ability.

[0050] In step S130, the feature vector is optimized and reduced in dimension by using the speaker recognition model, to obtain an identity vector of the speaker to be recognized.

[0051] Here, the process of optimizing and reducing the dimension of the feature vector by using the speaker recognition model can be realized by an X-vector-based framework.

[0052] The process of optimizing and reducing the dimension can be to optimize the regularity between different frames of the feature vector extracted by the neural network, optimize the relationship between the features within the frame, and map the feature vectors of different frames to a fixed dimension vector, and finally connect them into a sentence-level vector.

[0053] ​The identity vector is used to identify the identity of the speaker to be identified, for example, the identity vector is input into a back-end scoring model, and the similarity between the identity vector and the registered speaker feature vector is determined to identify the identity of the speaker to be identified. The mapping vector (embedding) output before the last normalization layer in the X-vector-based framework is usually extracted as the identity vector for subsequent identification.

[0054] Exemplarily, the similarity between the identity vector of the real-time audio data and the identity vector of the registered audio can be calculated by a general scoring method such as Probabilistic Linear Discriminative Analysis (PLDA), and the speaker recognition result is determined according to the size of the similarity and the set similarity threshold.

[0055] In the embodiment of the present application, first, audio data output by a speaker to be identified is obtained; then, a trained speaker recognition model is used to frame and extract features of the audio data to obtain a feature vector of the audio data; finally, the speaker recognition model is used to optimize and reduce the dimension of the feature vector to obtain an identity vector of the speaker to be identified; wherein the identity vector is used to identify the identity of the speaker to be identified; in this way, the framing and feature extraction of the original audio data are completed by the pre-trained speaker recognition model, and the extracted feature vector is further optimized and reduced in dimension to obtain an identity vector representing the identity of the speaker to be identified. In this way, the feature extraction of the audio data to be identified and the identification of the speaker to be identified are realized in an end-to-end manner, avoiding the use of short-time Fourier transform to process the original audio data to obtain frequency domain features, thereby solving the problem of using prior information to fix the frequency resolution of different frequency bands, making the recognition system less robust.

[0056] In some possible embodiments, the trained speaker recognition model includes a feature learning network and an optimization and dimension reduction network, Figure 2 A flowchart of a speaker recognition method provided by the embodiment of the present application is shown in Figure 2 The method includes at least the following steps:

[0057] Step S210, obtaining audio data output by a speaker to be identified;

[0058] Step S220, using a feature learning network to frame the audio data to obtain at least two frames of audio;

[0059] Here, the framing process is to segment the audio data according to a specified length (time period or number of samples) to structure it into a stable data structure.

[0060] The implementation manner of frame dividing the audio data by using the feature learning network can be any manner possible in the related art, for example, by windowing and setting a frame shift, and the embodiments of the present application do not limit this.

[0061] In step S230, a feature learning network is used to extract features from each frame of the audio according to a specific dilation scale, to obtain a feature vector of the audio data.

[0062] Here, the feature learning network is a network in a trained speaker recognition model, that is, the embodiments of the present application learn features with speaker representation ability by using a neural network, so as to realize direct feature extraction on original speech data in an end-to-end manner.

[0063] The specific dilation scale represents a step size used when extracting features, and different dilation scales reflect different receptive fields. Generally, the larger the receptive field, the more abstract the extracted features. Layers with different dilation scales are set in the feature learning network, and the dilation scale of the layer gradually increases as the layer deepens.

[0064] In some possible embodiments, the above process of extracting features from each frame of the audio according to a specific dilation scale is implemented in the following manner: the feature learning network includes at least a first sub-layer and a second sub-layer; the first sub-layer is used to extract features from each frame of the audio according to a first dilation scale, to obtain a first vector; the second sub-layer is used to extract features from each frame of the audio according to a second dilation scale, to obtain a second vector; and the first vector and the second vector are connected to obtain the feature vector of the audio data.

[0065] Here, the first sub-layer and the second sub-layer are hidden layers in the feature learning network, and the second dilation scale corresponding to the second sub-layer is greater than the first dilation scale corresponding to the first sub-layer. The finally obtained feature vector is a feature vector obtained by splicing the feature vectors extracted by each sub-layer according to weights.

[0066] It should be noted that the hidden layer does not directly receive signals from the outside world, nor does it directly send signals to the outside world. The hidden layer in the neural network is an intermediate black box, which can be considered as a generalization of other different functional layers.

[0067] It is worth noting that the selection of the hidden layer in the feature learning network can be flexible, that is, the first sub-layer and the second sub-layer can be adjacent layers or can be separated by at least one sub-layer. That is, the feature learning network includes a plurality of hidden layers, and the second dilation scale is greater than the first dilation scale.

[0068] In some possible embodiments, the expansion scale of adjacent sub-layers in the feature learning network increases in the form of 2 raised to the power of N; where N is an integer greater than or equal to 0. That is, assuming that a first sub-layer and a second sub-layer are adjacent sub-layers, the first sub-layer is used to extract features of each frame of the audio at time domain sample points with a step size of 2 N-1 , and the second sub-layer is used to extract features of each frame of the audio at time domain sample points with a step size of 2 N .

[0069] For example, when N = 1, the first sub-layer extracts features of each frame of the audio at time domain sample points 1, 2, 3, 4, 5, 6, 7, 8, 9, and the like with a step size of 1; and the second sub-layer extracts features of each frame of the audio at time domain sample points 1, 3, 5, 7, 9, and the like with a step size of 2. In this way, by expanding the scale layer by layer, the receptive field is expanded, the window length of each layer is increased in turn, the frequency resolution is reduced in turn, and thus more information can be learned through multi-layer processing.

[0070] In step S240, the feature vector is optimized and reduced in dimension by using an optimized and reduced dimension network, to obtain an identity vector of the speaker to be identified.

[0071] Here, the optimized and reduced dimension network at least includes a frame-level layer, a pooling layer, and a segment-level layer connected in turn; where: the frame-level layer is used to optimize the rules between different frames and the mutual relationship between the features within the frames; the pooling layer is implemented through two full connection layers, and is used to map the feature vector output by the frame-level layer to a fixed-dimension vector through attention pooling or statistic pooling; and the segment-level layer is used to send the output of the pooling layer to a normalization layer through an N-layer neural network, to perform speaker identity discrimination.

[0072] In one possible embodiment, the above-mentioned optimized and reduced dimension process can be implemented through the following process: the feature vector is optimized by using the frame-level layer, to obtain a frame-level feature vector; the frame-level vector is mapped to a fixed-dimension feature vector by using the pooling layer; the fixed-dimension feature vector is connected by using the segment-level layer, to obtain a segment-level feature vector; and the segment-level feature vector is taken as the identity vector of the speaker to be identified.

[0073] In the embodiment of the present application, layers with different expansion scales are arranged in the feature learning network in the speaker recognition model. By expanding the scale layer by layer, the receptive field is expanded, the window length of each layer is increased in turn, and the frequency resolution is reduced in turn. In this way, more information can be learned through multi-layer processing. At the same time, the feature vectors extracted by the feature learning network are processed by the optimization dimension reduction network in the speaker recognition model to determine the identity vector of the speaker to be identified, which is used for subsequent scoring model to determine the identity.

[0074] Figure 3 A flowchart of a speaker recognition method provided in the embodiment of the present application is shown in Figure 3 The training process of the speaker recognition model includes at least the following steps:

[0075] Step S310, obtaining a speech corpus sample labeled with speaker identity;

[0076] Here, the training is performed in a supervised manner through the labeled speech corpus sample. The supervised means that the machine learning task of inferring a function from a labeled training dataset.

[0077] Step S320, obtaining a built speaker recognition model;

[0078] Here, the built speaker recognition model includes an initial feature learning network and an initial optimization dimension reduction network.

[0079] Step S330, initializing the weight parameters of the initial feature learning network using the model parameters of the trained feature extraction network;

[0080] Here, the feature extraction network and the feature learning network in the speaker recognition model have the same network structure, and the model parameters of the feature extraction network are determined by pre-training the speech corpus without identity in a self-supervised manner. Therefore, after the feature extraction network is trained to reach a convergence state, the feature learning network in the built speaker recognition model can be directly initialized to train the speaker recognition model.

[0081] Initializing the feature learning network in the time domain speaker recognition system using the pre-trained feature extraction network can make the network learn more robust speaker features.

[0082] Step S340, training the initial feature learning network and the initial optimization dimension reduction network using the speech corpus sample to obtain the speaker recognition model.

[0083] Here, the training process is in an end-to-end manner, based on a supervised loss function, taking speech corpus samples as input, first passing through a feature learning network to extract a feature vector of the speech corpus sample, then taking the feature vector extracted by the feature learning network as the input of the optimization dimension reduction network, and finally outputting an identity vector representing the speaker identity through the optimization dimension reduction network.

[0084] Figure 4 A flowchart of a speaker recognition method provided by an embodiment of the present application is shown in FIG. 1, which shows the process of the pre-training feature extraction network as follows: Figure 4

[0085] In step S410, at least two types of unlabeled original audio samples are obtained.

[0086] Here, the training is in a self-supervised manner using unlabeled original audio samples. The self-supervised means that the model learns directly from unlabeled data without labeled data.

[0087] In step S420, pseudo audio samples corresponding to each type of original audio sample are obtained.

[0088] Here, the unlabeled original audio samples are first processed by a data augmentation module to generate corresponding pseudo audio samples through reverberation, noise, speed, masking, and spectrum enhancement.

[0089] The at least two types of unlabeled original audio samples include a first type of original audio output by the same speaker and a second type of original audio output by different speakers. In some possible implementation manners, the first type of original audio is processed by data augmentation to obtain pseudo audio data belonging to the same speaker as the first type of original audio. In other possible implementation manners, the second type of original audio is processed by data augmentation to obtain pseudo audio data belonging to different speakers as the first type of original audio. In the actual training process, the above at least one implementation manner can be selected according to the business requirements. The present application does not limit this.

[0090] In step S430, the initial feature extraction network is trained in a self-supervised manner based on a contrastive loss using each type of original audio sample and pseudo audio sample.

[0091] Here, the contrastive loss is a loss function that can well express the matching degree of a pair of samples, i.e., the original audio sample and the pseudo audio sample, and can also be well used to train the feature extraction model.

[0092] ​It's important to note that the value of the contrastive loss function depends on the Euclidean distance between the features of the two samples and a set threshold. When samples are originally similar, a larger Euclidean distance in the feature space results in a larger loss function (increasing function), indicating a poor model. Conversely, when samples are dissimilar, a smaller Euclidean distance in the feature space leads to a larger loss function value.

[0093] Pre-training uses generated pseudo-audio samples along with original audio samples, fed into the feature extraction network. Self-supervised training is performed based on contrastive loss. Pre-training stops when the loss decreases to a threshold. This improves recognition accuracy in cases of large channel differences or uneven data distribution.

[0094] Step S440: In response to the contrast loss being less than or equal to a specific threshold, determine the model parameters of the feature extraction network to obtain the trained feature extraction network.

[0095] Here, the feature extraction network model is trained until the contrastive loss value is less than or equal to a specific threshold through iterative training, indicating that the training is complete.

[0096] The speaker recognition method described above will be explained below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0097] To fully utilize the learning and representation capabilities of neural networks, this application proposes a time-domain speaker recognition system that learns the relationship between time-domain waveform audio through a neural network. Furthermore, this paper proposes a pre-training method to reduce data inhomogeneity or channel differences.

[0098] To address the problem of current speaker recognition systems relying on prior information to solidify frequency resolution across different frequency bands, this application proposes an end-to-end approach to train a time-domain speaker recognition system based on an X-vector framework. Furthermore, this application employs a pre-training method to construct a large amount of pseudo-data (equivalent to pseudo-audio samples) to learn the influence of different channels. The pre-trained model is then used to initialize the feature learning layer of the aforementioned time-domain speaker recognition system, enabling the neural network to learn more robust speaker features.

[0099] To fully utilize the learning and representation capabilities of neural networks, this application proposes a time-domain speaker recognition system that learns the relationship between time-domain waveform audio through a neural network. This time-domain speaker recognition system mainly comprises two parts: a model training part and a back-end scoring part. This application focuses on describing the model training part; the back-end scoring part is implemented similarly to related technologies and will not be described further in this application.

[0100] Figure 5A The schematic diagram of the speaker recognition method provided in the embodiments of this application shows that the model structure mainly includes a feature learning layer 51 (equivalent to a feature learning network), a frame-level layer 52, a pooling layer 53, and a segment-level layer 54.

[0101] Feature learning layer 51, such as Figure 5B As shown, the feature learning layer 51 includes several hidden layers. This embodiment proposes a layer with different expansion scales. As the hidden layers deepen, their expansion scales gradually increase. For example, the expansion scales of hidden layer 1, hidden layer 2, and hidden layer n are 20, 2... 1 ,2 2 ,…,2 n-1 In other words, the deeper the layer, the larger its receptive field. Here, all layers other than the input and output layers are called hidden layers. The selection of hidden layers can be flexible, and the deeper the layer, the more abstract the features extracted. The final feature vector 511 obtained after the feature learning layer 51 is formed by concatenating the weight vectors of each hidden layer.

[0102] Frame-level layer 52: This layer can learn frame-level information using network structures such as ResNet, Transformer, Conformer-based Convolutional Enhancement Transformer, Convolutional Neural Networks (CNN), and Recurrent Neural Networks (RNN).

[0103] Pooling layer 53: The output of the frame-level layer is processed by attention pooling or statistical pooling to map the vector of the frame-level layer to a vector with a fixed dimension.

[0104] Segment-level layer 54: The output of the pooling layer is sent to the normalization layer after passing through N layers of neural network to determine the speaker's identity.

[0105] This application proposes a time-domain speaker recognition system. Compared with existing speaker recognition systems, the proposed system does not require feature extraction and can learn features with speaker representation capabilities using neural networks during model training.

[0106] To address the issues of significant channel mismatches and severe shortages or uneven distribution of tag data in current speaker systems, this application proposes a data augmentation-based pre-training method. For example... Figure 5C As shown, the pre-training method includes the following steps:

[0107] Step S501, data augmentation is performed on the original unlabeled audio samples to generate pseudo audio samples.

[0108] The original unlabeled audio samples will first pass through a data augmentation module to generate corresponding pseudo audio samples. Here, the data is classified, and the original audio samples and their corresponding generated pseudo audio samples are divided into the same class, and the original audio samples are mutually different classes.

[0109] A large number of original unlabeled audio samples are data augmented, including but not limited to: adding reverberation, adding noise, changing speed, adding masking, spectral enhancement, etc., to obtain a large number of pseudo audio samples.

[0110] In the model training data set, the distribution of audio data is often not very uniform, and this uneven distribution is caused by the difference in acoustic path (i.e. channel). Because it is possible that most of the model training data set is clean speech, if the speech being tested is speech in a noisy environment, the speaker recognition model's ability to distinguish may not be very good. In model training, if the training data can contain more scene data, the robustness will be improved.

[0111] Step S502, the original audio samples and the pseudo audio samples are used to pretrain the feature extraction network 55.

[0112] Here, the network structure of the feature extraction network is the same as that of the feature learning layer 51 described above. The generated pseudo audio samples and the original audio samples are sent into the feature extraction network together. As shown in Figure 5D , the candidate audio 11 is data augmented to obtain positive pseudo audio 12 and negative pseudo audio 12, and then the candidate audio 11 and the positive pseudo audio 12 are input into the feature extraction network 55 for training, or the candidate audio 11 and the negative pseudo audio 13 are input into the feature extraction network 55 for training. Thus, through pre-training, the data unevenness or channel difference is reduced.

[0113] In the pre-training (pretrain) process, self-supervised training is performed based on the contrastive loss, and the pre-training is stopped when the loss is reduced to a threshold. The contrastive loss is a loss function for self-supervised pre-training, which is used to represent the matching degree between paired samples, and can also be well used to train the feature extraction model.

[0114] Step S503, the model parameters of the feature extraction network 55 are used to initialize the weight parameters of the feature learning layer 51.

[0115] Here, the model parameters of the pre-trained feature extraction network 55 are used to initialize Figure 5Athe weight parameters of the feature learning layer 51 in the model. Thus, the subsequent model training of the speaker recognition system is performed.

[0116] The speaker recognition system in the time domain provided by the embodiments of the present application can learn features with stronger speaker representation ability by taking advantage of the powerful learning ability of the neural network. Meanwhile, the embodiments of the present application propose a pre-training method based on data enhancement to improve the recognition accuracy in the case of large channel difference or uneven data distribution.

[0117] Based on the foregoing embodiments, the embodiments of the present application further provide a speaker recognition device, which includes various modules and units included in the modules, and can be implemented by a processor in an electronic device. Of course, the device can also be implemented by a specific logic circuit. In the implementation process, the processor can be a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or the like.

[0118] Figure 6 A schematic diagram of the composition structure of a speaker recognition device provided by the embodiments of the present application is shown in FIG. 6, which includes a sample acquisition module 610, a feature extraction module 620, and an optimization and dimension reduction module 630, wherein: Figure 6

[0119] The sample acquisition module 610 is configured to acquire audio data output by a speaker to be identified.

[0120] The feature extraction module 620 is configured to perform frame division and feature extraction on the audio data by using a trained speaker recognition model, to obtain a feature vector of the audio data.

[0121] The optimization and dimension reduction module 630 is configured to perform optimization and dimension reduction on the feature vector by using the speaker recognition model, to obtain an identity vector of the speaker to be identified. The identity vector is used to identify the identity of the speaker to be identified.

[0122] ​In some possible embodiments, the trained speaker recognition model comprises a feature learning network and an optimized dimension reduction network, and the feature extraction module comprises a framing sub-module and an extraction sub-module, where: the framing sub-module is configured to frame the audio data by using the feature learning network to obtain at least two frames of audio; the extraction sub-module is configured to extract features of each frame of the audio by using the feature learning network according to a specific dilation scale, to obtain a feature vector of the audio data; and the optimized dimension reduction module is further configured to perform optimized dimension reduction on the feature vector by using the optimized dimension reduction network, to obtain an identity vector of the speaker to be recognized.

[0123] In some possible embodiments, the optimized dimension reduction network comprises a frame-level layer, a pooling layer and a segment-level layer, and the optimized dimension reduction module comprises a frame processing unit, a pooling unit, a segment processing unit and a feature determination unit, where: the frame processing unit is configured to perform optimized processing on the feature vector by using the frame-level layer, to obtain a frame-level feature vector; the pooling unit is configured to map the frame-level feature vector to a fixed-dimension feature vector by using the pooling layer; the segment processing unit is configured to connect the fixed-dimension feature vector by using the segment-level layer, to obtain a segment-level feature vector; and the feature determination unit is configured to take the segment-level feature vector as the identity vector of the speaker to be recognized.

[0124] In some possible embodiments, the feature learning network comprises at least a first sub-layer and a second sub-layer, and the extraction sub-module comprises a first unit, a second unit and a concatenation unit, where: the first unit is configured to extract features of each frame of the audio by using the first sub-layer according to a first dilation scale, to obtain a first vector; the second unit is configured to extract features of each frame of the audio by using the second sub-layer according to a second dilation scale, to obtain a second vector; the second dilation scale is greater than the first dilation scale; and the concatenation unit is configured to connect the first vector and the second vector, to obtain the feature vector of the audio data.

[0125] In some possible embodiments, the dilation scales of adjacent sub-layers in the feature learning network increase in the form of 2 raised to the power of N, where N is an integer greater than or equal to 0.

[0126] In some possible embodiments, the apparatus 600 further includes an identification model training module, comprising: a first obtaining sub-module, configured to obtain speech corpus samples labeled with speaker identities; a second obtaining sub-module, configured to obtain a built speaker identification model, the built speaker identification model comprising an initial feature learning network and an initial optimization dimension reduction network; an initialization sub-module, configured to initialize weight parameters of the initial feature learning network by using model parameters of a trained feature extraction network; and a training sub-module, configured to train the initial feature learning network and the initial optimization dimension reduction network by using the speech corpus samples to obtain the speaker identification model, wherein a feature vector extracted by the feature learning network is used as input of the optimization dimension reduction network.

[0127] In some possible embodiments, the identification model training module further includes a feature learning sub-module, comprising: a first obtaining unit, configured to obtain at least two types of unlabeled original audio samples; a second obtaining unit, configured to obtain pseudo audio samples corresponding to each type of original audio sample; a training unit, configured to perform self-supervised training on an initial feature extraction network based on a contrastive loss by using each type of original audio sample and the pseudo audio sample; and a parameter determination unit, configured to determine model parameters of the feature extraction network to obtain a trained feature extraction network in response to a case where the contrastive loss is less than or equal to a specific threshold.

[0128] In some possible embodiments, the at least two types of unlabeled original audio samples comprise a first type of original audio output by a same speaker and a second type of original audio output by different speakers; the second obtaining unit is further configured to perform data enhancement processing on the first type of original audio to obtain pseudo audio data belonging to the same speaker as the first type of original audio, and / or perform data enhancement processing on the second type of original audio to obtain pseudo audio data belonging to different speakers as the second type of original audio.

[0129] It should be noted that: the above description of the apparatus embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the apparatus embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0130] It should be noted that, in the embodiments of the present application, if the speaker recognition method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a plurality of instructions for causing an electronic device (which can be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read Only Memory, ROM), a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific hardware and software combination.

[0131] Correspondingly, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the speaker recognition method in any of the above embodiments.

[0132] Correspondingly, in the embodiments of the present application, a chip is also provided, which includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the steps of the speaker recognition method in any of the above embodiments.

[0133] Correspondingly, in the embodiments of the present application, a computer program product is also provided, and when the computer program product is executed by a processor of an electronic device, it is used to implement the steps of the speaker recognition method in any of the above embodiments.

[0134] Based on the same technical concept, the embodiments of the present application provide an electronic device for implementing the speaker recognition method described in the above method embodiments. Figure 7 A hardware entity schematic diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 7, which includes a memory 710 and a processor 720, the memory 710 stores a computer program executable on the processor 720, and the processor 720 implements the steps of the speaker recognition method in any of the embodiments of the present application when executing the program. Figure 7

[0135] The memory 710 is configured to store instructions and applications executable by the processor 720, and can also cache data (for example, image data, audio data, voice communication data and video communication data) to be processed by the processor 720 and each module in the electronic device, which can be realized by a flash memory (FLASH) or a random access memory (Random Access Memory, RAM). ​

[0136] The processor 720 implements the steps of the speaker recognition method of any one of the above when executing a program. The processor 720 generally controls the overall operation of the electronic device 700.

[0137] The above processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, or a microprocessor. It can be understood that the electronic device implementing the above processor functions can also be other, and the embodiments of the present application are not specifically limited.

[0138] The above computer storage medium / memory can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash Memory, a magnetic surface memory, an optical disc, or a Compact Disc Read-Only Memory (CD-ROM), etc. It can also be various electronic devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0139] It should be noted here that the above description of the storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0140] It should be understood that every feature, structure, or characteristic described herein is within a preferred embodiment of the present application. It should be noted that the foregoing embodiments are merely exemplary and are not to be construed as limiting the present application. It should also be noted that features from one embodiment can be combined with features from another embodiment. It should also be noted that the words "comprise," "comprising," "comprises," "include," "including," and "includes" when used in this specification and in the following claims are not to be interpreted so as to exclude other additives, components, elements or steps. It should be understood that the terms "a" or "an," as used herein, mean "one or more" when applied to any feature in the specification and claims.

[0141] It should be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a plurality of components. In this specification and in the claims, the term "on" or "onto" means "directly on or onto," unless otherwise indicated. The term "coupled" means either directly connected to or indirectly connected with the aid of one or more intervening components.

[0142] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: a plurality of units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0143] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place or distributed on multiple network units; and some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0144] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or hardware plus software functional unit.

[0145] Alternatively, the above-mentioned integrated units of the present application, if realized in the form of software function modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to make the equipment test line execute the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROM, magnetic discs or optical discs and various media that can store program codes.

[0146] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0147] The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0148] The above is only an implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of speaker recognition, characterized by, The method comprises: obtaining audio data of a speaker to be identified; frame the audio data using a feature learning network in a trained speaker identification model to obtain at least two frames of audio; The feature learning network comprises at least a first sub-layer and a second sub-layer; the first sub-layer is used to extract features of each frame of audio according to a first expansion scale to obtain a first vector; the second sub-layer is used to extract features of each frame of audio according to a second expansion scale to obtain a second vector; wherein the second expansion scale is greater than the first expansion scale; the first vector and the second vector are connected to obtain a feature vector of the audio data; Optimize and reduce the dimension of the feature vector using the speaker identification model to obtain an identity vector of the speaker to be identified; wherein the identity vector is used to identify the identity of the speaker to be identified.

2. The method of claim 1, wherein, The trained speaker identification model comprises the feature learning network and an optimization and dimension reduction network, The optimization and dimension reduction of the feature vector using the speaker identification model to obtain the identity vector of the speaker to be identified comprises: Optimize and reduce the dimension of the feature vector using the optimization and dimension reduction network to obtain the identity vector of the speaker to be identified.

3. The method of claim 2, wherein, The optimization and dimension reduction network comprises a frame level layer, a pooling layer and a segment level layer, The optimization and dimension reduction of the feature vector using the optimization and dimension reduction network to obtain the identity vector of the speaker to be identified comprises: Optimize the feature vector using the frame level layer to obtain a frame level feature vector; Map the frame level vector to a fixed dimension feature vector using the pooling layer; Connect the fixed dimension feature vector using the segment level layer to obtain a segment level feature vector; The segment level feature vector is used as the identity vector of the speaker to be identified.

4. The method of claim 1, wherein, The expansion scales of adjacent sub-layers in the feature learning network increase in the form of 2 raised to the power of N; wherein N is an integer greater than or equal to 0.

5. The method according to any one of claims 1 to 3, wherein The speaker identification model is trained by the following steps: Obtain speech corpus samples labeled with speaker identity; Obtain a built speaker identification model, which comprises an initial feature learning network and an initial optimization and dimension reduction network; Initialize the weight parameters of the initial feature learning network using the model parameters of the trained feature extraction network; Train the initial feature learning network and the initial optimization and dimension reduction network using the speech corpus samples to obtain the speaker identification model; wherein the feature vector extracted by the feature learning network is used as the input of the optimization and dimension reduction network.

6. The method of claim 5, wherein, The feature extraction network is pre-trained by the following steps: Obtain at least two types of unlabeled original audio samples; Obtain a pseudo audio sample corresponding to each type of original audio sample; Use each type of original audio sample and pseudo audio sample to perform self-supervised training on the initial feature extraction network based on a contrastive loss; In response to the case where the contrastive loss is less than or equal to a certain threshold, determine the model parameters of the feature extraction network to obtain the trained feature extraction network.

7. The method of claim 6, wherein, The at least two types of unlabeled original audio samples include a first type of original audio output by a same speaker, and a second type of original audio output by different speakers; The obtaining of the pseudo-audio sample corresponding to each type of original audio sample comprises: performing data enhancement processing on the first type of original audio to obtain pseudo-audio data belonging to the same speaker as the first type of original audio; and / or, performing data enhancement processing on the second type of original audio to obtain pseudo-audio data belonging to different speakers from the first type of original audio.

8. A speaker recognition apparatus characterized by comprising: The device comprises a sample acquisition module, a feature extraction module, and an optimization and dimension reduction module, wherein: The sample acquisition module is configured to acquire audio data output by a speaker to be identified; The feature extraction module is configured to frame the audio data using a feature learning network in a trained speaker recognition model to obtain at least two frames of audio; the feature learning network comprises at least a first sub-layer and a second sub-layer; the first sub-layer is configured to perform feature extraction on each frame of audio according to a first expansion scale to obtain a first vector; the second sub-layer is configured to perform feature extraction on each frame of audio according to a second expansion scale to obtain a second vector; the second expansion scale is greater than the first expansion scale; and the first vector and the second vector are connected to obtain a feature vector of the audio data; The optimization and dimension reduction module is configured to perform optimization and dimension reduction on the feature vector using the speaker recognition model to obtain an identity vector of the speaker to be identified; the identity vector is used to identify the identity of the speaker to be identified.

9. An electronic device comprising a memory and a processor, the memory storing a computer program operable to run on the processor, characterized in that, The processor executes the program to implement the steps in the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 7.

Images