Speaker recognition method based on speaker voice micro-movements

Abstract

The present application relates to the technical field of electronic digital data processing, in particular to a speaker recognition method based on speaker voice micro-motion, comprising: after the voice stream is preprocessed, Fbank features are extracted and sent into a teacher network and a student network respectively to obtain respective corresponding feature embeddings; the feature embeddings obtained by the teacher network and the student network are sent into a loss function and back propagation is performed; the student network is normally iterated, and the teacher network is iterated through an EMA sliding average method; the voiceprint feature information obtained through an ECAPA-TDNN voiceprint model is aggregated and classified with the speaker voice micro-motion information obtained by training the accent data, and speaker recognition is performed; the present application improves the generalization performance of the model by using data enhancement and other methods, avoids fitting in the channel features, does not require manual labeling, and helps the speaker verification model to achieve the ability to identify speakers in a larger population by introducing new features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic digital data processing, and in particular to a speaker recognition method based on speaker voice micro-movement. BACKGROUND

[0002] Currently, text-independent speaker recognition technology often relies on voiceprint recognition technology, which can extract text-independent speaker voice features from speech. One of the most typical and best captured micro-movements of a speaker is the speaker's accent. Therefore, voice recognition can achieve voice micro-movement recognition. Accent is a specific way of pronunciation in a certain language affected by many factors such as region, native language, and education level. Accent classification technology can well help speech recognition (ASR) or voice assistant tasks, in addition to which it can also help perform speaker recognition tasks and measure the degree of pronunciation standardization.

[0003] In the new century of computer technology development, with the rise of machine learning and deep learning, deep neural networks such as CNN and RNN are applied to speaker verification. Starting from the initial machine learning method of GMM-UBM, the speaker verification method has experienced the i-vector method extracted by GMM and DNN network and the i-vector method extracted by using DNN-UBM. These methods all use artificial design features to let the learning algorithm extract the information needed for speaker verification through methods such as spectral processing and statistical calculation of signals. Subsequently, the d-vector method using DNN was introduced, which extracted frame-level features of a segment of speech and extracted features of a segment of audio through averaging. Subsequently, the method of using x-vector for speaker verification further aggregated frame-level features through learnable parameters to obtain features of a segment of speech. The above methods eventually evolved into the ECAPA-TDNN method. This method has achieved leading results on various data sets and is widely used in practical speech recognition applications.

[0004] However, the data generalization ability of machine learning using the above method is poor, and methods using machine learning and relying on low-level prosodic features require fine-grained human annotation and have high requirements for data integrity. These methods only point out the differences in time and frequency of different accents in phonology and can guide humans to classify through these differences, but they do not consider speech sound changes and have low practicality for a large amount of complex data in reality. SUMMARY

[0005] The application aims to provide a speaker recognition method based on speaker voice micro-movement, which can improve the generalization performance of the model by using data enhancement and other methods, avoid fitting in channel characteristics, and does not require manual annotation, and help the speaker verification model to achieve the ability to identify speakers in a larger population by introducing new features.

[0006] To achieve the above-mentioned purpose, the application provides a speaker recognition method based on speaker voice micro-movement, comprising:

[0007] After the voice stream is preprocessed, the Fbank features are extracted and sent into the teacher network and the student network respectively to obtain the respective corresponding feature embeddings;

[0008] The feature embeddings obtained by the teacher network and the student network are sent into the loss function and are back-propagated;

[0009] The student network is normally iterated, and the teacher network is iterated by the EMA sliding average method;

[0010] The voiceprint feature information extracted by the ECAPA-TDNN voiceprint model is aggregated and classified with the speaker voice micro-movement information trained by the accent data to perform speaker recognition.

[0011] The teacher network and the student network have the same structure and hyperparameters, and the difference between the teacher network and the student network lies in the different parameters in the model.

[0012] The running process of the teacher network and the student network comprises:

[0013] After the voice stream is cut and data enhanced, the Fbank features are extracted and sent into the accent universal acoustic model and the standard mandarin acoustic model;

[0014] The sound features extracted by the accent universal acoustic model and the standard mandarin acoustic model are fused;

[0015] The fused sound features are feature contrast coded, and the features are mapped back to the original data space.

[0016] The accent universal acoustic model and the standard mandarin acoustic model adopt the Uni-ASR model.

[0017] The data set of the accent general acoustic model includes an accent Mandarin data set, an AISHELL-1 data set and an AISHELL-2 data set, the accent Mandarin data set is collected by a crawler on a network and has accent Mandarin, and video subtitles or script content can be used as voice transcription, and the total time length of the AISHELL-1 data set and the AISHELL-2 data set is greater than 1100 hours.

[0018] The data set of the standard Mandarin acoustic model adopts broadcast data collected on the network, and after background music separation and advertisement cutting, ASR is performed to obtain annotation, a total of 2100 hours of audio is collected, and more than 1750 hours of effective data is obtained after data cleaning.

[0019] The specific steps of fusing the sound features extracted from the accent general acoustic model and the standard Mandarin acoustic model include:

[0020] The features output by the accent general acoustic model and the standard Mandarin acoustic model are spliced in the channel dimension, the spliced features are subjected to global maximum pooling and global average pooling, then subjected to a bottleneck layer, the dimension is compressed, the attention parameters are obtained, and the attention parameters are subjected to convolution with the original spliced features.

[0021] The speaker recognition method based on speaker voice micro-motion of the application is completed by means of the framework of accent recognition and the framework of contrast learning, a speaker verification method of voice micro-motion and voiceprint feature fusion is proposed by means of the framework, the generalization performance of the model is improved by using data enhancement and other methods under the condition of widely collecting data, fitting in the channel feature is avoided, and manual annotation is not required, the model can pay attention to the features of phonemes and above phonemes to the word level by the design of the model and the selection of the features, the model of the application can help the model to obtain better speaker recognition results by introducing new features, and the method of the application can help the speaker verification model to play a role in the number of millions to tens of millions of people. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows.

[0023] Figure 1 is a flow chart of the speaker recognition method based on speaker voice micro-motion of the application.

[0024] Figure 2 is a running flow chart of the teacher network and the student network of the application.

[0025] Figure 3 is a schematic diagram of the overall structure of the model of the present application.

[0026] Figure 4 is a schematic diagram of the structure of the feature encoder and the accompanying model of the present application.

[0027] Figure 5 is a schematic diagram of the structure of the Uni-ASR model of the present application.

[0028] Figure 6 is a schematic diagram of the structure of the feature aggregation module of the present application. DETAILED DESCRIPTION

[0029] Referring to Figures 1-6 , the present application provides a speaker recognition method based on speaker voice micro-movement, comprising:

[0030] S1 After the speech stream is preprocessed, Fbank features are extracted and sent into a teacher network and a student network respectively to obtain respective corresponding feature embeddings;

[0031] The teacher network and the student network have the same structure and hyperparameters, and the difference between the teacher network and the student network lies in different parameters in the model. The present application can be integrated into an end-to-end system after multi-stage training, as shown in Figure 3 The left and right system branches have the same structure and hyperparameters, and the difference between the two lies in different parameters in the model, which is a contrastive learning structure. After the speech stream is segmented and data augmented, Fbank features are extracted and sent into the model.

[0032] The running process of the teacher network and the student network comprises:

[0033] S11 After the speech stream is segmented and data augmented, Fbank features are extracted and sent into a universal acoustic model and a standard mandarin acoustic model;

[0034] The universal acoustic model and the standard mandarin acoustic model adopt a Uni-ASR model. The universal acoustic model and the standard mandarin acoustic model adopt the same structure, and through the same structure model, the model can give time-aligned acoustic features. The ASR model does not need to distinguish different accents, and the ASR model will ignore different accents and still map different pronunciations of the same text to the non-accent transcription of the data set. In other words, the ASR model will not try to explore the differences between different accents. On the contrary, accent classification needs to find the differences between the accent and the reference original non-accent. Therefore, by letting the deep neural network grasp the differences between the two, different accents can be better classified.

[0035] The data set of the accent general acoustic model includes an accent Mandarin data set, an AISHELL-1 data set and an AISHELL-2 data set, the accent Mandarin data set is collected by a crawler on a network with accent Mandarin, and video subtitles or script content carried thereby can be used as voice transcription, and the accent video is subjected to audio extraction, down-sampling and the like; the AISHELL-1 data set and the AISHELL-2 data set have a total time length of greater than 1100 hours, and can provide high-quality voice audio and voice transcription.

[0036] The data set of the standard Mandarin acoustic model adopts broadcast data collected on the network, and is subjected to ASR after background music separation and advertisement removal, a total of 2100 hours of audio is collected, and more than 1750 hours of effective data is obtained after data cleaning. The model is expected to reflect the pronunciation of the standard Mandarin, that is, to accurately represent the phoneme of the input voice, and to map the phonemes with accents in the dialect into different vectors.

[0037] The accent general acoustic model and the standard Mandarin acoustic model adopt two acoustic models (AM) with the same structure, and the AM model is Uni-ASR, and the model architecture is as shown in Figure 5 The two complete models are trained respectively for the standard Mandarin AM and the accent general AM. When performing an end-to-end task, only the encoder part of the two is used. The complete ASR model structure is introduced below.

[0038] The architecture is composed of online and offline components, and the two components share a dynamic block encoder to reduce computational complexity. The online component includes a dynamic block encoder and a SCAMA decoder. The encoder adopts an LC-SAN-M structure, and the decoder is mainly based on a unidirectional deep feed-forward sequential memory network (DFSMN) with a multi-head attention (MHA) layer, and the attention mechanism uses a SCAMA structure. The offline component includes a stride convolution, a full sequence encoder, an optional text encoder and a full sequence decoder. The stride convolution is a convolution layer with a stride of 2. The full sequence encoder and the text encoder both adopt a SAN-M structure, and the full sequence decoder includes a DFSMN and an MHA layer. The attention mechanism inside the full sequence decoder is composed of an acoustic context vector and a text context vector. The acoustic context vector is the output of the offline encoder, and the text context vector is the output of the online SCAMA layer, and then the two context vectors are connected to generate an attention vector.

[0039] (1) SAN-M layer

[0040] This layer enhances its self-attention capabilities through DFSMN memory blocks. SAN-M provides an efficient mechanism to incorporate FSMN memory blocks into self-attention, thereby achieving robust local and long-term dependency modeling capabilities. The MHA layer can be expressed using the following formula:

[0041] ;

[0042] ;

[0043] The FSMN memory block output is calculated as follows:

[0044] ;

[0045] in This represents the t-th time instance in the self-attention value. , These are the forward or backward ders of the FSMN memory block, respectively. ⊙ represents the Hadamard product. A DFSMN filter is added to the values ​​inside the MHA to output the memory block. Then the memory content is added to the output of the MHA, i.e.:

[0046] ;

[0047] (2) LC-SAN-M layer

[0048] Input sequence Based on the preset block size Divide into Block size Related to encoder delay. Indicates the first Samples in the nth block. For the nth block Each time instance in a block can only access samples from the current and previous blocks. Computation is used for... The output of LC-SAN-M can be expressed as:

[0049] ;

[0050] Then formula (3) can be rewritten in the following form:

[0051] ;

[0052] ;

[0053] Among them This represents the total number of filters in the FSMN memory block. Ultimately, LC-SAN-M can be written in the following form:

[0054] ;

[0055] S12 fuses the sound features extracted by the accent universal acoustic model and the standard mandarin acoustic model;

[0056] The specific steps include:

[0057] The features output by the accent universal acoustic model and the standard mandarin acoustic model are spliced in the channel dimension, the spliced features are subjected to a global maximum pooling and a global average pooling, and then subjected to a bottleneck layer, the dimension of the features is compressed, the attention parameters are obtained, and the attention parameters are convolved with the original spliced features.

[0058] The feature fusion module fuses the sound features extracted by the two acoustic models and provides the features to the subsequent feature coding comparison module for comparison. Obviously, there are various methods for aggregation, and the application selects the aggregation method of channel attention. This method can automatically highlight the features to enable the feature comparison module to more effectively capture the differences and similarities between the two, and to reduce the features that do not need to be focused on. The features of the application can be described by the following formula:

[0059] First, the features output by the two AMs are spliced in the channel dimension, wherein denotes the feature vectors output by the accent universal AM and the standard mandarin AM:

[0060] ;

[0061] Then, is subjected to a global maximum pooling and a global average pooling, and then subjected to a bottleneck layer, the dimension of the features is compressed. The attention parameters can be obtained thereafter, and the attention parameters are convolved with the original spliced features. The formula of each step is as follows:

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] The compression dimension of the bottleneck layer in formula (16) is 16 in this embodiment, that is, the original features are , and the dimension of the features after compression is .

[0067] S13 performs feature comparison coding on the fused sound features, and maps the features back to the original data space.

[0068] Feature contrast encoding employs a feature contrast encoding module, which uses the encoder module of the Transformer. This module has been proven to be an effective information extraction method for sequence information. The present application selects a stacked encoder module in order to perform high-level extraction of the accent attribute of each frame. This encoder module is composed of a multi-head attention mechanism and a feedforward neural network consisting of two layers of inverse bottleneck layers. The attention mechanism discovers the similarity between the K matrix and the Q matrix, and uses the similarity score to obtain the output of the V matrix. In the experiment, the attention mechanism is applied in the form of self-attention, where the Q, K and V matrices are the same (input of the self-attention layer). The self-attention layer discovers the relationship between the acoustic features at different time steps and outputs the processed features.

[0069] The present application inserts a [cls] token at the beginning of the sequence. For any input, the [cls] token always appears before all the actual input audio embeddings. The input sequence is processed by the Transformer architecture, and the output of each layer generates a vector representation for each token in the sequence. For sequences containing the [cls] token, the output vector at the [cls] token position is considered as the comprehensive feature representation of the entire feature sequence. This is because the Transformer model can learn the representation of each position contextually, and the [cls] token, which does not carry specific semantic information, can more "fairly" integrate the information of the entire feature sequence. Therefore, it can be considered that this token can capture the main information of the entire input feature sequence, so that the [cls] token as a feature vector representing the entire feature sequence greatly simplifies the subsequent model, so that the model does not need to consider the influence of the sequence length, and can better adapt to various types of input.

[0070] S2 sends the feature embeddings obtained by the teacher network and the student network into the loss function respectively and performs back propagation;

[0071] S3 the student network normally iterates, and the teacher network iterates through the EMA sliding average method;

[0072] The teacher network and student network are a contrastive learning structure, and the contrastive learning framework used is a self-distillation algorithm framework, in which the output of the teacher network is used as the optimization target for optimizing the student network. Through data augmentation means, the input speech is converted into multiple different kinds of lengths and "enhanced perspectives". On this basis, the audio will be divided into global and local perspectives according to the length of the input segment. The global perspective is sent into the teacher network, while the global and local perspective audios are input into the student network. Such a design encourages the student network to learn the correspondence between local features and global features. The two networks use the same structure and hyperparameters, but because the input features are different, the weights inside the model are independent of each other. The two networks are composed of a backbone network and a mapping layer. The backbone network is as described above, and the classification layer includes three fully connected layers and a normalization layer. Finally, the cross-entropy loss function is used to optimize the network, and the formula is as follows:

[0073] (19);

[0074] where represents the cross-entropy, represents two long segments, represents four short segments, and represent the probability distributions output by the teacher network and the student network, respectively. Specifically, the probability distribution output by the teacher network is calculated by the softmax function, and the sharpness of the output is controlled by the temperature parameter .

[0075] (20);

[0076] Similarly, the student network also has its corresponding temperature parameter . To avoid the model collapsing to a simple solution (for example, all embedding vectors tend to be consistent), the parameters of the teacher network are updated by the exponential moving average (Exponential Moving Average, EMA) of the student network parameters, and sharpening and centering are applied to the output of the teacher network.

[0077] To ensure the diversity of the embeddings within a batch, the invention proposes a diversity regularization term. It forces the embeddings of different utterances to distinguish from each other, preventing simple solutions. The regularization loss is calculated according to the following formula:

[0078] ;

[0079] where, and represent the embeddings of the teacher and student networks in dimension . denotes variance, is a small quantity close to 0. This regularization term encourages the standard deviation of each dimension within a batch to be close to 1, thus ensuring the diversity of the embedding.

[0080] The redundancy elimination regularization term aims to reduce the redundancy between each embedding variable while minimizing the degree of redundancy. It makes the covariance of all speaker embeddings in each dimension of the batch dimension approach zero, preventing the state of information collapse, i.e., the state of highly correlated variables. The redundancy elimination regularization loss is calculated by:

[0081] ;

[0082] Here, is the cross-correlation matrix between the global outputs of the teacher network and the student network, calculated along the batch dimension, while is defined as:

[0083] ;

[0084] where indexes the batch sample, indexes the embedding dimension. This regularization term de-correlates different vector components of the embedding by encouraging the off-diagonal elements of the cross-correlation matrix to approach 0.

[0085] S4 carries out feature aggregation and classification on the voiceprint feature information extracted by the ECAPA-TDNN voiceprint model and the speaker speech micro-motion information trained by the accent data, and carries out speaker recognition;

[0086] The present application does not use a new speaker recognition model. The present application carries out feature aggregation of the voiceprint feature information extracted by the ECAPA-TDNN voiceprint model and the speaker speech micro-motion information trained by the accent data in a carefully designed feature aggregation module. Through the feature aggregation module and the final classification module, a speaker verification system can be obtained. The feature aggregation module can also be used as a speaker feature embedding module. Obviously, such a feature aggregation mode can help the system to capture the voiceprint features of the speaker in a relatively short time layer, and also capture the micro-motion of the speaker that is difficult to change in a relatively long time layer. This can enable the subsequent model to have greater confidence when dealing with millions or even tens of millions of speaker verification requirements.

[0087] The speech recognition method based on speaker voice micro-movement of the application uses the learning strategy of contrast learning, enables the model to use a larger data set, and gets rid of the requirement of fine labeling, so that the difficulty of data collection is greatly reduced. The labeling of large and small pieces of specific dialects needs to be prepared by professionals. The change of dialect accent in geographical space is a continuous process, and the degree of accent makes the labeling of classification labels a relatively difficult task. Contrast learning can explore the relationship between different accents, spontaneously gather similar accents, and let dissimilar accents move away from each other. This is a great help to extract an embedding that can accurately reflect the relationship between different accents and the degree of accent. The accent model without using the ASR model embedding often fits the recording channel characteristics or the speaker's voiceprint characteristics. Because the model fitting the voiceprint characteristics is much easier than fitting the accent characteristics. Humans are also more likely to distinguish differences in tone, while distinguishing different dialects requires a period of professional learning. This cannot be avoided when not using a text-related model. Such fitting often leads to poor generalization performance of the model. The introduction of the ASR model enables the comparison of the differences between the feature level meaning content (accent general ASR) and the real pronunciation content (standard Mandarin ASR), which is the basis for linguists to judge different accents. The speaker micro-movement system used in the application can extract various micro-movements of the speaker when pronouncing, which is something that the original speaker verification model cannot do. Among hundreds of thousands or even millions of speakers, it is easy to find that there are many pairs of speaker voiceprint characteristics similar. After introducing new features, the speaker verification model can provide another perspective for speaker verification among millions of speakers or more, helping the model to distinguish speakers with similar voices. The application is completed with the help of the framework of accent recognition and the framework of contrast learning. With the help of this framework, the speaker verification method of voice micro-movement and voiceprint feature fusion is proposed. In the case of widely collecting data, the generalization performance of the model is improved by using data enhancement and other methods, avoiding fitting in the channel characteristics, and not needing manual labeling. Through the design of the model and the selection of features, the model can pay attention to the features of phonemes and above phonemes to the word level for a long time. The model of the application can help the model to achieve better speaker recognition by introducing new features. The method of the application can help the speaker verification model to work among millions to tens of millions of people.

[0088] The above only discloses one or more preferred embodiments of the application, and cannot limit the scope of the application. Those skilled in the art can understand that all or part of the above-mentioned embodiments can be implemented, and equivalent changes made to the claims of the application still fall within the scope of the application.

Claims

1. A speaker recognition method based on speaker voice micro-movements, characterized by, The Fbank features extracted after preprocessing of the voice stream are sent into the teacher network and the student network respectively to obtain respective corresponding feature embeddings. The feature embeddings obtained by the teacher network and the student network are sent into a loss function and backpropagation is performed. The student network is normally iterated, and the teacher network is iterated through the EMA sliding average method. The voiceprint feature information extracted by the ECAPA-TDNN voiceprint model is aggregated and classified with the speaker voice micro-motion information trained by the accent data to perform speaker recognition. The operation process of the teacher network and the student network comprises: After the voice stream is segmented and data enhanced, the Fbank features are extracted and sent into the accent universal acoustic model and the standard Mandarin acoustic model. The sound features extracted by the accent universal acoustic model and the standard Mandarin acoustic model are fused, and the specific steps comprise: the features output by the accent universal acoustic model and the standard Mandarin acoustic model are spliced in the channel dimension, the spliced features are subjected to global maximum pooling and global average pooling, then a bottleneck layer is used to compress the dimension, the attention parameters are obtained, and the attention parameters are convolved with the originally spliced features. The fused sound features are subjected to feature comparison coding, and the features are mapped back to the original data space.

2. The speaker recognition method based on speaker voice micro-motion according to claim 1, wherein the teacher network and the student network have the same structure and hyperparameters, and the difference between the teacher network and the student network lies in different parameters in the model.

3. The speaker recognition method based on speaker voice micro-motion according to claim 1, wherein the accent universal acoustic model and the standard Mandarin acoustic model adopt Uni-ASR models.

4. The speaker recognition method based on speaker voice micro-motion according to claim 3, wherein the data set of the accent universal acoustic model comprises an accent Mandarin data set, an AISHELL-1 data set and an AISHELL-2 data set, the accent Mandarin data set is collected through a crawler on a network, and the video subtitles or script content thereof can be used as voice transcription, and the total duration of the AISHELL-1 data set and the AISHELL-2 data set is greater than 1100 hours.

5. The speaker recognition method based on speaker voice micro-motion according to claim 4, wherein the data set of the standard Mandarin acoustic model adopts broadcast data collected on the network, and the ASR is performed after background music separation and advertisement cutting, a total of 2100 hours of audio is collected, and more than 1750 hours of effective data is obtained after data cleaning. ​ ​ ​ ​ ​

Images