A voiceprint recognition method based on phoneme information and electronic equipment

Abstract

The application provides a voiceprint recognition method based on phoneme information and an electronic device, and the method comprises the following steps: performing endpoint detection on the audio to be recognized to obtain an effective speech segment; the effective speech segment is subjected to phoneme labeling and phoneme level speech segment determination in the effective speech segment through a pre-trained automatic speech recognition model; whether all phonemes labeled in the effective speech segment meet a phoneme coverage rate condition is determined according to the phoneme labeling result; if the phoneme coverage rate condition is met, phoneme level voiceprint recognition is adopted; and if the phoneme coverage rate condition is not met, sentence level voiceprint recognition is adopted. The voiceprint recognition method fuses the speaker embedding of the sentence level and the phoneme level, and can improve the accuracy and reliability of the voiceprint recognition system in a complex actual environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of information technology, and particularly relates to a voiceprint recognition method based on phoneme information and an electronic device. BACKGROUND

[0002] Voiceprint recognition includes voiceprint identification and voiceprint verification, and can be divided into text-dependent and text-independent according to whether it depends on fixed speech semantic information. Voiceprint identification refers to identifying the speaker of unknown identity information by extracting voiceprint features and comparing them with voiceprint information registered and entered in the voiceprint database to obtain identity information, which is a one-to-many task. Voiceprint verification refers to comparing the detected voice of a speaker with the information saved in the database for the corresponding speaker to determine whether the declared identity is true, which is a one-to-one task.

[0003] Voiceprint, like fingerprint, iris and other features, can be used as unique biological identity information for each person, and has the advantage of non-contact collection. With the rapid development of deep neural networks, the performance of x-vector using PLDA as the discriminative backend exceeds that of the traditional machine learning technology combination of i-vector+PLDA. TDNN is actually one-dimensional convolution along the time axis, and the original x-vector uses a convolution kernel with a size of 5, and each convolution calculation includes 5 frames before and after the current frame, which makes TDNN pay more attention to local features on the data. Researchers have proposed many improvements. Snyder et al. proposed E-TDNN to use a wider convolution kernel to process more time series information at the same time, and inserted an affine transformation layer between each convolution layer to make the performance far exceed that of the original TDNN. Povey et al. proposed F-TDNN to reduce the number of model parameters, which decomposes the weight matrix of each layer of TDNN into the product of two matrices with smaller ranks, so as to reduce the number of parameters while not losing information. Chen et al. introduced LSTM into TDNN to better capture global features through LSTM to compensate for TDNN.

[0004] Another widely used network structure in speaker recognition is the residual network (ResNet) proposed by He et al. ResNet is a variant of CNN, which is stacked by a series of two-dimensional convolution kernels. The addition of residual solves the problem of gradient vanishing in the training process of deep neural networks. Since acoustic features can be regarded as two-dimensional pictures composed of time and frequency-related data, it is possible to use two-dimensional convolution on acoustic features. Chung et al. and Li et al. directly use ResNet as the front-end of speaker embedding vector extraction. Some other researchers use ResNet as the core structure of voiceprint feature extraction, and achieve some specific applications by adjusting the structure or changing the objective function. Xie et al. reduce the number of channels in each residual block of ResNet-34 to propose a thin-ResNet structure, which reduces the parameter size. Zhao et al. combine the bidirectional long short-term memory network (BLSTM) with ResNet, so that the entire system can model the context information in long-time speech. Jung et al.’s RawNet network structure directly takes the original audio as input, and improves the model performance by improving the loss function and attention mechanism.

[0005] However, in practical applications, due to the collection of corpus, the speech used for verification often has great differences in accent, pronunciation content, and emotion from the registered speech. This bias leads to great uncertainty in the voiceprint features extracted using deep neural networks, ultimately causing the system performance to decline. Introducing phoneme information is an effective way to reduce the influence of such uncertainty on voiceprint features. Zhou et al. add phoneme features extracted by the bottleneck layer of an automatic speech recognition (ASR) model in the training of the voiceprint extraction model. Liu et al. use multi-task learning to effectively reduce the noise influence in past frame-level multi-task learning by taking phoneme classification at the frame level as a secondary task and speaker classification at the segment level as a primary task. Chen et al. are inspired by the past GMM-UBM modeling of each phoneme and propose a TDNN structure based on phonemes for voiceprint recognition. Tawara et al. believe that the past pooling method and phoneme-aware training method are harmful to extracting speaker embedding vectors from short speech, and propose an adversarial training framework to remove the variance in frame-level speaker embedding vectors caused by phonemes. Kim et al. believe that it is not only necessary to obtain speaker embedding vectors that remove phoneme interference, but also necessary to be sensitive to different phoneme durations, and propose a temporal dynamic convolution network structure that dynamically adjusts the convolution kernel parameters according to phonemes of different lengths. These existing researches show that adding phoneme information in the speaker modeling stage can reduce the uncertainty of voiceprint features introduced by the bias between registration and verification, thereby improving the performance of the speaker recognition system.

[0006] Although the influence of uncertainty is effectively weakened by introducing phoneme information, the previous phoneme information-based voiceprint recognition method still has two serious problems. First, real audio often contains a large amount of "noise" data, such as silence, background music, environmental noise and other unreliable speech fragments. These "noise" data will introduce additional confusion information in the voiceprint recognition model training stage, and the past methods lack processing of these "noise" data. Second, the existing phoneme information-based voiceprint recognition method still uses a sentence-level speaker embedding vector as a voiceprint feature. Although the phoneme information in the speaker embedding vector can be removed to some extent through adversarial training or adaptive methods, this coarse-grained voiceprint feature still contains features of different pronunciations, such as jumps between phonemes, which blur the differences between speakers. SUMMARY

[0007] The purpose of the embodiments of the present specification is to provide a phoneme information-based voiceprint recognition method and electronic equipment.

[0008] To solve the above technical problems, the embodiments of the present application are implemented in the following way:

[0009] In a first aspect, the present application provides a phoneme information-based voiceprint recognition method, which comprises:

[0010] Performing endpoint detection on the audio to be recognized to obtain an effective speech segment;

[0011] The effective speech segment is subjected to phoneme labeling and determination of a phoneme-level speech segment in the effective speech segment by using a pre-trained automatic speech recognition model;

[0012] According to the phoneme labeling result, it is determined whether all the phonemes labeled in the effective speech segment meet a phoneme coverage rate condition. If the phoneme coverage rate condition is met, phoneme-level voiceprint recognition is adopted. If the phoneme coverage rate condition is not met, sentence-level voiceprint recognition is adopted.

[0013] In one of the embodiments, the effective speech segment is subjected to phoneme labeling and determination of a phoneme-level speech segment in the effective speech segment by using a pre-trained automatic speech recognition model, which comprises:

[0014] The pre-trained automatic speech recognition model is used to determine the posterior probability of each speech frame in the effective speech segment with respect to different phoneme labels;

[0015] According to the maximum posterior probability of each speech frame, the effective speech segment is segmented to obtain an effective speech segment of a single phoneme;

[0016] The phoneme-level speech segment is obtained by filtering out the speech segment marked with silence in the effective speech segment and the phoneme segment with a total frame number less than a length threshold.

[0017] In one of the embodiments, the valid speech segment of a single phoneme is segmented according to the maximum a posteriori probability of each speech frame, including:

[0018] For each speech frame, the phoneme with the maximum a posteriori probability is selected as the most possible phoneme, and if the corresponding a posteriori probability is greater than a preset confidence threshold, the most possible phoneme is the phoneme label of the speech frame; otherwise, it is filled with a silence label;

[0019] The continuous speech frames with the same phoneme label are aggregated to obtain the valid speech segment of a single phoneme.

[0020] In one of the embodiments, the phoneme coverage condition includes:

[0021] When the frequency of the phoneme appearing in the valid speech segment is greater than a probability threshold, the phoneme is determined as a valid phoneme; and

[0022] The number of valid phonemes is greater than a preset number.

[0023] In one of the embodiments, the frequency of the phoneme appearing in the valid speech segment is determined according to the number of phonemes contained in the valid speech segment and the total number of phonemes in the valid speech segment.

[0024] In one of the embodiments, the phoneme-level voiceprint recognition includes:

[0025] Each phoneme-level speech segment of each phoneme is input into a phoneme-level voiceprint feature extraction network to obtain a phoneme-level embedding vector corresponding to each phoneme-level speech segment of each phoneme;

[0026] For each phoneme, all phoneme-level embedding vectors are aggregated to obtain a final embedding vector of each phoneme;

[0027] The final embedding vectors of all phonemes are collected as a phoneme-level voiceprint feature;

[0028] The phoneme-level voiceprint feature is compared with a corresponding voiceprint feature in a phoneme-level voiceprint database, all comparison results of the phonemes are summarized, and a voiceprint recognition result is obtained through a voting manner.

[0029] In one of the embodiments, the phoneme-level voiceprint feature extraction network is trained by using a multi-gate hybrid expert mechanism.

[0030] In one of the embodiments, the multi-gate hybrid expert mechanism includes three expert sub-networks.

[0031] The multi-gate hybrid expert mechanism is used to remap the embedding vectors of sub-tasks to different feature sub-spaces, including:

[0032] The phoneme-level embedding vectors output by the phoneme-level voiceprint feature extraction network are projected to different subspaces by the following transformation:

[0033]

[0034] where y1 is a phoneme recognition subtask hidden vector, y2 is a speaker recognition subtask hidden vector, h k is a subtask classification layer, gate k is a gating network that adjusts the contribution to different subtasks:

[0035]

[0036] expert i is an expert subnetwork:

[0037]

[0038] wherein, are learnable parameters.

[0039] In one of the embodiments, the sentence-level voiceprint recognition includes:

[0040] The sentence-level embedding vectors corresponding to the valid speech segments are extracted using a sentence-level voiceprint feature extraction network;

[0041] All sentence-level embedding vectors are aggregated to obtain sentence-level voiceprint features;

[0042] The sentence-level voiceprint features are compared with the corresponding voiceprint features in the sentence-level voiceprint database, and the comparison results of all valid speech segments are summarized to obtain a voiceprint recognition result by voting.

[0043] In a second aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the phoneme information-based voiceprint recognition method of the first aspect when executing the program.

[0044] As can be seen from the technical solutions provided by the above embodiments of the present specification, the scheme: the voiceprint recognition using the speaker embedding fused with the sentence level and the phoneme level can improve the accuracy and reliability of the voiceprint recognition system in complex actual environments. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments of the present specification, and all other drawings obtained by those of ordinary skill in the art without creative labor should also belong to the protection scope of the present specification.

[0046] Figure 1 A flowchart of a voiceprint recognition method based on phoneme information provided by the present application is shown in the figure.

[0047] Figure 2 Another flowchart of a voiceprint recognition method based on phoneme information provided by the present application is shown in the figure.

[0048] Figure 3 A flowchart of a phoneme labeling method based on ASR provided by the present application is shown in the figure.

[0049] Figure 4 A flowchart of a phoneme-level voiceprint recognition method provided by the present application is shown in the figure.

[0050] Figure 5 A structure diagram of MMoE provided by the present application is shown in the figure.

[0051] Figure 6 A flowchart of a sentence-level voiceprint recognition method provided by the present application is shown in the figure.

[0052] Figure 7 A structure diagram of an electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0053] In order for those skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be described clearly and completely in the following description of the embodiments of the present specification, and obviously, the described embodiments only represent some of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor should belong to the protection scope of the present specification.

[0054] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application, but those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0055] Many modifications and variations to the illustrative embodiments described herein will be apparent to those of ordinary skill in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the applications. Other embodiments will be apparent to those of ordinary skill in the art from consideration of the description and practice of the applications disclosed herein. The description and examples are intended for purposes of illustration and are not intended to limit or restrict the scope of the applications.

[0056] As used herein, the terms "comprise", "comprising", "include", "including", "have", "having" and the like are open-ended and do not exclude additional elements or steps.

[0057] The defects of the related art are:

[0058] (1) Lack of filtering means for unreliable speech segments: traditional voiceprint recognition methods, including voiceprint recognition methods based on phoneme information, directly use speech signals as input. However, actual speech signals contain a large number of unreliable "noise" segments, which will introduce a large amount of uncertainty in the modeling and reasoning process, and especially when the text content, accent, language, emotion, etc. of the registered speech and the test speech are too different, the system recognition accuracy will be greatly reduced;

[0059] (2) The granularity of the voiceprint recognition modeling method based on phoneme information is relatively rough: introducing additional phoneme information to the traditional sentence-level voiceprint recognition system can effectively improve the accuracy of voiceprint recognition. However, existing voiceprint recognition methods based on phoneme information mostly use the adversarial training of speaker recognition and phoneme recognition to obtain sentence-level speaker embedding vectors with invariance. This modeling granularity is relatively rough, only removes phoneme components, and ignores the bias caused by phoneme transitions, so the voiceprint recognition accuracy is limited;

[0060] (3) Lack of suitable fusion means for sentence-level and phoneme-level speaker embedding vectors: existing voiceprint recognition methods based on phoneme level often concatenate phoneme-level speaker embedding vectors according to a certain text to form a sentence to approximate the text-dependent voiceprint recognition scene. This way of artificially synthesizing sentences seriously weakens the speaker prosody features contained in the original continuous sentences, reducing the discrimination of the final speaker embedding vectors.

[0061] Based on the above defects, the present application provides a phoneme information based voiceprint recognition method, aiming to improve the accuracy and reliability of the voiceprint recognition system in complex actual environment. The method of the present application combines phoneme level voiceprint recognition and sentence level voiceprint recognition. The phoneme level voiceprint recognition converts the more difficult text-independent voiceprint recognition scene into a simpler text-dependent voiceprint recognition scene by comparing the voiceprint features of the same phonemes, so as to reduce the influence of uncertainties such as accent, text content, emotion, etc. The sentence level voiceprint recognition processes the to-be-tested corpus that does not meet the phoneme level voiceprint recognition condition, thereby improving the robustness of the combined voiceprint recognition method.

[0062] The application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0063] Reference Figure 1 and Figure 2 which shows a flowchart of the phoneme information based voiceprint recognition method provided by the embodiments of the present application.

[0064] As Figure 1 and Figure 2 shown, the phoneme information based voiceprint recognition method can include:

[0065] S110, endpoint detection is performed on the to-be-identified audio to obtain an effective speech segment.

[0066] Specifically, the to-be-identified audio is the original audio (or original speech) to be subjected to voiceprint recognition. Since the original audio contains non-speech segments such as silence, noise, music segments, etc., it is necessary to filter out the non-speech segments in the original audio and retain the speech segments, i.e. the effective speech of the human voice segment, as the effective speech segment for subsequent use.

[0067] Among them, the VAD technology can be divided into three categories: signal processing based, machine learning based and deep learning based.

[0068] The VAD based on signal processing mainly includes short-time energy, zero-crossing rate detection and spectral entropy, etc.

[0069] Among them, the short-time energy: since there is a significant difference in energy between speech segments and non-speech segments, an energy threshold (which can be set according to actual needs) can be set, and the speech segments and non-speech segments can be determined by comparing the short-time energy with the energy threshold. When the short-time energy is greater than or equal to the energy threshold, it is determined as a speech segment, and when the short-time energy is less than the energy threshold, it is determined as a non-speech segment.

[0070] Among them, the short-time energy E n It is calculated by the following formula:

[0071]

[0072] Wherein, x is the audio sample point to be identified, N is the number of sample points detected each time.

[0073] Since the voice value in the audio to be identified in a period of time can be positive or negative, the number of times of voice value jump (i.e. zero-crossing rate) is more when it is a non-language segment, and the number of times of voice value jump is less when it is a voice segment, therefore, a jump threshold can be set, and by comparing the jump threshold with the zero-crossing rate, the voice segment and the non-voice segment can be determined. When the zero-crossing rate is greater than or equal to the jump threshold, it is determined as a non-voice segment, and when the zero-crossing rate is less than the jump threshold, it is determined as a voice segment.

[0074] Wherein, the zero-crossing rate Z n The zero-crossing rate Z is calculated by the following formula:

[0075]

[0076]

[0077] Wherein, x is the audio sample point to be identified, and sgn(x) is a sign function.

[0078] Spectral entropy: since the spectral entropy of the voice segment is larger, and the spectral entropy of the non-voice segment is smaller, therefore, a spectral entropy threshold can be set, and by comparing the spectral entropy with the spectral entropy threshold, the voice segment and the non-voice segment can be determined. That is, when the spectral entropy is greater than or equal to the spectral entropy threshold, it is determined as a voice segment, and when the spectral entropy is less than the spectral entropy threshold, it is determined as a non-voice segment.

[0079] Wherein, the spectral entropy H(s) is calculated by the following formula:

[0080]

[0081] Wherein, P is the normalized signal power spectral density.

[0082] The VAD based on machine learning is mainly realized by a Gaussian mixture model (GMM), a support vector machine (SVM), a random forest (RF), a deep belief network (Deep Belief Networks), a conditional random field (CRF) and the like.

[0083] The VAD based on signal processing and traditional machine learning is often limited by the model expression ability, and the performance will be significantly reduced when some noise signals are similar to effective speech in time-frequency. Benefiting from the rapid development of deep learning, some more complex models based on deep neural networks have also been used for VAD in recent years, such as long short-term memory network (LSTM), gated recurrent unit network (GRU) and time convolution network (TCN), etc. These models regard VAD as a sequence learning task and have achieved good results. Some network structures with larger parameter scale, such as Transformer, complete tasks such as VAD, speech enhancement and speech separation through self-supervised learning, so that the segmented speech signal only contains clean human voice.

[0084] In S120, the phoneme labeling and determination of the phoneme-level speech segment in the effective speech segment are implemented through a pre-trained automatic speech recognition model, including:

[0085] The posterior probability of each speech frame in the effective speech segment with respect to different phoneme labels is determined by using the pre-trained ASR model;

[0086] The effective speech segment of a single phoneme is segmented according to the maximum posterior probability of each speech frame;

[0087] The phoneme-level speech segment is obtained by filtering out the speech segment with a silence label in the effective speech segment and the phoneme segment with a total frame number less than a length threshold.

[0088] In S120, the phoneme labeling and determination of the phoneme-level speech segment in the effective speech segment are implemented through a pre-trained automatic speech recognition model, including:

[0089] For each speech frame, the phoneme with the maximum posterior probability is selected as the most possible phoneme, and if the corresponding posterior probability is greater than a pre-set confidence threshold, the most possible phoneme is the phoneme label of the speech frame; otherwise, it is filled with a silence label;

[0090] The continuous speech frames with the same phoneme label are aggregated to obtain the effective speech segment of a single phoneme.

[0091] Specifically, for the effective speech obtained through VAD, the most possible phoneme label of each speech frame is obtained by using a pre-trained automatic speech recognition (ASR) model, and the continuous speech frames with the same phoneme are aggregated to realize phoneme labeling.

[0092] Generally, assuming that there is a frame sequence {x i |i=1,2,…,T} of an effective speech segment with a total length of T, a total set of K phoneme categories {m j |j=1,2,…,K}, then each speech frame x iis the posterior probability of a certain phoneme p(m i |x t ) is given by:

[0093]

[0094] where is a pre-trained ASR model, which can be a traditional acoustic model- language model or an end-to-end technique such as GMM-HMM, RNN-Transducer, Transformer, etc.

[0095] For each speech frame x i , the phoneme with the maximum posterior probability is selected, i.e., the most likely phoneme, and if the corresponding posterior probability is greater than a pre-set confidence threshold Threshold1, the most likely phoneme is considered as the phoneme label of the speech frame, otherwise, it is filled with a silence label "sp", and Threshold1 is generally taken as 0.6. This process is denoted by m(t).

[0096]

[0097] Since the silence frame cannot provide effective information, the speech frame with the label sp needs to be filtered out, and after this, the start and end time stamps of each reliable phoneme (i.e., the effective speech segment of a single phoneme or phoneme segment) in the sentence are obtained. According to the start and end time stamps of the phoneme, it is judged whether the phoneme duration meets the condition, specifically: if the total frame number of the corresponding phoneme segment is greater than a pre-set length threshold Threshold2, the acoustic feature of the corresponding phoneme segment (i.e., the phoneme-level speech segment) is cut out, otherwise, the phoneme is further discarded, and Threshold2 is generally taken as 2.

[0098] As Figure 3 is an example of labeling the effective speech segment "Wo", where the phonemes of "Wo" are {w, o}, sp is the silence label that needs to be filtered out, and phoneme b is the subsequent speech.

[0099] S130, according to the phoneme labeling result, it is judged whether all the phonemes marked in the effective speech segment meet the phoneme coverage condition, if the phoneme coverage condition is met, phoneme-level voiceprint recognition is adopted; if the phoneme coverage condition is not met, sentence-level voiceprint recognition is adopted.

[0100] Specifically, only when all the phonemes marked in the speaker's effective speech segment meet the phoneme coverage condition, the system performs phoneme-level voiceprint recognition.

[0101] In one embodiment, the phoneme coverage condition includes:

[0102] When the frequency of the phoneme appearing in the valid speech segment is greater than the probability threshold, the phoneme is determined as a valid phoneme;

[0103] The number of valid phonemes is greater than a preset number.

[0104] The frequency of the phoneme appearing in the valid speech segment is determined according to the number of phonemes contained in the valid speech segment and the total number of phonemes in the valid speech segment.

[0105] Specifically, the probability threshold and the preset number can be set according to actual needs. For example, the probability threshold is set to 0.05, and the preset number is set to 10.

[0106] The phoneme m i The frequency f of the phoneme appearing in the valid speech segment mi is:

[0107]

[0108] The number of phonemes contained in the valid speech segment m i is Count(m i , and the total number of phonemes in the valid speech segment is Number.

[0109] In one embodiment, the phoneme-level voiceprint recognition includes:

[0110] Each phoneme-level speech segment of each phoneme is input into a phoneme-level voiceprint feature extraction network to obtain a phoneme-level embedding vector corresponding to each phoneme-level speech segment of each phoneme;

[0111] For each phoneme, all phoneme-level embedding vectors are aggregated to obtain a final embedding vector of each phoneme;

[0112] The final embedding vectors of all phonemes are collected as a phoneme-level voiceprint feature;

[0113] The phoneme-level voiceprint feature is compared with the corresponding voiceprint feature in the phoneme-level voiceprint database, and the comparison results of all phonemes are summarized to obtain a voiceprint recognition result by voting.

[0114] Specifically, the input of the phoneme-level voiceprint feature extraction network is a phoneme-level speech segment (or a reliable phoneme segment), and the output is a corresponding phoneme-level embedding vector.

[0115] For a certain speaker, each speech segment belonging to the i-th phoneme is input into the phoneme-level voiceprint feature extraction network respectively, and the corresponding phoneme-level embedding vector (or called phoneme-level feature vector) is obtained. Then, the feature vectors are weighted and averaged to obtain the final embedding vector (or called voiceprint feature vector) of the i-th phoneme of the specific speaker. Repeat this step until all phonemes are traversed, and the final embedding vector corresponding to each phoneme is collected as the phoneme-level voiceprint feature of the speaker.

[0116] For example, for a certain speaker, it is assumed that m j is the j-th phoneme, 1≤j≤K, K is the total number of phoneme categories. The n-th phoneme segment belonging to the phoneme m j is where 1≤n≤N mj , N mj is the total number of segments of the phoneme m j , then the n-th speaker phoneme-level embedding vector of the phoneme m j is .

[0117]

[0118] In the above formula, f represents the phoneme-level voiceprint feature extraction network. The network main structure of the phoneme-level voiceprint feature extraction network can be a deep neural network such as convolutional neural network, time delay neural network (TDNN), Transformer, ResNet (Residual Neural Network), etc.

[0119] The final speaker embedding vector e mj of the phoneme m j is obtained by averaging and aggregating the N mj speaker phoneme-level embedding vectors of the phoneme m j :

[0120]

[0121] The set of speaker embedding vectors {e m1 , e m2 , …, e mk} corresponding to each phoneme is taken as the phoneme-level voiceprint feature of the speaker.

[0122] Cosine similarity, PLDA (Probabilistic Linear Discriminant Analysis), Gaussian mixture model, etc. can be used as the basis for comparison and verification corpus phonemes and corresponding phonetic feature database, the comparison results of all phonemes in the verification corpus are summarized, and the final speaker identity is obtained by voting.

[0123] For example, the phonetic feature of the phoneme "w" of the speaker "woa" is given. Figure 4 The phonetic feature of the phoneme "w" of the speaker "woa" is given.

[0124] The phonetic feature of the phoneme "w" of the speaker "woa" is given. w ,e o ,e a} respectively, and the phonetic feature of the phoneme "w" of the speaker "woa" is given. w ,E o ,E a} respectively, and the phonetic feature of the phoneme "w" of the speaker "woa" is given.

[0125]

[0126] Where S i (e i ,E i ) is the phonetic similarity between the speaker and the known speaker, for example, S w (e w ,E w ) represents the similarity between the phonetic feature to be verified and the corresponding registered phonetic feature. The function represents the similarity calculation method, which can be cosine similarity, PLDA, Gaussian mixture model, etc.

[0127] The final similarity score is obtained by averaging all phonetic similarity scores. For this 3-phoneme example, the final similarity score is:

[0128]

[0129] In real life, when distinguishing different speakers, people often remember some people by some inherent pronunciation characteristics. Therefore, it is necessary to use multi-task learning training method to improve the phoneme discrimination ability of the model and realize the modeling of different pronunciations. In existing multi-task learning, the feature representation of phonemes and speakers often shares the top parameters of the network, which is equivalent to two sub-tasks having the same distribution in the feature space. However, the feature representation of different phonemes and the feature representation of speakers should have different distributions, and this assumption will greatly reduce the discrimination ability of the generated speaker embedding vector.

[0130] To further improve the sensitivity of the model to different pronunciation characteristics, the present application introduces a multi-gate mixture-of-experts mechanism (MMoE) for training a phoneme-level voiceprint feature extraction network to remap the embedding vectors of subtasks to different subspaces. The MMoE is implemented as shown in Figure 5 Figure 5 An MMoE composed of 3 expert subnetworks is shown.

[0131] The multi-gate mixture-of-experts mechanism is used to remap the embedding vectors of subtasks to different feature subspaces, including:

[0132] The phoneme-level embedding vectors output by the phoneme-level voiceprint feature extraction network are projected to different subspaces by the following transformation:

[0133]

[0134] where y1 is the phoneme recognition subtask hidden vector, y2 is the speaker recognition subtask hidden vector, h k () is a subtask classification layer, gate k () is a gating network that adjusts the contribution of different subtasks:

[0135]

[0136] expert i () is an expert subnetwork:

[0137]

[0138] where, are learnable parameters, respectively.

[0139] As shown in Figure 5 , each expert represents a weak learner. According to the concept of integration, a series of weak learners can be combined into a strong learner. The use of a gating subnetwork can assign different weights to each expert according to different learning tasks. The MMoE improves the sensitivity of the phoneme-level voiceprint feature to different pronunciations and improves the speaker representation ability by automatically adjusting the shared and unshared parameters to learn the relationship between different tasks.

[0140] Continuing to refer to Figure 4 is an example of phoneme-level voiceprint recognition from reliable phoneme segments "w o a" of a speaker "I". Three segments of reliable phoneme "w" are obtained by ASR labeling two segments of phoneme "o" one segment of phoneme "a" ​The MMoE-based voiceprint extraction model generates a speaker embedding vector corresponding to each phoneme segment, aggregates the embedding vectors of each phoneme to obtain a final speaker embedding vector, and compares the final speaker embedding vector with a database to obtain a corresponding recognition result.

[0141] When the phoneme coverage rate is not satisfied, the joint voiceprint recognition degenerates into sentence-level voiceprint recognition.

[0142] In one embodiment, the sentence-level voiceprint recognition includes:

[0143] A sentence-level embedding vector corresponding to the effective speech segment is extracted using a sentence-level voiceprint feature extraction network.

[0144] All sentence-level embedding vectors are aggregated to obtain a sentence-level voiceprint feature.

[0145] The sentence-level voiceprint feature is compared with the corresponding voiceprint feature in the sentence-level voiceprint database, the comparison results of all effective speech segments are summarized, and the voiceprint recognition result is obtained through voting.

[0146] Specifically, {s n |n=1,2,…,N} represents a total of N segments of all effective speech segments of a speaker,

[0147] {e n |n=1,2,…,N} is a speaker embedding vector e n (i.e., a sentence-level embedding vector) extracted from each effective speech segment:

[0148]

[0149] wherein, is a sentence-level voiceprint feature extraction subnetwork, and the network structure can be a deep neural network such as ResNet, TDNN, or Transformer.

[0150] The final speaker embedding vector e of the speaker (i.e., the sentence-level voiceprint feature) is obtained by averaging and aggregating N speaker embedding vectors:

[0151]

[0152] Cosine similarity, PLDA, and Gaussian mixture model can be used as a basis for judgment, the sentence-level voiceprint features in the verification corpus are compared with the corresponding sentence-level registered voiceprint database, the comparison results of all sentences in the verification corpus are summarized, and the final speaker identity is obtained through voting.

[0153] For example, Figure 6 The comparison process is illustrated by three sentence-level voiceprint features of a certain speaker.

[0154] The sentence-level speaker embedding features {e 1 2 3} corresponding to the speaker 3 segments are respectively compared with a known speaker embedding feature {E} in the pre-recorded embedding database (i.e., the sentence-level embedding database) for similarity:

[0155]

[0156] where S i (e i ,E i ) is the sentence-level similarity between the speaker and the known speaker, for example, S1(e1,E) represents the similarity between the first segment of the to-be-verified embedding feature e 1 and the corresponding registered embedding feature. The function represents the way of similarity calculation, which can be cosine similarity, PLDA, Gaussian mixture model, etc.

[0157] The final similarity score is obtained by averaging all the sentence-level similarities. For the example of the 3 segments, the final similarity score is:

[0158]

[0159] Continuing to refer to Figure 6 is an example of extracting corresponding embedding vectors {e 1 ,e 2 ,e 3} from the valid speech {s 1 ,s 2 ,s 3} of a certain speaker 3 sentences, obtaining a sentence-level embedding feature e by aggregation, and obtaining a recognition result by comparison.

[0160] Most existing text-independent speaker recognition technologies extract sentence-level speaker embedding vectors from sample speech and to-be-identified speech, and improve the robustness of the speaker embedding feature by averaging as many sentence speaker embedding vectors as possible. This approach is often limited by the number of sentences, and does not process noise data in the sentence. When the text content, accent, language, emotion, etc. of the registered corpus and the verification corpus are too different, the system recognition accuracy greatly decays. In comparison, the present application calibrates the phonemes in the sentence through pre-training, further filters the valid phoneme segments according to the phoneme confidence, and generates a speaker embedding vector corresponding to each high-confidence phoneme. For test samples that meet certain conditions, the text-independent speaker recognition scenario is converted to a text-dependent scenario by comparing each phoneme in the test sentence with the same registered phoneme-level embedding vector, thereby reducing the performance decay caused by uncertainty.

[0161] ​​The existing deep learning-based voiceprint recognition system introducing phoneme information mostly separates the voiceprint features with invariance from acoustic features through the adversarial training of the two tasks of sentence-level speaker recognition and frame-level phoneme recognition to improve the robustness of the sentence-level speaker embedding vector, but this approach only removes the interference caused by phonemes to a certain extent and does not pay attention to the fact that the jump between phonemes also has speaker individuality information; compared with the prior art, the model training process is redesigned, the adversarial training of the original sentence-level speaker recognition and frame-level phoneme recognition is changed to the multi-task training of the phoneme-level speaker recognition and phoneme-level phoneme recognition, the MMoE mechanism is introduced to improve the sensitivity of the voiceprint feature model to different pronunciations, the pronunciation manner of the speaker is modeled, and the system accuracy is improved.

[0162] The voiceprint feature extraction model is only trained to obtain the phoneme-level speaker embedding vector, instead of modeling each phoneme separately, so that the system is more efficient and consumes less resources; and the MMoE is introduced for the first time, the sensitivity of the voiceprint feature extraction model to different pronunciations is improved through the multi-task training of the phoneme-level speaker recognition and phoneme-level phoneme recognition, the discrimination of the generated speaker embedding vector is improved, and finally the accuracy of the voiceprint recognition is improved.

[0163] Figure 7 A structural schematic diagram of an electronic device is provided for the embodiments of the application. Figure 7 As shown in the figure, a structural schematic diagram of an electronic device 700 suitable for implementing the embodiments of the application is shown.

[0164] As shown in the figure, Figure 7 The electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or programs loaded from a storage portion 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0165] The following components are connected to the I / O interface 705: an input part 706 including a keyboard, a mouse, and the like; an output part 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 708 including a hard disk, and the like; and a communication part 709 including a network interface card such as a LAN card, a modem, and the like. The communication part 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 710 as necessary, so that a computer program read out therefrom is installed in the storage part 708 as necessary.

[0166] In particular, according to embodiments of the present disclosure, the processes described above with reference to Figure 1 may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for executing the above-described voiceprint recognition method based on phoneme information. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 709, and / or installed from the removable medium 711.

[0167] The flow diagrams and the block diagrams in the drawings are illustrations of possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0168] The units or modules involved in the embodiments described in the present application can be implemented by means of software, or can be implemented by means of hardware. The units or modules described can also be arranged in a processor. The names of these units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0169] The systems, apparatuses, modules or units in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a notebook computer, a mobile phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0170] It should be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0171] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

Claims

1. A voiceprint recognition method based on phoneme information, characterized in that, The method includes: Endpoint detection is performed on the audio to be recognized to obtain valid speech segments; The effective speech segments are identified and phoneme-level speech segments are determined by a pre-trained automatic speech recognition model. Based on the phoneme labeling results, it is determined whether all the phonemes labeled in the valid speech segment meet the phoneme coverage condition. If the phoneme coverage condition is met, then phoneme-level voiceprint recognition is used; if the phoneme coverage condition is not met, then sentence-level voiceprint recognition is used. The phoneme-level voiceprint recognition includes: Input all the phoneme-level speech segments of each phoneme into the phoneme-level voiceprint feature extraction network to obtain the phoneme-level embedding vector corresponding to each phoneme-level speech segment. For each phoneme, all the phoneme-level embedding vectors are aggregated to obtain the final embedding vector for each phoneme. The final embedding vectors of all the phonemes are set into phoneme-level voiceprint features; The phoneme-level voiceprint features are compared with the corresponding voiceprint features in the phoneme-level voiceprint database. The comparison results of all phonemes are summarized, and the voiceprint recognition result is obtained through voting. The phoneme-level voiceprint feature extraction network was trained using a multi-gated hybrid expert mechanism. The multi-gated hybrid expert mechanism includes three expert sub-networks; The multi-gated hybrid expert mechanism is used to remap the embedding vectors of sub-tasks to different feature subspaces, including: The phoneme-level embedding vector output by the phoneme-level voiceprint feature extraction network is projected to different subspaces through the following transformations: in, Latent vectors for the phoneme recognition subtask. Late vector for speaker identification subtask Subtask classification layer, To adjust the gating network for different contributions to subtasks: For expert sub-networks: in, These are the learnable parameters.

2. The method according to claim 1, characterized in that, The effective speech segments are identified through a pre-trained automatic speech recognition model, which performs phoneme labeling and determines the phoneme-level speech segments within the effective speech segments, including: A pre-trained automatic speech recognition model is used to determine the posterior probability of each speech frame in the effective speech segment with respect to different phoneme labels; Based on the maximum posterior probability of each speech frame, effective speech segments of individual phonemes are obtained; Filter out speech segments marked with silence and phoneme segments whose total number of frames is less than the length threshold from the effective speech segments to obtain phoneme-level speech segments.

3. The method according to claim 2, characterized in that, The step of segmenting a single effective speech segment based on the maximum posterior probability of each speech frame includes: For each speech frame, the phoneme with the highest posterior probability is selected as the most likely phoneme. If the corresponding posterior probability is greater than a preset confidence threshold, then the most likely phoneme is the phoneme label for that speech frame; otherwise, it is filled with a silence marker. By aggregating consecutive speech frames with the same phoneme tag, a valid speech segment of a single phoneme can be obtained.

4. The method according to claim 1, characterized in that, The phoneme coverage conditions include: When the frequency of a phoneme appearing in the valid speech segment is greater than the probability threshold, the phoneme is determined to be a valid phoneme; and The number of valid phonemes is greater than the preset number.

5. The method according to claim 4, characterized in that, The frequency of a phoneme appearing in a valid speech segment is determined based on the number of phonemes contained in the valid speech segment and the total number of phonemes in the valid speech segment.

6. The method according to claim 1, characterized in that, The sentence-level voiceprint recognition includes: A sentence-level voiceprint feature extraction network is used to extract the sentence-level embedding vectors corresponding to the effective speech segments; Aggregate all the sentence-level embedding vectors to obtain sentence-level voiceprint features; The sentence-level voiceprint features are compared with the corresponding voiceprint features in the sentence-level voiceprint database. The comparison results of all valid speech segments are summarized, and the voiceprint recognition result is obtained through voting.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the voiceprint recognition method based on phoneme information as described in any one of claims 1-6.

Images