A method for voiceprint recognition based on improved deep learning

Abstract

The application is a method based on improved deep learning voiceprint recognition, belonging to the technical field of voiceprint recognition, comprising the steps of data collection and preprocessing, model construction, voiceprint recognition, failure cause detection, age detection, voiceprint recognition result evaluation and the like. The application improves voiceprint recognition, introduces a data arrangement method of priority queue, so that the speed is increased, the user voice is recognized, the accuracy of voice recognition is improved, the distinguishability of voiceprint is increased, and the success or failure of voiceprint recognition is fed back by using an auxiliary standard of age detection. The model improves the security of voiceprint recognition, so that the safety of voiceprint recognition is more guaranteed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of voiceprint recognition, and relates to an error correction and detection and self-adaptive technology, in particular to a process method based on an improved voiceprint recognition model. BACKGROUND

[0002] Voiceprint recognition technology is a technology based on speech recognition and language model, which can recognize the voice features of individuals and associate them with personal identity information. It has very effective application value in many fields, such as personal identity verification, voice assistants, medical technology and police security monitoring. Speech signals are a kind of information body containing rich physiological and behavioral characteristics, and the physiological and behavioral characteristics can be used to distinguish the unique voice of individuals. Biological characteristics include the structure, size, shape, age and personality of the individual's vocal cords, which are important factors affecting a person's language ability; behavioral characteristics may be influenced by language, culture, social background, personal experience, personal preferences and personal emotions. These characteristics are combined together and reflected in the speech signal, making each person's voice have unique individual characteristics. There are many factors that affect voice recognition. At present, in the field of speech recognition involving physical characteristics (such as age difference), it is still difficult to accurately and flexibly recognize the voice information of different people. Therefore, it is crucial to compare in the voice library, analyze whether the voiceprint is affected by age change, and make corresponding processing according to this factor.

[0003] The human voice also has a complex production process. First, the gas from the lungs is controlled by the laryngeal cavity and the vocal cords vibrate to produce the human fundamental tone, and then the sound changes under the joint action of the organs such as the mouth, nose, pharynx and trachea, and finally becomes the sound heard by people. Therefore, voiceprint recognition has sufficient complex conditions, convenient access, low cost of data collection equipment, and only needs a microphone and other speech collection equipment to record data. The voice recognition technology is currently relatively mature, such as the commonly used module matching method, nearest neighbor method, neural network method, VQ clustering method, etc. However, if we pursue more efficient and fast recognition, we still need to improve the speed of voiceprint recognition. In this regard, there is no very mature academic research summary at present, so the present application seeks to find a solution to improve the speed of voice recognition, which can make frequently used audio users be recognized faster than before, thereby improving the efficiency and accuracy of voiceprint recognition.

[0004] Therefore, the present application improves the voiceprint recognition method process, proposes a voiceprint recognition method based on deep learning, which can better and more effectively improve the accuracy of voiceprint recognition, so that the voiceprint recognition technology can be better applied in people's daily life. SUMMARY

[0005] In view of the above problems, the application provides a method for voiceprint recognition based on improved deep learning, which improves the accuracy of voiceprint recognition by using a priority queue and a backend correction method.

[0006] The technical scheme adopted by the application is as follows:

[0007] A method for voiceprint recognition based on improved deep learning, comprising the following steps:

[0008] S1, data collection and preprocessing: collecting voice signals, pre-emphasizing the voice signals, speech framing, short frame speech windowing, and silence detection; and dividing into a training set, a validation set and a test set;

[0009] S2, model construction: an x-vector system based on a TDNN neural network model is constructed, and the preprocessed voice signals are input into the model for training;

[0010] S3, voiceprint recognition: input a test audio, perform voiceprint recognition, use a priority queue in the form of a large top heap to store pre-stored voice samples, each node in the heap represents a voice sample, and the priority of the node is determined by the similarity of the sample and the test voice, the higher the similarity, the higher the priority, and the closer the node to the top of the heap; the lower the similarity, the lower the priority, and the closer the node to the bottom of the heap; the test voice is first compared with the top voice, and then sequentially compared downward in order until the best match is found or the stopping condition is met; during the comparison process, if a new sample is found to have a higher similarity with the test voice, the heap structure needs to be updated and the priority needs to be adjusted, and a downward adjustment algorithm is used to maintain the properties of the heap;

[0011] S4, if the voiceprint recognition fails, detecting the failure reason and informing the user of the failure reason;

[0012] S5, if the voiceprint recognition is successful, using an SVC classifier to detect the age, if the difference between the detected age and the recorded age is within ±3 years, the detection is considered successful, and if the difference between the detected age and the recorded age exceeds ±3 years, the detection is considered failed;

[0013] S6, voiceprint recognition result evaluation: if the age detection is successful, the new sample is compared with the historical sample, if the average similarity between the new sample and the historical sample is between 75% and 80%, the new sample feature and the historical sample feature are weighted and averaged according to the weight to obtain a new voice sample, if the average similarity is less than 75%, the user is reminded to re-enter, and the original historical sample is replaced, if the similarity is higher than 80%, the historical sample is kept unchanged.

[0014] Further, the step S2 specifically comprises the following steps:

[0015] S21, input features: input is frame-level acoustic features, take 20-30ms speech frame, step 10ms, feature dimension is 40-60 dimensions, including first-order, second-order difference;

[0016] S22, extract dynamic features: TDNN captures the dynamic features of speech through local time context window;

[0017] S23, calculate global features: convert variable-length frame-level features into fixed-dimension speaker features; calculate the mean and standard deviation of all frames output by TDNN, and splice them as global features;

[0018] S24, dimension reduction processing: the pooled features pass through multiple fully connected layers, gradually reducing the dimension, and the output of the last layer is x-vector, which is used as the embedding vector of the speaker;

[0019] S25, training and reasoning: when training, a Softmax classification layer is connected after x-vector to predict the speaker ID, and when reasoning, the classification layer is discarded, and x-vector is directly used for similarity calculation.

[0020] Further, the step S3 specifically comprises the following steps:

[0021] S31, calculate the similarity between the to-be-tested speech and the pre-stored speech in the speech library

[0022] The loudness normalization is performed by using the following formula:

[0023]

[0024] y' = G*y

[0025] Wherein, G is a gain factor, y is the original audio signal, LUFS(y) is the integrated loudness of the audio signal, in LUFS unit, L target is the target loudness level, and y' is the normalized audio signal;

[0026] Mel normalization is performed by using the following formula

[0027]

[0028] Wherein, M' t,f is the normalized mel spectrum, M t,f is the original mel spectrum, M min is the minimum value of the mel spectrum, and M max is the maximum value of the mel spectrum;

[0029] The similarity is calculated by using the following formula

[0030] combined_similarity = ω1 x similarity1 + ω2 x similarity2 +... + ω n x similarity n

[0031] where w i is the weight of the i-th similarity measure, similarity i is the result of the i-th similarity measure, and combined_similarity is the final result of the similarity.

[0032] S32, using a large heap to manage the priority of pre-stored voices

[0033] S321, feature extraction: extracting feature vectors from pre-stored voices and test audio;

[0034] S322, building a large heap: initializing a large heap containing all pre-stored voices and their initial similarity scores;

[0035] S323, calculating similarity and updating the heap: calculating the similarity of each pre-stored voice with the test audio and updating the heap according to the similarity;

[0036] S324, comparison: taking nodes from the top of the heap one by one for comparison until a matching speaker is found or all nodes are traversed.

[0037] Further, the step S4 includes noise detection, decibel detection and silence detection.

[0038] Further, step S6 said the new sample features and historical sample features are weighted and averaged according to the weight to obtain a new sound sample, the specific weighting formula is as follows:

[0039]

[0040] ω hist = 1 - ω new

[0041] updated_features = ω new · new_features + ω hist · hist_features

[0042] where ω new is the weight of the new sample, ω hist is the weight of the historical sample, match_probaility is the average similarity of the new sample and the historical sample, new_features is the feature extracted from the new sample, and hist_features is the feature of the historical sample.

[0043] Advantages:

[0044] 1、The voiceprint recognition is improved, the data arrangement method of priority queue is introduced, speed is increased, user voice is recognized, the accuracy of voice recognition is improved, and the distinguishability of voiceprint is increased.

[0045] 2、The sound recognition is detected, if the sound recognition fails, the model can accurately point out which aspects of the user's sound recording need to be improved, the sound may be changed by the outside world, and therefore the model can avoid external reason errors, thereby improving the matching degree of the recognized sound.

[0046] 3、The back end of the voiceprint recognition can be corrected, the voiceprint recognition is corrected after success, the age is filled in during recording, the auxiliary standard of age is used to feed back whether the voiceprint recognition is successful, the safety of the voiceprint recognition is improved, and the safety of the voiceprint recognition is more guaranteed.

[0047] 4、The voiceprint recognition is self-adaptive, considering that the sound corresponding to the age will change, in the model, the sound is not constant, has updating capability, the system can update the sound change, thereby improving the variability of the sound, so that the sound can be used more persistently, thereby improving the experience of users. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 It is the overall flow chart of the present application.

[0049] Figure 2 It is the SVC structure diagram used in embodiment 6. DETAILED DESCRIPTION

[0050] The present application will be described in more detail below with reference to the accompanying drawings, and specific embodiments are given:

[0051] Embodiment 1 The overall process of the present application

[0052] As shown in Figure 1 The present application comprises the following steps:

[0053] S1, collect voice samples for data preprocessing, pre-emphasize the voice signal, frame the voice, window the short frame voice, etc.

[0054] S2, construct an x-vector system based on a TDNN neural network model, input the preprocessed voice sample into the model for training.

[0055] S3, input a test audio, perform voiceprint recognition, compare in the model in the way of priority queue, and establish a large top heap sequence about the pre-stored voice by adjusting the algorithm downward. Make the test voice compare with the top voice first, and then compare in order, complete the voiceprint recognition.

[0056] S4, if the voiceprint recognition fails, detect the failure reason and inform the user.

[0057] S5, if the voiceprint recognition is successful, perform age detection, evaluate the voiceprint recognition result, if the recorded age in the voiceprint library is different, the comparison fails, if the recorded age in the voiceprint library is the same, the comparison succeeds.

[0058] S6, if the comparison succeeds, compare the voice with the historical sample, if the average similarity between the current voice and the historical sample is between 75% and 80%, the new sample feature and the historical sample feature are weighted and averaged according to the weight, and a new voice sample is obtained; if it is lower than 75%, the user is reminded to re-enter and replace the original historical sample.

[0059] Example 2: Preprocessing of voice sample

[0060] In step S1, the collected voice sample is preprocessed, specifically including: pre-emphasis, which is to reduce the noise in the voice sample, so that the signal energy is as evenly distributed as possible in different frequency bands; frame division, which is to divide the continuous voice sample into equal-length sub-frames for short-time analysis of the signal; and windowing operation, which is to multiply the original voice frame waveform and the window function point by point to make the signal energy and the real spectrum more close.

[0061] Example 3: Model construction and training

[0062] Step S2 specifically includes the following steps:

[0063] S21, input features: input frame-level acoustic features (such as MFCC or FBank), take 20-30ms voice frame, step 10ms. Feature dimension is 40-60 dimensions (including first-order and second-order difference).

[0064] S22, TDNN layer (time delay layer): TDNN captures the dynamic features of voice through local time context window.

[0065] S221, the time context window of each layer is gradually expanded, for example:

[0066] The first layer: the context window is [-2, +2] (a total of 5 frames).

[0067] The second layer: the window is [-3, +3] (a total of 7 frames).

[0068] The subsequent layers cover longer context step by step (e.g. 15 frames or more).

[0069] S222, using ReLU activation function and Batch Normalization to accelerate convergence.

[0070] S23, Statistics Pooling: Key module: convert variable-length frame-level features into fixed-dimension speaker features. Calculate the mean and standard deviation of all frames output by TDNN, and concatenate them as global features. For example: if the TDNN output is 512 dimensions, the pooling is 1024 dimensions (512 mean + 512 standard deviation).

[0071] S24, Dense Layers: The pooled features are passed through multiple fully connected layers (e.g. 2-3 layers) to gradually reduce the dimension. The output of the last layer is an x-vector (usually 128-512 dimensions), which is used as the speaker embedding vector.

[0072] S25, Classification Layer: During training, a Softmax classification layer is added after the x-vector to predict the speaker ID. During inference, the classification layer is discarded and the x-vector is used directly for similarity calculation.

[0073] Embodiment 4: Voiceprint recognition

[0074] Step S3 specifically includes the following steps:

[0075] S31, calculate the similarity between the to-be-tested voice and the pre-stored voice in the voice library

[0076] S311, normalize the audio scale from both Mel and loudness aspects

[0077] Loudness normalization (based on ITU-R): Use integrated loudness as the measurement index, the formula is as follows:

[0078]

[0079] y' = G y

[0080] Where G is the gain factor, y is the original audio signal, LUFS(y) is the integrated loudness of the audio signal (in LUFS units), L target is the target loudness level (usually -23 LUFS), and y' is the normalized audio signal.

[0081] Mel normalization: Calculate the maximum and minimum values of the Mel spectrum, and use the formula to normalize. Where M't,f is the normalized mel-spectrogram, M t,f is the original mel-spectrogram, M min is the minimum value of the mel-spectrogram, M max is the maximum value of the mel-spectrogram.

[0082] S312 Calculate similarity

[0083] Formula: combined_similarity = ω1 × similarity1 + ω2 × similarity2 +... + ω n × similarity n

[0084] Where: w i is the weight of the i-th similarity measure, similarity i is the result of the i-th similarity measure, combined_similarity is the final result of the similarity.

[0085] In this paper, combined_similarity = 60% y' + 40% M' t,f .

[0086] S32, Use a large heap to manage the priority of pre-stored voice

[0087] S321, Feature extraction: extract feature vectors from pre-stored voice and test audio.

[0088] S322, Build a large heap: initialize a large heap containing all pre-stored voices and their initial similarity scores.

[0089] S323, Calculate similarity and update heap: calculate the similarity between each pre-stored voice and the test audio, and update the heap according to the similarity.

[0090] S324, Comparison: take out the nodes from the top of the heap one by one for comparison until the matching speaker is found or all nodes are traversed.

[0091] Embodiment 5 Failure reason detection

[0092] Step S4 Failure reason detection specifically includes the following steps:

[0093] S41, Noise detection

[0094] Preprocess the data and calculate the signal-to-noise ratio. The formula for measuring signal-to-noise ratio (SNR) in Python is usually based on the ratio of signal power to noise power, as follows:

[0095]

[0096] where P signal is the power of the signal, P noise is the power of the noise. Power is usually calculated by the mean square value of the signal, which is:

[0097]

[0098] where x[i] is the sample of the signal or noise, and N is the total number of samples. If the signal-to-noise ratio is less than 20 dB, output: the noise is too large, please ensure a quiet environment and re-enter.

[0099] S42, decibel detection

[0100] S421 sets the sampling rate (such as 16 kHz) and the sampling bit number (such as 16 bits).

[0101] S422 preprocessing: frame division: frame processing of the audio signal (such as 20 ms per frame); windowing: windowing of each frame signal (such as Hamming window) to reduce spectral leakage.

[0102] S423 volume calculation, calculate the frame energy:

[0103]

[0104] where x[i] is the sample in the frame, and N is the frame length.

[0105] Calculate the volume (decibel):

[0106] Volume (dB) = 10·log 10 (E)

[0107] S424 volume evaluation

[0108] According to the volume size, it is judged whether it meets the requirements: the volume is too low: Volume < 40 dB, then output: the sound is too small, please increase the volume and re-enter; the volume is normal: Volume ≥ 40 dB.

[0109] S43, short-time energy-based silence detection

[0110] Preprocess the data

[0111] S431, short-time division: divide the audio signal into multiple short-time windows.

[0112] S432, calculate the short-time energy: calculate the energy of the signal in each window, the formula is as follows:

[0113]

[0114] where x[i] is the sample of the audio signal, and N is the number of samples in the window.

[0115] S433, threshold judgment: compare the calculated energy with the preset threshold, if the energy is lower than the threshold, it is considered that the window is mute.

[0116] S434. Calculate the proportion of effective audio segment in total audio segment, the formula is as follows:

[0117]

[0118] Where, t i is the time length of the effective audio segment, T is the total audio segment time length. If d is less than 50%, output: the effective audio is too short, please input again.

[0119] Example 6 age detection

[0120] Step S5 specifically includes the following steps:

[0121] S51, first use the SVC (Support Vector Classification) classifier for age detection, and correct the voiceprint recognition result. The structure diagram of SVC is as Figure 2 shown. The SVC classifier is a classification algorithm based on support vector machine (SVM), and the support vector machine (SVM) is a supervised learning algorithm for classification tasks, and the core goal is to find an optimal decision boundary, i.e. hyperplane, which can maximize the margin between the two classes. The margin refers to the distance between the decision boundary and the nearest sample point, i.e. the distance between the decision boundary and the support vector. SVM ensures that this margin is as large as possible through optimization algorithm, thereby enhancing the generalization ability of the model.

[0122] S52, tolerance mechanism

[0123] Tolerance range:

[0124] If the difference between the predicted age and the recorded age is within ±3 years, the comparison is successful, i.e. the comparison success condition = |predicted age-recorded age| ≤ 3, if the difference between the predicted age and the recorded age is outside ±3 years, the comparison fails.

[0125] Example 7 voiceprint recognition result evaluation

[0126] Step S6 specifically includes the following steps:

[0127] S61 define weight allocation strategy

[0128] According to the average similarity match_probaility between the current sound and the historical sample, the weights of the new sample and the historical sample are dynamically adjusted. Assuming that the weight of the new sample is ω newThe lower the average similarity between the current sound and the historical samples, the higher the weight of the new sample, and the weight of the historical sample is ω hist The higher the similarity, the higher the weight of the historical sample. If the similarity is between 75% and 80%, the weight is defined as follows:

[0129]

[0130] ω hist = 1-ω new

[0131] S62 update the historical sample

[0132] The new sample features and the historical sample features are weighted and averaged according to the weight to generate a new sound sample. Assuming that new_features is the feature extracted from the current sound, and hist_features is the feature of the historical sample. The formula is as follows:

[0133] updated_features = ω new ·new_features + ω hist ·hist_features.

[0134] If the similarity is less than 75%, the user is prompted to re-enter, and if the similarity is greater than 80%, the historical sample feature remains unchanged.

Claims

1. A method for improved deep learning-based voiceprint recognition, comprising the following steps: S1, data collection and preprocessing: collecting voice signals, pre-emphasizing, speech framing, short frame speech windowing, and silence detection are performed on the voice signals; and the voice signals are divided into a training set, a validation set and a test set; S2, model construction: an x-vector system is constructed based on a TDNN neural network model, and the preprocessed voice signals are input into the model for training; S3, voiceprint recognition: a test audio is input, and voiceprint recognition is performed; a priority queue in the form of a large top heap is used to store pre-stored voice samples; each node in the heap represents a voice sample, and the priority of the node is determined by the similarity of the sample and the test voice; the higher the similarity, the higher the priority, and the closer the node to the top of the heap; the lower the similarity, the lower the priority, and the closer the node to the bottom of the heap; the test voice is first compared with the voice at the top of the heap, and then sequentially compared with the voices below in order until the best match is found or the stop condition is met; during the comparison process, if a new sample is found to have a higher similarity with the test voice, the heap structure needs to be updated and the priority needs to be readjusted; a downward adjustment algorithm is used to maintain the properties of the heap; S4, if the voiceprint recognition fails, the failure reason is detected, and the user is informed of the failure reason; S5, if the voiceprint recognition is successful, an SVC classifier is used for age detection; if the difference between the detected age and the recorded age is within ±3 years, the detection is considered successful; if the difference between the detected age and the recorded age exceeds ±3 years, the detection is considered failed; S6, voiceprint recognition result evaluation: if the age detection is successful, the new sample is compared with the historical sample; if the average similarity between the new sample and the historical sample is between 75% and 80%, the new sample feature and the historical sample feature are weighted and averaged according to the weight to obtain a new voice sample; if the average similarity is less than 75%, the user is reminded to re-enter, and the original historical sample is replaced; if the similarity is higher than 80%, the historical sample remains unchanged.

2. The method of claim 1, wherein, The step S2 specifically comprises the following steps: S21, input feature: input acoustic features at the frame level, take 20-30ms voice frames, step length 10ms, feature dimension 40-60 dimensions, including first-order and second-order differences; S22, extract dynamic features: TDNN captures dynamic features of voice through local time context window; S23, calculate global features: convert variable-length frame-level features into fixed-dimension speaker features; calculate the mean and standard deviation of all frames output by TDNN, and concatenate them as global features; S24, dimension reduction processing: the pooled features are processed through multiple fully connected layers, and the dimension is gradually reduced; the output of the last layer is x-vector, which is used as the embedding vector of the speaker; S25, training and reasoning: during training, a Softmax classification layer is connected after x-vector to predict the speaker ID; during reasoning, the classification layer is discarded, and x-vector is directly used for similarity calculation.

3. The method of claim 1, wherein, The step S3 specifically comprises the following steps: S31, calculate the similarity between the test voice and the pre-stored voice in the voice library The following formula is used for loudness normalization: y' = G y where G is a gain factor, y is the original audio signal, LUFS(y) is the integrated loudness of the audio signal in units of LUFS, L target is the target loudness level, and y' is the normalized audio signal. Mel normalization is performed using the following formula Among them, M′ t,f It is the normalized Mel spectrum, M t,f It is the original Mel spectrum, M min It is the minimum value of the Mel spectrum, M max It is the maximum value of the Mel spectrum; Similarity is calculated using the following formula combined_similarity = ω1 x similarity1 + ω2 x similarity2 +... + ω n x similarity n wherein: w i is the weight of the i-th similarity measure, similarity i is the result of the i-th similarity measure, combined_similarity is the final result of the similarity. S32, using a max heap to manage the priority of pre-stored voices S321, feature extraction: extract feature vectors from pre-stored voices and test audio; S322, build a max heap: initialize a max heap containing all pre-stored voices and their initial similarity scores; S323, calculate similarity and update the heap: calculate the similarity of each pre-stored voice with the test audio and update the heap according to the similarity; S324, perform comparison: take nodes from the top of the heap one by one for comparison until a matching speaker is found or all nodes are traversed.

4. The method of claim 1, wherein, The step S4 includes noise detection, decibel detection and silence detection.

5. The method of claim 1, wherein the method is based on improved deep learning speaker recognition. The step S6 includes weighting and averaging the new sample features and historical sample features according to the weight to obtain a new sound sample, and the specific weighting formula is as follows: ω hist = 1 - ω new updated_features = ω new • new_features + ω hist • hist_features where ω new is the weight of the new sample, ω hist is the weight of the historical sample, match_probaility is the average similarity between the new sample and the historical sample, new_features is the feature extracted from the new sample, and hist_features is the feature of the historical sample.

Images