Noise robust voice endpoint detection method and system
By introducing the Nospeech noise classification layer and attention weighting mechanism into the speech endpoint detection model, combining gradient inversion and smooth merge operation, the speech endpoint detection problem in scenarios with high noise is solved, and the accuracy of speech signal endpoint detection is improved and Identify effects.
Patent Information
- Application Number
- CN202411217935.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-06
AI Technical Summary
The effect of existing voice endpoint detection technology has decreased in scenarios with high noise, resulting in an increase in insertion and deletion errors in subsequent voice recognition tasks, the transliteration content is unavailable, and the effect of voice detection has decreased significantly.
Using the voice endpoint detection model of the cascading CNN, BILSTM, attention layer, DNN layer and Speech and Nospeech classification layers, the Nospeech noise classification layer was introduced in the training stage, and the feature extraction and the Classify the effects and perform a smooth merge operation during the test phase to reduce glitches in the identification results.
Effectively distinguish Speech and Nospirech intervals in high noise scenarios, improving the accuracy of voice signal endpoint detection and the effect of subsequent recognition tasks, and avoiding continuous glitch interleaving of recognition results.
Smart Images
Figure CN119091933A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech recognition, and in particular to a noise-robust speech endpoint detection method and system. Background Art
[0002] After decades of development and improvement, various detection algorithms have emerged. Some of them perform well in certain environments, while others perform well under certain conditions. In summary, speech endpoint detection can be roughly divided into the following categories: methods based on time domain features, methods based on transform domains (frequency domain, cepstrum domain, time-frequency domain), methods based on model matching, and methods based on neural networks.
[0003] With the promotion and application of deep learning methods, endpoint detection technology based on neural networks has become the mainstream algorithm. This method uses speech and no-speech training corpus to train a binary classification model to distinguish whether the current frame signal is valid speech. Then, the recognition result of the entire speech signal is regularized and smoothed to obtain the time interval of valid speech. These valid speech segments can be sent to subsequent models for attribute or text content recognition. Figure 1 The figure shows the structure of a typical speech endpoint detection network. The deep neural network used in the endpoint detection system is the DNN (Deep Neural Network). During training, the fb (filterbank) feature parameters of the speech are input into the DNN network. The DNN network transforms the features and then connects to the fully connected layer, and then gives the Speech and Nospeech classification results at the frame level.
[0004] These technologies work well for language endpoint detection in scenarios where the speech signal-to-noise ratio is high and the background noise is low. However, when the background noise is large and the signal-to-noise ratio is small, the effect of endpoint detection will drop significantly. For example, noisy segments will be classified as speech, and noisy speech segments will be classified as nospeech, resulting in a significant drop in the effectiveness of subsequent recognition tasks. For example, in continuous speech recognition, errors such as insertions and deletions will increase dramatically, and the transcribed content will become unusable. The effect of speech detection will drop significantly, greatly reducing the effectiveness of subsequent speech attribute and content recognition. Summary of the invention
[0005] The object of the present invention is to provide a noise-robust speech endpoint detection method and system to solve the speech endpoint detection problem in scenes with high noise in the background technology.
[0006] The technical solution to achieve the purpose of the present invention is:
[0007] A noise-robust speech endpoint detection method, comprising:
[0008] Step 1: Build a speech endpoint detection model, including cascaded CNN, BILSTM, attention layer, DNN layer, and Speech and Nospeech classification layers;
[0009] Step 2: After the BILSTM of the speech endpoint detection model, a Nospeech noise classification layer is introduced as a network training model. The introduced Nospeech noise classification layer is a branch connected in parallel with the attention layer, the DNN layer, and the Speech and Nospeech classification layers. The network training model is iteratively trained until the training stop condition is met to obtain a trained speech endpoint detection model.
[0010] Step 3: Get the frame-level binary classification results using the trained speech endpoint detection model.
[0011] Further, the attention layer is:
[0012] a c =tanhh(WX c (:))
[0013]
[0014]
[0015] Among them, X c (:) is the feature of the cth channel in feature X. The feature dimension size is T×1. There are C channels in total and there are C T×1 convolutional features. is the weighted representation vector, T is a fixed value, a c is the calculated value of the cth channel, w c is the superposition weight of the cth channel, and W is a 1×T vector.
[0016] Furthermore, the T=10.
[0017] Further, the iterative training of the speech endpoint detection model through the Nospeech noise classification layer specifically includes:
[0018] Step 2-1, collect Speech and Nospeech data as training data sets;
[0019] Step 2-2, determine two batches, randomly select K1 categories of Nospeech types as one batch of Nospeech labeled data, randomly select K2 data of each Nospeech type, the product of K1 and K2 is M, and the other batch is M number of Speech labeled data, extract the fb features of the two batches of labeled data;
[0020] Step 2-3: Input the features of the two batches into the network model for training. During the training, after the feature data of the two batches pass through CNN and BILSTM, only the Nospeech data is sent to the introduced Nospeech noise classification layer. The data of the two batches will then pass through the attention layer and the DNN layer and finally be sent to the Speech and Nospeech classification layers for binary classification. The introduced Nospeech noise classification layer performs gradient flipping on the classification results and returns them to BILSTM and CNN.
[0021] Step 2-4, repeatedly iterate step 2-2 to step 2-3 until the loss function of the network training model is stable or the maximum number of iterations N is reached.
[0022] Furthermore, the loss function of the network training model is:
[0023]
[0024] CELoss class1 Corresponding to the Speech and Nospeech classification layers, CELoss class1 (n) represents the classification loss calculated for the nth sample; CELoss class2 Corresponding to the introduced Nospeech noise classification layer, CELoss class2 (i) represents the classification loss of the i-th sample, and α is the setting parameter.
[0025] Further, α is set to 0.1.
[0026] Furthermore, the duration of the Speech and Nospeech data collected in step 2-1 is not less than 20 hours.
[0027] Further, the step 3 specifically includes:
[0028] Step 3-1, extracting fb features from the speech to be recognized;
[0029] Step 3-2: Input the fb feature into the trained speech endpoint detection model to obtain the frame-level Speech and Nospeech recognition results of the entire audio;
[0030] Step 3-3 smoothly merges the frame-level Speech and Nospeech recognition results to obtain the final recognition result.
[0031] Furthermore, the time interval threshold of the smooth merging operation is 0.2 seconds.
[0032] A noise-robust speech endpoint detection system, comprising:
[0033] A speech endpoint detection model building unit, used to build a speech endpoint detection model including a cascaded CNN, BILSTM, attention layer, DNN layer, and Speech and Nospeech classification layers;
[0034] The training unit introduces a Nospeech noise classification layer after the BILSTM of the speech endpoint detection model as a network training model. The introduced Nospeech noise classification layer is a branch connected in parallel with the attention layer, the DNN layer, and the Speech and Nospeech classification layers. The network training model is iteratively trained until the training stop condition is met, thereby obtaining a trained speech endpoint detection model.
[0035] The recognition unit obtains the frame-level binary classification results through the trained speech endpoint detection model, and uses smooth merging to obtain the endpoint detection results.
[0036] Different from the existing DNN-based speech signal endpoint detection technology, this solution implements a noise-robust speech signal endpoint detection technology, which can effectively solve the speech signal endpoint detection problem in the case of large background noise. The specific advantages are:
[0037] 1) In the training phase, a two-branch network was used, in which the gradient of the Nospeech classification branch was reversed, so that the Nospeech extracted by the common feature extraction network
[0038] The features are more stable, which is more conducive to the classification effect with the Speech and Nospeech classification networks;
[0039] 2) During the training phase, feature attention weighting is used for the Speech and Nospeech classification network branches, which is more conducive to the subsequent Speech and Nospeech classification;
[0040] 3) The smooth merging operation in the test phase avoids the phenomenon of continuous glitches in the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 A diagram of the structure of a typical voice endpoint detection network.
[0042] Figure 2 This is a diagram of the training structure for speech signal endpoint detection based on improved DNN.
[0043] Figure 3 This is a test structure diagram for speech signal endpoint detection based on improved DNN. DETAILED DESCRIPTION
[0044] This paper proposes a noise-robust speech detection method that can accurately distinguish between speech and nospeech intervals in high-noise scenes.
[0045] The speech detection method proposed in this case introduces an attention enhancement mechanism during the training phase, compared to the original DNN system, to improve the effect of speech and nospeech classification. In addition, a noise classification module is added to the training process. The noise classification module is used to distinguish noise categories, but the gradient is reversed in the network part that is fed back to feature extraction, so that the features of nospeech are as stable as possible, and there will be no large fluctuations due to different noise categories, which will affect the speech and nospeech classification of endpoint detection.
[0046] During the test phase, the noise classification module does not participate in the recognition task, and only the results of the Speech and Nospeech classification modules are used. Since the recognition result is a binary classification result at the frame level, more glitches may occur when the classification result is used directly, especially in the time period between Speech and Nospeech, the jitter result will increase. Therefore, this case adopts a merging and smoothing method to organize the recognition results and obtain the final Speech and Nospeech classification time interval.
[0047] The entire voice signal endpoint detection system is divided into two stages: training and testing, as follows:
[0048] 1. Training phase
[0049] Step 1-1: Business experts prepare training data and collect speech and nospeech data. Nospeech data is required to contain as many types as possible, such as silence, ringing sound, music, whistle, etc. The length of speech data should not be less than 20 hours, and the length of nospeech data should not be less than 20 hours.
[0050] Step 1-2: For the labeled data in step 1, extract the fb (filterbank) features.
[0051] Step 1-3: In Figure 1 The existing DNN network shown in the figure is connected to an intention layer, a smaller CNN and a BILSTM. As shown in the figure below, two batches of features are randomly sent to the network model for training. The Nospeech label data of one batch is obtained by randomly taking K1 categories of Nospeech types at a time, and randomly taking K2 data of each Nospeech type, so that the product of K1 and K2 is M. The number of the data type of the other batch is also M.
[0052] During training, after the feature data of the two batches pass through the CNN and BILSTM network modules, only the Nospeech data will be sent to the Nospeech linear classification layer, and the data of the two batches will pass through the attention layer and the DNN layer and finally be sent to the Speech and Nospeech linear classification layers for binary classification.
[0053] The attention layer mainly performs attention weighting operations:
[0054] The attention weights are as follows:
[0055] a c =tanhh(WX c (:))
[0056]
[0057]
[0058] Where X c (:) is the feature of the cth channel in feature X, and the feature dimension size is T×1. There are C channels in total, and there are C T×1 convolutional features. The contribution of these individual vectors to the subsequent recognition task is not equal. Simply splicing them into an embedding for classification is not the optimal solution. For this reason, we first pass these vectors through the attention layer before classification, and do a weighted representation vector It is sent as embedding to the subsequent recognition task to improve the recognition effect. When we do attention weighting, the size of T is generally a fixed value, such as 10.
[0059] The Nospeech linear classification layer is mainly used to distinguish different types of Nospeech. When its gradient is passed back to the BILSTM and CNN network modules during training, the gradient will be flipped to make the feature distribution of Nospeech feature data more uniform after passing through the CNN and BILSTM network modules, so as to avoid excessive differences in the feature distribution of different types of Nospeech. This will help the Speech and Nospeech linear classification layers to improve the classification effect.
[0060] Step 1-4: Set the joint training loss function to:
[0061]
[0062] CELoss class1 Corresponding to the Speech and Nospeech classification layers, CELoss class1 (n) represents the classification loss calculated for the nth sample; CELossclass2 Corresponding to the Nospeech classification layer, CELoss class2 (i) represents the classification loss of the i-th sample, in CELoss class2 The negative sign is used in front to achieve gradient reversal. α is a setting parameter, such as setting it to 0.1.
[0063] Step 1-5: Repeat steps 1-2 to 1-4 until the model Loss all When the model is stable or reaches the maximum number of iterations N, it is used to classify Speech and Nospeech.
[0064] 2. Identification stage
[0065] Step 2-1: Extract fb features from the speech to be recognized;
[0066] Step 2-2: Load the last generation of models obtained in step 1-5 of the training phase;
[0067] Step 2-3: Send the fb features to the model loaded in step 2-2 to obtain the frame-level Speech and Nospeech recognition results of the entire audio.
[0068] Step 2-4: Set the time interval threshold time_duration_thred for smooth merging, such as 0.2 seconds. Perform a smooth merging operation on the frame-level recognition results in step 2-3. This operation can merge the recognition results of the burr fragments into the most reasonable adjacent fragments, making the recognition results more reasonable and the recognition effect better; in addition, after merging the burr recognition results, if the recognition results of two adjacent fragments are consistent, then the recognition results of the adjacent fragments will be merged. The specific smooth merging method is as follows:
[0069] For example, a segment is identified as a 0.1 second Nospeech segment, and its adjacent segments before and after are 2 seconds of Speech and 4 seconds of Speech respectively. After smooth merging, these three adjacent segments are merged into a Speech segment, and the corresponding time interval is from the start time of the original first Speech segment to the end time of the original third Speech segment.
[0070] The above steps realize noise-robust speech signal endpoint detection.
[0071] This embodiment also provides a noise-robust speech endpoint detection system, including:
[0072] A speech endpoint detection model building unit, used to build a speech endpoint detection model including a cascaded CNN, BILSTM, attention layer, DNN layer, and Speech and Nospeech classification layers;
[0073] The training unit introduces a Nospeech noise classification layer after the BILSTM of the speech endpoint detection model as a network training model. The introduced Nospeech noise classification layer is a branch connected in parallel with the attention layer, the DNN layer, and the Speech and Nospeech classification layers. The network training model is iteratively trained until the training stop condition is met, thereby obtaining a trained speech endpoint detection model.
[0074] The recognition unit obtains the frame-level binary classification results through the trained speech endpoint detection model.
[0075] The present invention determines the starting point and the ending point of speech from a signal based on the input speech data. As the front end of speech attribute and content recognition, the detection effect of this technology directly affects the recognition effect of the back end. This scheme proposes an improved endpoint detection method for scenes with low speech signal-to-noise ratio and large background noise. Compared with the original DNN system, this method introduces an attention enhancement mechanism during the training process to improve the effect of Speech and Nospeech classification. In addition, the training process will also add a noise classification module, which is used to distinguish noise categories, but the network part that is fed back to feature extraction has a gradient inversion, so that the features of Nospeech are as stable as possible, and there will be no large fluctuations due to different types of noise, thereby affecting the Speech and Nospeech classification of endpoint detection.
[0076] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0077] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A noise-robust speech endpoint detection method, characterized in that: include: Step 1: Build a speech endpoint detection model, including cascaded CNN, BILSTM, attention layer, DNN layer, and Speech and Nospeech classification layers; Step 2: After the BILSTM of the speech endpoint detection model, a Nospeech noise classification layer is introduced as a network training model. The introduced Nospeech noise classification layer is a branch connected in parallel with the attention layer, the DNN layer, and the Speech and Nospeech classification layers. The network training model is iteratively trained until the training stop condition is met to obtain a trained speech endpoint detection model. Step 3: Get the frame-level binary classification results using the trained speech endpoint detection model.
2. A noise-robust speech endpoint detection method according to claim 1, characterized in that: The attention layer is: a c =tanh(WX c (:)) Among them, X c (:) is the feature of the cth channel in feature X. The feature dimension size is T×1. There are C channels in total and there are C T×1 convolutional features. is the weighted representation vector, T is a fixed value, a c is the calculated value of the cth channel, w c is the superposition weight of the cth channel, and W is a 1×T vector.
3. A noise-robust speech endpoint detection method according to claim 2, characterized in that: Said T=10.
4. The noise-robust speech endpoint detection method according to claim 1, characterized in that: The iterative training of the speech endpoint detection model through the Nospeech noise classification layer specifically includes: Step 2-1, collect Speech and Nospeech data as training data sets; Step 2-2, determine two batches, randomly select K1 categories of Nospeech types as one batch of Nospeech labeled data, randomly select K2 data of each Nospeech type, the product of K1 and K2 is M, and the other batch is M number of Speech labeled data, extract the fb features of the two batches of labeled data; Step 2-3: Input the features of the two batches into the network model for training. During the training, after the feature data of the two batches pass through CNN and BILSTM, only the Nospeech data is sent to the introduced Nospeech noise classification layer. The data of the two batches will then pass through the attention layer and the DNN layer and finally be sent to the Speech and Nospeech classification layers for binary classification. The introduced Nospeech noise classification layer performs gradient flipping on the classification results and returns them to BILSTM and CNN. Step 2-4, repeatedly iterate step 2-2 to step 2-3 until the loss function of the network training model is stable or the maximum number of iterations N is reached.
5. A noise-robust speech endpoint detection method according to claim 4, characterized in that: The loss function of the network training model is: CELoss class1 Corresponding to the Speech and Nospeech classification layers, CELoss class1 (n) represents the classification loss calculated for the nth sample; CELoss class2 Corresponding to the introduced Nospeech noise classification layer, CELoss class2 (i) represents the classification loss of the i-th sample, and α is the setting parameter.
6. A noise-robust speech endpoint detection method according to claim 5, characterized in that: α is set to 0.
1.
7. A noise-robust speech endpoint detection method according to claim 4, characterized in that: The duration of speech and nospeech data collected in step 2-1 is no less than 20 hours.
8. The noise-robust speech endpoint detection method according to claim 1, characterized in that: The step 3 specifically includes: Step 3-1, extracting fb features from the speech to be recognized; Step 3-2: Input the fb feature into the trained speech endpoint detection model to obtain the frame-level Speech and Nospeech recognition results of the entire audio; Step 3-3 smoothly merges the frame-level Speech and Nospeech recognition results to obtain the final recognition result.
9. A noise-robust speech endpoint detection method according to claim 8, characterized in that: The time interval threshold of the smooth merging operation is 0.2 seconds.
10. A system for implementing the noise-robust speech endpoint detection method according to any one of claims 1 to 9, characterized in that: include: A speech endpoint detection model building unit, used to build a speech endpoint detection model including a cascaded CNN, BILSTM, attention layer, DNN layer, and Speech and Nospeech classification layers; The training unit introduces a Nospeech noise classification layer after the BILSTM of the speech endpoint detection model as a network training model. The introduced Nospeech noise classification layer is a branch connected in parallel with the attention layer, the DNN layer, and the Speech and Nospeech classification layers. The network training model is iteratively trained until the training stop condition is met, thereby obtaining a trained speech endpoint detection model. The recognition unit obtains the frame-level binary classification results through the trained speech endpoint detection model.