A text-aware based voice endpoint detection method and system

Abstract

This application proposes a text-aware speech endpoint detection method and system. The method includes the following steps: S1, acquiring the original audio and extracting shallow acoustic features and text features from the original audio; S2, adaptively weighting and fusing the shallow acoustic features and text features to obtain fused audio features; S3, performing context encoding on the fused audio features to obtain temporal features; S4, performing binary classification prediction on the temporal features to obtain speech frame probabilities and non-speech frame probabilities; S5, making a decision on the binary classification prediction results and concatenating the decision results of each frame of the original audio to obtain the speech endpoint detection result. This method can effectively utilize the text information in the audio. By fusing shallow acoustic features and text features, it can improve the accuracy and stability of the backend speech endpoint detection model. In the absence of target domain training data, the pre-trained text feature extraction sub-network can ensure the robustness of the speech endpoint detection model.

Description

Technical Field

[0001] This application belongs to the field of speech processing technology, specifically relating to a speech endpoint detection method and system based on text perception. Background Technology

[0002] Voice Activity Detection (VAD) is an important branch of speech processing technology, playing a crucial role in speech tasks such as Automatic Speech Recognition (ASR), Speaker Recognition (ASV), and Speaker Diarization (SD). The purpose of VAD is to identify speech time segments and non-speech time segments in the original audio. Incorrect VAD results often severely interfere with downstream speech tasks, affecting the performance and stability of the speech system. Accurate VAD methods can effectively reduce noise interference and improve the robustness of the backend speech system.

[0003] Current speech endpoint detection methods can be categorized into unsupervised and supervised approaches. Early speech endpoint detection models primarily relied on unsupervised heuristic algorithms. Unsupervised speech endpoint detection models lack robustness in real-world environments; for example, algorithms based on energy features and zero-crossing rate (ZCR) are easily affected by environmental noise. With the widespread application of speech processing technology, speech endpoint detection technology is facing increasingly diverse and complex application scenarios. Traditional algorithms struggle to predict nonlinear relationships and cannot meet practical application requirements. In complex scenarios, audio exhibits low signal-to-noise ratios, accompanied by background noise, foreground noise, and indoor reverberation. For instance, audience laughter and applause in a stand-up comedy scene, transitional music in a television program, and car horns in an outdoor setting are all non-speech segments. This necessitates speech endpoint detection models with high noise robustness and detection accuracy. In recent years, many neural network-based frameworks have been proposed for speech endpoint detection tasks. Pyannote is a commonly used supervised speech endpoint detection framework. It utilizes a front-end feature extraction module to extract shallow acoustic features, which are then fed into a Long Short-Term Memory (LSTM) network for contextual information encoding. The network outputs frame-level silence and non-silence discrimination results. However, neural network-based speech endpoint detection frameworks rely on a large amount of target domain data to train model parameters. In practical applications, the target domain training data depends heavily on manual annotation. Without matching target domain training data, the performance of neural network-based speech endpoint detection models deteriorates drastically in noisy environments.

[0004] Chinese patent application CN201910917881.6 discloses a deep learning-based speech endpoint detection method, which includes the following steps: collecting audio data and performing frame segmentation to divide it into non-noise speech frames containing speech and noise speech frames not containing speech, thereby generating training data; extracting audio feature vectors using a convolutional neural network and sequentially inputting them into a long short-term memory layer and a fully connected layer; finally, the model outputs the probability that each audio frame belongs to a non-noise speech frame or a noise speech frame. The performance of the aforementioned deep learning-based speech endpoint detection model relies on a large amount of timestamped corpus; however, timestamped target domain data is often scarce. When in-domain training data is lacking, the performance of traditional deep learning-based speech endpoint detection models deteriorates drastically. Summary of the Invention

[0005] To address the aforementioned problems, the first aspect of this application proposes a text-aware speech endpoint detection method, comprising the following steps:

[0006] S1, Obtain the original audio and extract the shallow acoustic and text features of the original audio;

[0007] S2, adaptively weightedly fuses shallow acoustic features and text features to obtain fused audio features;

[0008] S3 involves contextually encoding the fused audio features to obtain temporal features;

[0009] S4, perform binary classification prediction on temporal features to obtain the probability of speech frames and the probability of non-speech frames;

[0010] S5 makes a decision on the binary classification prediction results and splices the decision results of each frame of the original audio to obtain the speech endpoint detection results.

[0011] The above method, by fusing text features, can effectively detect speaker text segments in audio, achieving a highly accurate speech endpoint detection model.

[0012] Furthermore, shallow acoustic features are extracted using a convolutional neural network.

[0013] Furthermore, text features are extracted using the front-end subnetwork of the end-to-end speech recognition model. The end-to-end speech recognition model includes convolutional neural network layers, Transformer coding modules, and several fully connected layers. The front-end subnetwork is the network of the end-to-end speech recognition model except for the last fully connected layer, and the text features are the output of the front-end subnetwork.

[0014] Furthermore, the number of nodes in the last layer of the fully connected layer is consistent with the number of characters in the dictionary of the corresponding language model.

[0015] Furthermore, in the adaptive weighted fusion process, the lower the signal-to-noise ratio of the original audio, the greater the weight of the shallow acoustic features. By adopting an adaptive weighted fusion mechanism to adaptively adjust the weight values ​​of different features in different scenarios, the stability of the speech endpoint detection model in different environments can be improved.

[0016] Furthermore, the temporal features are obtained through a temporally aware neural network.

[0017] Furthermore, S5 employs a dual-state, dual-threshold flip-flop mechanism for decision-making. Different states correspond to different flip-flop thresholds, which can further suppress interference from noise signals.

[0018] A second aspect of this application proposes a speech endpoint detection system, comprising a speech endpoint detection sub-model and a binarizer, wherein...

[0019] The speech endpoint detection sub-model includes:

[0020] A feature extraction subnetwork is configured to acquire the raw audio and extract shallow acoustic and textual features from the raw audio.

[0021] The feature fusion subnetwork is configured to adaptively weight and fuse shallow acoustic features and text features to obtain fused audio features.

[0022] A temporal modeling subnetwork is configured to perform contextual encoding on fused audio features to obtain temporal features;

[0023] as well as,

[0024] A binary classification prediction subnetwork is configured to perform binary classification prediction on temporal features to obtain the probability of speech frames and the probability of non-speech frames.

[0025] The binarizer is configured to make decisions on the results of binary classification predictions and concatenate the decision results of each frame of the original audio to obtain the speech endpoint detection results.

[0026] Furthermore, the training process of the speech endpoint detection sub-model includes the following steps:

[0027] S901 uses a binary classification cross-entropy function to calculate the loss function for the prediction results and the manual labels;

[0028] S902, the loss value calculated based on the loss function is backpropagated to the speech endpoint detection sub-model to obtain the gradient value of the network parameters of each layer;

[0029] S903 updates the parameters of each network layer based on the gradient value to obtain optimized model parameters;

[0030] S904, repeat S901-S903 until the calculated loss value converges.

[0031] Furthermore, the binarizer employs a dual-state dual-threshold flipping mechanism for decision-making, where the threshold is determined through grid search.

[0032] This application proposes a text-aware speech endpoint detection method that improves the accuracy of the model in noisy environments and effectively alleviates the problem of supervised models lacking matching target domain training data. This method is a frame-level sequence-to-sequence end-to-end speech endpoint detection approach. It takes raw audio as input and outputs frame-level speech or non-speech label sequences. It effectively utilizes textual information in the audio; when speaker text is present in an audio frame, the probability that the frame belongs to a speech frame increases accordingly. By fusing shallow acoustic features and textual features, the accuracy and stability of the backend speech endpoint detection model can be improved. In the absence of target domain training data, a pre-trained text feature extraction sub-network ensures the robustness of the speech endpoint detection model. Attached Figure Description

[0033] The accompanying drawings are provided to aid in further understanding of this application. The elements in the drawings are not necessarily to scale. For ease of description, only the parts relevant to the invention are shown in the drawings.

[0034] Figure 1 This is a schematic diagram of the steps of a voice endpoint detection method in one embodiment of this application;

[0035] Figure 2 This is a schematic diagram of the voice endpoint detection system in another embodiment of this application;

[0036] Figure 3 This is a schematic diagram of the dual-state dual-threshold flipping mechanism of the binarizer in another embodiment of this application;

[0037] Figure 4 This is a schematic diagram of the structure of a voice endpoint detection system in another embodiment of this application. Detailed Implementation

[0038] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0039] Figure 1 This is a schematic diagram of the steps of a voice endpoint detection method in one embodiment of the invention, which specifically includes:

[0040] S1, Obtain the original audio and extract the shallow acoustic and text features of the original audio;

[0041] S2, adaptively weightedly fuses shallow acoustic features and text features to obtain fused audio features;

[0042] S3 involves contextually encoding the fused audio features to obtain temporal features;

[0043] S4, perform binary classification prediction on temporal features to obtain the probability of speech frames and the probability of non-speech frames;

[0044] S5 makes a decision on the binary classification prediction results and splices the decision results of each frame of the original audio to obtain the speech endpoint detection results.

[0045] Figure 2 This is a schematic diagram of the structure of a speech endpoint detection system 200 in one embodiment of the present invention, which includes a speech endpoint detection model composed of subnetworks M1-M4 and a binarizer M5, specifically:

[0046] The feature extraction subnetwork M1 includes a shallow feature extraction subnetwork and a text feature extraction subnetwork, which extract shallow acoustic features and text features from the original audio, respectively. Specifically, the shallow feature extraction subnetwork consists of K layers of convolutional neural networks. For example, when K=4, the configuration of the convolutional layers (kernel size, number of channels) of this subnetwork can be set to (161, 80), (5, 60), (3, 60), and (3, 60). The convolutional layers downsample the input original audio and output the shallow acoustic features of the audio by learning the context information of the audio. The text feature extraction subnetwork is constructed using the front-end subnetwork of a pre-trained end-to-end speech recognition model. This end-to-end speech recognition model can be composed of convolutional neural network layers, a Transformer coding module, and several fully connected layers. This front-end subnetwork is the network of the end-to-end speech recognition model except for the last fully connected layer. Its prediction process can be represented as:

[0047]

[0048] Where X is the original input audio. Here, C is the character sequence predicted by the speech recognition model, and C is the set of characters in the target language dictionary. The text feature extraction subnetwork maps the original audio to the latent space of the text, outputting the text features of the audio. The output of the penultimate layer of the fully connected layer of the pre-trained end-to-end speech recognition model is used as the text feature, where the number of nodes in the last layer of the fully connected layer of the end-to-end speech recognition model is consistent with the number of characters in the dictionary of the language model corresponding to the speech recognition model. By extracting text representations using the pre-trained speech recognition model and leveraging a large amount of labeled speech recognition corpus, the performance of the text feature extraction subnetwork is improved, further enhancing the robustness of the speech endpoint detection model.

[0049] The feature extraction subnetwork M1 extracts two types of audio features simultaneously and outputs them to the subsequent feature fusion subnetwork for further encoding.

[0050] The feature fusion subnetwork M2 adaptively weights and fuses the shallow acoustic features and text features extracted from the feature extraction subnetwork M1 to obtain text-aware fused audio features. (Adaptive weight values ​​for shallow acoustic features) Adaptive weight values ​​for text features The calculation method is described by the following formula:

[0051]

[0052] Among them, e L ∈R D e is a shallow acoustic feature vector at the audio frame level. T ∈R D These are text feature vectors at the audio frame level. Let b be the weight matrix of the fully connected network, where b ∈ R. 2 Let f be the bias vector of the fully connected network. att (.) represents the soft attention mechanism function, and D is the dimension of the feature vector. The fused audio feature vector z∈R D It can be calculated using the following formula:

[0053]

[0054] Where, α L α represents the normalized weight value for shallow acoustic features. T These are the normalized weight values ​​for text features.

[0055] The feature fusion sub-network M2 adaptively assigns different weights to shallow acoustic features and text features based on different scenarios. In stable, quiet scenarios with a high signal-to-noise ratio, the feature fusion sub-network M2 tends to assign greater weights to shallow acoustic features; in noisy scenarios with a low signal-to-noise ratio, the feature fusion sub-network M2 tends to assign higher weights to text features.

[0056] The temporal modeling subnetwork M3 takes the fused audio features extracted from the feature fusion subnetwork M2 as its input and performs context-dependent temporal modeling on the audio. The temporal modeling subnetwork M3 can be constructed using an L-layer temporal-aware neural network; for example, when L=2, a 2-layer Long Short-Term Memory (LSTM) network can be used. The temporal modeling subnetwork M3 performs contextual encoding on the fused audio features; after observing the entire audio sequence, the network layers re-encode the current audio features. In other embodiments, the temporal modeling subnetwork M3 can also be implemented using recurrent neural networks (RNNs), time-delay neural networks (TDNNs), Transformer neural networks, etc. The temporal modeling subnetwork M3 outputs temporal features containing contextual information.

[0057] The binary classification prediction subnetwork M4 takes the temporal features generated in the M3 temporal modeling subnetwork and feeds them into it. M4 directly converts these temporal features into binary posterior probability outputs. The binary classification prediction subnetwork has two output nodes, corresponding to the probabilities of speech frames and non-speech frames, respectively.

[0058] During the model inference phase, the binarizer M5 receives the binary posterior probabilities generated by the binary classification prediction sub-network M4. Binarizer M5 then re-determines the results for both speech and non-speech frames, concatenating all frame-level decisions from the original audio to generate the speech endpoint detection result. In one specific embodiment, binarizer M5 employs a dual-state, dual-threshold flipping mechanism. Figure 3 This is a schematic diagram of the dual-state dual-threshold toggling mechanism of the binarizer in this embodiment. The dual-state dual-threshold toggling mechanism includes two states (speech state S1, non-speech state S2) and two thresholds (speech state threshold T1 and non-speech state threshold T2), where T1 > T2. Speech class and non-speech class correspond to speech state S1 and non-speech state S2, respectively. t This represents the posterior probability that the audio in frame t belongs to a speech frame. The speech state threshold T1 and the non-speech state threshold T2 are key parameters of the binarizer M5, effectively suppressing noise interference. Before audio input, the current state is randomly initialized to either a speech state or a non-speech state. If the current state is a speech state and the probability that the current frame belongs to a speech frame is lower than T2, the current state is converted to a non-speech state; if the current state is a non-speech state and the probability that the current frame belongs to a speech frame is higher than T1, the current state is converted to a speech state; otherwise, the current state remains unchanged.

[0059] In one specific embodiment, the training process of the voice endpoint detection model includes:

[0060] S101, in the pre-training stage, an end-to-end speech recognition model is pre-trained using audio corpora with text labels. The model weights of the trained end-to-end speech recognition model, except for the last fully connected layer, are saved as a text feature extraction sub-network. In the inference stage, the text feature extraction sub-network outputs frame-level text features of the audio.

[0061] S102, the training phase, uses timestamped audio corpora to train sub-networks M1-M4 of the speech endpoint detection model. During training, the model parameters of the text feature extraction sub-network are frozen and do not participate in backpropagation gradient updates. Specifically, the training process of sub-networks M1-M4 includes three sub-steps: S901, using a binary cross-entropy function to calculate the loss function on the prediction results and artificial labels, where the artificial labels are frame-level speech / non-speech labels; S902, backpropagating the loss value calculated by the loss function to sub-networks M1-M4 to obtain the gradient value of each layer's network parameters; S903, updating the gradient of each layer's network parameters based on the gradient values ​​to obtain the optimized model parameters. S901-S903 are repeated until the calculated loss value converges.

[0062] S103, Fine-tuning stage: The binarizer parameters for dual states and dual thresholds are fine-tuned using in-domain data. In-domain data refers to audio data collected in the same or similar application scenarios as the actual test data. Key parameters are the speech state threshold and non-speech state threshold of the binarizer. During fine-tuning, a grid search is performed on the thresholds, and the threshold that maximizes the accuracy of speech endpoint detection is selected as the final threshold.

[0063] After training, the speech endpoint detection model is used for speech endpoint detection in audio. Specifically, the steps include: using the pre-trained text feature extraction sub-network of S101 to frame the original audio and extract text features, mapping the original audio frames to the text latent space of the target language; adaptively weighting and fusing the text features and shallow acoustic features, and feeding them sequentially into the temporal modeling sub-network and the binary classification prediction sub-network; and outputting the final decision results of speech frames and non-speech frames by the binarizer with the dual-state dual-threshold flipping mechanism fine-tuned by S103.

[0064] Figure 4 This is a schematic diagram of the structure of a speech endpoint detection system 400 in another embodiment of this application, which includes a speech endpoint detection sub-model 41 and a binarizer 42, wherein,

[0065] The speech endpoint detection sub-model 41 includes:

[0066] Feature extraction subnetwork 411 is configured to acquire the raw audio and extract shallow acoustic and text features from the raw audio.

[0067] Feature fusion subnetwork 412 is configured to adaptively weight and fuse shallow acoustic features and text features to obtain fused audio features;

[0068] Temporal modeling subnetwork 413 is configured to perform contextual encoding on fused audio features to obtain temporal features;

[0069] as well as,

[0070] The binary classification prediction subnetwork 414 is configured to perform binary classification prediction on temporal features to obtain the probability of speech frames and the probability of non-speech frames.

[0071] Binarizer 42 is configured to make a decision on the result of binary classification prediction and splice the decision results of each frame of the original audio to obtain the speech endpoint detection result.

[0072] The speech endpoint detection model proposed in the above embodiments has high noise robustness and detection accuracy. Even in the absence of target domain training data, it can effectively ensure the robustness of the speech endpoint detection model by using a pre-trained text feature extraction sub-network.

[0073] Although the contents of this application have been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that any changes in form and detail made to this application without departing from the spirit and scope of this application as defined by the appended claims and without inventive effort are within the scope of protection of this application.

Claims

1. A text-aware speech endpoint detection method, characterized in that, Including the following steps: S1, Obtain the original audio, and extract the shallow acoustic features and text features of the original audio; S2, adaptively weightedly fuse the shallow acoustic features and the text features to obtain fused audio features; S3, perform context encoding on the fused audio features to obtain temporal features; S4, perform binary classification prediction on the temporal features to obtain the probability of speech frames and the probability of non-speech frames; as well as, S5, make a decision on the result of the binary classification prediction, and splice the decision results of each frame of the original audio to obtain the speech endpoint detection result.

2. The voice endpoint detection method according to claim 1, characterized in that, The shallow acoustic features are extracted using a convolutional neural network.

3. The voice endpoint detection method according to claim 1, characterized in that, The text features are extracted using the front-end subnetwork of an end-to-end speech recognition model. The end-to-end speech recognition model includes a convolutional neural network layer, a Transformer encoding module, and several fully connected layers. The front-end subnetwork is the network of the end-to-end speech recognition model except for the last fully connected layer. The text features are the output of the front-end subnetwork.

4. The voice endpoint detection method according to claim 3, characterized in that, The number of nodes in the last layer of the fully connected layer is consistent with the number of characters in the dictionary of the corresponding language model.

5. The voice endpoint detection method according to claim 1, characterized in that, In the process of adaptive weighted fusion, the lower the signal-to-noise ratio of the original audio, the greater the weight of the shallow acoustic features.

6. The voice endpoint detection method according to claim 1, characterized in that, The temporal features are obtained through a temporal-aware neural network.

7. The voice endpoint detection method according to claim 1, characterized in that, S5 employs a dual-state, dual-threshold flipping mechanism for decision-making.

8. A voice endpoint detection system, characterized in that, This includes a speech endpoint detection sub-model and a binarizer, among which, The speech endpoint detection sub-model includes: A feature extraction subnetwork is configured to acquire the original audio and extract shallow acoustic and textual features from the original audio. A feature fusion subnetwork is configured to adaptively weight and fuse the shallow acoustic features and the text features to obtain fused audio features; A temporal modeling subnetwork is configured to perform contextual encoding on the fused audio features to obtain temporal features; as well as, A binary classification prediction subnetwork is configured to perform binary classification prediction on the temporal features to obtain the probability of speech frames and the probability of non-speech frames. The binarizer is configured to make a decision on the result of the binary classification prediction and to concatenate the decision results of each frame of the original audio to obtain the speech endpoint detection result.

9. The voice endpoint detection system according to claim 8, characterized in that, The training process of the speech endpoint detection sub-model includes the following steps: S901 uses a binary classification cross-entropy function to calculate the loss function for the prediction results and the manual labels; S902, the loss value calculated based on the loss function is backpropagated to the speech endpoint detection sub-model to obtain the gradient value of the network parameters of each layer; S903, Update the network parameters of each layer according to the gradient value to obtain optimized network parameters; S904, repeat S901-S903 until the calculated loss value converges.

10. The voice endpoint detection system according to claim 8, characterized in that, The binarizer uses a dual-state dual-threshold flipping mechanism for decision-making, where the threshold is determined through grid search.

Images