Voice keyword detection method, storage medium and electronic equipment

Abstract

The invention provides a voice keyword detection method, a storage medium and electronic equipment, and relates to the technical field of voice signal processing, and the method comprises the steps: determining a phoneme scoring path and a background sound path corresponding to an acoustic feature sequence of a to-be-detected target voice; identifying a noise type corresponding to each noise background sound; constructing an initial first linear grammar finite state converter according to the phoneme scoring path, and configuring a corresponding wildcard arc for the first linear grammar finite state converter by using each extracted noise type to obtain a second linear grammar finite state converter based on noise perception; creating a composite weighted finite state converter through a composite operation; and decoding the composite weighted finite state converter to predict whether the target keyword exists in the target voice. Therefore, the keyword detection system can sense different noise environments, the sensitive degree of the system to noise is dynamically adjusted, and the keyword detection accuracy in the high-noise environment is guaranteed.

Description

Technical Field

[0001] The present application relates to the technical field of speech signal processing, and in particular to a speech keyword detection method, a storage medium and an electronic device. Background Art

[0002] Keyword Spotting (KWS), also known as Wake Word Detection (WWD), is an important interface for human-computer interaction. KWS systems are usually deployed on devices and run continuously. With the increasing diversity of application scenarios, improving noise robustness under resource-constrained conditions has become a key research focus in the KWS field.

[0003] In order to enhance the noise robustness of KWS systems, various methods have been developed. One of the methods is to introduce a single-channel / multi-channel speech enhancement (SE) module before the KWS module. Another method is to design a new training strategy or architecture to achieve a noise-robust KWS system. Regardless of the method, simulated noise data plays a vital role in improving KWS performance. Preliminary experiments in this application highlight two key observations: first, at low signal-to-noise ratio (SNR) levels, strong noise energy naturally leads to reduced recall; second, simulated data with too low SNR causes the model to overfit to the noise, ultimately reducing the overall performance. These phenomena occur because the noise in the challenging training data masks part of the keyword speech, causing the model to confuse the noise with the target keyword.

[0004] Therefore, speech errors caused by noise may lead to a mismatch between speech content and paired text transcription during training. In complex acoustic environments, especially in complex acoustic scenarios with high noise and low signal-to-noise ratio, the model overfits to noise during training, resulting in a low wake-up rate.

[0005] The industry has not yet proposed a better solution to the above problems. Summary of the invention

[0006] The present application provides a voice keyword detection method, a storage medium and an electronic device, which are used to at least solve the problem of poor performance of voice keyword detection in a noisy environment in the current related technology.

[0007] In a first aspect, an embodiment of the present application provides a method for detecting speech keywords, comprising: determining a phoneme scoring path and a background sound path corresponding to an acoustic feature sequence of a target speech to be detected; the phoneme scoring path comprises a plurality of keyword phonemes and corresponding keyword activation scores in sequence, the keyword activation score being used to indicate the activation score of the keyword phoneme for the corresponding keyword segment in the target keyword; identifying at least one noise background sound among the background sounds in the background sound path, and identifying the noise type corresponding to each of the noise background sounds; constructing an initial first linear grammar finite state converter according to the phoneme scoring path, and configuring corresponding wildcard arcs for the first linear grammar finite state converter using each extracted noise type to obtain a second linear grammar finite state converter based on noise perception; performing a composite operation on the second linear grammar finite state converter, a vocabulary, and the phoneme scoring path to create a composite weighted finite state converter; the vocabulary comprises a mapping relationship between normal phonemes of each keyword segment and corresponding tags; and decoding the composite weighted finite state converter to predict whether the target keyword exists in the target speech.

[0008] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the voice keyword detection method of any embodiment of the present application.

[0009] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the voice keyword detection method of any embodiment of the present application are implemented.

[0010] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the voice keyword detection method of any embodiment of the present application.

[0011] The beneficial effects of the embodiments of the present application are:

[0012] By distinguishing different types of noise backgrounds and introducing noise types as wildcard arcs in the grammatical finite state converter, constructing a search space through compound operations and decoding in the search space of the compound weighted finite state converter, it is possible to effectively integrate noise information while retaining phoneme information, so that the keyword detection system can perceive different noise environments, thereby dynamically adjusting the system's sensitivity to noise, and achieving a high keyword detection accuracy in a high-noise, low signal-to-noise ratio environment, and reducing the occurrence of overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A flowchart of an example of a method for detecting speech keywords according to an embodiment of the present application is shown;

[0015] Figure 2 A schematic diagram showing a comparison effect of a noise data simulation according to an embodiment of the present application and an example of a decoding diagram based on CTC and a decoding diagram based on WFST of NTC is shown;

[0016] Figure 3 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] It should be noted that the presence of preset keywords is continuously detected in the audio stream. The wake-up word detection (Keyword Spotting, KWS) systems of major voice providers and the wake-up systems on mobile devices, such as Huawei's "Xiaoyi", Baidu's "Xiaodu Xiaodu", Xiaomi's "Xiaoai Tongxue" and Apple's "hey Siri". At present, the KWS system runs a low-power wake-up word detection model on the end-side chip 24 hours a day to respond to user commands.

[0019] In recent years, there has been a growing interest in designing small and efficient KWS systems based on Connectionist Temporal Classification (CTC). These systems are usually deployed on low-resource computing platforms, and the limitations of model size and computing power create bottlenecks in complex acoustic scenarios. These limitations often lead to overfitting and confusion between keywords and background noise, resulting in high false positive rates.

[0020] Figure 1 An operation flow chart of an example of a method for detecting speech keywords according to an embodiment of the present application is shown.

[0021] like Figure 1 As shown, in step S110, the phoneme score path and the background sound path corresponding to the acoustic feature sequence of the target speech to be detected are determined.

[0022] Here, the phoneme score path includes a plurality of keyword phonemes and corresponding keyword activation scores in sequence, and the keyword activation score is used to indicate the activation score of the keyword phoneme for the corresponding keyword segment in the target keyword.

[0023] In some embodiments, the target speech can be processed by adopting various non-restrictive acoustic models (such as deep neural networks, etc.) to extract the corresponding factor scoring path and background sound path. Exemplarily, the acoustic feature sequence of the target speech to be detected is first extracted. The speech signal is feature extracted using methods such as Mel-frequency cepstral coefficients (MFCCs) or linear predictive coding (LPC). Then, based on these acoustic features, a phoneme scoring path is constructed. Each keyword phoneme is marked as a node and connected to other keyword factors according to the time sequence. Each factor can also be expressed by a keyword activation score to represent the relevance to the target keyword, and the keyword activation score can be measured by a confidence score.

[0024] In step S120 , at least one noise background sound among the background sounds in the background sound path is identified, and the noise type corresponding to each noise background sound is identified.

[0025] Here, various noise detection algorithms, such as short-time Fourier transform, spectral subtraction or blind source separation algorithms, can be used to distinguish noise from background sound of each frame. Furthermore, for each identified noise background sound, its noise type can also be identified by various machine learning methods. It should be understood that the noise type can be diverse and should not be limited here for the time being. On the one hand, the noise type can be a description based on environmental information, such as traffic noise, human voice interference, etc. On the other hand, the noise type can also be described based on the interference or error caused by its recognition of keywords, such as keyword insertion errors caused by noise or phoneme masking errors caused by excessive sound noise, etc., and all belong to the implementation scope of the embodiments of the present application.

[0026] In step S130, an initial first linear grammar FST (Finite State Transducer) is constructed according to the phoneme scoring path, and corresponding wildcard arcs are configured for the first linear grammar FST using each extracted noise type to obtain a second linear grammar FST based on noise perception.

[0027] Here, according to the phoneme scoring path, an initial first linear grammar finite state converter (FST) is constructed. At this time, according to the identified noise type, a wildcard arc is configured to form a second linear grammar finite state converter, and the wildcard arc implements the noise modeling path based on noise type perception, so that the keyword detection system can adapt to various noise environments and improve the accuracy and sensitivity of keyword detection.

[0028] In step S140, the second linear grammar FST, the vocabulary, and the phoneme score path are compounded to create a composite WFST (Weighted Finite State Transducer).

[0029] Here, the vocabulary contains the mapping relationship between the normal phonemes of each keyword segment and the corresponding tags. The noise-aware second linear grammar FST is combined with the vocabulary, and the mapping relationship between the phonemes and tags in the vocabulary is used to set weights for each phoneme so as to give corresponding probability weights during the decoding process. Then, a weighting strategy is used to dynamically adjust according to the importance of each phoneme and the influence of background noise (or transfer cost). As a result, the presence of different keyword segments in the target speech can be more accurately evaluated, thereby improving the overall detection performance.

[0030] In step S150, the composite WFST search space is decoded to predict whether the target keyword exists in the target speech.

[0031] Here, a dynamic programming algorithm, a Viterbi algorithm or other efficient decoding algorithms can be applied to search the composite WFST. By calculating the scores of different decoding paths, the path with the highest score is selected, and then it is determined whether it contains the target keyword, thereby quickly and accurately determining the target keyword and optimizing the real-time performance of speech keyword detection.

[0032] In some examples of the embodiments of the present application, before decoding, more abundant decoding path information can be added to the composite WFST to further improve the accuracy of speech decoding. Specifically, based on the acoustic posterior model, the emission dense graph ε(x) corresponding to the acoustic feature sequence x of the target speech is determined. Here, the acoustic posterior model is a pre-trained acoustic posterior model. Then, based on the emission dense graph and the composite weighted finite state converter, intersection processing is performed to obtain an updated graph structure. Furthermore, the updated graph structure is decoded by the Viterbi decoding algorithm to predict whether the target keyword exists in the target speech.

[0033] In some embodiments, the total score of each decoding path in the updated graph structure is calculated by the Viterbi decoding algorithm. Exemplarily, the Viterbi algorithm is used to traverse the updated graph structure, starting from the initial state, and each possible path is evaluated. The score of the path is updated at each step according to the current state, input features and transition probability. Here, the path score is obtained by the product and accumulation of the phoneme activation score and the transition probability. The best predecessor state of each state needs to be recorded during the calculation process for subsequent backtracking. Then, according to the total score of each decoding path, the optimal decoding path is searched from each decoding path. Exemplarily, the total scores of each decoding path are compared, and the path with the highest score is selected as the optimal decoding path. By accurately calculating the score of each path, it is ensured that the best phoneme combination can be captured. Then, the recognition result for the target speech is determined according to the optimal decoding path to predict whether the target keyword exists in the target speech. Exemplarily, according to the selected optimal decoding path, the initial state is traced back to extract the corresponding phoneme sequence, and it is matched with the keyword, and the final recognition result is output, so as to determine whether the target keyword exists in the target speech. More details about the decoding process of the Viterbi algorithm will be described below in conjunction with other examples.

[0034] It should be noted that, in the process of decoding the update graph structure, the correct phoneme alignment should be given priority and the selection of wildcard paths or states should be reduced, thereby reducing the interference of noise on keyword detection performance.

[0035] In some examples of the embodiments of the present application, pure noise decoding paths in the update graph structure are screened out, and the pure noise decoding paths are composed of wildcard markers. Then, for each of the screened out decoding paths, the total score of the decoding path is calculated by the Viterbi algorithm. Through the embodiments of the present application, noise decoding paths that do not contain keyword phoneme markers and are completely composed of wildcard markers are screened out to avoid invalid analysis of pure noise backgrounds, ensure that the decoding process is carried out for potentially effective decoding paths, and ensure the effectiveness and efficiency of the decoding process.

[0036] In some examples of the embodiments of the present application, corresponding transfer costs are configured for different types of wildcard arcs to quantify the degree of influence of different noise types on the speech keyword decoding process, thereby improving the adaptability and accuracy of the decoding system. For the details of setting the transfer costs (or influence weights) of different wildcard arcs, please refer to the example description in combination with other parts below. Specifically, the transfer costs corresponding to each type of wildcard arc are obtained. Furthermore, for the noise modeling path in each decoding path, the total score corresponding to the noise modeling path is calculated according to the transfer cost, and the noise modeling path is a decoding path containing wildcard arcs and keyword phoneme tags, for example, the path contains a state node configured with a wildcard arc. In this way, in the decoding process, for each noise modeling path, combined with the wildcard arc and the keyword phoneme tag, the path score can be combined by the activation score of the keyword phoneme and the transfer cost of the wildcard arc, thereby comprehensively considering the effectiveness of the keyword phoneme and the influence of noise on decoding, thereby improving the detection accuracy of the target keyword.

[0037] In some embodiments, the types of wildcard arcs include self-loop arcs and bypass arcs. The self-loop arc is used to indicate the noise type of noise insertion errors, and the bypass arc is used to indicate the noise type of masking or interference caused by excessive noise. In this way, the potential errors between the keyword segment sequence and the adjacent subwords are modeled by self-loop arcs (@) and bypass arcs (*), the self-loop arc @ is used to solve the noise insertion errors, and the bypass arc * is used to handle the masking and interference caused by excessive noise.

[0038] Regarding the implementation details of step S110, in some embodiments, based on the CTC (Connectionist Temporal Classification) acoustic model, the phoneme scoring path and background sound path corresponding to the acoustic feature sequence of the target speech are determined, and the encoder of the CTC acoustic model adopts a deep feedforward sequence memory network.

[0039] It should be noted that the CTC acoustic model is a deep learning model for sequence data processing, which allows the input and output sequence lengths to not match. In speech keyword detection, the number of input audio frames is usually greater than the number of characters of the target keyword. This variable-length feature makes CTC very suitable for handling such tasks. In addition, CTC does not require manually annotated alignment information (i.e., the correspondence between input and output), which simplifies the data preparation process, especially when the speech data set is large, reducing the annotation cost. Moreover, CTC can automatically learn how to align input and target output during training. During the training phase, the CTC loss function can be used for optimization so that the model can accurately predict the timing of keywords.

[0040] In addition, the encoder of the CTC acoustic model uses the DFSMN (Deep Feedforward Sequential Memory Networks) layer, which has good memory and modeling capabilities and can effectively capture long-term dependencies in speech signals. By introducing the memory module, the historical information in the sequence data can be used more effectively, thereby improving the modeling accuracy and performance of the signal. In addition, in the deep coding structure based on the DFSMN layer, each layer of DFSMN can perform specific processing and feature extraction on the input data, and the combination of multiple layers can gradually improve the coding performance.

[0041] Regarding the implementation details of step S130, in some embodiments, an initial first linear grammar FST is constructed according to the phoneme score path. Exemplarily, the first linear FST includes multiple states and transitions between states, and the states can be defined by each phoneme score path node in the phoneme score path, and the transitions (or edges) can be defined by the order of the corresponding phoneme score path nodes. Then, for each extracted noise type, a matching target state is determined according to the phoneme score node corresponding to the noise type, and a target wildcard arc matching the extracted noise type is automatically determined according to a preset noise wildcard relationship table, and a target wildcard is configured to the target state. Exemplarily, the noise wildcard relationship table records multiple noise types and corresponding wildcard arcs, for example, noise type 1 corresponds to wildcard arc a, noise type 2 corresponds to wildcard arc b, noise type 3 corresponds to wildcard arc c, and so on. Specifically, the corresponding keyword phonemes and phoneme scoring nodes are located according to the noise frame, and then the wildcard arc corresponding to the noise type of the noise frame is configured in the corresponding state, thereby realizing the association matching between the wildcard arc and the state information based on noise perception, so that different types of noise can be explicitly personalized modeled during the decoding process, and the impact of different noise environments can be perceived, thereby dynamically adjusting the system's sensitivity to noise and improving the detection performance of speech keyword detection in a noisy environment.

[0042] Through the embodiments of the present application, a noise-aware CTC-based KWS (NTC-KWS) framework is provided, which aims to enhance the robustness of the model in noisy environments, especially in extremely low signal-to-noise ratio conditions. In the embodiments of the present application, two additional noise modeling wildcard arcs are introduced in the training and decoding process based on the weighted finite state transducer (WFST) graph: the self-loop arc is used to solve noise insertion errors, and the bypass arc is used to deal with masking and interference caused by excessive noise. Experiments on clean and noisy Hey Snips datasets show that NTC-KWS outperforms the most advanced (SOTA) end-to-end system and CTC-KWS baseline under various acoustic conditions, especially in low signal-to-noise ratio scenarios.

[0043] Specifically, we first classify the possible types of errors in low signal-to-noise ratio training data, and then explicitly model these different noises in training and testing. Specifically, two additional wildcard arcs for noise modeling are introduced in the training based on the weighted finite state transducer (WFST) graph. The self-loop arc is used to solve the noise insertion error, and the bypass arc is used to deal with the masking and interference caused by excessive noise. The model can model the noise by modifying the original CTC loss function.

[0044] Furthermore, these two noise arcs are also introduced in the testing or inference phase. On the one hand, it can reduce the difficulty of decoding the model under low signal-to-noise ratio. If a part of the wake-up word is not easy to recognize, it can be skipped by modeling the noise. On the other hand, it ensures the consistency of NTC-KWS training and testing, making the overall performance of the system better.

[0045] The NTC-KWS system provided in the embodiment of the present application illustrates the possibility of joint modeling of noise during the training process of the wake-up system, which promotes further exploration of more fine-grained modeling of noise. For example, how to model human voice interference, how to model pure environmental noise. And whether the wake-up system can perceive different noise environments, so as to dynamically adjust the system's sensitivity to noise.

[0046] The following will describe the specific framework and details of the NTC-KWS system provided in the embodiment of the present application.

[0047] Figure 2 A schematic diagram showing the comparative effects of noise data simulation according to an embodiment of the present application and an example of decoding diagrams based on CTC and based on WFST NTC is shown.

[0048] exist Figure 2The left part of the figure shows three types of errors through a noise simulation example, masking and interference caused by excessive noise (in red) and insertion (in blue). Figure 2 In the right part of the example, the keyword is set to "ABC". The decoding transfer rules at the syntax and tag levels are respectively and S. In NTC, two additional wildcard arcs are highlighted in red and blue, corresponding to the error colors in the simulation example on the left. In the composite graph S, λ1 and λ2 represent the wildcard transition costs. ε on the left represents a null transition, while ε on the right represents a null output.

[0049] A. CTC Implementation Based on WFST

[0050] Connectionist Temporal Classification (CTC) is a widely used loss function for training sequence learning models with mismatched input and label lengths, especially for automatic speech recognition (ASR) or keyword spotting (KWS). For ASR, the input acoustic feature sequence can be represented as where each x t is a D-dimensional acoustic feature vector, and T represents the total number of frames. The output label sequence can be expressed as where y U is the label token, and U is the total number of tokens. It is worth noting that T must be much larger than U because CTC introduces a special blank token φ to model the empty output of each frame. The actual alignment sequence of the input is where each π t Denotes the CTC vocabulary V ctc The goal of CTC is to minimize the sum of negative log-likelihoods of aligned sequences given an input sequence. Therefore, the CTC loss can be expressed as:

[0051]

[0052] Where B is defined as the mapping of a legal CTC alignment π to a label y, and B -1 Represents the inverse mapping.

[0053] Furthermore, decoding is to find the most likely target of speech, which can be formulated as:

[0054]

[0055] Where the approximate derivation introduces Viterbi decoding to find the most likely alignment.

[0056] The weighted finite state transducer (WFST) aims to map the input sequence to the output sequence, providing an efficient implementation for the training and decoding of CTC. Specifically, the search space S is composed of and The composite operation definition of represents CTC merging (removing φ and merging repeated tokens), vocabulary (mapping from words to tokens), and grammar FST. In addition, an emission dense graph ε is constructed to model the acoustic posterior. Therefore, the training and decoding of CTC can be reformulated as:

[0057]

[0058]

[0059] Among them, ∩ and ° represent the intersection and composition operations on the WFST graph, respectively. Represents the linear FST constructed for the transcription y. Forward and Viterbi represent the forward computation and Viterbi decoding on the composite WFST graph, respectively. Specifically, token passing is an efficient way to implement Viterbi decoding within the WFST framework. The decoding path is considered complete when a token reaches the final node.

[0060] B. CTC-based keyword detection

[0061] Compared to the CTC-based automatic speech recognition (ASR) described in Section A above, the training process of KWS is almost identical, with only two small differences in the decoding stage.

[0062] In terms of decoding graph, KWS's grammar FST More compact and lightweight, containing only the keyword clip path and an optional background sound path, rather than the entire acoustic space.

[0063] In terms of confidence score metrics, the search results are converted into keyword activation scores by calculating the confidence scores of speech segments, and different strategies can be evaluated to calculate the confidence scores of CTC-based KWS.

[0064] C. Training and decoding of NTC-KWS

[0065] Although CTC-based KWS performs well under typical acoustic conditions, its performance degrades significantly when speech clips are drowned in noise. Figure 2As shown on the left side of , excessive noise introduces errors in the simulated speech, including content masking, noise artifacts, and pronunciation distortion. This leads to a mismatch between speech and transcription, increasing the possibility of false triggering caused by noise. To enhance noise robustness, this application proposes NTC-KWS, which aims to introduce two types of wildcard arcs in the training and decoding process: self-loop arcs and bypass arcs. These arcs represent Different label-level graph errors in Figure 1 As shown. When part of the keyword is masked by noise, the self-loop arc (@) models insertion errors in the keyword label (represented by blue errors and arcs). In scenarios where phonemes are distorted or masked by noise, the bypass arc (*) accounts for substitution or deletion errors (represented by red errors and arcs). These wildcards effectively capture the concept of "excess speech noise and interference". The acoustic posterior of the * and @ arcs, representing the average posterior of all speech, is calculated as follows:

[0066]

[0067] In the formula, π i (j) indicates that in frame x i The predicted vocabulary V ctc To ensure that the model prioritizes learning the correct alignment over wildcard paths, wildcard arcs are assigned a large initial penalty. These penalties decay exponentially over training epochs:

[0068]

[0069] In the formula, l represents the number of current rounds, β @ , β * is a hyperparameter that can be set empirically.

[0070] Although the model has learned to handle the noise-dominated part of the keywords through NTC-based training, the standard CTC-KWS decoding graph fails to fully utilize the enhanced training, especially in low signal-to-noise ratio (SNR) scenarios. In order to better align the training and decoding processes, the syntax graph can also be modified Expand the decoding search space and introduce wildcard paths, such as Figure 1 For example, assuming the keyword is "ABC", the vocabulary V ctc Including the marks "AB CDE" and the blank mark φ, in the figure Self-loop arcs (@) and bypass arcs (*) are introduced to model the potential errors between the “ABC” keyword sequence and adjacent subwords. Then, the NTC grammar graph G ntc Keyword component transfer path and vocabulary Combine to create the NTC-KWS search space S ntc . With CTC-KWS search space S ctc In comparison, S ntc An alternative noise modeling path is provided for token transfer, which improves the decoding topology of noisy KWS in the following aspects. First, the consistency of training and decoding methods is guaranteed. Second, when noise dominates, it becomes challenging to identify each token in the keyword. Therefore, it is reasonable to relax the decoding restrictions in complex acoustic environments to improve recall, as this enables the model to select wildcard arcs against noise. To balance recall and false positives, a transfer cost λ is introduced for these two types of arcs in the NTC decoding graph separately. @ and λ * ( Figure 1 However, if a decoding path consists of only wildcard tokens and does not contain any keyword phonemes, the path will be discarded.

[0071] Through the embodiments of the present application, the NTC-KWS framework is proposed for speech errors caused by noise, combining noise-aware training and decoding strategies to enhance the robustness of the model in noisy environments. More specifically, the embodiments of the present application incorporate two noise modeling wildcard arcs into the CTC training framework to adaptively model noise, and further improve the WFST-based decoding search space, thereby improving KWS performance. As a result, the noise-aware CTC training criterion can model noise-dominated speech segments in complex acoustic scenes, thereby preventing noise overfitting. In addition, by adopting a no-training decoding strategy, the noise modeling path is integrated into the WFST-based KWS decoding, which improves performance in noisy environments without NTC training, and provides further improvements when applied within the NTC-KWS framework. Compared with the SOTA end-to-end system and the CTC-KWS baseline, the proposed NTC-KWS achieves superior performance in both clean and noisy conditions, especially in extremely challenging scenarios.

[0072] In order to prove the advanced nature of the technical solution of the embodiment of the present application, the inventor of the present application also conducted experiments on the excellent performance of the NTC-KWS system in the process of practicing the present application. The corresponding experimental details will be described in detail below.

[0073] A. Dataset

[0074] In order to evaluate the performance and noise robustness of the system, this application constructed a dataset using the following open source corpus.

[0075] -LibriSpeech (LS): LibriSpeech is an open source, widely used ASR corpus containing 960 hours of English speech and its corresponding transcriptions. This dataset is used for pre-training of base models and general ASR data for KWS models.

[0076] -Hey Snips (Snips): Hey-Snips is an open source KWS dataset centered around the keyword "Hey Snips". Since negative utterances do not have transcribed text, these negative data are only used for evaluation and not for training. This application combines all the negative data in the training set, development set, and test set to create a large test set. The length of the new negative test set is about 97 hours.

[0077] -WHAM!: The WHAM! dataset contains various types of environmental noise, such as music, restaurant sounds, etc. This application uses this dataset to synthesize noisy speech data for training and evaluation.

[0078] During the training process, the base speech encoder is first pre-trained using the clean LibriSpeech dataset. Then, the KWS model is fine-tuned using Snips and an equal amount of the LibriSpeech dataset. To simulate a noisy dataset, each clean speech waveform of Snips and LS is mixed with a noise sample randomly selected from the WHAM! dataset, with a signal-to-noise ratio (SNR) selected from a uniform distribution between 0 and 20 dB. Clean and noisy speech data are collected to construct the training dataset.

[0079] During evaluation, a noisy negative test set is simulated in the same way as for training, and the positive speech for testing is simulated at different fixed SNR levels. All positive test samples in the Snips dataset are mixed with noise at different SNR levels ({0,5,10,15,20}dB). To further evaluate the robustness of the model, an additional dataset is also simulated at -5dB, which represents extremely low and out-of-domain SNR.

[0080] B. Configuration

[0081] In terms of models, the toolkit K2 is used to build differential WFST for training, and OpenFST is used to build the decoding system. The deep feedforward sequence memory network (DFSMN) is a lightweight and efficient speech encoder that is widely used in KWS tasks. This architecture is used as the backbone of the CTC and NTC KWS systems. All models are initialized from the same pre-trained checkpoint and optimized by SGD. The DFSMN architecture contains 6 layers, and the hidden layer and projection layer sizes are set to 512 and 320, respectively. The left and right context orders of each DFSMN layer are 8 and 2, respectively. The final output unit of the CTC-based model includes 70 phonemes derived from the CMU pronunciation dictionary "cmudict-0.7b", as well as a special blank symbol φ. In addition, the NTC model includes two additional special symbols: the self-loop symbol @ and the bypass symbol *.

[0082] In terms of acoustic features, 40-dimensional logarithmic Mel filter bank coefficients (FBank) are extracted from the original audio, using a 25 millisecond window and a 10 millisecond skip step. In addition, this application adopts two data enhancement strategies: online speech perturbation, whose deformation factors are uniformly sampled from {0.9, 1.0, 1.1}, and SpecAugment, using 2 frequency masks (maximum F=10) and 2 time masks (maximum T=50) for each speech segment. This application concatenates the current frame with the previous 5 frames and the next 5 frames to generate the model input. Here, downsampling with a frame skip factor of 3 is also applied to reduce computational cost.

[0083] In terms of hyperparameters, the decay coefficient β @ and β * are set to 0.999 and 0.975 respectively. Initial wildcard cost λ(0) @ and λ(0) * are set to -4 for training. During testing, the decoding coefficient λ @ and λ * During label transfer inference for both CTC and NTC systems, there are at most 20 active labels.

[0084] C. Evaluation Details

[0085] In terms of baselines, a comprehensive baseline is constructed to evaluate the NTC-KWS system provided by the embodiment of the present application. First, the CTC-based model is compared with the state-of-the-art (SOTA) end-to-end (E2E) system on the Hey Snips dataset, which is open source in WeKws, to demonstrate that the KWS model built by CTC or its variants can achieve excellent performance. Then, NTC-KWS and CTC-related baselines are further compared in detail at different in-domain and out-of-domain SNR levels to evaluate the improved training and decoding methods.

[0086] In terms of evaluation metrics, recall values ​​at different SNR levels at a specific false alarm rate (FAR) are reported. Published SOTA models typically present recall at a FAR level of 0.5 or 1 per hour. To ensure a fair comparison, this application uses the same settings when benchmarking with these systems. However, this evaluation standard is relatively loose because performance tends to be strong at this FAR level. In order to minimize the impact of bias and randomness, this application evaluates all systems under stricter conditions. Therefore, after a preliminary comparison, the recall at FAR = 0.05 per hour is shown.

[0087] Table 1. Comparison of E2E baseline with our CTC and NTC KWS systems on the clean Hey Snips dataset. “POS.SNIPS” refers to the positive part in the Snips dataset, while “EQU.LS” means that we added an equal amount of LibriSpeech speech as general ASR data.

[0088]

[0089] The experimental results and related analysis conclusions will be elaborated below.

[0090] A. Comparison with End-to-End (E2E) SOTA Baselines

[0091] The upper part of the table shows the results using the official Snips data, while the lower part reports the results based on reconstructed data due to the lack of transcribed text for Snips negative samples. As described above, the CTC model cannot be trained on speech data without transcribed text, so this reconstruction is necessary. The CTC and NTC models are compared with the SOTAE2E baseline on the clean Hey Snips dataset, with a false alarm rate (FAR) of 0.5 or 1.0 per hour. The model of this application can achieve performance comparable to that of the SOTA model (D&E vs. B or C1). When the models are trained on the same data, the results in the lower half of Table I show that both the CTC and NTC models outperform the SOTA baseline (D&E vs. C2) under the conditions of FAR of 0.5 or 1.0 per hour. In addition, under the more stringent test conditions of FAR=0.05, the recall rates of the CTC and NTC models are absolutely improved by more than 9%. The results in Table I further demonstrate that our model outperforms the baseline, especially under strict testing conditions, verifying the effectiveness of the CTC-KWS baseline and the proposed NTC-KWS.

[0092] B. Impact of NTC training and decoding

[0093] Here, the respective effects of NTC training and decoding are investigated. Table II shows the performance of CTC and NTC models under different combinations of training and inference strategies. For CTC inference (F vs. D), simply updating the training criterion to NTC does not improve performance, because NTC training does not force the model to consider noise as part of the speech signal. Therefore, model F performs similarly to the CTC model trained with clean, high-quality speech. Training the model with CTC loss and testing it with the NTC decoding method (D vs. G) results in an absolute improvement of 2.7% in recall. This gain is particularly significant at low SNR levels (SNR = -5,0), highlighting the challenges of labeled decoding under such conditions. The decoding strategy proposed in this application relaxes the restrictions in complex environments, resulting in significant performance improvements. This suggests that if the original CTC KWS model cannot be retrained, but performance improvements are needed in noisy environments, adopting the NTC decoding strategy is a practical solution. In addition, applying NTC decoding to the NTC model increases the average recall by 2.2% absolutely (E vs. G), mainly due to the improvement brought by the consistency between the training and inference processes. This improvement highlights the benefits of NTC training and decoding consistency. In summary, the proposed NTC-KWS framework achieves an absolute performance improvement of 4.9% over the CTC-KWS baseline (E vs. D). Notably, at low SNR levels of -5 and 0, significant performance improvements of 15% and 8.3% were obtained, respectively, highlighting the robustness of NTC-KWS under extremely noisy conditions.

[0094] Table II. Evaluation of different combinations of CTC-KWS and NTC-KWS. “TRAIN” refers to the specific training criterion used in the model training phase, while “TEST” indicates the type of decoding graph used in the inference phase.

[0095]

[0096] C. Finding the optimal decoding configuration

[0097] Here, we further explore the importance of two hyperparameters in the decoding configuration. @ and λ * A grid search was performed on the value of @ and λ *When both coefficients are set to positive values, the performance is best, which indicates that the NTC model is not overfitted to shortcuts during NTC training, and wildcard paths are still needed to enhance performance during inference. In addition, in the process of practicing this application, the applicant also tried to set the decoding transfer coefficient to a negative value. Although the improvement is not as obvious as when it is positive, its performance is still better than the CTC-KWS baseline. When both coefficients are set to -∞, the behavior of the NTC-KWS model is similar to model F in Table II, which indicates that the NTC decoding strategy degenerates into the CTC method. Another conclusion drawn from Table III is that the decoding stage has a significant effect on λ. * The sensitivity is higher than λ @ , and λ * Setting it too high is not advisable.

[0098] Table III. Self-loop weights λ in the inference phase @ and bypass weight λ * Negative values ​​indicate punishment, positive values ​​indicate encouragement.

[0099]

[0100] The present application embodiment provides NTC-KWS, which is a CTC-based KWS framework variant optimized for noisy environments, especially under extremely low signal-to-noise ratio (SNR) conditions. By introducing multiple wildcards in the training and decoding stages, the risk of overfitting the model to noise is reduced. Modeling noise-dominated segments as wildcard markers enables the model to better distinguish keyword speech from background noise. Compared with previous end-to-end (E2E) systems, the system provided by the present application embodiment achieves SOTA* performance on a clean Hey Snips dataset. Specifically, the NTC-KWS framework has an absolute improvement of 4.9% in average recall over the standard CTC-KWS system at all SNR levels. Under low SNR conditions, such as -5dB and 0dB, the absolute gains in recall rate reached 15% and 8.3%, respectively. These improvements highlight the robustness of the system provided by the present application embodiment in challenging, noisy environments.

[0101] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any of the above-mentioned voice keyword detection methods of the present application.

[0103] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any one of the above-mentioned voice keyword detection methods.

[0104] In some embodiments, an embodiment of the present application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a voice keyword detection method.

[0105] Figure 3 is a schematic diagram of the hardware structure of an electronic device for executing a voice keyword detection method provided by another embodiment of the present application, such as Figure 3 As shown, the device includes:

[0106] One or more processors 310 and memory 320, Figure 3 A processor 310 is taken as an example.

[0107] The device for executing the voice keyword detection method may further include: an input device 330 and an output device 340 .

[0108] The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0109] The memory 320 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the voice keyword detection method in the embodiment of the present application. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the voice keyword detection method in the above method embodiment is implemented.

[0110] The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 320 may optionally include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] The input device 330 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 340 can include a display device such as a display screen.

[0112] The one or more modules are stored in the memory 320, and when executed by the one or more processors 310, the voice keyword detection method in any of the above method embodiments is executed.

[0113] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0114] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0115] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0116] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0117] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0118] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting speech keywords, comprising: Determine a phoneme scoring path and a background sound path corresponding to an acoustic feature sequence of a target speech to be detected; the phoneme scoring path comprises a plurality of keyword phonemes and corresponding keyword activation scores in sequence, the keyword activation scores being used to indicate activation scores of keyword phonemes for corresponding keyword segments in a target keyword; Identifying at least one noise background sound among the background sounds in the background sound path, and identifying the noise type corresponding to each of the noise background sounds; Constructing an initial first linear grammar finite state converter according to the phoneme score path, and configuring corresponding wildcard arcs for the first linear grammar finite state converter using each extracted noise type to obtain a second linear grammar finite state converter based on noise perception; Compounding the second linear grammar finite state transducer, the vocabulary, and the phoneme score path to create a composite weighted finite state transducer; the vocabulary comprising a mapping relationship between normal phonemes of each keyword segment and corresponding tokens; The composite weighted finite state converter is decoded to predict whether the target keyword exists in the target speech.

2. The method according to claim 1, wherein: The step of decoding the composite weighted finite state converter to predict whether the target keyword exists in the target speech comprises: Based on the acoustic posterior model, determining an emission dense graph corresponding to the acoustic feature sequence of the target speech; Performing intersection processing based on the emission dense graph and the composite weighted finite state converter to obtain an updated graph structure; The updated graph structure is decoded by a Viterbi decoding algorithm to predict whether the target keyword exists in the target speech.

3. The method according to claim 1, wherein: The initial first linear grammar finite state converter is constructed according to the phoneme score path, and the corresponding wildcard arc is configured for the first linear grammar finite state converter by using each extracted noise type to obtain a second linear grammar finite state converter based on noise perception, including: constructing an initial first linear grammar finite state converter according to the phoneme score path; the state of the first linear grammar finite state converter is defined by each phoneme score path node in the phoneme score path; For each extracted noise type, a matching target state is determined according to the phoneme score node corresponding to the noise type, and a target wildcard arc matching the extracted noise type is automatically determined according to a preset noise wildcard relationship table, and the target wildcard is configured to the target state; the noise wildcard relationship table records multiple noise types and corresponding wildcard arcs.

4. The method according to claim 2, wherein: The step of decoding the updated graph structure by a Viterbi decoding algorithm to predict whether the target keyword exists in the target speech includes: Calculating the total score of each decoding path in the update graph structure by using a Viterbi decoding algorithm; According to the total scores of the decoding paths, searching for the optimal decoding path from the decoding paths; A recognition result for the target speech is determined according to the optimal decoding path to predict whether the target keyword exists in the target speech.

5. The method according to claim 4, wherein: The step of calculating the total score of each decoding path in the updated graph structure by using the Viterbi decoding algorithm includes: Screening out pure noise decoding paths in the update graph structure, wherein the pure noise decoding paths are composed of wildcard markers; For each eliminated decoding path, a total score of the decoding path is calculated using a Viterbi decoding algorithm.

6. The method according to claim 5, wherein: The calculating the total score of the decoding path by using the Viterbi decoding algorithm includes: Get the transfer costs corresponding to each type of wildcard arc; For each noise modeling path in the decoding path, a total score corresponding to the noise modeling path is calculated according to the transfer cost; the noise modeling path is a decoding path containing a wildcard arc and a keyword phoneme tag.

7. The method according to claim 6, wherein: The types of the wildcard arcs include self-loop arcs and bypass arcs; the self-loop arcs are used to indicate the noise type of noise insertion errors, and the bypass arcs are used to indicate the noise type of masking or interference caused by excessive noise.

8. The method according to any one of claims 1 to 7, wherein: The step of determining the phoneme score path and the background sound path corresponding to the acoustic feature sequence of the target speech to be detected includes: Based on the CTC acoustic model, a phoneme scoring path and a background sound path corresponding to the acoustic feature sequence of the target speech are determined; the encoder of the CTC acoustic model adopts a deep feedforward sequence memory network.

9. A storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 8.

Images