Speech keyword detection method, storage medium and electronic device

Abstract

The application provides a speech keyword detection method, a storage medium and an electronic device, and relates to the technical field of speech signal processing. The method comprises the following steps: performing deep feature coding on the speech to be detected based on a plurality of coding layers cascaded in a CTC acoustic model; decoding the intermediate coding features output by the first coding layer based on an intermediate decoding layer in the CTC acoustic model to determine a first prediction score sequence for the target keyword; decoding the deep coding features finally output by the cascaded coding layers based on a final decoding layer in the CTC acoustic model to determine a second prediction score sequence for the target keyword; calculating the similarity between the first prediction score sequence and the second prediction score sequence, and calibrating the second prediction score sequence based on the similarity to determine a target prediction score sequence corresponding to the target keyword. Thus, through speech keyword detection based on cross-layer discrimination consistency, a higher-precision keyword detection effect is achieved.

Description

Technical Field

[0001] This application relates to the field of speech signal processing technology, and in particular to a speech keyword detection method, storage medium, and electronic device. Background Technology

[0002] Keyword spotting (KWS), especially wake word detection (WWD), involves continuously detecting the presence of preset keywords in an audio stream. Major voice service providers offer KWS systems, and mobile devices use wake-up systems such as Huawei's "Xiaoyi," Baidu's "Xiaodu Xiaodu," Xiaomi's "Xiao Ai," and Apple's "Hey Siri." These technologies or products detect the appearance of preset keywords in a continuous speech stream and wake the device when the wake word appears.

[0003] Current keyword detection systems mainly include Automatic Speech Recognition (ASR) detection systems and detection systems based on Weighted Finite State Transducers (WFST). ASR-based decoding algorithms consider all possible phoneme sequences, resulting in an excessively large search space. This leads to inefficiency and prevents the keyword detection technology from achieving optimal performance. However, algorithms that limit the search space cause changes in the decoding output (from the phoneme sequence of the keyword to the probability of the keyword appearing in each frame), making them inefficient for streaming decoding. WFST-based keyword detection technology, using token passing, restricts the search space to the phonemes of the keyword and padding units; however, such systems have complex underlying code implementations, making them difficult to maintain and improve.

[0004] Currently, the industry has not proposed a better solution to the above problems. Summary of the Invention

[0005] This application provides a method, storage medium, and electronic device for voice keyword detection, which at least solves the problem of poor performance in voice keyword detection in current related technologies.

[0006] In a first aspect, embodiments of this application provide a speech keyword detection method, comprising: performing deep feature encoding on the speech to be detected based on multiple cascaded coding layers in a CTC acoustic model; decoding the intermediate coding features output by the first coding layer based on the intermediate decoding layer in the CTC acoustic model to determine a first prediction score sequence for a target keyword; the first coding layer being any one of the pre-coding layers except the last coding layer in the multiple cascaded coding layers; decoding the deep coding features finally output by the cascaded coding layers based on the final decoding layer in the CTC acoustic model to determine a second prediction score sequence for the target keyword; and calibrating the second prediction score sequence using the first prediction score sequence to determine a target prediction score sequence corresponding to the target keyword.

[0007] Secondly, embodiments of this application provide an electronic device, which includes: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the speech keyword detection method of any embodiment of this application.

[0008] Thirdly, embodiments of this application provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the speech keyword detection method of any embodiment of this application.

[0009] Fourthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the speech keyword detection method of any embodiment of this application.

[0010] The beneficial effects of the embodiments of this application are as follows:

[0011] By setting a dual decoding mechanism in the CTC (Connectionist Temporal Classification) acoustic model, the intermediate decoding layer processes the encoding features of the pre-encoding layer to achieve initial recognition of target keywords. The initial recognition result is then used to calibrate the final deep decoding result. By learning the difference patterns of keyword recognition in different layers, the speech keyword detection results can be optimized. Based on cross-layer discrimination consistency (CDC), speech keyword detection achieves higher accuracy. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating an example of a speech keyword detection method according to an embodiment of this application is shown;

[0014] Figure 2 A schematic diagram illustrating an example of the encoding and decoding process of a CTC acoustic model according to an embodiment of this application is shown;

[0015] Figure 3 A schematic diagram illustrating the operation flow of inference and training of an enhanced CTC acoustic model according to an embodiment of this application is shown.

[0016] Figure 4 The code screenshot shows an example of a CTC-based KWS streaming decoding algorithm provided according to an embodiment of this application;

[0017] Figure 5 The experimental simulation results of the frame-level decoding score mode for keyword detection of positive and negative samples are shown.

[0018] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that, given that KWS systems typically run on resource-constrained devices, reducing computational and memory costs is crucial. CTC avoids the need for frame-level alignment, providing an effective training standard for this task.

[0021] The non-autoregressive nature of CTC makes it naturally suitable for designing efficient streaming decoding algorithms. Existing continuous KWS decoding algorithms can be divided into two categories. The first category is based on Automatic Speech Reconciliation (ASR) inference. In this approach, the model processes speech and outputs a sequence of symbols in a streaming pattern, subsequently matching keywords in the updated decoded sequence. However, ASR decoding strategies aim to generate the most probable complete hypothesis across the entire space, rather than specifically detecting the presence of keywords, and are therefore not ideal for KWS. The second category relies on the KWS decoding graph, typically including keyword paths and optional padding paths. While this approach is specifically designed for KWS, implementing decoding based on graph and symbol transfer algorithms using Weighted Finite State Transducer (WFST) is complex, making it difficult to maintain and improve. Furthermore, these systems do not perform well enough in KWS at low signal-to-noise ratios, especially under extremely low SNR conditions, where they fail to achieve optimal results.

[0022] It should be noted that the search space of the ASR-based decoding algorithm is not limited for the KWS task, resulting in suboptimal performance. The WFST-based decoding algorithm, implemented based on automata theory, possesses a complete and systematic theoretical framework and offers better support for streaming decoding; therefore, systems built upon this theory are naturally more complex. Furthermore, the aforementioned algorithms are not optimal, thus their noise robustness is insufficient.

[0023] It should also be emphasized that current research and optimizations are focused on the two types of systems mentioned above. Specifically, for ASR-based decoding algorithms, the efficiency and performance of KWS decoding algorithms are improved by restricting the search space, but this fails to provide efficient support for streaming decoding. Furthermore, WFST systems are relatively difficult to improve and maintain.

[0024] Figure 1 A flowchart illustrating an example of a speech keyword detection method according to an embodiment of this application is shown.

[0025] like Figure 1 As shown, in step S110, deep feature encoding is performed on the speech to be detected based on the multiple cascaded coding layers in the CTC acoustic model.

[0026] Here, multiple coding layers in the CTC acoustic model can be used to frame the input speech signal and extract features, such as by using Mel-frequency cepstral coefficients (MFCC) or spectrograms for preprocessing. Then, more abstract features are extracted layer by layer through multiple coding layers to form a deep feature representation. It should be understood that the structure of the coding layers can be diverse, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), etc., and is not limited here.

[0027] It should be noted that the CTC acoustic model is a deep learning model for sequence data processing, which allows for mismatches between input and output sequence lengths. In speech keyword detection, the number of input audio frames is typically greater than the number of characters in the target keyword; this variable-length characteristic makes CTC well-suited for such tasks. Furthermore, CTC does not require manually labeled alignment information (i.e., the correspondence between input and output), simplifying the data preparation process, especially when dealing with large speech datasets, thus reducing annotation costs. Moreover, CTC can automatically learn how to align the input and target output during training. During the training phase, the CTC loss function can be used for optimization, enabling the model to accurately predict the temporal order of keywords.

[0028] In step S120, based on the intermediate decoding layer in the CTC acoustic model, the intermediate coding features output by the first coding layer are decoded to determine the first prediction score sequence for the target keyword.

[0029] Here, the first coding layer is any one of the cascaded coding layers except the last one. Unlike the conventional CTC acoustic model, in addition to decoding the final coded features, an intermediate decoding layer is used to preliminarily decode the intermediate coded features output by the first coding layer. The resulting prediction score sequence can represent the probability of the target keyword appearing in each time frame.

[0030] In step S130, based on the final decoding layer in the CTC acoustic model, the deep coding features output by the cascaded coding layers are decoded to determine the second prediction score sequence for the target keyword.

[0031] Here, the deep-coded features output by all cascaded coding layers are decoded by the final decoding layer, and a second prediction score sequence is generated by integrating more comprehensive information.

[0032] In step S140, the similarity between the first predicted score sequence and the second predicted score sequence is calculated, and the second predicted score sequence is calibrated based on the similarity to determine the target predicted score sequence for the corresponding target keyword.

[0033] Here, various unrestricted similarity algorithms can be used to quantify the similarity between the first and second predicted score sequences, such as cosine similarity, Euclidean distance, or Pearson correlation coefficient. Furthermore, the calculated similarity is used to adjust the second predicted score sequence, for example, by employing linear weighting or non-linear adjustment methods, so that the calibrated score sequence better matches the recognition of the actual target keywords.

[0034] Figure 2 A schematic diagram illustrating an example of the encoding and decoding process of a CTC acoustic model according to an embodiment of this application is shown.

[0035] like Figure 2 As shown, after the keyword audio enters the encoder, its intermediate and final layer representations are fed into the streaming decoding algorithm provided in this application; subsequently, the CDC module uses two score sequences to calculate the similarity, adds the similarity score to the final layer score and normalizes it to obtain the final output.

[0036] It should be noted that the embodiments of this application provide a frame synchronization streaming decoding algorithm for CTC-based KWS. During the implementation of this application, the inventors discovered that near the wake-up point, the behavior of the intermediate and final layers of positive samples and easily triggered negative samples differs significantly, meaning they exhibit substantial differences in similarity. Therefore, through the embodiments of this application, to further enhance the discrimination between challenging negative samples and positive samples, an effective CDC-enhanced streaming KWS decoding algorithm is implemented using Cross-layer Discrimination Consistency (CDC) information.

[0037] Experiments have shown that the decoding output of the intermediate layer can effectively help the system better distinguish between positive and negative examples. The cross-layer discrimination consistency enhancement further improves the performance of the streaming decoding algorithm. More details will be discussed in conjunction with other parts below.

[0038] In some examples of embodiments of this application, the CTC acoustic model is trained based on a speech sample set and a comprehensive loss function, which is:

[0039]

[0040] In the formula, Let w represent the loss function of the CTC acoustic model, where w is a hyperparameter and w∈[0,1). This represents the intermediate CTC loss term determined based on the prediction output of the intermediate decoding layer for the speech sample, and This represents the final CTC loss term determined based on the prediction output of the final decoding layer for the speech sample.

[0041] Specifically, given the acoustic feature vector of the discourse Where T is the number of frames and D is the feature dimension, the CTC model aims to predict target label sequences. Where U is the number of target tags. CTC allows repeated symbols and introduces a special whitespace symbol φ to facilitate alignment between the acoustic sequence and the tag sequence. The alignment path is... The CTC (Combined Criterion Tolerance) aims to maximize the likelihood of the target label sequence y across all possible alignment paths π. The loss function can be expressed as:

[0042]

[0043] In the formula, B represents the mapping from the alignment path π to the label sequence y, and B -1 This indicates the inverse mapping. All arithmetic operations are performed in the logarithmic domain. Additionally, intermediate CTC (ICTC) regularization is applied to the intermediate layers of the encoder, which can be used as an effective regularization strategy for CTC training.

[0044] In some examples of embodiments of this application, the coding layer employs a DFSMN (Deep Feedforward Sequential Memory Network) layer, which possesses excellent memory and modeling capabilities, effectively capturing long-term dependencies in speech signals. By introducing a memory module, historical information in the sequence data can be utilized more effectively, thereby improving the modeling accuracy and performance of the signal. In a deep coding structure based on DFSMN layers, each DFSMN layer can perform specific processing and feature extraction on the input data, and the combination of multiple layers can progressively improve coding performance. However, as mentioned earlier, the choice of coding layer structure can be diverse and is not limited here. DFSMN is merely a lightweight coding layer that can capture context dependencies well; replacing it with other encoders should also fall within the scope of implementation of embodiments of this application.

[0045] Regarding the placement of intermediate decoding layers, overly shallow coding layers primarily focus on feature extraction and cannot effectively capture keyword information, while overly deep coding layers learn content similar to the main CTC, thus failing to ideally extract CDC information. Therefore, in some examples of embodiments of this application, the first coding layer is positioned in the middle of a cascaded series of coding layers, for example, inserting an intermediate decoding layer near the halfway point of the total number of layers, which yields superior performance.

[0046] In some examples of embodiments of this application, intermediate decoding layers and final decoding layers are used to decode the encoded features corresponding to the speech to be detected frame by frame according to a preset time frame scale, and the target prediction score sequence is a frame-level decoding score accordingly. Here, the time frame scale can be determined comprehensively based on the current time frame and neighboring times, such as historical and / or future time frames, and can also be dynamically analyzed by using a sliding time window to analyze the similarity between the score sequences output by the intermediate decoding layer and the final decoding layer, thereby obtaining a dynamically analyzed and updated frame-level decoding score sequence.

[0047] In some examples of embodiments of this application, a preset time frame scale is used to indicate the current time frame, historical frame window, and future frame window. This allows for the inclusion of both historical and future information within the time window, comprehensively considering the effectiveness of the discriminative information provided by historical and future features near activation points, and further improving the performance of streaming keyword recognition based on CDC enhancement. The analysis of the historical / future window size related to CDC scores will be elaborated below with reference to other examples.

[0048] In some examples of embodiments of this application, the calculation details of the CDC score may be based on the cosine similarity function to calculate the score similarity sequence between the first predicted score sequence and the second predicted score sequence, and then add the score similarity sequence and the second predicted score sequence together and normalize them to determine the target predicted score sequence.

[0049] Figure 3 A schematic diagram illustrating the operation flow of inference and training of an enhanced CTC acoustic model according to an embodiment of this application is shown.

[0050] like Figure 3 As shown, this illustrates the model training and decoding framework using the "Hey Snips" dataset. An additional linear projection layer with ICTC is appended to the middle part of the encoder. L represents the number of DFSMN layers. CDC augmented decoding includes a refinement stage. s(init), s(inter), s(cdc), and s(refine) represent the initial CTC decoding score, the intermediate CTC decoding score, the CDC score between the initial and intermediate scores, and the final refined score, respectively. CDC This represents a function used to measure discriminative consistency. L His. and L Fut. These refer to the number of historical frames and the number of future frames used for CDC score calculation, respectively.

[0051] This application presents a novel streaming decoding algorithm for CTC-based KWS. Unlike ASR-based methods that require searching the entire utterance space and can only start from the beginning of the utterance, the algorithm provided in this application restricts the search space to keywords, allowing detection at any time point. Furthermore, this algorithm is easier to implement compared to complex graph decoding methods. Additionally, based on the finding that positive and negative samples behave differently across layers, cross-layer discriminant consistency is utilized to capture the differences between keyword and non-keyword speech segments. Combining CDC information further enhances the effectiveness of CTC-KWS streaming decoding. On the "Hey Snips" dataset, compared to various ASR-based and graph benchmarks, the CDC-enhanced streaming KWS decoding algorithm provided in this application demonstrates impressive performance at different signal-to-noise ratio (SNR) levels.

[0052] The description of CTC-based KWS streaming decoding highlights the core concept of limiting the search to keywords at each time step. Unlike ASR decoding, which aims to generate complete hypotheses, this application's method narrows the decoding space to specific segments related to keywords in the semantic space. Specifically, it considers the phoneme posterior related to keywords and selects the path with the highest keyword detection probability or partial occurrence probability at each decoding state.

[0053] Traditional ASR or graph-based decoding methods are inflexible and require significant effort for online adaptation. In contrast, the search algorithm provided in this application enables seamless keyword detection in streaming scenarios. At the beginning of each frame, two new path competitors are introduced with their probabilities initialized to 1. This allows them to seamlessly integrate into the decoding process and ensures that new competitors entering at each time step can effectively participate in decoding. Considering the potential length of keywords, a timeout mechanism is also implemented to improve search efficiency; more details can be found in... Figure 4 The provided screenshot shows the program code for the KWS streaming decoding algorithm based on CTC.

[0054] In addition to applying the aforementioned CTC-based KWS streaming decoding algorithm to the main CTC layer, inference can also be performed using shallow intermediate branches. The initial score and intermediate score calculated from the main CTC layer and intermediate CTC branches are represented as follows:

[0055]

[0056] Preliminary experiments show that the scores of the main CTC layer and the intermediate CTC layer exhibit significantly different characteristics when processing positive and negative samples.

[0057] Figure 5The experimental simulation results of frame-level decoding score mode for keyword detection for positive samples (top) and negative samples (bottom) are shown. Figure 5 As shown, near the wake-up point, the two score curves for positive samples appear stable and similar. However, for high-scoring negative samples, the activation time is shorter, and the two score curves diverge significantly. This difference may stem from the relatively consistent patterns in positive samples, allowing the ICTC layer to effectively learn these positive patterns despite its limited parameters. However, negative samples contain multiple patterns and conditions, making it difficult for shallow layers to fully capture all relevant aspects. Even if the final layer overfits to these false alarm patterns, ICTC can still serve as a regularization mechanism. The red box highlights the changes in CDC scores near the activation point. Notably, for positive samples, the score remains close to 1 near the wake-up point, while negative samples exhibit dramatic fluctuations.

[0058] Therefore, considering the different performances of positive and negative samples, the sliding window CDC score at time t is defined as:

[0059]

[0060] In the formula, f CDC This represents the function used to measure the consistency score; the cosine similarity function can be used here. L His. and L Fut. These represent historical frame counts and future frame counts, respectively. Figure 2 The diagram also shows the CDC score curve s(cdc). The refined score of the multi-stage decoding algorithm enhanced by CDC is the sum of the initial CTC score and the CDC score:

[0061]

[0062] In the formula, It is important to note that score calculation is performed frame-by-frame within a fully streaming decoding framework. Fut. The value can be adjusted to balance the trade-off between performance and system latency during the refining stage.

[0063] The following will elaborate on the relevant details of the experiments and evaluations related to the embodiments of this application.

[0064] The experiments in this application embodiment were conducted based on the following dataset:

[0065] LibriSpeech is a large-scale English Automatic Speech Recognition (ASR) dataset containing approximately 960 hours of speech and their corresponding transcriptions. LibriSpeech was used to initially train a CTC-based acoustic model to obtain a better foundational model.

[0066] Hey-Snips (Snips) is an open-source Keyword Detection (KWS) dataset containing the wake word "HeySnips". The positive sample portion of the dataset contains 5,799, 2,484, and 2,599 keyword speech samples in the training, development, and test sets, respectively. Due to a lack of transcribed content, the negative samples could not be used for ASR-based training, so the Snips dataset was reorganized. All the original negative samples were compiled into a large negative sample test set, approximately 97 hours of speech.

[0067] WHAM! is an environmental noise dataset containing various types of ambient noise. The robustness of the model at different signal-to-noise ratios (SNRs) is evaluated by combining clean Hey-Snips speech samples with randomly sampled noise from the WHAM! dataset to synthesize noisy keyword samples.

[0068] The configuration details are as follows:

[0069] During training, the model can first be pre-trained using the complete LibriSpeech dataset. In the fine-tuning phase, data of equal proportions are extracted from both LibriSpeech and Snips to construct a general ASR dataset. To enhance the model's performance in noisy environments, this application mixes each clean speech sample from Snips and LibriSpeech with noisy samples from WHAM!. The signal-to-noise ratio (SNR) of the mixed speech is uniformly distributed between 0 dB and 20 dB. Ultimately, the fine-tuning dataset contains approximately 24,000 speech samples in total (12,000 clean samples and 12,000 noisy samples).

[0070] The acoustic characteristics are explained below:

[0071] A 40-dimensional log-Melbourne filter bank (FBank) coefficient was extracted from each speech segment, using a 25-millisecond window and a 10-millisecond sliding window. Two data augmentation techniques were applied during training: (1) online speech perturbation, where the distortion factor was uniformly sampled from {0.9, 1.0, 1.1}; and (2) SpecAugment, using two temporal masks with a maximum mask length of Tmax = 50 and two frequency-domain masks with a maximum frequency width of fmax = 10. The FBank features from the first 5 frames and the last 5 frames were concatenated to generate a 440-dimensional input. To reduce the input frame rate, a 3-frame skipping factor was applied, resulting in each input frame containing 30 milliseconds of acoustic information.

[0072] The following explains the model and loss mechanism:

[0073] The encoder consists of 6 DFSMN layers with dimensions of 440, 512, and 320 for the input, hidden, and projection layers, respectively. The left and right order of the DFSMN layers are set to 8 and 2, respectively. The final acoustic output unit is defined using the CMU pronunciation dictionary “cmudict-0.7b”. In addition to 70 phonemes, a unique blank symbol is added for CTC modeling. During the pre-training phase, only the frame-level cross-entropy criterion can be used. During the fine-tuning phase, depending on the model, either CTC or a combined loss function combining CTC and ICTC is used. Unless otherwise stated, the weight w of the ICTC loss term in Equation (1) can be set to 0.3.

[0074] The following are explanations regarding the assessment:

[0075] To accurately evaluate the performance of the keyword detection system under different signal-to-noise ratio (SNR) scenarios, each positive sample was mixed with noise, with SNR levels ranging from {0, 5, 10, 15, 20} dB. Additionally, a positive sample set with an SNR of -5 dB was created to evaluate the system's robustness under extremely low SNR and out-of-domain SNR conditions. As mentioned earlier, all negative samples from the original Snips dataset were collected for testing. The negative sample set was also mixed with noise, with SNR ranging from 0 dB to 20 dB. Decoding hyperparameter S... Bonus and T out Set them to e respectively 3 And 3 seconds.

[0076] For evaluation metrics, recall (i.e., 1 - false negative rate) and macro-average recall can be assessed at different SNR levels, both under a fixed false positive rate (FAR). For ASR-based methods, this application reports accuracy (Acc.), which is equivalent to recall at FAR = 0, as these systems face challenges in decoding false positives. Furthermore, in comparison with the KWS graphical system, this application presents results with a false positive rate of 0.05 per hour, a more reasonable but rigorous evaluation criterion for the KWS system.

[0077] Table I compares three mainstream decoding baseline methods, all of which use the same CTC-based acoustic model. Both CTC greedy decoding and prefix beam search are ASR-based systems. Our streaming method significantly outperforms these three ASR-based and graph-based baseline methods. Compared to the two ASR-based baselines, streaming KWS decoding achieves absolute improvements of 10.4% and 6.6% at different SNR levels, respectively. Even compared to the robust graph-based baselines, the streaming decoding strategy delivers an absolute gain of 6.3%. The results demonstrate that the proposed method performs well in various noise environments.

[0078] Table I compares the accuracy of the proposed streaming algorithm with three baseline systems (FAR=0): (1) ASR greedy search,

[0079] (2) ASR prefix beam search, (3) graph-based KWS search.

[0080]

[0081] Several important details need to be noted. This method achieves an accuracy improvement of approximately 10% or more under low SNR conditions, such as -5 or 0 dB. This improvement is likely because at very low SNR, ASR-based decoding struggles to generate correct hypotheses due to excessive noise in the speech. For graph-based methods, decoded symbols have difficulty traversing the decoded graph, hindering their arrival at the final state and resulting in lower baseline accuracy for graph-based methods. However, in this application, the decoding is forced to output activation scores frame-by-frame, enabling it to continue operating in noisy environments. This demonstrates that the method provided in this application's embodiments is particularly effective in complex and challenging acoustic environments.

[0082] The importance of the CDC strategy for improving streaming decoding algorithms will be highlighted below. As shown in Table II, when comparing model pairs using the same decoding strategy (e.g., A vs. C and B vs. D), the gain of ICTC is more significant at low SNR levels, indicating the effectiveness of CTC regularization in noisy scenarios. Furthermore, the comparison of models E and D further demonstrates the effectiveness of the CDC strategy proposed in this application. Overall, compared to the baseline (A), the CDC-enhanced system (E) improves the average recall by 6.8% at different SNR levels and reduces the average false negative rate by 46.3%.

[0083] Table II. Comparison of recall rates of the proposed KWS streaming decoding and multi-stage decoding strategies with graph-based baselines at different SNR levels, with a fixed false alarm rate of 0.05 per hour.

[0084]

[0085] The KWS decoding algorithm provided in this application abandons the traditional ASR decoding implementation, which limits the decoding search space, supports streaming decoding, and introduces timeout and reward mechanisms to effectively solve the performance loss problem of streaming decoding implementation on KWS. Furthermore, the decoding output of the intermediate layer effectively helps the system better distinguish between positive and negative examples, and the decoding is enhanced by CDC information, further improving the performance of the streaming decoding algorithm. In addition, the system does not require WFST-related theory, only PyTorch and time concatenation implementation. Experiments show that the system has a significant effect on improving KWS performance at low signal-to-noise ratios, especially extremely low signal-to-noise ratios.

[0086] The CDC strategy utilizes the fractional pattern around the wake-up point, which inevitably introduces some latency. Therefore, the historical window size (L) was further analyzed. His. ) and future window size (L Fut. The impact on KWS performance. As described in other sections above, each frame contains 30 milliseconds of acoustic information. Here, fixing the total window size to 30 frames results in a maximum latency of 900 milliseconds, which is acceptable in practical deployments. As shown in Table III, increasing the number of future frames can improve performance, highlighting the effectiveness of the discriminative information provided by future features near activation points.

[0087] Table III. Comparison of recall performance using CDC-enhanced decoding with different window sizes. The results of the system proposed in this application are bolded and underlined, and the upper bound offline results used for reference are also bolded.

[0088]

[0089] Here, this application proposes two upper bound baselines: one using all future scores, and the other a fully offline baseline using all scores throughout the entire statement. The results show that when L... Fut. At a resolution of 30, the performance gain has reached its maximum, and further increases in future frames provide negligible performance improvements, indicating that excessive discriminative information yields diminishing returns. Introducing a slight delay (less than 1 second) is sufficient to achieve performance gains close to the upper bound.

[0090] Furthermore, the impact of the intermediate loss weight w in Equation (1) and the number of ICTC application layers on system performance was further analyzed. Table IV shows that when the number of layers is fixed, the value of w = 0.3 optimally balances L. ICTC The contribution of this technology is significant. Furthermore, experiments with the number of application layers show that placing the ICTC term near half the total number of layers L (i.e., L / 2) yields optimal performance. This is because shallow layers primarily focus on feature extraction and fail to effectively capture keyword information, while deep layers learn content similar to the main CTC, thus also failing to ideally extract CDC information.

[0091] Table IV. Recall results for different w values ​​and the number of ICTC layers applied. The encoder contains a total of L = 6 layers, with the optimal number of layers being L / 2 = 3.

[0092]

[0093] This application proposes a CDC-enhanced streaming KWS decoding algorithm specifically designed for CTC-based systems. Specifically, it presents a streaming decoding algorithm capable of detecting arbitrary starting points of keywords in an audio stream, further improving performance by utilizing CDC scores from intermediate and final layers. Compared to greedy search and prefix bundle search in the CTC ASR baseline, the streaming decoding method provided in this application achieves absolute improvements in accuracy of 10.4% and 6.6%, respectively. Furthermore, the CDC-enhanced version delivers an absolute gain of 6.8% in recall and reduces the false alarm rate by 46.3% with a false alarm rate of only 0.05 per hour, outperforming state-of-the-art graph-based CTC KWS methods. Therefore, it not only provides excellent performance but also demonstrates strong robustness under extremely low SNR and out-of-domain SNR conditions. Moreover, the method is easy to implement and maintain, making it highly suitable for practical applications and further improvements.

[0094] Therefore, the streaming decoding algorithm enhanced by CDC information not only compresses the decoding space and improves decoding efficiency, but also utilizes the discovered consistency of the score sequences between the intermediate and final layers to propose a CDC-enhanced decoding method, further improving the performance of the KWS task. In addition, the system has a significant effect on improving KWS performance under low signal-to-noise ratio conditions, which is of great significance for raising the performance ceiling of keyword detection technology and further promoting its application.

[0095] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0096] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the above-described speech keyword detection methods of this application.

[0097] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described speech keyword detection methods.

[0098] In some embodiments, this application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a voice keyword detection method.

[0099] Figure 6 This is a schematic diagram of the hardware structure of an electronic device for performing a speech keyword detection method according to another embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0100] One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0101] The device for performing the speech keyword detection method may further include an input device 630 and an output device 640.

[0102] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0103] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the speech keyword detection method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the speech keyword detection method in the above-described method embodiments.

[0104] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] Input device 630 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. Output device 640 may include display devices such as a display screen.

[0106] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, they execute the speech keyword detection method in any of the above method embodiments.

[0107] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0108] The electronic devices in this application embodiments exist in various forms, including but not limited to:

[0109] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0110] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include: PDAs, MIDs, and UMPCs, etc.

[0111] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0112] (4) Other airborne electronic devices with data interaction capabilities, such as vehicle-mounted systems installed on vehicles.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting speech keywords, comprising: Based on the multiple cascaded coding layers in the CTC acoustic model, deep feature encoding is performed on the speech to be detected. Based on the intermediate decoding layer in the CTC acoustic model, the intermediate coding features output by the first coding layer are decoded to determine the first prediction score sequence for the target keyword; the first coding layer is any one of the pre-coding layers in the cascaded multiple coding layers except the last coding layer. Based on the final decoding layer in the CTC acoustic model, the deep coding features output by the cascaded coding layers are decoded to determine the second prediction score sequence for the target keyword; Calculate the similarity between the first predicted score sequence and the second predicted score sequence, and calibrate the second predicted score sequence based on the similarity to determine the target predicted score sequence corresponding to the target keyword; The step of calculating the similarity between the first predicted score sequence and the second predicted score sequence, and calibrating the second predicted score sequence based on the similarity to determine the target predicted score sequence corresponding to the target keyword, includes: Based on the cosine similarity function, the score similarity sequence between the first predicted score sequence and the second predicted score sequence is calculated; The score similarity sequence and the second predicted score sequence are added together and normalized to determine the target predicted score sequence.

2. The method according to claim 1, wherein, The CTC acoustic model is trained based on a speech sample set and a comprehensive loss function; the comprehensive loss function is: ， In the formula, This represents the loss function of the CTC acoustic model. It is a hyperparameter and , This represents the intermediate CTC loss term determined based on the prediction output of the intermediate decoding layer for the speech sample, and This represents the final CTC loss term determined based on the prediction output of the final decoding layer for the speech sample.

3. The method according to claim 1, wherein, The intermediate decoding layer and the final decoding layer are used to decode the encoded features corresponding to the speech to be detected frame by frame according to a preset time frame scale, and the target prediction score sequence is a frame-level decoding score sequence accordingly.

4. The method according to claim 3, wherein, The preset time frame scale is used to indicate the current time frame, historical frame window, and future frame window.

5. The method according to claim 1, wherein, The first coding layer is the coding layer that is in the middle position among the multiple cascaded coding layers.

6. The method according to any one of claims 1-5, wherein, The encoding layer uses a DFSMN layer.

7. A storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

8. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-6.

9. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-6.

Images