Voice keyword detection method, storage medium and electronic equipment

Abstract

The invention provides a voice keyword detection method, a storage medium and electronic equipment, and relates to the technical field of voice signal processing, and the method comprises the steps: carrying out the depth feature coding of a to-be-detected voice based on a plurality of cascaded coding layers in a CTC acoustic model; based on an intermediate decoding layer in the CTC acoustic model, decoding an intermediate coding feature output by the first coding layer to determine a first prediction score sequence for the target keyword; based on a final decoding layer in the CTC acoustic model, decoding depth coding features finally output by the cascaded coding layers to determine a second prediction score sequence for the target keyword; and calculating the similarity between the first prediction score sequence and the second prediction score sequence, and calibrating the second prediction score sequence based on the similarity to determine a target prediction score sequence corresponding to the target keyword. Therefore, through voice keyword detection with cross-layer discrimination consistency, a keyword detection effect with higher precision is realized.

Description

Technical Field

[0001] The present application relates to the technical field of speech signal processing, and in particular to a speech keyword detection method, a storage medium and an electronic device. Background Art

[0002] Keyword Spotting (KWS), especially Wake Word Detection (WWD), involves continuously detecting the presence of preset keywords in the audio stream. The wake word detection (KWS) systems of major voice providers and the wake-up systems on mobile devices, such as Huawei's "Xiaoyi", Baidu's "Xiaodu Xiaodu", Xiaomi's "Xiaoai Tongxue" and Apple's "hey Siri". These technologies or products detect whether the preset keywords appear in the continuous speech stream and wake up the device when the wake word appears.

[0003] The current keyword detection system mainly includes the Automatic Speech Recognition (ASR) detection system and the Weighted Finite State Transducer (WFST)-based detection system. In the ASR-based decoding algorithm, all possible phoneme sequences are considered, resulting in a large search space: on the one hand, this leads to low efficiency, and on the other hand, the performance of the keyword detection technology cannot reach the optimal level; however, the algorithm that limits the search space will cause the decoding output to change (from the phoneme sequence of the keyword to the probability of the keyword appearing in each frame), resulting in the inability to efficiently support streaming decoding. In the keyword detection technology based on the WFST token passing technology, the search space is limited to the phonemes and filling units of the keyword, but the underlying implementation of such a system code is complex and difficult to maintain and improve.

[0004] The industry has not yet proposed a better solution to the above problems. Summary of the invention

[0005] The present application provides a voice keyword detection method, a storage medium and an electronic device, which are used to at least solve the problem of poor voice keyword detection performance in the current related technology.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting speech keywords, comprising: based on multiple cascaded coding layers in a CTC acoustic model, performing deep feature coding on the speech to be detected; based on an intermediate decoding layer in the CTC acoustic model, decoding the intermediate coding features output by the first coding layer to determine a first prediction score sequence for a target keyword; the first coding layer is any pre-coding layer except the last coding layer in the multiple cascaded coding layers; based on a final decoding layer in the CTC acoustic model, decoding the deep coding features finally output by the cascaded coding layers to determine a second prediction score sequence for the target keyword; and calibrating the second prediction score sequence using the first prediction score sequence to determine a target prediction score sequence corresponding to the target keyword.

[0007] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the voice keyword detection method of any embodiment of the present application.

[0008] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the voice keyword detection method of any embodiment of the present application are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the voice keyword detection method of any embodiment of the present application.

[0010] The beneficial effects of the embodiments of the present application are:

[0011] By setting up a double decoding mechanism in the CTC (Connectionist Temporal Classification, CTC) acoustic model, the intermediate decoding layer is used to process the coding features of the front coding layer to achieve initial recognition of the target keywords, and the initial recognition results are used to calibrate the final deep decoding results. The speech keyword detection results can be optimized by learning the difference patterns of keyword recognition in different layers, and the speech keyword detection based on cross-layer discrimination consistency (Cross-layerDiscrimination Consistency, CDC) can achieve a more accurate keyword detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of an example of a method for detecting speech keywords according to an embodiment of the present application is shown;

[0014] Figure 2 A schematic diagram showing an example of the encoding and decoding process of the CTC acoustic model according to an embodiment of the present application;

[0015] Figure 3 A schematic diagram of the operation flow of reasoning and training of an enhanced CTC acoustic model according to an embodiment of the present application is shown;

[0016] Figure 4 A code screenshot of an example of a CTC-based KWS streaming decoding algorithm provided in an embodiment of the present application is shown;

[0017] Figure 5 The experimental simulation effect diagram of the frame-level decoding score mode for keyword detection of positive samples and negative samples is shown;

[0018] Figure 6 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] It should be noted that given that KWS systems usually run on resource-constrained devices, reducing computational and memory costs is critical. Through CTC, the need for frame-level alignment is avoided, providing an effective training criterion for this task.

[0021] The non-autoregressive nature of CTC makes it a natural fit for designing efficient streaming decoding algorithms. Existing continuous KWS decoding algorithms can be divided into two categories. The first category is based on Automatic Speech Recongnition (ASR) reasoning. In this approach, the model processes speech and outputs a sequence of symbols in a streaming mode, and then matches keywords in the updated decoded sequence. However, the ASR decoding strategy aims to generate the most likely complete hypothesis in the entire space, rather than specifically detecting the presence of keywords, and is therefore not ideal for KWS. The second category relies on the KWS decoding graph, which typically includes a keyword path and an optional fill path. Although this approach is specifically targeted at KWS, it is complex to implement a graph and symbol transfer algorithm based on a weighted finite state transducer (WFST) for decoding, making it difficult to maintain and improve. In addition, the KWS performance of these systems at low signal-to-noise ratios is not good enough, especially in extremely low signal-to-noise ratios, where better results cannot be achieved.

[0022] It should be noted that the search space of the ASR-based decoding algorithm does not restrict the search space for the KWS task, resulting in suboptimal performance and results. The WFST-based decoding algorithm is based on automata theory, has a complete systematization theory, and better supports streaming decoding. Therefore, the system built based on such a theory is naturally more complex. In addition, the above algorithm is not optimal, so the system's anti-noise robustness is not strong enough.

[0023] It should also be emphasized that some current research and optimization are carried out for the above two types of systems. Specifically, for the ASR-based decoding algorithm, it improves the efficiency and performance of the KWS decoding algorithm by restricting the search space, but this cannot provide efficient support for streaming decoding. In addition, the WFST system is more difficult to improve and maintain.

[0024] Figure 1 An operation flow chart of an example of a method for detecting speech keywords according to an embodiment of the present application is shown.

[0025] like Figure 1 As shown, in step S110, based on multiple coding layers cascaded in the CTC acoustic model, deep feature encoding is performed on the speech to be detected.

[0026] Here, multiple coding layers in the CTC acoustic model can be used to frame the input speech signal and extract features, such as preprocessing using Mel frequency cepstral coefficients (MFCC) or spectrograms. Then, more abstract features are extracted layer by layer through multiple coding layers to form a deep feature representation. It should be understood that the structural selection of the coding layer can be diverse, such as convolutional neural network (CNN) or recurrent neural network (RNN), etc., which is not limited here.

[0027] It should be noted that the CTC acoustic model is a deep learning model for sequence data processing, which allows the input and output sequence lengths to not match. In speech keyword detection, the number of input audio frames is usually greater than the number of characters of the target keyword. This variable-length feature makes CTC very suitable for handling such tasks. In addition, CTC does not require manually annotated alignment information (i.e., the correspondence between input and output), which simplifies the data preparation process, especially when the speech data set is large, reducing the annotation cost. Moreover, CTC can automatically learn how to align input and target output during training. During the training phase, the CTC loss function can be used for optimization so that the model can accurately predict the timing of keywords.

[0028] In step S120, based on the intermediate decoding layer in the CTC acoustic model, the intermediate encoding features output by the first encoding layer are decoded to determine a first prediction score sequence for the target keyword.

[0029] Here, the first coding layer is any pre-coding layer except the last coding layer in the cascaded multiple coding layers. Different from the conventional CTC acoustic model, in addition to decoding the final coding features, the intermediate coding features output by the first coding layer are also preliminarily decoded through the intermediate decoding layer, and the resulting prediction score sequence can represent the probability of occurrence of the target keyword in each time frame.

[0030] In step S130 , based on the final decoding layer in the CTC acoustic model, the deep coding features finally output by the cascaded coding layers are decoded to determine a second prediction score sequence for the target keyword.

[0031] Here, the deep coding features finally output by all the cascaded coding layers are decoded by the final decoding layer, and a second prediction score sequence is generated by integrating more comprehensive information.

[0032] In step S140, the similarity between the first prediction score sequence and the second prediction score sequence is calculated, and the second prediction score sequence is calibrated based on the similarity to determine a target prediction score sequence corresponding to the target keyword.

[0033] Here, various non-limiting similarity algorithms can be used to quantify the similarity between the first prediction score sequence and the second prediction score sequence, such as cosine similarity, Euclidean distance or Pearson correlation coefficient. Then, the second prediction score sequence is adjusted using the calculated similarity, such as by using a linear weighting or nonlinear adjustment method, so that the calibrated score sequence is more consistent with the recognition of the actual target keyword.

[0034] Figure 2 A schematic diagram showing an example of the encoding and decoding process of the CTC acoustic model according to an embodiment of the present application is shown.

[0035] like Figure 2 As shown, after the keyword audio enters the encoder, the representations of the intermediate layer and the final layer will be sent to the streaming decoding algorithm provided in this application; then, the CDC module uses two score sequences to calculate the similarity, adds the similarity score to the final layer score and normalizes it as the final output.

[0036] It should be noted that a frame-synchronous streaming decoding algorithm for CTC-based KWS is provided in the embodiment of the present application. In the process of practicing the present application, the inventor of the present application found that near the wake-up point, there are significant differences in the behavior of the intermediate layer and the final layer of the positive sample and the easily triggered negative sample, that is, the two have a large difference in similarity performance. Therefore, through the embodiment of the present application, in order to further enhance the discrimination between challenging negative samples and positive samples, the cross-layer discrimination consistency (CDC) information is used to implement an effective CDC-enhanced streaming KWS decoding algorithm.

[0037] Experiments have shown that the decoding output assistance of the middle layer can effectively help the system better distinguish between positive and negative examples, and the performance of the streaming decoding algorithm can be further improved by enhancing the consistency of cross-layer discrimination. More details will be discussed in conjunction with other parts below.

[0038] In some examples of the embodiments of the present application, the CTC acoustic model is trained based on a speech sample set and a comprehensive loss function, where the comprehensive loss function is:

[0039]

[0040] In the formula, represents the loss function of the CTC acoustic model, w is a hyperparameter and w∈[0,1), represents the intermediate CTC loss term determined based on the predicted output of the intermediate decoding layer for the speech sample, and Represents the final CTC loss term determined based on the predicted output result of the final decoding layer for the speech sample.

[0041] Specifically, given the acoustic feature vector of an utterance Where T is the number of frames, D is the feature dimension, and the CTC model aims to predict the target label sequence. where U is the number of target labels. CTC allows repeated symbols and introduces a special blank symbol φ to facilitate the alignment between the acoustic sequence and the label sequence. The alignment path is denoted by Represented. CTC aims to maximize the likelihood of the target label sequence y on all possible alignment paths π. The loss function can be expressed as:

[0042]

[0043] Where B represents the mapping of the alignment path π to the label sequence y, and B -1 represents the inverse mapping. All arithmetic operations are performed in the logarithmic domain. In addition, intermediate CTC (ICTC) regularization is applied to the intermediate layers of the encoder, which can be used as an effective regularization strategy for CTC training.

[0044] In some examples of the embodiments of the present application, the coding layer adopts the DFSMN (Deep Feedforward Sequential Memory Networks) layer, which has good memory and modeling capabilities, and can effectively capture long-term dependencies in speech signals. By introducing a memory module, the historical information in the sequence data can be more effectively utilized, thereby improving the modeling accuracy and performance of the signal. In the deep coding structure based on the DFSMN layer, each layer of DFSMN can perform specific processing and feature extraction on the input data, and the combination of multiple layers can gradually improve the coding performance. However, as mentioned earlier, the structural selection of the coding layer can be diverse, and it is not limited here for the time being. DFSMN is just a lightweight coding layer that can better capture contextual dependencies. Replacing it with other encoders should also be within the scope of implementation of the embodiments of the present application.

[0045] Regarding the position of the intermediate decoding layer, a too shallow coding layer mainly focuses on feature extraction and cannot effectively capture keyword information, while a too deep coding layer learns content similar to the main CTC, so it is also impossible to extract CDC information ideally. In view of this, in some examples of the embodiments of the present application, the first coding layer is a coding layer in the middle position of the cascaded multiple coding layers, for example, inserting an intermediate decoding layer at a position near half of the total number of layers can achieve better performance.

[0046] In some examples of the embodiments of the present application, the intermediate decoding layer and the final decoding layer are used to decode the coding features corresponding to the speech to be detected frame by frame according to a preset time frame scale, and the corresponding target prediction score sequence is a frame-level decoding score. Here, the time frame scale can be comprehensively determined based on the current time frame and adjacent time, such as multiple historical and / or future time frames, and the similarity between the score sequences output by the intermediate decoding layer and the final decoding layer can also be dynamically analyzed by sliding the time window, and then a frame-level decoding score sequence that is dynamically analyzed and updated is obtained.

[0047] In some examples of the embodiments of the present application, a preset time frame scale is used to indicate the current time frame, the historical frame window, and the future frame window. In this way, by including both historical information and future information in the time window, the effectiveness of the discriminative information provided by the historical features and the future features near the activation point can be comprehensively considered, and the performance of the streaming keyword recognition based on CDC enhancement can be further improved. The analysis of the historical / future window size involved in the CDC score will be expanded in conjunction with other examples below.

[0048] In some examples of the embodiments of the present application, the calculation details of the CDC score can be to calculate the score similarity sequence between the first prediction score sequence and the second prediction score sequence based on the cosine similarity function, and then add and normalize the score similarity sequence and the second prediction score sequence to determine the target prediction score sequence.

[0049] Figure 3 A schematic diagram of the operational flow of reasoning and training of an enhanced CTC acoustic model according to an embodiment of the present application is shown.

[0050] like Figure 3 As shown in Figure 1, it shows the model training and decoding framework combined with the "Hey Snips" dataset. An additional linear projection layer with ICTC is attached to the middle part of the encoder. L represents the number of DFSMN layers. CDC enhanced decoding includes a refinement stage. s(init), s(inter), s(cdc), and s(refine) represent the initial CTC decoding score, the intermediate CTC decoding score, the CDC score between the initial score and the intermediate score, and the final refined score, respectively. f CDC represents the function used to measure the consistency of discrimination. His. and L Fut. Respectively refer to the number of historical frames and the number of future frames used for CDC score calculation.

[0051] Through the embodiments of the present application, a new streaming decoding algorithm is designed for CTC-based KWS. Unlike the ASR-based method that needs to search the entire space and can only search from the beginning of the speech, the algorithm provided by the present application limits the search space to keywords, allowing detection at any time point. In addition, compared with complex graphic decoding methods, this algorithm is easier to implement. In addition, based on the discovery that positive and negative samples perform differently between layers, the difference between keyword and non-keyword speech segments is captured using cross-layer discrimination consistency. The effectiveness of CTC-KWS streaming decoding is further improved by combining CDC information. On the "Hey Snips" dataset, compared with a variety of ASR-based and graphic benchmarks, the CDC enhanced streaming KWS decoding algorithm provided by the embodiments of the present application shows impressive performance at different signal-to-noise ratio (SNR) levels.

[0052] For the description of CTC-based KWS streaming decoding, the core concept of KWS streaming decoding is to limit the search to keywords at each time step. Unlike ASR decoding, which aims to generate complete hypotheses, the method of this application narrows the decoding space to specific paragraphs related to keywords in the semantic space. Specifically, for the phoneme posteriors related to keywords, the path with the highest probability of keyword detection or partial occurrence is selected at each decoding state.

[0053] Traditional ASR or graph-based decoding methods are less flexible and require a lot of effort for online adaptation. In contrast, the search algorithm provided in this application makes it possible to achieve seamless keyword detection in streaming scenarios. At the beginning of each frame, two new path competitors are introduced with their probabilities initialized to 1, which allows them to be seamlessly integrated into the decoding process and ensures that new competitors entering at time step can effectively participate in decoding. Considering the potential length of the keyword, a timeout mechanism is also implemented to improve the search efficiency. For more details, see Figure 4 A screenshot of the program code of the CTC-based KWS streaming decoding algorithm provided.

[0054] In addition to applying the above CTC-based KWS streaming decoding algorithm on the main CTC layer, shallow intermediate branches can also be used for inference. The initial scores and intermediate scores calculated from the main CTC layer and the intermediate CTC branch are expressed as:

[0055]

[0056] Preliminary experiments show that the scores of the main CTC layer and the intermediate CTC layer show significantly different characteristics when processing positive and negative samples.

[0057] Figure 5The experimental simulation effect diagram of the frame-level decoding score mode for keyword detection of positive samples (above) and negative samples (below) is shown. Figure 5 As shown, near the arousal point, the two score curves for positive samples appear stable and similar. However, for high-scoring negative samples, the activation time is shorter and the two score curves diverge significantly. This difference may stem from the fact that the patterns in the positive samples are relatively consistent, allowing the ICTC layer to effectively learn these positive patterns despite its limited parameters. However, negative samples contain a variety of patterns and conditions, making it difficult for shallow layers to fully capture all relevant aspects. Even if the final layer exhibits overfitting to these false alarm patterns, ICTC can still serve as a regularization mechanism. Among them, the red box highlights the change in the CDC score near the activation point. It is worth noting that for positive samples, the score always remains close to 1 near the arousal point, while negative samples show violent fluctuations.

[0058] Therefore, considering the different performances of positive and negative samples, the sliding window CDC score at time t is defined as:

[0059]

[0060] In the formula, f CDC represents the function used to measure the discriminative consistency score, where the cosine similarity function can be used. L His. and L Fut. Represent the historical frame number and the future frame number respectively. Figure 2 The CDC score curve s(cdc) is also plotted in . The refined score of the multi-stage decoding algorithm enhanced by CDC is the sum of the initial CTC score and the CDC score:

[0061]

[0062] In the formula, It should be noted that the score calculation is performed frame by frame within the full streaming decoding framework. Fut. The value of can be adjusted to balance the trade-off between performance of the refining phase and system latency.

[0063] The following will expand on the details of the experiments and evaluations related to the embodiments of the present application.

[0064] The experiments in this application example are based on the following data sets:

[0065] -LibriSpeech is a large-scale English automatic speech recognition (ASR) dataset containing about 960 hours of speech and its corresponding transcriptions. Use LibriSpeech to preliminarily train a CTC-based acoustic model to obtain a good base model.

[0066] -Hey-Snips (Snips) is an open source keyword detection (KWS) dataset that contains the wake-up word "HeySnips". The positive sample part of the dataset contains 5,799, 2,484, and 2,599 keyword speech samples in the training set, development set, and test set, respectively. Due to the lack of transcription content, negative samples cannot be used for ASR-based training, so the Snips dataset was reorganized. All original negative samples are compiled into a large negative sample test set, about 97 hours of speech.

[0067] -WHAM! is an environmental noise dataset that contains various types of environmental noise. The clean Hey-Snips speech samples are combined with the noise randomly drawn from the WHAM! dataset to synthesize the noisy keyword samples to evaluate the robustness of the model under different signal-to-noise ratios (SNR).

[0068] The configuration instructions are as follows:

[0069] During the training process, the model can be pre-trained using the complete LibriSpeech dataset first. In the fine-tuning stage, the same proportion of data is extracted from LibriSpeech and Snips to construct a general ASR dataset. In order to enhance the performance of the model in a noisy environment, this application mixes each clean speech in Snips and LibriSpeech with a noise sample in WHAM!. The signal-to-noise ratio (SNR) of the mixed speech is evenly distributed between 0dB and 20dB. In the end, the fine-tuning dataset contains a total of about 24,000 speech samples (12,000 clean samples and 12,000 noise samples).

[0070] The acoustic characteristics are as follows:

[0071] A 40-dimensional log-Mel filter bank coefficient (FBank) is extracted from each speech, using a window size of 25 milliseconds and a sliding window size of 10 milliseconds. Two data augmentation methods are applied during training: (1) online speech perturbation, where the distortion factor is uniformly sampled from {0.9, 1.0, 1.1}; (2) SpecAugment, using 2 time domain masks with a maximum mask length of Tmax=50 and 2 frequency domain masks with a maximum frequency domain width of fmax=10. The FBank features of the first 5 frames and the last 5 frames are concatenated to generate a 440-dimensional input. In order to reduce the input frame rate, a frame skip factor of 3 frames is applied, and each input frame finally contains 30 milliseconds of acoustic information.

[0072] The description of the model and loss is as follows:

[0073] The encoder consists of 6 DFSMN layers with dimensions of 440, 512, and 320 for the input, hidden, and projection layers, respectively. The left and right orders of the DFSMN layers are set to 8 and 2, respectively. The CMU pronunciation dictionary “cmudict-0.7b” is used to define the final acoustic output units. In addition to the 70 phonemes, a unique blank symbol is added for CTC modeling. In the pre-training stage, only the frame-level cross entropy criterion can be used. In the fine-tuning stage, depending on the model, CTC or a comprehensive loss function combining CTC with ICTC is used. Unless otherwise specified, the weight w of the ICTC loss term in Equation (1) can be set to 0.3.

[0074] The following are the descriptions of the assessment:

[0075] In order to accurately evaluate the performance of the keyword detection system under different signal-to-noise ratio (SNR) scenarios, each positive sample is mixed with noise with SNR levels of {0, 5, 10, 15, 20} dB. In addition, a positive sample set with an SNR of -5 dB is created to evaluate the robustness of the system under extremely low SNR and out-of-domain SNR conditions. As mentioned earlier, all negative samples in the original Snips dataset are collected for testing. The negative sample set is also mixed with noise with SNR ranging from 0 dB to 20 dB. Decoding hyperparameters S Bonus and T out Set to e 3 and 3 seconds.

[0076] For the evaluation metrics, the recall rate (i.e., 1-missing rate) and the macro-average recall rate at different SNR levels can be evaluated, all at a fixed false alarm rate (FAR). For ASR-based methods, this application reports the accuracy (Acc.), which is equivalent to the recall rate at FAR=0, because these systems have challenges in decoding false alarms. In addition, when comparing with the KWS graphics system, this application shows the results at a false alarm rate of 0.05 per hour, which is a more reasonable but strict evaluation standard for the KWS system.

[0077] In Table I, this application compares three mainstream decoding baseline methods, all of which use the same CTC-based acoustic model. CTC greedy decoding and prefix beam search are both ASR-based systems. The streaming method of this application significantly outperforms these three ASR-based and graph-based baseline methods. Compared with the two ASR-based baselines, the streaming KWS decoding achieved an absolute improvement of 10.4% and 6.6% at different SNR levels, respectively. Even compared with the strong graph-based baseline, the streaming decoding strategy brought an absolute gain of 6.3%. The results show that the proposed method can perform well in different noise environments.

[0078] Table I compares the accuracy of the proposed streaming algorithm with three baseline systems (FAR=0): (1) ASR greedy search,

[0079] (2) ASR prefix beam search, (3) graph-based KWS search.

[0080]

[0081] There are several important details to note. The method improves accuracy by about 10% or more under low SNR conditions, such as -5 or 0dB. This improvement may be because at very low SNRs, ASR-based decoding has difficulty generating correct hypotheses due to excessive noise in the speech. For graph-based methods, it is difficult for decoded symbols to pass through the decoding graph, which prevents them from reaching the final state, resulting in lower accuracy for the graph-based baseline. However, in the present application, the decoding is forced to output activation scores frame by frame, allowing it to continue working in strong noise environments. This shows that the method provided in the embodiments of the present application is particularly effective in complex and challenging acoustic environments.

[0082] The importance of the CDC strategy for improving streaming decoding algorithms is emphasized below. As shown in Table II, when comparing model pairs using the same decoding strategy (such as A and C and B and D), it is observed that the gain of ICTC at low SNR levels is more significant, which shows the effectiveness of CTC regularization in noisy scenes. In addition, the comparison of models E and D further demonstrates the effectiveness of the CDC strategy proposed in this application. Overall, compared with the baseline (A) of the CDC enhanced system (E), the average recall rate of this application at different SNR levels is improved by 6.8%, and the average false negative rate is relatively reduced by 46.3%.

[0083] Table II. Recall comparison of the proposed KWS streaming decoding and multi-stage decoding strategies with the graph-based baseline at different SNR levels, with a fixed hourly false alarm rate of 0.05.

[0084]

[0085] The KWS decoding algorithm provided by the embodiment of the present application abandons the traditional ASR decoding implementation, limits the decoding search space, supports streaming decoding, and introduces a timeout and reward mechanism, which effectively solves the problem of KWS performance loss for streaming decoding implementation. In addition, the decoding output of the middle layer will effectively help the system to better distinguish between positive and negative examples, and enhance the decoding through CDC information, so that the performance of the streaming decoding algorithm is further improved. In addition, the system does not require WFST-related theory, only PyTorch and time connection implementation are required. Experiments have shown that the system has a significant effect on improving low signal-to-noise ratios, especially extremely low signal-to-noise ratios, and KWS performance.

[0086] The CDC strategy exploits the fractional pattern around the wakeup point, which inevitably introduces some latency. Therefore, we further analyze the history window size (L His. ) and the future window size (L Fut. ) on the KWS performance. As described in other sections above, each frame contains 30 milliseconds of acoustic information. Here, the total window size is fixed to 30 frames, resulting in a maximum delay of 900 milliseconds, which is acceptable in practical deployments. As shown in Table III, increasing the number of future frames can improve the performance, which emphasizes the effectiveness of the discriminative information provided by future features near the activation point.

[0087] Table III. Comparison of recall performance of CDC enhanced decoding using different window sizes. The results of the system proposed in this application are bolded and underlined, and the upper bound offline results for reference are also bolded.

[0088]

[0089] Here, we propose two upper bound baselines: one that uses all future scores and another that uses all scores in the entire sentence. Fut. = 30, the performance gain has reached its maximum, and further adding future frames brings negligible performance improvement, which indicates that too much discriminative information will bring diminishing returns. Introducing a slight delay (less than 1 second) is enough to achieve a performance gain close to the upper bound.

[0090] In addition, the impact of the intermediate loss weight w in equation (1) and the number of layers of ICTC application on system performance is further analyzed. The results in Table IV show that when the number of layers is fixed, the value of w = 0.3 can best balance L ICTC In addition, experiments on the number of applied layers show that placing the ICTC term near half of the total number of layers L (i.e., L / 2) can achieve the best performance. The specific reason is that too shallow layers mainly focus on feature extraction and cannot effectively capture keyword information, while too deep layers learn content similar to the main CTC, so they cannot ideally extract CDC information.

[0091] Table IV. Recall results for different w values ​​and number of ICTC layers applied. The encoder contains L=6 layers in total, and the optimal number of layers is L / 2=3.

[0092]

[0093] Through the embodiments of the present application, a CDC enhanced streaming KWS decoding algorithm designed specifically for CTC-based systems is proposed. Specifically, a streaming decoding algorithm capable of detecting any starting point of a keyword in an audio stream is proposed, and the performance is further improved by utilizing the CDC scores from the intermediate and final layers. Compared with the greedy search and prefix beam search in the CTC ASR baseline, the streaming decoding method provided in the present application achieved an absolute improvement of 10.4% and 6.6% in accuracy, respectively. In addition, the CDC enhanced version brought an absolute gain of 6.8% in recall rate, and reduced the false alarm rate by 46.3% with a false alarm rate of only 0.05 times per hour, which is better than the current most advanced graph-based CTC KWS method. Therefore, it not only provides excellent performance, but also demonstrates strong robustness under extremely low signal-to-noise ratio and out-of-domain signal-to-noise ratio conditions. In addition, the method is easy to implement and maintain, and is very suitable for practical applications and further improvements.

[0094] Therefore, the streaming decoding algorithm enhanced by CDC information not only compresses the decoding space but also improves the decoding efficiency. By utilizing the consistency of the score sequence between the intermediate layer and the final layer, a CDC enhanced decoding method is proposed to further improve the performance of the KWS task. In addition, the system has a significant effect on improving the KWS performance under low signal-to-noise ratio, which is of great significance for improving the performance upper limit of keyword detection technology and further promoting its application.

[0095] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0096] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any of the above-mentioned voice keyword detection methods of the present application.

[0097] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any one of the above-mentioned voice keyword detection methods.

[0098] In some embodiments, an embodiment of the present application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a voice keyword detection method.

[0099] Figure 6 is a schematic diagram of the hardware structure of an electronic device for executing a voice keyword detection method provided by another embodiment of the present application, such as Figure 6 As shown, the device includes:

[0100] One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example.

[0101] The device for executing the voice keyword detection method may further include: an input device 630 and an output device 640 .

[0102] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0103] The memory 620 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the voice keyword detection method in the embodiment of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, the voice keyword detection method in the above method embodiment is implemented.

[0104] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] The input device 630 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 640 can include a display device such as a display screen.

[0106] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the voice keyword detection method in any of the above method embodiments is executed.

[0107] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0108] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0109] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0110] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0111] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0112] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0113] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting speech keywords, comprising: Based on multiple cascaded coding layers in the CTC acoustic model, deep feature encoding is performed on the speech to be detected; Based on the intermediate decoding layer in the CTC acoustic model, the intermediate coding features output by the first coding layer are decoded to determine a first prediction score sequence for the target keyword; the first coding layer is any pre-coding layer except the last coding layer in the cascaded multiple coding layers; Based on a final decoding layer in the CTC acoustic model, decoding the deep coding features finally output by the cascaded coding layer to determine a second prediction score sequence for the target keyword; A similarity between the first prediction score sequence and the second prediction score sequence is calculated, and the second prediction score sequence is calibrated based on the similarity to determine a target prediction score sequence corresponding to the target keyword.

2. The method according to claim 1, wherein: The CTC acoustic model is trained based on a speech sample set and a comprehensive loss function; the comprehensive loss function is: In the formula, represents the loss function of the CTC acoustic model, w is a hyperparameter and w∈[0,1), represents an intermediate CTC loss term determined based on the predicted output result of the intermediate decoding layer for the speech sample, and represents the final CTC loss term determined based on the predicted output result of the final decoding layer for the speech sample.

3. The method according to claim 1, wherein: The intermediate decoding layer and the final decoding layer are used to decode the coding features corresponding to the speech to be detected frame by frame according to a preset time frame scale, and accordingly the target prediction score sequence is a frame-level decoding score sequence.

4. The method according to claim 3, wherein: The preset time frame scale is used to indicate the current time frame, the historical frame window and the future frame window.

5. The method according to claim 1, wherein: The first coding layer is a coding layer located in the middle of the cascaded multiple coding layers.

6. The method according to any one of claims 1 to 5, wherein: The coding layer adopts the DFSMN layer.

7. The method according to claim 1, wherein: The calculating the similarity between the first prediction score sequence and the second prediction score sequence, and calibrating the second prediction score sequence based on the similarity to determine a target prediction score sequence corresponding to the target keyword, comprises: Calculating a score similarity sequence between the first prediction score sequence and the second prediction score sequence based on a cosine similarity function; The score similarity sequence and the second prediction score sequence are added and normalized to determine a target prediction score sequence.

8. A storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images