Keyword detection method, electronic device, and storage medium

Abstract

This invention discloses a keyword detection method, an electronic device, and a storage medium. The keyword detection method is used in a keyword detection system, which includes a pre-trained acoustic model, an attention module, and a triggering module. The method includes: processing audio input using the pre-trained acoustic model to obtain frame-level acoustic features; using the attention module to interact with the acoustic features and the textual information of the keywords to determine whether each phoneme in the keyword is found in the frame-level acoustic features and outputting a determination result; and using the triggering module to determine the final wake-up state based on the determination result. By introducing a pre-trained acoustic model, an attention module, and a triggering module into the keyword detection system, a streaming, customizable keyword discovery method can be provided. The search process of the attention module is decoupled from the acoustic model, thus eliminating reliance on the acoustic model and allowing easy expansion to different acoustic models.

Description

Technical Field

[0001] This invention belongs to the field of keyword detection technology, and particularly relates to keyword detection methods, electronic devices, and storage media. Background Technology

[0002] In related technologies, smart home appliances and voice assistants are being deployed more and more widely. As a cutting-edge component, the keyword discovery module (also known as wake word detection or voice triggering) should quickly and accurately trigger subsequent modules upon hearing the wake word. Therefore, the accuracy and latency of the keyword recognition system are crucial to user experience.

[0003] Keyword recognition tasks require models to discover keywords in continuous speech streams with limited computational power. Traditional keyword recognition frameworks are deep neural networks (DNNs) with unique non-neural search methods. Based on their dependence on Hidden Markov Models (HMMs), these methods can be categorized into HMM-based and HMM-free methods. HMM-based methods add an HMM to the DNN, which qualitatively classifies the audio input as keywords and fillers. With the success of deep learning, some works have proposed building HMM-free systems on top of DNNs without HMMs, directly predicting keywords or sub-word tags of keywords. These methods generally consist of an acoustic model and a post-processing module. The acoustic model architecture includes convolutional neural networks (CNNs), recurrent neural networks (RNNs), attention mechanisms, and other composite networks. Furthermore, some methods based on connectionist temporal classifier (CTC) systems and RNN-Transducer systems have been proposed to reduce dependence on permutations. These sequence-to-sequence methods typically require beam searching during the decoding or detection stages; however, the associated computational overhead limits their application in many cases.

[0004] Therefore, although the aforementioned efforts in keyword discovery tasks have greatly improved performance under certain specific conditions, some unresolved issues limit the generalizability of these methods.

[0005] In developing this application, the inventors discovered that: 1) Traditional keyword discovery pipelines are complex. The unique non-neural search process leads to additional overhead and a mismatch between training and testing. Furthermore, many tunable hyperparameters are typically required during the detection phase to reduce the false positive rate. 2) Most methods are designed for predefined keyword scenarios and are not suitable for supporting user-customizable keywords. In customizable keyword scenarios, their performance often drops sharply, or even becomes completely unsuitable. Although some work has addressed customizable keyword discovery tasks, practical challenges remain, such as the additional computational cost of adaptive detection, the need for streaming models, and insufficient data. Summary of the Invention

[0006] This invention provides a keyword detection method, an electronic device, and a storage medium to at least solve one of the above-mentioned technical problems.

[0007] In a first aspect, embodiments of the present invention provide a keyword detection method for a keyword detection system, wherein the keyword detection system includes a pre-trained acoustic model, an attention module, and a triggering module. The method includes: processing audio input using the pre-trained acoustic model to obtain frame-level acoustic features; using the attention module to interact with the acoustic features and the text information of the keywords to determine whether each phoneme in the keyword is found in the frame-level acoustic features and outputting a determination result; and using the triggering module to determine a final wake-up state based on the determination result.

[0008] In a second aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the keyword detection method of any embodiment of the present invention.

[0009] Thirdly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the keyword detection method of any embodiment of the present invention.

[0010] The method in this application embodiment, by introducing a pre-trained acoustic model, attention module, and triggering module into the keyword detection system, can provide a streaming customizable keyword discovery method. The search process of the attention module is decoupled from the acoustic model, so it does not depend on the acoustic model and can be easily extended to different acoustic models. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a keyword detection method provided in an embodiment of the present invention;

[0013] Figure 2 This is a keyword detection system framework provided in an embodiment of the present invention;

[0014] Figure 3 This is an embodiment of the wake word detection system structure of an end-to-end streaming neural network provided by the present invention;

[0015] Figure 4 A specific example of a keyword "keyword" multiple tags is provided in an embodiment of the present invention;

[0016] Figure 5 A comparison of the recall results of the model provided in this application embodiment, as an embodiment of the present invention, with the recall results of the baseline model at a false positive rate of 2 times on the test-clean and test-other datasets;

[0017] Figure 6 Evaluation results of different sample strategies provided in an embodiment of the present invention;

[0018] Figure 7 Evaluation results of different labeling methods provided in one embodiment of the present invention;

[0019] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please refer to Figure 1 The diagram illustrates a flowchart of an embodiment of the keyword detection method of this application. The method described above can be used in a keyword detection system, wherein the keyword detection system includes a pre-trained acoustic model, an attention module, and a triggering module.

[0022] like Figure 1 As shown, in step 101, the pre-trained acoustic model is used to process the audio input to obtain frame-level acoustic features;

[0023] In step 102, the attention module interacts with the text information of the acoustic features and keywords to determine whether each phoneme in the keyword is found in the frame-level acoustic features and outputs the determination result.

[0024] In step 103, the triggering module determines the final wake-up state based on the determination result.

[0025] In this embodiment, by introducing a pre-trained acoustic model, attention module, and triggering module into the keyword detection system, a streaming customizable keyword discovery method can be provided. The search process of the attention module is decoupled from the acoustic model, so it does not depend on the acoustic model and can be easily extended to different acoustic models.

[0026] In some optional embodiments, the pre-trained acoustic model uses phonemes as recognition symbols and is trained with frame-aligned labels. After training, all parameters of the pre-trained acoustic model are frozen. Searches can be performed without changing the output of the acoustic model, achieving decoupling from the acoustic model.

[0027] In some optional embodiments, the step of using the attention module to interact with the textual information of the acoustic features and keywords to determine whether each phoneme in the keyword is found in the frame-level acoustic features and outputting the determination result includes:

[0028] The phoneme sequence of the keyword is processed using a bidirectional long short-term memory module, and the last hidden state of the bidirectional long short-term memory module is used as the word embedding of the keyword.

[0029] The word embedding and the frame-level acoustic features are judged through a cross-attention mechanism and the judgment result is output. The judgment result includes the attention score of each frame in the frame-level acoustic features and the output of each frame in the frame-level acoustic features.

[0030] In some optional embodiments, the triggering module employs a multi-label construction mechanism to model the length of the phoneme sequence of the keyword and the multi-label of the triggering stage. Each label in the multi-label represents a specific phoneme in the phoneme sequence being triggered. When all phonemes in the phoneme sequence are triggered, the keyword is triggered, causing a wake-up. In a further optional embodiment, multiple binary classifiers are used to model the multi-label.

[0031] In some optional embodiments, the attention module and the trigger module are jointly trained using attention loss and multi-label loss, wherein the attention loss is related to the attention score of the last attention layer of the attention module, and the multi-label loss is related to the multi-label. The attention loss primarily helps fuse text and acoustic information, which is beneficial for subsequent detection, while the multi-label loss is the main loss function and is crucial to the subsequent output results.

[0032] In some optional embodiments, the attention score includes an attention score for positive word embeddings and / or an attention score for padding embeddings. When a frame corresponds to a keyword, attention is restricted to activation on the positive word embedding. The positive word embedding and padding embedding are used simultaneously, and this score represents the association between the acoustic and textual information calculated by the model, helping the model better distinguish between keyword-based and non-keyword-based audio.

[0033] In some optional embodiments, the keyword is a wake word, and the keyword detection system is a wake word detection system (KWS).

[0034] It should be noted that the above method steps are not intended to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the reverse order of the steps, and this application does not impose any restrictions on this.

[0035] The following description addresses some problems encountered by the inventors in realizing this invention and provides a specific embodiment of the final solution, so that those skilled in the art can better understand the solution of this application.

[0036] The inventors found that the above defects were mainly caused by the following reasons: the end-to-end customizable wake word technology still has defects, and the accuracy and latency performance need to be improved.

[0037] To address the aforementioned deficiencies in related technologies, those skilled in the art typically employ alternative solutions to convert customizability into selectability, allowing users to choose wake words from a set of preset words, or allowing users to combine and set wake words using a certain number of preset syllables. The solution in this application allows users to set wake words completely arbitrarily with very short latency, placing greater demands on the model's modeling capabilities and anti-interference abilities, while also requiring the model to run more quickly.

[0038] In this embodiment, during model training, keywords are randomly selected and distractor words are generated for comparative training, enabling the model to arbitrarily customize wake words during use. Furthermore, the model in this embodiment operates entirely in a streaming manner, without relying on any post-processing, significantly reducing detection latency.

[0039] Please refer to Figure 2 The diagram shows a block diagram of a keyword detection system provided in an embodiment of this application.

[0040] The solution proposed in this application not only makes full use of a large amount of readily available data, greatly improving the customizability and robustness of the model, but also allows for the random customization of various wake words during deployment and training. Furthermore, the training process proposed in this application significantly reduces the data acquisition difficulty of the wake word detection system, resulting in a robust and low-latency model.

[0041] The following specific experiments and experimental data verify the beneficial effects of the embodiments of this application compared to the prior art.

[0042] In this application embodiment, to address the issues of model complexity and detection latency, a fully end-to-end streaming customizable keyword discovery system is proposed. This system utilizes an attention mechanism to integrate textual information and a multi-label mechanism to trigger the final wake-up. The method of this application embodiment can be seen as an extension of related technologies in customizable keyword discovery scenarios. The core contributions of the work of this application embodiment can be summarized as follows:

[0043] This application proposes a fully end-to-end streaming customizable keyword detection method with only one adjustable hyperparameter during the detection phase: the wake-up threshold. The fully neural network architecture enables joint optimization of the model during training.

[0044] - This application introduces a multi-label mechanism, which models the length information of keywords and the triggering stage into an end-to-end model, and for the first time regards the keyword discovery task as a multi-label classification task.

[0045] The neural search process proposed in this application is decoupled from the acoustic model, which is only responsible for providing the acoustic posterior. The neural network search process does not depend on a specific acoustic model and can be easily transferred or extended to different acoustic models.

[0046] 2. End-to-end method

[0047] The architecture and training process of the end-to-end model in the embodiments of this application will be described in detail below. The overall framework is as follows: Figure 1 As shown, it consists of three main parts: 1) a pre-trained acoustic model (AM) to provide frame-level acoustic posterior; 2) an attention module that interacts between acoustic and textual information to determine whether a specific keyword is detected in the current time period; and 3) a triggering module that determines the detection stage of a custom keyword, such as non-triggered, semi-triggered, and fully triggered.

[0048] Figure 3The structure of the model proposed in this application is shown in the embodiment. It includes a pre-trained acoustic model to provide frame-level acoustic posterior; an attention module to fuse acoustic and textual information; and a triggering module to model length information and triggering phases into a multi-label mechanism to determine the final wake-up. The English-Chinese translations are as follows: Pre-trained Acoustic Model (AM), Attention module, Trigger module, exists the keyword, Classifier, CrossAttention, Bidirectional Long Short-Term Memory Network (BiLSTM).

[0049] 2.1. Pre-trained acoustic model

[0050] The end-to-end model in this embodiment is built on a pre-trained acoustic model that processes audio input to provide a frame-level acoustic posterior. The pre-trained acoustic model uses phonemes as recognition symbols and is trained with frame-aligned labels. Cross-entropy loss is used as the standard for the training procedure. Once pre-training is complete, all parameters of AM are frozen during subsequent training.

[0051] 2.2. Attention Module

[0052] Before going into detail about the constrained attention mechanism in the attention module, it is necessary to explain the construction of training samples during the training phase.

[0053] 2.2.1. Construction of Training Samples

[0054] Due to the lack of customizable keyword recognition training datasets, this embodiment constructs training samples by sampling from text records corresponding to audio. During sampling, the number of words N fed into the model is used as a hyperparameter. This embodiment samples two consecutive words in each text as positive keyword phrase inputs and uses a zero vector as padding input. To increase robustness to false wake-ups, this embodiment also introduces N-2 negative inputs during the training phase, inspired by contrastive learning and discriminative training. Two sampling strategies are used to form N-2 negative inputs. One is to randomly sample words from other corpora in the same mini-batch as negative inputs; the other is to perform several phoneme substitutions on the phoneme sequence of positive examples to construct a more complex negative input, further improving the model's discriminative ability. These substitutions are based on a pre-defined confusion set for each phoneme. Considering most keyword recognition application scenarios and experience, the maximum length of the sampled phoneme sequence is limited to no more than 20.

[0055] 2.2.2. Limited attention

[0056] The attention module uses a cross-attention mechanism for information fusion. The phoneme sequence of a word is first processed by a Bidirectional Long Short-Term Memory (BiLSTM) module, and the final hidden state is considered as the word embedding. After obtaining the word embeddings W1, W2, ..., W... N W i After ∈R1×h, the keyword information is fused with the acoustic features through a cross-attention mechanism. This embodiment utilizes an attention mechanism, treating audio features as queries and word embeddings as both keys and values. Therefore, the attention score and output At in frame t will be calculated as follows:

[0057]

[0058] Where the dimension d of the key k This is a proportional factor. To guide the distribution of attention scores to be more concentrated, an additional attention loss L is applied to the attention scores of the last attention layer. att L att It can be calculated as:

[0059]

[0060] Where a pt This refers to the score of positive keyword embedding in frame t, a ft This refers to the score of padding embedding in frame t, where N represents the frame corresponding to the positive keyword. Using this method, the loss L... att This is used to guide the model's attention process among positive keywords, filler inputs, and negative inputs. When a frame corresponds to a keyword, attention is restricted to activating on positive items, otherwise on filler items, and never on negative items.

[0061] Since the attention score can be considered as a measure of the joint probability distribution of the acoustic frame and keywords, the embodiments of this application directly use the attention score as an auxiliary filter in the detection to roughly filter out some negative examples that may lead to false positives.

[0062] 2.3. Trigger Module

[0063] The triggering module makes the final decision on whether the expected keyword has been discovered. Previous end-to-end methods optimized the model, directly generating the likelihood score of the keyword without any search procedure, thus solving the problem as a binary classification problem. To retain the advantages of end-to-end methods and better simulate keyword length information and the triggering stage, this application transforms the keyword discovery problem into a multi-label classification problem. Specifically, this application designs a label construction mechanism to model information about the keyword phoneme sequence length and the triggering stage.

[0064] Multiple tags represent the triggering phase of a custom keyword, making the process resemble a gradual triggering process. A tag of 0 means the keyword has not been triggered at all, a tag of 1 means the first phoneme has been triggered, and so on. Only when all phonemes of a keyword have been triggered is the keyword considered triggered and causes an awakening. For a word instance, the phoneme sequence of "keyword" is "k", "iy1", "w", "er2", "d". This embodiment assigns a tag of 0 to any frame in this sequence that is not related to the positive keyword. Then, this embodiment assigns a tag of 1 to the frame corresponding to the first phoneme ("K" in this example). Next, this embodiment assigns tags of 1 and 2 to the frame corresponding to the second phoneme ("IY1" in this example), and so on, until the frame corresponding to each phoneme is tagged. An example of the tag generation process is shown below. Figure 2 middle.

[0065] Figure 4 The multiple tags for the keyword "keyword" are shown.

[0066] This application's embodiments solve the aforementioned multi-label classification problem using multiple binary classifiers, assuming that the distribution of each frame and label is independent. The loss L for the multi-label classification problem... mul It can be calculated as:

[0067]

[0068] Where y i t ∈{0,1} represents the i-th tag in the t-th frame, x i t ∈(0,1) represents the predicted probability of having label i in frame t. Furthermore, the number l... max As mentioned earlier, the maximum length corresponding to the keyword phoneme sequence is set to 20 in this embodiment. The average score of all tags is used as the likelihood score of the keyword in each frame. The advantages of the multi-level mechanism will be discussed further later.

[0069] The attention module and the trigger module are jointly trained using the loss L in equation (4). α is used as a hyperparameter to balance the relative importance of each part.

[0070]

[0071] 3. Experimental Setup

[0072] 3.1. Dataset

[0073] Currently, customizable keyword discovery tasks lack a widely used and recognized standard dataset in the author's knowledge domain. Therefore, this application embodiment establishes a keyword detection version of the LibriSpeech dataset to evaluate the effectiveness of the proposed method. To maintain consistency with real-world applications and eliminate evaluation randomness, the selection of keywords in the test dataset is based on their phoneme sequence length and frequency. Specifically, this application embodiment selects 5 to 50 of the most frequent words from the test-clean and test-other datasets, respectively, with each word having at least 6 phonemes. Notably, keyword matching is based on phoneme sequences; that is, anything prefixed, suffixed, or subworded with a keyword is considered a positive sample. For example, "accompany" will also be considered a positive sample for the keyword "company".

[0074] To determine the wake-up threshold of the keyword recognition system during the testing phase, this embodiment assigns the remaining audio unrelated to any keyword to the false wake-up dataset. The duration of the false wake-up dataset is approximately 3 hours.

[0075] 3.2. Baseline Settings

[0076] The baseline model in this application consists of an Advanced Memory (AM) network and a post-processing algorithm. The frame-level AM consists of a 5-layer Deep Fast Sequential Memory (DFSMN) network, and the post-processing algorithm is a dynamically programmed algorithm that can effectively reduce the false alarm rate during decoding. The hidden size and projected size in the DFSMN are 256 and 128, respectively. The total trainable parameters are approximately 1440K.

[0077] 3.3. End-to-end model setup

[0078] The end-to-end model used in this application embodiment uses Figure 3 The architecture described in the embodiment is as follows. In the method of this application embodiment, the same DFSMN in the baseline model is used as the pre-trained AM. The BiLSTM in the attention module consists of a 3-layer network with a hidden size of 128. This application embodiment applies a 3-layer cross-attention between acoustic representation and textual information with a hidden size of 128 dimensions. The triggering module contains a 2-layer fully connected network. During the training phase, the number of words fed into the system is set to 4, including a positive keyword, a filler word, a random negative word, and a confusing negative word. α in equation (4) is empirically set to 1.1.

[0079] During the testing phase, the text input will consist of a keyword and a filler word. Because the attention score on the keyword item in the keyword frame is quite high under limited attention, only frames with a score higher than 0.8 are considered potential keyword activation frames. The average score of all predicted labels will be used as the confidence score. In other words, the keyword will be triggered when the confidence score of the potential frame is higher than the reawakening threshold. Furthermore, the reawakening threshold for the triggering module will be searched on the aforementioned false positive dataset, with constraints imposed on the number of false positives.

[0080] 3.4. Evaluation Indicators

[0081] In the experiments of this application embodiment, the wake-up precision was essentially 1, and there was no significant difference in performance among different keywords. Therefore, this application embodiment uses micro-average recall as an evaluation metric to evaluate the average performance of selected keywords under strict 2 false positive conditions (less than 1 time per hour).

[0082] Figure 5 The illustration shows a comparison between the end-to-end model recall results of this application embodiment and the recall results of the baseline model with a 2x false positive rate on the test-clean and test-other datasets. Here, recall@K means that K keywords are used in the evaluation, with the following English and Chinese translations: model, baseline model, end-to-end model, recall, test-clean, and test-other.

[0083] 4. Results and Analysis

[0084] 4.1. Results

[0085] First, the recall rates of keywords in the embodiments of this application and the baseline model were compared, and then... Figure 5 The evaluation results for different numbers of keywords are listed. The model in this application significantly improves recall compared to the baseline model in almost all results.

[0086] Meanwhile, this application embodiment compares the average processing time of the baseline model and the end-to-end model of this application embodiment under exactly the same settings. Compared with the baseline model, the end-to-end model of this application embodiment achieves a relative speedup of approximately 44 times. To further investigate the effectiveness of the sampling strategy and the multi-label mechanism introduced in this application embodiment, this application embodiment conducted a series of comparative experiments with 15 keywords.

[0087] 4.2. Impact of Negative Samples

[0088] To investigate the impact of negative samples during training, this application evaluated the model's performance using different negative sample construction strategies. Hybrid sampling refers to using one random sample and one hard negative sample. All_random and All_hard refer to using randomly sampled negative input and manually constructed jumbled negative input, respectively. No_neg refers to using only positive keywords and fillers, without using any negative input.

[0089] Figure 6 The evaluation results for different sampling strategies are shown. The English and Chinese translations are as follows: Sampling strategy, All_random, All_hard, No_neg, Hybridsample, Recall, test-clean, test-other.

[0090] like Figure 6 As shown, mixed samples achieved the best performance on both test sets. On the test-clean and test-other sets, all random samples outperformed no-negative samples, while all hard samples outperformed both all random samples and no-negative samples on the test-other set. The results for All_random and No_neg indicate that random negative samples can effectively improve the model's detection ability in continuous audio. This is likely due to the greater coverage of the text space, which better matches the test scenario of the keyword discovery task. Furthermore, the poor performance on the test-other dataset suggests that the model struggles in complex environments, resulting in limited recall with a fixed number of false alarms. Using additional chaotic negative samples for discrimination training can further improve performance.

[0091] 4.3. Impact of Multi-Label Mechanism

[0092] This section analyzes the impact of the multi-label mechanism in the method of this application's embodiments. This application's embodiments remove the multi-label mechanism and directly use the score of the first label or the last label as the confidence score for the detected keywords. In this case, the method of this application's embodiments degenerates into a binary classification method.

[0093] Figure 7The evaluation results of different labeling methods are shown. The English and Chinese translations are as follows: Label method, Recall, Test-clean, Test-other, First label, Last label, Multi-label.

[0094] exist Figure 7 The results are listed below. It is clear that the multi-label mechanism of this application embodiment can deliver significantly better performance. Using the first label is easily affected by words with similar pronunciations, such as common prefixes and suffixes. These false alarms can greatly degrade performance. Meanwhile, using the last label is similar to methods proposed in related technologies. However, in the experiments of this application embodiment, especially in complex acoustic environments, performance drops sharply, possibly due to a lack of training data specific to the task. The multi-label mechanism of this application embodiment better simulates the triggering process of keywords with different phoneme numbers, leading to a significant performance improvement.

[0095] 5. Conclusion

[0096] This application proposes a fully end-to-end customizable keyword discovery system for streaming media, eliminating the need for a post-processing module through neural search. The end-to-end method uses an attention module to fuse audio representations and textual information, and a triggering module to determine wake-up. This application introduces a novel multi-label mechanism to model length information and the triggering phase. Furthermore, the end-to-end model is decoupled from the acoustic model, allowing it to be transferred or extended to other acoustic models. This application evaluates its end-to-end method on a keyword discovery version of the Librispeech dataset. Experiments show that the model outperforms the baseline in micro-mean recall under low false positive rates and achieves approximately 44x relative speed. Future work will explore how to extend this model to scenarios with multiple keywords and further improve its robustness in complex acoustic environments.

[0097] In other embodiments, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the keyword detection method in any of the above method embodiments for use in a keyword detection system, wherein the keyword detection system includes a pre-trained acoustic model, an attention module, and a triggering module;

[0098] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0099] The pre-trained acoustic model is used to process audio input to obtain frame-level acoustic features;

[0100] The attention module interacts with the text information of the acoustic features and keywords to determine whether each phoneme in the keyword is found in the frame-level acoustic features and outputs the determination result.

[0101] The final wake-up state is determined using the triggering module based on the determination result.

[0102] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the keyword detection system, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely configured relative to the processor, and these remote memories may be connected to the keyword detection system via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-described keyword detection methods.

[0104] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 8 As shown, the device includes one or more processors 810 and a memory 820. Figure 8 Taking a processor 810 as an example, the keyword detection method and system may further include an input device 830 and an output device 840. The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8Taking a bus connection as an example, the memory 820 is the aforementioned non-volatile computer-readable storage medium. The processor 810 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the keyword detection method and system described in the above embodiments. The input device 830 can receive input digital or character information and generate key signal inputs related to user settings and function control of the communication compensation device. The output device 840 may include a display screen or other display device.

[0105] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0106] In one implementation, the above-described electronic device is applied to a keyword detection system, which includes a pre-trained acoustic model, an attention module, and a triggering module, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0107] The pre-trained acoustic model is used to process audio input to obtain frame-level acoustic features;

[0108] The attention module interacts with the text information of the acoustic features and keywords to determine whether each phoneme in the keyword is found in the frame-level acoustic features and outputs the determination result.

[0109] The final wake-up state is determined using the triggering module based on the determination result.

[0110] The electronic devices described in this application exist in various forms, including but not limited to:

[0111] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones, multimedia phones, feature phones, and low-end phones, etc.

[0112] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, etc.

[0113] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0114] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0115] (5) Other electronic devices with data interaction functions.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A keyword detection method for use in a keyword detection system, wherein, The keyword detection system includes a pre-trained acoustic model, an attention module, and a triggering module, and the method includes: The pre-trained acoustic model is used to process audio input to obtain frame-level acoustic features; The attention module interacts with the text information of the acoustic features and keywords to determine whether each phoneme in the keyword is found in the frame-level acoustic features and outputs the determination result. The triggering module determines the final wake-up state based on the determination result; The pre-trained acoustic model uses phonemes as recognition symbols and is trained using frame-aligned labels. After training, all parameters of the pre-trained acoustic model are frozen. The triggering module uses a multi-label construction mechanism to model the length of the phoneme sequence of the keyword and the multi-label of the triggering stage. Each label in the multi-label represents a certain phoneme in the phoneme sequence being triggered. When all phonemes in the phoneme sequence are triggered, the keyword is triggered to cause wake-up. The step of using the attention module to interact with the text information of the acoustic features and keywords to determine whether each phoneme in the keyword is found in the frame-level acoustic features and outputting the determination result includes: processing the phoneme sequence of the keyword using a bidirectional long short-term memory module, using the last hidden state of the bidirectional long short-term memory module as the word embedding of the keyword; determining the word embedding and the frame-level acoustic features through a cross-attention mechanism and outputting the determination result, wherein the determination result includes the attention score of each frame in the frame-level acoustic features and the output of each frame in the frame-level acoustic features.

2. The method according to claim 1, wherein, The attention module and the triggering module are jointly trained using attention loss and multi-label loss, wherein the attention loss is related to the attention score of the last attention layer of the attention module, and the multi-label loss is related to the multi-label.

3. The method according to claim 2, wherein, The attention score includes the attention score for positive word embeddings and / or the attention score for filler embeddings. When the frame corresponds to a keyword, attention is restricted to activation on positive word embeddings.

4. The method according to claim 2, wherein, The multi-label model is performed using multiple binary classifiers.

5. The method according to any one of claims 1-4, wherein, The keyword is a wake-up word, and the keyword detection system is a wake-up word detection system.

6. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 5.

7. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images