An end-to-end speech keyword detection method, device and medium

Abstract

The application discloses an end-to-end speech keyword detection method and device and a medium, wherein the method comprises the following steps: building a network model, extracting acoustic features, and generating sample labels; the model is regularly trained to make the model have the ability to preliminarily classify keywords and non-keywords; auxiliary categories are added according to the evaluation results of a verification set and the decoding results of a training set; the sample labels are updated and the model is continuously trained according to the addition of the auxiliary categories; and it is judged whether the sample contains a predefined keyword according to the decoding of the model output. The application is aimed at the problems of high false alarm rate of a commonly used keyword-filling model in speech keyword detection and influence of low-quality samples in a training set on the training effect, uses a method of dynamically adding auxiliary categories, sets two auxiliary categories, and respectively processes non-keyword samples prone to false alarms and low-quality keyword samples prone to missed detection, so that the utilization efficiency of training samples is effectively improved, and the performance of speech keyword detection is improved.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing, and in particular to an end-to-end speech keyword detection method, device and medium. Background Technology

[0002] Speech keyword detection technology refers to the detection of predefined keywords from a continuous audio stream. It is one of the current research hotspots in the field of speech and is widely used in terminal device control, smart home, voice information retrieval, audio monitoring and other fields.

[0003] However, for speech keyword detection with a fixed vocabulary, both traditional and deep learning methods generally employ a keyword-filling model to model keywords and non-keywords. Specifically, keywords in the keyword list are split into modeling units and classified as keyword categories, while all other sentences are split into modeling units and classified as non-keyword categories. This modeling method results in a smaller overall number of categories, requires less data, and has faster decoding speed, but it also has some drawbacks. Because non-keyword categories contain too much content, some non-keyword features are very similar to keyword features, making it difficult for the model to determine a good decision boundary during training, thus impairing the performance of the keyword detection model.

[0004] Taking tonal syllables as the modeling unit as an example, on the one hand, the keyword-filling model will include some non-keyword syllables in the non-keyword category that are very similar to the acoustic pronunciation of the syllables in the keyword category. This will increase the classification difficulty of the model during the training and inference process, resulting in a higher false alarm rate for the keyword detection model.

[0005] On the other hand, some training sentences containing keywords in the training corpus have low sample quality, resulting in unclear pronunciation of the segments containing the keywords. This leads to consistently high loss values ​​for these samples during training, meaning that the keyword samples are consistently missed. This is detrimental to model training and has a negative impact on the model's classification ability. Summary of the Invention

[0006] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide an end-to-end speech keyword detection method, device and medium based on dynamically adding auxiliary categories.

[0007] The first technical solution adopted in this invention is:

[0008] An end-to-end speech keyword detection method includes the following steps:

[0009] S1. Process the training sample labels and pre-add placeholders for auxiliary categories to add auxiliary categories during subsequent training. Among them, the auxiliary categories are divided into two major categories, which are respectively used to process non-keyword samples prone to false alarms and low-quality keyword samples prone to missed detections.

[0010] S2. Use the tonal syllables as modeling units to construct a speech keyword detection model, and normally train the model for E1 rounds to enable the model to have the basic ability to distinguish between keyword and non-keyword categories.

[0011] S3. Start adding the first type of auxiliary category from the E1 round, adding 1 auxiliary category every E gap rounds until N1 auxiliary categories are added. The method of adding auxiliary categories is as follows: for a tonal syllable belonging to the non-keyword category, its label is changed from the non-keyword category label to the corresponding label of the auxiliary category. N1 is a hyperparameter, which is similar to the number of syllable categories N keyword corresponding to the keyword table.

[0012] S4. Start adding the second type of auxiliary category from the E2 round, where E1 < E2. The method of adding auxiliary categories is as follows: for a low-quality training sample containing a keyword, change the label of the syllable corresponding to its keyword from the keyword category label to the auxiliary category label. Low-quality samples with multiple keyword syllables can belong to the same second type of auxiliary category label.

[0013] S5. Input the test sample into the trained speech keyword detection model to obtain the posterior probability output matrix, use the prefix beam search algorithm for decoding, and determine whether the decoded result contains the predefined keyword.

[0014] Furthermore, in step S1, the original label set C contains N start = N keyword + N nonkeyword + N blank output categories. Among them, N nonkeyword is 1, representing the non-keyword category, and N blank is 1, representing the CTC blank label. The label set C1 for adding the first type of auxiliary category has N1 auxiliary categories, and the label set C2 for adding the second type of auxiliary category has N2 auxiliary categories. After pre-adding the auxiliary categories, the label set becomes C ′ = {C, C1, C2}, with a total of N class = N start + N1 + N2 output categories.

[0015] Furthermore, the original label set C = {C kw1 , C kw2 , …, C kwN , C filler , Cblank}; where C kw1 C kw2 ,…,C kwN For N predefined keyword category labels, C filler For non-keyword category tags, C blank For CTC, leave blank labels;

[0016] After adding auxiliary category placeholders, the label set C ′ ={C,C1,C2}, where C2={C ax2}, For N1 predefined first-class auxiliary categories, C ax2 It is a pre-defined second auxiliary category.

[0017] Furthermore, the speech keyword detection model sequentially includes a convolutional downsampling layer, a backbone layer, and a classification layer, and the features input to the model are the log-Mel spectrum of the audio signal;

[0018] The backbone layer of the model uses a convolution-enhanced Transformer model, or simply Conformer, but other neural network models that support end-to-end training can also be used; the classification layer uses a fully connected neural network.

[0019] The training process calculates the posterior probability output matrix of the model and the CTC loss of the target label sequence:

[0020]

[0021] Where x represents the input feature sequence corresponding to the speech sample, l represents the target label sequence corresponding to the labeled text, and B represents the many-to-one mapping function between the target label sequence l and all feasible paths π; the loss is specifically calculated using a forward-backward algorithm based on dynamic programming.

[0022] Further, step S3 includes:

[0023] Starting from round E1, based on the model's recognition results on the validation set, every E gap In each round, one syllable with the highest number of false alarms that has not been set as an auxiliary category is selected from the syllables contained in the non-keyword category. Before the start of the next round of training, the labels of the selected syllables in the entire training corpus are set to the corresponding labels in the auxiliary category set C1; E gap The settings are based on the total number of training epochs, ensuring that N1 auxiliary classes are added before the model ends training, and that training continues for a certain number of epochs to allow the model to fully converge.

[0024] Furthermore, for weakly supervised speech keyword annotation samples, that is, sample annotations that only contain the text transcription of the corresponding speech sample without frame-level annotations or keyword location information of the speech sample, when a sample generates a false alarm, the similarity between a segment in text A that generated the false alarm and the false alarm keyword text W is calculated using a text pronunciation similarity calculation tool. A sliding window is used to select a segment in text A that is the same length as W, and its similarity is calculated with W. The higher the similarity score, the higher the similarity between the two text segments. The text segment with the highest similarity score, A, is selected. max Consider the text segment A as a false alarm fragment in the sample that generates a false alarm. max The included syllables are used to count the number of false alarms in the validation set for that round.

[0025] Furthermore, the selection method for the first type of auxiliary category is as follows:

[0026] After enabling the addition of the first auxiliary category, when evaluating the model using the validation set, the posterior probability matrix output by the model is decoded, and it is determined whether the decoding result contains predefined keywords. If the target label sequence does not contain keywords but keywords appear in the decoding result, it is determined that a false alarm has occurred.

[0027] In the weakly labeled target sequence, based on the keyword text sequence of the false alarm, the fuzzy sound matching tool library is used to perform similarity matching in the target sequence, find the segment in the target sequence with the highest similarity to the keyword of the false alarm as the segment of the false alarm, and record each syllable contained in the segment as a false alarm.

[0028] Count all syllables that generate false alarms in the current round of the validation set and their false alarm counts, and add the syllable with the highest false alarm count that does not belong to the keyword category and the syllable that has already been added to the auxiliary category to the auxiliary category;

[0029] After the auxiliary categories are added, based on the updated tag set C ′ Regenerate the labels for the training samples and save the label set C. ′ .

[0030] Further, step S4 includes:

[0031] Starting from round E2, the keyword confidence score S output by the keyword samples in the training set during training is statistically analyzed. This score S is used to determine whether the keyword has been learned well by the model. If E2 is consecutively positive, the confidence score S is calculated. cont If the confidence score S of a keyword sample in a certain round is less than the threshold θ, then the keyword sample is regarded as a low-quality sample, and the keyword syllables with low confidence scores in the keyword segment are changed to the corresponding labels in the auxiliary category set C2 before the start of the next round of training.

[0032] Furthermore, the selection method for the second type of auxiliary category is as follows:

[0033] After enabling the addition of a second auxiliary category, the CTC loss is calculated after the forward propagation in the training process, and the confidence score of the keyword is obtained. The sample number and frequency of the keyword whose confidence score is lower than the preset value are also counted.

[0034] If the keyword confidence score of a keyword sample is lower than θ for multiple consecutive rounds... ′ If the keyword sample is not found, it will be considered a low-quality sample and added to the low-quality sample list.

[0035] After this round of training is completed, the labels of all training samples are regenerated. The specified keyword fragments in the target sequence of low-quality samples are added to the second auxiliary category set C2. When generating labels, the labels of the low-quality keyword fragments in the target sequence are changed from the original keyword labels to the second auxiliary category labels.

[0036] Further, step S5 includes:

[0037] In the posterior probability matrix output by the model, the label types include keyword category labels, non-keyword labels, labels corresponding to auxiliary category sets C1 and C2, and CTC blank labels. During decoding, CTC blank labels are first removed, then consecutive identical labels are merged, and keyword category labels are searched for within them. At this time, the labels corresponding to auxiliary categories C1 and C2 are treated as non-keyword labels.

[0038] The second technical solution adopted in this invention is:

[0039] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement an end-to-end speech keyword detection method as described above.

[0040] The third technical solution adopted in this invention is:

[0041] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement an end-to-end speech keyword detection method as described above.

[0042] The fourth technical solution adopted in this invention is:

[0043] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the method described above.

[0044] The beneficial effects of this invention are as follows: This invention improves the keyword-filling model commonly used in speech keyword detection by adding two subcategories to the non-keyword category, namely the first auxiliary category and the second auxiliary category; the first auxiliary category targets non-keyword samples that are prone to false alarms, thus reducing the overall false alarm rate; the second auxiliary category targets keyword samples that are prone to missed detection, thus realizing the utilization of low-quality samples. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a simplified flowchart of the end-to-end speech keyword detection method based on dynamically adding auxiliary categories in an embodiment of the present invention;

[0047] Figure 2 This is a detailed flowchart illustrating the end-to-end speech keyword detection method based on dynamically adding auxiliary categories in an embodiment of the present invention.

[0048] Figure 3 This is a flowchart illustrating the steps of an end-to-end speech keyword detection method according to an embodiment of the present invention. Detailed Implementation

[0049] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0051] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0052] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0053] To address the existing technical problems, this invention dynamically adds auxiliary categories during the model training process. This reduces the acoustic similarity between keyword and non-keyword category samples and the negative impact of low-quality keyword samples on training without increasing the number of output categories excessively. Furthermore, this method is applied to weakly supervised training corpora, automatically identifying difficult samples and setting them as auxiliary categories during training, without requiring manual intervention.

[0054] This invention adds two major categories of auxiliary categories, referred to as the first auxiliary category and the second auxiliary category, respectively. The first auxiliary category is used to process non-keyword samples that are prone to false alarms, thereby reducing the false alarm rate. The second auxiliary category is used to process keyword samples that are prone to being missed in the training set, thereby making use of low-quality samples.

[0055] Example 1

[0056] like Figure 3 As shown, this embodiment provides an end-to-end speech keyword detection method, using basic language units (such as tone syllables) as recognition units, including the following steps:

[0057] S1. Process the training sample labels and pre-add placeholders for auxiliary categories to be added during subsequent training. The auxiliary categories are divided into two main categories, which are used to handle non-keyword samples that are prone to false alarms and low-quality keyword samples that are prone to missed detections.

[0058] For example, in step S1, the initial tag set C contains Nstart =N keyword +N nonkeyword +N blank There are N output categories, where N nonkeyword A value of 1 indicates a non-keyword category, and N represents the category of non-keywords. blank A value of 1 indicates a blank label in CTC; a label set C1 with the first auxiliary category is added, resulting in N1 auxiliary categories; a label set C2 with the second auxiliary category is added, resulting in N2 auxiliary categories; after adding auxiliary categories beforehand, the model output label set becomes C... ′ ={C,C1,C2}, there are N in total class =N start +N1+N2 output categories.

[0059] In some embodiments, step S1 uses a connectionist temporal classifier loss for training. The connectionist temporal classifier loss is abbreviated as CTC loss. The original label set C = {C...} kw1 C kw2 ,…,C kwN C filler C blank}; where C kw1 C kw2 ,…,C kwN For N predefined keyword category labels, C filler For non-keyword category tags, C blank For CTC, leave a blank label.

[0060] After adding auxiliary category placeholders, the label set C ′ ={C,C1,C2}, where C2={C ax2}, For N1 predefined first-class auxiliary categories, C ax2 It is a pre-defined second auxiliary category.

[0061] S2. Using toned syllables as modeling units, construct a speech keyword detection model. Train the model normally for E1 rounds to enable the model to have a basic ability to distinguish between keyword and non-keyword categories.

[0062] In some embodiments, the speech keyword detection model sequentially comprises a convolutional downsampling layer, a backbone layer, and a classification layer, with the input model using the log-Mel spectrum of the audio signal. Specifically, the backbone layer uses a convolutionally enhanced Transformer model, or simply Conformer, but other neural network models that support end-to-end training can also be used; the classification layer uses a fully connected neural network.

[0063] The training process calculates the posterior probability output matrix of the model and the CTC loss of the target label sequence:

[0064]

[0065] Where x represents the input feature sequence corresponding to the speech sample, l represents the target label sequence corresponding to the labeled text, and B represents the many-to-one mapping function between the target label sequence l and all feasible paths π; the loss is specifically calculated using a forward-backward algorithm based on dynamic programming.

[0066] S3. Starting from round E1, add the first auxiliary category, every E... gap One auxiliary category is added per round until N1 auxiliary categories are added. The method for adding auxiliary categories is as follows: for a toned syllable that belongs to a non-keyword category, its label is changed from the non-keyword category label to the corresponding auxiliary category label; N1 is a hyperparameter corresponding to the number of syllable categories in the keyword table. keyword similar.

[0067] In step S3, starting from round E1, based on the model's recognition results on the validation set, every E... gap In each round, one syllable with the highest number of false alarms that has not been set as an auxiliary category is selected from the syllables contained in the non-keyword category. Before the start of the next round of training, the labels of the selected syllables in the entire training corpus are set to the corresponding labels in the auxiliary category set C1; E gap The settings are based on the total number of training epochs, ensuring that N1 auxiliary classes are added before the model ends training, and that training continues for a certain number of epochs to allow the model to fully converge.

[0068] For weakly supervised speech keyword annotation samples, where the annotations only contain the text transcription of the corresponding speech sample and not the frame-level annotations or keyword location information, when a false alarm occurs, a text pronunciation similarity calculation tool (such as Dimsim) is used to calculate the similarity between a segment in text A that generated the false alarm and the false alarm keyword text W. A sliding window is used to select a segment in text A of the same length as W and perform a similarity calculation with W. The higher the similarity score, the higher the similarity between the two text segments. The text segment with the highest similarity score, A, is selected. max Consider A as a false alarm segment in the sample that generates a false alarm. max The included syllables are used to count the number of false alarms in the validation set for that round.

[0069] In some embodiments, the opening round E1 and the number of auxiliary categories N1 in step S3 are set as hyperparameters. The setting of E1 cannot be too small or too large. If the time point for starting to add the first type of auxiliary category is too early, the model does not yet have basic discrimination ability, and the selected difficult non-keyword syllable categories may not have obvious acoustic confusion at this time. If the time point for starting to add the first type of auxiliary category is too late, it will slow down the model's convergence speed and lead to a longer training time.

[0070] As an implementation manner, the specific process of selecting the first type of auxiliary category in step S3 includes:

[0071] S3.1. After starting to add the first type of auxiliary category, when using the validation set to evaluate the model, decode the posterior probability matrix output by the model and determine whether the decoded result contains a predefined keyword. If the target label sequence does not contain the keyword but the keyword appears in the decoded result, it is determined that a false alarm has occurred.

[0072] S3.2. In the weakly labeled target sequence, there is no position information of the keyword, but it is necessary to determine the position where the false alarm occurs in the input speech. The present invention adopts a fuzzy matching method to search for the false alarm segment of the target sequence. The specific method is to perform similarity matching in the target sequence using the fuzzy sound matching tool library Dimsim according to the keyword text sequence of the false alarm, find the segment with the highest similarity to the false alarm keyword in the target sequence as the false alarm segment, and record each syllable included in this segment as a false alarm once.

[0073] S3.3. According to the above method, count all the syllables that generate false alarms and their false alarm times on the validation set of the current round, and add the syllable with the highest false alarm times that does not belong to the keyword category and has not been added to the auxiliary category to the auxiliary category.

[0074] S3.4. After the addition of the auxiliary category is completed, according to the updated label set C ′ Regenerate the labels of the training samples and save the label set C ′ .

[0075] S4. Start adding the second type of auxiliary category from the E2 round, where E1 < E2; the method of adding the auxiliary category is: for a low-quality training sample containing a keyword, modify the label of the syllable corresponding to the keyword to the auxiliary category label. Multiple low-quality samples of keyword syllables can belong to the same second type of auxiliary category label.

[0076] In step S4, starting from the E2 round, count the keyword confidence score S output by the keyword samples in the training set during the training process, and use this score to judge whether the keyword is well learned by the model. If it is continuous for E contIf the confidence score S of a keyword sample in a certain round is less than the threshold θ, then the keyword sample is regarded as a low-quality sample, and the keyword syllables with low confidence scores in the keyword segment are changed to the corresponding labels in the auxiliary category set C2 before the start of the next round of training.

[0077] In some embodiments, the start time E2 in step S4 is set as a hyperparameter because the setting of E2 should also be moderate. If it is started too early, the model does not yet have basic discrimination ability, and at this time the confidence score of the keyword has little reference value for whether the sample is of low quality. If it is started too late, it will slow down the convergence speed of the model and result in a longer training time.

[0078] As an optional implementation, the specific process for selecting the second type of auxiliary category in step S4 includes:

[0079] S4.1 After enabling the addition of a second auxiliary category, during the forward propagation of the training process, while calculating the CTC loss, decode and obtain the confidence score of the keyword, and count the sample number and frequency of keywords with a confidence score lower than 0.1; the formula for calculating the confidence score of the keyword is:

[0080]

[0081] Where k′ (i = 1, 2, ... n) represents the characters contained in the keyword kw, which has a total of n characters; Indicates the output category k i The probability corresponding to the t-th frame.

[0082] Since the defined keyword lengths may vary, the confidence scores need to be normalized based on the number of characters contained in the keywords:

[0083]

[0084] Among them, Score′ kw The final value of S mentioned earlier is used as the keyword confidence score in this invention.

[0085] Because the Softmax function is used to calculate keyword scores, the CTC peak score is affected by the number of categories. With more categories, the absolute value of the peak decreases, and with fewer categories, the absolute value of the peak increases. Considering this factor, the actual threshold used is modified based on the threshold θ combined with the number of categories.

[0086] θ′=θlnN class

[0087] θ′ is the threshold ultimately adopted in this invention, where N classThis represents the total number of categories output.

[0088] S4.2 If the keyword confidence score of a keyword sample is lower than θ′ for 5 consecutive rounds, then the keyword sample is considered a low-quality sample and is added to the low-quality sample list.

[0089] S4.3 After this round of training is completed, the labels of all training samples are regenerated. The specified keyword fragments in the target sequence of low-quality samples are added to the second auxiliary category set C2. When generating labels, the labels of the low-quality keyword fragments in the target sequence are changed from the original keyword labels to the second auxiliary category labels.

[0090] S5. Input the test sample into the trained speech keyword detection model to obtain the posterior probability output matrix. Use the prefix beam search algorithm to decode the sample and determine whether the decoding result contains predefined keywords.

[0091] In step S5, the posterior probability matrix output by the model includes keyword category labels, non-keyword labels, labels corresponding to auxiliary category sets C1 and C2, and CTC blank labels. During decoding, CTC blank labels are first removed, then consecutive identical labels are merged, and keyword category labels are searched for within them. At this time, the labels corresponding to auxiliary categories C1 and C2 are treated as non-keyword labels.

[0092] During inference, the model output decoding graph with auxiliary categories has more categories than when no auxiliary categories are added. These auxiliary categories are treated as non-keyword categories during decoding, so they do not affect keyword search.

[0093] The following is in conjunction with the appendix Figure 1-2 The above method will be further explained with reference to specific embodiments.

[0094] See Figure 1 and Figure 2 This embodiment provides an end-to-end speech keyword detection method based on dynamically adding auxiliary categories, including the following steps:

[0095] Step 1: Preparations before training: Build the network model, extract acoustic features from the dataset, and generate training sample labels;

[0096] In this embodiment, the convolutional downsampling layer uses a 2-layer convolutional neural network with a kernel size of [3,3], performing 4x time downsampling to reduce computational complexity; the backbone layer uses a 6-layer Conformer neural network with an input dimension of 128, 4 attention heads, a feedforward layer dimension of 512, and a kernel size of 31; the classification layer uses a 2-layer fully connected neural network, ultimately resulting in N... class =N start+N1+N2 output categories.

[0097] In this embodiment, the training set, validation set, and test set are all from the AISHELL-2 Chinese Mandarin speech dataset. Eight keywords were selected: "company", "Internet", "journalist", "country", "Beijing", "Shanghai", "city", and "capital". The training set is 72.43 hours long, the validation set is 7.78 hours long, and the test set is 12.98 hours long. The ratio of positive samples to negative samples is 1:3, where positive samples are those containing the keywords and negative samples are those not containing the keywords. 80-dimensional log-Mel spectrum features were extracted from the speech dataset.

[0098] In this embodiment, the keyword modeling unit uses toned syllables.

[0099] Step 2: Train the model a certain number of times to enable it to have a basic ability to distinguish between keyword and non-keyword categories.

[0100] In this embodiment, the optimizer used during training is AdamW, with a maximum of 80 training epochs. The sample size of each mini-batch is 64. The learning rate increases linearly from 0 to 0.001 in the first 10 epochs, and then decays cosinely to 0 in the 80th epoch. Before adding auxiliary categories, the number of output categories is 1.

[0101] Step 3: After the model has a certain discrimination ability, start adding the first auxiliary category. Based on the model's classification results on the validation set, count the syllable categories with the most false alarms on the current round of validation set, obtain the list of false alarms in the validation set for this round, and add the syllables with the most false alarms that do not belong to keywords and have already been added as auxiliary categories to the auxiliary categories.

[0102] In this embodiment, the model adds the first auxiliary category starting from the 10th round, with a total of 10 auxiliary categories set, and one auxiliary category is added every one round.

[0103] In this embodiment, when the model detects a false alarm on the validation set, it is necessary to identify which segment of the non-keyword sample fragment was falsely identified as a keyword. However, since the samples lack frame-level annotation information, it is impossible to determine which word in the target sequence was falsely identified as a keyword based on the frame-level labels output by the model. This embodiment employs a fuzzy matching algorithm to identify the corresponding word in the target sequence that was falsely identified as a keyword: the fuzzy phonetic Chinese matching tool Dimsim is used. This tool can calculate the acoustic similarity between two given Chinese words. Based on the length of the keyword that caused the false alarm, Chinese words are extracted one by one from the target sequence using a sliding window of the same length, and the acoustic similarity with the false alarm keyword is calculated. Finally, the Chinese word fragment with the highest similarity is identified as the syllable that caused the false alarm.

[0104] For example, if the non-keyword target sequence is "This background is really beautiful", but the keyword "Beijing" is detected, a false alarm is generated. According to the acoustic similarity calculation, it is found that the similarity between "background" in the target sequence and the keyword "Beijing" is the highest. Therefore, the false alarm is identified as the keyword "Beijing" in the target sequence "background", and the syllables "bei4" and "jing3" corresponding to "background" are recorded as one false alarm each.

[0105] In this embodiment, after adding the syllable corresponding to the first round of auxiliary category in the current round, the sample labels are regenerated before the start of the next round of training, and the label of the corresponding syllable is changed from the non-keyword category to the corresponding auxiliary category.

[0106] Step 4: After the model has a certain discriminative ability, start adding a second auxiliary category. Based on the keyword confidence score obtained by the model's forward propagation on the training set, if the keyword confidence score of a certain keyword sample is low for multiple consecutive rounds, then the keyword fragment label in that sample is modified to the auxiliary category label.

[0107] In this embodiment, the model adds samples to the second auxiliary category starting from the 50th round. If the confidence score of the keywords of a positive sample is less than 0.1 for 5 consecutive rounds, it is considered a low-quality sample and the sample is added to the low-quality sample list. Before the next round of training begins, the sample labels are regenerated and the labels corresponding to the keyword fragments in the low-quality samples are changed from the keyword category to the second auxiliary category.

[0108] Step 5: After training is complete, input the test samples into the speech keyword detection model to obtain the posterior probability output matrix. Use the prefix beam search algorithm for decoding, and treat the auxiliary categories encountered in the decoding process as non-keyword categories.

[0109] In this embodiment, when the model shows no performance improvement on the validation set for 10 consecutive rounds, it is considered that the training has converged and training is stopped.

[0110] This invention proposes two auxiliary categories, focusing on training non-keyword samples that are prone to false alarms and keyword samples that are prone to missed detection, thereby improving the efficiency of training sample utilization and enhancing the performance of speech keyword detection.

[0111] Example 2

[0112] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 3 This illustrates an end-to-end speech keyword detection method.

[0113] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0114] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0115] Since this electronic device is the electronic device corresponding to the end-to-end voice keyword detection method in the embodiment of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0116] Example 3

[0117] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 3 This illustrates an end-to-end speech keyword detection method.

[0118] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0119] Since this storage medium is the storage medium corresponding to an end-to-end speech keyword detection method in an embodiment of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0120] Example 4

[0121] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of an end-to-end speech keyword detection method according to various exemplary embodiments of this application as described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0122] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0124] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An end-to-end speech keyword detection method, characterized in that, Includes the following steps: S1. Process the training sample labels and pre-add placeholders for auxiliary categories to be added during subsequent training. The auxiliary categories are divided into two main categories, which are used to handle non-keyword samples that are prone to false alarms and low-quality keyword samples that are prone to missed detection. S2. Construct a speech keyword detection model and train the model normally. The rounds enable the model to have a basic ability to distinguish between keyword and non-keyword categories; S3, from The first auxiliary category is added at the start of each round; the method for adding auxiliary categories is as follows: for a toned syllable that belongs to a non-keyword category, its label is changed from the non-keyword category label to the corresponding auxiliary category label; S4, from A second auxiliary category is added starting from the next round, in which The method for adding auxiliary categories is as follows: for a low-quality training sample containing keywords, change the label of the syllable corresponding to the keywords from the keyword category label to the auxiliary category label. S5. Input the test sample into the trained speech keyword detection model to obtain the posterior probability output matrix, decode it, and determine whether the decoding result contains predefined keywords.

2. The end-to-end speech keyword detection method according to claim 1, characterized in that, Original tag set ;in, for A predefined keyword category tag, Non-keyword category tags, For CTC, leave blank labels; After adding auxiliary category placeholders, the tag set ,in , , for A pre-defined first auxiliary category It is a pre-defined second auxiliary category.

3. The end-to-end speech keyword detection method according to claim 1, characterized in that, Step S3 includes: From the At the start of each round, based on the model's recognition results on the validation set, each... In each round, the syllable with the highest number of false alarms that is not set as an auxiliary category is selected from the syllables contained in the non-keyword category. Before the start of the next round of training, the labels of the selected syllables in the entire training corpus are set as the auxiliary category set. Corresponding labels; ensure that the model is labeled before training ends. Add one auxiliary category and continue training for a certain number of rounds to allow the model to fully converge.

4. The end-to-end speech keyword detection method according to claim 3, characterized in that, For weakly supervised speech keyword annotation samples, where the annotations only contain the text transcription of the corresponding speech sample and not the frame-level annotations or keyword location information, when a sample generates a false alarm, a text pronunciation similarity calculation tool is used to calculate the text of the false alarm sample. The fragments and the keyword text of the false alarm Similarity between them, using a sliding window to select text In and Equal length segments, and with Similarity calculations are performed; a higher similarity score indicates a higher similarity between the two text segments. The text segment with the highest similarity score is selected. The text fragment is considered as a false alarm segment in the sample that generates a false alarm. The included syllables are used to count the number of false alarms in the validation set for that round.

5. The end-to-end speech keyword detection method according to claim 1, characterized in that, The selection method for the first type of auxiliary category is as follows: After enabling the addition of the first auxiliary category, when evaluating the model using the validation set, the posterior probability matrix output by the model is decoded, and it is determined whether the decoding result contains predefined keywords. If the target label sequence does not contain keywords but keywords appear in the decoding result, it is determined that a false alarm has occurred. In the weakly labeled target sequence, based on the keyword text sequence of the false alarm, the fuzzy sound matching tool library is used to perform similarity matching in the target sequence, find the segment in the target sequence with the highest similarity to the keyword of the false alarm as the segment of the false alarm, and record each syllable contained in the segment as a false alarm. Count all syllables that generated false alarms in the current round of the validation set and their false alarm counts, and then group those with the highest false alarm counts that do not belong to the keyword category and those already added to the auxiliary category. Each syllable is added to an auxiliary category; After the auxiliary categories are added, based on the updated tag set Regenerate the labels for the training samples and save the label set. .

6. The end-to-end speech keyword detection method according to claim 1, characterized in that, Step S4 includes: From the At the start of each round, the keyword confidence scores of the keyword samples in the training set are calculated during the training process. And through that score To determine whether the keyword has been learned well by the model, if it is continuous Keyword confidence score of a keyword sample in a round All less than the threshold If the keyword sample is deemed low-quality, then the keyword syllables with lower confidence scores in that sample will be replaced with auxiliary category sets before the next training round begins. The corresponding tags in the text.

7. The end-to-end speech keyword detection method according to claim 1, characterized in that, The selection method for the second type of auxiliary category is as follows: After enabling the addition of a second auxiliary category, the CTC loss is calculated after the forward propagation in the training process, and the confidence score of the keyword is obtained. The sample number and frequency of the keyword whose confidence score is lower than the preset value are also counted. If the keyword confidence score of a keyword sample is lower than 1 for multiple consecutive rounds... If the keyword sample is not found, it will be considered a low-quality sample and added to the low-quality sample list. After this round of training, the labels for all training samples are regenerated, and the specified keyword fragments in the target sequences of low-quality samples are added to the second auxiliary category set. In the process of generating labels, the labels for the low-quality keyword fragments of the target sequence are changed from the original keyword labels to second-class auxiliary category labels.

8. The end-to-end speech keyword detection method according to claim 1, characterized in that, Step S5 includes: The posterior probability matrix output by the model includes label types such as keyword category labels, non-keyword labels, and auxiliary category sets. and Corresponding tags and CTC blank tags; during decoding, first remove CTC blank tags, then merge consecutive identical tags, and search for keyword category tags within them, at this point, auxiliary categories. and All corresponding tags are treated as non-keyword tags.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 8.

Images