A keyword detection method and a training method of a keyword detection model

Abstract

The embodiment of the application is suitable for the technical field of computer software, and provides a keyword detection method and a training method of a keyword detection model, the method comprising: obtaining a sound spectrum feature of a to-be-detected voice; dividing a plurality of first time windows which are at least partially overlapped in a time dimension of the sound spectrum feature; respectively generating short-time phoneme features corresponding to each of the first time windows according to the sound spectrum features in each of the first time windows; generating a global feature sequence according to the short-time phoneme features corresponding to each of the first time windows; and generating a keyword detection result according to the global feature sequence. The embodiment of the application can improve the stability and accuracy of the keyword stream detection of the voice.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a keyword detection method and a training method for a keyword detection model. Background Technology

[0002] Keyword Spotting (KWS) technology, as a core component of intelligent voice interaction systems, aims to detect specific trigger words in real time from a continuous speech stream. KWS is typically based on a deep learning model, using a combination of speech recognition and Voice Activity Detection (VAD) algorithms to decode the current recognition result and then match it with target keywords to achieve the wake-up function. Alternatively, it can achieve keyword wake-up through acoustic feature extraction and classification.

[0003] Keyword streaming detection refers to techniques for detecting specific keywords in continuous, real-time audio streams with low latency and low power consumption. Most existing streaming methods employ simple recurrent neural networks or convolutional networks for temporal modeling, but these models struggle to establish long-range contextual dependencies, leading to performance degradation when processing long speech sequences.

[0004] There is an urgent need for a method that can accurately detect keywords in speech. Summary of the Invention

[0005] In view of this, embodiments of this application provide a keyword detection method and a keyword detection model training method to improve the stability, efficiency and accuracy of streaming key detection of speech signals.

[0006] The first aspect of this application provides a keyword detection method, including: Obtain the spectral features of the speech to be detected; Divide the acoustic spectrum features into multiple at least partially overlapping first time windows along the time dimension; Based on the spectral features within each of the first time windows, short-time phoneme features corresponding to each of the first time windows are generated respectively; A global feature sequence is generated based on the short-time phoneme features corresponding to each of the first time windows; Based on the global feature sequence, keyword detection results are generated.

[0007] In some implementations of the first aspect, obtaining the spectral features of the speech to be detected includes: Obtain the spectrogram of the speech to be detected; A multi-level downsampling convolutional neural network is used to sequentially compress the feature dimensions of the spectrogram to obtain the spectrogram features corresponding to the spectrogram. The multi-level downsampling convolutional neural network is composed of multiple convolutional neural networks with a stride greater than 1.

[0008] In some implementations of the first aspect, the first time window contains multiple frames; the step of generating short-time phoneme features corresponding to each of the first time windows based on the spectral features within each first time window includes: The inter-frame dependencies of each frame contained in each first time window are determined. The inter-frame dependencies are obtained by training a first neural network model. The input layer of the first neural network model is connected to multiple frames in the same first time window. Based on the inter-frame dependency relationship and the spectral features corresponding to each frame, short-time phoneme features corresponding to each of the first time windows are generated.

[0009] In some implementations of the first aspect, generating a global feature sequence based on the short-time phoneme features corresponding to each of the first time windows includes: The contextual dependencies between the first time windows are determined, and the contextual dependencies are obtained by training a second neural network model, wherein the input layer of the second neural network model is connected to two or more first time windows; A global feature sequence is generated based on the contextual dependencies and the short-time phoneme features corresponding to each of the first time windows.

[0010] In some implementations of the first aspect, generating keyword detection results based on the global feature sequence includes: The global feature sequence is processed using a trained classifier to output classification results corresponding to each of the first time windows; the classification results include classification labels. Based on the classification results corresponding to each of the first time windows, a classification score corresponding to each of the first time windows is generated; the classification score is used to characterize the confidence level of the classification label of the first time window. Based on the classification score corresponding to each of the first time windows, the keyword detection result of the speech to be detected is determined.

[0011] In some implementations of the first aspect, dividing the acoustic spectrum features into multiple at least partially overlapping first time windows includes: Identify the keywords to be detected, and determine the time length based on the keywords; The acoustic spectrum feature is divided into multiple first time windows according to the time length.

[0012] A second aspect of this application provides a method for training a keyword detection model, including: Extract sample speech from a sample dataset; the sample dataset includes multiple positive samples and multiple negative samples; The sample speech is subjected to speech rate transformation to obtain speed-adjusted speech; Determine the sample labels for the speed-adjusted speech; Add noise information and / or room impulse response information to the speed-adjusted speech to generate training speech; Based on the training speech and the sample labels, training samples are generated; The keyword detection model is obtained by training the training samples, and the keyword detection model is used to implement the keyword detection method as described in the first aspect.

[0013] In some implementations of the second aspect, determining the sample labels of the speed-adjusted speech includes: Divide the time dimension of the speed-adjusting speech into multiple at least partially overlapping second time windows; Determine the energy characteristics of each of the second time windows; If the energy characteristics meet the preset flat-top detection conditions, a first detection label corresponding to the second time window is determined; If the energy characteristics do not meet the preset flat-top detection conditions, a second detection label corresponding to the second time window is determined; Sample labels are generated based on the first detection label and / or the second detection label corresponding to each of the second time windows.

[0014] A third aspect of this application provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the keyword detection method as described in the first aspect above.

[0015] A fourth aspect of this application provides a computer program product, including a computer program that, when run, causes the keyword detection method described in the first aspect above to be executed.

[0016] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the keyword detection method described in the first aspect above.

[0017] In this embodiment, the spectrogram features of the speech to be detected are obtained; multiple at least partially overlapping first time windows are divided along the time dimension of the spectrogram features; short-time phoneme features corresponding to each first time window are generated based on the spectrogram features within each first time window; a global feature sequence is generated based on the short-time phoneme features corresponding to each first time window; and a keyword detection result is generated based on the global feature sequence. The spectrogram features are segmented using partially overlapping first time windows based on a sliding window technique, and the spectrogram features within each first time window are processed to obtain short-time phoneme features. By combining the short-time phoneme features corresponding to different times, a global feature sequence incorporating contextual information can be obtained. This enables the determination of whether keywords exist in the speech to be detected and the generation of keyword detection results, thereby improving the stability and accuracy of streaming keyword detection in speech. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of a keyword detection method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a keyword detection model provided in an embodiment of this application; Figure 3 This is a schematic diagram of a training method for a keyword detection model provided in an embodiment of this application; Figure 4 This is a schematic diagram of a training sample generation process provided in an embodiment of this application; Figure 5 This is a schematic diagram of a keyword detection device provided in an embodiment of this application; Figure 6 This is a schematic diagram of a training device for a keyword detection model provided in an embodiment of this application; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0022] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0024] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0026] Most existing streaming processing methods employ simple recurrent neural networks or convolutional networks for temporal modeling. These models struggle to establish long-range contextual dependencies, leading to performance degradation when processing long speech sequences. This application addresses this by establishing and training a keyword detection model. This model analyzes speech features segment by segment using a sliding window technique and establishes both intra-window and inter-window feature interaction mechanisms. This allows for the acquisition and utilization of global contextual information in the speech, improving the stability, efficiency, and accuracy of keyword detection. For example, it prevents performance degradation when processing long speech sequences. Specific embodiments of this application illustrate the technical solution.

[0027] Reference Figure 1 The diagram illustrates a keyword detection method provided in an embodiment of this application, which may specifically include the following steps: Step 101: Obtain the spectral features of the speech to be detected.

[0028] When the speech to be detected is received, it can be transformed (e.g., STFT (Short-Time Fourier transform)) to obtain spectrogram features, which include information in two dimensions: time frame and frequency.

[0029] In some implementations, the speech to be detected is first processed to obtain a corresponding spectrogram. The keyword detection model receives the spectrogram and extracts spectrogram features from it. In other implementations, the keyword detection model receives externally generated spectrogram features.

[0030] For example, after receiving the speech to be detected, the speech can be converted into spectrogram features Y, then the spectrogram features Y can be moduloed to obtain amplitude spectrum features |Y|, and then the amplitude spectrum features |Y| can be compressed to obtain spectrogram Z (for example, the amplitude spectrum can be compressed using power law to obtain spectrogram Z=). (The exponent m can be determined based on the actual scenario), which improves the detail of the spectrogram and compresses the data space, enabling the spectrogram to be accurately processed by a lightweight model. Then, spectrogram features are extracted from the spectrogram Z.

[0031] Step 102: Divide the acoustic spectrum features into multiple first time windows that at least partially overlap in the time dimension.

[0032] The spectrogram features include a time dimension. Based on a preset sliding window strategy, multiple first time windows of the same length can be divided into the time dimension of the spectrogram features, thereby dividing the spectrogram features into blocks, and keyword detection can be performed based on the spectrogram features contained in each block.

[0033] Since the speech corresponding to the keyword in the to-be-detected speech is a continuous speech segment, similarly, the features corresponding to the keyword in the spectrogram features are continuous on the time axis. In order to achieve more accurate feature detection in the time dimension and ensure the interaction effect between windows, it is necessary to divide the first time windows in a partially overlapping manner, so that adjacent first time windows partially overlap.

[0034] Step 103: Generate short-time phoneme features corresponding to each of the first time windows respectively according to the spectrum features in each first time window.

[0035] A phoneme is a speech unit that can be used to distinguish word meanings. Normally, in order to improve the accuracy of triggering a wake-up event by a keyword, the keyword usually includes speech of multiple characters, that is, the speech of the keyword contains a phoneme combination. Therefore, the detection of the keyword requires accurate detection of different phonemes.

[0036] For example, the keyword includes several characters, including "hello", and the corresponding phoneme combination is / n / – / i / –(tone)– / h / – / a / – / u / .

[0037] Some phonemes correspond to relatively short time durations. Therefore, in the embodiments of the present application, the spectrum features are processed by setting the first time windows with appropriate time lengths, which can effectively extract the short-time phoneme features. Through the short-time phoneme features, phonemes with relatively short durations can be effectively detected, and the transition dynamics between phonemes can also be detected. The short-time phoneme feature is the acoustic representation of the phoneme within the first time window.

[0038] Step 104: Generate a global feature sequence according to the short-time phoneme features corresponding to each of the first time windows.

[0039] A plurality of short-time phoneme features can form an initial feature sequence. Since the time points corresponding to the first time windows are sorted in sequence, the sequential relationship between the respective short-time phoneme features can be determined. In the scenario of keyword streaming detection, each short-time phoneme feature can be processed according to time sorting. For the current short-time phoneme feature, context information can be captured by combining the short-time phoneme features of other first time windows in the same spectrogram, so as to adjust the current short-time phoneme feature to obtain a target short-time phoneme feature. A global feature sequence containing global context information is obtained from each target short-time phoneme feature.

[0040] Step 105: Generate a keyword detection result according to the global feature sequence.

[0041] Based on the global feature sequence, changes in short-term phoneme features and contextual information obtained from multiple short-term phoneme features can be determined. This contextual information can then be combined to determine whether any features in the global feature sequence correspond to keywords. In other words, the presence of keywords in the detected speech can be determined using the global feature sequence, generating keyword detection results. The keyword detection results indicate whether keywords are present in the detected speech or not.

[0042] In this embodiment, the spectrogram features of the speech to be detected are obtained; multiple at least partially overlapping first time windows are divided along the time dimension of the spectrogram features; short-time phoneme features corresponding to each first time window are generated based on the spectrogram features within each first time window; a global feature sequence is generated based on the short-time phoneme features corresponding to each first time window; and a keyword detection result is generated based on the global feature sequence. The spectrogram features are segmented using partially overlapping first time windows based on a sliding window technique, and the spectrogram features within each first time window are processed to obtain short-time phoneme features. By combining the short-time phoneme features corresponding to different times, a global feature sequence incorporating contextual information can be obtained. This enables the determination of whether keywords exist in the speech to be detected and the generation of keyword detection results, thereby improving the stability and accuracy of streaming keyword detection in speech.

[0043] The embodiments of this application can achieve low-latency streaming processing by controlling the hardware data flow, with each first window processing latency being less than 200ms.

[0044] For keyword wake-up scenarios, the following processing can be performed after step 105: If the keyword detection result shows the existence of a keyword corresponding to the wake-up function, a wake-up trigger event corresponding to the keyword is generated. Then, based on the application scenario, device, and other factors of this embodiment, the response strategy for the wake-up trigger event is determined. For example, this embodiment is applied to smart home devices, and the smart home devices can respond to the wake-up trigger event by outputting a preset audio.

[0045] For other functional scenarios, corresponding function trigger events can be generated, and corresponding function operations can be performed in response to the function trigger events.

[0046] In some implementations of this application, step 101 includes: obtaining the spectrogram of the speech to be detected; using a multi-level downsampling convolutional neural network to sequentially compress the feature dimensions of the spectrogram to obtain the spectrogram features corresponding to the spectrogram, wherein the multi-level downsampling convolutional neural network is composed of multiple convolutional neural networks with a convolution stride greater than 1.

[0047] Reference Figure 2 The diagram shows a structural schematic of a keyword detection model provided in this application. Embodiments of this application can be derived from... Figure 2 The keyword detection model shown is implemented. This keyword detection model consists of multiple convolutional neural networks with a stride greater than 1.

[0048] The keyword detection model can receive externally sent spectrograms (e.g., the compressed spectrogram Z mentioned above), and then use a multi-level downsampling convolutional neural network to extract multi-scale features from the spectrograms to obtain spectrogram features. Each level of the convolutional neural network includes convolutional layers, batch normalization layers, and activation functions (such as SELU), progressively compressing the dimensionality of the spectrogram features to capture acoustic patterns from local to global.

[0049] In each episode, the stride of the convolutional neural network is greater than 1, thereby reducing the feature dimension of the spectral features in sequence, thus adapting to the feature extraction needs of keywords of different durations, that is, it can adapt to the first time window of different price durations.

[0050] As an example, in a multi-level downsampling convolutional neural network, the stride of each convolutional neural network is 2, and the spectrogram feature size is reduced successively by 1 / 2, 1 / 4, 1 / 8, and 1 / 16. In a specific implementation, the stride of each convolutional neural network can be set according to actual needs.

[0051] In some implementations of this application, step 103 includes: determining the inter-frame dependency relationship of each frame contained in each first time window, wherein the inter-frame dependency relationship is obtained by training a first neural network model, and the input layer of the first neural network model is connected to multiple frames in the same first time window; and generating short-time phoneme features corresponding to each first time window based on the inter-frame dependency relationship and the spectral features corresponding to each frame.

[0052] The keyword detection model includes a first neural network model. The input layer of the first neural network model is connected to multiple frames within the same first time window. After training, the trained first neural network model can determine the inter-frame dependencies between different frames within the same first time window. Based on these dependencies, short-time phoneme features are generated, thereby fusing the spectrogram features corresponding to different frames within the same time window and generating enhanced short-time phoneme features within the first time window. This achieves an intra-window inter-frame interaction mechanism, accurately extracting short-time phoneme features within the window.

[0053] The embodiments of this application do not limit the structure of the first neural network model, only that it can determine the inter-frame dependency relationship within the first time window. For example, the first neural network model can realize the state transfer between different frames within the same first time window based on the gated recurrent unit (GRU) or attention mechanism (setting an attention layer), thereby determining the inter-frame dependency relationship within the first time window.

[0054] For the first neural network model, the same gated recurrent unit parameters or the same attention parameters can be used to process each frame, thereby making the first neural network model lightweight.

[0055] In some implementations of this application, step 104 includes: determining the contextual dependencies between first time windows, wherein the contextual dependencies are obtained by training a second neural network model, and the input layer of the second neural network model is connected to two or more first time windows; and generating a global feature sequence based on the contextual dependencies and the short-time phoneme features corresponding to each first time window.

[0056] The keyword detection model includes a first neural network model and a second neural network model whose input layer is connected to two or more first time windows. After training the second neural network model, the trained model can determine the contextual dependencies between different first time windows. This contextual dependency allows for the fusion of short-term phoneme features corresponding to different first time windows and the generation of a global feature sequence. This achieves an inter-window interaction mechanism, accurately extracting contextual information corresponding to multiple first time windows and fusing this contextual information into the global feature sequence.

[0057] The embodiments of this application do not limit the structure of the second neural network model, as long as it can determine the contextual dependencies between the first time windows. For example, the second neural network model can realize the state transfer between different first time windows based on a bidirectional gated recurrent unit (GRU) or an attention mechanism (setting an attention layer), thereby determining the contextual dependencies between the first time windows.

[0058] For the second neural network model, the same gated recurrent unit parameters or the same attention parameters can be used to process the short-term phoneme features of each first time window, thereby making the second neural network model lightweight. In this application embodiment, through the above-mentioned intra-window inter-frame interaction and inter-window interaction, local features and global features are effectively fused to obtain a global feature sequence, improving the accuracy of keyword detection in long speech.

[0059] In some implementations of this application, step 105 includes: processing the global feature sequence using a trained classifier and outputting classification results corresponding to each first time window; the classification results include classification labels; generating classification scores corresponding to each first time window based on the classification results corresponding to each first time window; the classification scores are used to characterize the confidence level of the classification labels of the first time window; and determining the keyword detection results of the speech to be detected based on the classification scores corresponding to each first time window.

[0060] like Figure 2 As shown, the global feature sequence can be input into the trained classifier, which consists of a fully connected layer and a SUM function. The classifier outputs the classification probability for each first time window and determines the classification label by combining the preset classification threshold and the magnitude of the classification probability. The classification label of a single first time window can include either a keyword or a non-keyword.

[0061] For example: if the classification probability of a certain first time window is 85%, which is greater than the classification threshold of 0.8, then the classification label is determined to be a keyword; if the classification probability of a certain first time window is 75%, which is less than the classification threshold of 0.8, then the classification label is determined to be a non-keyword.

[0062] The dimensions of the global feature sequence include frequency, time, and channel (one channel corresponds to one first time window). The classifier determines the classification label corresponding to each channel dimension. The output (classification result) of the classifier then includes frequency, time, and class dimensions (corresponding to the classification label).

[0063] To simplify the output dimensions and improve decision-making efficiency, further processing of the classification results is needed in the frequency dimension. The frequency dimension processing component in the keyword detection model can sum or pool the frequency dimension information of the classification results to obtain a classification score, which represents the confidence level of the classification label in the first time window. The keyword detection result is then determined based on the confidence level. For example, if there are several consecutive confidence levels greater than the confidence threshold and the classification label is a keyword, the keyword detection result is determined to be that a keyword exists.

[0064] The classifier in this embodiment is a lightweight classifier. Each first time window shares the classifier parameters. The classification results of the first time window are fused through the frequency dimension processing component, which reduces the number of model parameters and reduces the output latency of the classification results of the first time window to less than 1 second.

[0065] In some implementations of this application, step 102 includes: determining the keyword to be detected and determining the time length based on the keyword; dividing the spectral features into multiple first time windows according to the time length.

[0066] First, identify the keywords to be detected. Then, determine the duration of each keyword based on its phoneme combinations; for example, the more phoneme combinations, the longer the duration. Next, divide the spectrogram feature into multiple first-time windows based on this duration, thus segmenting the spectrogram feature for processing.

[0067] As an example, one could calculate the average duration based on the time length of multiple keywords, and then divide the spectral features into multiple first time windows according to the average duration in the time dimension.

[0068] By combining keywords to determine the time length, the time length of the first time window can be dynamically adjusted according to the keywords (100-500ms), avoiding output jumps caused by fixed frame lengths, and providing a window length adaptive strategy.

[0069] Since the first time window needs to be divided according to a partial overlap method, this application embodiment sets an adjustable overlap rate (e.g., 30%~50%). By setting an adjustable overlap rate, it can adapt to different detection performance requirements. For example, a higher overlap rate requires higher performance.

[0070] Reference Figure 3 This paper illustrates a training method for a keyword detection model provided in an embodiment of this application. The embodiment of this application includes the following steps: Step 301: Extract sample speech from the sample dataset.

[0071] The sample dataset includes multiple positive samples (positive sample data) and multiple negative samples (negative sample data).

[0072] Reference Figure 4 This illustration shows a schematic diagram of a training sample generation process provided in an embodiment of this application. Positive samples (speech containing keywords) and negative samples (speech without keywords) are stored in a database or file system, respectively. When a training sample needs to be generated, either a positive or negative sample is randomly selected as the sample speech. For example, the training data generation program can randomly sample speech segments from the positive and negative sample databases to ensure class balance. During sampling, the speech is truncated according to a preset sample length (e.g., 2 seconds) to avoid silent segments.

[0073] Step 302: Perform speech rate transformation on the sample speech to obtain speed-adjusted speech.

[0074] To simulate different speaking habits and improve the stability of speech detection at different speaking speeds, the speaking speed of the sampled speech is randomly adjusted using a time stretching algorithm (e.g., adjusted to 0.8-1.2 times the original speed) to obtain speed-adjusted speech.

[0075] Step 303: Determine the sample labels for the speed-adjusted speech.

[0076] The sample labels can be the same as the classification labels. Positive samples can be labeled with keywords, while negative samples can be labeled with non-keywords, thus ensuring the stability of the keyword detection model in keyword detection after training.

[0077] Step 304: Add noise information and / or room impulse response information to the speed-adjusted speech to generate training speech.

[0078] Since the speech to be detected may be collected in a variety of different scenarios, in order to provide data diversity and improve the generalization ability of the keyword detection model, noise information and / or room impulse response (RIR) information can be added to the regulated speech to obtain training speech.

[0079] Noise data (including various types of noise information) and room impulse response data (including various types of room impulse response information) can be pre-generated. Noise information can be extracted from the noise database, and room impulse response information can be extracted from the room impulse response data. This enables the reuse of noise data and room impulse response data without the need to generate new noise information and room impulse response information for each training sample, thus improving the efficiency of training sample generation.

[0080] For example, a first trigger probability for noise information and a second trigger probability for room impulse response information can be set to randomly add noise information and / or room impulse response information to the speed-adjusted speech. For the noise information, a target signal-to-noise ratio (SNR) can be determined within a certain SNR range, and noise information can be added based on the target SNR to further improve the data diversity of the training speech.

[0081] For example, a silence insertion strategy can be set to insert silence segments into the training speech, so that the trained model can maintain its recognition performance in a noisy environment with a signal-to-noise ratio of 0dB and at low volume.

[0082] For example, a reverb effect (convolutional RIR) can also be added to the training speech.

[0083] Step 305: Generate training samples based on the training speech and sample labels.

[0084] Training samples are obtained by combining corresponding training voices and sample labels. This application does not limit the training method; as an example, a contrastive learning task can be set during training to effectively improve the model's robustness in unknown scenarios.

[0085] Step 306: Train the keyword detection model using training samples.

[0086] The keyword detection model is used to implement the keyword detection method as described in the above embodiments, so that the keyword detection model can implement each step of the above keyword detection method embodiments.

[0087] It is understandable that multiple training samples can be generated at once to obtain a training sample set, which can then be used to train the keyword detection model.

[0088] In traditional keyword detection, the training of keyword detection models can only target a single keyword. However, in this embodiment, the generation, annotation, and training of training samples can all be automated. The model trained using training samples is based on phoneme features rather than specific keywords. Furthermore, during the detection of the speech to be detected, the length of the first time window for segmenting the spectral features can be determined based on the keyword to be detected. By combining the above-mentioned training sample generation method and keyword detection method, this embodiment only needs to add positive samples when new keywords need to be detected, without having to repeat the complete model training process, which greatly reduces the maintenance cost of the keyword detection model.

[0089] In this embodiment, sample speech is extracted from a sample dataset, which includes multiple positive and multiple negative samples. The speech rate of the sample speech is transformed to obtain speed-adjusted speech. The sample labels of the speed-adjusted speech are determined. Noise information and / or room impulse response information are added to the speed-adjusted speech to generate training speech. Training samples are generated based on the training speech and the sample labels. This achieves the combination of positive samples, negative samples, noise information, and room impulse response information, simulating various acoustic scenarios in real-world environments. The random speech rate transformation and dynamic signal-to-noise ratio adjustment mechanism make the model more adaptable to different speech rates and noise environments.

[0090] For example, the training data generation program described above can use an iterator protocol to load training samples, thereby supporting online generation of training samples. Combined with dynamic data augmentation strategies (such as random frequency response perturbations), it effectively reduces storage overhead and improves model robustness. Furthermore, it uses training samples to train a keyword detection model, achieving end-to-end automation from data synthesis to model inference, thus reducing deployment complexity. The training data generation program can process multiple sample speech in parallel using hardware resource pools (such as multi-core CPUs), improving synthesis efficiency.

[0091] In some implementations of this application, step 302 includes: dividing the time dimension of the speed-adjusted speech into multiple at least partially overlapping second time windows; determining the energy characteristics of each second time window; determining a first detection label corresponding to the second time window if the energy characteristics meet a preset flat-peak detection condition; determining a second detection label corresponding to the second time window if the energy characteristics do not meet the preset flat-peak detection condition; and generating sample labels based on the first detection label and / or the second detection label corresponding to each second time window.

[0092] The training speech can be segmented into partially overlapping second time windows. Energy features are calculated for each second time window, and keyword segments are automatically labeled using a flat-top peak detection algorithm (such as threshold-based peak finding). Algorithm parameters (such as thresholds) can be dynamically adjusted according to keyword characteristics.

[0093] For example, the flat-peak detection condition is that the energy feature reaches an energy threshold and the duration reaches a certain time threshold. The first detection label can be the classification label corresponding to the keyword, and the second detection label can be the classification label corresponding to the non-keyword. By comparing the energy feature with the flat-peak detection condition, the training samples are automatically labeled, and the speech segments corresponding to the keywords are determined. There is no need for manual labeling of the keyword segments of the training samples, thus improving the training efficiency of the keyword detection model.

[0094] It should be noted that for training samples obtained by using negative samples as sample speech, it is not necessary to determine the energy features of each second time window, but to directly determine the second detection label corresponding to the second time window.

[0095] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0096] Reference Figure 5 The diagram illustrates a keyword detection device provided in an embodiment of this application, which may specifically include: The spectral feature acquisition module 501 is used to acquire the spectral features of the speech to be detected. The first time window segmentation module 502 is used to segment multiple at least partially overlapping first time windows in the time dimension of the acoustic spectrum features. The short-time phoneme feature generation module 503 is used to generate short-time phoneme features corresponding to each first time window based on the spectral features within each first time window. The global feature sequence generation module 504 is used to generate a global feature sequence based on the short-time phoneme features corresponding to each first time window; The keyword detection result generation module 505 is used to generate keyword detection results based on the global feature sequence.

[0097] In some implementations of the embodiments of this application, the acoustic spectrum feature acquisition module 501 includes: The spectrogram acquisition submodule is used to acquire the spectrogram of the speech to be detected; The spectrogram feature acquisition submodule is used to sequentially compress the feature dimensions of the spectrogram using a multi-level downsampling convolutional neural network to obtain the spectrogram features corresponding to the spectrogram. The multi-level downsampling convolutional neural network consists of multiple convolutional neural networks with a stride greater than 1.

[0098] In some implementations of this application, the first time window includes multiple frames; the short-time phoneme feature generation module 503 includes: The inter-frame dependency determination submodule is used to determine the inter-frame dependency of each frame contained in each first time window. The inter-frame dependency is obtained by training the first neural network model. The input layer of the first neural network model is connected to multiple frames in the same first time window. The short-time phoneme feature generation submodule is used to generate short-time phoneme features corresponding to each first time window based on the inter-frame dependency relationship and the spectral features corresponding to each frame.

[0099] In some implementations of this application, the global feature sequence generation module 504 includes: The context dependency determination submodule is used to determine the context dependencies between the first time windows. The context dependencies are obtained by training the second neural network model, and the input layer of the second neural network model is connected to two or more first time windows. The global feature sequence generation submodule is used to generate a global feature sequence based on contextual dependencies and short-time phoneme features corresponding to each first time window.

[0100] In some implementations of this application, the keyword detection result generation module 505 includes: The classification result generation submodule is used to process the global feature sequence using a trained classifier and output the classification results corresponding to each first time window; the classification results include classification labels. The classification score determination submodule is used to generate a classification score corresponding to each first time window based on the classification results of each first time window; the classification score is used to characterize the confidence level of the classification label of the first time window. The keyword detection result generation submodule is used to determine the keyword detection results of the speech to be detected based on the classification score corresponding to each first time window.

[0101] In some implementations of the embodiments of this application, the first time window division module 502 includes: The time length determination submodule is used to determine the keywords to be detected and to determine the time length based on the keywords; The first time window division submodule is used to divide the acoustic spectrum features into multiple first time windows according to the time length.

[0102] This application provides a keyword detection device, which can be used to implement the steps in the aforementioned keyword detection method embodiments.

[0103] Reference Figure 6 The diagram illustrates a training apparatus for a keyword detection model provided in an embodiment of this application, which may specifically include: The sample speech extraction module 601 is used to extract sample speech from a sample dataset; the sample dataset includes multiple positive samples and multiple negative samples. Speed ​​adjustment module 602 is used to perform speech rate conversion on sample speech to obtain speed-adjusted speech; The sample label determination module 603 is used to determine the sample labels of the speed-adjusting voice. The training speech generation module 604 is used to add noise information and / or room impulse response information to the speed-adjusted speech to generate training speech; The training sample generation module 605 is used to generate training samples based on the training speech and sample labels; Training module 606 is used to train a keyword detection model using training samples. The keyword detection model is used to implement the various steps of the keyword detection method embodiment as described above.

[0104] In some implementations of the embodiments of this application, determining the sample labels of speed-adjusted speech includes: Divide the time dimension of the speed-adjusted speech into multiple at least partially overlapping second time windows; Determine the energy characteristics of each second time window; If the energy characteristics meet the preset flat-top detection conditions, determine the first detection label corresponding to the second time window; If the energy characteristics do not meet the preset flat-top detection conditions, determine the second detection label corresponding to the second time window; Sample labels are generated based on the first detection label and / or the second detection label corresponding to each second time window.

[0105] This application provides a training device for a keyword detection model. By using this device, the various steps in the aforementioned keyword detection model training method embodiments can be implemented.

[0106] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0107] Reference Figure 7 The diagram illustrates an electronic device according to an embodiment of this application. Figure 7 As shown, the electronic device 700 in this embodiment includes: a processor 710, a memory 720, and a computer program 721 stored in the memory 720 and executable on the processor 710. When the processor 710 executes the computer program 721, it implements the steps of various embodiments of the keyword detection method or keyword detection model training method described above, for example... Figure 1 Steps 101 to 105 shown, or Figure 3 Steps 301 to 306 are shown. Alternatively, when processor 710 executes computer program 721, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 Modules 501 to 505 shown Figure 6 The functions of modules 601 to 606 are shown.

[0108] For example, computer program 721 may be divided into one or more modules / units, one or more of which are stored in memory 720 and executed by processor 710 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which may be used to describe the execution process of computer program 721 in electronic device 700.

[0109] Electronic device 700 can be a desktop computer, a cloud server, or a smart home device. Electronic device 700 may include, but is not limited to, a processor 710 and a memory 720. Those skilled in the art will understand that... Figure 7 This is merely one example of electronic device 700 and does not constitute a limitation on electronic device 700. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 700 may also include input / output devices, network access devices, buses, etc.

[0110] The processor 710 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0111] The memory 720 can be an internal storage unit of the electronic device 700, such as a hard disk or RAM of the electronic device 700. The memory 720 can also be an external storage device of the electronic device 700, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, FlashCard, etc., equipped on the electronic device 700. Furthermore, the memory 720 can include both internal and external storage units of the electronic device 700. The memory 720 is used to store the computer program 721 and other programs and data required by the electronic device 700. The memory 720 can also be used to temporarily store data that has been output or will be output.

[0112] This application also discloses a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the keyword detection method or the keyword detection model training method as described in the foregoing embodiments.

[0113] This application also discloses a computer program product, including a computer program, which, when run, causes the keyword detection method or keyword detection model training method as described in the foregoing embodiments to be executed.

[0114] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A keyword detection method, characterized in that, include: Obtain the spectral features of the speech to be detected; Divide the acoustic spectrum features into multiple at least partially overlapping first time windows along the time dimension; Based on the spectral features within each of the first time windows, short-time phoneme features corresponding to each of the first time windows are generated respectively; A global feature sequence is generated based on the short-time phoneme features corresponding to each of the first time windows; Based on the global feature sequence, keyword detection results are generated; The first time window contains multiple frames; the step of generating short-time phoneme features corresponding to each of the first time windows based on the spectral features within each first time window includes: The inter-frame dependencies of each frame contained in each first time window are determined. The inter-frame dependencies are obtained by training a first neural network model. The input layer of the first neural network model is connected to multiple frames in the same first time window. Based on the inter-frame dependency relationship and the spectral features corresponding to each frame, short-time phoneme features corresponding to each first time window are generated. The step of generating a global feature sequence based on the short-time phoneme features corresponding to each of the first time windows includes: The contextual dependencies between the first time windows are determined, and the contextual dependencies are obtained by training a second neural network model, wherein the input layer of the second neural network model is connected to two or more first time windows; A global feature sequence is generated based on the contextual dependencies and the short-time phoneme features corresponding to each of the first time windows.

2. The method according to claim 1, characterized in that, The acquisition of the spectral features of the speech to be detected includes: Obtain the spectrogram of the speech to be detected; The feature dimensions of the spectrogram are sequentially compressed using a multi-level downsampling convolutional neural network to obtain the spectrogram features corresponding to the spectrogram. The multi-level downsampling convolutional neural network consists of multiple convolutional neural networks with a stride greater than 1.

3. The method according to claim 1, characterized in that, The step of generating keyword detection results based on the global feature sequence includes: The global feature sequence is processed using a trained classifier to output classification results corresponding to each of the first time windows; the classification results include classification labels. Based on the classification results corresponding to each of the first time windows, a classification score corresponding to each of the first time windows is generated; the classification score is used to characterize the confidence level of the classification label of the first time window. Based on the classification score corresponding to each of the first time windows, the keyword detection result of the speech to be detected is determined.

4. The method according to any one of claims 1-3, characterized in that, The process of dividing the acoustic spectrum features into multiple at least partially overlapping first time windows includes: Identify the keywords to be detected, and determine the time length based on the keywords; The acoustic spectrum feature is divided into multiple first time windows according to the time length.

5. A training method for a keyword detection model, characterized in that, include: Extract sample speech from a sample dataset; the sample dataset includes multiple positive samples and multiple negative samples; The sample speech is subjected to speech rate transformation to obtain speed-adjusted speech; Determine the sample labels for the speed-adjusted speech; Add noise information and / or room impulse response information to the speed-adjusted speech to generate training speech; Based on the training speech and the sample labels, training samples are generated; The keyword detection model is obtained by training the training samples, and the keyword detection model is used to implement the keyword detection method as described in any one of claims 1 to 4.

6. The method according to claim 5, characterized in that, Determining the sample label of the speed-adjusted speech includes: Divide the time dimension of the speed-adjusting speech into multiple at least partially overlapping second time windows; Determine the energy characteristics of each of the second time windows; If the energy characteristics meet the preset flat-top detection conditions, a first detection label corresponding to the second time window is determined; If the energy characteristics do not meet the preset flat-top detection conditions, a second detection label corresponding to the second time window is determined; Sample labels are generated based on the first detection label and / or the second detection label corresponding to each of the second time windows.

7. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device performs the method as described in any one of claims 1-6.

8. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1-6 to be performed.

Images