A training method for keyword recognition based on phrase corpus

Abstract

This invention discloses a training method for keyword recognition based on phrase corpora. The method involves acquiring target keywords and generating synthetic speech samples containing them, with samples including context lengths of varying lengths; constructing a negative sample set that is acoustically similar to or semantically unrelated to the keywords; acquiring reference speech data and performing distribution adaptation to generate augmented data; training the keyword recognition model based on different context samples, augmented data, and negative samples using a training method incorporating curriculum learning (from easy to difficult, from long to short, from clean to noisy, etc.); employing boundary attention mechanisms during training, including at least one of boundary-weighted loss, label smoothing, frame enhancement, alignment loss, CTC constraint, attention enhancement, neighborhood sampling, and start / end point prediction. This invention does not require real recordings, improves the model's adaptability to real-world scenarios and phrase corpus recognition accuracy, and reduces false wake-ups, making it suitable for custom wake-up scenarios.

Description

Technical Field

[0001] This invention relates to the fields of speech processing and artificial intelligence, specifically to a training method for keyword recognition based on phrase corpus, and more particularly to a method for model training based on synthesized speech samples and reference speech data without the need to collect real recording samples of target keywords. Background Technology

[0002] Keyword recognition technology is widely used in smart terminals, smart homes, in-vehicle interactions, wearable devices, and other voice interaction scenarios. With the increasing demand for customized voice interaction, users are increasingly inclined to use shorter words or phrases as trigger commands or wake words. Compared to long sentences or phrases, phrase-based keywords often have less context, shorter duration, and limited boundary transition information. Therefore, they are more susceptible to the effects of speaking style, background noise, accent differences, and variations in terminal acquisition conditions, leading to a decline in recognition performance.

[0003] In existing technologies, text-to-speech or other data generation methods are typically used to construct keyword recognition training sets to reduce the cost of training data collection. While these methods can quickly generate target keyword samples without large-scale manual recording, they still suffer from the following problems: First, synthesized speech and speech in real-world environments often differ in acoustic distribution, making models trained on synthesized speech insufficiently adaptable to real-world scenarios. Secondly, due to limited contextual information, the model does not learn the relevant features of keyword boundaries sufficiently in phrase corpus samples, resulting in insufficient recognition stability under short pronunciation, weak boundary pronunciation or noise interference. Third, existing enhancement or transfer strategies typically address the issues of data distribution bias and missing corpus features separately, lacking a collaborative training mechanism.

[0004] Therefore, there is an urgent need for a training method that can reduce the distribution deviation between synthesized speech and real speech and enhance the model's ability to learn the boundary features of phrase keywords without collecting real recording samples of target keywords. Summary of the Invention

[0005] (a) Purpose of the invention

[0006] The purpose of this invention is to provide a training method for phrase corpus keyword recognition, in order to solve at least one of the following problems in the prior art: the distribution of synthesized speech samples differs greatly from that of real-world usage scenarios, resulting in insufficient generalization ability of the model; the context of phrase corpus keywords is insufficient, resulting in inadequate learning of boundary features; and existing solutions lack collaborative modeling of distribution adaptation and phrase corpus training, affecting the actual deployment effect.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A training method for keyword recognition based on phrase corpora includes the following steps: Step 1: Data generation; The system acquires target keyword information and generates synthesized speech samples containing the target keywords. The synthesized speech samples include at least two types of samples with different context lengths. Simultaneously, it acquires reference speech data as a reference domain and constructs a negative sample set that is acoustically similar to, phoneme-similar to, semantically unrelated to, or related to background interference with the target keywords.

[0009] Step 2: Data Augmentation; Based on the reference speech data, the synthesized speech samples are subjected to distribution adaptation processing to generate enhanced training data.

[0010] Step 3: Model building and training; A keyword recognition model is constructed, and the keyword recognition model is jointly trained or trained in stages based on samples of different context lengths, the enhanced training data, and the negative sample set to obtain a target model for phrase corpus keyword recognition.

[0011] Further, in step 2, the distribution adaptation process includes alignment processing to reduce the distribution difference between the synthesized speech samples and the reference speech data; the alignment processing includes at least one of adversarial alignment, statistical distribution alignment, feature matching, mapping transformation, or a combination thereof.

[0012] Furthermore, the alignment process includes: constraining the feature extraction process to generate a representation with reduced domain relevance by discriminative learning that distinguishes between the synthesized speech sample and the source of the reference speech data; and / or correcting the feature representation of the synthesized speech sample based on similar samples, cluster centers, statistical features, or prototype representations in the reference speech data.

[0013] Furthermore, in step 2, the generation of enhanced training data also includes injecting natural variation features into the synthesized speech samples; the natural variation features include at least one of noise features, accent features, speech rate features, prosodic features, speaker difference features, channel features, and reverberation features.

[0014] Furthermore, in step 3, the training of the keyword recognition model includes progressive training based on samples of different context lengths. The progressive training is carried out in a manner that transitions from longer contexts to shorter contexts, so as to gradually improve the model's ability to recognize keywords in phrase corpus.

[0015] Furthermore, in step 3, a boundary attention mechanism is adopted during model training. The boundary attention mechanism includes at least one of the following: boundary weighted loss, boundary label smoothing, boundary frame enhancement, boundary alignment loss, CTC boundary constraint, attention weight enhancement, boundary neighborhood sampling, and start and end point assisted prediction. The boundary attention mechanism is used to enhance the temporal, frequency, or time-frequency features related to the keyword start boundary, end boundary, or boundary neighborhood.

[0016] Furthermore, the boundary attention mechanism includes boundary weighted loss, which is achieved by assigning a higher loss weight to boundary-related intervals than to non-boundary intervals, thereby enhancing the model's ability to learn keyword boundary features; or it employs at least one of boundary label smoothing, boundary frame enhancement, boundary alignment loss, CTC boundary constraint, attention weight enhancement, boundary neighborhood sampling, and start-end point assisted prediction.

[0017] Furthermore, the training in step 3 employs a multi-stage training strategy and / or a course learning strategy; the multi-stage training strategy includes any combination of at least two of the following stages: The pre-training phase based on longer context samples; The adaptation training phase based on enhanced training data; Transfer training phase based on shorter context samples; Joint optimization phase based on boundary concern mechanism; The course learning strategy includes organizing training samples or training tasks in the order of easy to difficult, long to short, clean to noisy, synthetic to real distribution, and ordinary samples to difficult samples.

[0018] Furthermore, the keyword recognition model is a lightweight neural network model suitable for edge deployment, and the lightweight neural network model includes at least one of convolutional networks, temporal networks, attention networks, transformer networks, or combinations thereof.

[0019] Furthermore, the lightweight neural network model includes depthwise separable convolutional layers and a global feature aggregation module.

[0020] Furthermore, it also includes: constructing negative samples that are acoustically similar to the target keyword, phoneme similar, semantically unrelated, or related to background interference, and jointly training the keyword recognition model based on the negative samples; the negative samples are used to reduce the false wake-up rate of the model.

[0021] Furthermore, the trained keyword recognition model is deployed to edge devices to achieve real-time keyword recognition.

[0022] (III) Beneficial Effects

[0023] Compared with the prior art, the present invention has at least the following beneficial effects: (1) Constructing target keyword synthesis samples with different context lengths helps to enhance the model's ability to learn keywords from phrase corpus step by step; (2) By introducing reference speech data and performing distribution adaptation processing, it is helpful to reduce the distribution deviation between the synthesized speech samples and the real scene speech; (3) By introducing a boundary attention mechanism, it helps to improve the model’s ability to model the start boundary, end boundary and neighborhood features of keywords; (4) By incorporating distribution adaptation, natural mutation injection, progressive training, curriculum learning strategies and boundary attention mechanisms into a unified training framework, it is helpful to improve the model’s recognition performance and adaptability in short corpora and real-world scenarios. (5) By constructing and utilizing a negative sample set for joint training, it is helpful to reduce the false wake-up rate of the model to acoustic similarity interference and background noise, and improve the reliability of actual deployment; (6) This method does not require the collection of real audio samples of target keywords, which can reduce the data construction cost of custom keyword recognition system and shorten the model development cycle. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] This embodiment provides a training method for keyword recognition based on phrase corpus, including the following steps: Step 1: Data generation; The system receives user-defined keyword text and generates speech samples containing the keywords using speech synthesis technology. The speech samples include at least long contextual samples with context and short word samples containing only the keywords. Simultaneously, reference speech data is acquired.

[0026] The reference speech data is speech data used to characterize the acoustic distribution of real-world usage scenarios. It may or may not contain target keywords and may originate from one or more real-world application scenarios (such as home, in-vehicle, etc.). The reference speech data can be used to construct realistic feature representations, statistical distribution representations, prototype representations, and / or natural variation resources.

[0027] The long context sample refers to speech containing keywords and complete context, preserving the pronunciation context of keywords in natural sentences, and providing a foundation for the model to learn core acoustic features (pitch, rhythm, pronunciation style).

[0028] The short word samples refer to speech containing only keywords, which directly correspond to the target recognition scenario and serve as core training data.

[0029] In addition, this step constructs a negative sample set. The negative samples include at least one of the following types: easily confused words or phrases with similar acoustic features to the target keyword (e.g., high phoneme overlap); interfering words with similar pronunciation to the target keyword but unrelated semantics; common interfering audio in the background environment (e.g., television sounds, knocking sounds, music clips); and samples containing partial syllables of the target keyword but with different overall semantics. The negative samples are constructed using methods such as automatic generation based on phoneme rules, mining similar segments from reference speech data, or generation through adversarial perturbation. The negative samples are used to provide contrastive supervision during training to reduce false wake-ups of the model to non-target signals.

[0030] Step 2: Data Augmentation; Based on the reference speech data, the synthesized speech samples are subjected to distribution adaptation processing to generate enhanced training data that transfers the acoustic feature distribution to the real scene.

[0031] The distribution adaptation process includes alignment processing to reduce the distribution differences between the synthesized speech samples and the reference speech data.

[0032] In some implementations, the alignment process can be achieved through a discriminative domain alignment mechanism so that the feature representations extracted by the model are insensitive to differences in data sources.

[0033] In some implementations, the alignment process can also be achieved through a reference sample matching mechanism, that is, by correcting the feature representation of the synthesized speech sample based on the similarity between the synthesized speech sample and the reference speech data. The discriminative domain alignment mechanism and the reference sample matching mechanism described above can be used individually or in combination.

[0034] In some implementations, the alignment process may also employ known distribution alignment techniques such as statistical distribution alignment (e.g., maximum mean difference MMD) and mapping transformations (e.g., style transfer based on generative adversarial networks).

[0035] Specifically, an optional discriminative domain alignment implementation involves constructing a joint training framework comprising a feature extractor, a keyword classifier, and a domain discriminator. The domain discriminator distinguishes whether input features originate from synthetic or real data, and a gradient reversal layer is used for inverse optimization, enabling the feature extractor to learn a representation with reduced domain relevance. The joint loss is minimized during training.

[0036] An optional reference sample matching implementation is as follows: from the real feature library (or cluster centers, prototype representations) constructed based on the reference speech data, retrieve the reference representation that is most similar to the acoustic features of the speech sample; calculate the feature distance between the speech sample and the retrieved reference representation to generate a correction vector; use the correction vector to fine-tune the acoustic features of the speech sample to make it closer to the real distribution.

[0037] To further enhance the diversity of training data, natural variation features are injected into the synthesized speech samples. This injection can be achieved through noise perturbation, articulation style transfer, duration transformation, prosody adjustment, spectral distortion, channel response simulation, or combinations thereof. These natural variation features can be derived from statistical extraction of reference speech data or obtained through generative methods.

[0038] Step 2 outputs augmented training data that is aligned with the real data distribution while also containing rich scene variations.

[0039] Step 3: Model building and training; A keyword recognition model is constructed, and the keyword recognition model is trained based on samples of different context lengths, the augmented training data, and the negative sample set to obtain the final target model.

[0040] In a preferred embodiment, a course-based learning strategy is employed: training samples or tasks are organized in the following order: from easy to difficult, from long to short, from clean to noisy, from synthetic to real-world distributions, and from ordinary samples to difficult samples. Specifically, basic training is first performed using clean synthetic samples with complete context, enabling the model to establish preliminary acoustic representation capabilities. Then, enhanced training data (including natural variations and domain alignment) and shorter contextual samples are gradually introduced. Finally, boundary attention mechanisms and negative samples are added for refined training. This course-based learning sequence helps the model maintain stability of core keyword features while gradually adapting to complex conditions.

[0041] During training, a boundary attention mechanism is employed. This boundary attention mechanism includes, but is not limited to, at least one of the following methods: Boundary-weighted loss: assigns higher loss weights to the feature contributions of the keyword's start frame, end frame, and their neighborhood; Boundary label smoothing: Smooths frame labels near the boundary to mitigate the impact of annotation noise; Boundary frame enhancement: During training, the temporal or frequency domain features near the boundary are randomly perturbed or masked to enhance robustness; Boundary alignment loss: Calculate the frame-level distance between the predicted boundary and the true boundary, and use it as an auxiliary loss term; CTC boundary constraints: Explicit constraints on boundary frames are added to the CTC loss, prompting the model to pay more attention to the boundary region; Attention weight enhancement: guides the attention module to assign higher weights to boundary regions; Boundary neighborhood sampling: Increase the sampling density of frames near the boundary in the training samples to enable the model to learn boundary transition patterns more fully; Start and End Point Auxiliary Prediction: Adds an extra auxiliary output branch to specifically predict the start and end positions of keywords.

[0042] In this embodiment, as an example, boundary-weighted loss is used as the implementation method: a time-weighted cross-entropy loss function is used to assign a higher loss weight to boundary-related intervals than to non-boundary intervals, thereby enhancing the model's ability to learn keyword boundary features. However, it should be understood that the present invention is not limited to this, and other boundary attention methods can also be used independently or in combination.

[0043] In addition, negative samples are used in conjunction with comparative learning during training. Specifically, positive samples (target keywords) and negative samples are input into the model simultaneously, and the discriminative constraint between positive and negative samples is added to the classification loss, such as using center loss, triplet loss, or cross-entropy loss with negative samples. This makes the model's response to target keywords significantly higher than that of negative samples, thereby reducing the false wake-up rate.

[0044] Step 4: Multi-stage training In one specific implementation, a phased training strategy is adopted, and course learning ideas are integrated, including at least two combinations of the following phases: Phase 1: Basic Pre-training (from easy to difficult). Initial training is conducted using long-context samples of synthesized speech to initialize model parameters and allow it to initially grasp the pronunciation patterns of keywords. This phase employs low augmentation intensity, and the samples primarily consist of clean environments.

[0045] Phase 2: Augmentation and Adaptation Training (from clean to noisy, from synthetic to real distribution). The augmented training data generated in Step 2 is used for training, gradually increasing the noise, mutation, and domain alignment intensity to improve the model's adaptability to real-world scene variations. This phase can jointly optimize the classification loss and domain discrimination loss.

[0046] Phase 3: Length Transfer Training (from Long to Short). Following the order of transitioning from longer to shorter contexts, the model is trained using sample sequences of different context lengths, allowing it to gradually adapt to the given corpus conditions.

[0047] Phase 4: Joint optimization of boundary and negative samples (from ordinary samples to difficult samples). When training with pure short word samples, a boundary attention mechanism is applied, and negative samples are introduced for comparative learning, focusing on improving the model's sensitivity to short word boundaries and its ability to suppress interference.

[0048] The number of training rounds, loss weights, and stopping conditions at each stage can be set according to the data scale, model scale, and target performance requirements; this invention does not impose any limitations on these settings.

[0049] Keyword recognition model structure: A lightweight neural network model suitable for edge deployment, including but not limited to convolutional networks, temporal networks, attention networks, transformer networks, or combinations thereof. In a preferred embodiment, a lightweight deep separable convolutional neural network structure is employed, which may include a hybrid pooling layer (whose output is the arithmetic mean of the max pooling and average pooling results) and a dual-path global pooling fusion layer (which concatenates the feature vectors output by global average pooling and global max pooling to form a comprehensive global feature). The above specific structures are merely examples and are not intended to limit the scope of protection of this invention.

[0050] Simulation experiment verification

[0051] This embodiment uses the following settings for simulation experiment verification: Training set: 10 custom keywords (such as "turn on the lights", "turn on the air conditioner", etc.) generated using TTS technology, with 1000 samples generated for each keyword. No real recorded data for these keywords is used (simulating an unsupervised / zero-resource scenario).

[0052] Test set: 20 volunteers (half male and half female, including 5 with slight accents) were invited to record the above keywords in three real environments, totaling 2,000 audio recordings.

[0053] Test scenario: Quiet indoor environment: background noise <40dB.

[0054] Noisy environment: Background noise 60-70dB (simulated TV sound, noisy human voices).

[0055] Short Word Speed ​​Mode: Volunteers are asked to speak only short words at a fast pace (e.g., only say "lights" instead of "turn on the lights") to test the robustness of short word boundaries.

[0056] Comparative example: (1) Pure TTS direct training: The model is trained directly using TTS data, without domain alignment or length transfer.

[0057] (2) Traditional transfer learning: pre-training with a general large dataset + fine-tuning with TTS data.

[0058] (3) Data augmentation: Only traditional augmentation such as adding noise and pitch shifting is used, without adversarial domain adaptation.

[0059] With the false wake-up rate fixed at 1 time / hour, the comparison results are shown in Table 1: Table 1 Pure TTS direct training 82.5% 65.3% 45.2% 64.3% Traditional transfer learning 88.1% 72.4% 58.6% 73.0% Data Augmentation 86.4% 78.9% 51.3% 72.2% This embodiment 96.8% 91.5% 84.2% 90.8% Significantly improved short word performance: In the "Short Word Speed ​​Test," the wake-up rate of this embodiment reached 84.2%, far exceeding the 58.6% of traditional transfer learning. This directly proves that the "progressive length transfer" and "boundary enhancement loss" strategies effectively solve the problems of blurred short word boundaries and missing features mentioned above.

[0060] Enhanced noise resistance and robustness: In noisy environments, this embodiment achieves an improvement of approximately 26 percentage points compared to pure TTS direct training. This verifies that "bidirectional domain alignment + mutation injection" successfully brings the TTS data distribution closer to the distribution of real-world noisy environments.

[0061] Feasibility of zero-resource startup: The experiment was trained entirely on TTS data, yet it achieved an overall wake-up rate of over 90% in real-world scenarios, proving that this solution can support the deployment of a low-cost, high-accuracy custom keyword KWS system.

[0062] This embodiment significantly outperforms traditional methods in all tests, especially in the most challenging "short word speed" scenario, where it improves performance by more than 25 percentage points. This fully demonstrates the effectiveness of the "progressive length transfer" and "boundary enhancement" mechanisms, as well as the key role of "bidirectional domain alignment" in improving the robustness of the model.

[0063] Finally, after optimizing the trained lightweight model through techniques such as quantization and pruning, it can be deployed to edge devices such as smart speakers and mobile phones to achieve a high-accuracy real-time wake-up function based on custom keywords.

[0064] After optimizing the trained lightweight model through techniques such as quantization and pruning, it can be deployed to edge devices such as smart speakers and mobile phones to achieve a high-accuracy real-time wake-up function based on custom keywords.

[0065] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A training method for keyword recognition based on phrase corpora, characterized in that, include: Obtain target keyword information and generate synthetic speech samples containing the target keywords, wherein the synthetic speech samples include at least two types of samples with different context lengths; construct a negative sample set that is acoustically similar to the target keywords, phoneme similar, semantically unrelated, or related to background interference; Acquire reference speech data, and perform distribution adaptation processing on the synthesized speech samples based on the reference speech data to generate enhanced training data; Construct a keyword recognition model, and perform joint training or phased training on the keyword recognition model based on samples of different context lengths, the augmented training data and the negative sample set to obtain a target model for phrase corpus keyword recognition; The training includes a course learning strategy, which organizes training samples or training tasks in the order of easy to difficult, long to short, clean to noisy, synthetic to real distribution, and ordinary samples to difficult samples.

2. The training method for keyword recognition based on phrase corpus according to claim 1, characterized in that, The distribution adaptation process includes alignment processing to reduce the distribution differences between the synthesized speech samples and the reference speech data; the alignment processing includes at least one of adversarial alignment, statistical distribution alignment, feature matching, mapping transformation, or a combination thereof.

3. The training method for keyword recognition based on phrase corpus according to claim 2, characterized in that, The alignment process includes: constraining the feature extraction process to generate a representation with reduced domain relevance by discriminative learning that distinguishes between synthesized speech samples and reference speech data sources; and / or correcting the feature representation of synthesized speech samples based on similar samples, cluster centers, statistical features, or prototype representations in the reference speech data.

4. The training method for keyword recognition based on phrase corpus according to claim 1, characterized in that, The generation of enhanced training data also includes injecting natural variation features into the synthesized speech samples. These natural variation features include at least one of noise features, accent features, speech rate features, prosodic features, speaker difference features, channel features, and reverberation features.

5. The training method for keyword recognition based on phrase corpus according to claim 1, characterized in that, The training of the keyword recognition model includes progressive training based on samples of different context lengths. The progressive training is carried out in a manner that transitions from longer contexts to shorter contexts, so as to gradually improve the model's ability to recognize keywords in short phrase corpora.

6. The training method for keyword recognition based on phrase corpus according to claim 1, characterized in that, A boundary attention mechanism is adopted during model training, which includes at least one of the following: boundary weighted loss, boundary label smoothing, boundary frame enhancement, boundary alignment loss, CTC boundary constraint, attention weight enhancement, boundary neighborhood sampling, and start and end point assisted prediction; the boundary attention mechanism performs reinforcement learning on temporal features, frequency features, or time-frequency features related to the keyword start boundary, end boundary, or boundary neighborhood.

7. The training method for keyword recognition based on phrase corpus according to claim 6, characterized in that, The boundary attention mechanism includes boundary-weighted loss, which is achieved by assigning a higher loss weight to boundary-related intervals than to non-boundary intervals, thereby enhancing the model's ability to learn keyword boundary features.

8. A training method for keyword recognition based on phrase corpora according to any one of claims 1 to 7, characterized in that, The training employs a multi-stage training strategy and / or a course learning strategy; the multi-stage training strategy includes any combination of at least two of the following stages: The pre-training phase based on longer context samples; The adaptation training phase based on enhanced training data; Transfer training phase based on shorter context samples; Joint optimization phase based on boundary concern mechanism; The course learning strategy includes organizing training samples or training tasks in the order of easy to difficult, long to short, clean to noisy, synthetic to real distribution, and ordinary to difficult samples.

9. The training method for keyword recognition based on phrase corpus according to claim 8, characterized in that, The keyword recognition model is a lightweight neural network model suitable for edge deployment. The lightweight neural network model includes at least one of convolutional networks, temporal networks, attention networks, transformer networks, or combinations thereof.

10. The training method for keyword recognition based on phrase corpus according to claim 9, characterized in that, The lightweight neural network model includes depthwise separable convolutional layers and a global feature aggregation module.