Noise environment keyword detection method based on multi-task joint learning

Abstract

This invention relates to a keyword detection method in noisy environments based on multi-task joint learning, belonging to the fields of speech signal processing and natural language processing. It extracts multi-layer acoustic representations from noisy signals using a pre-trained audio encoder model, then performs temporal and inter-layer aggregation through global average pooling layers to generate highly compact and noise-resistant shared features, eliminating local interference from transient noise. Finally, the features are input in parallel into detection and classification branches, and a multi-task joint optimization strategy is used for training to enhance the model's ability to distinguish between speech and background noise. A keyword detection test dataset in a real noisy environment is constructed, and a cascaded decision logic of detection followed by classification is adopted in the inference stage to effectively filter invalid inputs. This invention combines the generalization representation power of the pre-trained model with a multi-task collaborative learning mechanism, significantly improving the model's recognition accuracy, system stability, and engineering practicality in low signal-to-noise ratio and real non-stationary noise environments.

Description

Technical Field

[0001] This invention relates to a keyword detection method for noisy environments based on multi-task joint learning, and to the fields of speech signal processing and natural language processing technology. Background Technology

[0002] Keyword detection is one of the core technologies in voice interaction systems. Its main task is to detect predefined keywords from continuous audio streams or short speech segments and output the corresponding keyword categories. Keyword detection technology has been widely used in smart speakers, mobile terminals, in-vehicle interaction, smart homes, industrial voice control and other scenarios, playing an important role in realizing voice wake-up and voice command control.

[0003] With the development of deep learning and automatic speech recognition technologies, keyword detection models based on convolutional neural networks and recurrent neural network structures have emerged, achieving good detection results on standard speech test sets. However, most existing keyword detection methods are based on relatively ideal recording environments, or simply simulate real-world environments by superimposing artificially synthesized noise onto clean speech. While these methods can achieve high accuracy under controlled conditions, they often exhibit significant performance degradation in real-world, complex noisy environments.

[0004] The reason for this is that background noise in real-world applications is typically non-stationary, complex, and time-varying. For example, in environments such as server rooms, streets, canteens, subway stations, laboratories, and air-conditioned dormitories, background noise may include various interfering factors such as continuous mechanical noise, instantaneous impact noise, conversations among people, and environmental reverberation. These factors can mask key discriminative information in keyword speech, making it difficult for existing keyword detection models to reliably extract robust acoustic features, thus increasing the false negative rate, raising the false positive rate, and decreasing classification accuracy.

[0005] On the other hand, while some existing methods attempt to improve performance by introducing pre-trained speech models or employing multi-task learning strategies, they still suffer from the following shortcomings: First, they fail to effectively utilize the global information in multi-layer pre-trained representations, making it difficult to obtain stable noise-resistant shared features; second, they lack effective collaborative modeling mechanisms for the tasks of "whether a keyword exists" and "which specific keyword it belongs to"; and third, testing and evaluation are mostly based on synthetic noise data, which cannot fully reflect the model's generalization ability in real-world complex noise environments. Therefore, there is an urgent need to propose a keyword detection method that can adapt to real-world complex noise scenarios while balancing detection accuracy and practical robustness. Summary of the Invention

[0006] The purpose of this invention is to address the problems of insufficient robustness, high false trigger rate, and limited generalization ability of existing keyword detection models in real-world complex noise environments, and to provide a keyword detection method for noisy environments based on multi-task joint learning. This method utilizes a pre-trained audio encoder to extract multi-layer robust acoustic representations, and aggregates temporal and hierarchical information through a global average pooling mechanism to form a compact and noise-resistant shared feature representation. Simultaneously, it combines keyword classification and speech activity detection tasks for joint optimization, thereby improving the model's keyword detection capability in low signal-to-noise ratio and real-world complex noise scenarios.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] The technical feature of this invention is to construct a keyword detection framework and keyword detection training data for real complex noise scenarios, train the keyword detection framework using a multi-task joint training strategy, and finally apply the trained keyword detection framework to complete the keyword detection task in a real noise environment.

[0009] The keyword detection framework mainly consists of an audio encoder, a global average pooling layer, a detection head, and a classification head. The audio encoder extracts multi-layer acoustic representations from the noisy input speech signal. The global average pooling layer performs temporal and inter-layer aggregation on these multi-layer acoustic representations to generate compact and noise-resistant shared feature representations. The detection head determines whether keywords exist in the input speech. The classification head outputs the corresponding keyword category when a keyword is detected. Training data is obtained by mixing keyword speech data with environmental noise data, and different signal-to-noise ratio conditions are set to obtain multiple sets of noisy training samples. The keyword detection task is used to learn keyword category discrimination, and the speech activity detection task is used to learn whether speech or keywords exist in the input audio.

[0010] The keyword detection method for noisy environments based on multi-task joint learning is implemented through the following steps:

[0011] Step 1: Construct joint training and real test data with multiple signal-to-noise ratios;

[0012] Specifically, by constructing a standardized set of keyword speech samples, constructing a set of environmental noise samples, constructing noisy training samples according to multiple signal-to-noise ratios, and constructing detection labels and classification labels, standardized speech samples with keyword category labels are obtained.

[0013] Step 2: Multi-layer feature extraction based on pre-trained audio encoder;

[0014] After converting the standardized speech samples with keyword category labels obtained in step one into acoustic features, robust hierarchical acoustic representations are extracted. Temporal aggregation is performed on the temporal acoustic representations of each layer, and then inter-layer fusion is performed on the aggregation results of each layer. The shared feature representation is obtained through the output representation of the audio encoder. The shared feature representation is input into the detection head and the classification head respectively. The detection head outputs the detection result of whether there are keywords in the input audio, and the classification head outputs the keyword category prediction result corresponding to the input audio.

[0015] Step 3: Noise-resistant shared feature aggregation based on global average pooling (GAP);

[0016] The multi-layer temporal hidden representation extracted in step two is subjected to global average pooling in both the time dimension and the hierarchical dimension to smooth out local perturbations of transient noise and generate a highly compact and noise-resistant globally shared feature representation.

[0017] Step 4: Collaborative discriminative learning based on multi-task joint optimization;

[0018] After obtaining the globally shared feature representation in step three, it is input into the detection head and the classification head respectively. The speech activity detection loss and keyword classification loss are calculated. By jointly optimizing the total loss function, a multi-task joint learning keyword detection model for noisy environments is obtained, which enables the multi-task joint learning keyword detection model for noisy environments to have the ability to extract speech and distinguish word classes in complex noise.

[0019] Step 5: Cascaded inference detection for real noise environments.

[0020] During the inference phase, the multi-task joint learning keyword detection model for noisy environments trained in step four is used to perform cascaded decision-making on unknown noisy frequencies by first detecting and then classifying them. Invalid inputs are filtered out first, and then keyword category detection is performed to achieve the keyword category detection task in a real and complex noise background.

[0021] Beneficial effects:

[0022] 1. The keyword detection method for noisy environments based on multi-task joint learning disclosed in this invention utilizes a pre-trained audio encoder to extract multi-layer robust acoustic representations, which can fully inherit the representational and noise resistance capabilities of large-scale pre-trained models, thereby improving the keyword detection performance in complex noisy environments.

[0023] 2. The keyword detection method for noisy environments based on multi-task joint learning disclosed in this invention constructs a stable and compact shared feature representation by sequentially performing time dimension aggregation and inter-layer aggregation on multi-layer temporal representations, which can effectively suppress the influence of instantaneous noise and local disturbances on the detection results.

[0024] 3. The keyword detection method for noisy environments disclosed in this invention introduces a multi-task joint learning mechanism for keyword classification tasks and speech activity detection tasks, enabling the model to simultaneously possess keyword category recognition ability and speech / noise differentiation ability, thereby reducing the false trigger rate in complex noisy environments.

[0025] 4. The keyword detection method for noisy environments based on multi-task joint learning disclosed in this invention improves the model's adaptability and generalization ability to real complex noise scenarios by constructing noisy training data with multiple signal-to-noise ratios and real environmental noise test data, and has strong engineering application value.

[0026] 5. The keyword detection method for noisy environments based on multi-task joint learning disclosed in this invention forms a reasoning process of detection first and classification later, which improves the reliability and practicality of the system in actual deployment while ensuring detection accuracy. Attached Figure Description

[0027] Figure 1 This is a flowchart of the keyword detection method for noisy environments based on multi-task joint learning, as proposed in this invention. Detailed Implementation

[0028] To better illustrate the purpose, technical solution, and beneficial effects of the present invention, the following detailed description of the present invention is provided in conjunction with the accompanying drawings and specific embodiments.

[0029] This implementation example Figure 1 As shown, the data processing flow and specific implementation steps of the keyword detection method for noisy environments based on multi-task joint learning are as follows:

[0030] Step 1: Construct a standardized set of keyword speech samples;

[0031] Specifically, it involves constructing basic speech data for the keyword detection task, providing a data foundation with dual supervision signals for the subsequent multi-task learning of the model.

[0032] A publicly available keyword speech dataset was selected as the source of keyword speech samples, specifically the Google SpeechCommand (GSC) dataset. This dataset contains multiple predefined keyword categories, each containing several short speech segments recorded by different speakers. To enhance the system's scalability in practical applications, new command word categories can also be added to the GSC keyword set.

[0033] In the keyword speech sample preprocessing, the original audio is first processed to unify the sampling rate, ensuring all keyword speech samples meet a uniform input specification. Then, audio segments are aligned in duration, silence segments are removed, and labels are normalized to form a standardized set of keyword speech samples suitable for subsequent mixing, training, and inference. The resulting standardized speech samples with keyword category labels are used as the main speech component for constructing noisy training data.

[0034] Step 2: Construct an environmental noise sample set;

[0035] Specifically, background noise data from realistic and complex environments is used to construct an environmental noise sample set. Publicly available environmental sound datasets are selected as the source of environmental noise samples, using the ESC-50 dataset. These environmental noise samples cover multiple types of background interference signals, used to simulate non-stationary noise environments commonly encountered in keyword detection tasks. In addition to publicly available environmental sound datasets, background signals such as actually recorded equipment noise, traffic noise, crowd conversation noise, wind noise, and air conditioning noise can also be added.

[0036] When preprocessing the environmental noise samples, long-duration noise recordings are first sliced ​​to ensure their length meets the requirements for subsequent mixing. Then, the sampling rate is unified and amplitude normalization is performed to remove excessively short, silent, or abnormally quality noise segments, resulting in a set of environmental noise samples that can be randomly combined with keyword speech samples.

[0037] Step 3: Construct noisy training samples according to multiple signal-to-noise ratios;

[0038] Specifically, the FANT tool is used to mix the keyword speech samples in step one with the environmental noise samples in step two to construct a noisy data training set and a noisy data validation set, generating training samples covering different noise intensities, thereby improving the robustness of the audio encoder in complex noisy environments.

[0039] Multiple predefined signal-to-noise ratio (SNR) conditions are set during mixing, with four noise levels—0 dB, -5 dB, -10 dB, and -15 dB—preferably used to simulate different acoustic environments ranging from moderate to strong noise. For each keyword speech sample, an ambient noise sample is randomly selected. After calculating the noise scaling factor based on the target SNR, the samples are superimposed to obtain the corresponding noisy speech sample. After superposition, the output waveform is amplitude-limited and standardized to obtain multiple sets of noisy training and validation samples under different SNR conditions, which are used for subsequent multi-task joint training.

[0040] Step 4: Construct detection labels and classification labels;

[0041] Specifically, dual supervision signals are generated for both keyword existence detection and keyword category recognition tasks. For each noisy sample obtained in step three, if it contains valid keyword speech content, it is assigned the detection label "Keyword Exists" and the corresponding keyword category label; if it only contains background noise or does not contain the target keyword, it is assigned the detection label "Keyword Does Not Exist".

[0042] Therefore, this step ultimately constructs two types of labels: one is a detection label, used to indicate whether keywords exist in the input audio, corresponding to the subsequent speech activity detection / keyword presence detection task; the other is a classification label, used to indicate the keyword category to which the input audio belongs, corresponding to the subsequent Keyword Classification (KWS) task. Through this dual supervision method, the audio encoder can learn both "whether there are keywords in the input" and "which category the keyword belongs to" during training, thereby reducing the risk of false activation under noisy conditions.

[0043] Step 5: Multi-layer feature extraction based on pre-trained audio encoder;

[0044] This step aims to leverage the powerful acoustic representation capabilities of large-scale pre-trained models to initially extract acoustic features from the original waveform. Specifically, after converting the standardized speech samples with keyword category labels obtained in step four into acoustic features, robust hierarchical acoustic representations are extracted. Temporal aggregation is then performed on each layer's temporal acoustic representation, followed by inter-layer fusion of the aggregation results. The final representation is then expressed through the output of the audio encoder. The shared feature representation is obtained; the shared feature representation is input into the detection head and the classification head respectively, wherein the detection head outputs the detection result of whether there are keywords in the input audio, and the classification head outputs the keyword category prediction result corresponding to the input audio.

[0045] The output of the audio encoder represents As shown in equation (1):

[0046] (1)

[0047] in, This shows the log-Mel filter bank characteristics corresponding to the input audio. Indicates an audio encoder; Indicates the number of layers in the audio encoder; No. The temporal hidden representation of the layer output; Indicates the number of frames; Indicates the number of Mel bands; This represents the hidden dimension. The audio encoder extracts features layer by layer and feeds them into the classifier.

[0048] Step 6: Noise-resistant shared feature aggregation based on global average pooling;

[0049] For the multi-layer temporal hidden representation extracted in step five, the time series is compressed into a single vector. Its physical meaning is to use the global context information of the whole speech to fill in or smooth the feature loss caused by sudden impact noise such as door closing sound, heavy object falling sound, etc. in a few frames.

[0050] First, the time-dimensional average of each layer of hidden representation in the encoder is performed. As shown in equation (2):

[0051] (2)

[0052] in, Indicates the first The representation of the layers after time-dimensional averaging. Indicates the total number of audio frames. Indicates the first The hidden state sequence output by the layer encoder contains timing information.

[0053] Secondly, the vectors from each layer, after being averaged in the time dimension, are arithmetically averaged again along the hierarchical dimension to fuse the acoustic and semantic information from different layers and obtain the final shared feature representation, thus obtaining the global shared feature representation. As shown in equation (3):

[0054] (3)

[0055] in, This represents the total number of encoder layers. The final shared feature representation is obtained by averaging the outputs of each coding layer over time and then fusing them between layers. It is used to characterize the global acoustic features and keyword semantic features of the input noisy speech sample.

[0056] Step 7: Collaborative discriminative learning based on multi-task joint optimization;

[0057] After obtaining the globally shared feature representation z in step six, it is input into the detection head and the classification head respectively. The speech activity detection loss and keyword classification loss are calculated. By jointly optimizing the total loss function, a multi-task joint learning keyword detection model for noisy environments is obtained, which enables the multi-task joint learning keyword detection model for noisy environments to have the ability to extract speech and distinguish word classes in complex noise.

[0058] Specifically, it involves receiving the globally shared feature representation generated in step five. Subsequently, through multi-task joint training, the model is encouraged to "actively" search for speech signals in complex backgrounds. Globally shared feature representations are then used. Parallel input to two fully connected network branches:

[0059] 1) Detection head: Performs speech activity detection (VAD) and determines globally shared feature representations. Does the text contain a speech segment? Output the binary classification prediction probability. .

[0060] 2) Classification Head: Perform keyword classification using KWS, output the posterior probability distribution of the corresponding C keyword categories, and calculate the classification loss by combining the classification labels from step four. .

[0061] The total loss function is constructed by weighted summing of tasks 1) and 2) in this step. As shown in equation (4):

[0062] (4)

[0063] in, This represents the classification loss corresponding to the keyword detection task; This represents the detection loss corresponding to the voice activity detection task; and These represent the weight coefficients of the losses for the two tasks. The total loss function is iteratively optimized through backpropagation. This makes the VAD task a powerful auxiliary regularization mechanism here, forcing the feature extraction in step five and the aggregation network in step six to learn to distinguish between "noise" and "effective speech" when updating weights, thus obtaining a noisy environment keyword detection model that is jointly learned by multiple tasks.

[0064] Step 8: Cascaded inference detection for real-world noise environments;

[0065] During the inference phase, a real noise keyword test dataset is collected based on the real environment. The noise environment keyword detection model trained in step four is used to perform a cascaded decision of detecting and classifying noisy frequencies. First, invalid inputs are filtered out, and then keyword category detection is performed to achieve the keyword category detection task in a real complex noise background.

[0066] The keyword detection model in noisy environments, trained through multi-task joint learning in step seven, is utilized. A real-world dataset of noisy keywords is collected to evaluate the robustness of the keyword detection model under complex noise conditions.

[0067] The real-world environments in this step include indoor equipment noise scenarios and outdoor open noise scenarios. The indoor equipment noise scenarios include at least one or more of the following: server room, computer lab, and dormitory air-conditioned environment. The outdoor open noise scenarios include at least one or more of the following: street, canteen, subway station, and square. The test audio collected from these real-world environments is manually reviewed, segmented, and labeled to form a real-world noise keyword test dataset.

[0068] Unlike traditional testing methods that rely solely on synthetic noise, this step preserves real-world device noise, crowd interference, environmental echoes, and transient impact disturbances, enabling the test set to better reflect the complexity of actual deployment environments and thus more accurately assess the model's engineering usability and generalization capabilities.

[0069] A multi-task joint learning-based keyword detection model for noisy environments is used to perform inference on real noisy keyword test data. During inference, the detection head in step 7.1) first determines whether a keyword exists in the current input. If the detection result indicates that no keyword exists, an empty label is output; if the detection result indicates that a keyword exists, the classification head in step 7.2) outputs the corresponding keyword category. Based on the recognition results of the multi-task joint learning-based keyword detection model for noisy environments on the real noisy keyword test dataset, the recognition accuracy, false detection rate, and generalization performance of the keyword detection model in real complex noisy environments are evaluated.

[0070] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A keyword detection method for noisy environments based on multi-task joint learning, characterized in that: Specifically, this is achieved through the following steps: Step 1: Construct joint training and real test data with multiple signal-to-noise ratios; Specifically, by constructing a standardized keyword speech sample set, constructing an environmental noise sample set, constructing noisy training samples according to multiple signal-to-noise ratios, and constructing detection labels and classification labels, standardized speech samples with keyword category labels are obtained. Step 2: Multi-layer feature extraction based on pre-trained audio encoder; After converting the standardized speech samples with keyword category labels obtained in step one into acoustic features, robust hierarchical acoustic representations are extracted. Temporal aggregation is then performed on each layer's temporal acoustic representation, followed by inter-layer fusion of the aggregation results. The final representation is then expressed through the output of the audio encoder. The shared feature representation is obtained; the shared feature representation is input into the detection head and the classification head respectively, wherein the detection head outputs the detection result of whether there are keywords in the input audio, and the classification head outputs the keyword category prediction result corresponding to the input audio; Step 3: Noise-resistant shared feature aggregation based on global average pooling (GAP); The multi-layer temporal hidden representations extracted in step two are then averaged sequentially along the time dimension. Global average pooling along the hierarchical dimension smooths out local perturbations from transient noise, generating globally shared feature representations. ; Step 4: Collaborative discriminative learning based on multi-task joint optimization; The globally shared feature representation obtained in step three The inputs are fed into the detection head and the classification head respectively, and the speech activity detection loss and keyword classification loss are calculated. By jointly optimizing the total loss function, a multi-task joint learning keyword detection model for noisy environments is obtained, which enables the multi-task joint learning keyword detection model for noisy environments to have the ability to extract speech and distinguish word classes in complex noise. Step 5: Cascaded inference detection for real-world noise environments; During the inference phase, a real noise keyword test dataset is collected based on the real environment. The noise environment keyword detection model trained in step four is used to perform a cascaded decision of detecting and classifying noisy frequencies. First, invalid inputs are filtered out, and then keyword category detection is performed to achieve the keyword category detection task in a real complex noise background.

2. The keyword detection method for noisy environments based on multi-task joint learning as described in claim 1, characterized in that: The output representation of the audio encoder in step two As shown in equation (1): （1） in, This shows the log-Mel filter bank characteristics corresponding to the input audio. Indicates an audio encoder; Indicates the number of layers in the audio encoder; No. The temporal hidden representation of the layer output; Indicates the number of frames; Indicates the number of Mel bands; The hidden dimension is represented; the audio encoder extracts features layer by layer and feeds them into the classifier.

3. The keyword detection method for noisy environments based on multi-task joint learning as described in claim 1, characterized in that: In step three, the hidden representation of each layer of the encoder is averaged over time. As shown in equation (2): （2） in, Indicates the first The representation of the layers after time-dimensional averaging. Indicates the total number of audio frames. Indicates the first The hidden state sequence containing timing information is output by the layer encoder; Globally shared feature representation As shown in equation (3): （3） in, This represents the total number of encoder layers. The final shared feature representation is obtained by averaging the outputs of each coding layer over time and then fusing them between layers. It is used to characterize the global acoustic features and keyword semantic features of the input noisy speech sample.

4. The keyword detection method for noisy environments based on multi-task joint learning as described in claim 1, characterized in that: In step four, the globally shared feature representation generated in step three is received. Subsequently, through multi-task joint training, the model is encouraged to "actively" search for speech signals in complex backgrounds; globally shared feature representations are then used. Parallel input to two fully connected network branches: 1) Detection head: Performs speech activity detection (VAD) and determines globally shared feature representations. Does the text contain a speech segment? Output the binary classification prediction probability. ; 2) Classification Head: Perform keyword classification using KWS, output the posterior probability distribution of the corresponding C keyword categories, and calculate the classification loss by combining the classification labels from step four. ; The total loss function is constructed by weighted summing of tasks 1) and 2) in this step. As shown in equation (4): （4） in, This represents the classification loss corresponding to the keyword detection task; This represents the detection loss corresponding to the voice activity detection task; and These represent the weight coefficients of the losses for the two tasks; the total loss function is iteratively optimized through backpropagation. This makes the VAD task a powerful auxiliary regularization mechanism here, forcing the feature extraction in step five and the aggregation network in step six to learn to distinguish between "noise" and "effective speech" when updating weights, thus obtaining a noisy environment keyword detection model that is jointly learned by multiple tasks.

5. The keyword detection method for noisy environments based on multi-task joint learning as described in claim 1, characterized in that: In step five, the real environment includes indoor equipment noise scenarios and outdoor open noise scenarios. The indoor equipment noise scenarios include at least one or more of the following: server room, computer laboratory, and dormitory air-conditioned environment. The outdoor open noise scenarios include at least one or more of the following: street, canteen, subway station, and square. The test audio collected in the real environment is manually reviewed, segmented, and labeled to form a real noise keyword test dataset.

Images