Speech wake-up method and device based on multi-task joint model, and storage medium

Abstract

The application discloses a speech wake-up method and device based on a multi-task joint model and a storage medium, relates to the technical field of artificial intelligence, and comprises the following steps: acquiring speech real-time data of a target device, inputting speech representation of the speech real-time data into a pre-trained multi-task joint model, determining speech time-frequency features of the speech real-time data based on the multi-task joint model, further performing probability prediction on the speech time-frequency features, obtaining a wake-up probability, and using the wake-up probability to represent the probability that the speech real-time data comprises a target wake-up word. Since the multi-task joint model is a model obtained by jointly training a noise reduction task and a wake-up task, the multi-task joint model has certain robustness and accuracy, can effectively improve the accuracy of the wake-up probability, and finally, wake-up processing of the target device is carried out in combination with the wake-up probability, so that the false wake-up of the target device is reduced and the accuracy of speech wake-up is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a voice wake-up method, device, and storage medium based on a multi-task joint model. Background Technology

[0002] With the development of artificial intelligence technology, voice wake-up technology plays a crucial role, forming the basis for natural interaction between users and smart voice devices. For example, smart speakers and smart refrigerators in smart homes, or intelligent devices in industrial automation scenarios, can activate the device's voice function through specific wake words, thereby issuing commands or querying information.

[0003] Currently, most smart voice devices use neural network models for voice wake-up, and then determine whether to activate the voice function based on the wake-up result. However, current neural network models suffer from low accuracy in voice wake-up, which significantly affects the performance of smart voice devices.

[0004] Therefore, there is an urgent need for a new voice wake-up method to overcome the shortcomings of conventional technologies and improve the accuracy of voice wake-up. Summary of the Invention

[0005] This application provides a voice wake-up method, apparatus, and storage medium based on a multi-task joint model, to at least solve the problem of low voice wake-up accuracy.

[0006] According to one aspect of the embodiments of this application, a voice wake-up method based on a multi-task joint model is provided, comprising: acquiring real-time voice data of a target device, and inputting the voice representation of the real-time voice data into a pre-trained multi-task joint model; wherein the multi-task joint model is obtained by training a neural network model based on joint tasks; the joint tasks include: a noise reduction task and a wake-up task; determining the voice time-frequency features corresponding to the voice representation based on the multi-task joint model, and performing probability prediction on the voice time-frequency features to obtain a wake-up probability; wherein the wake-up probability is used to characterize the probability that the real-time voice data includes a target wake-up word; and performing wake-up processing on the target device when a preset wake-up condition is met based on the wake-up probability.

[0007] In an exemplary embodiment, the multi-task joint model includes: an encoder and a wake-up classifier; determining the speech time-frequency features corresponding to the speech representation based on the multi-task joint model, and performing probability prediction on the speech time-frequency features to obtain the wake-up probability, includes: determining the time-frequency features corresponding to the speech representation based on the encoder, performing correlation analysis on multiple time-frequency units included in the time-frequency features, and obtaining the speech time-frequency features based on the correlation analysis results; inputting the speech time-frequency features to the wake-up classifier, and having the wake-up classifier perform probability prediction based on the speech time-frequency features to obtain the wake-up probability.

[0008] In an exemplary embodiment, the encoder includes a convolutional layer, a downsampling layer, and an attention layer. The process involves determining the time-frequency features corresponding to the speech representation based on the encoder, performing correlation analysis on multiple time-frequency units included in the time-frequency features, and obtaining the speech time-frequency features based on the correlation analysis results. This includes: inputting the speech representation into the convolutional layer; convolving the speech representation according to a first preset dimension to obtain initial time-frequency features of the speech representation; inputting the initial time-frequency features into the downsampling layer; compressing the initial time-frequency features with a preset stride using a second preset dimension to obtain the time-frequency features; inputting the time-frequency features into the attention layer; where both the first and second preset dimensions include a time dimension and a frequency dimension; splitting the time-frequency features based on the attention layer to obtain multiple time-frequency units, establishing correlation relationships between the multiple time-frequency units, and obtaining correlation analysis results; and performing average pooling on the multiple time-frequency units and the correlation analysis results to obtain the speech time-frequency features.

[0009] In an exemplary embodiment, the speech time-frequency features include: multi-frame time-frequency segmented features; the wake-up classifier includes: a temporal modeling layer, a frame-level classification layer, and a time aggregation layer; the wake-up classifier performs probability prediction based on the speech time-frequency features to obtain the wake-up probability, including: inputting the multi-frame time-frequency segmented features into the temporal modeling layer, whereby the temporal modeling layer processes the multi-frame time-frequency segmented features to obtain hidden state features corresponding to each of the multi-frame time-frequency segmented features; inputting each hidden state feature into the frame-level classification layer, whereby the frame-level classification layer classifies and discriminates each hidden state feature to obtain multiple frame-level wake-up probabilities, and inputting the multiple frame-level wake-up probabilities into the time aggregation layer; and the time aggregation layer aggregates the multiple frame-level wake-up probabilities according to a set time window to obtain the wake-up probability.

[0010] In an exemplary embodiment, the step of inputting the multi-frame time-frequency segmentation features into the timing modeling layer, and having the timing modeling layer process the multi-frame time-frequency segmentation features to obtain the hidden state features corresponding to each of the multi-frame time-frequency segmentation features, includes: for any time-frequency segmentation feature t in the multi-frame time-frequency segmentation features, inputting the any time-frequency segmentation feature t into a first expression of the timing modeling layer, and determining the hidden state feature corresponding to the any time-frequency segmentation feature t based on the first expression; the first expression is: ;in, Refers to the hidden state feature corresponding to any time-frequency frame feature t; Refers to any of the aforementioned time-frequency frame features t; Refers to one-dimensional temporal convolution operation; Refers to the activation function.

[0011] In an exemplary embodiment, the hidden state features are classified and discriminated based on the frame-level classification layer to obtain multiple frame-level wake-up probabilities, including: for any one of the hidden state features... , and the hidden state feature The second expression input to the frame-level classification layer is used to evaluate the features of any hidden state based on the second expression. Perform classification and discrimination to obtain the features of any hidden state. The frame-level wake-up probability; the second expression is: ;in, The frame-level wake-up probability of any hidden state feature t; 1 refers to the activation function. Refers to the weight matrix; This refers to the bias parameter.

[0012] In an exemplary embodiment, when the wake-up probability determines that a preset wake-up condition is met, the target device is woken up, which includes: comparing the wake-up probability with a preset wake-up probability threshold to obtain a comparison result; and when the comparison result indicates that the wake-up probability is greater than the wake-up probability threshold, the target device is woken up.

[0013] In an exemplary embodiment, before acquiring real-time speech data from a target device and inputting the speech representation of the real-time speech data into a pre-trained multi-task joint model, the method further includes: acquiring speech training data and inputting sample speech representations of the speech training data into a neural network model to be trained; the neural network model includes: an encoder, a denoiser, and a wake-up classifier; performing feature extraction on the sample speech representation based on the encoder to obtain sample speech time-frequency features; inputting the sample speech time-frequency features into the denoiser, performing speech reconstruction processing on the sample speech time-frequency features based on the denoiser to determine a first denoising loss; and inputting the sample speech time-frequency features into the wake-up classifier, performing probability prediction on the sample speech time-frequency features based on the wake-up classifier to determine a wake-up loss; training the neural network model to be trained based on the first denoising loss and the wake-up loss to adjust the encoder parameters of the encoder, the denoiser parameters of the denoiser, and the classifier parameters of the wake-up classifier to obtain a trained multi-task joint model.

[0014] In an exemplary embodiment, the noise reduction device includes: a feature restoration layer, a fusion layer, and an output layer; performing speech reconstruction processing on the sample speech time-frequency features based on the noise reduction device to determine a first noise reduction loss includes: inputting the sample speech time-frequency features into the feature restoration layer; performing at least one upsampling process on the sample speech time-frequency features based on the feature restoration layer to obtain a feature restoration result, and inputting the feature restoration result into the fusion layer; performing fusion processing on the sample speech time-frequency features and the feature restoration result based on the fusion layer to obtain a fusion processing result, and inputting the fusion processing result into the output layer; outputting the mask parameters corresponding to the fusion processing result from the output layer, and performing speech reconstruction processing based on the mask parameters to determine the first noise reduction loss.

[0015] In an exemplary embodiment, the time-frequency features of the sample speech and the feature reconstruction result are fused based on the fusion layer to obtain a fusion processing result, including: inputting the time-frequency features of the sample speech and the feature reconstruction result into a third expression of the fusion layer, and performing fusion processing on the time-frequency features of the sample speech and the feature reconstruction result based on the third expression to obtain a fusion processing result; the third expression is: ;in, Refers to the feature reconstruction result, The jump-connection mechanism, in which, = , Refers to the time-frequency features of the sample speech; Transposed convolution; Refers to the result of the fusion process.

[0016] In an exemplary embodiment, the masking parameters include: amplitude masking values; the output layer outputs the masking parameters corresponding to the fusion processing result, and performs speech reconstruction processing based on the masking parameters to determine the first noise reduction loss, including: inputting the fusion processing result to a fourth expression of the output layer, and processing the fusion processing result with the fourth expression to obtain the amplitude masking value; wherein, the fourth expression is: ;in, Refers to the amplitude mask value. Activation function; Refers to one-dimensional temporal convolution operation; The result of the fusion processing is used; the amplitude mask value is input into the fifth expression, and the first clean speech spectrum is reconstructed from the fifth expression; the fifth expression is: ;in, Refers to the first pure speech spectrum; The noise amplitude spectrum is used as a reference; the first noise reduction loss is determined based on the first clean speech spectrum and the set amplitude spectrum label.

[0017] In an exemplary embodiment, the masking parameters include a complex mask matrix; the output layer outputs the masking parameters corresponding to the fusion processing result, and performs speech reconstruction processing based on the masking parameters to determine a first noise reduction loss, including: reconstructing the complex spectrum of the fusion processing result based on the output layer to obtain a complex mask matrix; converting the format of the complex mask matrix to obtain a complex domain, and reconstructing a second clean speech spectrum based on the complex domain; and determining the first noise reduction loss based on the second clean speech spectrum and a set amplitude spectrum label.

[0018] In an exemplary embodiment, the sample speech time-frequency features include: multi-frame sample time-frequency segmented features; the wake-up classifier includes: a temporal modeling layer, a frame-level classification layer, and a temporal aggregation layer; inputting the sample speech time-frequency features into the wake-up classifier, and performing probability prediction on the sample speech time-frequency features based on the wake-up classifier to determine the wake-up loss includes: inputting the multi-frame sample time-frequency segmented features into the temporal modeling layer, where the temporal modeling layer processes the multi-frame sample time-frequency segmented features respectively to obtain sample hidden state features corresponding to the multi-frame sample time-frequency segmented features; inputting each sample hidden state feature into the frame-level classification layer, where the frame-level classification layer classifies and discriminates each sample hidden state feature to obtain multiple sample frame-level wake-up probabilities, and inputting the multiple sample frame-level wake-up probabilities into the temporal aggregation layer; aggregating the multiple sample frame-level wake-up probabilities according to a set time window based on the temporal aggregation layer to obtain sample wake-up probabilities; and performing cross-entropy processing on the sample wake-up probabilities and preset label wake-up probabilities to obtain the wake-up loss.

[0019] In an exemplary embodiment, the neural network model to be trained is trained based on the first noise reduction loss and the wake-up loss to adjust the encoder parameters of the encoder, the noise reduction parameters of the noise reducer, and the classifier parameters of the wake-up classifier to obtain a trained multi-task joint model. This includes: obtaining a second noise reduction loss; wherein the second noise reduction loss is output by the speech noise reduction model after the speech training data is input into it; fusing the first noise reduction loss, the second noise reduction loss, and the wake-up loss to obtain a target loss; adjusting the encoder parameters of the encoder, the noise reduction parameters of the noise reducer, and the classifier parameters of the wake-up classifier according to the target loss to train the neural network model until a training stop condition is met to obtain a trained initial multi-task joint model; and pruning the initial multi-task joint model to obtain a multi-task joint model.

[0020] According to another aspect of the embodiments of this application, a voice wake-up device based on a multi-task joint model is also provided, comprising: an acquisition module, configured to acquire real-time voice data of a target device and input the voice representation of the real-time voice data into a pre-trained multi-task joint model; wherein the multi-task joint model is obtained by training a neural network model based on joint tasks; the joint tasks include: a noise reduction task and a wake-up task; a prediction module, configured to determine the voice time-frequency features corresponding to the voice representation based on the multi-task joint model, and perform probability prediction on the voice time-frequency features to obtain a wake-up probability; wherein the wake-up probability is used to characterize the probability that the real-time voice data includes a target wake-up word; and a wake-up module, configured to perform wake-up processing on the target device when a preset wake-up condition is met based on the wake-up probability.

[0021] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described voice wake-up method based on a multi-task joint model when it is run.

[0022] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described voice wake-up method based on a multi-task joint model through the computer program.

[0023] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program, which, when executed by a processor, is the above-described voice wake-up method based on a multi-task joint model.

[0024] This application acquires real-time speech data from a target device and inputs the speech representation of the real-time speech data into a pre-trained multi-task joint model. The multi-task joint model is trained by a neural network model based on joint tasks, including a noise reduction task and a wake-up task. The multi-task joint model then determines the speech time-frequency features of the real-time speech data and further performs probability prediction on these features to obtain the wake-up probability. The wake-up probability characterizes the probability that the real-time speech data includes the target wake-up word. Since the multi-task joint model is trained on the joint noise reduction and wake-up tasks, it possesses certain robustness and accuracy, effectively improving the accuracy of the wake-up probability. Finally, by combining the wake-up probability with the wake-up processing of the target device, false wake-ups of the target device can be reduced, and the accuracy of voice wake-up can be improved. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the hardware environment for a voice wake-up method based on a multi-task joint model according to an embodiment of this application;

[0028] Figure 2 This is one of the flowcharts of a voice wake-up method based on a multi-task joint model according to an embodiment of this application;

[0029] Figure 3 This is one of the flowcharts of a voice wake-up method based on a multi-task joint model according to an embodiment of this application;

[0030] Figure 4 This is a second flowchart of a voice wake-up method based on a multi-task joint model according to an embodiment of this application;

[0031] Figure 5 This is a second flowchart of a voice wake-up method based on a multi-task joint model according to an embodiment of this application;

[0032] Figure 6 This is the third flowchart of a voice wake-up method based on a multi-task joint model according to an embodiment of this application;

[0033] Figure 7 This is a structural block diagram of a voice wake-up device based on a multi-task joint model according to an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] According to one aspect of the embodiments of this application, a voice wake-up method based on a multi-task joint model is provided. This voice wake-up method based on a multi-task joint model is widely used in whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned voice wake-up method based on a multi-task joint model can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 can acquire real-time speech data of the target device collected by terminal device 102, and then input the speech representation of the real-time speech data into a pre-trained multi-task joint model. Based on the multi-task joint model, the speech time-frequency features corresponding to the speech representation are determined, and the speech time-frequency features are probabilistically predicted to obtain the wake-up probability. Thus, if the preset wake-up conditions are met based on the wake-up probability, the target device is woken up. Server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently to provide data processing services for server 104.

[0038] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0039] To address the aforementioned issues, this embodiment provides a voice wake-up method based on a multi-task joint model, which can be applied to whole-house smart digital control applications. Figure 2 This is a flowchart of a voice wake-up method based on a multi-task joint model according to an embodiment of this application. The process includes the following steps S202-S206:

[0040] Step S202: Acquire real-time speech data from the target device and input the speech representation of the real-time speech data into a pre-trained multi-task joint model; wherein, the multi-task joint model is obtained by training a neural network model based on joint tasks; the joint tasks include: noise reduction task and wake-up task;

[0041] It should be noted that real-time speech data is time-series data continuously output in frames, which is collected by the microphone built into the target device and preprocessed by the front end; the noise reduction task is to train a neural network model to learn to reconstruct speech from the speech training data in order to recover clean speech from the speech training data; the wake-up task is to train a neural network model to learn to perform binary classification on the speech training data and determine whether it contains the target wake-up word.

[0042] Step S204: Determine the speech time-frequency features corresponding to the speech representation based on the multi-task joint model, and perform probability prediction on the speech time-frequency features to obtain the wake-up probability; wherein, the wake-up probability is used to characterize the probability that the real-time speech data includes the target wake-up word;

[0043] Among them, speech time-frequency features are a quantitative expression of the energy distribution of speech at different time points and at different frequency components, which can capture the instantaneous dynamic changes and spectral structure characteristics of speech. Wake-up probability is the probability determined by the multi-task joint model based on its learned semantic discrimination ability whether the current input real-time speech data includes the target wake-up word.

[0044] Optionally, the multi-task joint model includes an encoder and a wake-up classifier. Based on the multi-task joint model, the model determines the speech time-frequency features corresponding to the speech representation and performs probability prediction on these features to obtain the wake-up probability. This includes: determining the speech time-frequency features corresponding to the speech representation based on the encoder, performing correlation analysis on multiple time-frequency units included in the time-frequency features, and obtaining the speech time-frequency features based on the correlation analysis results; inputting the speech time-frequency features into the wake-up classifier, which then performs probability prediction based on these features to obtain the wake-up probability. The encoder is responsible for extracting the speech time-frequency features from the speech representation, while the wake-up classifier makes the final wake-up decision based on these features. The wake-up probability is the endpoint of the entire multi-task joint model inference chain. As the result of the encoder's global semantic understanding of the speech time-frequency structure and the wake-up classifier's accurate capture of temporal dynamics, the wake-up probability relies on the encoder's noise and semantic separation capabilities and wake-up word structure perception capabilities learned during the training phase. This ensures that even in environments with low signal-to-noise ratios and background noise such as continuous motor noise or fan airflow, it can accurately identify genuine human voice wake-up commands and effectively suppress false wake-ups.

[0045] Optionally, the encoder includes a convolutional layer, a downsampling layer, and an attention layer. Based on the encoder, the time-frequency features corresponding to the speech representation are determined, and correlation analysis is performed on multiple time-frequency units included in the time-frequency features. The speech time-frequency features are obtained based on the correlation analysis results, including: inputting the speech representation into the convolutional layer; convolving the speech representation according to a first preset dimension to obtain initial time-frequency features of the speech representation; inputting the initial time-frequency features into the downsampling layer; compressing the initial time-frequency features with a preset stride in a second preset dimension to obtain time-frequency features; inputting the time-frequency features into the attention layer; both the first and second preset dimensions include a time dimension and a frequency dimension; decomposing the time-frequency features based on the attention layer to obtain multiple time-frequency units, establishing the correlation relationship between the multiple time-frequency units, and obtaining the correlation analysis results; and performing average pooling on the multiple time-frequency units and the correlation analysis results to obtain the speech time-frequency features.

[0046] It should be noted that in the multi-task joint model of this application, the encoder, as the core feature extraction module, consists of a three-level structure of convolutional layer, downsampling layer and attention layer. Its goal is to effectively suppress interference components in high-noise environment while preserving the semantic structure of speech, thereby generating speech time-frequency features with strong discriminative ability.

[0047] For example, speech is represented as Where T is the number of time frames and F is the frequency dimension; when the speech representation is input to the encoder, it first undergoes initial feature extraction by a convolutional layer. The convolutional layer uses a two-dimensional convolutional kernel, and its convolution operation is performed simultaneously in both the time and frequency dimensions, i.e., the first preset dimension is the joint dimension of time and frequency. This convolutional kernel does not simply slide along the time axis, but rather moves along K... T ×K F The rectangle of size is used for local scanning on the time-frequency graph, where K... T K is the kernel size in the time direction. F denoted as C1, where C1 is the kernel size in the frequency direction. Through this operation, the convolutional layer can simultaneously capture the dynamic changes of speech within a local time window (such as the start and end of syllables) and the energy distribution within a local frequency band (such as formant regions), generating a set of high-dimensional initial time-frequency features.

[0048] Subsequently, the initial time-frequency feature C1 is fed into a downsampling layer to reduce the spatiotemporal resolution of the features, enhance abstraction capabilities, and reduce computational burden. The downsampling layer compresses the initial time-frequency feature C1 using convolution or max-pooling operations with a stride greater than 1 in the second preset dimension, namely the time and frequency dimensions. The relevant formulas are as follows:

[0049]

[0050] in, The initial time-frequency characteristics output from the upper layer have the following shape: ; For convolution kernel; Used for simultaneous downsampling in both time and frequency directions; As a bias term, for example, downsampling with a step size of 2 in the time direction can halve the number of time frames, effectively capturing speech structures with longer periods; downsampling with a step size of 2 in the frequency direction can ignore excessively subtle frequency band changes, preserving the Mel frequency band backbone information that is sensitive to human hearing. After downsampling, the initial time-frequency features are compressed into time-frequency features C2.

[0051] Next, the time-frequency feature C2 is input to the attention layer, a key module for semantic focusing and relevance modeling. The attention layer first decomposes C2 into multiple independent time-frequency units along the time and frequency dimensions. Each time-frequency unit corresponds to a feature vector at position (t, f), where t refers to the frame number and f refers to the frequency band position. Subsequently, the attention layer generates a query vector, key vector, and value vector for each time-frequency unit to perform relevance analysis, calculate the dot product attention weights between all time-frequency units, and establish global associations between them. For example, when a time-frequency unit exhibits a high-frequency energy peak at time frame = 50 and frequency = 18, the attention layer can calculate the correlation between this point and all other units in the preceding and following frames and in the upper and lower frequency bands, determining whether it has semantic consistency with other time-frequency units. If it is found that the energy change of this time-frequency unit and the energy change at frame t = 53 and frequency band f = 22 exhibit a stable evolution in time and a resonant distribution in the spectrum, the attention weights will increase the connection strength between these two time-frequency units, forming a cross-temporal semantic anchor path. The establishment of this association enables the multi-task joint model to overcome noise interference and identify the non-local, structured semantic patterns that truly belong to the wake word. For example, even under strong motor noise masking, where energy fluctuations in certain key frequency bands are severely attenuated, the attention mechanism can still infer the formant peaks of the wake word that should exist at that location through semantic consistency, thereby enhancing its response in the feature space.

[0052] After obtaining all time-frequency units and their interrelationships, average pooling is performed on the entire feature map. Average pooling involves fusing the results of each time-frequency unit and its correlation analysis, and then averaging them along the spatial dimensions (time and frequency). Thus, the encoder, through a collaborative process of extracting the initial time-frequency structure via convolutional layers, compressing dimensions and enhancing abstraction via downsampling layers, establishing semantic relationships across time-frequency units via attention layers, and generating the final semantic time-frequency features via average pooling, solves the problem of speech semantic information being masked under strong self-noise. This enables the subsequent wake-up classifier to achieve high-precision, low-false-wake-up real-time wake-up detection based on semantically clear speech time-frequency features.

[0053] Optionally, the speech time-frequency features include: multi-frame time-frequency segmented features; the wake-up classifier includes: a temporal modeling layer, a frame-level classification layer, and a time aggregation layer; the wake-up classifier performs probability prediction based on the speech time-frequency features to obtain the wake-up probability, including: inputting the multi-frame time-frequency segmented features into the temporal modeling layer, where the temporal modeling layer processes the multi-frame time-frequency segmented features to obtain the hidden state features corresponding to each of the multi-frame time-frequency segmented features; inputting each hidden state feature into the frame-level classification layer, where the frame-level classification layer classifies and discriminates each hidden state feature to obtain multiple frame-level wake-up probabilities, and inputting the multiple frame-level wake-up probabilities into the time aggregation layer; and the time aggregation layer aggregates the multiple frame-level wake-up probabilities according to a set time window to obtain the wake-up probability.

[0054] Optionally, the multi-frame time-frequency segmentation features are input to the temporal modeling layer, which processes the multi-frame time-frequency segmentation features to obtain the hidden state features corresponding to each of the multi-frame time-frequency segmentation features. This includes: for any time-frequency segmentation feature t in the multi-frame time-frequency segmentation features, inputting any time-frequency segmentation feature t into a first expression of the temporal modeling layer, and determining the hidden state feature corresponding to any time-frequency segmentation feature t based on the first expression; the first expression is: ;in, The hidden state feature corresponding to any time-frequency frame feature t; Refers to any time-frequency frame feature t; Refers to one-dimensional temporal convolution operation; Refers to the activation function.

[0055] Optionally, the hidden state features are classified and discriminated based on the frame-level classification layer to obtain multiple frame-level wake-up probabilities, including: for any hidden state feature among the hidden state features. , take any hidden state feature The second expression, input to the frame-level classification layer, is used to evaluate the features of any hidden state based on the second expression. Perform classification and discrimination to obtain the features of any hidden state. The frame-level wake-up probability; the second expression is:

[0056] ;

[0057] in, The frame-level wake-up probability refers to any hidden state feature t; 1 refers to the activation function. Refers to the weight matrix; This refers to the bias parameter.

[0058] Optionally, if the wake-up probability determines that the preset wake-up conditions are met, the target device is wake-up processed, including: comparing the wake-up probability with a preset wake-up probability threshold to obtain a comparison result; and if the comparison result indicates that the wake-up probability is greater than the wake-up probability threshold, the target device is wake-up processed.

[0059] For example, the wake-up classifier consists of three levels: a temporal modeling layer, a frame-level classification layer, and a temporal aggregation layer. Its design goal is to achieve high-precision, low-latency, and interference-resistant recognition of target wake-up words based solely on the time-frequency features of the encoder output, without relying on external noise reduction modules. The time-frequency features output by the encoder are a feature sequence comprising multiple frames of time-frequency frame features. This feature sequence represents the dynamic semantic evolution of speech within a continuous time window, where each frame implicitly contains a mixture of local acoustic structure and noise interference information. The first level of the wake-up classifier is the temporal modeling layer, which performs contextual modeling on the time-frequency frame features of each frame to capture the continuous acoustic patterns of the wake-up word in the temporal dimension. This layer employs a lightweight temporal convolutional network (TCN) or a one-dimensional dilated convolutional layer structure, effectively expanding the receptive field while avoiding the introduction of excessive inference latency. For each input frame's time-frequency segmented feature t, the temporal modeling layer fuses information from several frames before and after it through convolutional operations, generating a hidden state feature ht that incorporates local context. This hidden state feature contains the semantic dependencies of the speech before and after that point in time. This modeling approach enables the multi-task joint model to recognize the complete acoustic trajectory of the wake word, rather than just an isolated spectral peak.

[0060] Subsequently, the hidden state features h1, h2…hT′ corresponding to all time-frequency frame features are sequentially input into the frame-level classification layer. This layer consists of a fully connected neural network, including linear transformations and a sigmoid activation function. For each hidden state ht, the frame-level classification layer performs the following operations:

[0061]

[0062] in, Let b be the learnable weight vector, and b be the bias term. ( () is the Sigmoid function, output value ∈[0,1] represents the frame-level wake-up probability, indicating the local confidence that the speech signal includes the target wake-up word. This probability is a direct discrimination based on the noise-resistant semantic features extracted by the encoder, and its value reflects whether the current frame highly matches the wake-up word acoustic template learned during the training phase. For example, when the hidden state of a frame contains a clear cooperative pattern of high-frequency formants and low-frequency initiation energy, The energy level will increase significantly; conversely, if the frame contains only random noise energy, then... It is close to 0.

[0063] At this point, a series of consecutive frame-level wake-up probabilities have been obtained. , … Due to transient noise interference in the voice signal (such as mechanical impacts from equipment or fan pulses), single-frame probabilities are prone to spurious fluctuations. Directly triggering wake-up based on a single-frame result would result in numerous false wake-ups. Therefore, a time aggregation layer is introduced as the final decision filtering module. The time aggregation layer receives all frame-level wake-up probabilities and aggregates consecutive frame-level probabilities within a preset time window (e.g., 64ms, corresponding to 8 frames). This window slides along the time axis to ensure continuous responsiveness to real-time voice streams. In this embodiment, the aggregation layer employs a maximum value aggregation mechanism:

[0064]

[0065] Specifically, a valid wake-up is determined as long as the wake-up probability is higher than the background noise level (i.e., p is close to 1) in at least one frame within any time window. This strategy ensures high sensitivity while effectively avoiding response delays caused by averaging, meeting the stringent requirements of edge devices for rapid response.

[0066] Step S206: If the preset wake-up conditions are met based on the wake-up probability, the target device is woken up.

[0067] Steps S202-S206 above involve acquiring real-time speech data from the target device and inputting the speech representation of the real-time speech data into a pre-trained multi-task joint model. The multi-task joint model is trained by a neural network model based on joint tasks, including noise reduction and wake-up. The multi-task joint model then determines the speech time-frequency features of the real-time speech data and performs probability prediction on these features to obtain the wake-up probability. The wake-up probability represents the probability that the real-time speech data includes the target wake-up word. Since the multi-task joint model is trained on the joint noise reduction and wake-up tasks, it possesses robustness and accuracy, effectively improving the accuracy of the wake-up probability. Finally, combining the wake-up probability with the wake-up processing of the target device reduces false wake-ups and improves the accuracy of voice wake-up.

[0068] In an exemplary embodiment, before acquiring real-time speech data from the target device and inputting the speech representation of the real-time speech data into a pre-trained multi-task joint model, the speech wake-up method based on the multi-task joint model further includes: acquiring speech training data and inputting the sample speech representation of the speech training data into a neural network model to be trained; the neural network model includes: an encoder, a denoiser, and a wake-up classifier; extracting features from the sample speech representation based on the encoder to obtain sample speech time-frequency features; inputting the sample speech time-frequency features into the denoiser, performing speech reconstruction processing on the sample speech time-frequency features based on the denoiser to determine a first denoising loss; and inputting the sample speech time-frequency features into the wake-up classifier, performing probability prediction on the sample speech time-frequency features based on the wake-up classifier to determine a wake-up loss; training the neural network model to be trained based on the first denoising loss and the wake-up loss to adjust the encoder parameters of the encoder, the denoiser parameters of the denoiser, and the classifier parameters of the wake-up classifier to obtain a trained multi-task joint model.

[0069] Understandably, speech training data consists of audio samples used to train neural network models. Speech training data can be obtained from various sources, including real-world recordings, simulated, or synthesized speech samples. It can also include sounds under various noise backgrounds to ensure the model's generalization ability and robustness.

[0070] An encoder is a component of a neural network model responsible for transforming input data into a more compact representation or feature vector. In speech processing tasks, the encoder is used to extract key time-frequency features from the input speech training data. These time-frequency features can include basic speech attributes such as pitch, timbre, and rhythm, and form the basis for subsequent noise reduction and classifier decisions. Encoder structures can be diverse, including those based on convolutional neural networks, recurrent neural networks (such as Long Short-Term Memory networks or gated recurrent units), and self-attention mechanisms such as the Transformer.

[0071] The denoiser is another crucial component of a neural network model. It receives the output features of the encoder and transforms them into an intelligible output, such as a speech waveform in a text-to-speech task or a clean speech spectrum used for noise reduction in a voice wake-up task. In the context of voice wake-up, the denoiser's primary task is to analyze the features of the encoder's output to recover a clean speech spectrum as close as possible to the original signal, thereby reducing the impact of noise on subsequent wake-up word recognition. The denoiser can also vary depending on the nature of the task. For example, in denoising scenarios, U-Net (U-shaped network) is a commonly used architecture that preserves high-resolution features through skip connections, helping to recover detailed information. The goal of the denoiser is to output a clean speech spectrum or time-domain waveform, which typically involves generating a spectral mask and then applying it to the spectrum of the input signal to eliminate background noise.

[0072] A wake-up classifier is a model or algorithm specifically designed to identify the presence or absence of wake words. Wake words are specific words that activate a device or service, such as "Xiao X, Xiao X." The task of the wake-up classifier is to locate and identify these words in a continuous audio stream. Wake-up classifiers can be based on various network architectures, such as lightweight convolutional neural networks (CNNs), time-delay neural networks (TDNNs), long short-term memory networks (LSTMs), or self-attention mechanisms like the Transformer. The input to the wake-up classifier is a feature representation processed by the encoder. Based on the input features, the wake-up classifier generates a probability value or score representing the likelihood of a wake word existing in the speech training data. By setting a threshold, if the score exceeds this threshold, the wake word is determined to exist, thereby activating the corresponding post-processing procedure or wake-up service. The determination of the first noise reduction loss and the wake-up loss are the two pillars of multi-task joint learning, working together on the time-frequency features of the sample speech output by the encoder to achieve the ultimate goal of noise reduction without losing semantics and wake-up without being disturbed by noise. In the process of reconstructing speech based on the time-frequency features of sample speech using a denoiser and determining the reconstruction loss, the denoiser can reconstruct a time-spectral representation that is as close as possible to the original clean speech, thereby reducing the masking interference of noise on subsequent wake-up classification. Thus, by combining the functions of the encoder, denoiser, and wake-up classifier, a complete neural network model can be constructed. This model starts from the original speech signal, goes through feature extraction and noise suppression, and finally accurately determines whether a wake-up word exists. This process ensures that smart devices can still respond to user commands in a timely manner under various noise conditions, improving the convenience and intelligence of human-computer interaction.

[0073] In an exemplary embodiment, the noise reduction unit includes: a feature restoration layer, a fusion layer, and an output layer; based on the noise reduction unit, speech reconstruction processing is performed on the time-frequency features of sample speech to determine a first noise reduction loss, including: inputting the time-frequency features of sample speech into the feature restoration layer; performing at least one upsampling process on the time-frequency features of sample speech based on the feature restoration layer to obtain a feature restoration result, and inputting the feature restoration result into the fusion layer; performing fusion processing on the time-frequency features of sample speech and the feature restoration result based on the fusion layer to obtain a fusion processing result, and inputting the fusion processing result into the output layer; outputting the mask parameters corresponding to the fusion processing result from the output layer, and performing speech reconstruction processing based on the mask parameters to determine the first noise reduction loss.

[0074] Understandably, in a multi-task joint model, the denoiser, as a branch module parallel to the wake-up classifier, has the core function of reconstructing noisy speech in the time-frequency domain based on the time-frequency features of the sample speech output by the encoder, thus providing supervision signals for joint training. This denoiser employs a three-level structure: a feature restoration layer, a fusion layer, and an output layer, working collaboratively. The time-frequency features of the sample speech are the mixed information of noise and speech extracted by the encoder from the original noisy speech. These features are fed into the feature restoration layer of the denoiser. The feature restoration layer consists of a series of transposed convolutions or upsampling-convolution combinations, which progressively expand and refine the low-resolution, highly abstract input features, restoring their spatial resolution in the time-frequency dimension. The feature restoration layer performs at least one upsampling process, progressively expanding the feature size in both the time and frequency dimensions through interpolation or transposed convolution operations, gradually bringing the features originally compressed by the encoder closer to the time-frequency resolution of the original speech.

[0075] After processing by the feature reconstruction layer, the obtained feature reconstruction result has a higher resolution, but it is still a high-level semantic mapping and lacks precise alignment with local details in the original input. At this point, the reconstruction result and the sample speech time-frequency features of the original input are fed into the fusion layer. The fusion layer has an element-wise addition or channel concatenation mechanism, and its core function is to semantically complement the high-level reconstructed features with the low-level input features. Specifically, the fusion layer aligns the feature reconstruction result with the sample speech time-frequency features output by the encoder in the spatial dimension and then fuses them to form the fused result.

[0076] The fusion result is then fed into the output layer. The output layer is a lightweight convolutional network, consisting of a single convolutional or fully connected layer. Its output is a mask parameter, which is used for speech reconstruction to determine the first denoising loss. The mask parameter is a two-dimensional matrix with the same size as the input time-frequency features, where each element's value is between 0 and 1, representing the proportion of original speech energy to be retained at the corresponding time frame and frequency point. Thus, the denoiser constructs a self-supervised mechanism that coordinates with the wake-up task, using a three-level structure: a feature restoration layer for semantic upscaling, a fusion layer for detail complementarity, and an output layer for generating a semantic mask. It treats denoising as an intermediate task, driving the encoder to learn semantically discriminative noise-resistant features, rather than independently performing speech enhancement. Through the reconstruction process guided by the mask parameter, the denoising loss not only provides a supervisory signal but also acts as a regularizing force constraining the feature space structure, enabling the model to retain the key acoustic fingerprint of the wake-up word even at extremely low signal-to-noise ratios, thereby laying a solid feature foundation for lightweight deployment on the edge.

[0077] In some embodiments, the time-frequency features of the sample speech and the feature reconstruction results are fused based on the fusion layer to obtain a fusion processing result, including: inputting the time-frequency features of the sample speech and the feature reconstruction results into a third expression of the fusion layer, and performing fusion processing on the time-frequency features of the sample speech and the feature reconstruction results based on the third expression to obtain a fusion processing result; the third expression is: ;in, Refers to the feature reconstruction result, The jump-connection mechanism, in which, = , Refers to the time-frequency features of the sample speech; Transposed convolution; Refers to the result of the fusion process; For activation functions; This refers to normalization processing.

[0078] It should be noted that, As a high-level reconstruction feature, its resolution is insufficient to align with the original input. Therefore, it must undergo upsampling to interpolate and enlarge it in both time and frequency dimensions, so that its spatial dimensions match those of the corresponding layer features in the encoder. Matching. This process ensures the feasibility of subsequent element-level fusion, avoiding computational limitations caused by dimensionality inconsistencies. The upsampled features are fed into a transposed convolutional layer. The transposed convolutional layer acts as a learnable feature adjuster, finely correcting artifacts, blurring, or structural distortions that may exist after upsampling. By learning a set of trainable convolutional kernels, the transposed convolution strengthens the structure of the upsampled features, making their semantic boundaries clearer and their energy distribution more consistent with the spectral characteristics of real speech. For example, if the energy of a certain frequency band appears diffuse after upsampling, the transposed convolution can learn a specific weight distribution to focus it into a concentrated, sharp formant shape, thus more closely resembling the acoustic essence of real speech. Subsequently, the output of the transposed convolution is combined with the jump features from the encoder. Element-wise addition is performed to semantically guide the lower-level features in the high-level reconstruction without overwriting the original details. This addition result is then passed through the GELU (Gaussian Error Linear Unit) activation function. Compared to ReLU, this non-linear activation function is smoother and better at gradient propagation, preserving low-energy speech components more delicately and avoiding speech distortion caused by hard thresholding. This is especially crucial for preserving weak speech components in extremely low signal-to-noise ratio environments. Finally, the entire result is fed into a Norm (normalization) layer. This layer uses layer-level or batch normalization to stabilize the training process, accelerate convergence, and eliminate training fluctuations caused by differences in feature scales between different samples. The normalized output is shown below. It becomes the final output of the decoding layer, serves as the input for the next layer's fusion or output, and provides high-quality intermediate features for subsequent mask prediction.

[0079] In the above embodiments, the jump-connection structure constructed through the fusion layer achieves structured preservation and semantic guidance of key acoustic features of the wake-up word during the noise reduction process, solving the defect of excessive smoothing leading to wake-up failure in traditional noise reduction methods. This mechanism enables the model to internalize noise reduction as a goal for recognition during the training phase, and even if the noise reduction branch is removed during the inference phase, it can still output a highly robust speech representation, thereby achieving lightweight, high recall, and low false wake-up capabilities on the device side.

[0080] In some embodiments, the mask parameters include: amplitude mask values; the mask parameters corresponding to the fusion processing result output by the output layer, and the speech reconstruction processing based on the mask parameters to determine the first noise reduction loss, include: inputting the fusion processing result to the fourth expression of the output layer, processing the fusion processing result by the fourth expression to obtain the amplitude mask value; wherein, the fourth expression is: ;in, Refers to the amplitude mask value. Activation function; Refers to one-dimensional temporal convolution operation; The result of the fusion process is used; the amplitude mask value is input into the fifth expression, and the first clean speech spectrum is reconstructed from the fifth expression; the fifth expression is: ;in, Refers to the first pure speech spectrum; The noise amplitude spectrum is used; the first noise reduction loss is determined based on the first clean speech spectrum and the set amplitude spectrum label. It should be noted that in this embodiment, the first clean speech spectrum is generated by the output layer and compared with the set amplitude spectrum label to calculate the first noise reduction loss. This constructs a noise reduction mechanism that is co-optimized with the wake-up task. It uses convolution and activation functions to directly generate amplitude mask values, which is lightweight, computationally efficient, and suitable for edge deployment; and the original phase is preserved through amplitude spectrum multiplication reconstruction, thus avoiding speech distortion.

[0081] Optionally, the mask parameters include: a complex mask matrix; the mask parameters corresponding to the fusion processing result output by the output layer are used, and speech reconstruction processing is performed based on the mask parameters to determine the first noise reduction loss, including: performing complex spectrum reconstruction on the fusion processing result based on the output layer to obtain a complex mask matrix; converting the format of the complex mask matrix to obtain a complex domain, and reconstructing a second clean speech spectrum based on the complex domain; determining the first noise reduction loss based on the second clean speech spectrum and a set amplitude spectrum label. By introducing the mechanism of complex mask matrix, complex domain reconstruction, and complex spectrum loss, the limitation of traditional amplitude masking that only adjusts energy and not phase under extremely low signal-to-noise ratio is overcome, and joint optimization reconstruction of speech amplitude and phase is achieved. Without increasing computational complexity, the naturalness and wake-up separability of the reconstructed speech are significantly improved, which is a core technological breakthrough for achieving highly robust voice wake-up in extreme noise scenarios (such as the operation of a robot vacuum cleaner or air conditioner noise).

[0082] Optionally, the sample speech time-frequency features include: multi-frame sample time-frequency segmented features; the wake-up classifier includes: a temporal modeling layer, a frame-level classification layer, and a temporal aggregation layer; inputting the sample speech time-frequency features into the wake-up classifier, and performing probability prediction on the sample speech time-frequency features based on the wake-up classifier to determine the wake-up loss, includes: inputting the multi-frame sample time-frequency segmented features into the temporal modeling layer, where the temporal modeling layer processes the multi-frame sample time-frequency segmented features respectively to obtain sample hidden state features corresponding to the multi-frame sample time-frequency segmented features; inputting each sample hidden state feature into the frame-level classification layer, where the frame-level classification layer classifies and discriminates each sample hidden state feature to obtain multiple sample frame-level wake-up probabilities, and inputting the multiple sample frame-level wake-up probabilities into the temporal aggregation layer; aggregating the multiple sample frame-level wake-up probabilities according to a set time window based on the temporal aggregation layer to obtain the sample wake-up probability; and performing cross-entropy processing on the sample wake-up probability and the preset label wake-up probability to obtain the wake-up loss. In other words, by establishing long-term dependencies through a temporal modeling layer, achieving local discrimination through a frame-level classification layer, completing stable decision-making through a temporal aggregation layer, and guiding global optimization with cross-entropy loss, an end-to-end, interference-resistant, and low-latency wake-up probability prediction system is constructed. The temporal modeling layer enables the model to understand the structure of wake-up words, rather than relying on local energy peaks; the temporal aggregation layer effectively suppresses false wake-ups while ensuring low latency through a maximum value strategy; the wake-up loss directly affects the aggregated sample-level probability, ensuring that the model training objective is completely consistent with the actual deployment objective. This mechanism enables the wake-up classifier to maintain stable performance with high recall and low false wake-ups even in extremely noisy environments.

[0083] Optionally, the neural network model to be trained is trained based on the first noise reduction loss and the wake-up loss to adjust the encoder parameters of the encoder, the noise reduction parameters of the denoiser, and the classifier parameters of the wake-up classifier, thereby obtaining a trained multi-task joint model. This includes: obtaining a second noise reduction loss; wherein the second noise reduction loss is output by the speech training data input into a pre-trained speech noise reduction model; fusing the first noise reduction loss, the second noise reduction loss, and the wake-up loss to obtain a target loss; adjusting the encoder parameters of the encoder, the noise reduction parameters of the denoiser, and the classifier parameters of the wake-up classifier according to the target loss to train the neural network model until the training stopping condition is met, thereby obtaining a trained initial multi-task joint model; and pruning the initial multi-task joint model to obtain a multi-task joint model.

[0084] It should be noted that when performing voice wake-up based on a multi-task joint model, in addition to the first noise reduction loss and the wake-up loss, a second noise reduction loss can be introduced, resulting in a target loss through these three losses. When determining the target loss, different weights can be assigned to the three losses; the specific weight values ​​can be set based on the importance of each loss to the voice wake-up task or historical experience. Furthermore, the encoder parameters of the encoder, the noise reduction parameters of the denoiser, and the classifier parameters of the wake-up classifier are adjusted using the target loss to train the neural network model until the training stopping condition is met, thus obtaining a trained multi-task joint model.

[0085] In the above embodiments, by introducing three types of losses and fusing them to determine the target loss, and then conducting model training based on the target loss, the robustness and accuracy of the multi-task joint model are effectively improved, laying the foundation for the accuracy of voice wake-up in subsequent practical applications.

[0086] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above method, the following description, in conjunction with embodiments, illustrates the process, but is not intended to limit the technical solutions of the embodiments of this application:

[0087] Currently, voice wake-up based on multi-task joint models is typically limited by three major bottlenecks: single-task training, lack of data annotation, and separation of noise reduction and wake-up tasks. This makes it difficult to achieve stable wake-up in real-world home appliance environments, and commonly results in low recall, high false wake-up rate, large edge latency, and poor generalization ability. Current home appliances (such as robot vacuums, vacuum cleaners, kitchen range hoods, and air conditioners) face multiple challenges in voice interaction scenarios with strong mechanical self-noise, including low signal-to-noise ratio, wide noise spectrum with high overlap with speech frequencies, close proximity of microphones and noise sources, and target speech originating from far-field sources. Therefore, this application proposes a voice wake-up method based on a multi-task joint model for high-noise, low-signal-to-noise ratio scenarios, referencing... Figure 3 The following is one of the flowcharts of the voice wake-up method based on a multi-task joint model according to this application:

[0088] This application focuses on three main aspects: multi-level data mixing, multi-task wake-up learning, and edge deployment. In multi-level data mixing, the speech training data employs a multi-level mixed data construction mechanism, integrating real device recordings, controllable semi-simulation data (precisely controlling distance, angle, and SNR (Signal-to-Noise Ratio)), and acoustic simulation data (virtual sound field). This constructs a high-fidelity, labelable, and highly generalizable training set covering multiple devices, noise sources, and spatial locations, fundamentally solving the industry challenges of scarce real-world data and difficult annotation.

[0089] refer to Figure 4 The diagram shown is the second flowchart of the voice wake-up method based on a multi-task joint model provided in this application: Multi-level mixed data is obtained by mixing multiple levels of data, originating from several different data layers, including: a real-recorded data layer, a semi-simulation data layer, and an acoustic simulation data layer. The real-recorded data layer uses an artificial mouth or human voice at different distances, angles, relative positions, and operating states on the device to play the wake-up dataset. The dataset is calibrated to be distributed according to the dynamic range of a normal human voice, with white noise superimposed at the beginning for time alignment. The semi-simulation data layer is obtained by mixing speech and noise, using joint probability and conditional probability to simulate complex acoustic conditions in a real environment. The acoustic simulation data layer is obtained using a virtual impact response generated based on acoustic geometric modeling.

[0090] In multi-task wake-up learning, an end-to-end joint learning architecture of encoder + denoiser + wake-up classifier is proposed. By unifying the feature extraction path, speech reconstruction (U-Net denoising branch) and wake-up discrimination (lightweight classification branch) are simultaneously optimized. A multi-task joint optimization mechanism of first denoising loss, wake-up loss, and second denoising loss is introduced to achieve a co-evolution where denoising serves wake-up and wake-up guides denoising, effectively suppressing feature distortion and task conflicts. (Reference) Figure 5 The following is the third flowchart of the voice wake-up method based on a multi-task joint model provided in this application:

[0091] The speech training data comes from a multi-level mixed data module, containing noise data and corresponding labels and metadata. The data is divided into two parts, provided to the model according to the set hyperparameters. The first part is real data, and the second part is simulated data. Both real and simulated data are input to a shared encoder (i.e., the encoder) for processing. The pre-trained speech denoising model only needs to be input with real data, and the pre-trained speech denoising model determines the second denoising loss based on the real data.

[0092] The shared encoder proposed in this application achieves synergistic optimization of engineering and theory in three aspects: low parameter count, high robustness, and strong time-frequency modeling capability. It possesses extremely high technical feasibility and practical deployment feasibility, which can be probabilistically evaluated from the following dimensions: The encoder employs 2D convolution (time × frequency joint convolution), which can simultaneously capture the temporal dynamics of speech (such as the "small → X" phoneme sequence) and spectral structure (such as formant morphology). Its ability to preserve speech semantics at low signal-to-noise ratios is significantly improved compared to 1D schemes, with measured feature separability (inter-class distance) improved by more than 40%, and the model exhibits stronger resistance to noise masking. Furthermore, combining 2D convolution with stride downsampling and depth-separable structures enables efficient dimensionality reduction, with the parameter count controlled to <150K. The encoder output is a compact time-frequency embedding, which can be seamlessly integrated into the U-Net denoising branch and wake-up classifier. The feature dimension is consistent with the time step, eliminating the need for interpolation or resampling, enabling end-to-end joint training, avoiding feature alignment errors, and achieving extremely high task collaboration efficiency.

[0093] The denoising decoder (i.e., the denoiser) receives the output of the shared encoder, processes it to determine the first denoising loss, and uses the second denoising loss output by the speech denoising model to jointly determine the first loss. The denoiser is based on the U-Net encoder-decoder architecture, specifically designed for high-fidelity speech time-frequency reconstruction in environments with extremely low signal-to-noise ratios and home appliance self-noise. Its core objective is to generate high-quality speech spectrum priors through end-to-end joint training, providing semantically preserved enhanced features for wake-up classification tasks. U-Net employs a symmetric encoder-denoiser structure. The encoder progressively downsamples through multiple layers of 2D convolutions to extract multi-level time-frequency features. Each layer of features is activated by GELU, normalized by Norm, and then saved as a jump path, ensuring that high-frequency detail information is not lost in the deep network. The denoiser uses an upsampling and jump-fusing mechanism, achieving feature upsampling through transposed convolution or interpolation / convolution, and then adding or concatenating the features at the same scale as the encoder element-wise, significantly improving the detail fidelity of the spectrogram reconstruction. In terms of output format, two robust reconstruction modes are supported to adapt to different training objectives and deployment constraints: (A) Amplitude mask output: A single-channel output spectral mask is multiplied with the input amplitude spectrum to achieve non-negative amplitude enhancement. The structure is simple, training is stable, and it is suitable for low-computing-power edge deployment. (B) Complex mask output: The output consists of dual-channel real and imaginary parts, or a complex mask is directly generated. The amplitude and phase are jointly reconstructed by multiplying the input complex spectrum through complex domain multiplication, which significantly improves the naturalness and intelligibility of speech and is suitable for high-precision reconstruction scenarios. Finally, the output layer uses an activation function to effectively constrain the range of complex components and avoid training divergence. This structure preserves the original local spectral structure through a jump-connection mechanism, models the noise-speech nonlinear relationship through time-frequency joint convolution, and combines the complex domain modeling capability. Under the premise of no clean speech labels, it can effectively recover the key frequency bands of wake words that are masked by strong mechanical noise, providing the wake classifier with feature inputs that have high semantic integrity and low noise interference.

[0094] The wake-up classifier employs a lightweight two-level architecture of temporal modeling and dynamic aggregation: Temporal Modeling Layer: Taking frame-level features from the encoder as input, a temporal dependency modeling module is constructed using a one-dimensional stacked or dilated convolutional temporal convolutional network. Pooling and Classification Layer: Global average pooling is applied to the temporal features to obtain a global representation, and a single-layer linear transformation is used to output the frame-level wake-up probability, thereby determining the second loss (i.e., the wake-up loss).

[0095] Finally, the first loss and the second loss are combined to determine the target loss. The neural network model is then trained based on the target loss to obtain the trained multi-task joint model.

[0096] In edge deployment, a lightweight inference optimization scheme is designed for edge use. This scheme employs hybrid quantization, structured pruning, and layer-freezing fine-tuning techniques to compress the model to less than 1.5M parameters without sacrificing recognition accuracy, achieving real-time wake-up. (Reference) Figure 6 The diagram shown is the fourth flowchart of the voice wake-up method based on a multi-task joint model provided in this application. Specifically, it is a flowchart for end-side deployment. End-side deployment mainly includes: end-side acquisition module, end-side preprocessing module, feature extraction module, processor, shared encoder, wake-up classifier, decision module, and other modules.

[0097] The edge-side acquisition module is responsible for acquiring the user's voice signal from the acoustic field and converting it into a digital signal, which serves as the input for subsequent feature extraction and embedding into the model. The edge-side preprocessing module integrates pre-emphasis, framing, windowing (Hamming window), and adaptive AGC (Automatic Gain Control) to dynamically compensate for far-field speech energy attenuation and environmental volume fluctuations, significantly improving the perceived stability of the signal-to-noise ratio. The feature extraction module uses MFCC (Mel-Frequency Cepstral Coefficients) as input features to extract MFCC features that characterize the acoustic feature distribution. At the model deployment level, the training and inference structures are decoupled on the processor: the encoder + dual-branch (denoiser, wake-up classifier) ​​model in the training phase is used, while only the encoder and wake-up classifier are retained during deployment, discarding the computationally redundant denoising branch. The model size is compressed to <1.5MB, the inference latency is <25ms, and the power consumption is <15mW.

[0098] The final decision module employs a dual guarantee of dynamic adaptive threshold mechanism and multi-frame stability decision: the threshold is adjusted in real time according to the variance of environmental noise to avoid false wake-ups due to increased noise. It uses the proportion of consecutive frames within a sliding window for judgment, effectively filtering out transient interference (such as knocking or discharging), reducing the false wake-up rate by more than 68%.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0100] This embodiment also provides a voice wake-up device based on a multi-task joint model, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0101] Figure 7 This is a structural block diagram of a voice wake-up device based on a multi-task joint model according to an embodiment of this application. The device includes:

[0102] The acquisition module 72 is used to acquire real-time speech data of the target device and input the speech representation of the real-time speech data into a pre-trained multi-task joint model; wherein, the multi-task joint model is obtained by training a neural network model based on joint tasks; the joint tasks include: noise reduction task and wake-up task;

[0103] The prediction module 74 is used to determine the speech time-frequency features corresponding to the speech representation based on the multi-task joint model, and to perform probability prediction on the speech time-frequency features to obtain the wake-up probability; wherein, the wake-up probability is used to characterize the probability that the real-time speech data includes the target wake-up word;

[0104] The wake-up module 76 is used to wake up the target device when the preset wake-up conditions are met based on the wake-up probability.

[0105] The aforementioned device acquires real-time speech data from the target device and inputs the speech representation of the real-time speech data into a pre-trained multi-task joint model. The multi-task joint model is obtained by training a neural network model based on joint tasks, including a noise reduction task and a wake-up task. Then, based on the multi-task joint model, the speech time-frequency features of the real-time speech data can be determined, and further probability prediction can be performed on the speech time-frequency features to obtain the wake-up probability. The wake-up probability is used to characterize the probability that the real-time speech data includes the target wake-up word. Since the multi-task joint model is a model obtained by jointly training the noise reduction and wake-up tasks, it has a certain degree of robustness and accuracy, which can effectively improve the accuracy of the wake-up probability. Finally, the wake-up process of the target device is carried out in combination with the wake-up probability, which can reduce false wake-ups of the target device and improve the accuracy of voice wake-up.

[0106] In an exemplary embodiment, the multi-task joint model includes: an encoder and a wake-up classifier; a prediction module 74 is further configured to determine the time-frequency features corresponding to the speech representation based on the encoder, and perform correlation analysis on the multiple time-frequency units included in the time-frequency features, and obtain speech time-frequency features based on the correlation analysis results; input the speech time-frequency features to the wake-up classifier, and the wake-up classifier performs probability prediction based on the speech time-frequency features to obtain the wake-up probability.

[0107] In an exemplary embodiment, the encoder includes a convolutional layer, a downsampling layer, and an attention layer; the prediction module 74 is further configured to input the speech representation into the convolutional layer, whereby the convolutional layer convolves the speech representation according to a first preset dimension to obtain initial time-frequency features of the speech representation, and inputs the initial time-frequency features into the downsampling layer; the downsampling layer performs compression processing on the initial time-frequency features with a preset stride in a second preset dimension to obtain the time-frequency features, and inputs the time-frequency features into the attention layer, wherein both the first preset dimension and the second preset dimension include a time dimension and a frequency dimension; the attention layer decomposes the time-frequency features to obtain multiple time-frequency units, and establishes the correlation relationship between the multiple time-frequency units to obtain a correlation analysis result; and average pooling is performed on the multiple time-frequency units and the correlation analysis result to obtain speech time-frequency features.

[0108] In an exemplary embodiment, the speech time-frequency features include: multi-frame time-frequency segmented features; the wake-up classifier includes: a temporal modeling layer, a frame-level classification layer, and a time aggregation layer; the prediction module 74 is further configured to input the multi-frame time-frequency segmented features into the temporal modeling layer, whereby the temporal modeling layer processes the multi-frame time-frequency segmented features to obtain hidden state features corresponding to each of the multi-frame time-frequency segmented features; input each hidden state feature into the frame-level classification layer, whereby the frame-level classification layer classifies and discriminates each hidden state feature to obtain multiple frame-level wake-up probabilities, and input the multiple frame-level wake-up probabilities into the time aggregation layer; and aggregate the multiple frame-level wake-up probabilities according to a set time window to obtain a wake-up probability.

[0109] In an exemplary embodiment, the prediction module 74 is further configured to, for any time-frequency frame feature t in the multi-frame time-frequency frame features, input the any time-frequency frame feature t into a first expression of the temporal modeling layer, and determine the hidden state feature corresponding to the any time-frequency frame feature t based on the first expression; the first expression is: ;in, Refers to the hidden state feature corresponding to any time-frequency frame feature t; Refers to any of the aforementioned time-frequency frame features t; Refers to one-dimensional temporal convolution operation; Refers to the activation function.

[0110] In an exemplary embodiment, the prediction module 74 is further configured to target any one of the hidden state features. , and the hidden state feature The second expression input to the frame-level classification layer is used to evaluate the features of any hidden state based on the second expression. Perform classification and discrimination to obtain the features of any hidden state. The frame-level wake-up probability; the second expression is: ;in, The frame-level wake-up probability of any hidden state feature t; 1 refers to the activation function. Refers to the weight matrix; This refers to the bias parameter.

[0111] In an exemplary embodiment, the wake-up module 76 is further configured to compare the wake-up probability with a preset wake-up probability threshold to obtain a comparison result; and to perform wake-up processing on the target device when the comparison result indicates that the wake-up probability is greater than the wake-up probability threshold.

[0112] In an exemplary embodiment, the apparatus further includes a training module; the training module is configured to acquire speech training data and input sample speech representations of the speech training data into a neural network model to be trained; the neural network model includes: an encoder, a denoiser, and a wake-up classifier; based on the encoder, feature extraction is performed on the sample speech representations to obtain sample speech time-frequency features; the sample speech time-frequency features are input into the denoiser, and speech reconstruction processing is performed on the sample speech time-frequency features based on the denoiser to determine a first denoising loss; and the sample speech time-frequency features are input into the wake-up classifier, and probability prediction is performed on the sample speech time-frequency features based on the wake-up classifier to determine a wake-up loss; the neural network model to be trained is trained based on the first denoising loss and the wake-up loss to adjust the encoder parameters of the encoder, the denoiser parameters of the denoiser, and the classifier parameters of the wake-up classifier to obtain a trained multi-task joint model.

[0113] In an exemplary embodiment, the noise reduction device includes: a feature restoration layer, a fusion layer, and an output layer; the training module is further configured to input the sample speech time-frequency features into the feature restoration layer, perform at least one upsampling process on the sample speech time-frequency features based on the feature restoration layer to obtain a feature restoration result, and input the feature restoration result into the fusion layer; perform fusion processing on the sample speech time-frequency features and the feature restoration result based on the fusion layer to obtain a fusion processing result, and input the fusion processing result into the output layer; output the mask parameters corresponding to the fusion processing result from the output layer, and perform speech reconstruction processing based on the mask parameters to determine a first noise reduction loss.

[0114] In an exemplary embodiment, the training module is further configured to input the sample speech time-frequency features and the feature reconstruction result into the third expression of the fusion layer, and perform fusion processing on the sample speech time-frequency features and the feature reconstruction result based on the third expression to obtain a fusion processing result; the third expression is: ;in, Refers to the feature reconstruction result, The jump-connection mechanism, in which, = , Refers to the time-frequency features of the sample speech; Transposed convolution; Refers to the result of the fusion process.

[0115] In an exemplary embodiment, the mask parameters include: amplitude mask values; the training module is further configured to input the fusion processing result into a fourth expression of the output layer, and process the fusion processing result by the fourth expression to obtain amplitude mask values; wherein, the fourth expression is: ;in, Refers to the amplitude mask value. Activation function; Refers to one-dimensional temporal convolution operation; The result of the fusion processing is used; the amplitude mask value is input into the fifth expression, and the first clean speech spectrum is reconstructed from the fifth expression; the fifth expression is: ;in, Refers to the first pure speech spectrum; The noise amplitude spectrum is used as a reference; the first noise reduction loss is determined based on the first clean speech spectrum and the set amplitude spectrum label.

[0116] In an exemplary embodiment, the mask parameters include: a complex mask matrix; a training module, further configured to perform complex spectrum reconstruction on the fusion processing result based on the output layer to obtain a complex mask matrix; perform format conversion on the complex mask matrix to obtain a complex domain; reconstruct a second clean speech spectrum based on the complex domain; and determine a first noise reduction loss based on the second clean speech spectrum and a set amplitude spectrum label.

[0117] In an exemplary embodiment, the sample speech time-frequency features include: multi-frame sample time-frequency segmented frame features; the wake-up classifier includes: a temporal modeling layer, a frame-level classification layer, and a temporal aggregation layer; the training module is further configured to input the multi-frame sample time-frequency segmented frame features into the temporal modeling layer, and the temporal modeling layer processes the multi-frame sample time-frequency segmented frame features respectively to obtain sample hidden state features corresponding to the multi-frame sample time-frequency segmented frame features respectively; input each sample hidden state feature into the frame-level classification layer, and classify and discriminate each sample hidden state feature based on the frame-level classification layer to obtain multiple sample frame-level wake-up probabilities, and input the multiple sample frame-level wake-up probabilities into the temporal aggregation layer; aggregate the multiple sample frame-level wake-up probabilities according to a set time window based on the temporal aggregation layer to obtain sample wake-up probabilities; and perform cross-entropy processing on the sample wake-up probabilities and preset label wake-up probabilities to obtain wake-up loss.

[0118] In an exemplary embodiment, the training module is further configured to obtain a second noise reduction loss; wherein the second noise reduction loss is output by the speech training data input to a pre-trained speech noise reduction model; the first noise reduction loss, the second noise reduction loss, and the wake-up loss are fused to obtain a target loss; the encoder parameters of the encoder, the noise reduction parameters of the denoiser, and the classifier parameters of the wake-up classifier are adjusted according to the target loss to train the neural network model until the training stopping condition is met, thereby obtaining an initial multi-task joint model that has been trained; and the initial multi-task joint model is pruned to obtain a multi-task joint model.

[0119] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0120] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0121] S1, acquire real-time voice data from the target device, and input the voice representation of the real-time voice data into a pre-trained multi-task joint model; wherein, the multi-task joint model is obtained by training a neural network model based on joint tasks; the joint tasks include: noise reduction task and wake-up task;

[0122] S2, based on the multi-task joint model, determine the speech time-frequency features corresponding to the speech representation, and perform probability prediction on the speech time-frequency features to obtain the wake-up probability; wherein, the wake-up probability is used to characterize the probability that the real-time speech data includes the target wake-up word;

[0123] S3, if the preset wake-up conditions are met based on the wake-up probability, the target device is woken up.

[0124] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0125] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0126] Embodiments of this application also provide an electronic device, such as... Figure 8 As shown, the electronic device includes a memory 802 and a processor 804. The memory 802 stores a computer program, and the processor 804 is configured to execute the steps in any of the above method embodiments via the computer program.

[0127] Optionally, in this embodiment, the processor 804 can be configured to perform the following steps via a computer program:

[0128] S1, acquire real-time voice data from the target device, and input the voice representation of the real-time voice data into a pre-trained multi-task joint model; wherein, the multi-task joint model is obtained by training a neural network model based on joint tasks; the joint tasks include: noise reduction task and wake-up task;

[0129] S2, based on the multi-task joint model, determine the speech time-frequency features corresponding to the speech representation, and perform probability prediction on the speech time-frequency features to obtain the wake-up probability; wherein, the wake-up probability is used to characterize the probability that the real-time speech data includes the target wake-up word;

[0130] S3, if the preset wake-up conditions are met based on the wake-up probability, the target device is woken up.

[0131] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0132] Alternatively, as those skilled in the art will understand, Figure 8 The structure shown is for illustrative purposes only. Figure 8 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 8 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 8 The different configurations shown.

[0133] The memory 802 can be used to store software programs and modules, such as the program instructions / modules corresponding to the voice wake-up method and device based on the multi-task joint model in this embodiment. The processor 804 executes various functional applications and data processing by running the software programs and modules stored in the memory 802, thereby realizing the aforementioned voice wake-up method based on the multi-task joint model. The memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 802 may further include memory remotely located relative to the processor 804, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 802 may be used, but is not limited to, to store information such as system configuration files. As an example, such as... Figure 8 As shown, the memory 802 may include, but is not limited to, the acquisition module 72, prediction module 74, and wake-up module 76 in the voice wake-up device based on the multi-task joint model. Furthermore, it may include, but is not limited to, other module units in the voice wake-up device based on the multi-task joint model, which will not be elaborated further in this example.

[0134] Optionally, the transmission device 806 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 806 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 806 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0135] In addition, the above-mentioned electronic device also includes: a display 808; and a connection bus 810 for connecting the various module components in the above-mentioned electronic device.

[0136] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0137] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0138] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0139] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0140] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0141] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A voice wake-up method based on a multi-task joint model, characterized in that, include: Acquire real-time speech data from the target device, and input the speech representation of the real-time speech data into a pre-trained multi-task joint model; wherein, the multi-task joint model is obtained by training a neural network model based on joint tasks; the joint tasks include: noise reduction task and wake-up task; Based on the multi-task joint model, the speech time-frequency features corresponding to the speech representation are determined, and the speech time-frequency features are probabilistically predicted to obtain the wake-up probability; wherein, the wake-up probability is used to characterize the probability that the real-time speech data includes the target wake-up word; If the preset wake-up conditions are met based on the wake-up probability, the target device is woken up.

2. The method according to claim 1, characterized in that, The multi-task joint model includes: an encoder and a wake-up classifier; based on the multi-task joint model, the speech time-frequency features corresponding to the speech representation are determined, and the wake-up probability is obtained by probabilistic prediction of the speech time-frequency features, including: Based on the encoder, the time-frequency features corresponding to the speech representation are determined, and correlation analysis is performed on multiple time-frequency units included in the time-frequency features. The speech time-frequency features are obtained based on the correlation analysis results. The speech time-frequency features are input into the wake-up classifier, which then performs probability prediction based on the speech time-frequency features to obtain the wake-up probability.

3. The method according to claim 2, characterized in that, The encoder includes a convolutional layer, a downsampling layer, and an attention layer. Based on the encoder, the time-frequency features corresponding to the speech representation are determined, and correlation analysis is performed on multiple time-frequency units included in the time-frequency features. The speech time-frequency features are obtained based on the correlation analysis results, including: The speech representation is input into the convolutional layer, which performs convolution on the speech representation according to a first preset dimension to obtain the initial time-frequency features of the speech representation, and then inputs the initial time-frequency features into the downsampling layer. The downsampling layer performs a preset step compression process on the initial time-frequency features in the second preset dimension to obtain the time-frequency features, and then inputs the time-frequency features into the attention layer. Both the first preset dimension and the second preset dimension include: a time dimension and a frequency dimension. The time-frequency features are split based on the attention layer to obtain multiple time-frequency units, and the correlation between the multiple time-frequency units is established to obtain the correlation analysis results. The speech time-frequency features are obtained by performing average pooling on the multiple time-frequency units and the correlation analysis results.

4. The method according to claim 2, characterized in that, The speech time-frequency features include: multi-frame time-frequency segmentation features; the wake-up classifier includes: a temporal modeling layer, a frame-level classification layer, and a time aggregation layer; the wake-up classifier performs probability prediction based on the speech time-frequency features to obtain the wake-up probability, including: The multi-frame time-frequency segmentation features are input into the temporal modeling layer, and the temporal modeling layer processes the multi-frame time-frequency segmentation features respectively to obtain the hidden state features corresponding to the multi-frame time-frequency segmentation features. Each of the hidden state features is input to the frame-level classification layer, and each of the hidden state features is classified and judged based on the frame-level classification layer to obtain multiple frame-level wake-up probabilities. The multiple frame-level wake-up probabilities are then input to the time aggregation layer. The time aggregation layer aggregates the multiple frame-level wake-up probabilities according to a set time window to obtain the wake-up probability.

5. The method according to claim 4, characterized in that, The multi-frame time-frequency segmentation features are input into the temporal modeling layer, which processes each feature to obtain hidden state features corresponding to the multi-frame time-frequency segmentation features, including: For any time-frequency frame segmentation feature t in the multi-frame time-frequency frame segmentation features, the any time-frequency frame segmentation feature t is input into the first expression of the time-series modeling layer, and the hidden state feature corresponding to the any time-frequency frame segmentation feature t is determined based on the first expression; the first expression is: ； in, Refers to the hidden state feature corresponding to any time-frequency frame feature t; Refers to any of the aforementioned time-frequency frame features t; Refers to one-dimensional temporal convolution operation; Refers to the activation function.

6. The method according to claim 4, characterized in that, Based on the frame-level classification layer, each hidden state feature is classified and discriminated to obtain multiple frame-level wake-up probabilities, including: For any of the hidden state features described above , and the hidden state feature The second expression input to the frame-level classification layer is used to evaluate the features of any hidden state based on the second expression. Perform classification and discrimination to obtain the features of any hidden state. The frame-level wake-up probability; the second expression is: ； in, Refers to any of the hidden state features Frame-level wake-up probability; 1 refers to the activation function. Refers to the weight matrix; This refers to the bias parameter.

7. The method according to claim 1, characterized in that, If, based on the wake-up probability, a preset wake-up condition is met, a wake-up process is performed on the target device, including: The wake-up probability is compared with a preset wake-up probability threshold to obtain a comparison result; If the comparison result indicates that the wake-up probability is greater than the wake-up probability threshold, the target device is woken up.

8. The method according to claim 1, characterized in that, Before acquiring real-time speech data from the target device and inputting the speech representation of the real-time speech data into a pre-trained multi-task joint model, the method further includes: Acquire speech training data and input the sample speech representations of the speech training data into the neural network model to be trained; the neural network model includes: an encoder, a noise denoiser, and a wake-up classifier; Based on the encoder, feature extraction is performed on the sample speech representation to obtain the time-frequency features of the sample speech; The sample speech time-frequency features are input into the denoiser, and speech reconstruction processing is performed on the sample speech time-frequency features based on the denoiser to determine the first denoising loss. The sample speech time-frequency features are also input into the wake-up classifier, and probability prediction is performed on the sample speech time-frequency features based on the wake-up classifier to determine the wake-up loss. The neural network model to be trained is trained based on the first noise reduction loss and the wake-up loss, so as to adjust the encoder parameters of the encoder, the noise reduction parameters of the noise reducer and the classifier parameters of the wake-up classifier, and obtain a trained multi-task joint model.

9. The method according to claim 8, characterized in that, The denoiser includes: a feature restoration layer, a fusion layer, and an output layer; based on the denoiser, speech reconstruction processing is performed on the time-frequency features of the sample speech to determine a first denoising loss, including: The time-frequency features of the sample speech are input into the feature restoration layer. Based on the feature restoration layer, the time-frequency features of the sample speech are upsampled at least once to obtain the feature restoration result, and the feature restoration result is input into the fusion layer. The time-frequency features of the sample speech and the feature restoration results are fused based on the fusion layer to obtain the fusion processing result, and the fusion processing result is input to the output layer. The output layer outputs the mask parameters corresponding to the fusion processing result, and performs speech reconstruction processing based on the mask parameters to determine the first noise reduction loss.

10. The method according to claim 9, characterized in that, Based on the fusion layer, the time-frequency features of the sample speech and the feature reconstruction results are fused to obtain the fusion processing result, including: The time-frequency features of the sample speech and the feature reconstruction result are input into the third expression of the fusion layer. Based on the third expression, the time-frequency features of the sample speech and the feature reconstruction result are fused to obtain the fusion result. The third expression is: ； in, Refers to the feature reconstruction result, The jump-connection mechanism, in which, = , Refers to the time-frequency features of the sample speech; Transposed convolution; Refers to the result of the fusion process.

11. The method according to claim 9, characterized in that, The masking parameters include: amplitude masking values; the output layer outputs the masking parameters corresponding to the fusion processing result, and performs speech reconstruction processing based on the masking parameters to determine the first noise reduction loss, including: The fusion result is input to the fourth expression of the output layer, and the fourth expression processes the fusion result to obtain the amplitude mask value; wherein, the fourth expression is: ;in, Refers to the amplitude mask value. Activation function; Refers to one-dimensional temporal convolution operation; Refers to the result of the fusion process; The amplitude mask value is input into the fifth expression, and the first clean speech spectrum is reconstructed from the fifth expression; the fifth expression is: ;in, Refers to the first pure speech spectrum; Noise amplitude spectrum; The first noise reduction loss is determined based on the first clean speech spectrum and the set amplitude spectrum label.

12. The method according to claim 9, characterized in that, The mask parameters include: a complex mask matrix; the output layer outputs the mask parameters corresponding to the fusion processing result, and performs speech reconstruction processing based on the mask parameters to determine the first noise reduction loss, including: Based on the output layer, the fusion processing result is reconstructed using a complex spectrum to obtain a complex mask matrix; The complex mask matrix is ​​format-converted to obtain the complex domain, and the second clean speech spectrum is reconstructed based on the complex domain; The first noise reduction loss is determined based on the second clean speech spectrum and the set amplitude spectrum label.

13. The method according to claim 8, characterized in that, The sample speech time-frequency features include: multi-frame sample time-frequency frame features; the wake-up classifier includes: a temporal modeling layer, a frame-level classification layer, and a temporal aggregation layer; the sample speech time-frequency features are input into the wake-up classifier, and based on the wake-up classifier, probability prediction is performed on the sample speech time-frequency features to determine the wake-up loss, including: The time-frequency frame segmentation features of the multi-frame samples are input into the temporal modeling layer, and the temporal modeling layer processes the time-frequency frame segmentation features of the multi-frame samples respectively to obtain the sample hidden state features corresponding to the time-frequency frame segmentation features of the multi-frame samples respectively. The hidden state features of each sample are input into the frame-level classification layer. Based on the frame-level classification layer, the hidden state features of each sample are classified and judged to obtain multiple sample frame-level wake-up probabilities. The multiple sample frame-level wake-up probabilities are then input into the time aggregation layer. Based on the time aggregation layer, the multiple sample frame-level wake-up probabilities are aggregated according to a set time window to obtain the sample wake-up probability; The wake-up loss is obtained by performing cross-entropy processing on the sample wake-up probability and the preset tag wake-up probability.

14. The method according to claim 8, characterized in that, The neural network model to be trained is trained based on the first noise reduction loss and the wake-up loss to adjust the encoder parameters of the encoder, the noise reduction parameters of the noise reducer, and the classifier parameters of the wake-up classifier, thereby obtaining a trained multi-task joint model, including: Obtain a second noise reduction loss; wherein the second noise reduction loss is obtained by inputting the speech training data into a pre-trained speech noise reduction model and outputting the speech noise reduction model. The first noise reduction loss, the second noise reduction loss, and the wake-up loss are fused together to obtain the target loss. The encoder parameters of the encoder, the denoiser parameters of the denoiser, and the classifier parameters of the wake-up classifier are adjusted according to the target loss to train the neural network model until the training stopping condition is met, and the initial multi-task joint model is obtained after training is completed. The initial multi-task joint model is pruned to obtain a new multi-task joint model.

15. A voice wake-up device based on a multi-task joint model, characterized in that, include: An acquisition module is used to acquire real-time speech data from a target device and input the speech representation of the real-time speech data into a pre-trained multi-task joint model; wherein, the multi-task joint model is obtained by training a neural network model based on joint tasks; the joint tasks include: noise reduction task and wake-up task; The prediction module is used to determine the speech time-frequency features corresponding to the speech representation based on the multi-task joint model, and to perform probability prediction on the speech time-frequency features to obtain the wake-up probability; wherein, the wake-up probability is used to characterize the probability that the real-time speech data includes the target wake-up word; The wake-up module is used to wake up the target device when the preset wake-up conditions are met based on the wake-up probability.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 14.

17. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 14 through the computer program.

Images