Model training method, voice wake-up method, device, equipment and medium

Abstract

This application discloses a model training method, a voice wake-up method, a device, an apparatus, and a medium. The model training method includes: extracting features from acquired voice training samples to obtain first audio features of the voice training samples; using the first audio features, training at least two wake-up sub-models, each including an encoding network and a decoding network; wherein the at least two encoding networks have different parameter counts, and all decoding networks have the same model structure and parameter counts; using the first audio features, jointly training the constructed initial wake-up model to obtain a voice wake-up model; wherein the initial wake-up model includes a joint decoding network and the encoding networks in the at least two wake-up sub-models; the initial model parameters of the joint decoding network are initialized based on the model parameters of the decoding networks in the at least two wake-up sub-models; the voice wake-up model is used to recognize user audio data to wake up electronic devices.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of speech technology, and in particular to a model training method, a voice wake-up method, a device, an equipment, and a storage medium. Background Technology

[0002] With the development of artificial intelligence and natural language processing technologies, voice interaction will become increasingly popular. In voice interaction scenarios for smart hardware, users need to use specific wake words to wake up the device.

[0003] To improve wake-up accuracy and reduce false wake-ups, common methods include: optimizing the voice wake-up model structure by using more complex and advanced network structures, such as increasing the number of layers and width, and increasing the number of model parameters to better fit the data and thus improve the wake-up rate; using model fusion methods, employing multiple models to ensure the wake-up rate, requiring the input audio to pass through two models with results greater than a threshold before wake-up can be achieved; using a two-level wake-up process on the device and in the cloud to recall missed samples; and incorporating other acoustic information to further enhance the voice wake-up effect.

[0004] However, the above methods usually have the problems of increased power consumption of the on-device model, needing to deploy multiple services, or training multiple models. Summary of the Invention

[0005] In view of the above, embodiments of this application provide at least one model training method, a voice wake-up method, a device, an apparatus, and a medium.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a model training method, including:

[0008] Feature extraction is performed on the acquired speech training samples to obtain the first audio features of the speech training samples; using the first audio features, at least two wake-up sub-models, each including an encoding network and a decoding network, are trained respectively; wherein the at least two encoding networks have different parameter counts, and all joint decoding networks have the same model structure and the same number of parameters; using the first audio features, the constructed initial wake-up model is jointly trained to obtain a speech wake-up model; wherein the initial wake-up model includes a joint decoding network and the encoding network in the at least two wake-up sub-models; the initial model parameters of the joint decoding network are initialized based on the model parameters of the decoding network in the at least two wake-up sub-models; the speech wake-up model is used to recognize user audio data to wake up electronic devices.

[0009] Secondly, embodiments of this application provide a voice wake-up method, the method comprising:

[0010] A second audio feature extracted from the user's audio data is obtained; the second audio feature is input into a trained voice wake-up model for recognition, and a recognition result is obtained; wherein, the voice wake-up model is trained by the above-mentioned model training method; if the wake-up word contained in the recognition result is consistent with the preset wake-up word, the electronic device is woken up based on the wake-up word.

[0011] Thirdly, embodiments of this application provide a model training apparatus, comprising:

[0012] The feature extraction module is used to extract features from the acquired speech training samples to obtain the first audio features of the speech training samples.

[0013] A separate training module is used to train at least two wake-up sub-models, each comprising an encoding network and a decoding network, using the first audio features; wherein the at least two encoding networks have different parameter counts, and all joint decoding networks have the same model structure and the same number of parameters;

[0014] A joint training module is used to jointly train the initial wake-up model using the first audio feature to obtain a voice wake-up model; wherein the initial wake-up model includes a joint decoding network and an encoding network in the at least two wake-up sub-models; the initial model parameters of the joint decoding network are initialized based on the model parameters of the decoding network in the at least two wake-up sub-models; the voice wake-up model is used to recognize user audio data to wake up electronic devices.

[0015] Fourthly, embodiments of this application provide a voice wake-up device, including:

[0016] The audio acquisition module is used to acquire the second audio features extracted from the user's audio data;

[0017] An audio recognition module is used to input the second audio feature into a trained voice wake-up model for recognition and obtain a recognition result; wherein the voice wake-up model is trained by any one of the model training methods in the first aspect;

[0018] The voice wake-up module is used to wake up the electronic device based on the wake-up word when the wake-up word contained in the recognition result is consistent with the preset wake-up word.

[0019] Fifthly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps in the method of the first aspect described above; or implements the steps in the method of the second aspect described above.

[0020] Sixthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method of the first aspect described above; or implements the steps in the method of the second aspect described above.

[0021] In this embodiment, firstly, features are extracted from the speech training samples to obtain corresponding first audio features; then, at least two wake-up sub-models composed of decoding and encoding networks are trained separately using the first audio features; then, an initial wake-up model is built using the trained wake-up sub-models, and the joint decoding network in the initial wake-up model is initialized based on the model parameters of the decoding network in each wake-up sub-model; finally, the initial wake-up model is jointly trained using the first audio features to obtain a speech wake-up model; thus, by configuring encoding networks with different parameter amounts in the initial wake-up model, different computational capabilities and accuracies can be generated, thereby achieving dynamic selection of different wake-up effects and corresponding model power consumption.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0024] Figure 1 A schematic diagram of an optional process for a model training method provided in an embodiment of this application;

[0025] Figure 2 A schematic diagram of an optional process for a model training method provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of an optional process for a model training method provided in an embodiment of this application;

[0027] Figure 4 A schematic flowchart of an optional voice wake-up method provided in an embodiment of this application;

[0028] Figure 5 A system framework for a voice wake-up model is provided in the embodiments of this application;

[0029] Figure 6 This is a schematic diagram of the structure of the wake-up sub-model provided in the embodiments of this application;

[0030] Figure 7 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the composition structure of a voice wake-up device provided in an embodiment of this application;

[0032] Figure 9 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0037] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0038] Voice wake-up is typically used in terminal devices or dedicated voice assistant hardware. These devices are programmed with a specific wake-up word, which users can say to indicate that they will issue a voice command to the device. While waiting for a voice command (standby mode), these devices continuously monitor their surroundings with low power consumption. When they hear the wake-up word, they enter the voice command execution state (working state). In other words, voice wake-up essentially switches the device from sleep mode to working mode using voice commands.

[0039] The basic process of voice wake-up is as follows: After receiving the voice signal, the voice device first performs signal processing (mainly including noise reduction and echo cancellation) and feature extraction on the voice signal, thereby converting the original input audio signal into features (i.e., the spectrum signal of the voice) that can be recognized by the wake-up engine on the end (e.g., the terminal device); then the features are input to the wake-up engine for comparison and recognition of the wake-up word; if the wake-up word is matched, the server will continue to be instructed to execute subsequent instructions, such as playing songs or crosstalk.

[0040] Voice wake-up capability primarily relies on the voice wake-up model, which is the core of the entire voice wake-up process. The voice wake-up model is mainly responsible for immediately switching to working mode upon hearing the wake-up word, so real-time monitoring is essential for timely feedback. Due to the need for real-time response and the relatively low computational requirements of the voice wake-up model, it is generally implemented locally on the device.

[0041] Voice wake-up has two optimization goals: first, to minimize the missed wake-up rate (usually expressed as a percentage) under a specific false wake-up rate (typically measured in times per hour); and second, to minimize power consumption. These two goals are clearly mutually restrictive. Due to the constraint of low power consumption, large-scale speech recognition systems are generally not feasible; instead, specialized small recognizers need to be designed for the wake words to be recognized.

[0042] In voice interaction scenarios for smart hardware, users need to wake up the device using a specific wake-up word, such as "Hi XX". Once the device is woken up, it immediately turns on its microphone to record the user's voice and transmits it in real time to a speech recognition system on the device or in the cloud for recognition. This process needs to ensure recognition accuracy to guarantee that downstream tasks, such as natural language understanding, can be processed correctly. Finally, the device will play the required feedback information through speech synthesis, thus completing the entire voice interaction chain. Therefore, waking up the device with voice is a crucial step and the first step in the entire interaction process.

[0043] To improve the accuracy of voice wake-up, several approaches are typically taken: First, optimizing the voice wake-up model structure by using more complex and advanced network structures, such as increasing the number of layers and width, and increasing the number of model parameters to better fit the data, thereby improving the wake-up rate; introducing attention mechanisms and contextual information into the model allows it to better focus on key features, thus improving the wake-up rate; for example, one related method uses an Attention+LSTM network model to output prediction results; another approach is to use model fusion, employing multiple models to ensure a high wake-up rate. For example, one related method prepares two neural networks with different structures; during decoding, the input audio must pass through both models to a threshold before wake-up is possible; the second approach uses the end-user... Two-level wake-up, both on-premises and in the cloud, is used to recall missed samples. For example, related methods use a terminal, including a first wake-up model to perform a first wake-up on the wake-up speech input to the first wake-up model, determine the first wake-up result, and output the wake-up speech to the cloud. The cloud includes a second wake-up model to perform a second wake-up on the wake-up speech input to the second wake-up model, determining the second wake-up result. Other methods incorporate other acoustic information to enhance the voice wake-up effect. For instance, related methods extract various voiceprint features from the preprocessed speech signal; sequentially merge each voiceprint feature to obtain a fused voiceprint feature; and input the fused voiceprint feature into a ResNet network combining an attention mechanism and an MFM activation function for training.

[0044] While these methods for increasing wake-up rate are effective, they present several problems in practical use:

[0045] 1) Optimize the voice wake-up model structure by using a more complex and advanced network structure. This requires time to design the model structure, and the power consumption of the model on the device will increase as the model becomes more complex.

[0046] 2) Using model fusion to ensure wake-up rate not only increases the number of parameters, but also increases power consumption as the number of models increases. At the same time, different thresholds need to be set for different models.

[0047] 3) To use two-level wake-up on the client and cloud, two services need to be deployed on both the client and cloud.

[0048] 4) Integrating other acoustic information to help improve the effect of voice wake-up usually requires additional training of a model that can extract other acoustic information.

[0049] This application provides a model training method that can be executed by a processor of an electronic device. The electronic device, i.e., the model training device, can be a terminal device, a server, a virtual machine, or a distributed computer system composed of one or more servers and / or computers. The terminal device includes, but is not limited to, smartphones, laptops, desktop computers, platform computers, in-vehicle devices, and smart wearable devices. The server can be a regular server or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system. The server can also be a server in a distributed system or a server integrated with blockchain.

[0050] Figure 1 A schematic diagram of an optional process for the model training method provided in the embodiments of this application is shown below. Figure 1 As shown, the method includes the following steps S110 to S130:

[0051] Step S110: Extract features from the acquired speech training samples to obtain the first audio features of the speech training samples.

[0052] Here, the first audio feature can be a filter bank (FBank) feature, a Mel-frequency cepstral coefficient (MFCC) feature, etc., and this application embodiment does not limit it.

[0053] The voice training samples include wake-up audio and non-wake-up audio. Wake-up audio is audio that can output a wake-up command or audio carrying a wake-up word, and is labeled with a tag related to the wake-up word. All audio other than wake-up audio is non-wake-up audio.

[0054] The speech training samples are the collected raw audio signals. Feature extraction refers to extracting features from the speech training samples, converting the raw audio signals into features that the model can recognize, i.e., the spectral signal of the speech. For example, Fbank feature extraction on speech training samples can typically extract 80 dimensions of features at a sampling rate of 16kHz.

[0055] Step S120: Using the first audio features, at least two wake-up sub-models, each including an encoding network and a decoding network, are trained.

[0056] Here, at least two of the expert encoders have different parameter counts, while all the decoders have the same model structure and parameter counts. In other words, the decoders for different wake-up sub-models can be initialized from the same network structure, where the initial model parameters of the decoders for different wake-up sub-models are determined through random initialization.

[0057] In some implementations, the encoding network can be a Long Short-Term Memory (LSTM) module or a transformer, and the decoding network can include an attention mechanism module, a Recurrent Neural Network (RNN) module, a fully connected module, and a softmax module.

[0058] It should be noted that, for wake-up scenarios, different parameter values ​​correspond to different power consumption. In this application embodiment, by configuring encoding networks with different parameter values, different wake-up effects and corresponding model power consumption can be dynamically selected.

[0059] Step S130: Using the first audio features, jointly train the initial wake-up model to obtain the voice wake-up model.

[0060] Here, the initial wake-up model includes at least a joint decoding network and an encoding network in the at least two wake-up sub-models; the initial model parameters of the joint decoding network are initialized based on the model parameters of the decoding networks in the at least two wake-up sub-models; the voice wake-up model is used to recognize user audio data to wake up the electronic device.

[0061] In this embodiment, firstly, features are extracted from the speech training samples to obtain corresponding first audio features; then, at least two wake-up sub-models composed of decoding and encoding networks are trained separately using the first audio features; then, an initial wake-up model is built using the trained wake-up sub-models, and the joint decoding network in the initial wake-up model is initialized based on the model parameters of the decoding network in each wake-up sub-model; finally, the initial wake-up model is jointly trained using the first audio features to obtain a speech wake-up model; thus, by selecting encoding networks with different parameter amounts in the initial wake-up model, different computational capabilities and accuracies can be generated, thereby achieving the goal of increasing the number of parameters to improve wake-up accuracy without affecting the power consumption of the overall speech wake-up model.

[0062] In some implementations, the initial wake-up model further includes an embedding layer and a gating network; the embedding layer generates a first embedding vector corresponding to the first audio feature; the gating network selects a target encoding network from the encoding networks of the at least two wake-up sub-models based on the first embedding vector. During joint training, the model parameters of the embedding layer, the gating network, and the joint decoding network are updated.

[0063] The target encoding network can be an encoding network used to determine the first encoding sequence in the joint decoding network to be input, and the target encoding network can be at least one of all encoding networks. For example, during training, the target encoding network can be the encoding network in all wake-up sub-models, or it can be the encoding network corresponding to the highest confidence level determined based on the first embedding vector. During prediction, the target encoding network can be one of K encoding networks determined based on a preset wake-up rate level, where K is different for different wake-up rate levels, and K is a positive integer.

[0064] It is worth noting that the embedding layer represents a feature mapping process, which refers to mapping the first audio features of the input into a hyperdimensional space. The first audio features extracted from the speech training samples are dimensionality reduced in the embedding layer to adapt to the lightweight gating network. On the other hand, the first audio features are downsampled in the embedding layer. For example, the features of the previous 10 seconds and the speech features of the next 10 seconds are incorporated into the features of the current 10 seconds. This makes the first embedding vector of the output encoding network have global information and can achieve better wake-up effect.

[0065] In some implementations, the gating network can be designed as a lightweight self-attention network. It should be noted that both the gating network and the embedding layer are lightweight neural network modules that do not introduce too many parameters. For example, the gating network can be designed as a lightweight self-attention network that only processes the output of the embedding layer. Its purpose is to provide a confidence score with similar probabilities for each encoding network. When the model predicts an audio sample, it first selects the encoding result of the target encoding network with the highest confidence or the outputs of the top K confidence encoding networks based on the output of the gating network, performs a weighted sum, and uses it as the input of the decoding network.

[0066] In implementation, after encoding the first embedding vector and the first audio feature through the target encoding network, the encoding result is input into the joint decoding network for decoding and recognition. Then, the loss is calculated using the recognition result, and backpropagation is performed to achieve the iterative training process of the model. Through the training process, the gating network learns which encoding network will achieve better results for different input audio features. Thus, after the gating network performs probability prediction on the first embedding vector, the output target encoding network has the highest probability, which means that the switch between the target encoding network and the joint decoding network is turned on, connecting the target encoding network and the joint decoding network.

[0067] Figure 2 A schematic diagram of an optional process for the model training method provided in the embodiments of this application is shown below. Figure 2 As shown, the method includes the following steps S210 to S250:

[0068] Step S210: Extract features from the acquired speech training samples to obtain the first audio features of the speech training samples.

[0069] Step S220: Using the first audio features, at least two wake-up sub-models, each including an encoding network and a decoding network, are trained.

[0070] Here, steps S210 to S220 correspond to steps S110 to S120, respectively. In practice, the specific implementation of steps S110 to S120 can be referred to.

[0071] Step S230: Determine the first model parameters of the encoding network and the second model parameters of the decoding network in each wake-up sub-model.

[0072] Here, each wake-up sub-model includes an encoding network and a decoding network. During training, the first audio feature is first input into the encoding network of each wake-up sub-model, the corresponding encoding result is output and input into the decoding network for decoding and recognition, and finally the loss of the first audio feature through each wake-up sub-model is calculated using the recognition result and the label of the first audio feature, and the gradient is backpropagated until the training converges, and the first model parameters of the encoding network and the second model parameters of the decoding network are obtained.

[0073] It should be noted that the at least two encoding networks can be designed with the same model structure and the same number of parameters, the same model structure but different number of parameters, or different model structures. Therefore, after training each encoding network and the wake-up sub-model composed of the encoding networks separately, the first model parameters of the different encoding networks can be the same or different.

[0074] Step S240: Average the second model parameters of the decoding network in the at least two wake-up sub-models to obtain the third model parameters as the initial model parameters of the joint decoding network.

[0075] Here, since the decoding networks included in different wake-up sub-modules have the same model structure and the same number of parameters, the average of the second model parameters of the decoding networks in at least two wake-up sub-models, i.e., the third model parameters, can be directly used as the initial model parameters of the joint decoding network in the initial wake-up model.

[0076] Step S250: Based on the first model parameters and the third model parameters, the initial wake-up model is jointly trained using the first audio features to obtain the voice wake-up model.

[0077] Here, during joint training, the first model parameters corresponding to each encoding network in the initial wake-up model can be fixed or fine-tuned, and the joint decoding network in the initial wake-up model can be initialized using the third model parameters, while other gating networks and embedding layers are randomly initialized.

[0078] In implementation, the first audio feature is input into the embedding layer and mapped to a first embedding vector. This first embedding vector is then input into a gating network and each encoding network. The gating network outputs a confidence score with a similar probability to each encoding network. During model training, the encoding result of the target encoding network with the highest confidence score is selected as the input to the joint decoding network, or the weighted sum of the encoding results from each encoding network is used as the input to the joint decoding network. During model prediction, if the confidence score of an encoding network is not among the top K (K is determined based on the user-selected wake-up rate level), that encoding network will not participate in prediction. In other words, these encoding networks that do not belong to the top K confidence scores will not be activated, thus reducing the computational complexity of the final voice wake-up model.

[0079] In this embodiment, each wake-up sub-model in the initial wake-up model is first trained to obtain the first model parameters of each encoding network and the second model parameters of each decoding network. The initial model parameters of the joint decoding network are then calculated. Based on the first model parameters of each encoding network and the second model parameters of the decoding network, the model parameters of other network structures in the initial wake-up model are trained using the first audio features to obtain the final voice wake-up model. This approach, by training each set of encoding and decoding networks separately to obtain the first and second model parameters, which are then used to fix or initialize the parameters of the corresponding network structures in the initial wake-up model to be trained, accelerates the joint training process. It also facilitates the direct selection of the target encoding network corresponding to the user's input audio during the deployment phase for inference computation, improving wake-up efficiency without affecting power consumption.

[0080] Figure 3 A schematic diagram of an optional process for the model training method provided in the embodiments of this application is shown below. Figure 3 As shown, step S130, "using the first audio features to jointly train the initial wake-up model to obtain a voice wake-up model," may include the following steps S310 to S350:

[0081] Step S310: The first audio feature is mapped through the embedding layer to obtain the first embedding vector.

[0082] Step S320: Input the first embedding vector into the gated network and each of the encoding networks to obtain the confidence level of each encoding network and the encoding features generated by each encoding network based on the first embedding vector.

[0083] Here, the first audio feature is processed by the embedding layer and the gating network to obtain the confidence level corresponding to each encoding network. The first audio feature is then processed by the embedding layer and each encoding network to obtain the encoded feature output by the same encoding network.

[0084] In some implementations, the first embedding vector is probabilistically predicted in the gating network to obtain the confidence level corresponding to each encoding network; in each encoding network, the first embedding vector and the first audio feature are concatenated and encoded to generate the encoded feature corresponding to the first embedding vector. Here, since the first embedding vector and the first audio feature may reside in different vector spaces, concatenating these two features and inputting them into each encoding network for encoding can obtain more global information from the speech training samples.

[0085] Step S330: Based on the coding features generated by each coding network and the confidence level corresponding to each coding network, determine the first coding sequence to be input to the joint decoding network.

[0086] Here, the coding features of all coding networks are weighted and summed using the weights corresponding to the confidence levels of each coding network to obtain the first coding sequence input to the joint decoding network.

[0087] Step S340: Determine the learning loss based on the output of the joint decoding network and the label of the first audio feature.

[0088] Here, the labels for the first audio feature can include: wake word labels and labels for phonemes, silence categories, and other pronunciation categories contained in the wake word. In implementation, the decoding network parses the first encoded sequence to calculate the confidence scores of the wake words contained in the speech training samples, and then combines the wake word labels to calculate the classification loss, thus obtaining the learning loss of the first audio feature through the initial speech model.

[0089] Step S350: Using the learning loss, update the model parameters of the embedding layer, the gating network, and the joint decoding network until the convergence condition is met to obtain the voice wake-up model.

[0090] Here, the convergence conditions include, but are not limited to, the number of iterations reaching a preset number, the training time meeting a preset duration, or the contrast loss value being lower than a preset threshold. The preset number of iterations is an empirical value, such as 300,000 or 50 million iterations, meaning that steps S310 to S340 are repeated until the preset number of iterations is reached, after which the training process is considered complete.

[0091] In some embodiments, step S320 may include the following steps S321 to S322:

[0092] Step S321: Perform probability prediction on the first embedding vector in the gating network to obtain the confidence level corresponding to each encoding network.

[0093] Step S322: In each of the coding networks, the first embedding vector and the first audio feature are concatenated and encoded to generate the encoded feature corresponding to the first embedding vector.

[0094] In the above embodiment, the first embedding vector output from the embedding layer is input into a gating network for probability prediction to obtain the confidence level of each encoding network. Simultaneously, the first embedding vector is concatenated with the first audio feature and then input into each encoding network to obtain the encoded feature. In this way, by using the confidence levels of different encoding networks output by the gating network to select the target encoding network corresponding to the first audio feature, the wake-up accuracy can be improved. Furthermore, by concatenating the first embedding vector with the first audio feature, global information can be obtained to achieve a better wake-up effect.

[0095] In some implementations, step S330 further includes: performing a weighted summation of the coding features generated by the coding network corresponding to each normalized confidence level to determine the first coding sequence input to the joint decoding network; or, determining the first coding sequence input to the joint decoding network based on the coding features generated by the coding network corresponding to the highest confidence level. In this way, since different coding networks are selected for joint training based on different power consumption requirements, only the coding features output by the coding networks participating in joint training are processed for different situations to obtain the first coding sequence input to the joint decoding network, thus improving the efficiency of model training.

[0096] In some implementations, the method further includes: during joint training, fine-tuning the first model parameters of each of the weighted encoding networks using the learning loss, or fine-tuning the first model parameters of the encoding network corresponding to the highest confidence level. For example, after obtaining the first model parameters of each encoding network through individual training, assuming the learning rate of the joint decoding network in joint training is 0.01, the parameters of the encoding networks participating in the training are adjusted with a learning rate of one-tenth or one-hundredth of 0.01. This allows for continuous adjustment of the encoding network parameters to optimize the accuracy of model wake-up while accelerating joint training.

[0097] In some implementations, the at least two encoding networks are networks with the same or different model structures. Since different parameter values ​​correspond to different power consumption for wake-up scenarios, gating is used to select models with different parameter values ​​to achieve different wake-up effects. In other scenarios, power consumption may not be a primary concern; the aim is to combine more models (equivalent to encoding networks) to obtain better results. This application embodiment, by configuring model structures with different parameter values, can dynamically select different wake-up effects and corresponding model power consumption.

[0098] This application provides a voice wake-up method, which can be executed by the processor of an electronic device. The electronic device, i.e., the voice wake-up device, can be a terminal device, a server, a virtual machine, or a distributed computer system composed of one or more servers and / or computers. The terminal device includes, but is not limited to, smartphones, laptops, desktop computers, platform computers, in-vehicle devices, and smart wearable devices. The server can be a regular server or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system. The server can also be a server in a distributed system or a server integrated with blockchain.

[0099] Figure 4 The following is a schematic diagram of an optional flow of the voice wake-up method provided in an embodiment of this application, such as... Figure 4As shown, the method includes the following steps S410 to S430:

[0100] Step S410: Obtain the second audio feature extracted from the user's audio data.

[0101] Here, the second audio feature can be an FBank feature, a Mel frequency cepstral coefficient (MFCC) feature, etc., and this application embodiment does not limit it.

[0102] Step S420: Input the second audio feature into the trained voice wake-up model for recognition to obtain the recognition result.

[0103] Here, the voice wake-up model is trained using the model training method proposed in the embodiments of this application. In some embodiments, the voice wake-up model includes a joint decoding network and encoding networks in at least two wake-up sub-models; the initial model parameters of the joint decoding network are initialized based on the model parameters of the decoding networks in the at least two wake-up sub-models; in some embodiments, the voice wake-up model further includes an embedding layer and a gating network; the embedding layer is used to generate a second embedding vector corresponding to the second audio feature; the gating network is used to select a target encoding network from the at least two encoding networks based on the second embedding vector.

[0104] Through the training process, the gating network learns which coding network will achieve better results for different audio features of the input. As a result, in the inference stage, the gating network in the voice wake-up model outputs the highest probability of the target coding network after making probability predictions on the second embedding vector.

[0105] Step S430: If the wake word contained in the recognition result is consistent with the preset wake word, wake up the electronic device based on the wake word.

[0106] Here, if the input user audio data matches the preset wake word, the resistive device will continue to be instructed to execute subsequent commands, such as playing a song or opening navigation.

[0107] In this embodiment, a second audio feature extracted from the user's audio data is first obtained. Then, the second audio feature is input into a trained voice wake-up model for recognition to obtain a recognition result. Finally, if the wake-up word contained in the recognition result matches the preset wake-up word, the electronic device is woken up based on the wake-up word. In this way, by training the voice wake-up model, different input audio features can be encoded by the corresponding target encoding network during the inference stage, and some neural networks can be dynamically activated. This achieves improved wake-up accuracy without affecting power consumption, and can be applied to both on-device wake-up systems and cloud wake-up systems.

[0108] In some implementations, the voice wake-up model includes an embedding layer, a gating network, a joint decoding network, and encoding networks in at least two wake-up sub-models; step S420 above can be implemented through steps S421 to S424:

[0109] Step S421: The second audio feature is mapped to the second embedding vector through the embedding layer.

[0110] Step S422: Based on the second embedding vector, select a target encoding network from the encoding networks of the at least two wake-up sub-models through the gating network.

[0111] Here, through the training process, the gating network learns which encoding network can achieve better results for different audio features of the input. The probability, i.e., the confidence score, of the corresponding encoding network output by the gating network is the highest, and the corresponding switch is turned on, controlling the output of the target encoding network to be sent to the decoding network.

[0112] For example, audio A is louder and has better sound quality, so the gating network is trained to select encoding network E1 (with fewer parameters) as the target encoding network for encoding; audio B has noise, so the gating network is trained to select encoding network E2 (with more parameters) as the target encoding network for encoding.

[0113] Step S423: Encode the first audio feature and the second embedding vector using the target coding network to obtain the second coding sequence corresponding to the user audio data.

[0114] Step S424: The second encoded sequence is decoded and analyzed by the joint decoding network to obtain the recognition result.

[0115] In the above embodiments, since the gating network only controls the output of the target encoding network to the joint decoding network for subsequent decoding and recognition, the output probabilities of the embedding layer and the gating network can be calculated first, and the corresponding target encoding network can be selected to participate in the inference calculation. Therefore, it will not affect the power consumption of the overall voice wake-up model. For different wake-up scenarios, different parameter counts in different encoding networks correspond to different power consumptions in the voice wake-up model. By configuring encoding network structures with different parameter counts and training the gating network to select encoding networks with different parameter counts, different wake-up effects can be achieved.

[0116] In some implementations, step S422 above is further implemented as follows: receiving a model configuration instruction input by the user; wherein the model configuration instruction is used to instruct the electronic device to be woken up according to a preset wake-up rate level; performing probability prediction on the second embedding vector through the gating network to obtain the confidence score corresponding to each of the encoding networks; determining the number K of the target encoding networks based on the preset wake-up rate level; K is a natural number greater than or equal to 1; and selecting the encoding networks corresponding to the K largest confidence scores as the target encoding networks. Here, the encoding networks to be encoded are determined according to the confidence score output by the gating network and the wake-up rate level specified by the user.

[0117] For example, suppose the voice wake-up model includes three encoding networks E1, E2, and E3, with wake-up rate levels of low, medium, and high, corresponding to selecting one, two, and three target encoding networks, respectively. Based on the user-input model configuration command, if two target encoding networks are required, the switches corresponding to the two encoding networks with the highest confidence scores are activated. For instance, if the gating network outputs probabilities of 0.5, 0.3, and 0.2 from the three encoding networks E1, E2, and E3, then a new encoding sequence is calculated by adding the encoding features from E1 (0.5) to E2 (0.3) and fed into the joint decoding network for decoding and recognition.

[0118] In this way, the user can choose to turn on K switches (for example, turning on 2 switches will cause the outputs of the two encoding networks with the highest confidence scores from the gating network to be fed into the decoding network, thus improving the wake-up effect, but the corresponding power consumption will also increase). This allows for the dynamic selection of different wake-up effects and corresponding model power consumption.

[0119] The above model training method will be described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0120] This application proposes a voice wake-up scheme that can be applied to both on-device and cloud-based wake-up models to improve wake-up accuracy. While increasing the number of parameters, it does not affect the power consumption of the model. Furthermore, this application proposes a method for users to dynamically select different wake-up effects and corresponding model power consumption (e.g., memory usage).

[0121] Figure 5 A system framework for a voice wake-up model provided in this application embodiment, such as Figure 5As shown, this application designs a voice wake-up model based on MOE, which includes three expert encoders (E1, E2, and E3), an embedding layer N1, a gating network G1, and a decoding network D1. The training objective is to enable the gating network G1 to learn which encoding network achieves better results for different input audio features 501. Thus, the probability of the corresponding encoding network output by the gating network G1 (i.e., the target encoding network) is highest, and the switch between the target encoding network and the decoding network D1 is opened. Therefore, the decoding network D1 only decodes the encoded features output by the target encoding network to obtain the output probability 502. For example, if the probability value corresponding to E1 output by the gating network G1 is 1, and the probabilities of E2 and E3 are 0, then only the switch between E1 and D1 is opened.

[0122] E1, E2, and E3 can be designed with the same model structure (e.g., all Transformers), but with different numbers of parameters (e.g., 5 layers, 2 layers, and 1 layer respectively), or they can be designed with different model structures. Models with different numbers of parameters will have different numbers of layers and weights, resulting in different computational power and accuracy. For example, a model with more layers can capture more speech features and may therefore be more accurate, but it will also increase computational complexity and training difficulty; alternatively, they can be designed with completely identical model structures and parameter numbers. The training phase is implemented through the following process:

[0123] S1 trains each encoder network (E1, E2, or E3) and decoder network D1 separately, such that each encoder network and decoder network D1 is a separate wake-up sub-model, and D1 is randomly initialized for each training session.

[0124] like Figure 6 As shown, taking the wake-up sub-model composed of encoding network E1 and decoding network D1 as an example, the audio features 601 extracted from the audio data are first input into the encoding network E1. After being encoded by the encoding network E1 to obtain the encoded features, they are directly input into the decoding network D1. In D1, the above encoded features are parsed and calculated to obtain the output probability 602 of whether the audio data contains a wake-up word.

[0125] S2, joint training, fixes or only allows fine-tuning of the parameters of E1, E2, and E3, and randomly initializes N1, G1, and D1 using the average value of the three D1 values ​​obtained from individual training.

[0126] During joint training, of the three probabilities output by G1, only the highest one can be switched on; the others are switched off. During training, the three confidence scores output by G1 are used to calculate probabilities using Softmax, ensuring that the sum of the three probability values ​​equals 1. The node with the highest probability is switched on, allowing the output of the corresponding encoder network to be fed into the decoder network for loss calculation and gradient backpropagation, until training converges.

[0127] During the deployment phase, since the gating switch only runs the output of one target encoding network to the decoding network D1, the output probabilities of the embedding layer N1 and the gating network G1 can be calculated first, and the corresponding target encoding network (e.g., E1) can be selected to participate in the inference calculation, so it will not affect the power consumption.

[0128] On the other hand, users can choose to turn on several switches, such as two switches. In this case, the first two of the three confidence probabilities output by the gating network G1 will correspond to the outputs of the encoding network and will be sent to the decoding network D1. This can improve the wake-up effect compared to one encoding network, but the power consumption will also increase.

[0129] This application proposes a voice wake-up method that can be applied to both on-device and cloud-based wake-up models to improve wake-up accuracy. Furthermore, the proposed voice wake-up method increases the number of parameters without affecting the model's power consumption. Finally, this application also allows users to dynamically select different wake-up effects and corresponding model power consumption (e.g., memory usage).

[0130] Based on the foregoing embodiments, this application provides a voice wake-up device, which includes the included modules, as well as the sub-modules and units included in each module. It can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0131] Figure 7 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of this application, as shown below. Figure 7 As shown, the model training device 700 includes: a feature extraction module 710, a separate training module 720, and a joint training module 730, wherein:

[0132] The feature extraction module 710 is used to extract features from the acquired speech training samples to obtain the first audio features of the speech training samples.

[0133] A separate training module 720 is used to train at least two wake-up sub-models, each including an encoding network and a decoding network, using the first audio features; wherein the at least two encoding networks have different parameter counts, and all joint decoding networks have the same model structure and the same number of parameters.

[0134] The joint training module 730 is used to jointly train the constructed initial wake-up model using the first audio feature to obtain a voice wake-up model; wherein, the initial wake-up model includes a joint decoding network and an encoding network in the at least two wake-up sub-models; the initial model parameters of the joint decoding network are initialized based on the model parameters of the decoding network in the at least two wake-up sub-models; the voice wake-up model is used to recognize user audio data to wake up electronic devices.

[0135] In some possible embodiments, the initial wake-up model further includes an embedding layer and a gating network, wherein the embedding layer is used to generate a first embedding vector corresponding to the first audio feature; the gating network is used to select a target encoding network from the encoding networks of the at least two wake-up sub-models based on the first embedding vector; wherein, during joint training, the model parameters of the embedding layer, the gating network and the joint decoding network are updated.

[0136] In some possible embodiments, the joint training module 730 includes: a first determining submodule, configured to determine a first model parameter of the encoding network and a second model parameter of the decoding network in each wake-up sub-model; a second determining submodule, configured to average the second model parameters of the decoding network in the at least two wake-up sub-models to obtain a third model parameter as the initial model parameter of the joint decoding network; and a joint training submodule, configured to perform joint training on the initial wake-up model using the first audio feature based on the first model parameter and the third model parameter to obtain the voice wake-up model.

[0137] In some possible embodiments, the joint training module includes: a feature mapping submodule, used to perform feature mapping on the first audio features through the embedding layer to obtain the first embedding vector; a vector processing submodule, used to input the first embedding vector into the gating network and each of the encoding networks to obtain the confidence level corresponding to each of the encoding networks and the encoding features generated by each of the encoding networks based on the first embedding vector; an encoding sequence submodule, used to determine the first encoding sequence input to the joint decoding network based on the encoding features generated by each of the encoding networks and the confidence level corresponding to each of the encoding networks; a loss determination submodule, used to determine the learning loss based on the output of the joint decoding network and the label of the first audio features; and a parameter update submodule, used to update the model parameters of the embedding layer, the gating network, and the joint decoding network using the learning loss until the convergence condition is met to obtain the voice wake-up model.

[0138] In some possible embodiments, the encoding sequence submodule includes: a first encoding unit, configured to perform a weighted summation of the encoding features generated by the encoding network corresponding to each normalized confidence level, to determine a first encoding sequence input to the joint decoding network; or, a second encoding unit, configured to determine a first encoding sequence input to the joint decoding network based on the encoding features generated by the encoding network corresponding to the highest confidence level.

[0139] In some possible embodiments, the joint training module further includes a fine-tuning submodule, used to fine-tune the first model parameters of each of the weighted summed encoding networks using the learning loss, or to fine-tune the first model parameters of the encoding network corresponding to the highest confidence level.

[0140] In some possible embodiments, the vector processing submodule includes: a prediction unit, used to perform probability prediction on the first embedding vector in the gating network to obtain the confidence level corresponding to each encoding network; and an encoding unit, used to concatenate and encode the first embedding vector and the first audio feature in each encoding network to generate the encoded feature corresponding to the first embedding vector.

[0141] In some possible embodiments, the at least two encoding networks are networks with the same or different model structures.

[0142] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0143] Based on the foregoing embodiments, this application provides a voice wake-up device, which includes the included modules, as well as the sub-modules and units included in each module. It can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit, a microprocessor, a digital signal processor, or a field-programmable gate array, etc.

[0144] Figure 8 This is a schematic diagram of the composition structure of a voice wake-up device provided in an embodiment of this application, as shown below. Figure 8 As shown, the voice wake-up device 800 includes: an audio acquisition module 810, an audio recognition module 820, and a voice wake-up module 830, wherein:

[0145] The audio acquisition module 810 is used to acquire the second audio feature extracted from the user's audio data;

[0146] The audio recognition module 820 is used to input the second audio feature into a trained voice wake-up model for recognition and obtain a recognition result; wherein, the voice wake-up model is trained by the above-described model training method;

[0147] The voice wake-up module 830 is used to wake up the electronic device based on the wake-up word when the wake-up word contained in the recognition result is consistent with the preset wake-up word.

[0148] In some possible embodiments, the voice wake-up model includes an embedding layer, a gating network, a joint decoding network, and encoding networks in at least two wake-up sub-models; the audio recognition module 820 includes: a feature mapping sub-module, used to map the second audio feature to the second embedding vector through the embedding layer; a gating selection sub-module, used to select a target encoding network from the encoding networks in the at least two wake-up sub-models based on the second embedding vector through the gating network; a vector encoding sub-module, used to encode the first audio feature and the second embedding vector using the target encoding network to obtain a second encoded sequence corresponding to the user audio data; and a decoding recognition sub-module, used to decode and analyze the second encoded sequence through the joint decoding network to obtain the recognition result.

[0149] In some possible embodiments, the gating selection submodule includes: an instruction acquisition unit, configured to receive a model configuration instruction input by a user; wherein the model configuration instruction is used to instruct the electronic device to wake up according to a preset wake-up rate level; a probability prediction unit, configured to perform probability prediction on the second embedding vector through the gating network to obtain a confidence level corresponding to each of the encoding networks; a switch determination unit, configured to determine the number K of the target encoding networks based on the preset wake-up rate level; K is a natural number greater than or equal to 1; and a network selection unit, configured to select the encoding networks corresponding to the K largest confidence levels as the target encoding networks.

[0150] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0151] It should be noted that, in the embodiments of this application, if the above-mentioned model training method or voice wake-up method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0152] This application provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described model training method or voice wake-up method.

[0153] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0154] This application provides a computer program including computer-readable code. When the computer-readable code is run in an electronic device, the processor in the electronic device executes some or all of the steps for implementing the above-described model training method or voice wake-up method.

[0155] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the aforementioned model training method or voice wake-up method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0156] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0157] It should be noted that, Figure 9 This is a schematic diagram of a hardware entity of an electronic device in an embodiment of this application, wherein the electronic device can be a model training device or a voice wake-up device. Figure 9 As shown, the hardware entity of the electronic device 900 includes: a processor 901, a communication interface 902, and a memory 903, wherein:

[0158] Processor 901 typically controls the overall operation of electronic device 900.

[0159] Communication interface 902 enables electronic devices to communicate with other terminals or servers via a network.

[0160] The memory 903 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 901 and various modules in the electronic device 900. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 901, the communication interface 902, and the memory 903 can be performed via bus 904.

[0161] It should be noted that the model training device or voice wake-up device in the embodiments of this application can be a terminal device, a server or virtual machine, or a distributed computer system composed of one or more servers and / or computers.

[0162] The terminal device includes, but is not limited to, smartphones, laptops, desktop computers, platform computers, in-vehicle devices, and smart wearable devices, etc., and this application embodiment does not limit the scope. The server can be a regular server or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system. The server can also be a server for a distributed system, or a server combined with blockchain.

[0163] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0164] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0166] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0167] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0168] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0169] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0170] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: Feature extraction is performed on the acquired speech training samples to obtain the first audio features of the speech training samples; Using the first audio feature, at least two wake-up sub-models, each including an encoding network and a decoding network, are trained respectively; wherein, the number of parameters in the at least two encoding networks is different, and the model structure and the number of parameters in all the decoding networks are the same; Using the first audio feature, the initial wake-up model is jointly trained to obtain a voice wake-up model; wherein, the initial wake-up model includes a joint decoding network and an encoding network in the at least two wake-up sub-models; the initial model parameters of the joint decoding network are initialized based on the model parameters of the decoding network in the at least two wake-up sub-models; the voice wake-up model is used to recognize user audio data to wake up electronic devices.

2. The method according to claim 1, characterized in that, The initial wake-up model further includes an embedding layer and a gating network. The embedding layer is used to generate a first embedding vector corresponding to the first audio feature. The gating network is used to select a target coding network from the coding networks of the at least two wake-up sub-models based on the first embedding vector. During joint training, the model parameters of the embedding layer, the gating network, and the joint decoding network are updated.

3. The method according to claim 1, characterized in that, The step of jointly training the initial wake-up model using the first audio feature to obtain a voice wake-up model includes: Determine the first model parameters of the encoding network and the second model parameters of the decoding network in each of the wake-up sub-models; The second model parameters of the decoding network in the at least two wake-up sub-models are averaged to obtain the third model parameters, which are then used as the initial model parameters of the joint decoding network. Based on the first model parameters and the third model parameters, the initial wake-up model is jointly trained using the first audio features to obtain the voice wake-up model.

4. The method according to claim 2, characterized in that, The step of jointly training the initial wake-up model using the first audio feature to obtain a voice wake-up model includes: The first audio feature is mapped through the embedding layer to obtain the first embedding vector; The first embedding vector is input into the gated network and each of the coding networks to obtain the confidence level of each coding network and the coding features generated by each coding network based on the first embedding vector. Based on the coding features generated by each coding network and the confidence level corresponding to each coding network, a first coding sequence is determined for input to the joint decoding network; Based on the output of the joint decoding network and the label of the first audio feature, the learning loss is determined; The learning loss is used to update the model parameters of the embedding layer, the gating network, and the joint decoding network until the convergence condition is met, thus obtaining the voice wake-up model.

5. The method according to claim 4, characterized in that, The step of determining the first encoded sequence input to the joint decoding network based on the encoded features generated by each of the encoded networks and the confidence level corresponding to each of the encoded networks includes: The weighted sum of the encoded features generated by the encoding network corresponding to each normalized confidence level is used to determine the first encoded sequence input to the joint decoding network; or, Based on the coding features generated by the coding network corresponding to the highest confidence level, the first coding sequence input to the joint decoding network is determined.

6. The method according to claim 5, characterized in that, The method further includes: The learning loss is used to fine-tune the first model parameters of each of the weighted summation encoding networks, or to fine-tune the first model parameters of the encoding network corresponding to the highest confidence level.

7. The method according to claim 4, characterized in that, The step of inputting the first embedding vector into the gated network and each of the encoding networks to obtain the confidence level of each encoding network and the encoding features generated by each encoding network based on the first embedding vector includes: In the gating network, the first embedding vector is probabilistically predicted to obtain the confidence level corresponding to each encoding network. In each of the coding networks, the first embedding vector and the first audio feature are concatenated and encoded to generate the encoded feature corresponding to the first embedding vector.

8. The method according to any one of claims 1 to 7, characterized in that, The at least two encoding networks are networks with the same or different model structures.

9. A voice wake-up method, characterized in that, The method includes: Obtain the second audio feature extracted from the user's audio data; The second audio feature is input into the trained voice wake-up model for recognition to obtain the recognition result; wherein the voice wake-up model is trained by any one of the model training methods in claims 1 to 8; If the wake word contained in the recognition result is consistent with the preset wake word, the electronic device is woken up based on the wake word.

10. The wake-up method according to claim 9, characterized in that, The voice wake-up model includes an embedding layer, a gating network, a joint decoding network, and encoding networks in at least two wake-up sub-models; the step of inputting the second audio feature into the trained voice wake-up model for recognition to obtain the recognition result includes: The second audio feature is mapped to a second embedding vector through the embedding layer; Based on the second embedding vector, a target encoding network is selected from the encoding networks of the at least two wake-up sub-models through the gating network; The first audio feature and the second embedding vector are encoded using the target encoding network to obtain the second encoded sequence corresponding to the user audio data; The recognition result is obtained by decoding and analyzing the second encoded sequence through the joint decoding network.

11. The wake-up method according to claim 10, characterized in that, The step of selecting a target encoding network from the encoding networks of the at least two wake-up sub-models based on the second embedding vector through the gating network includes: Receive a model configuration instruction input by the user; wherein the model configuration instruction is used to instruct the electronic device to wake up according to a preset wake-up rate level; The second embedding vector is probabilistically predicted using the gating network to obtain the confidence level corresponding to each of the coding networks. Based on the preset wake-up rate level, the number K of the target coding networks is determined; K is a natural number greater than or equal to 1. The coding networks corresponding to the top K largest confidence scores are used as the target coding networks.

12. A model training device, characterized in that, The model training device includes: The feature extraction module is used to extract features from the acquired speech training samples to obtain the first audio features of the speech training samples. A separate training module is used to train at least two wake-up sub-models, each comprising an encoding network and a decoding network, using the first audio features; wherein the at least two encoding networks have different parameter counts, and all decoding networks have the same model structure and the same number of parameters. A joint training module is used to jointly train the initial wake-up model using the first audio feature to obtain a voice wake-up model; wherein the initial wake-up model includes a joint decoding network and an encoding network in the at least two wake-up sub-models; the initial model parameters of the joint decoding network are initialized based on the model parameters of the decoding network in the at least two wake-up sub-models; the voice wake-up model is used to recognize user audio data to wake up electronic devices.

13. A voice wake-up device, characterized in that, The voice wake-up device includes: The audio acquisition module is used to acquire the second audio features extracted from the user's audio data; An audio recognition module is used to input the second audio feature into a trained voice wake-up model for recognition and obtain a recognition result; wherein the voice wake-up model is trained by any one of the model training methods in claims 1 to 8; The voice wake-up module is used to wake up the electronic device based on the wake-up word when the wake-up word contained in the recognition result is consistent with the preset wake-up word.

14. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8; or implements the steps of the method according to any one of claims 9 to 11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8; or implements the steps of the method according to any one of claims 9 to 11.

Images