A voice wake-up method and related device

Abstract

The present application discloses a voice wake-up method and related devices, which pre-train a trained multilingual wake-up word detection model. The trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, and the first decoder is a lightweight decoder. After acquiring the voice signal to be recognized, the voice signal is encoded using the trained first encoder to obtain a coding feature, and the coding feature is decoded using the trained first decoder to obtain the wake-up word detection result; according to the wake-up word detection result, the wake-up result of the voice signal can be determined. On the one hand, this solution can use a multilingual wake-up word detection model to achieve the purpose of voice wake-up of wake-up words in multiple languages. On the other hand, the model is a lightweight model, which is easier to deploy, reducing the difficulty of implementing multilingual voice wake-up.

Description

Technical Field

[0001] The present application relates to the field of voice processing technology, and in particular to a voice wake-up method and related devices. Background Art

[0002] Voice wake-up technology refers to the process in which a smart device (such as a robot, mobile phone, wearable device, smart home, car, etc.) detects a user's specific voice (usually a set wake-up word) in a dormant state, and then puts the device into a waiting state and enters the voice interaction process. With the growing market demand, smart devices are widely used in many countries around the world. Due to the different languages ​​used in different countries, the voice wake-up function of smart devices needs to support wake-up words in multiple languages.

[0003] However, the existing models for implementing voice wake-up have a large number of parameters, which affects the implementation of multi-language voice wake-up.

[0004] Therefore, how to provide a voice wake-up method that can more easily implement voice wake-up with wake-up words in multiple languages ​​has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0005] In view of the above problems, the present application provides a voice wake-up method and related devices to more easily achieve the purpose of voice wake-up of wake-up words in multiple languages. The specific scheme is as follows:

[0006] The first aspect of the present application provides a voice wake-up method, comprising:

[0007] Acquire a speech signal to be recognized;

[0008] Performing wake-up word detection on the speech signal using a trained multilingual wake-up word detection model to obtain a wake-up word detection result, wherein the trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, used to encode the speech signal to obtain a coding feature, and the first decoder is a lightweight decoder, used to decode the coding feature to obtain the wake-up word detection result;

[0009] The wake-up result of the voice signal is determined according to the wake-up word detection result.

[0010] In a possible implementation, the training method of the multilingual wake-up word detection model includes:

[0011] Obtain multi-language speech recognition training data and multi-language speech wake-up training data;

[0012] Acquire a pre-built multilingual speech recognition model, wherein the multilingual speech recognition model includes a second encoder and a second decoder;

[0013] Using the multilingual speech recognition training data, training the multilingual speech recognition model to obtain a trained second encoder;

[0014] Based on the trained second encoder, construct a third encoder;

[0015] Constructing a fine-tuning network, wherein the fine-tuning network includes the third encoder, a network layer, and the first decoder to be trained;

[0016] Fine-tune the fine-tuning network using the multilingual voice wake-up training data to obtain a fine-tuned third encoder and network layer;

[0017] Constructing a multilingual wake-up word detection model training network, wherein the multilingual wake-up word detection model training network comprises a first wake-up word detection branch and a second wake-up word detection branch, wherein the first wake-up word detection branch comprises the fine-tuned third encoder, the network layer, and the first decoder to be trained, and the second wake-up word detection branch comprises the first encoder to be trained and the first decoder to be trained;

[0018] The multilingual voice wake-up training data is used to train the multilingual wake-up word detection model training network to obtain a trained multilingual wake-up word detection model.

[0019] In a possible implementation, constructing a third encoder based on the trained second encoder includes:

[0020] Freeze the parameters of the trained second encoder, and add a weight parameter to each hidden layer in the trained second encoder to obtain a second encoder with weight parameters;

[0021] A LORA module is added to the weighted second encoder to obtain the third encoder.

[0022] In a possible implementation, fine-tuning the fine-tuning network using the multilingual voice wake-up training data to obtain a fine-tuned third encoder and network layer includes:

[0023] After the multilingual speech wake-up training data is input into the fine-tuning network, the output weight parameters of each hidden layer in the third encoder and the parameters of the LORA module are fine-tuned by using the cross entropy loss between the wake-up word detection result output by the fine-tuning network and the wake-up label to obtain the fine-tuned third encoder and network layer.

[0024] In a possible implementation, the using the multilingual voice wake-up training data to train the multilingual wake-up word detection model training network to obtain a trained multilingual wake-up word detection model includes:

[0025] After the multilingual speech wake-up training data is used to input the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, and the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder. The trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

[0026] In a possible implementation, the using the multilingual voice wake-up training data to train the multilingual wake-up word detection model training network to obtain a trained multilingual wake-up word detection model includes:

[0027] For each training data in the multilingual voice wake-up training data, obtain its category label from the K-Means clusterer and mark it;

[0028] Adding a classifier after the first encoder to be trained in the multilingual wake-up word detection model training network;

[0029] After the multilingual speech wake-up training data is input into the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label, and the cross entropy loss between the category prediction result output by the classifier and the category label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder, and the trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

[0030] In a possible implementation, performing wake-up word detection on the speech signal using a trained multilingual wake-up word detection model to obtain a wake-up word detection result includes:

[0031] Acquiring prior information of a language corresponding to the speech signal;

[0032] After concatenating the prior information of the language corresponding to the speech signal and the speech signal, the information is input into the trained multilingual wake-up word detection model to obtain a wake-up word detection result.

[0033] A second aspect of the present application provides a voice wake-up device, comprising:

[0034] An acquisition unit, used for acquiring a speech signal to be recognized;

[0035] a wake-up word detection unit, configured to perform wake-up word detection on the speech signal using a trained multilingual wake-up word detection model to obtain a wake-up word detection result, wherein the trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, configured to encode the speech signal to obtain a coding feature, and the first decoder is a lightweight decoder, configured to decode the coding feature to obtain the wake-up word detection result;

[0036] A wake-up result determination unit is used to determine the wake-up result of the voice signal according to the wake-up word detection result.

[0037] In a possible implementation, the voice wake-up device further includes a multilingual wake-up word detection model training unit;

[0038] The multilingual wake-up word detection model training unit includes:

[0039] A training data acquisition unit, used to acquire multilingual speech recognition training data and multilingual speech wake-up training data;

[0040] A multilingual speech recognition model acquisition unit, used to acquire a pre-built multilingual speech recognition model, wherein the multilingual speech recognition model includes a second encoder and a second decoder;

[0041] a multilingual speech recognition model training unit, configured to train the multilingual speech recognition model using the multilingual speech recognition training data to obtain a trained second encoder;

[0042] A third encoder construction unit, configured to construct a third encoder based on the trained second encoder;

[0043] A fine-tuning network construction unit, used to construct a fine-tuning network, wherein the fine-tuning network includes the third encoder, a network layer, and the first decoder to be trained;

[0044] A fine-tuning unit, configured to fine-tune the fine-tuning network using the multilingual speech wake-up training data to obtain a fine-tuned third encoder and network layer;

[0045] a multilingual wake-up word detection model training network construction unit, used to construct a multilingual wake-up word detection model training network, wherein the multilingual wake-up word detection model training network comprises a first wake-up word detection branch and a second wake-up word detection branch, wherein the first wake-up word detection branch comprises the fine-tuned third encoder, the network layer, and the first decoder to be trained, and the second wake-up word detection branch comprises the first encoder to be trained and the first decoder to be trained;

[0046] The multilingual wake-up word detection model training network training unit is used to train the multilingual wake-up word detection model training network using the multilingual speech wake-up training data to obtain a trained multilingual wake-up word detection model.

[0047] In a possible implementation, the third encoder construction unit is specifically configured to:

[0048] Freeze the parameters of the trained second encoder, and add a weight parameter to each hidden layer in the trained second encoder to obtain a second encoder with weight parameters;

[0049] A LORA module is added to the weighted second encoder to obtain the third encoder.

[0050] In a possible implementation, the fine-tuning unit is specifically configured to:

[0051] After the multilingual speech wake-up training data is input into the fine-tuning network, the output weight parameters of each hidden layer in the third encoder and the parameters of the LORA module are fine-tuned by using the cross entropy loss between the wake-up word detection result output by the fine-tuning network and the wake-up label to obtain the fine-tuned third encoder and network layer.

[0052] In a possible implementation, the multilingual wake-up word detection model training network training unit is specifically used to:

[0053] After the multilingual speech wake-up training data is used to input the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, and the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder. The trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

[0054] In a possible implementation, the multilingual wake-up word detection model training network training unit is specifically used to:

[0055] For each training data in the multilingual voice wake-up training data, obtain its category label from the K-Means clusterer and mark it;

[0056] Adding a classifier after the first encoder to be trained in the multilingual wake-up word detection model training network;

[0057] After the multilingual speech wake-up training data is input into the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label, and the cross entropy loss between the category prediction result output by the classifier and the category label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder, and the trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

[0058] In a possible implementation, the wake-up word detection unit is specifically used to:

[0059] Acquiring prior information of a language corresponding to the speech signal;

[0060] After concatenating the prior information of the language corresponding to the speech signal and the speech signal, the information is input into the trained multilingual wake-up word detection model to obtain a wake-up word detection result.

[0061] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the voice wake-up method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0062] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0063] The memory is used to store computer programs;

[0064] The processor is used to execute the computer program so that the electronic device can implement the voice wake-up method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0065] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the voice wake-up method of the above-mentioned first aspect or any implementation of the first aspect.

[0066] By means of the above technical scheme, a voice wake-up method and related devices provided in the present application are pre-trained to obtain a trained multilingual wake-up word detection model. The trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, and the first decoder is a lightweight decoder. After acquiring the voice signal to be recognized, the voice signal is encoded using the trained first encoder to obtain the encoding features, and the encoding features are decoded using the trained first decoder to obtain the wake-up word detection result; according to the wake-up word detection result, the wake-up result of the voice signal can be determined. On the one hand, this scheme can use the multilingual wake-up word detection model to achieve the purpose of voice wake-up of wake-up words in multiple languages. On the other hand, the model is a lightweight model, which is easier to deploy, reducing the difficulty of implementing multilingual voice wake-up. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0068] Figure 1 A flow chart of a voice wake-up method disclosed in an embodiment of the present application;

[0069] Figure 2 A flowchart of a training method for a multilingual wake-up word detection model disclosed in an embodiment of the present application;

[0070] Figure 3 A structural schematic diagram of a fine-tuning network is disclosed for an embodiment of the present application;

[0071] Figure 4 A schematic diagram of the structure of a multilingual wake-up word detection model training network disclosed in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of another multilingual wake-up word detection model training network structure disclosed in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of the structure of a voice wake-up device disclosed in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0076] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0077] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0078] Voice wake-up technology refers to the process in which a smart device (such as a robot, mobile phone, wearable device, smart home, car, etc.) detects a user's specific voice (usually a set wake-up word) in a dormant state, and then puts the device into a waiting state and enters the voice interaction process. With the growing market demand, smart devices are widely used in many countries around the world. Due to the different languages ​​used in different countries, the voice wake-up function of smart devices needs to support wake-up words in multiple languages.

[0079] However, the existing models for implementing voice wake-up have a large number of parameters, which affects the implementation of multi-language voice wake-up.

[0080] Therefore, how to provide a voice wake-up method that can more easily implement voice wake-up with wake-up words in multiple languages ​​has become a technical problem that needs to be urgently solved by those skilled in the art.

[0081] In order to solve the above problems, the embodiment of the present application provides a voice wake-up method. The voice wake-up method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0082] Reference Figure 1 , Figure 1 A flow chart of a voice wake-up method disclosed in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method may include the following steps:

[0083] S101: Acquire a speech signal to be recognized;

[0084] In the present application, the voice signal to be recognized is a wake-up word voice signal input by the user. The wake-up word voice signal can be a voice signal input by the user that only contains the wake-up word, or it can be a part containing the wake-up word extracted from a voice signal input by the user that contains the wake-up word and an interactive instruction. This application does not impose any limitations on this.

[0085] It should be noted that in the present application, the wake-up word voice signal may include wake-up words in any language, including but not limited to Chinese wake-up words (such as Xiaofei Xiaofei, Xiaodu Xiaodu, etc.), English wake-up words (such as Hey Siri, Hi Bixby, etc.), etc., and the present application does not impose any limitations on this.

[0086] S102: Performing wake-up word detection on the speech signal using a trained multilingual wake-up word detection model to obtain a wake-up word detection result, wherein the trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, used to encode the speech signal to obtain a coding feature, and the first decoder is a lightweight decoder, used to decode the coding feature to obtain the wake-up word detection result;

[0087] In this application, the lightweight encoder is any neural network architecture with a simple structure and a small number of parameters, such as ShuffleNet, and this application does not make any restrictions on this. The lightweight decoder is any neural network architecture with a simple structure and a small number of parameters, such as MDTC (Multi-scale dilated temporal convolutional network), and this application does not make any restrictions on this.

[0088] S103: Determine a wake-up result of the voice signal according to the wake-up word detection result.

[0089] In the present application, the wake-up result includes a first wake-up result and a second wake-up result, the first wake-up result is used to indicate that the wake-up is successful, and the second wake-up result is used to indicate that the wake-up fails. The wake-up result of the voice signal is ultimately the first wake-up result or the second wake-up result. If the wake-up word detection result indicates that there is a wake-up word in the voice signal, the wake-up result of the voice signal is determined to be the first wake-up result, and if the wake-up word detection result indicates that there is no wake-up word in the voice signal, the wake-up result of the voice signal is determined to be the second wake-up result.

[0090] A voice wake-up method provided in this embodiment is pre-trained to obtain a trained multilingual wake-up word detection model. The trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, and the first decoder is a lightweight decoder. After acquiring the voice signal to be recognized, the voice signal is encoded using the trained first encoder to obtain a coding feature, and the coding feature is decoded using the trained first decoder to obtain the wake-up word detection result; according to the wake-up word detection result, the wake-up result of the voice signal can be determined. On the one hand, this solution can use a multilingual wake-up word detection model to achieve the purpose of voice wake-up of wake-up words in multiple languages. On the other hand, the model is a lightweight model, which is easier to deploy, reducing the difficulty of implementing multilingual voice wake-up.

[0091] In another embodiment of the present application, the training method of the multilingual wake-up word detection model is described in detail, referring to Figure 2 , the training method may include the following steps:

[0092] S201: Acquire multi-language speech recognition training data and multi-language speech wake-up training data;

[0093] In the present application, multilingual speech recognition audio samples and corresponding recognition labels can be collected and organized as multilingual speech recognition training data. For each multilingual speech recognition audio sample, its corresponding recognition label is the text corresponding to the multilingual speech recognition audio sample.

[0094] In the present application, speech wake-up audio samples covering different languages ​​and different wake-up words and corresponding wake-up labels can be prepared as multi-language speech wake-up training data. For each speech wake-up audio sample, each audio frame is marked with a wake-up label. The wake-up label can include multiple wake-up positive labels and one wake-up negative label. Each wake-up word in each language corresponds to a wake-up positive label. The wake-up positive label corresponding to the wake-up word of language 1 can be represented by "1", the wake-up positive label corresponding to the wake-up word of language 2 can be represented by "2", and the wake-up negative label can be represented by "0".

[0095] The speech wake-up training audio samples include positive samples and negative samples. For each positive sample, part of its audio frames correspond to a wake-up positive label, and the remaining audio frames correspond to the wake-up negative label. For each negative sample, all of its audio frames correspond to the wake-up negative label.

[0096] As an implementable method, when assigning wake-up labels to each audio frame of the positive sample, voice activity detection can be performed on the positive sample first to determine the wake-up word end point, and the wake-up positive label can be assigned to the n audio frames before the wake-up word end point (including the wake-up word end point), and the remaining audio frames can be assigned wake-up negative labels. In the present application, a voice activity detection model can be used to perform voice activity detection on positive samples, and the voice activity detection model can adopt a hybrid architecture based on the combination of LSTM and convolutional neural network. The value of n is determined according to the standard filter feature extraction configuration. Generally speaking, 1 second of audio corresponds to 100 audio frames. The range of n is set to be roughly one-quarter of the audio length of the voice wake-up audio sample, and in principle, it is ensured that a complete syllable is contained in the n audio frames.

[0097] The above processing of voice wake-up audio samples is to prepare for the maximum pooling loss in subsequent training. Through such data preparation and label allocation, high-quality data support is provided for subsequent model training, ensuring that the trained multilingual wake-up word detection model has excellent accuracy.

[0098] S202: Acquire a pre-built multilingual speech recognition model, where the multilingual speech recognition model includes a second encoder and a second decoder;

[0099] In the present application, the multilingual speech recognition model includes a second encoder and a second decoder. In order to improve the multilingual speech recognition effect, in the present application, the network architecture of the multilingual speech recognition model can consider increasing the network depth and width. As an implementable method, the second encoder of the multilingual speech recognition model can be composed of 32 ConformerBlocks, and the second decoder can be composed of 16 Transformer Blocks. This network architecture can ensure that the multilingual speech recognition model shows excellent results when performing multilingual speech recognition.

[0100] S203: Training the multilingual speech recognition model using the multilingual speech recognition training data to obtain a trained second encoder;

[0101] In the present application, when the multilingual speech recognition model is trained using multilingual speech recognition training data, the loss function may be a cross entropy loss function, specifically, the cross entropy loss between the recognition result output by the multilingual speech recognition model and the recognition label. The optimization goal is to minimize the loss function or reach a preset number of iterative optimization times.

[0102] After the multilingual speech recognition model is trained using the multilingual speech recognition training data, the parameters of the second encoder and the second decoder in the multilingual speech recognition model can be determined. The second encoder after the parameters are determined is the trained second encoder.

[0103] S204: constructing a third encoder based on the trained second encoder;

[0104] In the present application, the third encoder is constructed based on the trained second encoder; specifically, the parameters of the trained second encoder are frozen, and a weight parameter is added to each hidden layer in the trained second encoder to weight the output of the hidden layer to obtain a second encoder with weight parameters, and then a LORA module is added to the weighted second encoder to obtain the third encoder.

[0105] Exemplarily, assuming that the second encoder is composed of 32 Conformer Blocks, it is necessary to add a weight parameter to the output of each Conformer Block to weight the output of the Conformer Block.

[0106] S205: constructing a fine-tuning network, wherein the fine-tuning network includes the third encoder, a network layer, and the first decoder to be trained;

[0107] For ease of understanding, refer to Figure 3 , Figure 3 The present application discloses a structural diagram of a fine-tuning network, such as Figure 3 As shown, in the fine-tuning network, the input end of the network layer is connected to the output end of the third encoder, and the output end of the network layer is connected to the first decoder to be trained.

[0108] S206: Fine-tune the fine-tuning network using the multilingual voice wake-up training data to obtain a fine-tuned third encoder and network layer;

[0109] In a possible implementation, fine-tuning the fine-tuning network using the multilingual voice wake-up training data to obtain a fine-tuned third encoder and network layer includes:

[0110] After the multilingual speech wake-up training data is input into the fine-tuning network, the output weight parameters of each hidden layer in the third encoder and the parameters of the LORA module are fine-tuned by using the cross entropy loss between the wake-up word detection result output by the fine-tuning network and the wake-up label to obtain the fine-tuned third encoder and network layer.

[0111] Specifically, the speech wake-up audio sample of the multilingual speech wake-up training data is input into the third encoder. After the third encoder encodes the speech wake-up audio sample, it outputs the encoding feature, and the encoding feature is input into the network layer. The network layer performs dimensionality reduction processing on the encoding feature to obtain the reduced-dimensional feature; the reduced-dimensional feature is input into the first decoder, and the first decoder decodes it to obtain the predicted probability of the wake-up word corresponding to each audio frame in the speech wake-up audio sample, and the predicted probability is used to indicate the possibility of the existence of the wake-up word.

[0112] In this application, the maximum pooling loss is used to optimize the multilingual speech wake-up training model. That is, for each audio frame of the speech wake-up audio sample, the maximum value of the prediction probabilities of each audio frame is selected as the final prediction probability, and the final prediction probability is the wake-up word detection result.

[0113] The loss function may be a cross entropy loss function, specifically a cross entropy loss between the wake-up word detection result output by the fine-tuning network and the wake-up tag. The optimization goal is to minimize the loss function or reach a preset number of optimization iterations.

[0114] It should be noted that during the fine-tuning process, the adjusted parameters are limited to the output weight parameters of each hidden layer, the parameters of the LORA module, the parameters of the network layer, and the parameters of the first decoder. Some parameters in the third encoder corresponding to the second encoder are not adjusted.

[0115] After fine-tuning, we can obtain a multilingual wake-up word detection model based on the multilingual speech recognition model. This model not only inherits the powerful semantic space modeling capability of the multilingual speech recognition model, but can also effectively adapt to multilingual speech wake-up tasks, providing strong support for subsequent voice interaction applications.

[0116] However, the multilingual wake-up word detection model based on the multilingual speech recognition model has a large number of parameters and is not easy to deploy on resource-constrained devices. Therefore, it is particularly necessary to train a lightweight multilingual wake-up word detection model.

[0117] S207: constructing a multilingual wake-up word detection model training network, wherein the multilingual wake-up word detection model training network comprises a first wake-up word detection branch and a second wake-up word detection branch, wherein the first wake-up word detection branch comprises the fine-tuned third encoder, the network layer, and the first decoder to be trained, and the second wake-up word detection branch comprises the first encoder to be trained and the first decoder to be trained;

[0118] In order to train a lightweight multilingual wake-up word detection model and to make full use of the rich acoustic knowledge contained in the multilingual speech recognition model, in the present application, a multilingual wake-up word detection model training network can be constructed. The multilingual wake-up word detection model training network includes two wake-up word detection branches. The first wake-up word detection branch includes a fine-tuned third encoder, a network layer, and a first decoder to be trained. The second wake-up word detection branch includes the first encoder to be trained and the first decoder to be trained. The two wake-up word detection branches share the same decoder. For ease of understanding, refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a multilingual wake-up word detection model training network disclosed in an embodiment of the present application. Figure 4 As shown, the input end of the network layer is connected to the output end of the fine-tuned third encoder, and the output end of the network layer and the output end of the first encoder to be trained are both connected to the first decoder to be trained.

[0119] S208: Using the multilingual voice wake-up training data to train the multilingual wake-up word detection model training network to obtain a trained multilingual wake-up word detection model.

[0120] In the present application, this step can be implemented in a variety of ways, which will be described in detail through the following embodiments.

[0121] In another embodiment of the present application, a method for training the multilingual wake-up word detection model training network using the multilingual voice wake-up training data to obtain a trained multilingual wake-up word detection model is described in detail, and the method is specifically:

[0122] After the multilingual speech wake-up training data is used to input the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, and the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder. The trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

[0123] For ease of understanding, refer to Figure 4 , the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label is , the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label is , in order to adjust the two cross entropy losses ( and ), the two cross entropy losses can be weighted and used as the final loss function to optimize the model. The optimization goal is to minimize the final loss function.

[0124] It should be noted that, during the training process, the adjusted parameters are limited to the parameters of the first encoder to be trained and the first decoder to be trained, and the parameters of the encoder and the network layer in the first wake-up word detection branch are not adjusted.

[0125] In another embodiment of the present application, another implementation method of training the multilingual wake-up word detection model training network using the multilingual voice wake-up training data to obtain a trained multilingual wake-up word detection model is introduced in detail, and the method specifically includes:

[0126] S301: For each training data in the multilingual voice wake-up training data, obtain its category label from the K-Means clusterer and mark it;

[0127] In this application, a K-Means clusterer can be trained first. Specifically, a multilingual speech recognition model encoder is used to extract the embeddings of each speech wake-up audio sample in the multilingual speech wake-up training data, and the K-Means clustering algorithm is applied to these embeddings to divide them into 4096 categories, and then a K-Means clusterer is trained. It should be noted that the annotation can be done at the frame level, that is, each audio frame is annotated with a classification label. Based on this, the category prediction results output by the subsequent classifier are the category prediction results of each audio frame.

[0128] S302: Add a classifier after the first encoder to be trained in the multilingual wake-up word detection model training network;

[0129] For ease of understanding, refer to Figure 5 , Figure 5 This is another schematic diagram of a multilingual wake-up word detection model training network structure disclosed in an embodiment of the present application. Figure 5 As shown, the output end of the first encoder is connected to a classifier.

[0130] S303: After the multilingual speech wake-up training data is input into the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label, and the cross entropy loss between the category prediction result output by the classifier and the category label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain a trained first encoder and a trained first decoder, and the trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

[0131] For ease of understanding, refer to Figure 5 , the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label is , the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label is , the cross entropy loss between the category prediction result output by the classifier and the category label is , in order to adjust the three cross entropy losses ( and and ), the three cross entropy losses can be weighted and used as the final loss function to optimize the model. The optimization goal is to minimize the final loss function.

[0132] It should be noted that, during the training process, the adjusted parameters are limited to the parameters of the first encoder to be trained and the first decoder to be trained, and the parameters of the encoder and the network layer in the first wake-up word detection branch are not adjusted.

[0133] Based on the multilingual wake-up word detection model trained in this way, the detection results are more accurate.

[0134] In addition, in the present application, in order to effectively reduce the training time, the speech wake-up training data of different languages ​​are fused and used to train the multi-language wake-up word detection model training network. Compared with training with single-language speech wake-up training data, fusion training using multi-language speech wake-up training data is likely to lead to increased errors in wake-up word detection results. To solve this problem, in the present application, an initial prior information can be tailored for each language. When the multi-language speech wake-up training data is used to train the multi-language wake-up word detection model training network, the wake-up training data of each language and the initial prior information of the corresponding language are jointly input into the multi-language wake-up word detection model training network for training. During the training process, the initial prior information of each language is optimized synchronously to obtain the prior information of each language.

[0135] As another possible implementation method, the method of using a trained multilingual wake-up word detection model to perform wake-up word detection on the voice signal to obtain a wake-up word detection result includes: obtaining prior information of the language corresponding to the voice signal; concatenating the prior information of the language corresponding to the voice signal and the voice signal, and inputting the trained multilingual wake-up word detection model to obtain a wake-up word detection result. Based on this, the accuracy of the wake-up word detection result can be further improved.

[0136] A voice wake-up method provided in an embodiment of the present application is introduced above, and a device for executing the voice wake-up method is introduced below.

[0137] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a voice wake-up device provided in an embodiment of the present application. Figure 6 As shown, the voice wake-up device includes:

[0138] An acquisition unit 11, used to acquire a speech signal to be recognized;

[0139] a wake-up word detection unit 12, configured to perform wake-up word detection on the speech signal using a trained multilingual wake-up word detection model to obtain a wake-up word detection result, wherein the trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, configured to encode the speech signal to obtain a coding feature, and the first decoder is a lightweight decoder, configured to decode the coding feature to obtain the wake-up word detection result;

[0140] The wake-up result determination unit 13 is used to determine the wake-up result of the voice signal according to the wake-up word detection result.

[0141] In a possible implementation, the voice wake-up device further includes a multilingual wake-up word detection model training unit;

[0142] The multilingual wake-up word detection model training unit includes:

[0143] A training data acquisition unit, used to acquire multilingual speech recognition training data and multilingual speech wake-up training data;

[0144] A multilingual speech recognition model acquisition unit, used to acquire a pre-built multilingual speech recognition model, wherein the multilingual speech recognition model includes a second encoder and a second decoder;

[0145] a multilingual speech recognition model training unit, configured to train the multilingual speech recognition model using the multilingual speech recognition training data to obtain a trained second encoder;

[0146] A third encoder construction unit, configured to construct a third encoder based on the trained second encoder;

[0147] A fine-tuning network construction unit, used to construct a fine-tuning network, wherein the fine-tuning network includes the third encoder, a network layer, and the first decoder to be trained;

[0148] A fine-tuning unit, configured to fine-tune the fine-tuning network using the multilingual speech wake-up training data to obtain a fine-tuned third encoder and network layer;

[0149] a multilingual wake-up word detection model training network construction unit, used to construct a multilingual wake-up word detection model training network, wherein the multilingual wake-up word detection model training network comprises a first wake-up word detection branch and a second wake-up word detection branch, wherein the first wake-up word detection branch comprises the fine-tuned third encoder, the network layer, and the first decoder to be trained, and the second wake-up word detection branch comprises the first encoder to be trained and the first decoder to be trained;

[0150] The multilingual wake-up word detection model training network training unit is used to train the multilingual wake-up word detection model training network using the multilingual speech wake-up training data to obtain a trained multilingual wake-up word detection model.

[0151] In a possible implementation, the third encoder construction unit is specifically configured to:

[0152] Freeze the parameters of the trained second encoder, and add a weight parameter to each hidden layer in the trained second encoder to obtain a second encoder with weight parameters;

[0153] A LORA module is added to the weighted second encoder to obtain the third encoder.

[0154] In a possible implementation, the fine-tuning unit is specifically configured to:

[0155] After the multilingual speech wake-up training data is input into the fine-tuning network, the output weight parameters of each hidden layer in the third encoder and the parameters of the LORA module are fine-tuned by using the cross entropy loss between the wake-up word detection result output by the fine-tuning network and the wake-up label to obtain the fine-tuned third encoder and network layer.

[0156] In a possible implementation, the multilingual wake-up word detection model training network training unit is specifically used to:

[0157] After the multilingual speech wake-up training data is used to input the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, and the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder. The trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

[0158] In a possible implementation, the multilingual wake-up word detection model training network training unit is specifically used to:

[0159] For each training data in the multilingual voice wake-up training data, obtain its category label from the K-Means clusterer and mark it;

[0160] Adding a classifier after the first encoder to be trained in the multilingual wake-up word detection model training network;

[0161] After the multilingual speech wake-up training data is input into the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label, and the cross entropy loss between the category prediction result output by the classifier and the category label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder, and the trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

[0162] In a possible implementation, the wake-up word detection unit is specifically used to:

[0163] Acquiring prior information of a language corresponding to the speech signal;

[0164] After concatenating the prior information of the language corresponding to the speech signal and the speech signal, the information is input into the trained multilingual wake-up word detection model to obtain a wake-up word detection result.

[0165] The present application also provides an electronic device in an embodiment. Figure 7As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0166] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0167] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0168] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any voice wake-up method provided in the embodiment of the present application.

[0169] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any voice wake-up method provided in the embodiment of the present application.

[0170] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0171] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0172] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0173] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A voice wake-up method, characterized in that: include: Acquire a speech signal to be recognized; Performing wake-up word detection on the speech signal using a trained multilingual wake-up word detection model to obtain a wake-up word detection result, wherein the trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, used to encode the speech signal to obtain a coding feature, and the first decoder is a lightweight decoder, used to decode the coding feature to obtain the wake-up word detection result; Determining a wake-up result of the voice signal according to the wake-up word detection result; The training method of the multilingual wake-up word detection model includes: Obtain multi-language speech recognition training data and multi-language speech wake-up training data; Acquire a pre-built multilingual speech recognition model, wherein the multilingual speech recognition model includes a second encoder and a second decoder; Using the multilingual speech recognition training data, training the multilingual speech recognition model to obtain a trained second encoder; Based on the trained second encoder, construct a third encoder; Constructing a fine-tuning network, wherein the fine-tuning network includes the third encoder, a network layer, and the first decoder to be trained; Fine-tune the fine-tuning network using the multilingual voice wake-up training data to obtain a fine-tuned third encoder and network layer; Constructing a multilingual wake-up word detection model training network, wherein the multilingual wake-up word detection model training network comprises a first wake-up word detection branch and a second wake-up word detection branch, wherein the first wake-up word detection branch comprises the fine-tuned third encoder, the network layer, and the first decoder to be trained, and the second wake-up word detection branch comprises the first encoder to be trained and the first decoder to be trained; The multilingual voice wake-up training data is used to train the multilingual wake-up word detection model training network to obtain a trained multilingual wake-up word detection model.

2. The method according to claim 1, characterized in that The step of constructing a third encoder based on the trained second encoder comprises: Freeze the parameters of the trained second encoder, and add a weight parameter to each hidden layer in the trained second encoder to obtain a second encoder with weight parameters; A LORA module is added to the second encoder with weight parameters to obtain the third encoder.

3. The method according to claim 2, characterized in that The method of fine-tuning the fine-tuning network using the multilingual voice wake-up training data to obtain a fine-tuned third encoder and network layer includes: After the multilingual speech wake-up training data is input into the fine-tuning network, the output weight parameters of each hidden layer in the third encoder and the parameters of the LORA module are fine-tuned by using the cross entropy loss between the wake-up word detection result output by the fine-tuning network and the wake-up label to obtain the fine-tuned third encoder and network layer.

4. The method according to claim 1, characterized in that: The method of training the multilingual wake-up word detection model training network using the multilingual voice wake-up training data to obtain a trained multilingual wake-up word detection model includes: After the multilingual speech wake-up training data is used to input the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, and the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder. The trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

5. The method according to claim 1, characterized in that The method of training the multilingual wake-up word detection model training network using the multilingual voice wake-up training data to obtain a trained multilingual wake-up word detection model includes: For each training data in the multilingual voice wake-up training data, obtain its category label from the K-Means clusterer and mark it; Adding a classifier after the first encoder to be trained in the multilingual wake-up word detection model training network; After the multilingual speech wake-up training data is input into the multilingual wake-up word detection model training network, the cross entropy loss between the wake-up word detection result output by the first wake-up word detection branch and the wake-up label, the cross entropy loss between the wake-up word detection result output by the second wake-up word detection branch and the wake-up label, and the cross entropy loss between the category prediction result output by the classifier and the category label are used to adjust the parameters of the first encoder to be trained and the first decoder to be trained to obtain the trained first encoder and the trained first decoder, and the trained first encoder and the trained first decoder are combined to obtain a trained multilingual wake-up word detection model.

6. The method according to claim 1, characterized in that The step of performing wake-up word detection on the speech signal using the trained multilingual wake-up word detection model to obtain a wake-up word detection result includes: Acquiring prior information of a language corresponding to the speech signal; After concatenating the prior information of the language corresponding to the speech signal and the speech signal, the information is input into the trained multilingual wake-up word detection model to obtain a wake-up word detection result.

7. A voice wake-up device, characterized in that: include: An acquisition unit, used for acquiring a speech signal to be recognized; a wake-up word detection unit, configured to perform wake-up word detection on the speech signal using a trained multilingual wake-up word detection model to obtain a wake-up word detection result, wherein the trained multilingual wake-up word detection model includes a trained first encoder and a trained first decoder, the first encoder is a lightweight encoder, configured to encode the speech signal to obtain a coding feature, and the first decoder is a lightweight decoder, configured to decode the coding feature to obtain the wake-up word detection result; A wake-up result determination unit, configured to determine a wake-up result of the voice signal according to the wake-up word detection result; A multilingual wake-up word detection model training unit; the multilingual wake-up word detection model training unit comprises: A training data acquisition unit, used to acquire multilingual speech recognition training data and multilingual speech wake-up training data; A multilingual speech recognition model acquisition unit, used to acquire a pre-built multilingual speech recognition model, wherein the multilingual speech recognition model includes a second encoder and a second decoder; a multilingual speech recognition model training unit, configured to train the multilingual speech recognition model using the multilingual speech recognition training data to obtain a trained second encoder; A third encoder construction unit, configured to construct a third encoder based on the trained second encoder; A fine-tuning network construction unit, used to construct a fine-tuning network, wherein the fine-tuning network includes the third encoder, a network layer, and the first decoder to be trained; A fine-tuning unit, configured to fine-tune the fine-tuning network using the multilingual speech wake-up training data to obtain a fine-tuned third encoder and network layer; a multilingual wake-up word detection model training network construction unit, used to construct a multilingual wake-up word detection model training network, wherein the multilingual wake-up word detection model training network comprises a first wake-up word detection branch and a second wake-up word detection branch, wherein the first wake-up word detection branch comprises the fine-tuned third encoder, the network layer, and the first decoder to be trained, and the second wake-up word detection branch comprises the first encoder to be trained and the first decoder to be trained; The multilingual wake-up word detection model training network training unit is used to train the multilingual wake-up word detection model training network using the multilingual speech wake-up training data to obtain a trained multilingual wake-up word detection model.

8. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the voice wake-up method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the voice wake-up method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The computer storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the voice wake-up method as described in any one of claims 1 to 6.

Images