Voice wake-up method and device for customizing wake-up word

Abstract

The invention provides a voice wake-up method and device for customizing a wake-up word, and the method comprises the steps: voice data are inputted into a voice wake-up model with a customizable size, the voice wake-up model outputs a result used for determining whether to carry out voice wake-up or not, the voice wake-up model can customize the wake-up word, and the voice wake-up model outputs a result used for determining whether to carry out voice wake-up or not; and the size of the voice wake-up model can be determined according to the custom wake-up word length. The voice wake-up model is small in parameter quantity, high in wake-up accuracy, small in computing power consumption and low in energy consumption, can be directly deployed on end-side equipment, and supports a user to customize wake-up words according to requirements.

Description

Technical Field

[0001] The present application relates to the field of speech processing, and more specifically, to a speech wake-up method and device for a custom wake-up word. Background Art

[0002] Voice wake-up technology is a technology that detects predefined wake-up words from continuous sound signals. It allows users to activate devices by saying predefined wake-up words. It is an indispensable part of intelligent voice interaction. KWS can be divided into two categories according to application scenarios, closed-vocabulary KWS and open-vocabulary KWS. Closed-vocabulary KWS means that the model can only recognize predefined keywords, such as Apple's "hey siri", Amazon's "Alexa", Huawei's smart device "Xiaoyi Xiaoyi", etc. Since the wake-up words are fixed, closed-vocabulary KWS generally defines the problem as a binary classification problem (only judge "yes" or "no"), and the wake-up model is also a binary classification model (only used to judge whether it contains predefined wake-up words). Open-vocabulary KWS (OpenKWS) allows users to customize their favorite wake-up words to wake up the device. For example, for Huawei phones, users can set a favorite wake-up word in "Settings" to replace the system's built-in "Xiaoyi Xiaoyi" to wake up the device. In the OpenKWS scenario, since the wake-up words defined by each user may vary greatly, the requirements for the wake-up solution will be more challenging, and the designed wake-up model will be more complex, and the model will have higher parameter volume and computational complexity.

[0003] The current solutions for implementing OpenKWS can be mainly divided into two categories: the first category is based on speech recognition, that is, the speech recognition model converts the speech signal into text units (such as phonemes, characters, etc.), and then retrieves whether it contains the text units corresponding to the target keywords; the second category is the end-to-end voice wake-up solution, which jointly models the speech signal with the keyword text and builds a unified end-to-end model (generally, the model consists of three modules, namely the acoustic model, the keyword encoder and the joint classifier. The first two are used to extract audio features and keyword features, respectively, and the extracted features are then fused and input into the joint classifier to output the final wake-up result). In comparison, the first category is based on speech recognition. The end-to-end voice wake-up technology in the second category has better performance because it fully considers the joint modeling of keywords and speech, and has gradually been adopted by various commercial products. In order to achieve excellent wake-up performance, most current models directly deploy pre-trained acoustic models on the end-side device for inference. The acoustic models relied on by the above two solutions are very powerful and effective, which leads to a large number of parameters and calculations of the acoustic model, which will seriously consume the computing resources of the end-side device and reduce the battery life. Therefore, how to effectively improve the battery life of the end-side device and reduce the energy consumption of the end-side device while ensuring the wake-up performance of the wake-up model is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a voice wake-up method with a customized wake-up word, which is used to reduce computing resource consumption and energy consumption.

[0005] On the first aspect, the present application proposes a voice wake-up method that supports user-defined wake-up words, which can be used in any possible voice interaction scenarios including work computers, mobile smart devices, wearable devices and robots. The method includes: generating a target text unit sequence from the target wake-up word input by the user; extracting a target acoustic model from a pre-trained acoustic model according to the target text unit sequence, the pre-trained acoustic model is composed of multiple sub-acoustic models, and the target acoustic model is composed of some sub-acoustic models in the multiple sub-acoustic models; receiving the voice input by the user; determining whether to respond to the voice according to the target acoustic model.

[0006] In the related technology, the voice wake-up method can realize the online voice wake-up function, allowing the user to wake up the device according to the defined wake-up word, and the voice wake-up function consumes a lot of energy on the end-side device, and the real-time calculation and reasoning of the model will consume a lot of computing resources and energy. The voice wake-up method proposed in this application is a voice wake-up method that supports user-defined wake-up words. This method not only allows users to customize wake-up words according to personal preferences and needs, but also realizes the positive correlation between the length of the defined target wake-up word and the size of the voice wake-up model. The target acoustic model in the voice wake-up model is part of the original acoustic model and is a more lightweight model. Therefore, the wake-up word can be defined according to the requirements of the model size while ensuring the wake-up accuracy, thereby reducing the consumption of computing resources and energy.

[0007] In one possible implementation, the method further includes: obtaining a pre-trained wake-up recognizer; determining whether to respond to the speech according to the target acoustic model includes: determining whether to respond to the speech according to the target acoustic model and the wake-up recognizer, the target acoustic model is used to extract speech features from the speech, and the wake-up recognizer is used to determine whether to respond based on the speech features.

[0008] It can be seen that in the method provided in this embodiment, the target wake-up model includes a target acoustic model and a pre-trained wake-up identifier.

[0009] In one possible implementation, determining whether to respond to the speech based on the target acoustic model and the wake-up recognizer includes: extracting the speech features from the speech using the target acoustic model; comparing the speech features with target speech features using the wake-up recognizer to determine whether to respond to the speech, wherein the target speech features are speech features obtained by encoding the target text unit sequence using a pre-trained text encoder.

[0010] It can be seen that before the voice wake-up service, the defined target wake-up word can be encoded using a pre-trained text encoder to obtain the target voice features, and the target voice features are saved in the terminal device; in the voice service stage, the acoustic model extracts the voice features from the user's input, and the voice features and the target voice features obtained above are fused and input into the wake-up recognizer. The wake-up recognizer compares the input target voice features with the voice features, and determines whether to give a response based on the comparison results.

[0011] In one possible implementation, comparing the voice feature with the target voice feature to determine whether to respond to the voice includes: if the content of the target voice feature contains the content of the voice feature, responding; if the content of the target voice feature does not contain the content of the voice feature, not responding.

[0012] It can be seen that the wake-up recognizer determines whether the voice contains the target wake-up word by comparing the content of the input voice feature with the content of the target voice feature, and by judging whether the content of the voice feature contains the content of the target voice feature, and then determines whether to give a response.

[0013] For example, the wake-up identifier evaluates the probability that the content of the voice feature contains the content of the target voice feature through a probability value, and compares the probability value with a preset threshold. In one way, it can be considered that when the preset threshold is less than the probability value, the content of the voice feature is considered to contain the content of the target voice feature, and a response is given; correspondingly, when the probability value is less than or equal to the preset threshold, the content of the voice feature is considered not to contain the content of the target voice feature, and no response is given.

[0014] In a possible implementation, extracting the target acoustic model from a pre-trained acoustic model according to the target text unit sequence includes: determining the sub-acoustic model corresponding to each text unit in the target text unit sequence according to the acoustic model; and combining the corresponding sub-acoustic models of each text unit in the target text unit sequence into the target acoustic model.

[0015] In a possible implementation, before generating the target text unit sequence from the target wake-up word, the method further includes: receiving a custom wake-up word input by a user; and converting the custom wake-up word into the corresponding target wake-up word.

[0016] In a possible implementation, the target text unit sequence includes but is not limited to phonemes, subphonemes, and Byte Pair Encoder (BPE) encoding.

[0017] In a possible implementation, the size of the target acoustic model is positively correlated with the length of the target text unit sequence.

[0018] In a second aspect, a voice wake-up device is proposed, the device comprising: a text conversion module, used to convert a target wake-up word input by a user into a target text unit sequence; a target acoustic model extraction module, used to extract a target acoustic model from a pre-trained acoustic model according to the target text unit sequence, the pre-trained acoustic model is composed of a plurality of sub-acoustic models, and the target acoustic model is composed of some sub-acoustic models in the plurality of sub-acoustic models;

[0019] A voice receiving module is used to receive the voice input by the user; and a voice wake-up module is used to determine whether to respond to the voice according to the target acoustic model.

[0020] In one possible implementation, the voice wake-up module is also used to determine whether to respond to the voice based on the target acoustic model and a pre-trained wake-up recognizer, the target acoustic model is used to extract voice features from the voice, and the wake-up recognizer is used to determine whether to respond based on the voice features.

[0021] In one possible implementation, the voice wake-up module is also used to: extract the voice features from the voice using the target acoustic model; use the wake-up identifier to compare the voice features with target voice features to determine whether to respond to the voice, and the target voice features are voice features obtained by encoding the target text unit sequence using a pre-trained text encoder.

[0022] In a possible implementation, the voice wake-up module is further used to: respond if the content of the target voice feature includes the content of the voice feature; and not respond if the content of the target voice feature does not include the content of the voice feature.

[0023] In a possible implementation, the target acoustic model extraction module is also used to: determine the sub-acoustic model corresponding to each text unit in the target text unit sequence based on the acoustic model; and compose the target acoustic model with the corresponding sub-acoustic models of each text unit in the target text unit sequence.

[0024] In a possible implementation, the device further includes: a custom wake-up word module, configured to receive a custom wake-up word input by a user, and convert the custom wake-up word into the corresponding target wake-up word.

[0025] In a possible implementation, the target text unit sequence includes but is not limited to phonemes, subphonemes, and Byte Pair Encoder (BPE) encoding.

[0026] In a third aspect, a device is proposed, comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory; wherein, when the program stored in the memory is executed, the processor is used to implement the method described in the first aspect or any possible implementation thereof.

[0027] In a fourth aspect, a computer-readable storage medium is proposed, wherein the computer-readable storage medium stores a computer program. When the computer program runs on a processor, the processor is used to implement the method described in the first aspect or any possible implementation method thereof.

[0028] In a fifth aspect, a computer program product is proposed, characterized in that when the computer program product runs on a processor, the processor is used to implement the method described in the above-mentioned first aspect or any possible implementation method thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is an application scenario of a voice wake-up method with a custom wake-up word provided in an embodiment of the present application.

[0030] Figure 2 It is a simple flow chart of a voice wake-up method with a custom wake-up word provided in an embodiment of the present application.

[0031] Figure 3 It is a schematic diagram of a specific implementation flow of a voice wake-up method with a custom wake-up word provided in an embodiment of the present application.

[0032] Figure 4 A schematic diagram of a joint training acoustic model and decoder provided in an embodiment of the present application;

[0033] Figure 5 It is a flow chart of a custom wake-up word in a voice wake-up method of a custom wake-up word provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described in detail below, and examples of the embodiments of the present application are shown in the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0036] The terms "first", "second", "third", "fourth", etc. (if any) in this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0037] In the embodiments of the present application, "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "at least one of A, B and C" means that (A); (B); (C); (A and B); (A and C); (B and C); or (A, B and C) may exist.

[0038] In order to make the technical problems solved by the present application, the technical solutions adopted and the technical effects achieved more clear, the technical solutions of the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0039] In order to balance the contradiction between low power consumption and high effect requirements of voice wake-up technology, we can start from one aspect, and try to reduce the size of the voice wake-up model and the computing power brought by the voice wake-up model when the effect meets the requirements or when the effect is controllable, thereby reducing the excessive power consumption brought by the online voice wake-up technology. Based on this, the present application provides a voice wake-up method, which is described in detail as follows:

[0040] First, some professional terms that may be used in the following embodiments are introduced.

[0041] Automatic Speech Recognition (ASR): is a technology that converts speech into text. It is one of the key technologies for natural communication between people and machines, and between people. ASR is the entrance for people to interact with smart devices. Its function is to make the device "understand" human language, so that it can complete what humans want it to do based on the recognized content.

[0042] Voice wake-up / Keyword Spotting (KWS): or keyword detection, is a branch of the voice recognition task. It needs to detect a limited number of pre-defined activation words or keywords from a voice stream, without recognizing all voices. This type of technology is the basis for enabling voice interaction capabilities of embedded devices and can be applied to various fields, such as mobile phones, smart speakers, robots, smart homes, car devices, wearable devices, etc. The wake-up word is pre-set. When the device is in sleep mode, it continues to pick up sound and detect the wake-up word. Once the wake-up word is detected, the device switches from sleep mode to working mode and waits for subsequent interaction.

[0043] Acoustic Model: The acoustic model is a knowledge representation of the differences in acoustics, phonetics, environmental variables, speaker gender, accent, etc., while the language model is a knowledge representation of a sequence of words. The acoustic model is used to calculate the probability of the model generating a speech waveform. It is one of the most important parts of a speech recognition system, accounting for most of the computational overhead and determining the performance of the speech recognition system.

[0044] Classifier: The concept of classification is to learn a classification function or construct a classification model (i.e., the classifier we usually mention) based on existing data. This function or model can map the data records in the database to a certain category in the given categories, so as to be applied to data prediction. In short, the classifier is a general term for the methods of classifying samples in data mining, including algorithms such as decision trees, logistic regression, naive Bayes, neural networks, etc. Here, it has the same function as the wake-up recognizer.

[0045] Neural Network: A neural network is an artificial intelligence method used to teach a computer to process data in a way inspired by the human brain. This is a machine learning process called deep learning, which uses interconnected nodes or neurons in a hierarchical structure similar to the human brain. It can create an adaptive system that the computer uses to learn from mistakes and continuously improve.

[0046] Phoneme: It is the smallest speech unit divided according to the natural attributes of speech. Analyzing according to the pronunciation actions in a syllable, one action constitutes one phoneme. Phonemes are divided into two major categories: vowels and consonants. For example, the Chinese syllable "ā" has only one phoneme, "ài" has two phonemes, and "dài" has three phonemes, etc. The method of using the International Phonetic Alphabet to mark speech is called the transcription method, which has two types: broad transcription and narrow transcription.

[0047] Loss Function: The loss function or cost function is a function that maps the value of a random event or its related random variable to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the learning criterion and the optimization problem, that is, the model is solved and evaluated by minimizing the loss function.

[0048] CTC: CTC is the acronym of the English "Connectionist Temporal Classification", which is translated into Chinese as "Connectionist Temporal Classification". Through CTC, the frame sequence of speech in time and the corresponding transcribed text sequence can be automatically aligned during the model training process without annotating the start and end time periods of the appearance of each character or phoneme, so as to directly classify on the time series and avoid the relatively complex alignment operations of traditional algorithms.

[0049] The CTC method is mainly divided into two parts: one is the label alignment and loss function loss calculation during training, and the other is the calculation of the final prediction result during inference decoding. Among them, the decoding part is divided into three types: Greedy, Beam Search and Prefix Beam Search. Among them, the most commonly used is the simple and efficient Greedy method. It is obtained by finding the value with the highest probability at each time step as the output of the final prediction and removing continuous repetitions and blank blocks. The training algorithm based on CTC loss is relatively complex to implement, so it is generally done directly using the existing computing framework.

[0050] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0051] Figure 1 This is an application scenario of a voice wake-up method with a custom wake-up word provided in an embodiment of the present application.

[0052] The present application proposes a voice wake-up method that supports user-defined wake-up words. The method can be used in any scenario where voice interaction may be used, including on platforms such as work computers, mobile smart devices, wearable devices and robots.

[0053] like Figure 1 As shown, in Figure 1 The far right side is the database and server, where the server is mainly used for pre-training models, and the database is used to store training sample data and trained models; the server and database are generally deployed in the cloud, and the trained models can be transmitted to the end-side device through network transmission methods such as routers, so as to facilitate interaction between the end-side device and the user to realize the functions of custom wake-up words and voice wake-up.

[0054] The user implements the custom wake-up word function on the terminal side: the terminal side includes the user and the terminal side device. The terminal side device generally includes an entity that can output personalized voice, which can be a smart terminal (smartphone, smart speaker, etc.), a wearable terminal or a robot. These entities themselves have certain computing capabilities and can complete certain model reasoning and computing tasks. The implementation scenario of the present invention can be a voice editing software running on a desktop computer, smart phone and other platforms, or an application or software with a voice wake-up function integrated on a computer, smart phone or wearable device that can make sound, a smart speaker or a robot that can talk to people.

[0055] In order to simplify the description, the interaction process between the terminal device and the user is described in detail by taking the "terminal" as an example. During the interaction between the terminal and the user, the terminal receives a wake-up word definition request and receives a custom wake-up word manually entered by the user. The terminal system automatically converts the target wake-up word into a target wake-up word pronunciation and returns the target wake-up word pronunciation to the user. The user receives a confirmation request for the target wake-up word pronunciation and completes the confirmation. Finally, the target wake-up word pronunciation and the target wake-up word are stored on the terminal device.

[0056] After determining the target wake-up word, the terminal also provides an online voice wake-up service: the terminal receives the user's interactive voice in real time, compares the received user's interactive voice with the target wake-up word, and gives a corresponding predefined response. Specifically, when the interactive voice input by the user contains the target wake-up word, the terminal device recognizes the target wake-up word and gives a response; when the interactive voice input by the user does not contain the target wake-up word, the terminal device does not respond.

[0057] In order to more clearly describe the voice wake-up method of a custom wake-up word proposed in this application, Figure 2 Take the voice wake-up process as an example to briefly describe it. Figure 2 This is a simple flow chart of a voice wake-up method with a custom wake-up word provided in an embodiment of the present application. It should be noted that this simple flow chart only simply shows a process implementation example of the voice wake-up method. The specific details are not elaborated in detail in this embodiment, and the specific details will be elaborated in detail in the following embodiments. Specifically, the flow chart is divided into the following four steps:

[0058] Step 1: The server generates the initial wake-up model through model training

[0059] The initial wake-up model includes an acoustic model and a voice wake-up model, wherein the acoustic model includes multiple target acoustic models for learning voice features; the voice wake-up model includes a text encoder and a wake-up recognizer for determining whether the input voice data contains the target wake-up word and whether to give a response. The following describes the training process of the acoustic model and the voice wake-up model respectively:

[0060] For acoustic model training, in one implementation, the initial acoustic model and the initial decoder are jointly trained using speech-text annotation data D (D = {Audio, Text}) until the parameters converge to obtain a trained acoustic model. For the sake of convenience, the "initial acoustic model" refers to an acoustic model that has not been trained in parameters, and the "initial decoder" also refers to a decoder that has not been trained in parameters; the acoustic model and decoder obtained after training until the parameters converge are respectively called the "trained acoustic model" and the "trained decoder".

[0061] The following mainly describes the detailed training process of the acoustic model. First, the training data is prepared. The training data refers to the speech text annotation data D (D = {Audio, Text}), where Audio represents speech data and Text represents speech text data. Before the acoustic model is trained, the speech text data needs to be preprocessed to obtain the corresponding speech text sequence. For example, the speech text sequence can be a phoneme sequence. In addition to the training sample data, the parameters of the initial acoustic model need to be initialized. The initialization operation is to allow the network to update the learning features and update the parameters of the network efficiently. Each convolutional layer in the initial acoustic model contains a gating unit, which is used to learn the characteristics of each phoneme, so that each sub-neural network corresponds to the characteristics of each phoneme in the learning phoneme sequence during the training process. After the parameter convergence training is completed, a sub-acoustic model corresponding to each phoneme can be obtained, and the multiple sub-acoustic models are combined into a trained acoustic model. In addition, the acoustic model training process also includes the training of the encoder. The initial encoder and the initial acoustic model are jointly trained until the parameters converge to obtain the trained acoustic model and the trained encoder. The training of the acoustic model is mainly used to learn the speech features of the speech data, and the training of the encoder is mainly used to decode the speech features to obtain decoded speech text data. During the training process, the difference between the decoded speech text data and the real speech text data is continuously reduced by means of a constrained loss function. This process can improve the ability of the acoustic model to learn speech features by improving the decoding ability of the decoder. That is, the acoustic model can only extract more accurate speech features to make the difference between the speech text decoded by the decoder and the real speech text smaller.

[0062] For the speech wake-up model, in one implementation, the speech wake-up model also needs to be obtained through training, but the training sample data of the acoustic model and the speech wake-up model are different. The training sample data of the speech wake-up model is obtained by processing and constructing the speech text annotation data D. Specifically, the wake-up task corpus is constructed from the speech text annotation data D (D = {Audio, Text}), that is, a wake-up word set 〖{Keyword〗_w, w = 1,2,…,W} consisting of words and phrases with a length of four characters (assuming that the wake-up word is specified as four characters, the length setting here needs to be based on the length specified for the specific wake-up word) is randomly selected from the text, then the speech-text pair containing the wake-up word is the positive example of the wake-up word, and the annotation label is set to true, while the speech-text pair excluding the wake-up word is a negative example, and the annotation label is set to false. According to this annotation method, a wake-up training corpus D^' (D^' = {Audio, Text, label}) containing W wake-up words is constructed. Each training corpus in the wake-up training corpus is a triple {text Tkeyword containing the keyword, positive sample speech Apositive of the keyword, negative sample speech Anegative of the keyword}.

[0063] After obtaining the wake-up training corpus, the voice wake-up model can be trained. Before training the voice wake-up model, it is also necessary to load the trained acoustic model and fix the parameters of the trained acoustic model during the training process to use the trained acoustic model to assist in training the initial wake-up model, wherein the initial wake-up model includes an initial text encoder and an initial wake-up recognizer. The trained acoustic model is used to extract the voice features of the voice data, and the initial text encoder is used to learn to encode the preprocessed target voice text into the target voice features. After obtaining the target voice features, the target voice features can be stored in the terminal device. During the online service stage, the voice features extracted from the target acoustic model and the target voice features obtained from the terminal device are fused and input into the initial wake-up recognizer. The initial wake-up recognizer is used to implement the recognition wake-up function. Specifically, the recognition wake-up refers to comparing the voice features with the target voice features, judging whether the content of the voice features contains the content of the target voice features and giving an evaluation index. Exemplarily, a probability value can be used to represent it, and a threshold is pre-set. The threshold can be set according to specific usage requirements and is a value between 0 and 1; then it is determined whether to respond. Specifically, when the probability value is greater than the preset threshold, it is considered that the content of the voice feature contains the content of the target voice feature, that is, it is considered that the online input voice contains the target wake-up word, and a response is given; when the probability value is less than the preset threshold, it is considered that the content of the voice feature does not contain the content of the target voice feature, that is, it is considered that the online input voice does not contain the target wake-up word, and no response is given. Usually the final output result is expressed as on / off, where on means that the current corpus contains the target wake-up word, that is, the probability that the content of the speech feature contains the content of the target speech feature is greater than the preset threshold (such as 0.7), and the result is a response; off means that the current corpus does not contain the target wake-up word, that is, the probability that the content of the speech feature contains the content of the target speech feature is less than the threshold (such as 0.7), and no response is given. The preset threshold can be set according to specific needs.

[0064] The above steps can obtain the trained acoustic model and the trained voice wake-up model. The following is a brief introduction to the steps of custom wake-up.

[0065] Step 2: The terminal generates a target text unit sequence based on the target wake-up word input by the user.

[0066] The custom voice wake-up method can realize the function of user-defined wake-up words on the terminal device. The terminal device can be an interactive terminal device, including a smart phone, a smart speaker, a robot, a smart home, a vehicle-mounted device, a wearable device, etc. It receives the request for the custom wake-up word submitted by the user, and completes the pronunciation confirmation of the custom wake-up word through interaction with the user. The custom wake-up word is confirmed as the target wake-up word and is stored in the terminal device. The terminal system can convert the target wake-up word into a target text unit sequence through a phonetic conversion tool. The target text unit sequence includes but is not limited to phonemes, subphonemes, byte pairs (Byte Pair Encoder, BPE) encoding, or pseudo labels converted by other acoustic wake-up models. After obtaining the target unit sequence, the operation of reorganizing the target wake-up model can be performed.

[0067] Step 3: The terminal reorganizes the initial wake-up model to generate the target wake-up model

[0068] After determining the target wake-up word and the target text unit sequence, the trained acoustic model and the trained speech wake-up model obtained in step one can be assembled based on the target text unit sequence to obtain a target wake-up model.

[0069] In one implementation, the target text unit sequence can be a target phoneme sequence, and each target phoneme has a unique corresponding sub-acoustic model in the acoustic model. The terminal extracts the corresponding trained target sub-acoustic model from the trained acoustic model according to the target phoneme sequence, and combines these target sub-acoustic models to obtain the target acoustic model. Therefore, the parameter amount of the target acoustic model is related to the target phoneme sequence, that is, the parameter amount of the target acoustic model is related to the defined target wake-up word.

[0070] The target acoustic model obtained is combined with the trained wake-up identifier to obtain a target wake-up model, the parameter quantity of the target acoustic model is positively correlated with the target wake-up word, and the target wake-up model can be applied to the terminal device to realize the online voice wake-up service.

[0071] Step 4: The terminal uses the target wake-up model to implement online voice wake-up service.

[0072] The target wake-up model runs on the interactive end-side device to implement the online voice wake-up function. Specifically, the end-side device receives the interactive voice input by the user in real time, and uses the target wake-up model to perform inference calculations to determine whether a response is required. If the interactive voice contains the target wake-up word, a response is given; if the interactive voice does not contain the target wake-up word, no response is given.

[0073] The above briefly introduces a simple flowchart of a voice wake-up method for a custom wake-up word proposed in this application. The following describes the specific implementation details of the voice wake-up method for a custom wake-up word. Figure 3 For example,

[0074] Figure 3 This is a schematic diagram of a specific implementation process of a voice wake-up method with a custom wake-up word provided in an embodiment of the present application. The process includes model training 100, definition of wake-up words 200, and online wake-up 300. The following is an introduction to each stage of the process:

[0075] Model training 100

[0076] This stage is generally performed on a cloud server. The model training 100 stage includes two key steps: training the acoustic model and training the voice wake-up model. The following is an introduction to the corresponding steps:

[0077] S1. Train the acoustic model.

[0078] This stage is mainly based on the annotated data D (D = {Audio, Text}) of the speech text as data samples to jointly train the acoustic model and the decoder. The "Audio" in the sample data is the speech data, and the "Text" is the speech text annotated data corresponding to the speech data. The acoustic model is used to learn the speech features of the speech data, and the encoder is used to decode the speech features to obtain the decoded speech text data. During the training process, the difference between the decoded speech text data and the real speech text data is continuously reduced by means of a constrained loss function. This process can improve the ability of the acoustic model to learn speech features by improving the decoding ability of the decoder, that is, the acoustic model can only extract more accurate speech features to reduce the difference between the speech text decoded by the decoder and the real speech text.

[0079] In order to more rigorously and clearly distinguish the acoustic model and sub-acoustic model before and after training, in the description of the training process, the acoustic model and sub-acoustic model before training are recorded as "initial acoustic model" and "sub-neural network", and the acoustic model and sub-acoustic model after training are respectively called: "trained acoustic model" and "trained sub-acoustic model"; the decoder before training is called "initial decoder", and the decoders after training are respectively called "trained decoders".

[0080] In one implementation, the speech text annotation data D (D = {Audio, Text}) is first preprocessed to obtain the processed speech text data. For a simple example, assume the speech text data is "Hello, Xiaoyi". After passing through the data preprocessing process, this speech text data obtains a phoneme sequence {"n i3 h ao3 x i ao3 i4"}, a total of 8 phonemes. These 8 phonemes include types of initials, finals + tones (where 3 represents the third tone and 4 represents the fourth tone). The phoneme sequence is used as a training sample and fed into the initial acoustic model for parameter learning. The initial acoustic model includes a gated unit, which is inserted after each convolutional layer of the initial acoustic model, dividing the initial acoustic model into individual sub-neural networks. These individual sub-neural networks continuously learn the features of each phoneme during the training process to generate a sub-acoustic model AME corresponding to each phoneme. For example, it can be learned respectively that: sub-acoustic model AME1 corresponds to "n", AME2 corresponds to "i3", AME3 corresponds to "h", AME4 corresponds to "ao3", AME5 corresponds to "x", AME6 corresponds to "i", AME7 corresponds to "ao3", and AME8 corresponds to "i4". Therefore, the trained acoustic model AM contains the trained sub-acoustic models AMEi corresponding to each phoneme, where i is the number of phonemes. After the initial acoustic model is trained, the trained acoustic model AM and the trained sub-acoustic models AME can be obtained, and each of the trained sub-acoustic models AME exists in the trained acoustic model AM.

[0081] Specifically, to describe this process more clearly, take Figure 4 as an example to expand the details in depth. Figure 4 FIG. is a schematic diagram of jointly training an acoustic model and a decoder provided by an embodiment of the present application. Among them, Audio represents speech data, Text represents the true speech text data corresponding to the speech data. In this embodiment, the true speech text data is a phoneme sequence, that is, obtained by preprocessing the true speech text data. Acoustic model represents an acoustic model (for distinction, it is the "initial acoustic model" before training and the "trained acoustic model" after training). Acoustic feature represents the speech feature extracted by the acoustic model from the input speech data. Decoder is a decoder (for distinction, it is the "initial decoder" before training and the "trained decoder" after training). The decoder outputs the speech text data obtained by decoding the speech feature. There is a difference between this speech text data and the true speech text data. The source of this difference is the error of the speech feature extracted by the acoustic model and the decoding error of the decoder. Therefore, a loss function is needed during the training process to constrain and reduce each error.

[0082] In this embodiment, the speech data ("Hello Xiaoyi") is input into the initial acoustic model, the initial acoustic model learns the speech features of the speech data, and performs data preprocessing on the speech text data "Hello Xiaoyi" to obtain the corresponding phoneme sequence data {"n i3 h ao3 x i4"}. After the initial decoder receives the phoneme sequence data and the speech features, it learns the correspondence between the speech data and the phoneme data, so that the input speech data can be accurately decoded into its corresponding phoneme sequence. The loss function is used to constrain the training parameters of the initial acoustic model, so that the trained acoustic model can accurately extract the effective features of the speech data, which is conducive to the trained decoder to accurately decode the obtained speech features into the corresponding phoneme sequence.

[0083] First, there is the process of sample preparation and parameter initialization. Here, the number of phonemes is p (p=6) for explanation. The phoneme sample data used for training contains p, and the initial sub-acoustic model is AMEi (i=1,2,…,p). Before model training, the initial acoustic model AM, the initial decoder, and the initial sub-acoustic model AMEi (i=1,2,…,p) are initialized respectively. Specifically, the parameters of the model can be initialized by random initialization for the initial acoustic model AM and the initial decoder. For the initial sub-acoustic model AMEi (i=1,2,…,p), it is randomly assigned 0 or 1 according to the target sparsity (it can be a hyperparameter used to indicate sparsity. For example, if =10%, then about 10% of the positions in AMEi (the value of i is less than or equal to the number of phonemes) are assigned 1, and the others are 0).

[0084] In fact, it is a training process. During the training process, the input of the initial acoustic model AM is speech data, and the output is the speech features corresponding to the speech data; the input of the initial encoder is the real phoneme sequence and speech features, and the output is On / Off, where On means that the phoneme sequence obtained by decoding the speech features contains the real phoneme sequence, that is, a response is given; Off means that the phoneme sequence obtained by decoding the speech features does not contain the real phoneme sequence, that is, no response is given. The initial decoder can obtain the phoneme sequence obtained by decoding the speech features. There is a difference between the phoneme sequence and the real phoneme sequence. The source of the difference is the error of the speech features extracted by the acoustic model and the decoding error of the decoder. Therefore, a loss function is needed in the training process to constrain and reduce the various errors. Here, the Loss loss function is used to constrain the parameters in the initial acoustic model A and the initial encoder. Specifically, the Loss loss function includes L CTC , L sim , L sparsity , and Loss = L CTC +L sim +L sparsity , where LCTC It represents the loss between the phoneme sequence predicted by the decoder and the real phoneme sequence, which is used to constrain the error between the predicted phoneme sequence and the real phoneme sequence. For example, the CTC loss function (Connectionist Temporal Classification loss) can be used. sim is the contrast loss between sub-acoustic models, which is used to constrain the same category of phonemes in training Statistics i The number of activated (non-zero) parameters in the (i-th sub-neural network); k represents the total number of parameters of the entire acoustic model; and Sparity target Represents the target sparsity after pruning, which can be set as needed. In this embodiment, it is set to 0.5. Since the value of Mi is a floating-point probability value rather than a binary value of 0 or 1 (binary, meaning only two values, either 0 or 1) in the calculation process, a threshold γ can be set. When the probability value is greater than γ, the final output is 1, otherwise the final output is 0.

[0085] In order to better constrain the parameters of each sub-acoustic model, the momentum update method is used to update the parameters of each initial sub-acoustic model AMEi, that is, 〖AME〗_(i)=〖momentum*AME〗_(i)+(1-momentum)*M_i, where momentum is the preset momentum parameter.

[0086] Finally, the trained acoustic model can be obtained by training the sample data until the parameters converge, and the trained acoustic model includes the trained sub-acoustic model.

[0087] S2. Train the voice wake-up model.

[0088] In this step, first, the wake-up task corpus (labeled positive and negative samples) is constructed from the speech text annotation data D (D = {Audio, Text}), that is, a wake-up word set 〖{Keyword〗_w, w = 1, 2, ..., W} consisting of words and phrases with a length of four characters (assuming that the wake-up word is specified as four characters, the length setting here needs to be based on the length specified by the specific wake-up word) is randomly selected from the text, then the speech text pair containing the wake-up word is the positive sample of the wake-up word, and the label is set to true, while the speech text pair excluding the wake-up word is a negative sample, and the label is set to false. According to this labeling method, a wake-up training corpus set D^' (D^' = {Audio, Text, label}) containing W wake-up words is constructed, where each training corpus in the wake-up corpus is a triple {text containing the keyword Tkeyword, positive sample speech of the keyword Apositive, negative sample speech of the keyword Anegative}.

[0089] The voice wake-up model is trained using the wake-up training corpus D^' (D^' = {Audio, Text, label}). The voice wake-up model includes an initial text encoder and an initial wake-up recognizer. Before training, it is necessary to load the trained acoustic model and fix the parameters of the acoustic model to implement parameter training of the initial text encoder and the initial wake-up recognizer. Specifically, a voice data Audio is input into the acoustic model. The acoustic model is used to learn the voice features corresponding to the voice data Audio, and the voice text annotation data Text corresponding to the voice data Audio is input into the initial text encoder. The initial text encoder is used to learn to encode the voice text annotation data to obtain the target voice features, which can be stored on the end-side device. In the online service stage, the voice features and the target voice features are fused and input into the initial wake-up recognizer. The initial wake-up recognizer compares the content of the voice features with the content of the target voice features to determine whether the content of the voice features contains the content of the target voice features, thereby determining whether to respond. During the inference process, the wake-up recognizer can identify whether the input speech contains the predefined target wake-up word, and then determine whether a response is required (On corresponds to giving a response, and Off corresponds to not responding).

[0090] Secondly, before training, you need to first w Perform text-to-phoneme transformation, assuming that the phoneme sequence after text conversion is {P w1 , P w2 , ..., P wj}, then when the positive and negative sample data of the keyword are input, its actual parameters and the parameters of the calculated initial acoustic model AM can be dynamically determined by the initial sub-acoustic models corresponding to the phoneme sequence. Specifically, it is expressed as This expression represents the union of the activated parameters corresponding to all the phonemes (W phonemes). The loss function L loss function is used for constraint during the training process of the voice wake-up model. The L loss function can be defined as:

[0091] 〖L=CE(Classifier(f〗_postive,f_keyword),1)+〖CE(Classifier(f〗_negative,f_keyword),0), where f_postive, f_negative and f_keyword are respectively the positive sample speech features, negative sample speech features and the keyword text features extracted by the text encoder after the trained acoustic model AM. Classifier is the wake-up identifier, and CE represents the cross entropy loss function. This loss function is mainly used to constrain the difference between the predicted phoneme and the target phoneme. The larger the output value of the loss function, the greater the difference between the predicted result and the target result. Therefore, during the training process, the speech wake-up model is continuously trained with the goal of reducing the loss function until convergence.

[0092] Define wake-up word 200

[0093] This stage is mainly used to define the wake-up words and reorganize the target wake-up model based on the target wake-up words.

[0094] First, the user on the terminal side interacts with the terminal device to implement the operation of customizing the wake-up word. Specifically, as described in step S3:

[0095] S3. Customize the wake-up word.

[0096] Set a custom wake-up word input by the user on the client side. The main process includes:

[0097] In one implementation, the terminal receives a wake-up word definition request and receives a custom wake-up word manually input by a user. The terminal system uses a character-to-phonetic conversion tool to automatically convert the target wake-up word into a target wake-up word voice, and returns the target wake-up word voice to the user. The user receives a pronunciation confirmation request for the target wake-up word voice and completes the confirmation. Finally, the target wake-up word and the target wake-up word voice are stored on the terminal device.

[0098] In another implementation, this embodiment takes the wake-up word "Hello Xiaowei" with ambiguous pronunciation as an example to describe the pronunciation confirmation process of the wake-up word voice in this technical solution. Existing custom wake-up word solutions generally require users to provide at least one piece of voice reading the wake-up word when defining the wake-up word, and the system then determines the correct pronunciation of the wake-up word based on the matching of the voice and the wake-up word text. This embodiment provides an implementation method that does not require users to provide wake-up word voice. When the system receives the wake-up word "Hello Xiaowei", it first uses a phonetic conversion tool to convert it into a text pronunciation sequence (such as a pinyin phoneme sequence). Since there are two polyphonic characters, "hao" and "wei", in it, there will be four pronunciation combinations: |{hao3 hao4}|×|{wei2 wei4}|. At the same time, the phonetic conversion tool will also give the priority order of different combinations when predicting the pinyin of the text voice. Thus, the possible order of these four pronunciation combinations may be, {{n i2 h ao3 x iao3 w ei2}, {n i2 hao3 x iao3 w ei4}, {n i2 hao4 x iao3 w ei2}, {n i2 h ao4 x iao3 w ei4}}.

[0099] After confirming the pronunciation set, the system will sequentially return each pronunciation combination to the user for confirmation. The presentation method of the returned pronunciation is not limited to pinyin text, using the system's speech generation (TTS) tool to synthesize the voice of phonemes, etc. If the current pronunciation is not confirmed by the user, the next pronunciation will be returned. If all pronunciations are not confirmed by the user, the user will be reminded to change a wake-up word. In this example, if the pronunciation expected by the user is {ni2 h ao3 x iao3 w ei4}, the first pronunciation result returned by the system, {n i2 h ao3 x iao3 w ei2}, will not be adopted by the user. Then, after the system returns the next pronunciation, the user confirmation can be obtained at this time. Finally, the system constructs the wake-up model corresponding to the wake-up word "Hello Xiaowei" based on these phonemes {n i2 h ao3 x iao3 w ei4}.

[0100] After obtaining the target wake-up word, the target wake-up model can be assembled based on the target wake-up word. The specific implementation is as described in S4 below:

[0101] S4. Assemble the target wake-up model.

[0102] Replace the target wake-up word with a target text unit. For example, the target text unit can be a target phoneme sequence or a target syllable sequence. Here, take the target phoneme sequence as an example: Suppose the user newly defines a wake-up word, such as "Hello, Xiaoyi". After the system receives the request for defining the wake-up word, it first uses a Chinese-character-to-phoneme conversion tool to convert the target wake-up word into a target phoneme sequence, that is, {n i2 h ao3 x iao3 i4} (a total of 8 phonemes). This target phoneme sequence includes initials, finals + tones (3 represents the third tone, 4 represents the fourth tone). Each target phoneme can uniquely determine a target acoustic model AME. Finally, 8 target acoustic models corresponding to this target phoneme sequence are obtained: the sub-acoustic model AME1 corresponds to "n", AME2 corresponds to "i3", AME3 corresponds to "h", AME4 corresponds to "ao3", AME5 corresponds to "x", AME6 corresponds to "i", AME7 corresponds to "ao3", and AME8 corresponds to "i4". Combine these 8 sub-acoustic models to obtain a target acoustic model, and obtain a trained text encoder and a trained wake-up recognizer from the voice wake-up model. Combine the above target acoustic model and the wake-up recognizer to obtain a target wake-up model.

[0103] Online Wake-up 300

[0104] After obtaining the target wake-up word, deploy the target wake-up model on the edge device to implement the online voice wake-up service. The specific implementation is as described in S5 below:

[0105] S5. Online Voice Wake-up Service.

[0106] The terminal executes the online voice wake-up service, and the edge device receives the user's interactive voice in real time and uses the target wake-up model to perform inference calculations to obtain the response result of the voice, so as to supply the terminal device to make corresponding interactive actions. Specifically, when the received interactive voice contains the target wake-up word, the terminal device gives a response; when the received interactive voice does not contain the target wake-up word, the terminal device does not give a response.

[0107] To better elaborate on the specific process of how the voice wake-up model implements the online wake-up service in the edge device, as Figure 5 shown, Figure 5It is a schematic diagram of an online voice wake-up service provided by an embodiment of the present application, wherein Acoustic model is a trained acoustic model, Keyword Acoustic model is a target acoustic model obtained by combining multiple sub-acoustic models corresponding to the target wake-up word, Keyword Encoder is a trained text encoder, and Joint classifier is a trained wake-up recognizer; Audio is the input voice data, Keyword is a phoneme sequence obtained by data preprocessing of the target wake-up word, and Acoustic Feature is the voice feature of the voice data extracted by the target acoustic model.

[0108] The following is an elaboration of this embodiment. The acoustic model is extracted according to the custom target wake-up word to obtain a target acoustic model. The target acoustic model is combined with the wake-up recognizer to form a target wake-up model and deployed on the terminal device. Before the target wake-up model is deployed on the terminal device, the target phoneme sequence needs to be encoded using a text encoder to obtain the target voice feature, which can be obtained online in real time. This situation generally consumes more computing resources; in addition, another way is to store the obtained target voice feature in the terminal device, which can be obtained from the terminal device in real time during the online service stage.

[0109] Figure 5 What is shown in the figure is a scenario where the target voice features are stored in the terminal device. Specifically, when in the online service state, the terminal device obtains online voice data in real time, and the target acoustic model is used to extract the voice features of the voice data. Then, the voice features obtained from the target acoustic model and the target voice features obtained from the memory of the terminal device are fused and input into the wake-up recognizer. The wake-up recognizer compares the content of the voice features with the content of the target voice features to determine whether the content of the voice feature contains the content of the target voice features, and then determines whether to respond. If the target voice text data contains the voice text data, it is considered that the voice data contains the target wake-up word, and the terminal device responds in a preset manner; if the comparison result is that the voice text data does not contain the target voice text data, it is considered that the voice data does not contain the target wake-up word, and the terminal device does not respond.

[0110] The above descriptions are only several embodiments of the present invention. Those skilled in the art may make various changes or modifications to the present invention based on the disclosure of the application documents without departing from the spirit and scope of the present invention.

Claims

1. A voice wake-up method, characterized in that: include: Generate a target text unit sequence from the target wake-up word input by the user; Extracting a target acoustic model from a pre-trained acoustic model according to the target text unit sequence, wherein the pre-trained acoustic model is composed of a plurality of sub-acoustic models, and the target acoustic model is composed of some sub-acoustic models in the plurality of sub-acoustic models; Receiving a voice input by the user; Determining whether to respond to the speech is performed according to the target acoustic model.

2. The method according to claim 1, characterized in that The method further comprises: Get a pre-trained wakeup recognizer; The determining whether to respond to the speech according to the target acoustic model comprises: Determining whether to respond to the speech is performed according to the target acoustic model and the wake-up recognizer, wherein the target acoustic model is used to extract speech features from the speech, and the wake-up recognizer is used to determine whether to respond based on the speech features.

3. The method according to claim 2, characterized in that The determining whether to respond to the voice according to the target acoustic model and the wake-up identifier comprises: extracting the speech feature from the speech using the target acoustic model; The wake-up identifier is used to compare the speech feature with a target speech feature to determine whether to respond to the speech, wherein the target speech feature is a speech feature obtained by encoding the target text unit sequence using a pre-trained text encoder.

4. The method according to claim 3, characterized in that The comparing the speech feature with the target speech feature to determine whether to respond to the speech comprises: If the content of the target voice feature includes the content of the voice feature, responding; If the content of the target voice feature does not include the content of the voice feature, no response is made.

5. The method according to any one of claims 1 to 4, characterized in that: The step of extracting a target acoustic model from a pre-trained acoustic model according to the target text unit sequence comprises: Determining a sub-acoustic model corresponding to each text unit in the target text unit sequence according to the acoustic model; The corresponding sub-acoustic model of each text unit in the target text unit sequence is combined into the target acoustic model.

6. The method according to claims 1-5, characterized in that: Before generating the target text unit sequence from the target wake-up word, the method further includes: Receive a custom wake-up word input by the user; The custom wake-up word is converted into the corresponding target wake-up word.

7. The method according to any one of claims 1 to 6, characterized in that The target text unit sequence includes but is not limited to phonemes, subphonemes, and Byte Pair Encoder (BPE) encoding.

8. The method according to claims 1-7, characterized in that The size of the target acoustic model is positively correlated with the length of the target text unit sequence.

9. A voice wake-up device, characterized in that: include: A text conversion module, used to convert the target wake-up word input by the user into a target text unit sequence; A target acoustic model extraction module, used for extracting a target acoustic model from a pre-trained acoustic model according to the target text unit sequence, wherein the pre-trained acoustic model is composed of a plurality of sub-acoustic models, and the target acoustic model is composed of some sub-acoustic models in the plurality of sub-acoustic models; A voice receiving module, used for receiving the voice input by the user; A voice wake-up module is used to determine whether to respond to the voice according to the target acoustic model.

10. The device according to claim 9, characterized in that The voice wake-up module is also used for: Determining whether to respond to the speech is based on the target acoustic model and a pre-trained wake-up recognizer, wherein the target acoustic model is used to extract speech features from the speech, and the wake-up recognizer is used to determine whether to respond based on the speech features.

11. The device according to claim 10, characterized in that The voice wake-up module is also used for: extracting the speech feature from the speech using the target acoustic model; The wake-up identifier is used to compare the speech feature with a target speech feature to determine whether to respond to the speech, wherein the target speech feature is a speech feature obtained by encoding the target text unit sequence using a pre-trained text encoder.

12. The device according to claim 11, characterized in that The voice wake-up module is also used for: If the content of the target voice feature includes the content of the voice feature, responding; If the content of the target voice feature does not include the content of the voice feature, no response is made.

13. The device according to any one of claims 9 to 12, characterized in that: The target acoustic model extraction module is also used for: Determining a sub-acoustic model corresponding to each text unit in the target text unit sequence according to the acoustic model; The corresponding sub-acoustic model of each text unit in the target text unit sequence is combined into the target acoustic model.

14. The device according to claims 9-13, characterized in that The device also includes: The custom wake-up word module is used to receive a custom wake-up word input by a user and convert the custom wake-up word into the corresponding target wake-up word.

15. The device according to any one of claims 9 to 14, characterized in that The target text unit sequence includes but is not limited to phonemes, subphonemes, and Byte Pair Encoder (BPE) encoding.

16. The device according to claims 9-15, characterized in that The size of the target acoustic model is positively correlated with the length of the target text unit sequence.

17. A device, characterized in that include: at least one memory for storing a program; at least one processor, configured to execute the program stored in the memory; Wherein, when the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1-8.

18. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on a processor, the processor is caused to execute the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that When the computer program product runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 8.

Images