Voice wake-up method and device supporting wake-up word customization

Abstract

The present application provides a voice wake-up method and device supporting wake-up word customization. The method comprises: inputting voice data into a voice wake-up model having a customizable size, and the voice wake-up model outputting a result for determining whether to perform voice wake-up, wherein a wake-up word can be customized in the voice wake-up model, and the size of the voice wake-up model is determined according to the length of the customized wake-up word. The voice wake-up model is characterized in small parameter quantity, high wake-up accuracy, low computing power consumption and low energy consumption, is directly deployed on a terminal-side apparatus, and supports a user to customize a wake-up word according to requirements.

Description

A voice wake-up method and device for custom wake-up words Technical Field

[0001] The present application relates to the field of speech processing, and more specifically, to a speech wake-up method and device for customizing a wake-up word. Background Art

[0002] Voice wake-up technology detects predefined wake-up words from continuous sound signals, allowing users to activate devices by speaking the predefined wake-up words. It is an essential component of intelligent voice interaction. Key words (KWS) can be categorized into two types based on application scenarios: closed-vocabulary KWS and open-vocabulary KWS. Closed-vocabulary KWS models only recognize predefined keywords, such as Apple's "hey siri," Amazon's "Alexa," and Huawei smart devices' "Xiaoyi Xiaoyi." Because the wake-up words are fixed, closed-vocabulary KWS typically defines the problem as a binary classification problem (determining only "yes" or "no"), and the wake-up model is also a binary classification model (determining only whether the predefined wake-up word is present). Open-vocabulary KWS (OpenKWS) allows users to customize their own wake-up words to wake up the device. For example, on Huawei phones, users can set a preferred wake-up word in Settings to replace the default "Xiaoyi Xiaoyi" to wake up the device. In the OpenKWS scenario, since the wake-up words defined by each user may vary greatly, the requirements for the wake-up solution will be more challenging, and the designed wake-up model will be more complex, with higher parameter and computational complexity.

[0003] The current solutions for implementing OpenKWS can be mainly divided into two categories: the first category is based on speech recognition, that is, the speech recognition model converts the speech signal into text units (such as phonemes, characters, etc.), and then searches for the text units corresponding to the target keywords; the second category is the end-to-end voice wake-up solution, which jointly models the speech signal with the keyword text and builds a unified end-to-end model (generally, the model consists of three modules, namely the acoustic model, the keyword encoder and the joint classifier. The first two are used to extract audio features and keyword features, respectively. The extracted features are then fused and input into the joint classifier to output the final wake-up result). In comparison, the first category is based on speech recognition. The end-to-end voice wake-up technology in the second category has better performance because it fully considers the joint modeling of keywords and speech, and has gradually been adopted by various commercial products. To achieve excellent wake-up performance, current models typically deploy pre-trained acoustic models directly on the device for inference. Both of these solutions rely on powerful and efficient acoustic models, which results in a large number of acoustic model parameters and computational complexity, severely draining the device's computing resources and reducing battery life. Therefore, effectively improving device battery life and reducing energy consumption while ensuring the wake-up model's performance is a pressing issue.

[0004] Summary of the Invention

[0005] The present application provides a voice wake-up method with a custom wake-up word, which is used to reduce computing resource consumption and energy consumption.

[0006] On the first aspect, the present application proposes a voice wake-up method that supports user-defined wake-up words. The method can be used in any possible voice interaction scenario including platforms such as work computers, mobile smart devices, wearable devices and robots. The method includes: generating a target text unit sequence from the target wake-up word input by the user; extracting a target acoustic model from a pre-trained acoustic model according to the target text unit sequence, the pre-trained acoustic model is composed of multiple sub-acoustic models, and the target acoustic model is composed of some sub-acoustic models in the multiple sub-acoustic models; receiving the voice input by the user; and determining whether to respond to the voice according to the target acoustic model.

[0007] In the related art, the voice wake-up method can realize the online voice wake-up function, allowing the user to wake up the device according to the defined wake-up words, and the voice wake-up function consumes a lot of energy on the end-side device, and the real-time calculation and inference of the model will consume a lot of computing resources and energy. The voice wake-up method proposed in this application is a voice wake-up method that supports user-defined wake-up words. This method not only allows users to customize wake-up words according to personal preferences and needs, but also realizes that the length of the defined target wake-up words is positively correlated with the size of the voice wake-up model. The target acoustic model in the voice wake-up model is part of the original acoustic model and is a more lightweight model. Therefore, the wake-up words can be defined according to the requirements of the model size while ensuring the wake-up accuracy, thereby reducing the consumption of computing resources and energy.

[0008] In one possible implementation, the method further includes: obtaining a pre-trained wake-up recognizer; and determining whether to respond to the speech based on the target acoustic model includes: determining whether to respond to the speech based on the target acoustic model and the wake-up recognizer, the target acoustic model being used to extract speech features from the speech, and the wake-up recognizer being used to determine whether to respond based on the speech features.

[0009] It can be seen that in the method provided in this embodiment, the target wake-up model includes a target acoustic model and a pre-trained wake-up identifier.

[0010] In one possible implementation, determining whether to respond to the speech based on the target acoustic model and the wake-up recognizer includes: extracting the speech features from the speech using the target acoustic model; and comparing the speech features with target speech features using the wake-up recognizer to determine whether to respond to the speech, where the target speech features are speech features obtained by encoding the target text unit sequence using a pre-trained text encoder.

[0011] It can be seen that before the voice wake-up service, the defined target wake-up word can be encoded using a pre-trained text encoder to obtain the target voice features, and the target voice features are saved in the terminal device; in the pre-voice service stage, the acoustic model extracts the voice features from the user input, and the voice features and the target voice features obtained above are fused and input into the wake-up recognizer. The wake-up recognizer compares the input target voice features with the voice features, and determines whether to give a response based on the comparison results.

[0012] In one possible implementation, comparing the voice feature with the target voice feature to determine whether to respond to the voice includes: responding if the content of the target voice feature contains the content of the voice feature; and not responding if the content of the target voice feature does not contain the content of the voice feature.

[0013] It can be seen that the wake-up recognizer determines whether the voice contains the target wake-up word by comparing the content of the input voice feature with the content of the target voice feature, and by judging whether the content of the voice feature contains the content of the target voice feature, and then determines whether to give a response.

[0014] For example, the wake-up identifier evaluates the probability that the content of the voice feature contains the content of the target voice feature through a probability value, and compares the probability value with a preset threshold. In one way, it can be considered that when the preset threshold is less than the probability value, the content of the voice feature is considered to contain the content of the target voice feature, and a response is given; correspondingly, when the probability value is less than or equal to the preset threshold, the content of the voice feature is considered not to contain the content of the target voice feature, and no response is given.

[0015] In one possible implementation, extracting a target acoustic model from a pre-trained acoustic model based on the target text unit sequence includes: determining a sub-acoustic model corresponding to each text unit in the target text unit sequence based on the acoustic model; and combining the sub-acoustic models corresponding to each text unit in the target text unit sequence into the target acoustic model.

[0016] In a possible implementation, before generating the target text unit sequence from the target wake-up word, the method further includes: receiving a custom wake-up word input by a user; and converting the custom wake-up word into the corresponding target wake-up word.

[0017] In a possible implementation, the target text unit sequence includes but is not limited to phonemes, subphonemes, and Byte Pair Encoder (BPE) encoding.

[0018] In a possible implementation, the size of the target acoustic model is positively correlated with the length of the target text unit sequence.

[0019] In a second aspect, a voice wake-up device is proposed, comprising: a text conversion module for converting a target wake-up word input by a user into a target text unit sequence; a target acoustic model extraction module for extracting a target acoustic model from a pre-trained acoustic model based on the target text unit sequence, wherein the pre-trained acoustic model is composed of multiple sub-acoustic models, and the target acoustic model is composed of some sub-acoustic models among the multiple sub-acoustic models;

[0020] A voice receiving module is used to receive the voice input by the user; and a voice wake-up module is used to determine whether to respond to the voice based on the target acoustic model.

[0021] In one possible implementation, the voice wake-up module is further used to determine whether to respond to the voice based on the target acoustic model and a pre-trained wake-up recognizer, wherein the target acoustic model is used to extract voice features from the voice, and the wake-up recognizer is used to determine whether to respond based on the voice features.

[0022] In one possible implementation, the voice wake-up module is further used to: extract the voice features from the voice using the target acoustic model; and compare the voice features with target voice features using the wake-up recognizer to determine whether to respond to the voice, where the target voice features are voice features obtained by encoding the target text unit sequence using a pre-trained text encoder.

[0023] In a possible implementation, the voice wake-up module is further configured to: respond if the content of the target voice feature includes the content of the voice feature; and not respond if the content of the target voice feature does not include the content of the voice feature.

[0024] In a possible implementation, the target acoustic model extraction module is also used to: determine the sub-acoustic model corresponding to each text unit in the target text unit sequence based on the acoustic model; and combine the sub-acoustic models corresponding to each text unit in the target text unit sequence into the target acoustic model.

[0025] In a possible implementation, the device further includes: a custom wake-up word module, configured to receive a custom wake-up word input by a user and convert the custom wake-up word into the corresponding target wake-up word.

[0026] In a possible implementation, the target text unit sequence includes but is not limited to phonemes, subphonemes, and Byte Pair Encoder (BPE) encoding.

[0027] In a third aspect, a device is proposed, comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory; wherein, when the program stored in the memory is executed, the processor is used to implement the method described in the first aspect or any possible implementation thereof.

[0028] In a fourth aspect, a computer-readable storage medium is proposed, which stores a computer program. When the computer program runs on a processor, the processor is used to implement the method described in the first aspect or any possible implementation method thereof.

[0029] In a fifth aspect, a computer program product is proposed, characterized in that when the computer program product runs on a processor, the processor is used to implement the method described in the above-mentioned first aspect or any possible implementation method thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] FIG1 is a diagram showing an application scenario of a voice wake-up method using a custom wake-up word according to an embodiment of the present application.

[0031] FIG2 is a simple flow chart of a voice wake-up method using a custom wake-up word provided in an embodiment of the present application.

[0032] FIG3 is a schematic diagram of a specific implementation flow of a voice wake-up method using a custom wake-up word provided in an embodiment of the present application.

[0033] FIG4 is a schematic diagram of a joint training acoustic model and decoder provided in an embodiment of the present application.

[0034] FIG5 is a flow chart of a custom wake-up word in a voice wake-up method of a custom wake-up word provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following describes embodiments of the present application in detail, and examples of the embodiments of the present application are shown in the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present application and are not to be construed as limiting the present application.

[0036] The terms "first", "second", "third", "fourth", etc. (if any) in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0037] In the embodiments of the present application, "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship. In addition, "at least one of A, B, and C" means that (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C) can exist.

[0038] In order to make the technical problems solved by this application, the technical solutions adopted and the technical effects achieved clearer, the technical solutions of the embodiments of this application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0039] In order to balance the contradiction between low power consumption and high performance requirements of voice wake-up technology, we can start from one aspect: when the performance meets the requirements or when the performance degradation is controllable, we can try to reduce the size of the voice wake-up model and the computing power required by the voice wake-up model, thereby reducing the excessive power consumption caused by online voice wake-up technology. Based on this, this application provides a voice wake-up method, which is detailed as follows:

[0040] First, some professional terms that may be used in the following embodiments are introduced.

[0041] Automatic Speech Recognition (ASR) is a technology that converts speech into text and is a key enabler for natural communication between humans and machines, and between humans. ASR is the gateway for interaction between humans and smart devices. Its function is to enable devices to "understand" human speech, enabling them to perform tasks based on the recognized content.

[0042] Voice wake-up / keyword spotting (KWS): Also known as keyword detection, this is a subset of speech recognition tasks. It involves detecting a limited number of predefined activation words or keywords from a speech stream, rather than recognizing all speech. This type of technology is fundamental to enabling voice interaction in embedded devices and can be applied to a variety of fields, such as mobile phones, smart speakers, robots, smart homes, automotive devices, wearables, and more. The wake-up word is pre-set. When the device is in sleep mode, it continuously picks up audio and detects the wake-up word. Once the wake-up word is detected, the device switches from sleep mode to active mode, awaiting further interaction.

[0043] Acoustic Model: The acoustic model is a knowledge representation of the differences in acoustics, phonetics, environmental variables, speaker gender, accent, etc., while the language model is a knowledge representation of a sequence of words. The acoustic model is used to calculate the probability of the model generating a speech waveform. It is one of the most important parts of a speech recognition system, accounting for most of the computational overhead and determining the performance of the speech recognition system.

[0044] Classifier: The concept of classification is to learn a classification function or construct a classification model (i.e., the classifier we usually refer to) based on existing data. This function or model can map the data records in the database to one of the given categories, thus enabling data prediction. In short, a classifier is a general term for methods of classifying samples in data mining, including algorithms such as decision trees, logistic regression, naive Bayes, neural networks, etc. Here, it has the same function as the wake-up recognizer.

[0045] Neural Network: A neural network is an artificial intelligence method used to teach a computer to process data in a way inspired by the human brain. This is a machine learning process called deep learning, which uses interconnected nodes or neurons in a hierarchical structure similar to the human brain. It can create an adaptive system that the computer uses to learn from mistakes and continuously improve.

[0046] Phoneme: It is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phoneme. Phonemes are divided into two major categories: vowels and consonants. For example, the Chinese syllable "啊 (ā)" has only one phoneme, "爱 (ài)" has two phonemes, "代 (dài)" has three phonemes, etc. The method of annotating speech with the International Phonetic Alphabet is called transcription, and there are two types: broad transcription and narrow transcription.

[0047] Loss Function: A loss function or cost function is a function that maps the values of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the learning criterion and optimization problems, that is, by minimizing the loss function to solve and evaluate the model.

[0048] CTC: CTC is the acronym of the English "Connectionist Temporal Classification", which is translated into Chinese as "Connectionist Temporal Classification". Through CTC, the frame sequence of speech in time and the corresponding transcribed text sequence can be automatically aligned during the model training process, without annotating the start and end time periods of each character or phoneme appearance, so as to directly classify on the time series and also avoid the relatively complex alignment operations of traditional algorithms.

[0049] The CTC method consists of two main parts: label alignment and loss calculation during training, and the calculation of the final prediction result during inference decoding. The decoding part is categorized into three types: greedy, beam search, and prefix beam search. The most commonly used is the simple and efficient greedy method. This method finds the value with the highest probability at each time step as the final prediction output, removing consecutive repetitions and blank blocks. However, due to the complexity of implementing CTC loss-based training algorithms, they are generally implemented directly within existing computing frameworks.

[0050] The technical solution in this application will be described below with reference to the accompanying drawings.

[0051] FIG1 is a diagram showing an application scenario of a voice wake-up method using a custom wake-up word according to an embodiment of the present application.

[0052] This application proposes a voice wake-up method that supports user-defined wake-up words. This method can be used in any scenario where voice interaction may be used, including on platforms such as work computers, mobile smart devices, wearable devices, and robots.

[0053] As shown in Figure 1, on the far right are the database and server. The server is mainly used for pre-training models, and the database is used to store training sample data and trained models. The server and database are generally deployed in the cloud, and the trained models can be transmitted to the end device through network transmission methods such as routers to facilitate interaction between the end device and the user to realize the functions of customized wake-up words and voice wake-up.

[0054] The user implements the custom wake-up word function on the end side: the end side includes the user and the end-side device. The end-side device generally includes an entity that can output personalized voice, which can be a smart terminal (smartphone, smart speaker, etc.), a wearable terminal or a robot. These entities themselves have certain computing capabilities and can complete certain model reasoning and computing tasks. The implementation scenario of the present invention can be on voice editing software running on platforms such as desktop computers and smart phones, or on an application or software with a voice wake-up function integrated on a computer, smart phone or wearable device that can speak, a smart speaker or a robot that can talk to people.

[0055] In order to simplify the description, the interaction process between the terminal device and the user is described in detail using the "terminal" as an example. During the interaction between the terminal and the user, the terminal receives a wake-up word definition request and receives a custom wake-up word manually entered by the user. The terminal system automatically converts the target wake-up word into a target wake-up word pronunciation and returns the target wake-up word pronunciation to the user. The user receives a confirmation request for the target wake-up word pronunciation and completes the confirmation. Finally, the target wake-up word pronunciation and the target wake-up word are stored on the terminal device.

[0056] After determining the target wake-up word, the terminal also provides an online voice wake-up service: the terminal receives the user's interactive voice in real time, compares the received user's interactive voice with the target wake-up word, and gives a corresponding predefined response. Specifically, when the interactive voice input by the user contains the target wake-up word, the terminal-side device recognizes the target wake-up word and gives a response; when the interactive voice input by the user does not contain the target wake-up word, the terminal-side device does not respond.

[0057] In order to more clearly describe the voice wake-up method of a custom wake-up word proposed in this application, the voice wake-up process is briefly described below using Figure 2 as an example. Figure 2 is a simple flow chart of a voice wake-up method of a custom wake-up word provided in an embodiment of the present application. It should be noted that this simple flow chart only simply shows a process embodiment of the voice wake-up method. The specific details are not expanded in detail in this embodiment, and the specific details will be elaborated in detail in the following embodiments. Specifically, the flow chart is divided into the following four steps:

[0058] Step 1: The server generates an initial wake-up model through model training

[0059] The initial wake-up model includes an acoustic model and a voice wake-up model. The acoustic model contains multiple target acoustic models for learning speech features. The voice wake-up model includes a text encoder and a wake-up recognizer for determining whether the input speech data contains the target wake-up word and whether to respond. The following describes the training process of the acoustic model and the voice wake-up model respectively:

[0060] For acoustic model training, in one implementation, the initial acoustic model and the initial decoder are jointly trained using speech-text annotation data D (D = {Audio, Text}) until the parameters converge to obtain a trained acoustic model. For the sake of convenience, the "initial acoustic model" refers to the acoustic model that has not been trained with parameters, and the "initial decoder" also refers to the decoder that has not been trained with parameters; the acoustic model and decoder obtained after training until the parameters converge are respectively called the "trained acoustic model" and the "trained decoder".

[0061] The following mainly describes the detailed training process of the acoustic model. The first step is to prepare the training data. The training data refers to the speech text annotation data D (D = {Audio, Text}), where Audio represents speech data and Text represents speech text data. Before training the acoustic model, the speech text data needs to be preprocessed to obtain the corresponding speech text sequence. For example, the speech text sequence can be a phoneme sequence. In addition to the training sample data, the parameters of the initial acoustic model also need to be initialized. The initialization operation is to allow the network to update the learning features and update the parameters of the network efficiently. Each convolution layer in the initial acoustic model contains a gating unit, which is used to learn the characteristics of each phoneme, so that during the training process, each sub-neural network corresponds to the characteristics of each phoneme in the learning phoneme sequence. After the parameter convergence training is completed, a sub-acoustic model corresponding to each phoneme can be obtained. The multiple sub-acoustic models are combined into a trained acoustic model. In addition, the acoustic model training process also includes the training of the encoder. The initial encoder and the initial acoustic model are jointly trained until the parameters converge to obtain the trained acoustic model and the trained encoder. The training of the acoustic model is mainly used to learn the speech features of the speech data, and the training of the encoder is mainly used to decode the speech features to obtain decoded speech text data. During the training process, the difference between the decoded speech text data and the real speech text data is continuously reduced by constraining the loss function. This process can improve the ability of the acoustic model to learn speech features by improving the decoding ability of the decoder. That is, only when the acoustic model extracts more accurate speech features can the difference between the speech text decoded by the decoder and the real speech text be smaller.

[0062] For the voice wake-up model, in one implementation, the voice wake-up model also needs to be obtained through training, but the training sample data of the acoustic model and the voice wake-up model are different. The training sample data of the voice wake-up model is obtained by processing and constructing the voice text annotation data D. Specifically, the wake-up task corpus is constructed from the voice text annotation data D (D = {Audio, Text}), that is, a wake-up word set {Keyword} consisting of words and phrases with a length of four words (assuming that the wake-up word is specified as four words, the length setting here needs to be based on the length specified for the specific wake-up word) is randomly selected from the text. w, w = 1, 2, ..., W}, then the audio-text pairs containing the wake-up word are positive examples of the wake-up word, and the label is set to true, while the audio-text pairs excluding the wake-up word are negative examples, and the label is set to false. According to this labeling method, a wake-up training corpus D′ (D′ = {Audio, Text, label}) containing W wake-up words is constructed. Each training corpus in the wake-up training corpus is a triplet {text containing the keyword Tkeyword, positive example speech of the keyword Apositive, negative example speech of the keyword Anegative}.

[0063] After obtaining the wake-up training corpus, the speech wake-up model can be trained. Before training the speech wake-up model, it is necessary to load the trained acoustic model and fix the parameters of the trained acoustic model during the training process to use the trained acoustic model to assist in training the initial wake-up model, where the initial wake-up model includes an initial text encoder and an initial wake-up recognizer. The trained acoustic model is used to extract speech features from speech data, and the initial text encoder is used to learn to encode the preprocessed target speech text into target speech features. After obtaining the target speech features, the target speech features can be stored on the end-side device. During the online service phase, the speech features extracted from the target acoustic model and the target speech features obtained from the end-side device are fused and input into the initial wake-up recognizer. The initial wake-up recognizer is used to implement the recognition wake-up function. Specifically, the recognition wake-up refers to comparing the speech features with the target speech features, judging whether the content of the speech features contains the content of the target speech features and giving an evaluation index. For example, a probability value can be used to represent it, and a threshold value can be set in advance. The threshold value can be set according to specific usage requirements and is a value between 0 and 1; then it is determined whether to respond. Specifically, when the probability value is greater than the preset threshold, it is considered that the content of the speech feature contains the content of the target speech feature, that is, it is considered that the online input speech contains the target wake-up word, and a response is given; when the probability value is less than the preset threshold, it is considered that the content of the speech feature does not contain the content of the target speech feature, that is, it is considered that the online input speech does not contain the target wake-up word, and no response is given. The final output result is usually expressed as on / off, where on means that the current corpus contains the target wake-up word, that is, the probability that the content of the speech feature contains the target speech feature is greater than a preset threshold (such as 0.7), and the result is a response; off means that the current corpus does not contain the target wake-up word, that is, the probability that the content of the speech feature contains the target speech feature is less than the threshold (such as 0.7), and no response is given. The preset threshold can be set according to specific needs.

[0064] The above steps will produce a trained acoustic model and a trained voice wake-up model. The following briefly describes the steps for customizing the wake-up.

[0065] Step 2: The terminal generates a target text unit sequence based on the target wake-up word input by the user

[0066] The custom voice wake-up method can realize the function of user-defined wake-up words on the terminal device. The terminal device can be an interactive terminal device, including a smart phone, smart speaker, robot, smart home, car-mounted device, wearable device, etc. It receives the request for the custom wake-up word submitted by the user and completes the pronunciation confirmation of the custom wake-up word through interaction with the user. The custom wake-up word is confirmed as the target wake-up word and is stored in the terminal device. The terminal system can convert the target wake-up word into a target text unit sequence through a phonetic conversion tool. The target text unit sequence includes but is not limited to phonemes, subphones, byte pair (Byte Pair Encoder, BPE) encoding, or pseudo labels converted by other acoustic wake-up models. After obtaining the target unit sequence, the operation of reorganizing the target wake-up model can be performed.

[0067] Step 3: The terminal reorganizes the initial wake-up model to generate the target wake-up model

[0068] After determining the target wake-up word and the target text unit sequence, the trained acoustic model and the trained speech wake-up model obtained in step 1 can be assembled based on the target text unit sequence to obtain a target wake-up model.

[0069] In one implementation, the target text unit sequence can be a target phoneme sequence, and each target phoneme has a unique corresponding sub-acoustic model in the acoustic model. The terminal extracts the corresponding trained target sub-acoustic model from the trained acoustic model according to the target phoneme sequence, and combines these target sub-acoustic models to obtain the target acoustic model. Therefore, the parameter amount of the target acoustic model is related to the target phoneme sequence, that is, the parameter amount of the target acoustic model is related to the defined target wake-up word.

[0070] The target acoustic model obtained is combined with the trained wake-up recognizer to obtain a target wake-up model. The parameter quantity of the target acoustic model is positively correlated with the target wake-up word. The target wake-up model can be applied to the terminal device to realize online voice wake-up service.

[0071] Step 4: The terminal uses the target wake-up model to implement online voice wake-up service.

[0072] The target wakeup model runs on the interactive end-side device to implement the online voice wakeup function. Specifically, the end-side device receives the interactive voice input of the user in real time and uses the target wakeup model to perform inference calculations to determine whether a response is required. If the interactive voice contains the target wakeup word, a response is given; if the interactive voice does not contain the target wakeup word, no response is given.

[0073] The above briefly introduces a simple flow chart of a voice wake-up method for a custom wake-up word proposed in this application. The following describes the specific implementation details of the voice wake-up method for the custom wake-up word. Taking Figure 3 as an example,

[0074] Figure 3 is a schematic diagram of a specific implementation process of a voice wake-up method for a custom wake-up word provided in an embodiment of the present application. The process includes model training, defining the wake-up word, and online wake-up. The following describes each stage of the process separately:

[0075] Model training

[0076] This phase is generally performed on a cloud server. The model training phase includes two key steps: training the acoustic model and training the voice wake-up model. The following describes the corresponding steps in detail:

[0077] S1. Train the acoustic model.

[0078] This stage is mainly based on the annotated data D (D = {Audio, Text}) of the speech text as a data sample to jointly train the acoustic model and the decoder. The "Audio" in the sample data is the speech data, and the "Text" is the speech text annotated data corresponding to the speech data. The acoustic model is used to learn the speech features of the speech data, and the encoder is used to decode the speech features to obtain the decoded speech text data. During the training process, the difference between the decoded speech text data and the real speech text data is continuously reduced by means of a constrained loss function. This process can improve the ability of the acoustic model to learn speech features by improving the decoding ability of the decoder. That is, only when the acoustic model extracts more accurate speech features can the difference between the speech text decoded by the decoder and the real speech text be smaller.

[0079] In order to more rigorously and clearly distinguish the acoustic model and sub-acoustic model before and after training, in the description of the training process, the acoustic model and sub-acoustic model before training are recorded as "initial acoustic model" and "sub-neural network", and the acoustic model and sub-acoustic model after training are respectively called "trained acoustic model" and "trained sub-acoustic model"; the decoder before training is called "initial decoder", and the decoders after training are respectively called "trained decoders".

[0080] In one implementation, the speech text annotation data D (D = {Audio, Text}) is first preprocessed to obtain the processed speech text data. To illustrate simply, assume the speech text data is "Hello, Xiaoyi". After passing through the data preprocessing process, this speech text data yields a phoneme sequence {"n i3 h ao3 x i ao3 i4"}, a total of 8 phonemes. These 8 phonemes include types of initials, finals + tones (where 3 represents the third tone and 4 represents the fourth tone). The phoneme sequence is used as a training sample and fed into the initial acoustic model for parameter learning. The initial acoustic model contains a gated unit, which is inserted after each convolutional layer of the initial acoustic model, dividing the initial acoustic model into individual sub-neural networks. These individual sub-neural networks continuously learn the features of each phoneme during training to generate a sub-acoustic model AME corresponding to each phoneme. For example, it can be learned respectively that: sub-acoustic model AME1 corresponds to "n", AME2 corresponds to "i3", AME3 corresponds to "h", AME4 corresponds to "ao3", AME5 corresponds to "x", AME6 corresponds to "i", AME7 corresponds to "ao3", and AME8 corresponds to "i4". Therefore, the trained acoustic model AM contains the trained sub-acoustic models AMEi corresponding to each phoneme, where i is the number of phonemes. After the initial acoustic model is trained, the trained acoustic model AM and the trained sub-acoustic models AME can be obtained, and each of the trained sub-acoustic models AME exists in the trained acoustic model AM.

[0081] Specifically, to describe this process more clearly, taking Figure 4 as an example, the in-depth details are expanded. Figure 4 is a schematic diagram of jointly training an acoustic model and a decoder provided by an embodiment of this application. The figure includes speech data and the corresponding real speech text data. In this embodiment, the real speech text data is a phoneme sequence, which is obtained by preprocessing the real speech text data. The figure also includes an acoustic model (for distinction, it is the "initial acoustic model" before training and the "trained acoustic model" after training), the acoustic feature represents the speech feature extracted by the acoustic model from the input speech data, a decoder (for distinction, it is the "initial decoder" before training and the "trained decoder" after training). The decoder outputs the speech text data obtained by decoding the speech feature. There is a difference between this speech text data and the real speech text data. The source of this difference is the error of the speech feature extracted by the acoustic model and the decoding error of the decoder. Therefore, a loss function is needed during training to constrain and reduce each error.

[0082] In this embodiment, the speech data ("Hello Xiaoyi") is input into the initial acoustic model, the initial acoustic model learns the speech features of the speech data, and the speech text data "Hello Xiaoyi" is preprocessed to obtain the corresponding phoneme sequence data {"n i3 h ao3 x i4"}. After the initial decoder receives the phoneme sequence data and the speech features, it learns the correspondence between the speech data and the phoneme data, so that the input speech data can be accurately decoded into its corresponding phoneme sequence. The loss function is used to constrain the training parameters of the initial acoustic model, so that the trained acoustic model can accurately extract the effective features of the speech data, which is conducive to the trained decoder to accurately decode the obtained speech features into the corresponding phoneme sequence.

[0083] First, the process of sample preparation and parameter initialization is described here with the number of phonemes being p (p=6). The phoneme sample data used for training contains p, and the initial sub-acoustic model is AME. i (i=1,2,…,p). Before model training, the initial acoustic model AM, initial decoder and initial sub-acoustic model AME are trained. i (i=1,2,…,p) respectively perform initialization operations. Specifically, for the initial acoustic model AM and the initial decoder, the parameters of the model can be initialized by random initialization. For the initial sub-acoustic model AME i (i=1,2,…,p) is randomly assigned 0 or 1 according to the target sparsity α (α can be a hyperparameter used to represent sparsity. For example, α=10%, then AME i (The value of i is less than or equal to the number of phonemes) About 10% of the positions are assigned 1, and the rest are 0).

[0084] In fact, it is a training process. During the training process, the input of the initial acoustic model AM is speech data, and the output is the speech features corresponding to the speech data; the input of the initial encoder is the real phoneme sequence and speech features, and the output is "on / off", where "on" means that the phoneme sequence obtained by decoding the speech features contains the real phoneme sequence, that is, a response is given; "off" means that the phoneme sequence obtained by decoding the speech features does not contain the real phoneme sequence, that is, no response is given. The initial decoder can obtain the phoneme sequence obtained by decoding the speech features. There is a difference between the phoneme sequence and the real phoneme sequence. The source of the difference is the error of the speech features extracted by the acoustic model and the decoding error of the decoder. Therefore, a loss function is needed in the training process to constrain and reduce the various errors. Here, the Loss loss function is used to constrain the parameters in the initial acoustic model A and the initial encoder. Specifically, the Loss loss function includes L CTC , L sim , L sparsity, and Loss=L CTC +L sim +L sparsity , where L CTC Represents the loss between the phoneme sequence predicted by the decoder and the true phoneme sequence, which is used to constrain the error between the predicted phoneme sequence and the true phoneme sequence. For example, the CTC loss function (Connectionist Temporal Classification loss) can be used; L sim is the contrast loss between sub-acoustic models, which is used to constrain the phonemes of the same category in training

[0085] Statistics M i The number of activated (non-zero) parameters in the (i-th sub-neural network); k represents the total number of parameters of the entire acoustic model; and Sparity target Indicates the target sparsity after pruning, which can be set as needed. In this embodiment, it is set to 0.5. i The value of is a floating-point probability value, rather than a binary value including only 0 or 1 (binary, meaning there are only two values, either 0 or 1). Therefore, a threshold γ can be set. When the probability value is greater than γ, the final output is 1, otherwise the final output is 0.

[0086] In order to better constrain the parameters of each sub-acoustic model, for each initial sub-acoustic model AME i Use momentum update to update parameters, i.e. AME i =momentum*AME i +(1-momentum)*M i , where momentum is the preset momentum parameter.

[0087] Finally, the trained acoustic model can be obtained by training the sample data until the parameters converge. The trained acoustic model includes the trained sub-acoustic model.

[0088] S2. Train the voice wake-up model.

[0089] In this step, we first construct the wake-up task corpus (labeled positive and negative samples) from the speech and text annotation data D (D = {Audio, Text}), where Audio represents speech data and Text represents the text data corresponding to the speech data. That is, we randomly select a wake-up word set {Keyword} from the text that consists of words and phrases with a length of four characters (assuming that the wake-up word is specified as four characters, the length setting here needs to be based on the length specified for the specific wake-up word). w, w=1,2,...,W}, then the speech-text pair containing the wake-up word is the positive example of the wake-up word, and the label is set to true, while the speech-text pair excluding the wake-up word is the negative example, and the label is set to false. According to this labeling method, a wake-up training corpus D′ (D′={Audio,Text,label}) containing W wake-up words is constructed, where label is the label of the speech data. Each training corpus in the wake-up corpus is a triple {text containing the keyword keyword, positive example speech of the keyword Apositive, negative example speech of the keyword Anegative}.

[0090] The voice wakeup model is trained using the wakeup training corpus D′ (D′={Audio, Text, Label}). This voice wakeup model includes an initial text encoder and an initial wakeup recognizer. Before training, it is necessary to load the trained acoustic model and fix its parameters to train the parameters of the initial text encoder and the initial wakeup recognizer. Specifically, a piece of speech data Audio is input into the acoustic model. The acoustic model is used to learn the speech features corresponding to the speech data "speech". The speech text annotation data "text" corresponding to the speech data "speech" is then input into the initial text encoder. The initial text encoder is used to learn to encode the speech text annotation data to obtain the target speech features, which can be stored on the end device. During the online service phase, the speech features and the target speech features are fused and input into the initial wakeup recognizer. The initial wakeup recognizer compares the content of the speech features with the content of the target speech features to determine whether the speech features contain the content of the target speech features, thereby determining whether to respond. During the inference process, the wake-up recognizer can identify whether the input speech contains the predefined target wake-up word, and then determine whether a response is required (where "on" corresponds to giving a response, and "off" corresponds to not responding).

[0091] Secondly, before training, you need to first w Perform text-to-phoneme conversion, assuming that the phoneme sequence after text conversion is {P w1 , P w2 ,...,P wj}, then when the positive and negative sample data of the keyword are input, its actual parameters and the parameters of the calculated initial acoustic model AM can be dynamically determined by the initial sub-acoustic models corresponding to the phoneme sequence. Specifically, it is expressed as This expression represents the union of the activated parameters corresponding to all the phonemes (W phonemes). The loss function L loss function is used for constraint during the training of the voice wake-up model. The L loss function can be defined as:

[0092] L=CE(Classifier(f postive , f keyword ), 1)+CE(Classifier(f negative , f keyword ), 0), where f postive , f negtive and f keyword The three components are the positive and negative speech features after processing by the trained acoustic model AM, as well as the keyword text features extracted by the text encoder. The "classifier" is the wakeup identifier, and CE represents the cross entropy loss function. This loss function is primarily used to constrain the difference between the predicted and target phonemes. A larger loss function output indicates a greater difference between the predicted and target results. Therefore, during training, the speech wakeup model is trained continuously until convergence with the goal of minimizing the loss function.

[0093] Define the wake word

[0094] This stage is mainly used to define the wake-up words and reorganize the target wake-up model based on the target wake-up words.

[0095] First, the user interacts with the terminal device to implement the custom wake-up word operation. Specifically, as described in step S3:

[0096] S3. Customize the wake-up word.

[0097] Set a custom wake-up word for user input on the client side. The main process includes:

[0098] In one implementation, the terminal receives a wake-up word definition request and a custom wake-up word manually input by the user. The terminal system uses a character-to-sound conversion tool to automatically convert the target wake-up word into a target wake-up word voice, and returns the target wake-up word voice to the user. The user receives a pronunciation confirmation request for the target wake-up word voice and completes the confirmation. Finally, the target wake-up word and the target wake-up word voice are stored on the terminal device.

[0099] In another implementation, this embodiment takes the wake-up word "Hello, Xiaowei" with ambiguous pronunciation as an example to describe the pronunciation confirmation process of the wake-up word voice in this technical solution. Existing custom wake-up word solutions generally require users to provide at least one piece of voice reading the wake-up word when defining the wake-up word, and the system then determines the correct pronunciation of the wake-up word based on the matching of the voice and the wake-up word text. This embodiment provides an implementation method that does not require users to provide wake-up word voice. When the system receives the wake-up word "Hello, Xiaowei", it first uses a phonetic conversion tool to convert it into a text pronunciation sequence (such as a pinyin phoneme sequence). Since there are two polyphonic characters, "hao" and "wei", there will be four pronunciation combinations: |{hao3 hao4}| × |{wei2 wei4}|. At the same time, the phonetic conversion tool will also give the priority order of different combinations when predicting the pinyin of the text voice. Thus, the possible order of these four pronunciation combinations may be, {{n i2 h ao3 x iao3 w ei2}, {n i2 h ao3 x iao3 w ei4}, {n i2 h ao4 x iao3 w ei2}, {n i2 h ao4 x iao3 w ei4}}.

[0100] After confirming the pronunciation set, the system will sequentially return each pronunciation combination to the user for confirmation. The presentation method of the returned pronunciation is not limited to pinyin text, using the system's speech generation (TTS) tool to synthesize the voice of phonemes, etc. If the current pronunciation is not confirmed by the user, the next pronunciation will be returned. If none of the pronunciations are confirmed by the user, the user will be reminded to change a wake-up word. In this example, if the pronunciation expected by the user is {n i2 h ao3 x iao3 w ei4}, the first pronunciation result returned by the system, {n i2 h ao3 x iao3 w ei2}, will not be adopted by the user. Then, after the system returns the next pronunciation, the user confirmation can be obtained at this time. Finally, the system constructs the wake-up model corresponding to the wake-up word "Hello, Xiaowei" based on these phonemes {n i2 h ao3 x iao3 w ei4}.

[0101] After obtaining the target wake-up word, the target wake-up model can be assembled based on the target wake-up word. The specific implementation is as described in S4 below:

[0102] S4. Assemble the target wake-up model.

[0103] Replace the target wake-up word with a target text unit. For example, the target text unit can be a target phoneme sequence or a target syllable sequence. Here, take the target phoneme sequence as an example: Suppose the user newly defines a wake-up word, such as "Hello, Xiaoyi". After the system receives the definition wake-up word request, it first uses a Chinese character to phoneme conversion tool to convert the target wake-up word into a target phoneme sequence, that is, {n i2 h ao3 x iao3 i4} (a total of 8 phonemes). This target phoneme sequence contains initials, finals + tones (3 represents the third tone, 4 represents the fourth tone). Each target phoneme can uniquely determine a target acoustic model AME. Finally, 8 target acoustic models corresponding to this target phoneme sequence are obtained: the sub-acoustic model AME1 corresponds to "n", AME2 corresponds to "i3", AME3 corresponds to "h", AME4 corresponds to "ao3", AME5 corresponds to "x", AME6 corresponds to "i", AME7 corresponds to "ao3", AME8 corresponds to "i4". Combine these 8 sub-acoustic models to obtain a target acoustic model, and obtain a trained text encoder and a trained wake-up recognizer from the speech wake-up model. Combine the above target acoustic model and wake-up recognizer to obtain a target wake-up model.

[0104] Online wake-up

[0105] After obtaining the target wake-up word, deploy the target wake-up model on the end device to implement the online speech wake-up service, and the specific implementation is as described in S5 below:

[0106] S5. Online speech wake-up service.

[0107] The terminal executes the online speech wake-up service, and the end device receives the user's interactive speech in real time, and uses the target wake-up model to perform inference calculation to obtain the response result of the speech, so as to supply the terminal device to make corresponding interactive actions. Specifically, when the received interactive speech contains the target wake-up word, the terminal device gives a response; when the received interactive speech does not contain the target wake-up word, the terminal device does not give a response.

[0108] To better elaborate on the specific process of how the speech wake-up model implements the online wake-up service in the end device, as shown in Figure 5, Figure 5 is a schematic diagram of an online speech wake-up service provided by an embodiment of the present application. Among them, the "acoustic model" is the trained acoustic model, the "keyword acoustic model" is the target acoustic model obtained by combining multiple sub-acoustic models corresponding to the target wake-up word, the "keyword encoder" is the trained text encoder, and the "joint classifier" is the trained wake-up recognizer; the "speech" is the input speech data, the "keyword" is the phoneme sequence obtained by preprocessing the target wake-up word, and the "acoustic feature" is the speech feature of the speech data extracted by the target acoustic model.

[0109] This embodiment is described below. The acoustic model extracts a target acoustic model based on a custom target wake-up word. This target acoustic model is then combined with a wake-up recognizer to form a target wake-up model, which is then deployed on the client device. Before deploying this target wake-up model on the client device, a text encoder is used to encode the target phoneme sequence to obtain target speech features. These target speech features can be obtained online in real time, which is generally computationally expensive. Alternatively, the obtained target speech features can be stored on the client device, which can then be retrieved from the client device in real time during the online service phase.

[0110] Figure 5 shows a scenario where the target voice features are stored in the end-side device. Specifically, when in the online service state, the end-side device obtains online voice data in real time, and the target acoustic model is used to extract the voice features of the voice data. The voice features obtained from the target acoustic model and the target voice features obtained from the memory of the end-side device are then fused and input into the wake-up recognizer. The wake-up recognizer compares the content of the voice features with the content of the target voice features to determine whether the content of the voice special offer contains the content of the target voice features, and then determines whether to respond. If the target voice text data contains the voice text data, it is considered that the voice data contains the target wake-up word, and the end-side device responds in a preset manner; if the comparison result shows that the voice text data does not contain the target voice text data, it is considered that the voice data does not contain the target wake-up word, and the end-side device does not respond.

[0111] The above descriptions are only several embodiments of the present invention. Those skilled in the art may make various changes or modifications to the present invention based on the disclosure of the application documents without departing from the spirit and scope of the present invention.

Claims

1. A voice wake-up method, characterized in that: include: Generate a target text unit sequence from the target wake-up word input by the user; Extracting a target acoustic model from a pre-trained acoustic model according to the target text unit sequence, wherein the pre-trained acoustic model is composed of a plurality of sub-acoustic models, and the target acoustic model is composed of some sub-acoustic models in the plurality of sub-acoustic models; Receiving a voice input by the user; Determining whether to respond to the speech is performed according to the target acoustic model.

2. The method according to claim 1, characterized in that: The method further comprises: Get a pre-trained wakeup recognizer; The determining whether to respond to the speech according to the target acoustic model comprises: Determining whether to respond to the speech is performed according to the target acoustic model and the wake-up recognizer, wherein the target acoustic model is used to extract speech features from the speech, and the wake-up recognizer is used to determine whether to respond based on the speech features.

3. The method according to claim 2, characterized in that The determining whether to respond to the voice according to the target acoustic model and the wake-up identifier comprises: extracting the speech feature from the speech using the target acoustic model; The wake-up identifier is used to compare the speech feature with a target speech feature to determine whether to respond to the speech, wherein the target speech feature is a speech feature obtained by encoding the target text unit sequence using a pre-trained text encoder.

4. The method according to claim 3, characterized in that The comparing the speech feature with the target speech feature to determine whether to respond to the speech comprises: If the content of the target voice feature includes the content of the voice feature, responding; If the content of the target voice feature does not include the content of the voice feature, no response is made.

5. The method according to any one of claims 1 to 4, characterized in that: The step of extracting a target acoustic model from a pre-trained acoustic model according to the target text unit sequence comprises: Determining a sub-acoustic model corresponding to each text unit in the target text unit sequence according to the acoustic model; The corresponding sub-acoustic model of each text unit in the target text unit sequence is combined into the target acoustic model.

6. The method according to any one of claims 1 to 5, characterized in that Before generating the target text unit sequence from the target wake-up word, the method further includes: Receive a custom wake-up word input by the user; The custom wake-up word is converted into the corresponding target wake-up word.

7. The method according to any one of claims 1 to 6, characterized in that The target text unit sequence includes but is not limited to phonemes, subphonemes, and Byte Pair Encoder (BPE) encoding.

8. The method according to any one of claims 1 to 7, characterized in that The size of the target acoustic model is positively correlated with the length of the target text unit sequence.

9. A voice wake-up device, characterized in that: include: A text conversion module, used to convert the target wake-up word input by the user into a target text unit sequence; A target acoustic model extraction module, used for extracting a target acoustic model from a pre-trained acoustic model according to the target text unit sequence, wherein the pre-trained acoustic model is composed of a plurality of sub-acoustic models, and the target acoustic model is composed of some sub-acoustic models in the plurality of sub-acoustic models; A voice receiving module, used for receiving the voice input by the user; A voice wake-up module is used to determine whether to respond to the voice according to the target acoustic model.

10. The device according to claim 9, characterized in that The voice wake-up module is also used for: Determining whether to respond to the speech is based on the target acoustic model and a pre-trained wake-up recognizer, wherein the target acoustic model is used to extract speech features from the speech, and the wake-up recognizer is used to determine whether to respond based on the speech features.

11. The device according to claim 10, characterized in that The voice wake-up module is also used for: extracting the speech feature from the speech using the target acoustic model; The wake-up identifier is used to compare the speech feature with a target speech feature to determine whether to respond to the speech, wherein the target speech feature is a speech feature obtained by encoding the target text unit sequence using a pre-trained text encoder.

12. The device according to claim 11, characterized in that The voice wake-up module is also used for: If the content of the target voice feature includes the content of the voice feature, responding; If the content of the target voice feature does not include the content of the voice feature, no response is made.

13. The device according to any one of claims 9 to 12, characterized in that: The target acoustic model extraction module is also used for: Determining a sub-acoustic model corresponding to each text unit in the target text unit sequence according to the acoustic model; The corresponding sub-acoustic model of each text unit in the target text unit sequence is combined into the target acoustic model.

14. The device according to any one of claims 9 to 13, characterized in that The device also includes: The custom wake-up word module is used to receive a custom wake-up word input by a user and convert the custom wake-up word into the corresponding target wake-up word.

15. The device according to any one of claims 9 to 14, characterized in that The target text unit sequence includes but is not limited to phonemes, subphonemes, and Byte Pair Encoder (BPE) encoding.

16. The device according to any one of claims 9 to 15, characterized in that The size of the target acoustic model is positively correlated with the length of the target text unit sequence.

17. A device, characterized in that include: at least one memory for storing a program; at least one processor, configured to execute the program stored in the memory; Wherein, when the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1-8.

18. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on a processor, the processor is caused to execute the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that When the computer program product runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 8.

Images