Training method and apparatus for wake-on-voice model, executing method for wake-on-voice model, and device and storage medium

Abstract

A training method and apparatus for a wake-on-voice model, an executing method for a wake-on-voice model, and a device and a storage medium. The wake-on-voice model comprises a neural network structure, a wake-up word classification branch and a starting / ending point determination branch. The training method comprises: acquiring an audio sample, and performing acoustic feature processing on the audio feature, so as to obtain an acoustic feature frame (S201); inputting the acoustic feature frame into a neural network structure, so as to generate an output vector (S202); and on the basis of the output vector, separately training a wake-up word classification branch and a starting / ending point determination branch (S203).

Description

Training and execution method, device, equipment and storage medium of voice wake-up model Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, execution method, apparatus, device, and storage medium for a voice wake-up model. Background Art

[0002] With the advancement of computer technology, more and more smart devices are supporting voice interaction. For example, a large number of devices, such as smart speakers, smart TVs, and cars, are equipped with voice interaction functions, allowing users to directly control and operate the devices using voice, providing great convenience. During the voice interaction process, wake-up word detection generally marks the beginning of each interaction. Once the wake-up word is triggered, the device will actively collect the user's voice commands. To reduce the risk of device triggering caused by false wake-up word recognition, many solutions implement a second-level wake-up word verification in the cloud, in addition to the first-level wake-up word verification performed locally. This requires accurate interception of the wake-up word.

[0003] In traditional technologies, common methods for intercepting wake-up words include:

[0004] First, at the wake-up word trigger position, a fixed-length audio segment is intercepted forward. However, considering the different pronunciation speeds of different users, the fixed-length interception is generally set according to the length of slow speech. In this way, for a fast-speech wake-up word trigger, some irrelevant audio may be intercepted, thereby affecting the accuracy of the secondary wake-up verification.

[0005] Second, voice activation detection (VAD) is used to capture the starting point of the wake-up word. However, this method can only be used in relatively quiet environments. In a noisy environment, the performance of VAD drops rapidly, and it is impossible to effectively obtain the starting position of the wake-up word.

[0006] Therefore, with the rapid development of voice interaction functions, there is an urgent need for an accurate and efficient voice wake-up technology.

[0007] Summary of the Invention

[0008] The present application aims to solve at least one of the technical problems existing in the prior art or related art. To this end, the present application provides a training method, execution method, device, equipment and storage medium for a voice wake-up model.

[0009] According to a first aspect of the present application, a method for training a voice wake-up model is provided, wherein the voice wake-up model includes a neural network structure, a wake-up word classification branch, and a start and end point judgment branch. The training method includes:

[0010] Acquiring an audio sample and performing acoustic feature processing on the audio sample to obtain an acoustic feature frame;

[0011] Inputting the acoustic feature frame into a neural network structure to generate an output vector, wherein the output vector has a first attribute and / or a second attribute, the first attribute being whether the wake-up word is included, and the second attribute being the starting point position and the ending point position of the wake-up word in the output vector; and

[0012] Based on the output vector, the wake-up word classification branch and the start and end point judgment branch are trained separately, including:

[0013] The output vector is input to the wake-up word classification branch and the start-end point judgment branch respectively, and the wake-up word classification branch outputs the probability that the audio sample contains the wake-up word, and the start-end point judgment branch outputs the start point position and end point position of the wake-up word in the audio sample; and

[0014] The output results of the wake-up word classification branch and the output results of the start-end point judgment branch are used to adjust the parameters of the wake-up word classification branch and the parameters of the start-end point judgment branch respectively to obtain updated wake-up word classification branch and start-end point judgment branch.

[0015] As an embodiment of the present application, the wake-up word classification branch includes at least a first fully connected layer structure connected to the neural network structure, and the output vector of the neural network structure includes first label data for the wake-up word classification branch, the first label data involves a first attribute, and the first label data is used to characterize whether the output vector contains the wake-up word and is input into the first fully connected layer structure as a first supervised learning target.

[0016] As an embodiment of the present application, adjusting the parameters of the wake-up word classification branch using the output result of the wake-up word classification branch includes: determining the first loss function of the wake-up word classification branch based on the first attribute of the output vector and the output result of the wake-up word classification branch; and adjusting the parameters of the wake-up word classification branch based on the first loss function.

[0017] As an embodiment of the present application, adjusting the parameters of the wake-up word classification branch based on the first loss function includes: iteratively updating the parameters of the wake-up word classification branch according to the calculated value of the first loss function until convergence to obtain an updated wake-up word classification branch.

[0018] As an embodiment of the present application, the first loss function of the wake-up word classification branch is cross entropy.

[0019] As an embodiment of the present application, the calculation formula of the first loss function is:

[0020] in, is the label of the wake-up word; when the output vector contains the wake-up word, When the output vector does not contain the wake-up word, y is the wake-up word probability output by the wake-up word classification branch, and its range is [0,1].

[0021] As an embodiment of the present application, the start and end point judgment branch includes at least a second fully connected layer structure connected to the neural network structure, and the output vector of the neural network structure includes second label data for the start and end point judgment branch, the second label data involves a second attribute, and the second label data is used to represent the starting point position and the ending point position of the wake-up word in the output vector and is input into the second fully connected layer structure as a second supervised learning target.

[0022] As an embodiment of the present application, adjusting the parameters of the start-end point judgment branch using the output results of the start-end point judgment branch includes: determining the second loss function of the start-end point judgment branch based on the second attribute of the output vector and the output results of the start-end point judgment branch; and adjusting the parameters of the start-end point judgment branch based on the second loss function.

[0023] As an embodiment of the present application, adjusting the parameters of the start and end point judgment branch based on the second loss function includes: iteratively updating the parameters of the start and end point judgment branch according to the calculated value of the second loss function until convergence to obtain an updated start and end point judgment branch.

[0024] As an embodiment of the present application, the second loss function of the start-end point judgment branch is the mean square error between the output start point position and end point position and the true start point position and end point position.

[0025] As an embodiment of the present application, the calculation formula of the second loss function is: L2 = (s1-k1) 2 +(s2-k2) 2

[0026] Among them, L2 is the mean square error value; s1 is the true starting point position of the wake-up word in the output vector; s2 is the true ending point position of the wake-up word in the output vector; k1 is the starting point position of the start-end point judgment branch output; k2 is the ending point position of the start-end point judgment branch output.

[0027] As an embodiment of the present application, the training method further includes: adjusting the parameters of the neural network structure using the output results of the wake-up word classification branch and / or the output results of the start and end point judgment branch to obtain an updated neural network structure.

[0028] As an embodiment of the present application, feature processing of audio samples includes: performing endpoint detection on the wake-up word in the audio sample, starting from the starting point of the wake-up word, intercepting data frames according to a time window of a fixed length until the end point of the wake-up word, so as to obtain a plurality of data frames of the same time length; the plurality of data frames are in the data form of a one-dimensional array; and extracting audio spectrum features from the one-dimensional array, and outputting a two-dimensional feature array.

[0029] As an embodiment of the present application, in response to the intercepted data frame including a plurality of wake-up words, the data frame including the first wake-up word is retained.

[0030] As an embodiment of the present application, inputting the acoustic feature frame into the neural network structure to generate an output vector includes: inputting a two-dimensional feature array into the neural network structure, and generating a one-dimensional vector through a convolution algorithm.

[0031] According to the second aspect of the present application, a method for executing a voice wake-up model is also provided, including:

[0032] Obtain the voice data to be detected;

[0033] Input the voice data to be detected into any of the above-mentioned trained voice wake-up models;

[0034] Obtain the output results of the wake-up word classification branch and the output results of the start and end point judgment branch; and

[0035] The wake-up result of the voice wake-up model is output based on the output result of the wake-up word classification branch and / or the output result of the start-end point judgment branch.

[0036] As an embodiment of the present application, the execution method further includes: sending the output result of the wake-up word classification branch and / or the output result of the start and end point judgment branch to the speech recognition cloud platform for secondary recognition of the wake-up word.

[0037] According to a third aspect of the present application, a training device for a voice wake-up model is also provided. The voice wake-up model includes a neural network structure, a wake-up word classification branch, and a start and end point judgment branch. The training device includes:

[0038] An acquisition module is used to acquire audio samples and perform acoustic feature processing on the audio samples to obtain acoustic feature frames;

[0039] A generation module, configured to input the acoustic feature frame into a neural network structure to generate an output vector, wherein the output vector has a first attribute and / or a second attribute, the first attribute being whether the wake-up word is included, and the second attribute being the starting point position and the ending point position of the wake-up word in the output vector; and

[0040] The training module trains the wake-up word classification branch and the start and end point judgment branch based on the output vector, including:

[0041] The output vector is input to the wake-up word classification branch and the start-end point judgment branch respectively, and the wake-up word classification branch outputs the probability that the output vector contains the wake-up word, and the start-end point judgment branch outputs the start point position and end point position of the wake-up word in the output vector; and

[0042] The output results of the wake-up word classification branch and the output results of the start-end point judgment branch are used to adjust the parameters of the wake-up word classification branch and the parameters of the start-end point judgment branch respectively to obtain an updated voice wake-up model.

[0043] According to the fourth aspect of the present application, a voice wake-up device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any of the above-mentioned training methods and / or any of the above-mentioned execution methods is implemented.

[0044] According to the fifth aspect of the present application, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned training methods and / or any of the above-mentioned execution methods are implemented.

[0045] According to the sixth aspect of the present application, a computer program is also provided, which implements any of the above-mentioned training methods and / or any of the above-mentioned execution methods when executed by a processor.

[0046] The training method of the voice wake-up model of the embodiment of the present application adopts a multi-task training method, and the wake-up word classification branch and the start and end point judgment branch are set in parallel at the end of the wake-up word detection network, and the classification task of the wake-up word and the regression task of the start and end points of the wake-up word are trained at the same time, so as to realize whether the output audio sample contains the wake-up word, and also output the starting point position and the end point position of the wake-up word in the audio sample. Therefore, the voice wake-up model trained by the embodiment of the present application can quickly and accurately obtain the starting point and end point of the wake-up word, which helps to more accurately intercept the wake-up word in the audio frame. In addition, since the classification task of the wake-up word and the regression task of the start and end points of the wake-up word are trained at the same time, it can help to enhance the effect of the neural network structure, and can further improve the accuracy of the wake-up word detection compared with a single training task.

[0047] In addition, the voice wake-up execution method of the embodiment of the present application is based on the above-mentioned trained voice wake-up model. It can output whether the audio sample contains the wake-up word, and also output the starting point and ending point of the wake-up word in the audio sample, thereby significantly improving the accuracy of wake-up word detection and greatly improving the performance of voice wake-up. In addition, the present application can more accurately intercept the wake-up word in the audio frame in scenarios that require cloud-based secondary verification. Therefore, it can especially demonstrate its advantages in scenarios that require cloud-based secondary verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. The drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] FIG1 is a schematic diagram of the structure of a voice wake-up model provided in an embodiment of the present application.

[0050] FIG2 is a flow chart of a method for training a voice wake-up model according to an embodiment of the present application.

[0051] FIG3 is a flow chart of a method for training a wake-up word classification branch in a voice wake-up model according to an embodiment of the present application.

[0052] FIG4 is a flow chart of a training method for a start and end point judgment branch of a voice wake-up model provided in an embodiment of the present application.

[0053] FIG5 is another flowchart of the method for training a voice wake-up model provided in an embodiment of the present application.

[0054] FIG6 is a flow chart of a method for executing a voice wake-up model according to an embodiment of the present application.

[0055] FIG7 is a schematic diagram of the structure of a training device for a voice wake-up model provided in an embodiment of the present application.

[0056] FIG8 is a schematic diagram of the structure of a voice wake-up device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] The embodiments of the present application provide a training method and an execution method for a voice wake-up model, which are applied to the fields of machine learning and voice technology in the field of artificial intelligence. Specifically, the embodiments of the present application can be applied to scenarios requiring voice interaction, such as smart speaker voice interaction, smart car voice interaction, and smart TV voice interaction, to improve the detection accuracy of wake-up words.

[0059] In order to facilitate understanding of the technical solutions provided by this application, exemplary application scenarios of the embodiments of this application are described below.

[0060] The training method of the voice wake-up model provided in the embodiment of the present application can be executed by the training device of the voice wake-up model provided in the embodiment of the present application. The training device of the voice wake-up model provided in the embodiment of the present application can be a terminal device or a server. Exemplarily, the training method of the voice wake-up model provided in the embodiment of the present application can be applied to a terminal device. For example, it can be implemented by a processor, an application or a web page in the terminal device. There is data communication between the terminal device and the server. The embodiment of the present application does not limit this. The embodiment of the present application does not limit the specific type of the terminal device. For example, the terminal device can be a smart phone, a personal computer, a tablet computer, a wearable device, a vehicle-mounted terminal, etc. The embodiment of the present application does not limit the type and number of servers. For example, the server can be a single independent server or a server cluster. The embodiment of the present application only takes this as an example and is not limited to this.

[0061] After the voice wake-up model is trained, it can be applied to terminal devices that require voice recognition to perform the voice wake-up function. For example, the voice wake-up model can be applied to terminal devices such as smart speakers, smart home devices, smartphones, car terminals, wearable devices, etc., which are not limited in this embodiment of the application.

[0062] Figure 1 is a structural diagram of a voice wake-up model provided in an embodiment of the present application. As shown in Figure 1, the voice wake-up model provided in an embodiment of the present application includes an acoustic feature processing unit, a neural network structure, a wake-up word classification branch, and a start-end point judgment branch. Among them, the acoustic feature processing unit is used to extract acoustic features of the input audio signal, and input the extracted acoustic features into the neural network structure. The output of the neural network structure is used as the input of the wake-up word classification branch and the start-end point judgment branch, and is used to simultaneously train the wake-up word classification branch to recognize the classification task of wake-up words and non-wake-up words, and calculate the regression task of the start-end point judgment branch to calculate the start and end point of the wake-up word. Advantageously, the wake-up word classification branch and the start-end point judgment branch share the output of a neural network structure, which helps to save computing costs.

[0063] FIG2 is a flow chart of a method for training a voice wake-up model according to an embodiment of the present application.

[0064] As shown in FIG2 , the training method of the voice wake-up model provided in the embodiment of the present application includes the following steps:

[0065] Step S201: Acquire an audio sample, and perform acoustic feature processing on the audio sample to obtain an acoustic feature frame.

[0066] In this step, the audio samples include wake-up word samples and non-wake-up word samples. In some examples, the wake-up word samples can be manually recorded audio containing the wake-up word. In addition, the diversity and complexity of the audio samples can be increased by adding noise later, changing the position of the wake-up word in the audio, etc., thereby enhancing the training effect. The non-wake-up word samples can be audio files that do not contain the wake-up word. They can be human voices in everyday scenes, such as audio from movies and TV series, or they can be automatically collected and generated by a computer.

[0067] Before training the voice wake-up model, the type of each audio sample can be labeled through machine recognition or manual recognition, that is, whether the audio sample contains the wake-up word.

[0068] At the same time, since the training method of this embodiment sets the start and end point training branch, before training the voice wake-up model, the start and end points of the wake-up word in the wake-up word sample can also be accurately marked by machine recognition or manual recognition. Among them, the start and end points of the wake-up word are the starting time point (also called the starting point) and the ending time point (also called the ending point) of the wake-up word in the audio sample. It can be understood that in some examples, non-wake-up word samples do not need to be marked with the starting point and the ending point.

[0069] In this step, performing acoustic feature processing on the audio sample includes: for the input audio sample, starting from the starting point of the wake-up word, intercepting the data frame according to a time window of a fixed length until the end point of the wake-up word, so as to obtain a plurality of data frames of the same time length from the audio sample. The plurality of data frames can be in the form of one-dimensional array data. In some examples, the starting point and the end point of the wake-up word can be identified from the audio sample based on the voice endpoint detection (VAD) technology. In some examples, in response to the intercepted data frame including a plurality of the wake-up words, the data frame containing the first wake-up word is retained, thereby avoiding repeated calculations to save computing resources and improve recognition speed.

[0070] Further, acoustic feature extraction is performed on the data frame of the above-mentioned one-dimensional array, and a two-dimensional feature array is output. In some examples, the Mel-Frequency Cepstral Coefficients (MFCC) can be used to perform acoustic feature extraction and feature enhancement on the data frame of the one-dimensional array to output a two-dimensional feature array. Specifically, in the MFCC feature extraction method, the audio data is first subjected to a fast Fourier transform (FFT), and then the spectra of different frequency bands are filtered and compressed according to the human auditory characteristics to obtain MFCC features. In other examples, a static filter network (Filter Bank, FBANK), a normalized energy coefficient (Power-Normalized Coefficients, PNCC), etc. can also be used to extract audio spectrum features, and this application does not impose specific restrictions on this.

[0071] For example, for an input audio sample, starting from the start of the audio information containing the wake-up word identified in the audio sample, data frames are intercepted through the time window every a seconds until the end of the audio information containing the wake-up word. Based on this, m data frames of the same length can be intercepted from the audio sample and used as training samples for the input neural network in the form of a one-dimensional array. Furthermore, MFCC coefficients can be used to extract and enhance features of the input training samples, and n×m 32-bit two-dimensional feature arrays can be output as acoustic feature frames for subsequent use.

[0072] Step S202: Input the acoustic feature frame into the neural network structure to generate an output vector, wherein the output vector has a first attribute and / or a second attribute, the first attribute is whether it contains a wake-up word, and the second attribute is the starting point position and the ending point position of the wake-up word in the output vector.

[0073] In this step, the neural network structure can be the backbone structure of the wake-up word recognition network. The backbone structure can be constructed using a convolutional neural network. Generally speaking, it can be composed of a convolutional layer and a pooling layer. The backbone structure can be a convolutional neural network (CNN), a recurrent neural network (RNN), an attention network, or other types of recognition network structures, which are not specifically limited in this application.

[0074] When the acoustic feature frame is input into the Backbone structure, the Backbone structure compresses it through a convolution algorithm to generate a one-dimensional vector and then outputs it. This one-dimensional vector serves as the common input for the subsequent wake-up word classification branch and the start and end point judgment branch.

[0075] It can be understood that as mentioned in the aforementioned step S201, before training the voice wake-up model, the audio samples include wake-up word samples and non-wake-up word samples, that is, they are marked as to whether they contain the wake-up word. In addition, for the wake-up word samples, the starting point and ending point of the wake-up word are marked. Therefore, in this step, the output vector of the neural network structure has a first attribute and / or a second attribute, where the first attribute is whether it contains the wake-up word, and the second attribute is the starting point position and ending point position of the wake-up word in the output vector.

[0076] Step S203: Based on the output vector, the wake-up word classification branch and the start and end point judgment branch are trained respectively.

[0077] In this step, the wake-up word classification branch is used to train the task of determining whether an audio sample is a wake-up word. During training, the wake-up word classification branch uses all audio samples for training. Specifically, the input to the wake-up word classification branch is the one-dimensional vector data obtained after processing by the neural network structure, including two types of samples: wake-up word and non-wake-up word. The training output is the probability that the sample is a wake-up word and the probability that the sample is not a wake-up word. As you can see, the sum of these two probabilities is 1.

[0078] The start-end point judgment branch is used to perform the regression task of calculating the start and end points of the wake-up word and output the calculated start and end points of the wake-up word. For example, during training, the start-end point judgment branch only inputs samples containing the wake-up word for training. Specifically, the start-end point judgment branch inputs the one-dimensional vector data obtained after processing by the neural network structure, and outputs the location of the start and end points of the wake-up word in the sample through training.

[0079] Continuing to refer to FIG. 2 , step S203 includes the following steps:

[0080] Step S2031: Input the output vector of the neural network structure to the wake-up word classification branch and the start-end point judgment branch respectively, and the wake-up word classification branch outputs the probability that the audio sample contains the wake-up word, and the start-end point judgment branch outputs the start point position and end point position of the wake-up word in the audio sample; and

[0081] Step S2032: Use the output results of the wake-up word classification branch and the output results of the start-end point judgment branch to adjust the parameters of the wake-up word classification branch and the parameters of the start-end point judgment branch respectively to obtain updated wake-up word classification branch and start-end point judgment branch.

[0082] In steps S2031 and S2032, taking the wake-up word classification branch as an example, in some examples, it includes at least a first fully connected layer structure connected to the neural network structure, and the output vector of the neural network structure includes first label data for the wake-up word classification branch. The first label data involves a first attribute, and the first label data is used to characterize whether the output vector contains the wake-up word and is input into the first fully connected layer structure as a first supervised learning target.

[0083] In some examples, see FIG3 , which is a flowchart of a method for training a wake-up word classification branch of a voice wake-up model provided in an embodiment of the present application. Adjusting the parameters of the wake-up word classification branch using the output result of the wake-up word classification branch in step S2032 includes:

[0084] Step S301: Determine a first loss function of the wake-up word classification branch based on the first attribute of the output vector and the output result of the wake-up word classification branch.

[0085] In this step, the first loss function of the wake-up word classification branch is cross entropy.

[0086] Step S302: Adjust the parameters of the wake-up word classification branch based on the first loss function.

[0087] In this step, the parameters of the wake-up word classification branch are iteratively updated according to the calculated value of the first loss function until convergence to obtain an updated wake-up word classification branch.

[0088] In one example, the calculation formula of the first loss function is:

[0089] in, is the label of the wake-up word; when the output vector contains the wake-up word, When the output vector does not contain the wake-up word, y is the wake-up word probability output by the wake-up word classification branch, and its range is [0,1].

[0090] Based on this, the calculated wake-up word probability can be compared with the information in the audio sample that has been labeled to determine whether the audio sample contains the wake-up word. On the one hand, it can be used to evaluate the training effect, and on the other hand, it can be used to adjust the parameters to optimize the wake-up word classification branch, and the wake-up word classification branch with optimized parameters is used as the updated wake-up word classification branch. In this embodiment, by adjusting the parameters, the wake-up word probability output by the wake-up word classification branch can be close to or equal to the label of the labeled wake-up word.

[0091] In step S2031 and step S2032, taking the start and end point judgment branch as an example, it at least includes a second fully connected layer structure connected to the neural network structure, and the output vector of the neural network structure includes second label data for the start and end point judgment branch. The second label data involves a second attribute, and the second label data is used to represent the starting point position and the ending point position of the wake-up word in the output vector and is input into the second fully connected layer structure as a second supervised learning target.

[0092] In some examples, referring to FIG. 4 , which is a flowchart of a method for training a start-end point determination branch of a voice wake-up model provided in an embodiment of the present application, wherein adjusting the parameters of the start-end point determination branch using the output results of the start-end point determination branch in step S2032 includes:

[0093] Step S401: Determine a second loss function of the start-end point judgment branch according to the second attribute of the output vector and the output result of the start-end point judgment branch.

[0094] In this step, the second loss function of the start and end point judgment branch is the mean square error between the output start point position and end point position and the true start point position and end point position.

[0095] Step S402: adjusting the parameters of the start and end point judgment branch based on the second loss function.

[0096] In this step, the parameters of the start and end point judgment branch are iteratively updated according to the calculated value of the second loss function until convergence to obtain an updated start and end point judgment branch.

[0097] In one example, the second loss function is calculated as: L2 = (s1-k1) 2 +(s2-k2) 2

[0098] Among them, L2 is the mean square error value;

[0099] s1 is the actual starting point of the wake-up word in the output vector;

[0100] s2 is the actual end point position of the wake-up word in the output vector;

[0101] k1 is the starting point position of the start and end point judgment branch output;

[0102] k2 is the end point position of the start and end point judgment branch output.

[0103] Based on this, the calculated start and end points of the wake-up word can be compared with the start and end point information marked in the audio sample. On the one hand, it can be used to evaluate the training effect, and on the other hand, it can be used to adjust the parameters to optimize the start and end point judgment branch, and the start and end point judgment branch with optimized parameters is used as the updated start and end point judgment branch. In this embodiment, by adjusting the parameters, the start point position k1 and end point position k2 output by the start and end point judgment branch can be close to or equal to the actual start point position s1 and the actual end point position s2 of the wake-up word in the output vector.

[0104] Referring again to Figure 1, in some examples, the output of the wake-up word classification branch and / or the output of the start-end point determination branch can be used to adjust the parameters of the neural network structure to obtain an updated neural network structure. For example, the neural network structure can use an error backpropagation algorithm to correct the parameters of the initial neural network structure during training, so that the reconstruction error loss of the neural network structure becomes smaller and smaller until the error loss converges.

[0105] To help those skilled in the art better understand the present application, a more detailed embodiment is listed below. As shown in FIG5 , the present embodiment provides a method for training a voice wake-up model, including the following steps:

[0106] Step S501: Generate wake-up word samples and non-wake-up word samples.

[0107] In this step, the wake-up word for the voice wake-up model is set to "Hi Lexin," and all other audio samples are non-wake-up words. The wake-up word can be sampled by manually recording the voice "Hi Lexin" in a quiet, low-noise environment. To enhance training effectiveness, after collecting the wake-up word samples, the complexity of the wake-up word samples is increased by adding noise and adjusting the position of the wake-up word in the sample audio, thereby improving training results. Non-wake-up word samples are obtained by intercepting audio from TV dramas, movies, and other audio sources that contain human voices but do not contain wake-up words.

[0108] Step S502: Mark the start and end points of the wake-up word.

[0109] In this step, the start and end points of the wake-up word in the audio sample are required for training purposes. This can be done by machine or manual annotation, marking the start and end points of the wake-up word in the audio sample. Non-wake-up word samples do not need to be annotated.

[0110] Step S503: intercepting the data frame.

[0111] In this step, a 1-second window is used, starting from the start point where the wake-up word in the audio sample is recognized, and moving at intervals of 32ms to intercept several data frames, and optionally upload them to the cloud for secondary recognition of the wake-up word.

[0112] In one example, in order to avoid repeated calculations to save computing resources and improve recognition speed, among the several intercepted data frames containing the wake-up word, only the first data frame can be uploaded to the cloud for secondary recognition of the wake-up word.

[0113] Step S504: feature extraction and feature enhancement of the data frame.

[0114] In this step, the one-dimensional array in the speech sample is subjected to feature extraction and feature enhancement through MFCC, and 50*32 32-bit feature two-dimensional arrays are output.

[0115] Step S505: Output a one-dimensional vector through convolution calculation

[0116] In this step, 50*32 32-bit feature two-dimensional array samples are input into the neural network structure, and a one-dimensional vector is output after convolution calculation.

[0117] Step S506: Training the wake-up word classification branch and the start and end point judgment branch

[0118] In this step, the one-dimensional vector output by the neural network structure is input into the wake-up word classification branch and the start and end point judgment branch respectively, and the wake-up word and non-wake-up word classification task and the regression task of calculating the start and end points of the wake-up word are run simultaneously.

[0119] Both the wake-up word classification branch and the start-end point judgment branch adopt a fully connected layer structure, also known as a linear layer structure, which is used for the output of the last layer of the neural network for subsequent calculations. Among them, the loss function of the wake-up word classification branch is cross entropy. The loss function of the start-end point judgment branch is the mean square error between the output start point position and end point position and the true start point position and end point position. Accordingly, the specific data types and parameter adjustment methods of the wake-up word classification branch and the start-end point judgment branch refer to the specific description of the first loss function and the second loss function above, which will not be repeated here. Taking into account the different parameters used in the subsequent calculations of the wake-up word classification branch and the start-end point judgment branch, the two tasks need to be trained separately.

[0120] The wake-up word classification branch is used to train the wake-up word classification task. The wake-up word classification branch takes as input a one-dimensional vector data sample of a wake-up word and a non-wake-up word. After training, it outputs two sets of data: the probability that the sample is a wake-up word and the probability that the sample is not a wake-up word, with the sum of these two probabilities being 1. The output of the wake-up word classification branch is compared with the audio information to evaluate the model training results and is used to adjust parameters to optimize the wake-up word classification branch.

[0121] The start-end point determination branch is used to train the regression task of the wake-up word's starting and ending points. The start-end point determination branch's input consists only of one-dimensional vector data samples of the wake-up word. After training, it outputs two sets of data: the start and end points of the wake-up word. The output of the wake-up word classification branch is compared with the audio information to evaluate the model's training results and is used to adjust parameters to optimize the start-end point determination branch.

[0122] The embodiments of the present application have been extensively tested and proven effective. For example, on an ESP32-S3 with very limited resources, it achieved a recognition rate of over 95% using only 5% of the CPU's computing resources and passed Amazon's Alexa performance test.

[0123] In summary, the training method of the voice wake-up model of the embodiment of the present application adopts a multi-task training method, and the wake-up word classification branch and the start and end point judgment branch are set in parallel at the end of the wake-up word detection network, and the classification task of the wake-up word and the regression task of the start and end points of the wake-up word are trained at the same time, so as to realize whether the output audio sample contains the wake-up word, and also output the starting point position and the end point position of the wake-up word in the audio sample. Therefore, the voice wake-up model trained by the embodiment of the present application can quickly and accurately obtain the starting point and end point of the wake-up word, which helps to more accurately intercept the wake-up word in the audio frame. In addition, since the classification task of the wake-up word and the regression task of the start and end points of the wake-up word are trained at the same time, it can help to enhance the effect of the neural network structure, and can further improve the accuracy of the wake-up word detection compared to a single training task. In addition, since the embodiment of the present application can intercept the wake-up word more accurately, it can be particularly advantageous in scenarios where secondary verification in the cloud is required.

[0124] Based on the above embodiments, the trained voice wake-up model can be pre-installed in the voice wake-up device, so that the voice wake-up device has the voice wake-up function. Accordingly, the embodiment of the present application provides an execution method of the voice wake-up model, as shown in Figure 6, the execution method includes the following steps:

[0125] Step S601: Acquire voice data to be detected.

[0126] Step S602: inputting the voice data to be detected into the trained voice wake-up model.

[0127] Step S603: Obtain the output result of the wake-up word classification branch and the output result of the start and end point judgment branch.

[0128] Step S604: outputting a wake-up result of the speech wake-up model based on the output result of the wake-up word classification branch and / or the output result of the start-end point judgment branch.

[0129] In some embodiments, the execution method of the above-mentioned voice wake-up model further includes: sending the output result of the wake-up word classification branch and / or the output result of the start and end point judgment branch to the voice recognition cloud platform for secondary recognition of the wake-up word.

[0130] The voice wake-up execution method of the embodiment of the present application is based on the above-mentioned trained voice wake-up model. It can output whether the audio sample contains the wake-up word, and also output the starting point and ending point of the wake-up word in the audio sample, thereby significantly improving the accuracy of wake-up word detection and greatly improving the performance of voice wake-up. In addition, the present application can more accurately intercept the wake-up word in the audio frame in scenarios requiring cloud-based secondary verification. Therefore, it can be particularly advantageous in scenarios requiring cloud-based secondary verification.

[0131] Referring to Figure 7, an embodiment of the present application also provides a training device for a voice wake-up model, including an acquisition module, a generation module and a training module. The acquisition module is used to acquire audio samples and perform acoustic feature processing on the audio samples to obtain acoustic feature frames. The generation module is used to input the acoustic feature frame into the neural network structure to generate an output vector, wherein the output vector has a first attribute and / or a second attribute, the first attribute is whether it contains a wake-up word, and the second attribute is the starting point position and the ending point position of the wake-up word in the output vector. The training module is used to train the wake-up word classification branch and the start-end point judgment branch according to the output vector, including: inputting the output vector into the wake-up word classification branch and the start-end point judgment branch respectively, and the wake-up word classification branch outputs the probability that the output vector contains the wake-up word, and the start-end point judgment branch outputs the starting point position and the ending point position of the wake-up word in the output vector; and, using the output result of the wake-up word classification branch and the output result of the start-end point judgment branch to adjust the parameters of the wake-up word classification branch and the parameters of the start-end point judgment branch respectively to obtain an updated voice wake-up model.

[0132] In this embodiment, the specific limitations, implementation principles, and beneficial effects of the training device for the voice wake-up model can be found in the description of the training method for the voice wake-up model above, and will not be repeated here. The above modules can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0133] Referring to Figure 8, an embodiment of the present application also provides a voice wake-up device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the training steps of the voice wake-up model and / or the execution steps of the voice wake-up model in the above-mentioned embodiments can be implemented.

[0134] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the training steps of the voice wake-up model and / or the execution steps of the voice wake-up model in the above-mentioned embodiments can be implemented.

[0135] An embodiment of the present application further provides a computer program, which, when executed by a processor, can implement the training steps of the voice wake-up model and / or the execution steps of the voice wake-up model in the above-mentioned embodiments.

[0136] In the above embodiments, the implementation principles and beneficial effects of the voice wake-up device, computer-readable storage medium, and computer program of the voice wake-up model can be found in the description of the training method and execution method of the voice wake-up model above, and will not be repeated here.

[0137] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0138] Although various embodiments of various aspects of the present application have been described for the purposes of this disclosure, it should not be understood that the teachings of this disclosure are limited to these embodiments. Features disclosed in a specific embodiment are not limited to that embodiment, but can be combined with features disclosed in different embodiments. For example, one or more features and / or operations of the method according to the present application described in one embodiment may also be applied individually, in combination or as a whole in another embodiment. It should be understood by those skilled in the art that there are possible more optional implementations and variations, and various changes and modifications may be made to the above-mentioned system without departing from the scope defined by the claims of this application.

Claims

1. A method for training a voice wake-up model, characterized in that: The voice wake-up model includes a neural network structure, a wake-up word classification branch, and a start and end point judgment branch. The method includes: Acquiring an audio sample, and performing acoustic feature processing on the audio sample to obtain an acoustic feature frame; Inputting the acoustic feature frame into a neural network structure to generate an output vector, wherein the output vector has a first attribute and / or a second attribute, the first attribute being whether the wake-up word is included, and the second attribute being a starting point position and an ending point position of the wake-up word in the output vector; and According to the output vector, the wake-up word classification branch and the start and end point judgment branch are trained respectively, including: The output vector is input to the wake-up word classification branch and the start-end point judgment branch respectively, and the wake-up word classification branch outputs the probability that the audio sample contains the wake-up word, and the start-end point judgment branch outputs the start point position and the end point position of the wake-up word in the audio sample; and The output result of the wake-up word classification branch and the output result of the start-end point judgment branch are used to adjust the parameters of the wake-up word classification branch and the parameters of the start-end point judgment branch respectively to obtain the updated wake-up word classification branch and the start-end point judgment branch.

2. The method according to claim 1, wherein: The wake-up word classification branch includes at least a first fully connected layer structure connected to the neural network structure, and the output vector of the neural network structure includes first label data for the wake-up word classification branch, the first label data involves a first attribute, and the first label data is used to characterize whether the output vector contains the wake-up word and is input into the first fully connected layer structure as a first supervised learning target.

3. The method according to claim 1, wherein: Adjusting the parameters of the wake-up word classification branch using the output result of the wake-up word classification branch includes: Determining a first loss function of the wake-up word classification branch according to the first attribute of the output vector and the output result of the wake-up word classification branch; The parameters of the wake-up word classification branch are adjusted based on the first loss function.

4. The method according to claim 3, wherein: The adjusting the parameters of the wake-up word classification branch based on the first loss function includes: Iteratively update the parameters of the wake-up word classification branch according to the calculated value of the first loss function until convergence to obtain the updated wake-up word classification branch.

5. The method according to claim 3, wherein: The first loss function of the wake-up word classification branch is cross entropy.

6. The method according to claim 5, wherein: The calculation formula of the first loss function is: in, is the label of the wake-up word; when the output vector contains the wake-up word, When the output vector does not contain the wake-up word, y is the wake-up word probability output by the wake-up word classification branch, and its range is [0,1].

7. The method according to claim 1, wherein: The start-end point judgment branch includes at least a second fully connected layer structure connected to the neural network structure, and the output vector of the neural network structure includes second label data for the start-end point judgment branch, the second label data involves a second attribute, and the second label data is used to characterize the starting point position and the ending point position of the wake-up word in the output vector and is input into the second fully connected layer structure as a second supervised learning target.

8. The method according to claim 1, wherein: Adjusting the parameters of the start-end point judgment branch using the output result of the start-end point judgment branch includes: Determine a second loss function of the start-end point judgment branch according to the second attribute of the output vector and the output result of the start-end point judgment branch; The parameters of the start and end point determination branch are adjusted based on the second loss function.

9. The method according to claim 8, wherein: The adjusting the parameters of the start and end point judgment branch based on the second loss function includes: The parameters of the start-end point judgment branch are iteratively updated according to the calculated value of the second loss function until convergence, so as to obtain the updated start-end point judgment branch.

10. The method according to claim 8, wherein: The second loss function of the start-end point judgment branch is the mean square error between the output start point position and end point position and the actual start point position and end point position.

11. The method according to claim 10, wherein: The calculation formula of the second loss function is: L2 = (s1-k1) 2 +(s2-k2) 2 Among them, L2 is the mean square error value; s1 is the actual starting point position of the wake-up word in the output vector; s2 is the actual end point position of the wake-up word in the output vector; k1 is the starting point position output by the start and end point judgment branch; k2 is the end point position output by the start and end point judgment branch.

12. The method according to claim 1, further comprising: The parameters of the neural network structure are adjusted using the output result of the wake-up word classification branch and / or the output result of the start and end point judgment branch to obtain an updated neural network structure.

13. The method according to claim 1, wherein: The performing feature processing on the audio sample comprises: Performing endpoint detection on the wake-up word in the audio sample, starting from the start point of the wake-up word, intercepting data frames according to a time window of a fixed length until the end point of the wake-up word, so as to obtain a plurality of data frames of the same time length; the plurality of data frames are in the data form of a one-dimensional array; and Audio spectrum features are extracted from the one-dimensional array, and a two-dimensional feature array is output.

14. The method according to claim 13, wherein: In response to the intercepted data frame including a plurality of the wake-up words, the data frame including the first wake-up word is retained.

15. The method according to claim 13, wherein: Inputting the acoustic feature frame into a neural network structure to generate an output vector comprises: The two-dimensional feature array is input into the neural network structure, and a one-dimensional vector is generated through a convolution algorithm.

16. A method for executing a voice wake-up model, comprising: Obtain the voice data to be detected; Inputting the voice data to be detected into the voice wake-up model trained by any one of claims 1 to 15; Obtaining the output result of the wake-up word classification branch and the output result of the start and end point judgment branch; as well as The wake-up result of the voice wake-up model is output based on the output result of the wake-up word classification branch and / or the output result of the start and end point judgment branch.

17. The method according to claim 16, further comprising: The output result of the wake-up word classification branch and / or the output result of the start and end point judgment branch are sent to the speech recognition cloud platform for secondary recognition of the wake-up word.

18. A training device for a voice wake-up model, characterized in that: The voice wake-up model includes a neural network structure, a wake-up word classification branch and a start and end point judgment branch, and the device includes: An acquisition module, used for acquiring audio samples and performing acoustic feature processing on the audio samples to obtain acoustic feature frames; A generating module, configured to input the acoustic feature frame into a neural network structure to generate an output vector, wherein the output vector has a first attribute and / or a second attribute, the first attribute being whether the wake-up word is included, and the second attribute being a starting point position and an ending point position of the wake-up word in the output vector; and A training module, which trains the wake-up word classification branch and the start and end point judgment branch respectively according to the output vector, including: The output vector is input to the wake-up word classification branch and the start-end point judgment branch respectively, and the wake-up word classification branch outputs the probability that the output vector contains the wake-up word, and the start-end point judgment branch outputs the start point position and the end point position of the wake-up word in the output vector; and The output result of the wake-up word classification branch and the output result of the start-end point judgment branch are used to adjust the parameters of the wake-up word classification branch and the parameters of the start-end point judgment branch respectively to obtain an updated voice wake-up model.

19. A voice wake-up device, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the processor executes the computer program, the training method described in any one of claims 1 to 15 and / or the execution method described in claim 16 or 17 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method according to any one of claims 1 to 15 and / or the execution method according to claim 16 or 17 are implemented.

Images