Voice wake-up method, training method of acoustic model, and related devices

Abstract

The application discloses a voice wake-up method, a training method of an acoustic model and related devices. The voice wake-up method comprises the following steps: receiving a voice signal to be recognized, and obtaining an acoustic feature of the voice signal to be recognized; inputting the acoustic feature into a trained acoustic model to obtain a probability of a wake-up word; wherein the step of training the acoustic model comprises the following steps: constructing an enhanced model, the enhanced model comprising a first branch which is the same as the acoustic model; training the enhanced model and the acoustic model together by using training data, and synchronously updating and keeping the same parameters of the first branch and the acoustic model; and determining whether to wake up based on the probability. In the foregoing manner, the application can improve the frame classification accuracy of the acoustic model in a noise scene, thereby greatly improving the wake-up rate in the noise scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of voice wake-up, and particularly relates to a voice wake-up method, a training method of an acoustic model and related devices. BACKGROUND

[0002] With the rapid development of the Internet of Things and artificial intelligence technology, intelligent terminal devices such as sound boxes and large screens are gradually popularized. In addition, the voice interaction mode is gradually replacing the traditional manual interaction mode, and as the first step to start the voice interaction of the intelligent terminal, the voice wake-up system has always been a research hotspot in the related field.

[0003] The existing voice wake-up system includes an acoustic model and a decoder, wherein the acoustic model takes an acoustic state as a modeling unit, and the decoder takes the acoustic posterior probability of each frame of voice as input to perform Viterbi decoding. When the probability of the wake-up word obtained by decoding is greater than a preset threshold, the intelligent device enters a working state from a dormant state and receives the user's instruction. Limited by the limited storage and computing capacity of the intelligent terminal, the acoustic model carried by the intelligent terminal usually adopts a lightweight network structure (such as mobilenet_v2, shufflenet_v2, etc.). Due to the small number of parameters of the lightweight network, the representation ability is limited, which leads to a low classification accuracy of the voice frame. In particular, in a noisy environment, the underfitting phenomenon of the acoustic model is more obvious, and the frame classification accuracy sharply decreases, which further leads to a sharp decrease in the wake-up rate in a noisy environment. SUMMARY

[0004] The application provides a voice wake-up method, a training method of an acoustic model and related devices to improve the frame classification accuracy of the acoustic model in a noisy environment, and then greatly improve the wake-up rate in a noisy environment.

[0005] To solve the above technical problems, one technical solution adopted by the application is to provide a voice wake-up method, which includes: receiving a to-be-recognized voice signal and obtaining acoustic features of the to-be-recognized voice signal; inputting the acoustic features into a trained acoustic model to obtain a probability of a wake-up word; wherein the step of training the acoustic model includes: constructing an enhanced model, the enhanced model including a first branch identical to the acoustic model; training the enhanced model and the acoustic model together using training data, and synchronously updating and keeping the same parameters of the first branch and the acoustic model; and determining whether to wake up based on the probability.

[0006] To solve the above technical problems, another technical solution adopted by the present application is to provide a training method of an acoustic model, comprising: constructing an enhanced model; wherein the enhanced model comprises a first branch same as the acoustic model to be trained; training the enhanced model and the acoustic model together by using training data, and the parameters of the first branch and the acoustic model are updated synchronously and remain the same.

[0007] To solve the above technical problems, another technical solution adopted by the present application is to provide a voice wake-up device, comprising: a first obtaining module configured to receive a voice signal to be recognized and obtain acoustic features of the voice signal to be recognized; a second obtaining module connected with the first obtaining module, configured to input the acoustic features into a trained acoustic model to obtain a probability value of a wake-up word; an execution model connected with the second obtaining module, configured to determine whether to wake up based on the probability value; a training module connected with the second obtaining module, configured to construct an enhanced model, wherein the enhanced model comprises a first branch same as the acoustic model; and configured to train the enhanced model and the acoustic model together by using training data, and the parameters of the first branch and the acoustic model are updated synchronously and remain the same.

[0008] To solve the above technical problems, another technical solution adopted by the present application is to provide an electronic device, comprising a memory and a processor coupled with each other, the memory has program instructions stored therein, and the processor is configured to execute the program instructions to implement the method in any of the above embodiments.

[0009] To solve the above technical problems, another technical solution adopted by the present application is to provide a storage device having program instructions capable of being executed by a processor, wherein the program instructions are configured to implement the method in any of the above embodiments.

[0010] Differently from the prior art, the application has the following beneficial effects: in the application, an enhanced model is introduced when training the acoustic model, and the acoustic model is a submodel of the enhanced model, and the two share model parameters. On the one hand, the acoustic model is trained in the enhanced model with larger parameter capacity, and the enhanced model can provide additional supervision information, so that the acoustic model jumps out of the local optimum and converges to a better state. On the other hand, since the acoustic model is a submodel of the enhanced model, the improvement of the acoustic model further improves the effect of the enhanced model, and the better enhanced model further improves the effect of the acoustic model, and so on, so that the acoustic model and the enhanced model finally achieve higher effects. On the other hand, the enhanced model is only started in the training stage, and only the acoustic model is deployed to the intelligent device for forward inference calculation, without additional calculation overhead. Therefore, the speech wake-up method provided by the application can improve the underfitting problem of the acoustic model, improve the frame classification accuracy of the acoustic model in a noisy environment, and thus greatly improve the wake-up rate in a noisy environment; and it will not bring additional calculation overhead and will not increase the response time of the device.

[0011] In addition, the application adopts a knowledge distillation mode for training, takes the enhanced model as a teacher model and the acoustic model as a student model, adds a KL divergence loss to the student model to make it as close as possible to the teacher model. Due to the larger model capacity, the teacher model can achieve a higher frame classification accuracy, and under the constraint of the teacher model, the model parameters are redundant, and compared with direct training which is easy to fall into a local optimum, this training mode can make the student model achieve a better parameter configuration and achieve a higher frame classification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings, wherein:

[0013] Figure 1 Flowchart of an embodiment of the speech wake-up method of the application;

[0014] Figure 2 Flowchart of an embodiment of the training method of the acoustic model of the application;

[0015] Figure 3 Structure diagram of an embodiment of the acoustic model and the enhanced model;

[0016] Figure 4 Structure diagram of an embodiment of the speech wake-up device of the application;

[0017] Figure 5 Fig. 1 is a structural schematic diagram of an embodiment of an electronic device according to the present application;

[0018] Figure 6 Fig. 2 is a structural schematic diagram of an embodiment of a storage device according to the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0020] In order to improve the wake-up rate in a noise scene, the existing method generally adopts the following two methods to solve the problem.

[0021] Method one: a secondary confirmation model is introduced to correct the wake-up result. Specifically, a binary classification model can be added on the basis of the existing acoustic model. In a noise scene, the frame accuracy of the acoustic model classification is low, resulting in a low decoding score of the voice containing the wake-up word. In order to reduce the missed wake-up as much as possible, the threshold needs to be appropriately reduced; when the total decoding score of the acoustic model exceeds the threshold, the hidden layer features of the continuous multiple frames of the acoustic model are input into the secondary classification model to further confirm whether the current voice contains the wake-up word. That is, this method recalls the voice segment suspected to contain the wake-up word as much as possible through the first-level acoustic model, and then confirms whether the voice segment contains the wake-up word through the secondary classification model, which can greatly improve the wake-up rate in a noise scene. However, due to the increase of the secondary confirmation model, the parameter quantity and the calculation quantity of the wake-up model increase, which not only brings additional storage and calculation resource burden to the intelligent terminal, but also increases the power consumption and response time of the intelligent terminal, affecting the user experience.

[0022] Method two: a multi-path PK (challenge) mode is adopted. Specifically, after the user voice and the background noise are received by the intelligent terminal, they are sent to the front-end module for noise reduction processing; after the sound source positioning, echo cancellation, beam forming, voice enhancement and other processing steps, the multi-path noise reduction audio is finally output; multiple wake-up processes are started, and the multi-path audio is sent into the corresponding wake-up process for forward decoding calculation; as long as one path successfully wakes up in the same time interval, it is considered that there is a wake-up word in the voice segment, and the intelligent terminal enters the working state from the sleep state, preparing to receive the user's voice instruction. This multi-path PK mode can effectively improve the wake-up rate in a noise scene. However, since multiple wake-up processes need to be started at the same time to perform forward decoding calculation on the multi-path audio, not only does it bring a huge calculation burden to the intelligent terminal, but also the multi-path PK strategy brings a certain time delay, resulting in an increase in the response time of the intelligent terminal.

[0023] To solve the above technical problems, please refer to Figure 1 , Figure 1 The flowchart of an embodiment of the voice wake-up method of the present application specifically includes:

[0024] S101: receiving a to-be-identified voice signal and obtaining an acoustic feature of the to-be-identified voice signal.

[0025] Specifically, in the present embodiment, the microphone of the intelligent terminal obtains the to-be-identified voice signal, and then sends the to-be-identified voice signal into the processor of the intelligent terminal; the processor sequentially performs frame division, windowing and Fourier transform processing on the to-be-identified voice signal to obtain a plurality of to-be-identified frequency domain features, wherein the plurality of to-be-identified frequency domain features constitute the acoustic feature. For example, the number of to-be-identified frequency domain features after feature extraction and conversion of a to-be-identified voice signal with a time length of one second can be 100 frames.

[0026] S102: inputting the acoustic feature into the trained acoustic model to obtain a probability of a wake-up word.

[0027] Specifically, in the present embodiment, the acoustic model can include at least one convolutional layer, pooling layer and fully connected layer, etc.; the structure of the acoustic model can be any one of the prior art, for example, the acoustic model can be a Gaussian-mixture hidden Markov model GMM-HMM, a deep neural network-hidden Markov model DNN-HMM, a deep recurrent neural network-hidden Markov model RNN-HMM, a deep convolutional neural network-hidden Markov model CNN-HMM, a connection time sequence classification-long short-term memory model CTC-LSTM, etc.

[0028] In one embodiment, please refer to Figure 2 , Figure 2 The flowchart of an embodiment of the training method of the acoustic model of the present application, the steps of training the acoustic model specifically include:

[0029] S201: constructing an enhanced model; wherein the enhanced model includes a first branch identical to the acoustic model to be trained.

[0030] Specifically, before the above-mentioned construction of the enhanced model, it can also include: expanding the width of at least part of the layers of the acoustic model by an integer multiple to construct an initial model; wherein the initial model includes a first branch identical to the acoustic model and a second branch different from the acoustic model. Since the size of the acoustic model increases linearly with the increase of the parameter quantity, in order to reduce the training overhead, only the initial model (i.e. the maximum enhanced model) is maintained. In addition, the above-mentioned way of constructing the initial model by increasing the width of at least part of the layers of the acoustic model produces less training time overhead on the processor compared with increasing the depth of at least part of the layers of the acoustic model.

[0031] Preferably, each layer width of the acoustic model can be enlarged by an integer multiple to construct the initial model. In this way, the diversity of the enhanced model constructed based on the initial model can be stronger. Moreover, each layer width in the initial model has a first ratio with the corresponding layer width in the acoustic model, and the first ratios of all layers in the initial model are the same. This design can reduce the difficulty of constructing the initial model.

[0032] For example, the acoustic model includes a convolution layer, a pooling layer and a full connection layer, and the channel number of each of the convolution layer, the pooling layer and the full connection layer can be expanded to r times of the original number to construct the initial model; r is the first ratio mentioned above; according to experience, r can be 3, etc. At this time, assuming that the width (i.e. the number of convolution kernels) of a certain layer of the original acoustic model is w, the width of the corresponding layer in the initial model is r*w.

[0033] Further, the step of constructing the enhanced model in the step S201 includes: constructing the enhanced model from the first branch and at least part of the second branch in the initial model. The process of constructing the enhanced model is relatively simple.

[0034] Generally, the number of times of training the acoustic model and the enhanced model can be multiple, and at this time, the structure of the enhanced model for different training times can be different, so that the corresponding enhanced model becomes diversified for each training, and the accuracy of the trained acoustic model is improved. For example, the step of constructing the enhanced model includes: A, obtaining a diversity parameter matrix based on the current training time; each element s in the diversity parameter matrix is an integer greater than 1 and less than the first ratio r, and each element s corresponds to each layer in the initial model; optionally, the diversity parameter matrix can be randomly generated. B, for each layer in the initial model, obtaining a width selection range corresponding to the current layer, wherein the width selection range includes the width w of the corresponding layer of the acoustic model, the product s*w of the width of the corresponding layer of the acoustic model and the element, and the width r*w of the corresponding layer of the initial model. C, selecting any one from the width selection range as the width of the corresponding layer of the enhanced model, and the corresponding layer of the enhanced model includes the corresponding layer of the acoustic model.

[0035] Optionally, the values of each element s in the diversity parameter matrix are the same. This design can reduce the computational load of the processor and reduce the difficulty of constructing the enhanced model.

[0036] In one application scenario, as shown in Figure 3 Figure 3 ​Fig. 1 is a structural schematic diagram of an embodiment of the acoustic model and the enhanced model. Only three layers of the acoustic model 10 and the enhanced model 12 are shown respectively. Assuming that each layer of the acoustic model 10 contains two convolution kernels, the first ratio r corresponding to each layer is 3, each layer of the initial model contains six convolution kernels, and each element s of the diversity parameter matrix corresponding to the current training number is 2; at this time, when the enhanced model 12 is constructed, the width corresponding to each layer can be selected from any one of w (i.e., 2), s*w (i.e., 2*2) and r*w (i.e., 3*2). For example, Figure 3 In the case of Fig. 1, the width of the first layer of the enhanced model 12 is selected as s*w, the width of the second layer is selected as r*w, and the width of the third layer is selected as w, and the enhanced model must contain Figure 3 In the case of Fig. 1, the acoustic model 10 contains two columns.

[0037] S202: The enhanced model and the acoustic model are jointly trained using the training data, and the parameters of the first branch and the acoustic model are updated synchronously and kept the same.

[0038] Specifically, in one embodiment, the above step S202 or the above step S201 includes constructing training data. The specific process of constructing the training data can be as follows:

[0039] A, obtaining a plurality of sample data; wherein the sample data is positive example voice data or negative example voice data; the positive example voice data contains a wake-up word, the negative example voice data does not contain a wake-up word, and the positive example voice data and the negative example voice data are provided with corresponding text labels.

[0040] B, forcibly aligning each sample data and the corresponding text label to obtain a state label corresponding to each frame of audio in the sample data.

[0041] Specifically, a decoding path of the sample data can be constructed according to the corresponding text label of the sample data by using a larger model that has been trained by the cloud, and then the scores of all alignment paths (each frame of speech feature corresponds to an acoustic state state label) are calculated by using the Viterbi algorithm, and the alignment path corresponding to the maximum value is taken as the state label of the audio.

[0042] C, constructing spliced data using a plurality of sample data, and the plurality of spliced data constitute the training data; wherein the spliced data is formed by splicing at least one positive example voice data and at least one negative example voice data, and the state label of the spliced data is sequentially composed of the state labels of the positive example voice data and the negative example voice data forming the spliced data.

[0043] Since the wake-up word audio segment is usually followed by the non-wake-up word audio segment in actual use scenarios, the long sentence splicing manner for constructing the long spliced data can make the training data more matched with the actual use scenarios, and increase the robustness of the wake-up system. In addition, the long sentence splicing manner increases the complexity of the training data, which is beneficial to the subsequent training process, and makes the wake-up model converge to a better state.

[0044] Optionally, for each spliced data, the number of the negative example voice data constituting the spliced data can be more than the number of the positive example voice data. For example, the ratio of the number of the negative example voice data to the number of the positive example voice data is 3, etc. Generally speaking, too much positive example voice data can make the acoustic model have a large false wake-up probability, and too little positive example voice data can make the acoustic model have a large non-wake-up probability, so the ratio is given to balance the two.

[0045] Another option is that different sample data can have different corresponding amplitudes. In order to eliminate the influence of the inconsistent amplitude distribution of each sample data when forming the spliced data, the following manner can be taken:

[0046] A. Obtain the local amplitude maximum value of each sample data constituting the current spliced data, and the global amplitude maximum value among all the local amplitude maximum values.

[0047] For example, it is assumed that 3 positive example voice data (a1, a2 and a3) and 1 negative example voice data (b1) are needed to constitute the spliced data; the local amplitude maximum value of a1 is t1, the local amplitude maximum value of a2 is t2, the local amplitude maximum value of a3 is t3, and the local amplitude maximum value of b1 is t4, and t4 is the global amplitude maximum value among t1, t2, t3 and t4.

[0048] B. Perform amplitude normalization processing on the sample data based on the local amplitude maximum value of each sample data.

[0049] For example, taking the positive example voice data a1 as an example, the amplitude intensity at each position of the positive example voice data a1 is divided by t1 to limit the amplitude intensity at each position of the positive example voice data a1 to the range of 0-1.

[0050] C. After multiplying each sample data after amplitude normalization by the global amplitude maximum value, perform splicing processing to obtain the spliced data.

[0051] For example, taking the positive example voice data a1 as an example, the amplitude intensity at each position after normalization is multiplied by t4 to limit the amplitude intensity at each position of the positive example voice data a1 to the range of 0-t4, and at this time the amplitude intensity at each position of each positive example voice data and negative example voice data forming the spliced data is limited to the range of 0-t4.

[0052] Of course, in other embodiments, the implementation of step C above can also be: splicing the amplitude-normalized sample data of each sample, and then multiplying the spliced data by the global maximum amplitude.

[0053] In addition, it should be noted that the global maximum amplitude corresponding to different spliced data can be different, and this way can increase the diversity of training data. In other embodiments, the spliced data can also be augmented by adding noise, adding reverberation, and changing speed to obtain augmented spliced data and corresponding state labels.

[0054] In another embodiment, please refer to Figure 3 , the step of training the enhanced model and the acoustic model together using the training data in step S202 specifically includes:

[0055] A, input the training data into the enhanced model 12 to obtain the first prediction result p t (y|x), and input the training data into the acoustic model 10 to obtain the second prediction result p s (y|x).

[0056] B, based on the first prediction result p t (y|x) and the state label of the training data, obtain the first loss ce loss (p t ), based on the second prediction result p s (y|x) and the state label of the training data, obtain the second loss ce loss (p s ), based on the first prediction result p t (y|x) and the second prediction result, obtain the ce loss (p t )KL divergence loss kl div (p s ,p t ).

[0057] Specifically, the first loss ce loss (p t ) and the second loss ce loss (p s ) can be cross-entropy loss, etc.

[0058] C, based on the first loss ce loss (p t ), the second loss ce loss (p s ) and the KL divergence loss kl div (p s ,p t ), obtain the total loss.

[0059] Specifically, the total loss loss total is a sum of a first loss ce loss (p t ), a second loss ce loss (p s ), and a KL divergence loss kl div (p s , p t ).

[0060] D, updating parameters of the enhanced model, the acoustic model, and the initial model based on the total loss; wherein parameters of the first branch in the enhanced model and the initial model are kept the same as parameters of the acoustic model, and parameters of the remaining branches in the enhanced model are the same as corresponding branch parameters in the initial model.

[0061] Specifically, gradients of the two models can be calculated according to the total loss, and the gradients of the two models are superimposed, and then the parameters of the enhanced model and the parameters of the acoustic model are synchronously adjusted according to the superimposed gradients; and the enhanced model includes a module of the acoustic model, and the parameters of the same convolution kernel part in the acoustic model and the enhanced model are kept consistent during parameter updating. In the above step D, the parameters in the initial model can be synchronously updated according to the enhanced model, and in the subsequent training process, the enhanced model is randomly selected from the updated initial model for training. That is, in response to the fact that the preset condition for stopping training is not reached (for example, in response to the fact that the number of training times does not reach the preset value, or in response to the fact that the total loss does not converge, etc.), the enhanced model is reconstructed based on the updated initial model, and the reconstructed enhanced model and the updated acoustic model are jointly trained using the next batch of training data. The process of reconstructing the enhanced model and the training process can be referred to the above embodiments, which will not be described here.

[0062] For example, the parameters of the acoustic model are (x, y), the parameters of the enhanced model are (x, y, z), and it is assumed that the back propagation gradients calculated by the acoustic model are A1 and the back propagation gradients calculated by the enhanced model are A2 under the current training number, then the parameter update of the enhanced model is (x-x1, y-y1, z-z1), and the parameter update of the acoustic model is (x-x1, y-y1), not only the parameter update of the enhanced model, at this time the parameter update of the maximum enhanced model is (x-x1, y-y1, z-z1, u, v, w); wherein the adjustment values x1 and y1 of the same parameters x and y in the acoustic model and the enhanced model are obtained based on A1 and A2; the adjustment value of the parameter z different from the acoustic model in the enhanced model is obtained based on A2. The above can be understood as gradient superposition, the gradient of the shared parameter part of the two models is superimposed, such as x and y, and the unshared part is updated separately, such as z, which is only the parameter of the enhanced model and only updates the gradient calculated by the enhanced model. The specific parameter update method corresponds to the following calculation formula:

[0063]

[0064] wherein, (w b i , w a i ) represents the enhanced model parameters of the current training number, w b i is the parameter of the acoustic model, a is the scaling parameter of the enhanced model gradient, according to the experience value, a can be 1, and η is the learning rate, in addition, in the training step, the new parameter w a i ) is also updated by gradient descent.

[0065] In the above design, the enhanced model is used as the teacher model, the acoustic model is used as the student model, the KL divergence loss is added to the student model, the acoustic model and the enhanced model are jointly trained, the acoustic model is the submodel of the enhanced model, the multi-task learning method is used for training, and the knowledge of the enhanced model is migrated to the acoustic model; as one of the main methods of model compression, knowledge distillation is more and more used for end-side model effect optimization, the traditional knowledge distillation method needs to train a larger teacher model in advance, then the parameters of the teacher model are fixed, the student model is constrained by the KL divergence to approximate the teacher model, and the teacher model is discarded after the student model is trained; unlike the traditional knowledge distillation method, the application constructs the enhanced model of the acoustic model as the teacher model, and the teacher model and the student model are jointly trained, without training the teacher model in advance, in addition, since the final optimization target is the student model, the traditional knowledge distillation method keeps the teacher model parameters fixed and adjusts the student model parameters, the application adjusts the student model parameters and the teacher model parameters simultaneously according to the gradients calculated based on the student model and the teacher model, keeps the parameters of the same convolution kernel in the model consistent, after one batch of training is completed, the student model and the teacher model are optimized cooperatively through parameter sharing, the teacher model is reconstructed based on the updated student model (the maximum enhanced model is updated first, and then the enhanced model is randomly generated as the teacher model) for the next training, and the next batch of training is performed. When the preset number of training words or the number of epochs (one epoch is to traverse all training data) is reached, the training is stopped.

[0066] S103: determining whether to wake up based on the probability.

[0067] Specifically, in this embodiment, when the probability of the wake-up word is greater than or equal to the threshold value, the intelligent terminal enters the working state from the sleep state; when the probability of the wake-up word is less than the threshold value, the intelligent terminal remains in the sleep state.

[0068] In the above design manner, the enhanced model is introduced when training the acoustic model, and the acoustic model is a sub-model of the enhanced model, and the two share model parameters. On the one hand, the acoustic model is trained by being embedded into the enhanced model with larger parameter capacity, the enhanced model can provide additional supervision information, so that the acoustic model jumps out of the local optimum and converges to a better state. On the other hand, since the acoustic model is a sub-model of the enhanced model, the improvement of the acoustic model further improves the effect of the enhanced model, and the better enhanced model further improves the effect of the acoustic model, and so on, so that the acoustic model and the enhanced model finally achieve higher effects. On the other hand, the enhanced model is only started in the training stage, and only the acoustic model is deployed to the intelligent device for forward inference calculation, which does not bring additional computational overhead. Therefore, the speech wake-up method provided by the present application can improve the under-fitting problem of the acoustic model, improve the frame classification accuracy of the acoustic model in a noisy environment, and thus greatly improve the wake-up rate in a noisy environment. Moreover, it will not bring additional computational overhead and will not increase the response time of the device.

[0069] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of an embodiment of a speech wake-up device of the present application. The speech wake-up device comprises a first obtaining module 20, a second obtaining module 22, an executing module 24 and a training module 26.

[0070] Specifically, the first obtaining module 20 is configured to receive a to-be-recognized speech signal and obtain an acoustic feature of the to-be-recognized speech signal. The second obtaining module 22 is connected with the first obtaining module 20 and is configured to input the acoustic feature into the trained acoustic model to obtain a probability value of a wake-up word. The executing module 24 is connected with the second obtaining module 22 and is configured to determine whether to wake up based on the probability value. The training module 26 is connected with the second obtaining module 22, and the training module 26 comprises a constructing sub-module 260 and a training sub-module 262 connected with each other. The training sub-module 262 is connected with the second obtaining module 22. The constructing sub-module 260 is configured to construct an enhanced model, and the enhanced model comprises a first branch same as the acoustic model. The training sub-module 262 is configured to train the enhanced model and the acoustic model together by using training data, and the parameters of the first branch and the acoustic model are updated synchronously and kept the same.

[0071] In one embodiment, the above constructing sub-module 260 is specifically configured to expand the width of at least part of layers of the acoustic model by an integer multiple to construct an initial model; wherein the initial model comprises a first branch same as the acoustic model and a second branch different from the acoustic model; and the enhanced model is constructed by selecting the first branch and at least part of the second branch from the initial model.

[0072] Further, the step of selecting the first branch and at least part of the second branch from the initial model to construct the enhanced model comprises: obtaining a diversity parameter matrix based on the current training number; each element in the diversity parameter matrix is an integer greater than 1 and less than a first ratio, each element corresponds to each layer in the initial model, and the first ratio is the ratio of the width of the corresponding layer in the initial model to the width of the corresponding layer in the acoustic model; for each layer in the initial model, a width selection range corresponding to the current layer is obtained, wherein the width selection range includes the width of the corresponding layer in the acoustic model, the product of the width of the corresponding layer in the acoustic model and the element, and the width of the corresponding layer in the initial model; and any one of the width selection range is selected as the width of the corresponding layer in the enhanced model, and the corresponding layer in the enhanced model includes the corresponding layer in the acoustic model.

[0073] Optionally, the step of expanding the width of at least part of the layers of the acoustic model by an integer multiple to construct the initial model comprises: expanding the width of each layer of the acoustic model by an integer multiple to construct the initial model, and the first ratio of all layers in the initial model is the same.

[0074] Optionally, the value of each element in the diversity parameter matrix is the same.

[0075] In an embodiment, the training submodule 262 is specifically configured to input the training data into the enhanced model to obtain a first prediction result, and input the training data into the acoustic model to obtain a second prediction result; obtain a first loss based on the first prediction result and the state label of the training data, obtain a second loss based on the second prediction result and the state label of the training data, and obtain a KL divergence loss based on the first prediction result and the second prediction result; obtain a total loss based on the first loss, the second loss and the KL divergence loss; update the parameters in the enhanced model, the acoustic model and the initial model based on the total loss; wherein the parameters of the first branch in the enhanced model and the initial model remain the same as the parameters of the acoustic model, and the parameters of the remaining branches in the enhanced model are the same as the parameters of the corresponding branches in the initial model.

[0076] In addition, the training submodule 262 is further configured to: in response to not reaching the preset condition for stopping training (for example, in response to the training number not reaching the preset value, or in response to the total loss not converging, etc.), reconstruct the enhanced model based on the updated initial model, and jointly train the reconstructed enhanced model and the updated acoustic model using the next batch of training data.

[0077] In another implementation, the training module 26 can further include a splicing sub-module connected with the constructing sub-module, configured to obtain a plurality of sample data; wherein the sample data is positive example voice data or negative example voice data; the positive example voice data contains the wake-up word, the negative example voice data does not contain the wake-up word, and the positive example voice data and the negative example voice data are provided with corresponding text labels; each sample data and the corresponding text label are forcedly aligned to obtain a state label corresponding to each frame of audio in the sample data; a plurality of sample data are used to construct splicing data, and the plurality of splicing data constitute the training data; wherein the splicing data are formed by splicing at least one positive example voice data and at least one negative example voice data, and the state label of the splicing data is sequentially constituted by the state labels of the positive example voice data and the negative example voice data forming the splicing data.

[0078] Optionally, the step of constructing the splicing data by using the plurality of sample data comprises: obtaining a local amplitude maximum value of each sample data used to construct the current splicing data, and a global amplitude maximum value among all the local amplitude maximum values; performing amplitude normalization processing on the sample data based on the local amplitude maximum value of each sample data; and multiplying each sample data after the amplitude normalization by the global amplitude maximum value, and then performing splicing processing to obtain the splicing data.

[0079] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of an embodiment of the electronic device. The electronic device comprises a memory 32 and a processor 30 coupled with each other, the memory 32 stores program instructions, and the processor 30 is configured to execute the program instructions to implement any of the voice wake-up methods or training methods described above. Specifically, the electronic device includes but is not limited to a desktop computer, a notebook computer, a tablet computer, a server, etc., which are not limited herein. In addition, the processor 30 can also be referred to as a CPU (Center Processing Unit). The processor 30 can be an integrated circuit chip with signal processing capability. The processor 30 can also be a general processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 30 can be jointly implemented by integrated circuit chips.

[0080] Please refer to Figure 6 , Figure 6An implementation of the storage device of the present application is shown in a structural schematic diagram. The storage device 40 stores program instructions 400 that can be executed by the processor, and the program instructions 400 are used to implement any of the voice wake-up methods or training methods described above.

[0081] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0082] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0083] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0084] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0085] The above merely describes the embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the present application specification and drawings, is also included in the patent protection scope of the present application.

Claims

1. A voice wake-up method, characterized in that, include: Receive the speech signal to be recognized and obtain the acoustic features of the speech signal to be recognized; The acoustic features are input into the trained acoustic model to obtain the probability of the wake word; wherein, the step of training the acoustic model includes: constructing an enhancement model, the enhancement model including the same first branch as the acoustic model; and training the enhancement model and the acoustic model together using training data, wherein the parameters of the first branch and the acoustic model are updated synchronously and remain the same; Whether to wake up is determined based on the probability; Before constructing the enhanced model, the method includes: increasing the width of at least some layers of the acoustic model by an integer multiple to construct an initial model; wherein the initial model includes a first branch that is the same as the acoustic model and a second branch that is different from the acoustic model; The construction of the enhanced model includes: selecting the first branch and at least a portion of the second branch from the initial model to construct the enhanced model; wherein the second branch is updated separately.

2. The voice wake-up method according to claim 1, characterized in that, The step of selecting the first branch and at least a portion of the second branch from the initial model to construct the enhanced model includes: A diversity parameter matrix is ​​obtained based on the current number of training iterations; wherein each element in the diversity parameter matrix is ​​an integer greater than 1 and less than a first ratio, each element corresponds to each layer in the initial model, and the first ratio is the ratio of the width of the corresponding layer in the initial model to the width of the corresponding layer in the acoustic model; For each layer in the initial model, a width selection range corresponding to the current layer is obtained, wherein the width selection range includes the width of the layer corresponding to the acoustic model, the product of the width of the layer corresponding to the acoustic model and the element, and the width of the layer corresponding to the initial model; one of the width selection ranges is randomly selected as the width of the layer corresponding to the enhanced model, and the layer corresponding to the enhanced model includes the layer corresponding to the acoustic model.

3. The voice wake-up method according to claim 2, characterized in that, The step of expanding the width of at least some layers of the acoustic model by an integer multiple to construct an initial model includes: expanding the width of each layer of the acoustic model by an integer multiple to construct the initial model, and the first ratio of all layers in the initial model is the same; And / or, the value of each element in the diversity parameter matrix is ​​the same.

4. The voice wake-up method according to claim 1, characterized in that, The step of jointly training the augmented model and the acoustic model using training data, and synchronously updating and maintaining the same parameters for the first branch and the acoustic model, includes: The training data is input into the enhancement model to obtain a first prediction result, and the training data is input into the acoustic model to obtain a second prediction result; A first loss is obtained based on the first prediction result and the state label of the training data; a second loss is obtained based on the second prediction result and the state label of the training data; and a KL divergence loss is obtained based on the first prediction result and the second prediction result. The total loss is obtained based on the first loss, the second loss, and the KL divergence loss; The parameters in the augmented model, the acoustic model, and the initial model are updated based on the total loss; wherein the parameters of the first branch in the augmented model and the initial model are the same as the parameters of the acoustic model, and the parameters of the remaining branches in the augmented model are the same as the parameters of the corresponding branches in the initial model.

5. The voice wake-up method according to claim 4, characterized in that, Also includes: In response to the failure to meet the preset condition for stopping training, the augmented model is reconstructed based on the updated initial model, and the reconstructed augmented model and the updated acoustic model are jointly trained using the training data.

6. The voice wake-up method according to claim 1, characterized in that, Prior to the step of jointly training the augmentation model and the acoustic model using training data, the following steps are included: Multiple sample data are obtained; wherein, the sample data is positive example speech data or negative example speech data; the positive example speech data contains a wake word, the negative example speech data does not contain a wake word, and the positive example speech data and the negative example speech data are set with corresponding text annotations; Force alignment is performed on each of the sample data and the corresponding text annotation to obtain the status label corresponding to each frame of audio in the sample data; The training data is constructed using multiple sample data sets; wherein the concatenated data sets are formed by concatenating at least one positive example speech data set and at least one negative example speech data set, and the state labels of the concatenated data sets are formed sequentially by the state labels of the positive example speech data set and the negative example speech data set that formed the concatenated data sets.

7. The voice wake-up method according to claim 6, characterized in that, The step of constructing spliced ​​data using multiple sample data includes: Obtain the maximum local amplitude of each sample data in the current spliced ​​data, and the maximum global amplitude among all the maximum local amplitudes; The sample data is normalized based on the local maximum amplitude value of each sample data. Each sample data point after amplitude normalization is multiplied by the global maximum amplitude value, and then the data is spliced ​​together to obtain the spliced ​​data.

8. A method for training an acoustic model, characterized in that, include: Construct an enhancement model; wherein the enhancement model includes the same first branch as the acoustic model to be trained; The enhancement model and the acoustic model are trained together using training data, and the parameters of the first branch and the acoustic model are updated synchronously and remain the same. Before constructing the enhanced model, the method includes: increasing the width of at least some layers of the acoustic model by an integer multiple to construct an initial model; wherein the initial model includes a first branch that is the same as the acoustic model and a second branch that is different from the acoustic model; The construction of the enhanced model includes: selecting the first branch and at least a portion of the second branch from the initial model to construct the enhanced model; wherein the second branch is updated separately.

9. A voice wake-up device, characterized in that, include: The first acquisition module is used to receive the speech signal to be recognized and obtain the acoustic features of the speech signal to be recognized; The second acquisition module, connected to the first acquisition module, is used to input the acoustic features into the trained acoustic model to obtain the probability value of the wake word; An execution model, connected to a second acquisition module, is used to determine whether to wake up based on the probability value; A training module, connected to a second acquisition module, is used to construct an augmentation model, the augmentation model including the same first branch as the acoustic model; And for jointly training the augmented model and the acoustic model using training data, wherein the parameters of the first branch and the acoustic model are updated synchronously and remain the same; Before constructing the enhanced model, the method includes: increasing the width of at least some layers of the acoustic model by an integer multiple to construct an initial model; wherein the initial model includes a first branch that is the same as the acoustic model and a second branch that is different from the acoustic model; The construction of the enhanced model includes: selecting the first branch and at least a portion of the second branch from the initial model to construct the enhanced model; wherein the second branch is updated separately.

10. An electronic device, characterized in that, It includes a memory and a processor coupled to each other, the memory storing program instructions, and the processor executing the program instructions to implement the voice wake-up method of any one of claims 1 to 7, or the training method of claim 8.

11. A storage device, characterized in that, The system stores program instructions that can be executed by a processor, the program instructions being used to implement the voice wake-up method of any one of claims 1 to 7, or the training method of claim 8.

Images