A speech wake-up method and device based on a continuous learning model, and a storage medium

Abstract

The application relates to a voice wake-up method and device based on a continuous learning model and a storage medium. The method comprises the following steps: constructing a teacher model and a student model based on a lightweight deep residual convolution network, and assembling the teacher model and the student model into a continuous learning model; acquiring voice sample data; inputting the voice sample data into the continuous learning model as training samples for training; and using the trained continuous learning model to wake up target voice. The application can improve prediction accuracy and wake-up efficiency. In particular, in the model training process, the teacher model and the student model are fused in a training mode, and a distillation loss, a feature map loss and a cross-entropy loss are combined, so that the continuous learning model is optimized, the student model is responsible for learning new knowledge, the teacher model is responsible for guiding the student model to not excessively learn new knowledge, the distribution difference between old knowledge and new knowledge is reduced, and the learning of new knowledge and the retention of old knowledge are realized.

Description

Technical Field

[0001] This application relates to the fields of deep learning and voice wake-up technology, and more specifically, to a voice wake-up method, device and storage medium based on a continuous learning model. Background Technology

[0002] With the rise of the human-computer interaction concept, voice wake-up technology is being applied more and more widely. As the first step in voice interaction products, impacting the human-computer interaction experience is currently one of the hot research topics for enterprises. For example, after enterprises set wake words for their products, the products determine whether to enter the question-and-answer service based on the keywords spoken by the user.

[0003] With the rise of deep learning, more and more companies are deploying end-to-end deep learning voice wake-up models in their products. Currently, there are two main learning methods for these models. The first method involves adding wake-up words and non-wake-up words from specific environments to the original speech data for retraining. This method has the advantage of high overall model accuracy, but each retraining session is time-consuming, and all training data must be completely saved, resulting in high storage resource consumption. The second method involves fine-tuning the existing model using wake-up words and non-wake-up words from specific environments. This method has the advantage of short training time and does not require the original speech data. However, fine-tuning speech models is prone to the curse of forgetting, where the model overlearns new knowledge and forgets old knowledge. Therefore, while fine-tuned models achieve high accuracy on new data, their accuracy drops significantly when learning from the original data. Summary of the Invention

[0004] Based on the above-mentioned technical problems, the present invention aims to construct a teacher model and a student model based on a lightweight deep residual convolutional network, and assemble the teacher model and the student model into a continuous learning model to improve the continuous learning ability of the model. In this way, when using the trained continuous learning model for voice wake-up related applications, the prediction accuracy and wake-up efficiency can be improved.

[0005] The first aspect of this invention provides a voice wake-up method based on a continuous learning model, the method comprising:

[0006] A teacher model and a student model are constructed based on a lightweight deep residual convolutional network, and the teacher model and the student model are assembled into a continuous learning model;

[0007] Acquire speech sample data;

[0008] The speech sample data is used as training samples and input into the continuous learning model for training.

[0009] A pre-trained continuous learning model is used to wake up the target speech.

[0010] In some embodiments of the present invention, the teacher model and student model constructed based on a lightweight deep residual convolutional network are configured with one first fully connected layer, five residual convolutional layers, two second fully connected layers, one feature layer and one probability layer in sequence, with each layer stacked in the order described.

[0011] In some embodiments of the present invention, each residual convolutional layer consists of two feedforward networks and a gated causal extended convolution located between the two feedforward networks. The two feedforward networks and the gated causal extended convolution are connected by residual connections.

[0012] In some embodiments of the present invention, the speech sample data includes first speech data representing past knowledge and second speech data representing new knowledge. The speech sample data is input as training samples into the continuous learning model for training, comprising: inputting the first speech data into the teacher model for pre-training; simultaneously inputting the second speech data into both the student model and the pre-trained teacher model, and loading preset weights onto both the student model and the pre-trained teacher model so that the pre-trained teacher model and the student model acquire the same past knowledge; during training, freezing the weights of the pre-trained teacher model so that the parameters of the pre-trained teacher model are not updated, and performing gradient descent learning on the student model; using the pre-trained teacher model to predict the second speech data, and guiding the student model based on the prediction results, so that the student model learns new knowledge based on past knowledge.

[0013] In some embodiments of the present invention, the training method of the continuous learning model further includes:

[0014] During training, the weighted sum of the feature map loss function, knowledge distillation loss function, and cross-entropy loss function is used as the overall loss function.

[0015] The continuous learning model is optimized using the overall loss function.

[0016] Training stops when the overall loss function value calculated by the optimized continuous learning model converges.

[0017] In some embodiments of the present invention, before waking up the target speech using a trained continuous learning model, the method further includes:

[0018] Convert the target speech into original speech features;

[0019] The original speech features are input into the trained continuous learning model.

[0020] In some embodiments of the present invention, converting the target speech into original speech features includes:

[0021] The target speech is sequentially pre-emphasized, framed, windowed, and subjected to discrete Fourier transform.

[0022] The speech features are extracted from the results obtained after discrete Fourier transform based on the Mel frequency cepstral coefficients to obtain the original speech features.

[0023] In some embodiments of the present invention, before inputting the speech sample data as training samples into the continuous learning model for training, the method further includes: performing speech enhancement transformation operations on the speech sample data, wherein the speech enhancement transformation operations include at least tone transformation, volume transformation, fade-in / fade-out transformation, speech rate transformation, beginning and end silence cutting transformation, bandpass filtering transformation, and bandstop filtering transformation.

[0024] A second aspect of the present invention provides a voice wake-up device based on a continuous learning model, the device comprising:

[0025] A building module is used to construct teacher and student models based on a lightweight deep residual convolutional network, and to assemble the teacher and student models into a continuous learning model.

[0026] The acquisition module is used to acquire voice sample data;

[0027] The training module is used to input the speech sample data as training samples into the continuous learning model for training.

[0028] The wake-up module is used to wake up the target speech using a trained continuous learning model.

[0029] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0030] A teacher model and a student model are constructed based on a lightweight deep residual convolutional network, and the teacher model and the student model are assembled into a continuous learning model.

[0031] Acquire speech sample data;

[0032] The speech sample data is used as training samples and input into the continuous learning model for training.

[0033] A trained continuous learning model is used to wake up the target speech.

[0034] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0035] This application first constructs a teacher model and a student model based on a lightweight deep residual convolutional network, and then assembles the teacher model and the student model into a continuous learning model. Speech sample data is acquired and used as training samples to input into the continuous learning model for training, which improves the model's continuous learning ability. Using the trained continuous learning model to wake up the target speech improves prediction accuracy and wake-up efficiency. Specifically, during training, the introduction of a feature map loss function optimizes the model. Based on the knowledge distillation loss function and cross-entropy loss function, a feature map loss function is introduced. The weighted sum of the feature map loss function, knowledge distillation loss function, and cross-entropy loss function is used as the overall loss function to continuously optimize the model, enabling continuous learning. Furthermore, no original data needs to be saved during training, reducing data storage consumption and significantly improving training speed. Moreover, the student model is responsible for learning new knowledge, while the teacher model guides the student model to avoid over-learning new knowledge. By reducing the distribution difference between old and new knowledge, new knowledge is learned while old knowledge is retained, thereby improving the application efficiency of voice wake-up.

[0036] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0037] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0038] Figure 1 The illustration shows a schematic diagram of the steps of a voice wake-up method based on a continuous learning model in an exemplary embodiment of this application;

[0039] Figure 2 A schematic diagram illustrating the structure and working process of a continuous learning model in an exemplary embodiment of this application is shown.

[0040] Figure 3 A schematic diagram of the residual convolutional layer structure in an exemplary embodiment of this application is shown;

[0041] Figure 4 A schematic diagram of the model parameters used during model training in an exemplary embodiment of this application is shown;

[0042] Figure 5 This illustration shows a schematic diagram of the process of converting the target speech into original speech features in an exemplary embodiment of this application;

[0043] Figure 6A schematic diagram of the structure of a voice wake-up device based on a continuous learning model in an exemplary embodiment of this application is shown;

[0044] Figure 7 This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0045] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application. It will be apparent to those skilled in the art that this application can be implemented without one or more of these details. In other instances, some technical features well-known in the art have not been described to avoid confusion with this application.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof.

[0047] Exemplary embodiments according to this application will now be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments may be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. The drawings are not drawn to scale, and some details may be enlarged and omitted for clarity. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0048] The following is in conjunction with the instruction manual appendix. Figure 1 - Appendix Figure 7 Several embodiments are given to describe exemplary implementations according to this application. It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the implementations of this application are not limited in any way. Rather, the implementations of this application can be applied to any applicable scenario.

[0049] Currently, voice interaction products are a hot research topic, and more and more companies are deploying end-to-end deep learning voice wake-up models in their products. There are two main learning methods for these models. The first method involves adding wake-up words and non-wake-up words from specific environments to the original speech data for retraining. This method has the advantage of high overall model accuracy, but each retraining session is time-consuming, and all training data must be completely saved, resulting in high storage resource consumption. The second method involves fine-tuning the existing model using wake-up words and non-wake-up words from specific environments. This method has the advantage of short training time and does not require the use of the original speech data. However, fine-tuning voice models is prone to the curse of forgetting, where the model overlearns new knowledge and forgets old knowledge. Therefore, while fine-tuned models achieve high accuracy on new data, their accuracy drops significantly when learning from the original data.

[0050] Therefore, in order to enable the voice wake-up model to have continuous learning capabilities and thus be better applied in the field of voice wake-up, this application provides a voice wake-up method based on a continuous learning model in some exemplary embodiments, such as... Figure 1 As shown, the method includes: S1, constructing a teacher model and a student model based on a lightweight deep residual convolutional network, and assembling the teacher model and the student model into a continuous learning model; S2, acquiring speech sample data; S3, inputting the speech sample data as training samples into the continuous learning model for training; S4, using the trained continuous learning model to wake up the target speech. These steps are described in detail below.

[0051] The first step is to construct a teacher model and a student model based on a lightweight deep residual convolutional network, and then assemble the teacher model and the student model into a continuous learning model.

[0052] In one specific implementation, refer to Figure 2 The teacher and student models, built based on a lightweight deep residual convolutional network, are configured with one first fully connected layer, five residual convolutional layers, two second fully connected layers, one feature layer, and one probability layer, stacked in the aforementioned order. Figure 2 As shown, the teacher model and the student model have the same underlying structure. The teacher model can be viewed as a pre-trained model used for prediction, while the student model can be viewed as a newly trained model. Here, the first and second fully connected layers are both fully connected layers in existing technologies; the terms "first" and "second" are merely for formal distinction.

[0053] In another specific implementation, refer to Figure 3Each residual convolutional layer consists of two feedforward layers and a gated causal extended convolution layer located between them. The two feedforward layers and the gated causal extended convolution layer are connected via residual connections. It should be noted that a feedforward neural network (FNN) is a type of artificial neural network. A feedforward neural network employs a unidirectional multi-layer structure. Each layer contains several neurons. In this type of neural network, each neuron receives signals from the neurons in the previous layer and generates an output for the next layer. Layer 0 is called the input layer, the last layer is called the output layer, and the other intermediate layers are called hidden layers. Hidden layers can be single or multiple. In the five residual convolutional layers, the input of the first residual convolutional layer is the output of the fully connected layer, and the input of the subsequent four residual convolutional layers is the output of the previous residual convolutional layer. It's worth noting that the residual convolutional layer has two outputs: the overall output of the residual convolutional layer and the intermediate output of the feedforward network. The feedforward network output of each residual convolutional layer is retained. After five residual convolutional layers have been processed, a skip connection is established, and the output of the skip connection is used as the input to the next layer. The feature layer is a gated linear layer, with a structure similar to a gated causal extended convolution. The probability layer is a softmax layer used to obtain all possible probabilities at the current time point. In this implementation, there are two categories: keywords and non-keywords.

[0054] The second step is to obtain voice sample data.

[0055] The speech sample data here includes keyword speech sample data and non-keyword speech sample data. For keyword speech sample data, this application recorded 2000 clean keyword speech data samples and 2000 noisy keyword speech data samples from 50 participants, with an average of 60 speech samples recorded per participant. The speech recording equipment was a professional microphone with a sampling frequency of 16kHz and 1 sampling channel. During the recording process, the participants changed their tone, speaking speed, and volume to ensure speech diversity as much as possible. The duration of the speech data samples ranged from 1 to 7 seconds, with an average duration of approximately 3 seconds for all keyword speech samples and a total duration of approximately 2.5 hours.

[0056] For non-keyword speech data, this application collected several publicly available speech datasets, publicly available noise datasets, and background noise recorded in noisy environments. The publicly available speech datasets include: aidatatang_200zh, aishell3, aishell4, primewords_md_2018_set1, free_st_mandrain, magic_mandrain, and thchs30. Among them, the aishell4 dataset consists of 8-channel long speech segments recorded in a classroom environment, and therefore preprocessed it. Specifically, the 8-channel speech data was first split into multiple single-channel speech data, and then the long speech data was randomly sliced ​​according to intervals of 1-7 seconds, finally yielding non-keyword speech data suitable for training. The publicly available noise dataset and the background noise recorded in noisy environments underwent the same preprocessing process. The total duration of all non-keyword speech data is approximately 750 hours.

[0057] In a preferred implementation, the speech sample data also needs to undergo speech enhancement transformation to strengthen and expand the speech samples. Here, the speech enhancement transformation operation includes at least tone transformation, volume transformation, fade-in / fade-out transformation, speech rate transformation, beginning and end silence cutting transformation, bandpass filtering transformation, and bandstop filtering transformation. The total duration of the enhanced keyword speech is approximately 17.5 hours.

[0058] It should be noted that learning new knowledge using the student model is continuous learning, so the sample data includes new speech data, which differs from the pre-training data used by the teacher model. This new speech data only contains a small amount of keyword and non-keyword speech. This application collected 100 instances of keyword and non-keyword speech that the model predicted incorrectly in a new environment. By learning from these incorrectly predicted speech data, the model can adapt to new environments. In other words, the speech sample data includes first speech data representing past knowledge and second speech data representing new knowledge. The 100 collected new speech data are the second type of speech data. This speech sample data is then used as training samples to input into the continuous learning model for training. This training process includes two stages: the first stage inputs the first speech data into the teacher model for pre-training; the second stage inputs the second speech data into both the student model and the pre-trained teacher model simultaneously, using the prediction results output by the pre-trained teacher model to guide the student model.

[0059] The third step is to input the speech sample data as training samples into the continuous learning model for training.

[0060] For example Figure 2 As shown, during training Figure 2The teacher model on the left is used for prediction, while the student model on the right is used for continuous learning. The student model's incorrect predictions differ significantly from the teacher model's pre-trained speech data. Therefore, when fine-tuning new speech data, it's easy to forget old knowledge due to fitting new knowledge, requiring strict control during training. Specifically, the speech sample data includes first speech data representing past knowledge and second speech data representing new knowledge. The first speech data is input into the teacher model for pre-training; the second speech data is simultaneously input into both the student model and the pre-trained teacher model, with pre-defined weights applied to both to ensure they acquire the same past knowledge. During training, the weights of the pre-trained teacher model are frozen to prevent parameter updates, while gradient descent learning is applied to the student model. The pre-trained teacher model predicts the second speech data, and the prediction results guide the student model, enabling it to learn new knowledge based on past knowledge. The model parameters used during training are as follows: Figure 4 As shown, the continuous learning model consists of two stages. The first stage is the pre-training stage, which uses the initial speech data representing past knowledge. The second stage is the continuous learning stage, which guides the continuous learning of the student model based on the prediction results output by the pre-trained teacher model. In other words, the experimenter can continuously learn from new training data without using the original pre-training data. The training speed is fast and the knowledge forgetting rate is low. The knowledge forgetting rate is defined as:

[0061]

[0062] Among them, acc p acc represents the accuracy of the pre-trained model on the pre-training dataset. c This represents the accuracy of the pre-trained model on the pre-training dataset after continuous learning.

[0063] During training, the weights of the teacher model are frozen to prevent parameter updates, while gradient descent learning is applied to the student model. This allows the teacher model to predict new inputs based on past knowledge, and the student model to learn new knowledge based on past knowledge. Upon receiving input, the teacher model obtains predicted probabilities through a fully connected layer -> residual convolutional layer -> fully connected layer -> feature layer -> probability layer. The student model performs two calculations: feedforward and feedback. The feedforward calculation for the student model is consistent with that of the teacher model. Feedback calculations are performed based on the errors between the predicted probabilities output by the student model and those output by the teacher model, as well as the errors between the feature maps of the teacher and student models. This updates the parameters of the student model, enabling it to learn new knowledge. In this structure, the student model is responsible for learning new knowledge, while the teacher model guides it to avoid overlearning new knowledge. By reducing the distributional difference between old and new knowledge, the learning of new knowledge and the retention of old knowledge are achieved.

[0064] During training, the weighted sum of the feature map loss function, knowledge distillation loss function, and cross-entropy loss function is used as the overall loss function to optimize the continuous learning model. Training stops when the overall loss function value calculated by the optimized continuous learning model converges. The first aspect is calculating the feature map loss, the purpose of which is to make the distance between the feature map output by the student model and the feature map output by the teacher model as close as possible, thereby limiting the student model's influence on the representation of old knowledge when learning new knowledge. Its loss function is as follows:

[0065]

[0066] Where M is the length of the time dimension and N is the length of the feature dimension. The feature values ​​representing the output of the teacher model, These represent the feature values ​​output by the student model.

[0067] The second perspective is distillation loss. The purpose of distillation loss is to make the probability distribution output by the student model as similar as possible to the probability distribution output by the teacher model, thereby avoiding the forgetting of old knowledge. Its loss function is as follows:

[0068]

[0069] in, This represents the probability value output by the teacher model. This represents the probability value output by the student model.

[0070] In addition, this application also uses cross-entropy loss to learn new knowledge, and its calculation formula is as follows:

[0071]

[0072] Among them, y m,n The probability of being the correct label.

[0073] The overall loss used for model training is the weighted sum of the losses of the three factors mentioned above, and its calculation formula is as follows:

[0074] Loss=w1*fmloss+w2*dtloss+w3*celoss

[0075] Among them, w1, w2, and w3 are weights that can be adjusted according to the specific task. In the experimental task conducted in this application, the values ​​of the three are 0.1, 0.2, and 0.7, respectively.

[0076] It should be noted that the introduction of a feature map loss function further optimizes the model. Based on the knowledge distillation loss function and cross-entropy loss function, a feature map loss function is introduced. The weighted sum of the feature map loss function, knowledge distillation loss function, and cross-entropy loss function is used as the overall loss function to continuously optimize the model, enabling continuous learning. Furthermore, no original data needs to be saved during training, reducing data storage consumption and significantly improving training speed. After each loss function value calculation, the model can be improved based on the calculated loss function value until the calculated loss function value converges after a certain training iteration, i.e., after reaching a preset number of training iterations. The application of the above loss function, which integrates feature map loss on top of knowledge distillation loss, optimizes the continuous learning model. Experiments show that its technical performance is significantly improved. The achieved technical effect is that the student model is responsible for learning new knowledge, while the teacher model guides the student model to avoid over-learning new knowledge. By reducing the distribution difference between old and new knowledge, it achieves the learning of new knowledge and the retention of old knowledge, thereby improving the application efficiency of voice wake-up.

[0077] The fourth step is to use a trained continuous learning model to wake up the target speech.

[0078] Continuous learning models need to be properly trained before they can be used in real-world applications. The specific process of using a trained model to wake up target speech is similar to that of using speech sample data. Using a trained continuous learning model to wake up target speech includes: extracting features from the target speech input to convert it into raw speech features; inputting the raw speech features into the continuous learning model to obtain target speech features, which are then processed through fully connected layers, residual convolutional layers, fully connected layers, feature layers, and finally a probability layer to obtain the keyword probabilities corresponding to the target speech.

[0079] In practice, feature extraction of the target speech is required first, such as MFCC speech feature extraction. MFCC (Mel Frequency Cepstrum Coefficient) refers to the Mel-frequency cepstrum coefficients. Figure 5 As shown, the target speech is sequentially pre-emphasized, framed, windowed, and subjected to Discrete Fourier Transform. The speech features are then extracted from the results obtained after the Discrete Fourier Transform based on the Mel frequency cepstral coefficients to obtain the original speech features.

[0080] For example, let the input speech signal signal = [s1, s2, ..., s T Where T = 16kHz is the signal sampling frequency. Pre-emphasis can remove the influence of lip radiation and increase the high-frequency resolution of speech. Its implementation is as follows:

[0081] signal i =signal i+1 -α*signal i

[0082] Where α is the pre-emphasis coefficient, typically taken as 0.97. To ensure the balance of the input signal, the speech is divided into multiple segments using a framing method, as follows:

[0083] numframes = 1 + ceil((T-frame) length ) / frame step )

[0084] Among them, frame length The frame length is [frame length]. step The translation distance between adjacent frames is used to pad signals shorter than one frame with zeros. The resulting frame segmentation is numframes frames. Next, a windowing method is applied to avoid spectral leakage caused by non-periodic truncation of the signal. The formula for windowing is:

[0085] x i =x i *ω i

[0086] Where, ω i This is the Hanning window. Since it's often difficult to discern the characteristics of a signal from its time-domain transformations, a discrete Fourier transform is used after windowing to obtain the spectrum of each frame. The formula is:

[0087]

[0088] Where N is the number of sampling points, F i,k Let x represent the frequency of the k-th point in the i-th frame. i,nThis represents the nth point in the i-th frame. The Discrete Fourier Transform uses 512 sampling points, a window length of 32ms, and a window shift of 20ms. The pitch of a sound perceived by the human ear is not linearly proportional to its frequency, but the Mel frequency scale better matches the characteristics of human hearing. Therefore, a Mel filter is used for processing. The conversion formula between Mel frequency and actual frequency is as follows:

[0089]

[0090] The above formula is used to convert the audio frequency range into a Mel frequency range, which is then evenly divided into L parts, i.e., L Mel filters. Each Mel filter is triangular in shape, and the resulting Mel frequency is the center frequency of the Mel filter. Let o... l ,c l ,h l Let the lower cutoff frequency, center frequency, and upper cutoff frequency of the Mel filter be respectively. The following relationship can be obtained:

[0091] c l =h l-1 =o l+1

[0092] It is evident that the center frequency of a Mel filter is equal to the upper limit frequency of the previous Mel filter and also equal to the lower limit frequency of the next Mel filter. The formula for a triangular Mel filter can be expressed as:

[0093]

[0094] The final output of the Mel filter can be expressed as:

[0095]

[0096] Perform a discrete cosine transform on the output of the Mel filter. Figure 5 The final Mel-frequency cepstral coefficients are obtained from the DCT in the DCT.

[0097]

[0098] Where I represents the number of Mel cepstral coefficients. The frequency range for audio extraction using Mel cepstral coefficients is 20-8000 Hz. The Mel filter bank has 26 elements, and the number of Mel cepstral coefficients is 26. In this implementation, the first 13 Mel cepstral coefficients are used as input features. Cepstral coefficients conform to the principles of human hearing and can effectively extract acoustic features of speech.

[0099] Continuous learning models need to be properly trained before they can be used in real-world applications. The process of using a trained model to wake up target speech is similar to that of processing speech sample data. When using a trained continuous learning model to wake up target speech, the target speech is first processed by feature extraction, for example, by converting it into raw speech features using MFCC (Multi-Functional Convolutional Layer). Then, the raw speech features are input into the continuous learning model to obtain target speech features, which are processed through fully connected layers, residual convolutional layers, fully connected layers, feature layers, and finally a probability layer to obtain the probability of the corresponding keywords in the target speech. Therefore, the application of speech wake-up is essentially keyword detection, that is, detecting target keywords in continuous speech. If the model detects the target keywords, it can be considered that the target speech has been successfully woken up. It can be seen that the continuous learning model of this application is more suitable for various applications of speech wake-up, and its continuous learning capability ensures the accuracy of speech wake-up.

[0100] To verify the accuracy of the continuous learning model in this application, the pre-trained dataset was divided into training, validation, and test sets in an 8:1:1 ratio. The pre-trained model achieved an F1 score of 98.21% on the test set. Without using distillation loss and feature map loss, the model fine-tuned directly on 200 new speech samples showed an F1 score of 95.14% on the test set. However, the model fine-tuned using distillation loss and feature map loss showed an F1 score of 97.73% on the test set, with the knowledge forgetting rate decreasing from 3.13% to 0.49%. The formula for the F1 score is:

[0101]

[0102] Experiments show that the method described in this application can effectively avoid forgetting old knowledge while learning new knowledge, and can directly fine-tune new speech data without pre-training data, thus reducing training time and data storage consumption. Therefore, the voice wake-up method based on a continuous learning model proposed in this application is both efficient and accurate, thereby significantly improving prediction efficiency and voice wake-up efficiency.

[0103] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention.

[0104] In some exemplary embodiments of this application, a voice wake-up device based on a continuous learning model is also provided to execute the voice wake-up method based on a continuous learning model described in various embodiments of this application, such as... Figure 6 As shown, the device includes:

[0105] A building module is used to construct teacher and student models based on a lightweight deep residual convolutional network, and to assemble the teacher and student models into a continuous learning model.

[0106] The acquisition module is used to acquire voice sample data;

[0107] The training module is used to input the speech sample data as training samples into the continuous learning model for training.

[0108] The wake-up module is used to wake up the target speech using a trained continuous learning model.

[0109] The device improves prediction accuracy and wake-up efficiency. During model training, the fusion of teacher and student models, combined with distillation loss, feature map loss, and cross-entropy loss, enables continuous model learning. Furthermore, training does not require saving original data, reducing data storage consumption and significantly increasing training speed.

[0110] It is understood that the voice wake-up device based on the continuous learning model has supporting hardware and equipment, which are not limited herein. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application.

[0111] It should also be emphasized that the system provided in this application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0112] Please refer to the following. Figure 7 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 7As shown, the computer device 2 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program that can run on the processor 200. When the processor 200 runs the computer program, it executes the voice wake-up method based on a continuous learning model provided in any of the foregoing embodiments of this application.

[0113] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 203 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0114] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. After receiving an execution instruction, the processor 200 executes the program. The voice wake-up method based on a continuous learning model disclosed in any of the foregoing embodiments of this application can be applied to the processor 200, or implemented by the processor 200.

[0115] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the above method.

[0116] This application also provides a computer-readable storage medium corresponding to the voice wake-up method based on a continuous learning model provided in the foregoing embodiments, wherein a computer program (i.e., a program product) is stored thereon, and the computer program, when run by a processor, executes the voice wake-up method based on a continuous learning model provided in any of the foregoing embodiments.

[0117] In addition, examples of the computer-readable storage medium may include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be described in detail here.

[0118] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the voice wake-up method based on a continuous learning model provided in any of the foregoing embodiments, including: constructing a teacher model and a student model based on a lightweight deep residual convolutional network, and assembling the teacher model and the student model into a continuous learning model; acquiring voice sample data; inputting the voice sample data as training samples into the continuous learning model for training; and using the trained continuous learning model to wake up the target voice.

[0119] It should be noted that the algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application. Numerous specific details are set forth in the specification provided herein. However, it is to be understood that embodiments of this application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0120] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0121] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed herein and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed herein may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0122] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation apparatus according to embodiments of this application. This application can also be implemented as a device or apparatus program for performing part or all of the methods described herein. The program implementing this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0123] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A voice wake-up method based on a continuous learning model, characterized in that, The method includes: A teacher model and a student model are constructed based on a lightweight deep residual convolutional network, and the teacher model and the student model are assembled into a continuous learning model. Acquire speech sample data; The speech sample data is used as training samples and input into the continuous learning model for training. The target speech is awakened using a pre-trained continuous learning model; The speech sample data includes first speech data representing past knowledge and second speech data representing new knowledge. The speech sample data is used as training samples and input into the continuous learning model for training, including: The first voice data is input into the teacher model for pre-training; The second voice data is simultaneously input into the student model and the pre-trained teacher model, and preset weights are loaded onto both the student model and the pre-trained teacher model so that the pre-trained teacher model and the student model obtain the same past knowledge. During training, the weights of the pre-trained teacher model are frozen so that the parameters of the pre-trained teacher model are not updated, and gradient descent learning is performed on the student model. The teacher model, having completed pre-training, makes predictions on the second speech data, and the student model is guided based on the prediction results, so that the student model can learn new knowledge based on past knowledge. Specifically, the pre-trained teacher model predicts the second speech data, and guides the student model based on the prediction results. This includes performing feedback calculations based on the errors between the predicted probabilities output by the student model and those output by the teacher model, the errors between the predicted probabilities output by the student model and the true labels, and the errors between the feature maps output by the teacher model and the student model. This process updates the parameters in the student model, enabling the learning of new knowledge and the retention of old knowledge.

2. The voice wake-up method based on a continuous learning model according to claim 1, characterized in that, The teacher and student models, built on a lightweight deep residual convolutional network, are configured with one first fully connected layer, five residual convolutional layers, two second fully connected layers, one feature layer, and one probability layer, stacked in the order described above.

3. The voice wake-up method based on a continuous learning model according to claim 2, characterized in that, Each residual convolutional layer consists of two feedforward layers and a gated causal extended convolution layer located between the two feedforward layers. The two feedforward layers and the gated causal extended convolution layer are connected by residual connections.

4. The voice wake-up method based on a continuous learning model according to claim 1, characterized in that, The training method for the continuous learning model also includes: During training, the weighted sum of the feature map loss function, knowledge distillation loss function, and cross-entropy loss function is used as the overall loss function. The continuous learning model is optimized using the overall loss function. Training stops when the overall loss function value calculated by the optimized continuous learning model converges.

5. The voice wake-up method based on a continuous learning model according to claim 1, characterized in that, Before using the trained continuous learning model to wake up the target speech, the method further includes: Convert the target speech into original speech features; The original speech features are input into the trained continuous learning model.

6. The voice wake-up method based on a continuous learning model according to claim 5, characterized in that, Converting the target speech into original speech features includes: The target speech is sequentially pre-emphasized, framed, windowed, and subjected to discrete Fourier transform. The speech features are extracted from the results obtained after discrete Fourier transform based on the Mel frequency cepstral coefficients to obtain the original speech features.

7. The voice wake-up method based on a continuous learning model according to claim 1, characterized in that, Before inputting the speech sample data as training samples into the continuous learning model for training, the method further includes: performing speech enhancement transformation operations on the speech sample data, wherein the speech enhancement transformation operations include at least tone transformation, volume transformation, fade-in / fade-out transformation, speech rate transformation, first and last silence cutting transformation, bandpass filtering transformation, and bandstop filtering transformation.

8. A voice wake-up device based on a continuous learning model, characterized in that, The device includes: A building module is used to construct teacher and student models based on a lightweight deep residual convolutional network, and to assemble the teacher and student models into a continuous learning model. The acquisition module is used to acquire voice sample data; The training module is used to input the speech sample data as training samples into the continuous learning model for training. The wake-up module is used to wake up the target speech using a trained continuous learning model; The speech sample data includes first speech data representing past knowledge and second speech data representing new knowledge. The speech sample data is used as training samples and input into the continuous learning model for training, including: The first voice data is input into the teacher model for pre-training; The second voice data is simultaneously input into the student model and the pre-trained teacher model, and preset weights are loaded onto both the student model and the pre-trained teacher model so that the pre-trained teacher model and the student model obtain the same past knowledge. During training, the weights of the pre-trained teacher model are frozen so that the parameters of the pre-trained teacher model are not updated, and gradient descent learning is performed on the student model. The teacher model, having completed pre-training, makes predictions on the second speech data, and the student model is guided based on the prediction results, so that the student model can learn new knowledge based on past knowledge. Specifically, the pre-trained teacher model predicts the second speech data, and guides the student model based on the prediction results. This includes performing feedback calculations based on the errors between the predicted probabilities output by the student model and those output by the teacher model, the errors between the predicted probabilities output by the student model and the true labels, and the errors between the feature maps output by the teacher model and the student model. This process updates the parameters in the student model, enabling the learning of new knowledge and the retention of old knowledge.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1-7.

Images