Voice keyword detection and model training method and system, storage medium and electronic equipment

Abstract

The invention provides a voice keyword detection and model training method and system, a storage medium and electronic equipment, and the model training method comprises the steps: constructing a feature denoising neural network, and training the feature denoising neural network; constructing a time binary weight neural network, and training the time binary weight neural network; and carrying out joint training on the feature denoising neural network and the time binary weight neural network so as to carry out voice keyword detection based on the trained feature denoising neural network and the time binary weight neural network. According to the voice keyword detection and model training method and system, the storage medium and the electronic equipment, voice keyword detection is realized based on the feature denoising-time binary weight neural network, and the accuracy of voice keyword detection is effectively improved.

Description

Technical Field

[0001] The present application belongs to the technical field of speech recognition, and in particular relates to a speech keyword detection and model training method, system, storage medium and electronic device. Background Art

[0002] Voice keyword detection refers to the identification of specific keywords or phrases from voice signals. Keyword detection systems can identify specific words or phrases spoken by users from voice streams in real time. Typically, only certain specific words need to be recognized, rather than the entire voice content. In other words, the system determines whether the input voice data is a keyword. If it is a keyword, the keyword name is displayed; otherwise, non-keywords are displayed or not displayed. Since voice keyword detection systems typically run on terminal devices with limited memory and computing resources, such as edge applications such as mobile phone voice assistants, smart homes, and in-vehicle voice systems, the voice command word recognition system should simultaneously meet the requirements of high accuracy, small memory usage, and low computational complexity.

[0003] Keyword detection, as a multi-classification task, is highly susceptible to background noise, resulting in reduced recognition accuracy and increased false positives. Keyword detection systems often operate in non-quiet environments, such as on noisy streets, in shopping malls, or in vehicles with roaring engines. Consequently, keyword detection accuracy is often reduced. Therefore, improving the noise robustness of keyword detection systems while maintaining low memory usage and computational complexity is a major challenge in current speech keyword detection technology.

[0004] In recent years, deep learning theory and technology have gradually matured, and neural network-based keyword detection has become the mainstream solution due to its excellent performance. Compared with traditional solutions such as template matching and Gaussian mixture hidden Markov models, it uses fewer parameters and requires less computation, while achieving higher recognition accuracy. Furthermore, neural network-based noise reduction methods, due to their more significant noise reduction effects and ability to cope with more complex noise environments, have gradually replaced traditional methods such as spectral subtraction and filtering to become the mainstream solution.

[0005] In a speech keyword detection method of the prior art, multiple keywords to be detected are preprocessed to obtain embedded keywords; the acoustic embedding features of the speech are obtained; the embedded keywords and the acoustic embedding features are passed through a preset number of sequentially connected transformer coding blocks to obtain the keyword start embedding vector, the keyword end embedding vector and the acoustic output vector; the similarity score between the keyword and the acoustic embedding feature is calculated based on the keyword start embedding vector, the keyword end embedding vector and the acoustic output vector; the keyword deviation of each keyword is obtained based on the similarity score; the acoustic output vector and the keyword deviation are element-wise multiplied and then decoded to obtain the keyword detection result. However, this method has the following shortcomings: 1) The encoding model used in this method is a transformer model, which has a relatively complex structure and is prone to excessive memory usage and computational complexity, making it difficult to deploy on the edge with limited resources; 2) This method includes two stages, encoding and decoding. Compared with the current mainstream end-to-end keyword detection method, it often has a longer wake-up delay, which affects the user experience. In addition, the decoding stage needs to calculate the similarity with the pre-stored acoustic feature vectors, and therefore is highly dependent on the quality of the pre-stored acoustic feature vectors; 3) Failure to properly optimize the solution or training data set for the noisy environment can easily lead to poor noise resistance performance.

[0006] In addition, a voice keyword detection system for edge intelligent applications in the prior art includes a voice keyword detection processor, an audio acquisition module, and a KWS voice recognition accelerator; the voice keyword detection processor is used to detect whether the command recognition button is pressed; the audio acquisition module is used to drive the PDM to collect voice data input by the microphone when the command recognition button is pressed; the voice keyword detection processor is used to send the voice data to the KWS voice recognition accelerator; the KWS voice recognition accelerator is used to extract MFCC features from the voice data and calculate the template matching distance between the MFCC features and the template; the voice keyword detection processor is used to use the minimum template matching distance between the MFCC features and the template as the voice detection result. However, the system has the following shortcomings: 1) This solution is essentially a keyword detection method based on template matching, which can only recognize isolated words and cannot effectively recognize continuous voice streams; 2) This solution also has the disadvantages of long wake-up delay and poor noise robustness; 3) When used, a large amount of voice template data needs to be stored, resulting in excessive memory space consumption.

[0007] Application Contents

[0008] In view of the shortcomings of the existing technology mentioned above, the purpose of this application is to provide a speech keyword detection and model training method, system, storage medium and electronic device, which realizes speech keyword detection based on feature denoising-time binary weighted neural network, effectively improving the accuracy of speech keyword detection.

[0009] In a first aspect, the present application provides a method for training a speech keyword detection model, the method comprising the following steps: extracting amplitude spectra of speech data from a noise-free speech dataset and a noise-added speech dataset; constructing a feature denoising neural network, and training the feature denoising neural network based on the amplitude spectra; wherein the feature denoising neural network is used to denoise the amplitude spectra of the speech data to obtain a denoised amplitude spectra; extracting log-mel spectrum features of the speech data from a keyword speech dataset and a non-keyword speech dataset; constructing a time binary weighted neural network, and training the time binary weighted neural network based on the log-mel spectrum features; wherein the time binary weighted neural network is used to obtain probability score values of keywords and non-keywords in the speech data based on the log-mel spectrum features; based on the keyword speech dataset and the non-keyword speech dataset, jointly training the feature denoising neural network and the time binary weighted neural network to perform speech keyword detection based on the trained feature denoising neural network and the time binary weighted neural network; wherein the amplitude spectrum of the speech dataset is input into the feature denoising network to obtain a denoised amplitude spectrum, log-mel spectrum features are extracted based on the denoised amplitude spectrum, and the log-mel spectrum features are input into the time binary weighted neural network.

[0010] In an implementation of the first aspect, extracting the amplitude spectrum of the speech data includes the following steps:

[0011] Performing frame division and windowing on the voice data;

[0012] Performing short-time Fourier transform on the framed and windowed speech data to obtain a time-frequency spectrum of the speech data;

[0013] The absolute value of the time-frequency spectrum is calculated to obtain the amplitude spectrum.

[0014] In an implementation of the first aspect, the feature denoising neural network includes N first fully connected layers, N ReLU activation functions, N second fully connected layers, and M Sigmoid activation functions connected in sequence; where M and N are natural numbers;

[0015] The N first fully connected layers are used to perform feature compression on the amplitude spectrum to obtain a compressed amplitude spectrum;

[0016] The N ReLU activation functions are used to perform nonlinear processing on the compressed amplitude spectrum to obtain a nonlinear amplitude spectrum;

[0017] The N second fully connected layers are used to perform feature expansion on the nonlinear amplitude spectrum to obtain an extended amplitude spectrum, where the extended amplitude spectrum has the same size as the amplitude spectrum;

[0018] The M Sigmoid activation functions are used to perform nonlinear processing on the expanded amplitude spectrum, obtain a mask for removing noise components in the amplitude spectrum, and multiply the mask by the amplitude spectrum of the speech data to obtain a denoised amplitude spectrum.

[0019] In an implementation of the first aspect, extracting the logarithmic mel-spectrogram features of the speech data includes the following steps:

[0020] Mel filtering and logarithmic processing are performed on the denoised amplitude spectrum to obtain the logarithmic Mel spectrum feature.

[0021] In an implementation of the first aspect, the temporal binary weighted neural network includes a binary weighted convolution layer, a batch normalization layer, a ReLU activation function, multiple temporal binary weighted convolution blocks, a global average pooling layer, and a third fully connected layer, which are sequentially connected;

[0022] The temporal binary weighted convolution block includes a binary weighted convolution layer, a batch normalization layer and a ReLU activation function connected in sequence;

[0023] The output channels of the multiple binary weight convolution layers in the temporal binary weight neural network are increased in sequence;

[0024] The weight parameters in the binary weight convolution layer are binarized to 1 and -1.

[0025] In an implementation of the first aspect, when the feature denoising neural network and the temporal binary weighted neural network are jointly trained, the parameters of the feature denoising neural network remain unchanged, and only the parameters of the temporal binary weighted neural network are updated.

[0026] In a second aspect, the present application provides a speech keyword detection model training system, the system comprising a first extraction module, a first training module, a second extraction module, a second training module, and a third training module;

[0027] The first extraction module is used to extract the amplitude spectrum of the speech data in the noise-free speech data set and the noisy speech data set;

[0028] The first training module is used to construct a feature denoising neural network and train the feature denoising neural network based on the amplitude spectrum; wherein the feature denoising neural network is used to denoise the amplitude spectrum of the speech data to obtain a denoised amplitude spectrum;

[0029] The second extraction module is used to extract the logarithmic Mel spectrum features of the speech data in the keyword speech data set and the non-keyword speech data set;

[0030] The second training module is used to construct a time binary weighted neural network and train the time binary weighted neural network based on the logarithmic Mel spectrum feature; wherein the time binary weighted neural network is used to obtain probability score values of keywords and non-keywords in the speech data based on the logarithmic Mel spectrum feature;

[0031] The third training module is used to jointly train the feature denoising neural network and the time binary weighted neural network based on the keyword speech data set and the non-keyword speech data set, so as to perform speech keyword detection based on the trained feature denoising neural network and time binary weighted neural network; wherein the amplitude spectrum of the speech data set is input into the feature denoising network to obtain the denoised amplitude spectrum, the logarithmic Mel spectrum feature is extracted based on the denoised amplitude spectrum, and the logarithmic Mel spectrum feature is input into the time binary weighted neural network.

[0032] In a third aspect, the present application provides a method for detecting speech keywords, the method comprising the following steps:

[0033] Extracting the amplitude spectrum of the speech data to be detected;

[0034] Inputting the amplitude spectrum into the feature denoising neural network trained by the above-mentioned speech keyword detection model training method to obtain a denoised amplitude spectrum;

[0035] Acquire a logarithmic Mel spectrum feature based on the denoised amplitude spectrum;

[0036] Inputting the logarithmic Mel spectrum features into the time binary weighted neural network trained by the above-mentioned speech keyword detection model training method to obtain the probability score values of keywords and non-keywords in the speech data;

[0037] It is determined whether the voice data contains the keyword based on the probability score value.

[0038] In one implementation of the third aspect, determining the keywords contained in the speech data based on the probability score value includes the following steps:

[0039] Smoothing the probability score value to obtain a sliding average;

[0040] When the maximum value of the sliding average values is greater than or equal to a preset threshold and the maximum value corresponds to a keyword, the keyword contained in the voice data is determined.

[0041] In a fourth aspect, the present application provides a voice keyword detection system, the system comprising an extraction module, a first acquisition module, a second acquisition module, a third acquisition module and a detection module;

[0042] The extraction module is used to extract the amplitude spectrum of the speech data to be detected;

[0043] The first acquisition module is used to input the amplitude spectrum into the feature denoising neural network trained by the above-mentioned speech keyword detection model training method to obtain the denoised amplitude spectrum;

[0044] The second acquisition module is used to obtain logarithmic Mel spectrum features based on the denoised amplitude spectrum;

[0045] The third acquisition module is used to input the logarithmic Mel spectrum features into the time binary weighted neural network trained by the above-mentioned voice keyword detection model training method to obtain the probability score values of keywords and non-keywords in the voice data;

[0046] The detection module is used to determine the keywords contained in the speech data based on the probability score value.

[0047] In a fifth aspect, the present application provides an electronic device, comprising: a processor and a memory;

[0048] The memory is used to store computer programs;

[0049] The processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned voice keyword detection model training method or the above-mentioned voice keyword detection method.

[0050] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned speech keyword detection model training method or the above-mentioned speech keyword detection method.

[0051] As described above, the speech keyword detection and model training method, system, storage medium, and electronic device described in this application have the following beneficial effects:

[0052] (1) Speech keyword detection is realized based on feature denoising-time binary weighted neural network. In the feature extraction stage, the amplitude spectrum features are denoised. Compared with other keyword detection systems based on neural networks, it can cope with more complex noise environments, that is, it can still obtain high accuracy and low recognition rate in low signal-to-noise ratio environments. In addition, since the speech input length of keyword detection is short (generally about 1 second), and there is no need to convert the denoised amplitude spectrum into a speech signal through inverse short-time Fourier transform, the feature denoising neural network used can still obtain obvious feature denoising effect despite its simple structure, extremely small number of parameters and computational complexity. Due to the obvious noise reduction effect of the feature denoising neural network, the input of the time binary weighted neural network is relatively pure, the classification difficulty is relatively low, and it can achieve satisfactory classification accuracy under the conditions of using extremely low number of parameters and computational complexity.

[0053] (2) The temporal binary weighted neural network uses binary weighted convolution instead of traditional convolution, binarizing the convolution weight parameter values into single-bit binary data, significantly reducing memory usage. In addition, since the weights are all 1 and -1, multiplication calculations are not required in the convolution operation, greatly reducing the computational complexity of the model;

[0054] (3) First, the feature denoising neural network and the time binary weight neural network are trained separately. Then, the trained feature denoising neural network and the time binary weight neural network are combined into a feature denoising-time binary weight neural network for joint training. This allows the model parameters of the time binary weight neural network to be reasonably initialized and adaptively adjusted, thereby significantly reducing the difficulty of model training and effectively improving the classification accuracy of the feature denoising-time binary weight neural network.

[0055] (4) While ensuring extremely low memory usage and computational complexity, high accuracy and low misrecognition rate can be achieved in both quiet and noisy environments.

[0056] (5) Compared with traditional methods such as template matching method and Gaussian mixture model-hidden Markov model (GMM-HMM) method, it has higher recognition accuracy and lower recognition delay, and has stronger noise resistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Shown is a flow chart of a method for training a speech keyword detection model in one embodiment of the present application;

[0058] Figure 2 Shown is a schematic diagram of the structure of a feature denoising neural network in one embodiment of the present application;

[0059] Figure 3 Shown is a schematic diagram of the structure of a temporal binary weighted neural network in one embodiment of the present application;

[0060] Figure 4 Shown is a structural diagram of a speech keyword detection model training system in one embodiment of the present application;

[0061] Figure 5 Shown is a flow chart of a method for detecting speech keywords in one embodiment of the present application;

[0062] Figure 6 Shown is a schematic diagram of the structure of a speech keyword detection system in one embodiment of the present application;

[0063] Figure 7 Shown is a schematic structural diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0064] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0065] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0066] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings in the embodiments of the present application.

[0067] like Figure 1 As shown, in one embodiment, the voice keyword detection method of the present application includes steps S11 to S15.

[0068] Step S11: extracting the amplitude spectra of the speech data in the noise-free speech dataset and the noise-added speech dataset.

[0069] Specifically, in this application, a noise-free speech dataset and a noise-added speech dataset are used as training datasets to train a feature denoising neural network, wherein the noise can be traffic noise, wind noise, music, venue noise, etc.

[0070] When extracting the amplitude spectrum of the speech data in the noise-free speech dataset and the noisy speech dataset, the speech data is first framed and windowed. For example, the frame length used for framing is generally 10ms-30ms, the frame shift is generally about half the frame length, and a Hanning window is used for windowing. Next, a short-time Fourier transform is performed on the framed and windowed speech data to obtain a two-dimensional time-frequency spectrum of the speech data. Then, the absolute value of the time-frequency spectrum is calculated, thereby retaining the distribution characteristics of frequency and amplitude while removing phase information and simplifying calculations. The amplitude spectrum is then obtained and used as the input of the feature denoising neural network.

[0071] Step S12: construct a feature denoising neural network, and train the feature denoising neural network based on the amplitude spectrum; wherein the feature denoising neural network is used to denoise the amplitude spectrum of the speech data to obtain a denoised amplitude spectrum.

[0072] Specifically, the feature denoising neural network described in this application is used to denoise the amplitude spectrum of speech data to obtain a denoised amplitude spectrum. That is, the amplitude spectrum of the speech data is input into the feature denoising neural network, and the corresponding denoised amplitude spectrum can be output.

[0073] The feature denoising neural network includes N first fully connected layers, N ReLU (Rectified Linear Unit, linear rectifier function) activation functions, N second fully connected layers and M Sigmoid activation functions (also known as S-shaped growth curves) connected in sequence; where M and N are natural numbers. Taking N=1 and M=1 as an example, the feature denoising neural network is as follows Figure 2As shown. Among them, the N first fully connected layers are used to perform feature compression on the amplitude spectrum, that is, the number of output channels of the first fully connected layer is less than the number of input channels, thereby obtaining a compressed amplitude spectrum. The N ReLU activation functions are used to perform nonlinear processing on the compressed amplitude spectrum to obtain a nonlinear amplitude spectrum. The N second fully connected layers are used to perform feature expansion on the nonlinear amplitude spectrum, that is, the number of output channels of the second fully connected layer is greater than the number of input channels, thereby obtaining an expanded amplitude spectrum, and the expanded amplitude spectrum is the same size as the amplitude spectrum. The M Sigmoid activation functions are used to perform nonlinear processing on the expanded amplitude spectrum to obtain a mask for removing noise components in the amplitude spectrum, and multiply the mask with the amplitude spectrum of the speech data to obtain a denoised amplitude spectrum (MASK). It should be noted that the number of fully connected layers in the feature denoising neural network and the number of input and output channels of the fully connected layers can be adjusted according to the application scenario requirements or the complexity of the noise environment. For example, when the noise environment is more complex, such as in conference venues, stations, etc., the number of fully connected layers can be appropriately increased, or the number of input and output channels of each fully connected layer can be appropriately increased to improve network performance. That is, the improvement of feature denoising performance is achieved by sacrificing the number of parameters and the amount of calculation.

[0074] When training the feature denoising neural network, the amplitude spectrum is first input into the neural network to obtain the corresponding denoised amplitude spectrum. Next, the signal-to-noise ratio loss between the denoised amplitude spectrum and the amplitude spectrum of the corresponding noise-free speech data is calculated during forward propagation. Then, based on the gradient of the loss function for each model parameter (weight and bias) during backpropagation, a gradient backpropagation is performed to optimize the parameters of each model layer to minimize the signal-to-noise ratio loss, thereby obtaining a trained feature denoising neural network.

[0075] Step S13: extracting the logarithmic Mel-spectrogram features of the speech data in the keyword speech data set and the non-keyword speech data set.

[0076] Specifically, the keyword speech data set and the non-keyword speech data set are used as training data sets to train a time binary weighted neural network, wherein the keywords and the non-keywords are both predefined.

[0077] For the speech data of the keyword speech dataset and the non-keyword speech dataset, the speech data is first input into the feature denoising neural network to obtain denoised amplitude spectrum features. The denoised amplitude spectrum is then subjected to mel filtering and logarithmic processing to obtain the logarithmic mel spectrum features. Preferably, the number of mel filters used in the mel filtering is 10, and the logarithm is taken with a base of 2.

[0078] Step S14: construct a time binary weighted neural network and train the time binary weighted neural network based on the logarithmic Mel spectrum feature; wherein the time binary weighted neural network is used to obtain probability score values of keywords and non-keywords in the speech data based on the logarithmic Mel spectrum feature.

[0079] Specifically, the temporal binary weighted network in this application is a lightweight neural network for speech keyword detection and classification tasks, which takes logarithmic Mel spectrum features as input and outputs probability score values of keywords and non-keywords in speech data. It should be noted that the convolution layers used in the temporal binary weighted network are all binary weighted convolution layers, that is, the convolution weight parameter values are all binarized into single-bit binary data (1 and -1), which significantly reduces the memory usage of the neural network. In addition, since the weights are all 1 and -1, no multiplication calculation is required in the convolution operation, which greatly reduces the computational complexity of the neural network.

[0080] In one embodiment, if Figure 3 As shown, the temporal binary weighted neural network includes a 1x3 binary weighted convolution layer, a batch normalization layer (BN layer), a ReLU activation function, multiple temporal binary weighted convolution blocks, a global average pooling layer (Global_AvgPooL) and a third fully connected layer (Fully connectedlayer) connected in sequence. Preferably, three temporal binary weighted convolution blocks are used. The temporal binary weighted convolution block includes a 1x9 binary weighted convolution layer, a batch normalization layer (BN layer) and a ReLU activation function connected in sequence. The output channels of the four binary weighted convolution layers in the temporal binary weighted neural network increase in sequence. For example, the output channels are selected in the form of (2x, 3x, 4x, 6x). When x=8, the number of output channels of each binary weighted convolution layer is 16, 24, 32, and 48, respectively, that is, gradually increasing from top to bottom, in order to ensure that the neural network can further extract richer and more diverse feature information. It should be noted that the number of input and output channels of each convolution layer in the time binary weighted neural network and the number of time binary weighted convolution blocks can be appropriately adjusted according to the number of keywords to be detected or the complexity of the noise environment. For example, when the number of keywords to be detected is greater, the number of time binary weighted convolution blocks can be appropriately increased, or the number of input and output channels of each time binary weighted convolution layer can be appropriately increased, thereby improving network performance. Alternatively, when the noise environment is very complex and the performance of the feature denoising neural network is not enough to remove the noise completely, the number of time binary weighted convolution blocks can be appropriately increased, or the number of input and output channels of each time binary weighted convolution layer can be appropriately increased, thereby improving the accuracy of keyword detection by improving the classification performance and noise resistance of the time binary weighted neural network itself.

[0081] The processing flow of the temporal binary weighted neural network is as follows: First, the single-channel two-dimensional logarithmic Mel spectrum feature of the speech data is transformed into a multi-channel one-dimensional time domain feature, that is, the frequency domain dimension and the channel dimension are exchanged. Taking the 1×10×61 logarithmic Mel spectrum feature as an example, where 1 is the number of channels, 10 is the frequency domain dimension (the number of Mel filters), and 61 is the time domain dimension (the number of frames), it is transformed into a 10×1×61 multi-channel one-dimensional time domain feature and used as the input of the temporal binary weighted neural network. Next, the multi-channel one-dimensional time domain feature is input into the first binary weighted convolution layer of the temporal binary weighted neural network, and a binary weighted convolution operation is performed to further extract the key information in the feature input and expand the number of channels of the feature input. Specifically, in the binary weighted convolution operation, since the weights are only 1 and -1, the multiplication operation in the traditional convolution operation can be simplified to a simple sign judgment operation. That is, when the weight is 1, the multiplication result is the input value multiplied by it itself, and when the weight is -1, the multiplication result is the opposite of the input value multiplied by it. Ultimately, the multiplication and addition calculation required by the traditional convolution can be simplified to an addition calculation. Subsequently, the normalization processing of the BN layer and the nonlinear processing of the ReLU activation function are performed. These two steps do not change the dimension of the feature. Then, the multi-channel one-dimensional time domain features obtained by the above processing are sequentially input into the three time binary weighted convolution blocks connected in series, and the binary weighted convolution operation, BN, and ReLU processing are performed in each time binary weighted block. In this process, the number of channels of the feature input will increase with the increase in the number of output channels of the binary weighted convolution layer, thereby obtaining richer and more diverse key feature information. Next, the multi-channel one-dimensional time-domain features obtained in the above process are input into the global average pooling layer for dimensionality reduction, i.e., feature integration. Specifically, this is done by accumulating and averaging the second dimension, i.e., the time-domain dimension, to obtain a one-dimensional feature vector for classification. Finally, the one-dimensional feature vector obtained in the previous step is input into the final fully connected layer, i.e., the classification layer, to obtain the probability scores for each keyword and non-keyword.

[0082] When training the time binary weighted neural network, the data set consisting of the keyword speech data set and the non-keyword speech data set can be randomly noised, such as adding traffic noise, wind noise, music, venue noise, etc., to improve the noise robustness of the neural network. The loss function used in neural network training is the cross-entropy loss (Cross-Entropy Loss) function. During the training process, the logarithmic Mel spectrum features of the speech data are input into the time binary weighted neural network for forward propagation, further extracting feature information, and outputting the classification probability value. When performing forward propagation, the weight values in the binary weighted convolution layer need to be binarized during the training process. That is, in wb They represent the binarized weights, and w represents the full-precision weight. r Represents the normalized full-precision weight, w.mean represents the mean of the full-precision weight w; w.std represents the standard deviation of the full-precision weight w. Before the full-precision weight w is binarized, it will be standardized, that is, w will be normalized, in order to balance the weight distribution while reducing the quantization error when w is binarized to 1 and -1. Then, the cross-entropy loss is calculated according to the probability score value and the speech data set label, and the gradient of the loss function for each model parameter (weight and bias) during the back propagation process is used to optimize the parameters of each layer of the model to minimize the cross-entropy loss, thereby obtaining a trained time binary weight neural network. During back propagation, due to the non-differentiability (non-differentiable) of the binarization function Sign used in the binary weight convolution layer, the continuously differentiable Hardtanh activation function (hard tangent activation function) method is used to approximate and replace the Sign function for reverse gradient.

[0083] Step S15: Based on the keyword speech data set and the non-keyword speech data set, the feature denoising neural network and the time binary weighted neural network are jointly trained to perform speech keyword detection based on the trained feature denoising neural network and time binary weighted neural network; wherein the amplitude spectrum of the speech data set is input into the feature denoising network to obtain a denoised amplitude spectrum, logarithmic Mel spectrum features are extracted based on the denoised amplitude spectrum, and the logarithmic Mel spectrum features are input into the time binary weighted neural network.

[0084] Specifically, the feature denoising neural network and the time binary weighted neural network are combined into a feature denoising-time binary weighted neural network for joint training. The data sets used are the keyword speech data set and the non-keyword speech data set, and the noise addition ratio during training is appropriately increased. The loss function used is the cross entropy loss function.

[0085] It should be noted that when jointly training the feature denoising neural network and the temporal binary weighted neural network, the parameters of the feature denoising neural network are fixed and participate only in forward propagation, not in backpropagation. That is, no parameter updates are performed, and only the parameters of the temporal binary weighted neural network are adaptively updated and adjusted. Therefore, separate training of the temporal binary weighted neural network can be regarded as reasonable parameter initialization for the temporal binary weighted neural network during joint training, thereby reducing the difficulty of network training and thus improving network performance.

[0086] During the training process, the amplitude spectrum of the speech data set is input into the feature denoising network to obtain the denoised amplitude spectrum, the logarithmic Mel spectrum features are extracted based on the denoised amplitude spectrum, and the logarithmic Mel spectrum features are input into the time binary weighted neural network to obtain the probability score values of keywords and non-keywords in the speech data. Then, the cross entropy loss is calculated based on the probability score value and the speech data set label, and the gradient of the loss function to the parameters (weights and bias) of the time binary weighted neural network during the backpropagation process is used to perform gradient backpropagation to optimize the parameters of each layer of the time binary weighted neural network to minimize the cross entropy loss, so as to obtain the trained feature denoising neural network and time binary weighted neural network. Among them, the weights of the binary weighted convolution layer are also binarized during the joint training. Through the above-mentioned joint training method, the model parameters of the time binary weighted neural network can be adaptively adjusted, thereby effectively improving the classification accuracy of the feature denoising-time binary weighted neural network.

[0087] The protection scope of the voice keyword detection method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, subtracting, or replacing steps in the existing technology based on the principles of the present application are included in the protection scope of the present application.

[0088] An embodiment of the present application also provides a speech keyword detection model training system, which can implement the speech keyword detection method described in the present application. However, the implementation device of the speech keyword detection model training system described in the present application includes but is not limited to the structure of the speech keyword detection model training system listed in this embodiment. All structural deformations and replacements of the existing technology made according to the principles of the present application are included in the protection scope of the present application.

[0089] like Figure 4 As shown, in one embodiment, the speech keyword detection model training system of the present application includes a first extraction module 41, a first training module 42, a second extraction module 43, a second training module 44 and a third training module 45.

[0090] The first extraction module 41 is used to extract the amplitude spectrum of the speech data in the noise-free speech dataset and the noise-added speech dataset.

[0091] The first training module 42 is connected to the first extraction module 41, and is used to construct a feature denoising neural network and train the feature denoising neural network based on the amplitude spectrum; wherein the feature denoising neural network is used to denoise the amplitude spectrum of the speech data to obtain a denoised amplitude spectrum.

[0092] The second extraction module 43 is used to extract the logarithmic Mel-spectrogram features of the speech data in the keyword speech data set and the non-keyword speech data set.

[0093] The second training module 44 is connected to the second extraction module 43, and is used to construct a time binary weighted neural network and train the time binary weighted neural network based on the logarithmic Mel spectrum feature; wherein the time binary weighted neural network is used to obtain the probability score values of keywords and non-keywords in the speech data based on the logarithmic Mel spectrum feature.

[0094] The third training module 45 is connected to the first training module 42 and the second training module 44, and is used to jointly train the feature denoising neural network and the time binary weighted neural network based on the keyword speech data set and the non-keyword speech data set, so as to perform speech keyword detection based on the trained feature denoising neural network and the time binary weighted neural network; wherein the amplitude spectrum of the speech data set is input into the feature denoising network to obtain the denoised amplitude spectrum, the logarithmic Mel spectrum feature is extracted based on the denoised amplitude spectrum, and the logarithmic Mel spectrum feature is input into the time binary weighted neural network.

[0095] Among them, the structures and principles of the first extraction module 41, the first training module 42, the second extraction module 43, the second training module 44 and the third training module 45 correspond one to one with the steps in the above-mentioned speech keyword detection model training method, so they are not repeated here.

[0096] like Figure 5 As shown, in one embodiment, the voice keyword detection method of the present application includes the following steps S51-S55.

[0097] Step S51: extract the amplitude spectrum of the speech data to be detected.

[0098] Specifically, when performing voice keyword detection, the amplitude spectrum of the voice data to be detected is first extracted. When performing amplitude spectrum extraction on the voice data to be monitored, the voice data is first framed and windowed; then the framed and windowed voice data is short-time Fourier transformed to obtain a two-dimensional time-frequency spectrum of the voice data; then the absolute value of the time-frequency spectrum is calculated to obtain the amplitude spectrum.

[0099] Step S52: input the amplitude spectrum into the feature denoising neural network trained by the above-mentioned speech keyword detection model training method to obtain a denoised amplitude spectrum.

[0100] Step S53: Obtain logarithmic Mel spectrum features based on the denoised amplitude spectrum.

[0101] Specifically, Mel filtering and logarithmic processing are performed on the denoised amplitude spectrum to obtain the logarithmic Mel spectrum feature.

[0102] Step S54: Input the logarithmic Mel-spectrogram features into the temporal binary weighted neural network trained by the above-mentioned speech keyword detection model training method to obtain probability score values of keywords and non-keywords in the speech data.

[0103] Step S55: Determine the keywords contained in the speech data based on the probability score value.

[0104] Specifically, when performing voice keyword detection, since voice is input continuously, in order to avoid result overlap caused by the continuous output of voice keyword detection results, it is necessary to smooth the output results of the time binary weighted neural network, that is, to take a sliding average of the results of each frame within the smoothing window. In this application, the smoothing window is the length of 10 frames of continuous speech, that is, the results of 10 consecutive frames are accumulated and averaged, which is used as the final probability value after the sliding average of each keyword and non-keyword.

[0105] The maximum value of the probability score values is compared with a keyword detection threshold, assuming that the detection threshold is set to 0.7. If the maximum value is greater than or equal to the detection threshold 0.7 and the maximum value corresponds to a keyword, it is determined that the voice data contains the keyword.

[0106] The protection scope of the voice keyword detection method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, subtracting, or replacing steps in the existing technology based on the principles of the present application are included in the protection scope of the present application.

[0107] An embodiment of the present application also provides a voice keyword detection system, which can implement the voice keyword detection method described in the present application. However, the implementation device of the voice keyword detection system described in the present application includes but is not limited to the structure of the voice keyword detection system listed in this embodiment. All structural deformations and replacements of the existing technology made according to the principles of the present application are included in the protection scope of the present application.

[0108] like Figure 6 As shown, in one embodiment, the present application provides a voice keyword detection system including an extraction module 61 , a first acquisition module 62 , a second acquisition module 63 , a third acquisition module 64 and a detection module 65 .

[0109] The extraction module 61 is used to extract the amplitude spectrum of the speech data to be detected.

[0110] The first acquisition module 62 is connected to the extraction module 61 and is used to input the amplitude spectrum into the feature denoising neural network trained by the above-mentioned speech keyword detection model training method to obtain a denoised amplitude spectrum.

[0111] The second acquisition module 63 is connected to the first acquisition module 62 and is configured to acquire logarithmic Mel spectrum features based on the denoised amplitude spectrum.

[0112] The third acquisition module 64 is connected to the second acquisition module 63 and is used to input the logarithmic Mel spectrum features into the time binary weighted neural network trained by the above-mentioned speech keyword detection model training method to obtain the probability score values of keywords and non-keywords in the speech data.

[0113] The detection module 65 is connected to the third acquisition module 64 and is configured to determine the keywords contained in the speech data based on the probability score value.

[0114] Among them, the structures and principles of the extraction module 61, the first acquisition module 62, the second acquisition module 63, the third acquisition module 64 and the detection module 65 correspond one-to-one to the steps in the above-mentioned voice keyword detection method, so they are not repeated here.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices or methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.

[0116] The modules / units described as separate components may or may not be physically separate, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into a processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into a single module / unit.

[0117] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] The embodiment of the present application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state drive, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid-state drive (SSD)), etc.

[0119] An embodiment of the present application further provides an electronic device comprising a processor and a memory.

[0120] The memory is used to store computer programs.

[0121] The memory includes various media that can store program codes, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0122] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned voice keyword detection model training method or the above-mentioned voice keyword detection method.

[0123] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0124] like Figure 7 As shown, the electronic device of the present application is in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units 71, a memory 72, and a bus 73 connecting different system components (including the memory 72 and the processing unit 71).

[0125] Bus 73 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0126] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0127] The memory 72 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 721 and / or cache memory 722. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 723 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive"). Although Figure 7Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 73 via one or more data medium interfaces. The memory 72 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present application.

[0128] A program / utility 724 having a set (at least one) of program modules 7241 may be stored, for example, in memory 72. Such program modules 7241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 7241 generally implement the functions and / or methods of the embodiments described herein.

[0129] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 74. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 75. Figure 7 As shown, the network adapter 75 communicates with other modules of the electronic device via the bus 73. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0130] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A speech keyword detection model training method, characterized in that: The method comprises the following steps: Extracting the amplitude spectrum of speech data from a noise-free speech dataset and a noise-added speech dataset; Constructing a feature denoising neural network, and training the feature denoising neural network based on the amplitude spectrum; wherein the feature denoising neural network is used to denoise the amplitude spectrum of the speech data to obtain a denoised amplitude spectrum; Extract the logarithmic Mel spectrum features of speech data from the keyword speech dataset and the non-keyword speech dataset; Constructing a time binary weighted neural network and training the time binary weighted neural network based on the logarithmic Mel spectrum feature; wherein the time binary weighted neural network is used to obtain probability score values of keywords and non-keywords in the speech data based on the logarithmic Mel spectrum feature; Based on the keyword speech data set and the non-keyword speech data set, the feature denoising neural network and the time binary weighted neural network are jointly trained to perform speech keyword detection based on the trained feature denoising neural network and the time binary weighted neural network; wherein the amplitude spectrum of the speech data set is input into the feature denoising network to obtain a denoised amplitude spectrum, logarithmic Mel spectrum features are extracted based on the denoised amplitude spectrum, and the logarithmic Mel spectrum features are input into the time binary weighted neural network.

2. The speech keyword detection model training method according to claim 1, characterized in that: Extracting the amplitude spectrum of speech data includes the following steps: Performing frame division and windowing on the voice data; Performing short-time Fourier transform on the framed and windowed speech data to obtain a time-frequency spectrum of the speech data; The absolute value of the time-frequency spectrum is calculated to obtain the amplitude spectrum.

3. The speech keyword detection model training method according to claim 1, characterized in that: The feature denoising neural network includes N first fully connected layers, N ReLU activation functions, N second fully connected layers and M Sigmoid activation functions connected in sequence; where M and N are natural numbers; The N first fully connected layers are used to perform feature compression on the amplitude spectrum to obtain a compressed amplitude spectrum; The N ReLU activation functions are used to perform nonlinear processing on the compressed amplitude spectrum to obtain a nonlinear amplitude spectrum; The N second fully connected layers are used to perform feature expansion on the nonlinear amplitude spectrum to obtain an extended amplitude spectrum, where the extended amplitude spectrum has the same size as the amplitude spectrum; The M Sigmoid activation functions are used to perform nonlinear processing on the expanded amplitude spectrum, obtain a mask for removing noise components in the amplitude spectrum, and multiply the mask by the amplitude spectrum of the speech data to obtain a denoised amplitude spectrum.

4. The speech keyword detection model training method according to claim 1, characterized in that: Extracting the logarithmic Mel-spectrogram features of speech data includes the following steps: Mel filtering and logarithmic processing are performed on the denoised amplitude spectrum to obtain the logarithmic Mel spectrum feature.

5. The speech keyword detection model training method according to claim 1, wherein: The temporal binary weighted neural network includes a binary weighted convolution layer, a batch normalization layer, a ReLU activation function, a plurality of temporal binary weighted convolution blocks, a global average pooling layer and a third fully connected layer connected in sequence; The temporal binary weighted convolution block includes a binary weighted convolution layer, a batch normalization layer and a ReLU activation function connected in sequence; The output channels of the multiple binary weight convolution layers in the temporal binary weight neural network are increased in sequence; The weight parameters in the binary weight convolution layer are binarized to 1 and -1.

6. The speech keyword detection model training method according to claim 1, wherein: When the feature denoising neural network and the time binary weighted neural network are jointly trained, the parameters of the feature denoising neural network remain unchanged, and only the parameters of the time binary weighted neural network are updated.

7. A speech keyword detection model training system, characterized in that: The system includes a first extraction module, a first training module, a second extraction module, a second training module and a third training module; The first extraction module is used to extract the amplitude spectrum of the speech data in the noise-free speech data set and the noisy speech data set; The first training module is used to construct a feature denoising neural network and train the feature denoising neural network based on the amplitude spectrum; wherein the feature denoising neural network is used to denoise the amplitude spectrum of the speech data to obtain a denoised amplitude spectrum; The second extraction module is used to extract the logarithmic Mel spectrum features of the speech data in the keyword speech data set and the non-keyword speech data set; The second training module is used to construct a time binary weighted neural network and train the time binary weighted neural network based on the logarithmic Mel spectrum feature; wherein the time binary weighted neural network is used to obtain probability score values of keywords and non-keywords in the speech data based on the logarithmic Mel spectrum feature; The third training module is used to jointly train the feature denoising neural network and the time binary weighted neural network based on the keyword speech data set and the non-keyword speech data set, so as to perform speech keyword detection based on the trained feature denoising neural network and time binary weighted neural network; wherein the amplitude spectrum of the speech data set is input into the feature denoising network to obtain the denoised amplitude spectrum, the logarithmic Mel spectrum feature is extracted based on the denoised amplitude spectrum, and the logarithmic Mel spectrum feature is input into the time binary weighted neural network.

8. A method for detecting speech keywords, characterized in that: The method comprises the following steps: Extracting the amplitude spectrum of the speech data to be detected; Inputting the amplitude spectrum into the feature denoising neural network trained by the speech keyword detection model training method according to any one of claims 1 to 6 to obtain a denoised amplitude spectrum; Acquire a logarithmic Mel spectrum feature based on the denoised amplitude spectrum; Inputting the logarithmic Mel-spectrogram feature into the temporal binary weighted neural network trained by the speech keyword detection model training method according to any one of claims 1 to 6 to obtain probability score values of keywords and non-keywords in the speech data; Keywords contained in the speech data are determined based on the probability score value.

9. The method for detecting speech keywords according to claim 8, wherein: Determining the keywords contained in the voice data based on the probability score value includes the following steps: Smoothing the probability score value to obtain a sliding average; When the maximum value of the sliding average values is greater than or equal to a preset threshold and the maximum value corresponds to a keyword, it is determined that the voice data contains the keyword.

10. A speech keyword detection system, characterized by: The system includes an extraction module, a first acquisition module, a second acquisition module, a third acquisition module and a detection module; The extraction module is used to extract the amplitude spectrum of the speech data to be detected; The first acquisition module is used to input the amplitude spectrum into the feature denoising neural network trained by the speech keyword detection model training method according to any one of claims 1 to 6 to obtain the denoised amplitude spectrum; The second acquisition module is used to obtain logarithmic Mel spectrum features based on the denoised amplitude spectrum; The third acquisition module is used to input the logarithmic Mel spectrum feature into the time binary weighted neural network trained by the speech keyword detection model training method according to any one of claims 1 to 6 to obtain the probability score values of keywords and non-keywords in the speech data; The detection module is used to determine the keywords contained in the speech data based on the probability score value.

11. An electronic device, characterized in that: The electronic device includes: a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the electronic device performs the voice keyword detection model training method according to any one of claims 1 to 6 or the voice keyword detection method according to any one of claims 8 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the speech keyword detection model training method according to any one of claims 1 to 6 or the speech keyword detection method according to any one of claims 8 to 9 is implemented.

Images