A lightweight speech keyword recognition method, device, and medium

Abstract

The application relates to a lightweight speech keyword recognition method, device and medium. The method comprises the following steps: obtaining to-be-recognized speech, converting the to-be-recognized speech into a feature frame vector, inputting the feature frame vector into a keyword recognition model, obtaining a probability array of a classification label matched with a target keyword, and determining whether the target keyword is contained in the to-be-recognized speech according to the probability array. The keyword recognition model comprises the following modules: a time convolution module comprising a plurality of convolution layers combined according to a preset hole coefficient array, which is used for extracting time sequence correlation information; a nesting module; a compression module comprising a plurality of convolution layers combined according to a preset step length, which is used for compressing and extracting features of data obtained by splicing the outputs of the time convolution module and the nesting module in the channel depth; and a full-connection classification output module used for obtaining the probability array. Under the constraint of small parameter quantity and operation complexity, the method has a good recognition accuracy, and is more suitable for deployment on an embedded system.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a lightweight speech keyword recognition method, device, and medium. Background Technology

[0002] With the advancement of speech processing technology and the popularization of edge computing, the application scenarios of speech recognition and interaction are becoming more extensive. However, edge intelligent devices such as smartwatches face constraints such as privacy protection, real-time communication, and power consumption when conducting real-time voice interaction applications. This has led to a need for lightweight, localized speech keyword recognition mechanisms with low parameter and computational requirements. Traditional speech keyword recognition methods include Hidden Markov Models (HMMs) and Gaussian mixture models (GMMs), which require high computational power and are difficult to apply efficiently to embedded devices and other systems. In recent years, neural network models have become increasingly popular in speech keyword recognition, including: methods based on deep neural networks (DNNs), but DNN models generally have a large number of parameters and are difficult to effectively extract temporal correlation information of speech; methods based on convolutional neural networks (CNNs) have achieved good recognition results with a smaller number of parameters, but there is still considerable room for improvement in recognition efficiency; methods based on residual networks (ResNets) use dilated convolutions to expand the receptive field of the network, but ResNet methods often require hundreds of thousands or more parameters, making the network model heavy; and methods based on temporal convolutional neural networks (TCNs) have achieved good results in areas such as feature extraction of temporal data. Based on this, existing methods face the following challenges: to achieve high recognition accuracy, the neural network models involved in general methods often have a large number of parameters, frequently exceeding 100KB, which is difficult for embedded systems with limited RAM / flash space to effectively support and run; at the same time, the high computational complexity brought by the complex model structure also leads to problems such as computational latency.

[0003] In summary, current keyword recognition methods suffer from problems such as large number of neural network parameters and high model complexity. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a lightweight speech keyword recognition method, device, and medium. Through a specific neural network structure design, it achieves a high recognition accuracy at the cost of a small total number of model parameters and low model complexity, making it more suitable for deployment in embedded systems such as MCUs with small memory and low computing power.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] One aspect of the present invention provides a lightweight speech keyword recognition method, comprising the following steps:

[0007] The process involves acquiring the speech to be recognized, converting it into a feature frame vector that matches a preset format, inputting it into a pre-trained keyword recognition model, obtaining a probability array of classification labels matching the target keyword, and determining whether the speech contains the target keyword based on the probability array.

[0008] The keyword recognition model includes:

[0009] The temporal convolution module includes multiple convolutional layers combined according to a preset dilation coefficient, used to extract intertemporal correlation information based on the feature frame vector and obtain first output data;

[0010] A nested module is used to obtain second output data based on the feature frame vector;

[0011] The compression module includes multiple convolutional layers combined with a preset stride, used to compress and extract features from the first output data and the data concatenated with the second output data to obtain the third output data;

[0012] The fully connected classification output module is used to obtain the probability array based on the third output data.

[0013] As a preferred technical solution, the temporal convolution module includes multiple sets of Conv1D and / or Conv2D layers combined and sequentially connected according to the hole coefficient, as well as corresponding activation function layers.

[0014] As a preferred technical solution, the compression module includes multiple sets of Conv1D and / or Conv2D layers and corresponding activation function layers that are combined and sequentially connected according to the step size, for concatenating the first output data and the second output data in the channel dimension and performing feature extraction and compression processing on the concatenated data.

[0015] As a preferred technical solution, the fully connected classification output module includes a straightening layer and a corresponding classification layer connected in sequence.

[0016] As a preferred technical solution, the temporal convolution module includes four sets of Conv1D layers and corresponding ReLU activation function layers, which are combined and connected sequentially according to a preset dilation coefficient; the nested module includes one Conv1D layer and a corresponding ReLU activation function layer; the compression module includes two sets of Conv2D layers and corresponding ReLU activation function layers, which are combined and connected sequentially according to a preset stride; the fully connected classification output module includes a flattening layer and a softmax classification layer connected sequentially.

[0017] As a preferred technical solution, the void coefficient is [1,2,4,8] and the step size is [2,2].

[0018] As a preferred technical solution, the process of obtaining the feature frame vector includes the following steps:

[0019] The Mel frequency cepstral coefficients of the speech to be recognized are obtained using a Mel filter bank, and the feature frame vector with a multi-channel time-series data format is obtained based on the Mel frequency cepstral coefficients.

[0020] As a preferred technical solution, the data input process for the pre-trained keyword recognition model is as follows:

[0021] Determine whether the feature frame vector is less than or equal to a preset length. If yes, the first frame of the feature frame vector is taken as the starting frame, and the part of the feature frame vector that is less than the preset length is filled with 0. If no, starting from the first frame of the feature frame vector, shift in the time direction according to the preset frame step size, and cyclically determine the starting frame.

[0022] Starting from the starting frame, a feature frame vector of a preset length is read as input data for the keyword recognition model.

[0023] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for performing the lightweight speech keyword recognition method described above.

[0024] In another aspect, the present invention provides a computer-readable storage medium comprising one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the lightweight speech keyword recognition method described above.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] (1) By introducing a temporal convolution module into the keyword recognition model, compared with the classic CNN convolutional neural network, it can effectively extract the temporal correlation information features of speech data, achieve a high recognition accuracy under specific parameter constraints, and improve the model's ability to extract temporal correlation information.

[0027] (2) By introducing nested modules, the output data of the temporal convolution module and the nested modules are spliced ​​together in the channel depth, thereby increasing the ability of subsequent modules to extract features from the original input data.

[0028] (3) By introducing a compression module into the keyword recognition model, the output data volume of the model is greatly reduced while increasing the nonlinearity of the model. This greatly reduces the number of parameters of the fully connected classification output module and the compression module, so that while achieving a high recognition accuracy, the total number of parameters of the neural network model is greatly reduced, and the model occupies less memory during runtime. Attached Figure Description

[0029] Figure 1 The diagrams show the structure of the neural network models in Examples 1 and 2.

[0030] Figure 2 This is a flowchart of the steps in Example 1;

[0031] Figure 3 This is a flowchart of the algorithm in Example 1;

[0032] Figure 4 This is a schematic diagram illustrating the training and testing results of the neural network model in Example 2 on a general dataset. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] Example 1

[0035] This embodiment provides a lightweight speech keyword recognition method. For example... Figure 1 The above is a structural diagram of the neural network model used in this embodiment. Figure 2 The above is a flowchart illustrating the identification method of this embodiment. Figure 3 The above is a schematic diagram of the algorithm flow in this embodiment.

[0036] The keyword recognition method in this embodiment includes the following steps:

[0037] Step S1: MFCC feature extraction of the speech to be recognized. MFCC feature data of a specific length (t_1 ≥ 1s) of speech is extracted through specific operations. In this embodiment, the relevant settings are: sampling rate of 16k; frame length of 480, corresponding to a duration of 0.03s; step size of 320, corresponding to a duration of 0.02s. After performing MFCC feature extraction operations on the speech to be recognized, two-dimensional time-frequency feature data with an output size of (t_1 / 0.02s rounded up) columns * 40 rows can be obtained, where 1s of duration corresponds to 50*40 two-dimensional time-frequency feature data.

[0038] Step S1 specifically includes the following operations:

[0039] S11, framing: The voice data is divided into segments of 0.03s in length and with a step size of 0.02s. Each segment is a frame of data.

[0040] S12, Windowing, performing window function processing on each frame of data;

[0041] S13, FFT transform, performs Fourier transform (FFT) on a windowed frame of data;

[0042] S14, Mel filtering: Based on the characteristics of the human auditory perception system, the amplitude spectrum obtained by FFT transformation is multiplied by the frequency of each filter and accumulated. The resulting value is the energy value of the frame data in the corresponding frequency band of the filter.

[0043] S15, DCT Transform, applies Discrete Cosine Transform (DCT) to perform correlation processing on the filter bank coefficients.

[0044] Step S2: In the neural network model operation section, the MFCC feature data extracted in step S1 is read from the starting column of the data in this loop for 50 columns. The starting column of the data is column 1 when the loop is executed for the first time, corresponding to reading 50*40 two-dimensional feature data, which is input into the trained neural network model. After the model operation, a probability array is output.

[0045] In a model that can recognize 10 speech keywords, the output probability array is a 1-dimensional array of length 11, corresponding to 11 classification labels. The first 10 values ​​are the probability values ​​corresponding to the labels of the recognizable target keywords, and the last value is the probability value corresponding to the unrecognized / unknown labels.

[0046] In the training part of the neural network model, MFCC features are extracted from the training speech dataset in a Python environment, with the sampling rate, frame length, stride, and other settings exactly the same as in step S1. Then, the TFrecord tool is used to create a dataset format readable by the TensorFlow V2.x machine learning framework from the extracted MFCC feature data. This is implemented using the Keras framework within TensorFlow V2.x. Figure 1-2 The diagram shows modules M1, M2, M3, and M4 in the neural network model, and the overall model setup. After the model is built, the created dataset is imported into the model, the relevant training hyperparameters are adjusted, the model training is completed, and the model is saved.

[0047] Step S3: The MFCC feature data extracted in step S1 is processed at a specific loop frequency of 0.1s, with a step size of 5 frames / column. The starting point of the Nth loop is the (N-1)*5+1th column. 50 columns are read continuously, corresponding to 50*40 two-dimensional feature data, and input into the trained neural network model for calculation. After each execution of step S2, N is incremented by 1.

[0048] Step S4: The maximum probability value P and corresponding classification label Y in the probability array output by the model during a single loop are used as the recognition result for that single loop. The classification label Yn corresponding to the maximum probability value Pn = max{Pi} in the output results of N loops is used to determine whether the target keyword exists in the speech data to be recognized. If the classification label corresponding to the maximum probability value in the output results of N loops is unknown, then the target keyword does not exist in the speech data. If the classification label corresponding to the maximum probability value in the output results of N loops is not unknown, then the keyword corresponding to the non-unknown label with the highest probability value is the recognized target keyword.

[0049] Example 2

[0050] This embodiment illustrates the effectiveness of the lightweight speech keyword recognition method provided in Embodiment 1 by combining a specific speech dataset:

[0051] (1) Speech dataset

[0052] The `data_speech_commands_v0.02` dataset (Google Brain, 2018) is the second version of the English speech dataset provided by Google. This dataset contains 105,829 WAV-formatted speech samples across 35 words. Each speech sample is a single-channel PCM value, with a duration of 1 second and containing only one English word. Following the common model accuracy measurement approach in current speech keyword recognition research, 10 words—"yes," "no," "up," "down," "left," "right," "on," "off," "stop," and "go"—were selected as target keywords for recognition. The remaining speech samples were labeled "unknown" to correspond to speech samples outside of these 10 words, resulting in a total of 11 labels. In this embodiment, the dataset is divided into three sets in an 8:1:1 ratio: a training set, a validation set, and a test set.

[0053] (2) Implementation process

[0054] In the neural network model training part, the librosa (Python) library tool is used to extract MFCC features from the speech data of the aforementioned three datasets. The relevant settings in this embodiment are: sampling rate of 16k; frame length of 480, corresponding to a duration of 0.03s; stride of 320, corresponding to a duration of 0.02s; each speech data point yields 50*40 one-dimensional (50 groups) feature data with a channel depth of 40. The TFrecord tool is used to create a dataset readable by the TensorFlow V2.x machine learning framework, ultimately including three datasets: a training dataset, a validation dataset, and a test dataset.

[0055] The training process employs a multi-step continuous training approach, specifically consisting of six steps. Each step has an epoch of [130, 70, 70, 70, 70, 130] and a corresponding drop rate of [0.31, 0.21, 0.16, 0.12, 0.1, 0.1]. In the sixth step, the ReduceLROnPlateau tool is used to automatically reduce the learning step size during model training. Simultaneously, the Adam optimization strategy is employed during training. The training monitor objective is set as validation loss, with the loss function provided by the Keras framework. The model's recognition performance is statistically analyzed using recognition accuracy, which is then normalized.

[0056] (3) Implementation effect

[0057] To evaluate the effectiveness of the model and method in this invention, this embodiment uses the above-mentioned dataset for experiments and effect analysis. Figure 4 As shown, the loss and accuracy process of the model and method in this invention on the training set and validation set during the training process is recorded. Table 1 shows the inference accuracy of the trained model on the above three datasets and the dataset as a whole, where the recognition accuracy on the test set is 97.84%.

[0058] Table 1: Test results of the neural network model in this invention on the dataset

[0059]

[0060] Based on a literature review (López-Espejo I, Tan ZH, Hansen J, et al. Deep spoken keyword spotting: An overview [J]. IEEE Access, 2021.), this embodiment is compared with current major speech keyword recognition network methods. As shown in Table 2, the model and method in this invention achieve a high recognition accuracy with a smaller number of parameters and computational complexity. Other models with higher recognition accuracy often have more than 100 KB of parameters, making them difficult to deploy effectively on embedded systems with limited memory and computing power.

[0061] Table 2: Comparison of the neural network model in this invention with existing research

[0062]

[0063] As can be seen from the figures and tables above in this embodiment, on a general speech dataset, the model and method of this invention can achieve a high recognition accuracy with a relatively small number of parameters and computational complexity constraints, thus enabling better matching of embedded systems with limited memory and low computing power.

[0064] Example 3

[0065] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the lightweight speech keyword recognition method as described in Embodiment 1.

[0066] Example 4

[0067] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the lightweight speech keyword recognition method as described in Embodiment 1.

[0068] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A lightweight speech keyword recognition method, characterized in that, Includes the following steps: The process involves acquiring the speech to be recognized, converting it into a feature frame vector that matches a preset format, inputting it into a pre-trained keyword recognition model, obtaining a probability array of classification labels matching the target keyword, and determining whether the speech contains the target keyword based on the probability array. The keyword recognition model includes: The temporal convolution module includes multiple convolutional layers combined according to a preset dilation coefficient, used to extract intertemporal correlation information based on the feature frame vector and obtain first output data; A nested module is used to obtain second output data based on the feature frame vector; The compression module includes multiple convolutional layers combined with a preset stride, used to compress and extract features from the first output data and the data concatenated with the second output data to obtain the third output data; The fully connected classification output module is used to obtain the probability array based on the third output data.

2. The lightweight speech keyword recognition method according to claim 1, characterized in that, The temporal convolution module includes multiple sets of Conv1D and / or Conv2D layers combined and sequentially connected according to the hole coefficient, as well as corresponding activation function layers.

3. The lightweight speech keyword recognition method according to claim 1, characterized in that, The compression module includes multiple Conv1D and / or Conv2D layers and corresponding activation function layers that are combined and sequentially connected according to the step size, for concatenating the first output data and the second output data in the channel dimension and performing feature extraction and compression processing on the concatenated data.

4. The lightweight speech keyword recognition method according to claim 1, characterized in that, The fully connected classification output module includes a straightening layer and a corresponding classification layer connected in sequence.

5. The lightweight speech keyword recognition method according to claim 1, characterized in that, The temporal convolution module includes four sets of Conv1D layers and corresponding ReLU activation function layers, which are combined and connected sequentially according to a preset dilation coefficient; the nested module includes one Conv1D layer and a corresponding ReLU activation function layer; the compression module includes two sets of Conv2D layers and corresponding ReLU activation function layers, which are combined and connected sequentially according to a preset stride; the fully connected classification output module includes a flattening layer and a softmax classification layer connected sequentially.

6. The lightweight speech keyword recognition method according to claim 5, characterized in that, The void coefficient is [1,2,4,8], and the step size is [2,2].

7. The lightweight speech keyword recognition method according to claim 1, characterized in that, The process of obtaining the feature frame vector includes the following steps: The Mel frequency cepstral coefficients of the speech to be recognized are obtained using a Mel filter bank, and the feature frame vector with a multi-channel time-series data format is obtained based on the Mel frequency cepstral coefficients.

8. The lightweight speech keyword recognition method according to claim 1, characterized in that, The specific data input process for the pre-trained keyword recognition model is as follows: Determine whether the feature frame vector is less than or equal to a preset length. If yes, the first frame of the feature frame vector is taken as the starting frame, and the part of the feature frame vector that is less than the preset length is filled with 0. If no, starting from the first frame of the feature frame vector, shift in the time direction according to the preset frame step size, and cyclically determine the starting frame. Starting from the starting frame, a feature frame vector of a preset length is read as input data for the keyword recognition model.

9. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for performing the lightweight speech keyword recognition method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It includes one or more programs that are executed by one or more processors of an electronic device, the one or more programs including instructions for performing the lightweight speech keyword recognition method as described in any one of claims 1-8.

Images