Keyword identification method, system and equipment based on double-flow network and medium
By constructing a speech recognition system based on a dual-stream network, the problems of accuracy and computational cost in keyword recognition technology in embedded devices were solved, and high-precision and stable recognition was achieved in noisy environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI MISTAR MEDICAL TECH CO LTD
- Filing Date
- 2024-10-15
- Publication Date
- 2026-04-17
AI Technical Summary
Existing keyword recognition technologies in embedded devices suffer from a trade-off between accuracy and computational cost, making it difficult to maintain high accuracy and stability in environments with noise interference and audio distortion.
A dual-stream network-based approach is adopted to construct a speech recognition stream and an endpoint detection stream. A mapping relationship is established through the SincNet network and the ASPP network, and deep learning is used to train the model to improve the robustness and accuracy of keyword recognition.
It improves the robustness and accuracy of keyword recognition, reduces computational costs, and enhances the recognition accuracy and efficiency of speech recognition streams.
Smart Images
Figure CN121884789A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech recognition, and more particularly to a keyword recognition method, system, device, and medium based on a two-stream network. Background Technology
[0002] Speech recognition technology systems are mainly divided into continuous speech recognition systems and keyword (isolated word) recognition systems. In practical applications, continuous speech recognition systems require a resource-rich equipment environment to operate normally, such as voice assistants, real-time transcription, and chatbots. However, in many scenarios, transcribing the entire speech signal is unnecessary because the speech signal itself has a large degree of redundancy, and its key information often exists in a part of the entire signal, namely, keywords.
[0003] Keyword recognition systems are much easier to implement than continuous speech recognition systems and generally do not require a large amount of memory, making them particularly suitable for embedded devices with small size and diverse application scenarios. They are widely used in mobile devices, wearable devices, and in-vehicle voice control. However, in real-world applications, the reliability and accuracy of keyword recognition are often prerequisites for effective interaction. Noise interference and audio distortion can significantly impact recognition accuracy and stability, and the algorithm's running speed directly affects the user experience. Current deep learning-based keyword recognition technologies generally suffer from a trade-off between accuracy and computational cost. Increasing accuracy requires a significant increase in the number of parameters, consuming the limited memory of embedded systems; conversely, decreasing the algorithm's noise resistance will decrease. Therefore, designing a keyword recognition method that is highly accurate, stable, and computationally inexpensive is essential. Summary of the Invention
[0004] The purpose of this invention is to provide a keyword recognition method, system, device, and medium based on a two-stream network (SRSEDS Net) to solve the problems mentioned in the background art.
[0005] To achieve the above-mentioned objectives, one aspect of the present invention provides a keyword recognition method based on a two-stream network, comprising the following steps:
[0006] Step S1: Process the original keyword speech data to construct a dual-channel dataset for the dual-stream network model;
[0007] Step S2: Construct a two-stream network model for dual-channel speech recognition;
[0008] Step S3: Train the two-stream network model and establish the mapping relationship between noisy speech and keyword categories and speech endpoint information;
[0009] Step S4: Predict keyword categories using the trained two-stream network model.
[0010] Furthermore, step S1 includes the following steps:
[0011] Step S101: The original keyword speech data is selected from the speech command dataset. The dataset is divided into training set, validation set and test set, and 3000 clean speech data in the dataset are selected as the speech data required for training.
[0012] Step S102: Add noise to the 3000 clean speech data in step S101. The noise sources are three types of noise from the GSCD library: pink, white, and exercise_bike. Set the signal-to-noise ratio level to 0dB, 5dB, and 10dB for each type of noise to add noise to the speech signal.
[0013] Step S103: The original 3,000 clean speech data are used as speech keyword category data, and 27,000 sets of speech categories-noisy speech datasets are formed with the noisy speech data. The speech category-noisy speech datasets are then normalized.
[0014] Step S104: Process the speech category - noisy speech dataset, and generate a one-to-one corresponding txt tag index for the noisy speech and speech keywords as the training set;
[0015] Step S105: Use the dual-threshold method to perform endpoint detection on the 3000 clean speech data to obtain the start and end points of the speech segments. Mark the signal points of the speech segments as 1 and the segments without speech signals as 0, forming a binary classification data with a quantity of 27000 and a length of 16000 for each segment.
[0016] Furthermore, the dual-stream network is a dual-channel neural network, wherein the dual channels are a speech recognition stream and an endpoint detection stream, and the speech data outputs different feature data through the dual channels.
[0017] Furthermore, the speech recognition stream is a SincNet network for a multi-classification task. The input is noisy speech data and the output is keyword categories. It establishes a mapping relationship between noisy speech and keyword categories, which is used to fuse endpoint information from the endpoint detection stream at the starting end.
[0018] Furthermore, the endpoint detection stream is an ASPP (Atrous Spatial Pyramid Pooling) network. The input is noisy speech data, and the output is endpoint detection data. It establishes a mapping relationship between noisy speech and speech endpoints, and is used to feed the detected speech endpoint information into the beginning of the speech recognition stream, assisting the speech recognition stream network in determining the start and end positions of speech segments and performing effective recognition.
[0019] Furthermore, the speech recognition stream consists of three ASPP layers and two convolutional layers. The ASPP layers are used to extract multi-directional semantic information of the speech, and the convolutional layers are used to provide the speech recognition stream with a more specific recognition location, so as to improve the accuracy of the speech recognition stream in recognizing keywords.
[0020] Furthermore, in step S3, the loss function for the speech recognition stream in the training network is cross-entropy, the loss function for the endpoint detection stream is binary cross-entropy, and the total network loss function is the sum of the two loss functions.
[0021] A second aspect of the present invention provides a keyword recognition system based on a two-stream network, comprising a dataset module, a two-stream network module, a training module, and a prediction module, wherein:
[0022] The dataset module is used to process the raw keyword speech data and construct a dual-channel dataset for the dual-stream network model;
[0023] The dual-stream network module is used to build a dual-stream network model for dual-channel speech recognition;
[0024] The training module is used to train the two-stream network model and establish a mapping relationship between noisy speech and keyword categories and speech endpoint information;
[0025] The prediction module uses a trained two-stream network model to predict keyword categories.
[0026] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a keyword recognition method based on a two-stream network as described above.
[0027] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a keyword recognition method based on a two-stream network as described above.
[0028] Compared with existing technologies, this system and method have the following advantages:
[0029] 1. This method improves the robustness and accuracy of keyword recognition by creating a corresponding speech dataset and using deep learning technology to train a dual-stream network model that can obtain corresponding keyword category labels and speech start and end points from noisy speech.
[0030] 2. In this invention, the speech recognition stream performs a keyword recognition task. Therefore, the endpoint detection stream incorporates the detected endpoint information into the initial segment of the speech recognition stream, which can help the speech recognition stream quickly determine the start and end positions of the speech segment and perform effective recognition, thereby improving the recognition accuracy and efficiency of the speech recognition stream. Attached Figure Description
[0031] Figure 1 This is a flowchart of a keyword recognition method based on a two-stream network.
[0032] Figure 2 This is a pure audio image.
[0033] Figure 3 This is a noisy speech graph.
[0034] Figure 4 This is an endpoint detection graph.
[0035] Figure 5 This is a structural diagram of a two-stream network model.
[0036] Figure 6 This is a diagram illustrating the training principle. Detailed Implementation
[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] like Figure 1 The diagram shown is a flowchart of the method of the present invention. An embodiment of the present invention provides a keyword recognition method based on a two-stream network, with the specific steps as follows:
[0039] Step S1: Create a suitable two-channel dataset. This includes the following steps:
[0040] Step S101: The original speech data used is the Google SpeechCommands Dataset (GSCD), an open-source dataset provided by Google for keyword search (KWS) and primarily used in natural language processing and computational linguistics research. The dataset contains 105,899 sentences, each with a fixed duration of 1 second and a sampling rate of 16,000 Hz. It is divided into eleven categories: "yes," "no," "up," "down," "left," "right," "on," "off," "stop," and "go," with additional command words used to fill in unknowns, totaling 11 categories. The training set comprises 68.4%, the validation set 11.2%, and the test set 20.4%. 3,000 sentences from the dataset are selected as the speech data required for training.
[0041] Step S102: Add noise to the 3000 clean speech data mentioned in step S101. The noise sources are three types of noise from the GSCD library: pink, white, and exercise_bike. Set the signal-to-noise ratio (SNR) level to 0dB, 5dB, and 10dB for each type of noise to add noise to the speech signal.
[0042] Step S103, the speech keyword category data mentioned in step S101, and the noisy speech with three types of noise and three signal-to-noise ratios added in step S102, formed 3000*9=27000 sets of one-to-one corresponding speech category-noisy speech datasets, and the datasets were normalized.
[0043] Step S104: Process the data described in step S103 to generate a one-to-one corresponding txt tag index for the noisy speech and speech keywords for subsequent training.
[0044] Step S105: Use the dual-threshold method to perform endpoint detection on the 3000 clean speech data mentioned in step S101, that is, obtain the start and end points of the speech segments, mark the speech segment signal points as 1, and mark the no speech signal segments as 0, forming a binary classification data of "either 0 or 1" with a quantity of 27000 and a length of 16000 for each segment.
[0045] The training dataset can be obtained by following the steps above, which consists of two channels: noisy speech, keyword categories, and speech segment markers.
[0046] Step S2: Construct a dual-stream network framework suitable for dual-channel speech recognition.
[0047] A two-stream network is essentially a dual-channel neural network where input data simultaneously enters two independent, parallel neural networks. After processing by the two networks, they output different feature data. Specifically, the method of this invention constructs a two-stream network consisting of a speech recognition stream and an endpoint detection stream.
[0048] Speech recognition stream is a SincNet network that takes noisy speech data as input and outputs keyword categories. It establishes a mapping relationship between noisy speech and keywords, and is a multi-classification task. The network integrates endpoint information from endpoint detection stream at the beginning, which can effectively combine speech endpoint information for the next step of recognition.
[0049] Endpoint detection stream is an ASPP (Atrous Spatial Pyramid Pooling) network that takes noisy speech data as input and outputs endpoint detection data. It establishes a mapping relationship between noisy speech and speech endpoints. The network merges the detected speech endpoint information with the input data into the beginning of the speech recognition stream, which can help the speech recognition stream network quickly determine the start and end positions of speech segments and perform effective recognition, thereby improving the accuracy of speech recognition and the ability to distinguish keywords.
[0050] Two-stream network model structure as follows Figure 5 As shown in the diagram. Here, SincNet Filters represents filters, Pooling represents pooling, Layer Norm represents layer normalization, Leaky ReLU represents activation functions, Dropout represents node dropping, CNN / DNN Layers represents deep convolutional layers, Softmax represents classification functions, Conv represents ordinary convolutional layers, and rate is the dilation factor.
[0051] Specifically, the speech data obtained in step S1 first enters the speech recognition stream. The speech recognition stream consists of three ASPP layers and two convolutional layers. The speech data enters the first ASPP layer, which is composed of five different parallel processing modules: ordinary convolution, dilation factors rate=6, rate=12, rate=18, and average pooling layers. ASPP layers are commonly used to extract multi-directional semantic information from speech. After passing through three ASPP modules, the network can extract relatively accurate speech endpoint information. After the speech data passes through the first ordinary convolution layer, it is concatenated with the input data as key information and fed into the input of the speech recognition stream network. This provides the speech recognition stream with a clearer recognition location for effective recognition, thus improving the accuracy of keyword recognition in the speech recognition stream. After the final convolution layer, the speech data finally outputs endpoint information.
[0052] Input data incorporating endpoint information enters the speech recognition stream. The speech recognition stream uses a SincNet structure. The input speech data, incorporating endpoint information, enters SincNet Filters, which are interpretable and trainable bandpass filters with good interpretability in the frequency domain. The data then passes through pooling layers, batch normalization, Leaky ReLU activation, node dropping, and convolutional layers before entering the final Softmax classification layer, ultimately outputting a keyword category with high accuracy.
[0053] Step S3: Train the two-stream network model and establish the mapping relationship between noisy speech, keyword categories, and speech endpoint information. First, divide the dataset of 27,000 data sets into a training set of 21,600 sets and a validation set of 5,400 sets, with 80% for training and 20% for validation. The loss function for the speech recognition stream is cross-entropy, and the loss function for the endpoint detection stream is binary cross-entropy. The total network loss function is the sum of these two loss functions. Figure 6 The diagram shown illustrates the principle of training.
[0054] The initial learning rate was 0.001, the optimizer was an adaptive moment estimator, the minimum training batch size was 8, and the number of training epochs was 50. The model development platform used Python 3.8 and Keras 2.7.0. The main parameters of the computer used for network model training and experimental testing are as follows: NVIDIA GeForce RTX 3060 GPU, Core i11-11400F CPU + 64GB RAM.
[0055] Step S4: Predict the speech category using the trained network. After training, the dual-stream network SRSEDS Net can effectively complete the keyword recognition task for noisy speech. The noisy speech to be tested enters the endpoint detection stream input of the dual-stream network. Its ASPP structure can accurately identify the location of speech segments. This information is incorporated into the input data and enters the speech recognition stream SincNet network. Compared with the input data, this information has richer semantic information and more obvious features, which can help the speech recognition stream output more accurate keyword speech categories.
[0056] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for keyword spotting based on a two-stream network, characterized in that, Includes the following steps: Step S1: Process the original keyword speech data to construct a dual-channel dataset for the dual-stream network model; Step S2: Construct a two-stream network model for dual-channel speech recognition; Step S3: Train the two-stream network model and establish the mapping relationship between noisy speech and keyword categories and speech endpoint information; Step S4: Predict keyword categories using the trained two-stream network model.
2. The keyword spotting method based on dual-stream networks according to claim 1, wherein, Step S1 includes the following steps: Step S101: The original keyword speech data is selected from the speech command dataset. The dataset is divided into training set, validation set and test set, and 3000 clean speech data in the dataset are selected as the speech data required for training. Step S102: Add noise to the 3000 clean speech data in step S101. The noise sources are three types of noise from the GSCD library: pink, white, and exercise_bike. Set the signal-to-noise ratio level to 0dB, 5dB, and 10dB for each type of noise to add noise to the speech signal. Step S103: The original 3,000 clean speech data are used as speech keyword category data, and 27,000 sets of speech categories-noisy speech datasets are formed with the noisy speech data. The speech category-noisy speech datasets are then normalized. Step S104: Process the speech category - noisy speech dataset, and generate a one-to-one corresponding txt tag index for the noisy speech and speech keywords as the training set; Step S105: Use the dual-threshold method to perform endpoint detection on the 3000 clean speech data to obtain the start and end points of the speech segments. Mark the signal points of the speech segments as 1 and the segments without speech signals as 0, forming a binary classification data with a quantity of 27000 and a length of 16000 for each segment.
3. The keyword recognition method based on a two-stream network according to claim 1, characterized in that, The dual-stream network is a dual-channel neural network, with the dual channels being a speech recognition stream and an endpoint detection stream, and the speech data outputs different feature data through the dual channels.
4. The keyword recognition method based on a two-stream network according to claim 3, characterized in that, Speech recognition stream is a SincNet network for a multi-classification task. The input is noisy speech data and the output is keyword categories. It establishes a mapping relationship between noisy speech and keyword categories, and is used to fuse endpoint information from endpoint detection stream at the starting end.
5. The keyword recognition method based on a two-stream network according to claim 3, characterized in that, Endpoint detection stream is an ASPP network that takes noisy speech data as input and outputs endpoint detection data. It establishes a mapping relationship between noisy speech and speech endpoints, and is used to feed the detected speech endpoint information into the beginning of the speech recognition stream to help the speech recognition stream network determine the start and end positions of speech segments and perform effective recognition.
6. The keyword recognition method based on a two-stream network according to claim 3, characterized in that, The speech recognition stream consists of three ASPP layers and two convolutional layers. The ASPP layers are used to extract multi-directional semantic information of the speech, and the convolutional layers are used to provide the speech recognition stream with a more specific recognition location to improve the accuracy of the speech recognition stream in recognizing keywords.
7. The keyword recognition method based on a two-stream network according to claim 1, characterized in that, In step S3, the loss function of the speech recognition stream in the training network is cross-entropy, the loss function of the endpoint detection stream is binary cross-entropy, and the total network loss function is the sum of the two loss functions.
8. A keyword recognition system based on a two-stream network, characterized in that, It includes a dataset module, a two-stream network module, a training module, and a prediction module, among which: The dataset module is used to process the raw keyword speech data and construct a dual-channel dataset for the dual-stream network model; The dual-stream network module is used to build a dual-stream network model for dual-channel speech recognition; The training module is used to train the two-stream network model and establish a mapping relationship between noisy speech and keyword categories and speech endpoint information; The prediction module uses a trained two-stream network model to predict keyword categories.
9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a keyword recognition method based on a dual-stream network as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a keyword recognition method based on a two-stream network as described in any one of claims 1-7.