A speech keyword recognition system and method based on graph convolutional neural network
By introducing graph convolutional neural network and narrow channel bottleneck structure in the speech keyword recognition system, the problem of high complexity and difficulty in extracting global information is solved, and efficient speech keyword recognition is achieved, suitable for low resources equipment.
Patent Information
- Application Number
- CN202011057897.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-09-29
AI Technical Summary
The existing speech keyword recognition method based on convolutional neural networks has high network complexity and intensive calculations, making it difficult to extract global information.
The speech keyword recognition system based on graph convolution neural network is adopted, and the network complexity is compressed by introducing bottleneck structures and residual connections of narrow channels, and the global context information is modeled through graph convolution neural network module.
It significantly compresses the network complexity and computing volume, improves the accuracy of speech keyword recognition, and is suitable for low-resource device scenarios.
Smart Images

Figure CN112435652B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of lightweight, low-power speech keyword recognition methods based on neural networks, and in particular, relates to a speech keyword recognition system and method based on a graph convolutional neural network. Background Art
[0002] Keyword recognition is often used as the first step in voice interaction to determine whether the user has the intention to interact. When the user has the intention to interact, the system responds accordingly according to the user's instructions. When the user has no intention to interact, the system should always be in standby and dormant mode. The keyword recognition model is generally configured on the end side and generally runs offline to protect the user's privacy. The computing and storage resources on the end side are relatively limited, and the voice keyword recognition system often runs on the end side all the time, which has relatively strict restrictions on the size, accuracy and amount of operations required for prediction. At present, there are two problems with the keyword recognition method based on convolutional neural networks: the first is that the network complexity is still relatively high and the calculation is still relatively intensive. The second is that convolutional neural networks are difficult to extract global information. Summary of the invention
[0003] The purpose of the present invention is to provide a speech keyword recognition system and method based on a graph convolutional neural network, so as to solve the problems in the prior art that the network complexity of the keyword recognition method based on the convolutional neural network is still relatively high, the calculation is still relatively intensive, and the convolutional neural network is difficult to extract global information.
[0004] In order to achieve the above object, the present invention provides the following technical solutions:
[0005] A speech keyword recognition system based on graph convolutional neural network, comprising:
[0006] A voice data collection module is used to collect the wake-up words uttered by the operator.
[0007] A bandpass filter that can filter out noise in the wake-up word of the received voice data acquisition module.
[0008] An acoustic feature extraction module, which can receive the wake-up word collected by the voice data collection module, and extract the feature information of the wake-up word through the acoustic feature extraction module.
[0009] A neural network classifier that can classify command words based on acoustic features.
[0010] A basic network structure, which includes an initial block, several stages and a neural network classifier.
[0011] Several stages are composed of different numbers of bottelneck blocks. The number of bottelneck blocks is adjusted according to the complexity of the model. The neural network classifier includes a global pooling layer, a linear layer, and a Softmax module.
[0012] The initial block has a 3*3 convolution without bias. The bottelneck block includes three layers of convolution, the first and third layers of convolution are 1*1 convolution, and the second layer of convolution is 3*3 convolution.
[0013] Several stages are composed of different numbers of bottelneck blocks, and the number of bottelneck blocks is adjusted according to the complexity of the model.
[0014] A graph convolutional neural network module is inserted into the basic network structure. The graph convolutional neural network module can model the global context information through the graph convolutional neural network.
[0015] On the basis of the above technical solution, the present invention can also be improved as follows:
[0016] Furthermore, the graph convolutional neural network regards the convolutional feature map as a fully connected graph. Through the information propagation of the graph convolutional neural network, the output feature map encodes the global information.
[0017] The graph convolutional neural network module regards the feature graph of the convolutional neural network as a fully connected graph. Through the information propagation of graph convolution, the association between each node on the feature can be directly modeled.
[0018] Furthermore, the frequency range of the bandpass filter is 20 Hz to 4 kHz.
[0019] Furthermore, the acoustic feature extraction module can divide the speech into frames according to a frame length of 30ms and a frame shift of 10ms, and the acoustic feature extraction module can extract Fbank features of each frame of speech.
[0020] Furthermore, the process of extracting the Fbank feature of each frame of speech by the acoustic feature extraction module includes windowing, pre-emphasis, FFT and energy logarithm operation.
[0021] The Fbank feature is I∈ t represents the number of frames in time, and f represents the feature dimension in the frequency domain.
[0022] Furthermore, the 3*3 convolutional layer without bias can extract feature representation from MFCC features and convert the single-channel Fbank features into multi-channel convolutional feature maps.
[0023] Furthermore, Ratio represents the ratio of dimensionality reduction, and Ratio is less than 1. A Ratio less than 1 can compress network parameters and computational complexity.
[0024] Furthermore, the difference between several stages lies in the number of channels. The depth of the stage channel is proportional to the width of the stage channel number. The number of channels is consistent within the stage. The last bottleneck structure of each stage is used to increase the dimension.
[0025] Furthermore, the global pooling layer can transform the three-dimensional feature map extracted by convolution into a one-dimensional vector.
[0026] A speech keyword recognition method based on graph convolutional neural network, comprising:
[0027] S101, configuring a voice data collection module, and collecting the wake-up word uttered by the operator through the voice data collection module.
[0028] S102, configuring a bandpass filter that can filter out noise in the wake-up word received by the voice data acquisition module.
[0029] S103, configuring an acoustic feature extraction module, which can receive the wake-up word collected by the voice data collection module, and extract feature information of the wake-up word through the acoustic feature extraction module.
[0030] S104, configuring a basic network structure, which includes an initial block group, a plurality of stages, and a neural network classifier.
[0031] Several stages are composed of different numbers of bottelneck blocks. The number of bottelneck blocks is adjusted according to the complexity of the model. The neural network classifier includes a global pooling layer, a linear layer, and a Softmax module.
[0032] The initial block has a 3*3 convolution without bias. The bottelneck block includes three layers of convolution, the first and third layers of convolution are 1*1 convolution, and the second layer of convolution is 3*3 convolution.
[0033] Several stages are composed of different numbers of bottelneck blocks, and the number of bottelneck blocks is adjusted according to the complexity of the model.
[0034] S105, a graph convolutional neural network module, is inserted into the basic network structure. The graph convolutional neural network module can model the global context information through the graph convolutional neural network.
[0035] The present invention has the following advantages: the bottleneck network structure and residual connection are applied to the command word recognition task, so that the relatively complex convolution kernel acts on a relatively low dimension, compressing the size and amount of calculation of the model. The graph convolutional neural network is introduced to model the global context information. The graph convolutional neural network regards the convolutional feature map as a fully connected graph, and the output feature map encodes the global information through the information propagation of the graph convolutional neural network.
[0036] The bottleneck structure of narrow channels and the residual connection method significantly compress the complexity of the network while maintaining comparable accuracy, achieving efficient network calculations and making it more suitable for applications in low-resource device scenarios. The graph convolutional network is introduced to model the global context of the convolutional feature graph, improving the accuracy of speech keyword recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0038] Figure 1 The present invention is a flowchart of a method for speech keyword recognition.
[0039] Figure 2 This is a diagram of the working principle of the voice keyword recognition system.
[0040] Figure 3 This is the basic network structure diagram of the speech keyword recognition system.
[0041] Figure 4 This is the information propagation flow chart of graph convolutional network in speech keyword recognition system.
[0042] Figure 5 Schematic diagram of the location where the graph convolution module is inserted into the basic network in the speech keyword recognition system.
[0043] Figure 6 This is a diagram showing the comparison data between the speech keyword recognition decoder and the original decoder.
[0044] Description of symbols
[0045] Speech data acquisition module 10, bandpass filter 20, acoustic feature extraction module 30, neural network classifier 40, global pooling layer 401, linear layer 402, Softmax module 403, basic network structure 50, initial block group 501, first module 502, second module 503, third module 504, graph convolutional neural network module 60. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] like Figure 1-6 As shown, the embodiment of the present invention provides a speech keyword recognition system based on graph convolutional neural network, which includes: a speech data acquisition module 10, a bandpass filter 20, an acoustic feature extraction module 30, a neural network classifier 40 and a basic network structure 50.
[0048] The speech keyword recognition system based on graph convolutional neural network uses a narrow channel bottleneck structure and residual connection method to significantly compress the complexity of the network while maintaining comparable accuracy, achieving efficient network calculation and being more suitable for applications in low-resource device scenarios. The graph convolutional network is introduced to model the global context of the convolutional feature graph, which improves the accuracy of speech keyword recognition.
[0049] like Figure 1 As shown, the broken line represents the residual connection, that is, when learning the network, the feature F(X) is not learned directly, but F(X)-X, that is, the residual.
[0050] The voice data collection module 10 collects the wake-up words uttered by the operator.
[0051] The bandpass filter 20 can filter out the noise in the wake-up word received by the voice data acquisition module 10. In the data preprocessing stage, we first use the 20Hz-4kHz bandpass filter 20 to filter out the noise in the voice.
[0052] The acoustic feature extraction module 30 can receive the wake-up word collected by the voice data collection module 10, and extract the feature information of the wake-up word through the acoustic feature extraction module 30. The acoustic feature extraction module 30 extracts acoustic features of sound intensity, loudness, fundamental frequency and voicing degree in the dimensions of each frame of the wake-up word, and obtains 40-dimensional feature values.
[0053] The acoustic feature extraction module 30 divides the speech into frames according to the frame length of 30ms and the frame shift of 10ms, and then extracts the Fbank feature from each frame of speech. Assume that the extracted Fbank feature is I∈ t represents the number of frames in time, and f represents the feature dimension in the frequency domain. Because the data is a fixed-length speech of 1 second and uses 40-dimensional MFCC features, t = 101 and f = 40.
[0054] MFCC is another commonly used feature in speech. It is obtained by performing DCT (discrete cosine transform) on the Fbank feature. 40 dimensions is one of the commonly used acoustic feature dimensions and is an empirical value.
[0055] The basic network structure 50 includes an initial block group 501, several stages and a neural network classifier 40.
[0056] The bottleneck structure is the basic unit of residual connection, with ratio < 1, and the basic network is constructed. The ratio is always less than 1, which has the effect of compressing network parameters and computation. The Bottlebeck structure locks the 3*3 convolution operation in a low dimension, thereby reducing the number of parameters of the convolution kernel and the number of operations of the convolution layer, and improving the efficiency of network calculation.
[0057] Several stages are composed of different numbers of bottelneck blocks. The number of bottelneck blocks is adjusted according to the complexity of the model. The neural network classifier 40 includes a global pooling layer 401, a linear layer 402 and a Softmax module 403.
[0058] The initial block 501 has a 3*3 convolution without bias. This layer can extract feature representation from MFCC features and convert single-channel Fbank features into multi-channel convolution feature maps. In order to reduce the size of the feature map and remove redundant information, a 2*2 average pooling layer is followed by the 3*3 convolution.
[0059] The information of a single channel is relatively simple. It is also a common operation to convert it into multi-channel through a neural network with multiple convolution kernels to model the information more richly.
[0060] The bottelneck block includes three layers of convolution. The first and third layers of convolution are 1*1 convolutions. The first and third layers of convolution are used for dimensionality reduction and dimension reconstruction respectively. Dimension reduction and dimension reconstruction are achieved by controlling the number of convolution kernels. The number of kernels in the first layer of convolution is relatively small, so the feature map obtained is of reduced dimension. The number of kernels in the third layer of convolution is increased to reconstruct the dimension. Dimension reduction is to compress the size of the network and the amount of computation required for the model, and dimension reconstruction is to meet the requirements of residual links. Because the original feature map channel is C, the feature map that may be obtained after the dimensionality reduction of the first layer of convolution is C / 4. For the residual link, the number of channels must be consistent with C, so the third layer of convolution is required to reconstruct the channel dimension.
[0061] The second convolution layer is a 3*3 convolution. The second 3*3 convolution layer performs convolution on a relatively low dimension. Ratio represents the ratio of dimensionality reduction (Ratio<1). In the first module 502, the feature map itself has a relatively low dimension, so we did not choose a too small Ratio, so Ratio=0.5. For the second module 503 and the third module 504, we set the Ratio to 0.25.
[0062] Several stages are composed of different numbers of bottelneck blocks. The number of bottelneck blocks is adjusted according to the complexity of the model. The difference between different stages is the number of channels. The deeper the stage, the wider the number of channels. However, the number of channels is consistent within the stage. The last bottleneck structure of each stage is used to increase the dimension.
[0063] Neural network classifier 40 (Classification Block): It consists of a global average pooling layer 401, a linear layer 402 and a Softmax module 403. The function of the global pooling layer 401 is to convert the three-dimensional feature map extracted by convolution into a one-dimensional vector. Assuming that the size of the original feature map is h*w*c, the function of global pooling is to average the h*w points of each channel, and each channel obtains an average value, that is, a 1*c vector, and the linear layer 402 is used for classification.
[0064] A graph convolutional neural network module 60, namely a GCN module (Graph Convolutional Network module), is inserted into the basic network structure 50. The graph convolutional neural network module 60 can model global context information through a graph convolutional neural network.
[0065] The limitation of convolutional neural networks is that they can only extract local information. The speech keyword recognition system of the present invention introduces a graph convolutional neural network module 60, which regards the feature graph of the convolutional neural network as a fully connected graph. Through the information propagation of graph convolution, the association between each node on the feature can be directly modeled. Rich context information is introduced to improve the performance of speech command word recognition.
[0066] In order to evaluate the graph convolutional neural network's ability to model rich contexts and its impact on the task of voice command word recognition, we inserted the GCN module into the basic network mentioned above to improve the accuracy of voice command word recognition.
[0067] like Figure 6 As shown, the space occupied by the original recognition network is 19.9Kb, the accuracy of the original recognition network is 90.1%, and the number of multiplications predicted by the original recognition network at one time is 5.65M. The space occupied by the basic network of the present invention is 16.2Kb, the accuracy of the basic network of the present invention is 93.9%, and the number of multiplications predicted by the basic network of the present invention at one time is 1.95M. The space occupied by the basic network + graph convolution module is 27.6KB, the accuracy of the basic network + graph convolution module is 95.2%, and the number of multiplications predicted by the basic network + graph convolution module at one time is 2.55M.
[0068] The keyword recognition model is usually configured on the client side and usually runs offline to protect user privacy. The computing and storage resources on the client side are relatively limited, and the voice keyword recognition system is often always running on the client side, which has strict restrictions on the model size, accuracy, and the amount of operations required for prediction.
[0069] First, we tried a more efficient network structure in the command word task. Inspired by the optimization method of deep residual neural network for training time and computation, we applied the bottleneck network structure and residual connection to the command word recognition task, so that the more complex convolution kernel acts on a relatively low dimension, compressing the size of the model and the amount of computation.
[0070] The speech keyword recognition system of the present invention introduces a graph convolutional neural network to model global context information. The graph convolutional neural network regards the convolutional feature graph as a fully connected graph, and the output feature graph encodes global information through information propagation of the graph convolutional neural network. The graph convolutional network can model the relationship between any pair of nodes, regardless of the distance between the nodes. The speech signal is a sequential signal with context dependence, and global information can help improve the performance of speech command word recognition. By enhancing the context modeling capability of the model, the recognition effect is improved. In addition to demonstrating the rationality of the method, the speech keyword recognition system of the present invention also verifies the effectiveness of the method through experiments.
[0071] The speech keyword recognition system of the present invention designs a basic speech command word recognition basic network based on bottleneck. Then, a graph convolutional neural network module 60 is added to the basic network, and the performance of the network is further improved compared with the basic network.
[0072] A speech keyword recognition method based on graph convolutional neural network, comprising:
[0073] S101, configuring a voice data collection module to collect a wake-up word uttered by an operator.
[0074] In this step, a voice data collection module 10 is configured to collect the wake-up word uttered by the operator.
[0075] S102, configuring a bandpass filter to filter out noise in the wake-up word received by the voice data acquisition module.
[0076] In this step, a bandpass filter 20 is configured, which can filter out the noise in the wake-up word received by the voice data acquisition module 10.
[0077] S103: configuring an acoustic feature extraction module to extract feature information of the wake-up word.
[0078] In this step, an acoustic feature extraction module 30 is configured, which can receive the wake-up word collected by the voice data collection module 10 and extract feature information of the wake-up word through the acoustic feature extraction module 30.
[0079] S104: Configure a basic network structure.
[0080] In this step, a basic network structure 50 is configured, which includes an initial block group 501, several stages, and a neural network classifier 40.
[0081] Several stages are composed of different numbers of bottelneck blocks. The number of bottelneck blocks is adjusted according to the complexity of the model. The neural network classifier 40 includes a global pooling layer 401, a linear layer 402 and a Softmax module 403.
[0082] The initial block 501 has a 3*3 convolution without bias. The bottelneck block includes three layers of convolution, the first and third layers of convolution are 1*1 convolution, and the second layer of convolution is 3*3 convolution.
[0083] Several stages are composed of different numbers of bottelneck blocks, and the number of bottelneck blocks is adjusted according to the complexity of the model.
[0084] S105, the graph convolutional neural network module can model the global context information through the graph convolutional neural network.
[0085] In this step, a graph convolutional neural network module 60 is inserted into the basic network structure 50. The graph convolutional neural network module 60 can model global context information through a graph convolutional neural network.
[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A speech keyword recognition system based on graph convolutional neural network, characterized in that: include: a voice data collection module, through which the wake-up word uttered by the operator is collected; A bandpass filter capable of filtering out noise in the wake-up word received by the voice data acquisition module; an acoustic feature extraction module, which can receive the wake-up word collected by the voice data collection module and extract feature information of the wake-up word through the acoustic feature extraction module; A basic network structure, which includes an initial block group, several stages and a neural network classifier; Several stages are composed of different numbers of bottelneck blocks; the number of bottelneck blocks is adjusted according to the complexity of the model; the neural network classifier includes a global pooling layer, a linear layer and a Softmax module; The initial block group has a 3*3 convolution without bias; the bottelneck block includes three layers of convolution, the first and third layers of convolution are 1*1 convolution, and the second layer of convolution is 3*3 convolution; Several stages are composed of different numbers of bottelneck blocks, and the number of bottelneck blocks is adjusted according to the complexity of the model; A graph convolutional neural network module, which is inserted into the basic network structure; the graph convolutional neural network module can model global context information through a graph convolutional neural network; The 3*3 convolutional layer without bias can extract feature representation from MFCC features and convert single-channel Fbank features into multi-channel convolutional feature maps; Ratio represents the ratio of dimensionality reduction, and the Ratio is less than 1. The Ratio less than 1 can compress network parameters and computational complexity; The stages include: a first module, a second module and a third module, wherein the Ratio of the first module is 0.5, and the Ratio of the second module and the third module is 0.
25.
2. The speech keyword recognition system based on graph convolutional neural network as claimed in claim 1, characterized in that: The graph convolutional neural network regards the convolutional feature map as a fully connected graph, and the output feature map encodes global information through information propagation of the graph convolutional neural network; The graph convolutional neural network module regards the feature graph of the convolutional neural network as a fully connected graph, and through the information propagation of graph convolution, the association between each node on the feature can be directly modeled.
3. The speech keyword recognition system based on graph convolutional neural network as claimed in claim 1, characterized in that: The frequency range of the bandpass filter is 20HZ to 4KHZ.
4. The speech keyword recognition system based on graph convolutional neural network as claimed in claim 3, characterized in that: The acoustic feature extraction module can divide the speech into frames according to a frame length of 30ms and a frame shift of 10ms, and the acoustic feature extraction module can extract the Fbank features of each frame of speech.
5. The speech keyword recognition system based on graph convolutional neural network as claimed in claim 4, characterized in that: The process of extracting the Fbank feature of each frame of speech by the acoustic feature extraction module includes windowing, pre-emphasis, FFT and energy logarithm operation; The Fbank feature is that t represents the number of frames in time, and f represents the feature dimension in the frequency domain.
6. The speech keyword recognition system based on graph convolutional neural network as claimed in claim 1, characterized in that: The difference between the stages is the number of channels. The depth of the stage channel is proportional to the width of the stage channel. The number of channels is consistent inside the stage. The last bottleneck structure of each stage is used to increase the dimension.
7. The speech keyword recognition system based on graph convolutional neural network as claimed in claim 6, characterized in that: The global pooling layer can convert the three-dimensional feature map extracted by convolution into a one-dimensional vector.
8. A method for speech keyword recognition based on graph convolutional neural network, characterized in that: include: S101, configuring a voice data collection module to collect a wake-up word issued by an operator through the voice data collection module; S102, configuring a bandpass filter that can filter out noise in the wake-up word received by the voice data acquisition module; S103, configuring an acoustic feature extraction module, which can receive the wake-up word collected by the voice data collection module, and extract feature information of the wake-up word through the acoustic feature extraction module; S104, configuring a basic network structure, which includes an initial block group, a plurality of stages and a neural network classifier; S105, a graph convolutional neural network module, which is inserted into the basic network structure; the graph convolutional neural network module can model global context information through the graph convolutional neural network; Several stages are composed of different numbers of bottelneck blocks; the number of bottelneck blocks is adjusted according to the complexity of the model; The initial block group has a 3*3 convolution without bias; the bottelneck block includes three layers of convolution, the first and third layers of convolution are 1*1 convolution, and the second layer of convolution is 3*3 convolution; Several stages are composed of different numbers of bottelneck blocks, and the number of bottelneck blocks is adjusted according to the complexity of the model; The 3*3 convolutional layer without bias can extract feature representation from MFCC features and convert single-channel Fbank features into multi-channel convolutional feature maps; Ratio represents the ratio of dimensionality reduction, and the Ratio is less than 1. The Ratio less than 1 can compress network parameters and computational complexity; The stages include: a first module, a second module and a third module, wherein the Ratio of the first module is 0.5, and the Ratio of the second module and the third module is 0.25.
