Speech recognition system based on deep learning
By adopting a variety of feature extraction methods, limited weight sharing strategies, deep learning output layer and hidden Markov model decoder in the speech recognition system, combined with backpropagation algorithm and pre-training technology, the problem of large amount of calculation and difficulty in improving the accuracy of speech recognition systems in the existing technology is solved, and the speech recognition effect with high accuracy and low calculation is achieved.
Patent Information
- Application Number
- CN202411979166.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-04-15
AI Technical Summary
The existing deep learning-based speech recognition system has a large amount of computing and is difficult to improve the accuracy rate in complex environments, and cannot effectively integrate speech features and attributes.
By combining various ways to extract spectrum features, local features are extracted using convolutional layers and pooled layers to reduce dimensionality and noise immunity, and a limited weight sharing strategy is adopted to reduce computational complexity. Combining the deep learning output layer and the hidden Markov model decoder, the text output is generated using the Softmax function and the Viterbi algorithm, and the parameters are optimized through backpropagation algorithm and pre-training techniques.
It realizes speech recognition with high accuracy and low computational volume, effectively integrates speech characteristics and attributes, and overcomes the problem that the existing technology has a large amount of computation and difficulty in improving accuracy in complex environments.
Smart Images
Figure CN119832903A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and more specifically, to a speech recognition system based on deep learning. Background Art
[0002] With the development of speech recognition technology, early recognition methods were mainly based on hidden Markov models (HMMs), which modeled speech signals in short-term static mode, converted them into time series features, and combined them with Gaussian mixture models (GMMs) for probability calculation. However, the traditional HMM-GMM method has poor robustness in noisy environments and limited modeling capabilities for complex data. In addition, these methods rely on shallow neural networks (such as single-layer perceptrons or simple feedforward networks) and cannot effectively extract high-dimensional and nonlinear features, making it difficult to improve recognition accuracy. In recent years, with the rise of deep learning, speech recognition systems based on deep neural networks (DNNs) and convolutional neural networks (CNNs) have gradually become a research hotspot. Through multi-layer nonlinear transformations, deep neural networks can extract more discriminative high-level features from large-scale data, greatly improving speech recognition performance.
[0003] In the existing public document 1 (Research on speech recognition technology based on deep learning [D]. North China University of Technology, 2024), the proposed joint gated self-attention network can effectively capture local features and global dependencies. However, when trained on large-scale data sets, because the model needs to establish long-range dependencies between more data points, the complex attention mechanism in the network will lead to excessive computation, affecting the response speed and real-time performance of the model. In the existing public document 2 (Design and Industrial Application of Speech Recognition System Based on FPGA Deep Learning [D]. University of Electronic Science and Technology of China, 2023), a speech recognition neural network model is proposed. The model uses FPGA to build a speech recognition system and conducts tests to complete the speech recognition task. However, the speech recognition algorithm has a complex network layer and a large number of parameters. Therefore, how to use the deep learning model to integrate speech features and speech attributes to build a speech recognition system with high accuracy and low computation is a technical problem that needs to be solved urgently. Summary of the invention
[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a speech recognition system based on deep learning, which extracts spectral features by combining multiple methods, extracts local features by the convolution layer, reduces the dimension and resists noise by the pooling layer, and improves the adaptability to frequency changes and reduces the computational complexity by the limited weight sharing strategy. It combines the deep learning output layer and the hidden Markov model decoder, utilizes the Softmax function and the Viterbi algorithm to accurately generate text output, and improves the accuracy of speech recognition, so as to solve the problems raised in the above-mentioned background technology.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] A speech recognition system based on deep learning comprises a feature extraction module, a convolutional neural network module, an output layer and a decoding module, and a training and optimization module; the feature extraction module extracts the spectral features of a speech signal through Mel frequency cepstral coefficients (MFCC) and logarithmic Mel filter energy coefficients (MFSC), and captures the dynamic changes of speech in combination with first-order and second-order differences (delta and delta-delta); the convolutional neural network module extracts the local features of the speech signal through a convolution layer and a pooling layer, and adopts a limited weight sharing (LWS) strategy so that each frequency subband uses an independent convolution kernel for feature extraction; the output layer and the decoding module combine a convolutional neural network and a hidden Markov model (HMM), and map the high-dimensional features of the convolutional neural network to the probability distribution of phonemes or words through a Softmax activation function, and generate speech text output according to the probability distribution and the transition probability of the hidden Markov model using a Viterbi algorithm;
[0007] The limited weight sharing strategy includes the following contents: Assume that the frequency is divided into S sub-bands, where the convolution kernel w of the s-th sub-band s (m, f) only convolves the spectral features within the frequency range belonging to the s-th subband, so the convolution operation with limited weight sharing is expressed as:
[0008]
[0009] Where: B s is the frequency index set of the s-th subband, W s (m,f) is the weight of the convolution kernel of the sth frequency subband in the mth channel and fth frequency band, b s (m) is the bias term of the s-th subband in the m-th channel, Y s (m, t) is the value of the feature map output at the mth channel and sth subband at the tth moment after the convolution operation.
[0010] As a further solution of the present invention, the feature extraction module extracts the spectral features of the speech signal through Mel frequency cepstral coefficients (MFCC) and logarithmic Mel filter energy coefficients (MFSC), and combines the first-order and second-order differences (delta and delta-delta) to capture the dynamic changes of the speech, including the following specific contents: the feature extraction module uses Mel frequency cepstral coefficients and logarithmic Mel filter energy coefficients as the main feature extraction methods, and combines the first-order and second-order differences to describe the dynamic changes of the speech signal. The feature extraction module first preprocesses the input speech signal, and the preprocessing includes: energy-based endpoint detection, identifying valid speech segments from the original speech signal, removing the front and back silent or noise parts, and framing the speech signal. The energy of each frame is calculated by the following formula: Where E(n) is the energy of the nth frame, x(ni) represents the sampling value of the signal in the nth frame, M is the frame length, and i is the index of the weighted sum of the first M sampling points. In a noisy environment, spectral subtraction is used to suppress noise by estimating the spectrum of the background noise and subtracting the noise part from the spectrum of the input signal. The energy-based silence removal method is used to remove the silence part.
[0011] The spectral analysis of the speech signal is performed through the Mel-Frequency Filter Bank to obtain the spectral information of each frame. The spectral information is discretized on the Mel scale so that the output of the filter bank can be closer to the perceptual characteristics of human hearing. The features of each frame of speech signal are composed of the output of the Mel-Frequency Filter Bank, which contains multiple filters and is configured as 40 Mel-frequency bands, which can effectively represent the spectral features of speech. The feature extraction module further calculates the logarithmic Mel filter energy coefficient (MFSC) of each frame of speech signal, that is, the energy value output by the Mel-Frequency Filter Bank is taken logarithmically to enhance the resolution of the high-frequency part while reducing the weight of the low-frequency part; the advantage of this process is that it can retain more spectral information without using the traditional discrete cosine transform (DCT), avoiding the local information loss that may be introduced by DCT.
[0012] The feature extraction module also introduces first-order difference and second-order difference features. The difference features help the feature extraction module capture dynamic changes in speech by calculating the rate of change between adjacent frames.
[0013] As a further solution of the present invention, the convolutional neural network module extracts local features of the speech signal through the convolution layer and the pooling layer, and adopts the limited weight sharing (LWS) strategy so that each frequency subband uses an independent convolution kernel for feature extraction, including the following specific contents: In the convolutional neural network, the task of the convolution layer is to perform local scanning of the input feature map through the convolution operation, and extract the local features in the frequency and time domains. In each convolution layer, the input feature map is processed by multiple convolution kernels, and the convolution kernel slides in the frequency and time domains, thereby performing weighted summation on the local information of the input signal and extracting the corresponding features.
[0014] In the convolutional neural network, the pooling layer is after the convolutional layer, and the pooling operation of the pooling layer is used to reduce the spatial size of the feature map, reduce the amount of calculation and improve the anti-noise ability of the network. The pooling operation adopts the maximum pooling method, and the maximum pooling method retains the most significant features by selecting the maximum value in the local area.
[0015] In the convolutional neural network, the present invention adopts a limited weight sharing strategy. Compared with the full weight sharing strategy used in the traditional convolutional neural network, the limited weight sharing strategy can improve the flexibility and expressiveness of the model when processing speech signals, especially when processing features of different frequency bands. The traditional convolutional neural network uses the same convolution kernel for all frequency bands when processing speech spectrum features. Although this full weight sharing method can simplify the network structure and reduce the amount of calculation, it may not be able to adapt to the characteristic differences of different frequency bands, especially in complex speech signals and noise environments. The traditional convolution operation calculation formula is: Among them, W(m,f) represents the weight of the convolution kernel in the mth channel and the fth frequency band, X(f,t) is the value of the input feature map at the tth time and the fth frequency band, b(m) represents the bias term in the mth channel, and Y(m,t) is the value of the output feature map, which represents the output of the mth channel at the tth time.
[0016] Under the strategy of limited weight sharing, the convolution kernel W(m,f) is no longer globally shared, but localized to each frequency band. The convolution kernel is assigned to different frequency sub-bands, and each frequency band uses an independent convolution kernel.
[0017] As a further solution of the present invention, the output layer and the decoding module combine the convolutional neural network and the hidden Markov model (HMM), and the high-dimensional features of the convolutional neural network are mapped to the probability distribution of phonemes or words through the Softmax activation function, and the Viterbi algorithm is used to generate speech text output according to the probability distribution and the transition probability of the hidden Markov model, including the following specific contents: the output layer and the decoding module This module combines the output layer of deep learning and the traditional hidden Markov model decoder to achieve efficient and accurate speech recognition. The function of the output layer is to map the high-dimensional features processed by the convolutional neural network to the probability distribution of each phoneme or word, and the decoding module is responsible for generating the final text output according to the probability distribution.
[0018] In the output layer, the convolutional neural network maps the feature maps obtained by convolution and pooling operations to a new space through several fully connected layers. The output layer uses the Softmax activation function for classification, converting the features of each audio frame into the probability distribution of different speech units (such as phonemes, syllables or words). The output of each node represents the probability of the phoneme or word appearing at the current moment. The Softmax function normalizes the exponential values of all possible categories to ensure that the sum of the probabilities of each category is 1.
[0019] In the output layer and decoding module, the hidden Markov model is combined with the output of the convolutional neural network. During the decoding process, the probability distribution of the phonemes or words output by the convolutional neural network will be combined with the prior probability and transition probability in the hidden Markov model to update the state probability at each moment. In order to select the most likely one from multiple possible state sequences, the Viterbi algorithm is used. The Viterbi algorithm is a dynamic programming algorithm that can find the most likely state sequence under a given model and observation sequence. The Viterbi algorithm gradually calculates the maximum probability path of each state and finally obtains the most likely word or phoneme sequence for the entire speech segment. The Viterbi algorithm recursively calculates the optimal path for each frame, and the calculation formula is:
[0020] δ t (i) = max j [δ t-1 (j)·A ji ·B i (O t )];
[0021] Where: t (i) represents the probability of the optimal path in state i at time t, A ji is the transition probability from state j to state i, B i (O t ) generates the observation symbol O for state i at time tt probability.
[0022] As a further solution of the present invention, the training and optimization module optimizes the parameters of the convolutional neural network through the back propagation algorithm and stochastic gradient descent (SGD), and uses the convolution restricted Boltzmann machine (CRBM) for pre-training, including the following specific contents: the training and optimization module uses the back propagation algorithm to update the weights, calculates the gradient of the loss function for each network parameter, and updates the parameters through gradient descent, thereby gradually reducing the error. In order to accelerate the training process, the training and optimization module uses a pre-training technology, which is to pre-train the convolutional neural network through a convolution restricted Boltzmann machine, and the convolution restricted Boltzmann machine extracts features from the data through unsupervised learning and initializes the weights of the convolution layer.
[0023] During the training process, the training and optimization module uses a stochastic gradient descent algorithm to perform parameter optimization. The stochastic gradient descent algorithm updates network parameters by calculating the gradient of each small batch of data, thereby avoiding the computational bottleneck during large-scale data processing.
[0024] The technical effects and advantages of a speech recognition system based on deep learning of the present invention are as follows: the present invention combines multiple methods to extract spectral features, extracts local features through a convolution layer, reduces dimension and resists noise through a pooling layer, and uses a limited weight sharing strategy to improve adaptability to frequency changes and reduce computational complexity; then, the deep learning output layer and a hidden Markov model decoder are combined, and a Softmax function and a Viterbi algorithm are used to accurately generate text output; then, a back propagation algorithm and a pre-training technique are used, weights are initialized through convolution restricted Boltzmann machine pre-training, and parameters are optimized using a stochastic gradient descent algorithm to accelerate the training process; the speech recognition system effectively integrates speech features and attributes, overcomes the problems of large computational complexity and difficulty in improving accuracy in the prior art in complex environments, and realizes speech recognition with high accuracy and low computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a structural schematic diagram of a speech recognition system based on deep learning in the present invention.
[0026] Figure 2 This is a structural diagram of the joint gated self-attention network in the prior art.
[0027] Figure 3 The present invention is a flowchart of training a speech recognition neural network model in the prior art.
[0028] Figure 4 Schematic diagram of a conventional convolutional neural network with weight sharing.
[0029] Figure 5Schematic diagram of a convolutional neural network with limited weight sharing of the present invention. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] Example 1
[0032] See also Figure 1 The structural schematic diagram shown in the figure shows that the embodiment of the present invention provides a speech recognition system based on deep learning, which includes a feature extraction module, a convolutional neural network module, an output layer and a decoding module, and a training and optimization module; the feature extraction module extracts the spectral features of the speech signal through Mel frequency cepstral coefficients (MFCC) and logarithmic Mel filter energy coefficients (MFSC), and combines the first-order and second-order differences (delta and delta-delta) to capture the dynamic changes of the speech; the convolutional neural network module extracts the local features of the speech signal through convolution layers and pooling layers, and adopts limited weight sharing (LWS) ) strategy, so that each frequency subband uses an independent convolution kernel for feature extraction; the output layer and decoding module combine the convolutional neural network and the hidden Markov model (HMM), and the high-dimensional features of the convolutional neural network are mapped to the probability distribution of phonemes or words through the Softmax activation function, and the Viterbi algorithm is used to generate speech text output according to the probability distribution and the transition probability of the hidden Markov model; the training and optimization module optimizes the parameters of the convolutional neural network through the back propagation algorithm and stochastic gradient descent (SGD), and uses the convolution restricted Boltzmann machine (CRBM) for pre-training.
[0033] In this embodiment, see Figure 2 The structure shown in the figure is a joint gated self-attention network in the prior art. The network can effectively capture local features and global dependencies. However, when trained on large-scale datasets, the complex attention mechanism in the network will lead to excessive computation, affecting the response speed and real-time performance of the model because the model needs to establish long-range dependencies between more data points. Figure 3 The flowchart shown is a speech recognition neural network model proposed in the prior art. The model uses FPGA to build a speech recognition system and conducts testing to complete the speech recognition task. However, the speech recognition algorithm has a complex network layer number and too many parameters.
[0034] Furthermore, the feature extraction module extracts the spectral features of the speech signal through Mel frequency cepstral coefficients (MFCC) and logarithmic Mel filter energy coefficients (MFSC), and combines the first-order and second-order differences (delta and delta-delta) to capture the dynamic changes of the speech, including: the feature extraction module uses Mel frequency cepstral coefficients and logarithmic Mel filter energy coefficients as the main feature extraction methods, and combines the first-order and second-order differences to describe the dynamic changes of the speech signal. The feature extraction module first preprocesses the input speech signal, and the preprocessing includes: energy-based endpoint detection, identifying valid speech segments from the original speech signal, removing the front and back silent or noisy parts, and framing the speech signal. The energy of each frame is calculated by the following formula: Where E(n) is the energy of the nth frame, x(ni) represents the sampling value of the signal in the nth frame, M is the frame length, and i is the index of the weighted sum of the first M sampling points. In a noisy environment, spectral subtraction is used to suppress noise by estimating the spectrum of the background noise and subtracting the noise part from the spectrum of the input signal. The energy-based silence removal method is used to remove the silence part.
[0035] The spectral analysis of the speech signal is performed through the Mel-Frequency Filter Bank to obtain the spectral information of each frame. The spectral information is discretized on the Mel scale so that the output of the filter bank can be closer to the perceptual characteristics of human hearing. The features of each frame of speech signal are composed of the output of the Mel-Frequency Filter Bank, which contains multiple filters and is configured as 40 Mel-frequency bands, which can effectively represent the spectral features of speech. The feature extraction module further calculates the logarithmic Mel filter energy coefficient (MFSC) of each frame of speech signal, that is, the energy value output by the Mel-Frequency Filter Bank is taken logarithmically to enhance the resolution of the high-frequency part while reducing the weight of the low-frequency part; the advantage of this process is that it can retain more spectral information without using the traditional discrete cosine transform (DCT), avoiding the local information loss that may be introduced by DCT.
[0036] The feature extraction module also introduces first-order difference and second-order difference features. The difference features help the feature extraction module capture dynamic changes in speech by calculating the rate of change between adjacent frames.
[0037] Furthermore, the convolutional neural network module extracts local features of the speech signal through convolutional layers and pooling layers, and adopts a limited weight sharing (LWS) strategy so that each frequency subband uses an independent convolution kernel for feature extraction, including: In the convolutional neural network, the task of the convolutional layer is to perform a local scan of the input feature map through a convolution operation and extract local features in the frequency and time domains. In each convolutional layer, the input feature map is processed by multiple convolution kernels, and the convolution kernels slide in the frequency and time domains, thereby performing a weighted summation on the local information of the input signal and extracting the corresponding features.
[0038] In the convolutional neural network, the pooling layer is after the convolutional layer, and the pooling operation of the pooling layer is used to reduce the spatial size of the feature map, reduce the amount of calculation and improve the anti-noise ability of the network. The pooling operation adopts the maximum pooling method, and the maximum pooling method retains the most significant features by selecting the maximum value in the local area.
[0039] In the convolutional neural network, the present invention adopts a limited weight sharing strategy, see Figure 4 As shown in the schematic diagram, the traditional convolutional neural network uses a full weight sharing strategy, which uses the same convolution kernel for all frequency bands when processing speech spectrum features. Although this full weight sharing method can simplify the network structure and reduce the amount of calculation, it may not be able to adapt to the characteristic differences of different frequency bands, especially in complex speech signals and noise environments. The traditional convolution operation calculation formula is: Among them, W(m,f) represents the weight of the convolution kernel in the mth channel and the fth frequency band, X(f,t) is the value of the input feature map at the tth time and the fth frequency band, b(m) represents the bias term in the mth channel, and Y(m,t) is the value of the output feature map, which represents the output of the mth channel at the tth time.
[0040] See also Figure 5 As shown in the schematic diagram, under the strategy of limited weight sharing, the flexibility and expressiveness of the model can be improved when processing speech signals, especially when processing features of different frequency bands. The convolution kernel W(m,f) is no longer globally shared, but localized to each frequency band. The convolution kernel is assigned to different frequency sub-bands, and each frequency band uses an independent convolution kernel. Assume that the frequency is divided into S sub-bands, where the convolution kernel W of the sth sub-band is s (m, f) only convolves the spectral features within the frequency range belonging to the sth subband. Therefore, the convolution operation with limited weight sharing is expressed as:
[0041]
[0042] Where: B sis the frequency index set of the s-th subband, W s (m,f) is the weight of the convolution kernel of the sth frequency subband in the mth channel and fth frequency band, b s (m) is the bias term of the s-th subband in the m-th channel, Y s (m, t) is the value of the feature map output at the mth channel and the sth subband at the tth moment after the convolution operation. The limited weight sharing strategy improves the adaptability to frequency changes. Different frequency bands may contain different speech information, especially in the speech signals of different speakers, the characteristics of the high-frequency and low-frequency parts are obviously different. The limited weight sharing can ensure that the feature extraction method of each frequency subband is specialized and can better capture the frequency characteristics in the speech signal. Compared with the full weight sharing strategy, the limited weight sharing can reduce unnecessary weight sharing, so that the convolution kernel can adapt to the characteristics of different frequency bands more flexibly, while avoiding the problem that local features cannot be effectively learned when using a unified convolution kernel, reducing the complexity of neural network calculations.
[0043] Furthermore, the output layer and the decoding module combine the convolutional neural network and the hidden Markov model (HMM), and the high-dimensional features of the convolutional neural network are mapped to the probability distribution of phonemes or words through the Softmax activation function, and the Viterbi algorithm is used to generate speech text output according to the probability distribution and the transition probability of the hidden Markov model, including the following specific contents: The output layer and the decoding module This module combines the output layer of deep learning and the traditional hidden Markov model decoder to achieve efficient and accurate speech recognition. The function of the output layer is to map the high-dimensional features processed by the convolutional neural network to the probability distribution of each phoneme or word, and the decoding module is responsible for generating the final text output according to the probability distribution.
[0044] In the output layer, the convolutional neural network maps the feature maps obtained by convolution and pooling operations to a new space through several fully connected layers. The output layer uses the Softmax activation function for classification, converting the features of each audio frame into the probability distribution of different speech units (such as phonemes, syllables or words). The output of each node represents the probability of the phoneme or word appearing at the current moment. The Softmax function normalizes the exponential values of all possible categories to ensure that the sum of the probabilities of each category is 1.
[0045] In the output layer and decoding module, the hidden Markov model is combined with the output of the convolutional neural network. During the decoding process, the probability distribution of the phonemes or words output by the convolutional neural network will be combined with the prior probability and transition probability in the hidden Markov model to update the state probability at each moment. In order to select the most likely one from multiple possible state sequences, the Viterbi algorithm is used. The Viterbi algorithm is a dynamic programming algorithm that can find the most likely state sequence under a given model and observation sequence. The Viterbi algorithm gradually calculates the maximum probability path of each state and finally obtains the most likely word or phoneme sequence for the entire speech segment. The Viterbi algorithm recursively calculates the optimal path for each frame, and the calculation formula is:
[0046] δ t (i) = max j [δ t-1 (j)·A ji ·B i (O t )];
[0047] Where: t (i) represents the probability of the optimal path in state i at time t, A ji is the transition probability from state j to state i, B i (O t ) generates the observation symbol O for state i at time t t probability.
[0048] Furthermore, the training and optimization module optimizes the parameters of the convolutional neural network through the back propagation algorithm and stochastic gradient descent (SGD), and uses the convolution restricted Boltzmann machine (CRBM) for pre-training, including the following specific contents: the training and optimization module uses the back propagation algorithm to update the weights, calculates the gradient of the loss function for each network parameter, and updates the parameters through gradient descent, thereby gradually reducing the error. In order to accelerate the training process, the training and optimization module uses a pre-training technology, which is to pre-train the convolutional neural network through a convolution restricted Boltzmann machine, and the convolution restricted Boltzmann machine extracts features from the data through unsupervised learning and initializes the weights of the convolution layer.
[0049] During the training process, the training and optimization module uses a stochastic gradient descent algorithm to perform parameter optimization. The stochastic gradient descent algorithm updates network parameters by calculating the gradient of each small batch of data, thereby avoiding the computational bottleneck during large-scale data processing.
[0050] The present invention combines multiple methods to extract spectral features, extracts local features through the convolution layer, reduces the dimension and resists noise through the pooling layer, and uses a limited weight sharing strategy to improve adaptability to frequency changes and reduce computational complexity; then, the deep learning output layer and the hidden Markov model decoder are combined to accurately generate text output using the Softmax function and the Viterbi algorithm; then, the back propagation algorithm and pre-training technology are used to initialize weights through convolution restricted Boltzmann machine pre-training, and the stochastic gradient descent algorithm is used to optimize parameters to accelerate the training process; the speech recognition system effectively integrates speech features and attributes, overcomes the problems of large computational complexity and difficulty in improving accuracy in complex environments in the prior art, and realizes speech recognition with high accuracy and low computational complexity.
[0051] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0052] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A speech recognition system based on deep learning, characterized in that: It includes a feature extraction module, a convolutional neural network module, an output layer and a decoding module, and a training and optimization module; the feature extraction module extracts the spectral features of the speech signal through the Mel frequency cepstral coefficient and the logarithmic Mel filter energy coefficient, and combines the first-order and second-order differences to capture the dynamic changes of the speech; the convolutional neural network module extracts the local features of the speech signal through the convolution layer and the pooling layer, and adopts a limited weight sharing strategy so that each frequency subband uses an independent convolution kernel for feature extraction; the output layer and the decoding module combine the convolutional neural network and the hidden Markov model, and map the high-dimensional features of the convolutional neural network to the probability distribution of phonemes or words through the Softmax activation function, and use the Viterbi algorithm to generate speech text output according to the probability distribution and the transition probability of the hidden Markov model; The limited weight sharing strategy includes the following contents: Assume that the frequency is divided into S sub-bands, where the convolution kernel W of the s-th sub-band s (m, f) only convolves the spectral features within the frequency range belonging to the s-th subband, so the convolution operation with limited weight sharing is expressed as: Where: B s is the frequency index set of the s-th subband, W s (m,f) is the weight of the convolution kernel of the sth frequency subband in the mth channel and fth frequency band, b s (m) is the bias term of the s-th subband in the m-th channel, Y s (m, t) is the value of the feature map output at the mth channel and sth subband at the tth moment after the convolution operation.
2. A speech recognition system based on deep learning according to claim 1, characterized in that The feature extraction module preprocesses the input speech signal. The preprocessing includes: energy-based endpoint detection, identifying valid speech segments from the original speech signal, removing the front and back silent or noisy parts, and processing the speech signal by framing. The energy of each frame is calculated by the following formula: Where E(n) is the energy of the nth frame, x(ni) represents the sampling value of the signal in the nth frame, M is the frame length, and i is the index of the weighted sum of the first M sampling points. In a noisy environment, spectral subtraction is used to suppress noise by estimating the spectrum of the background noise and subtracting the noise part from the spectrum of the input signal. The energy-based silence removal method is used to remove the silence part.
3. A speech recognition system based on deep learning according to claim 1, characterized in that The Viterbi algorithm used by the output layer and the decoding module is a dynamic programming algorithm. By gradually calculating the maximum probability path of each state, the most likely word or phoneme sequence of the entire speech segment is finally obtained. The Viterbi algorithm calculates the optimal path of each frame in a recursive manner. The calculation formula is: d t (i)=max j [d t-1 (j)·A ji ·B i (The t )]; Where: t (i) represents the probability of the optimal path in state i at time t, A ji is the transition probability from state j to state i, B i (O t ) generates the observation symbol O for state i at time t t probability.
4. A speech recognition system based on deep learning according to claim 1, characterized in that: The feature extraction module performs spectrum analysis on the speech signal through a Mel frequency filter group to obtain spectrum information of each frame, and the filter group is configured as 40 Mel frequency bands; the feature extraction module further calculates the logarithmic Mel filter energy coefficient of each frame of speech signal, that is, takes the logarithm of the energy value output by the Mel frequency filter group.
5. The deep learning-based speech recognition system according to claim 1, characterized in that: In the convolutional neural network, the convolution layer processes the input feature map through the convolution kernel, the convolution kernel slides in the frequency and time domains to extract local features, and uses weight sharing, so that each convolution kernel shares the same weight at all positions of the input signal.
6. A speech recognition system based on deep learning according to claim 1, characterized in that: In the convolutional neural network, the pooling layer is after the convolutional layer. The pooling operation is used to reduce the spatial size of the feature map. The maximum pooling method is adopted to retain the most significant features by selecting the maximum value in the local area.
7. A speech recognition system based on deep learning according to claim 1, characterized in that: In the output layer and the output layer in the decoding module, the convolutional neural network maps the feature map obtained by the convolution and pooling operations to a new space through a fully connected layer, uses the Softmax activation function for classification, and converts the features of each audio frame into the probability distribution of different speech units.
8. The deep learning-based speech recognition system according to claim 1, characterized in that: The training and optimization module uses a back-propagation algorithm to update weights, calculates the gradient of the loss function for each network parameter, and updates the parameters by gradient descent; at the same time, a pre-training technique is used to pre-train the convolutional neural network by a convolution restricted Boltzmann machine, which extracts features from data through unsupervised learning and initializes the weights of the convolutional layer.