Multilingual real-time speech recognition system and method based on deep learning

Abstract

The invention relates to the technical field of speech recognition, in particular to a multilingual real-time speech recognition system and method based on deep learning. The method comprises the following steps: inputting a voice signal to be recognized into a Wiener filtering model, and outputting an enhanced voice signal; processing the discrete energy spectrum by using a group of triangular band-pass filters which are uniformly distributed on frequency to obtain corresponding output, performing logarithmic operation on the output of all the filters to obtain a logarithmic power spectrum, and obtaining characteristic parameters of the voice signal based on discrete cosine transform of the logarithmic power spectrum; extracting voice signal feature vectors based on the feature parameters; and inputting the voice signal characteristic vector into a vector quantizer, outputting a code vector label formed by quantization, inputting the code vector label into the HMM voice recognition model, and outputting a recognition result. According to the invention, the accuracy of speech recognition can be improved, and the stability of real-time speech recognition service is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a multilingual real-time speech recognition system and method based on deep learning. Background Art

[0002] As an important branch of artificial intelligence, speech recognition technology aims to convert human speech into computer-readable input, such as keystrokes, binary codes or character sequences. This technology involves many disciplines such as computers, communications, speech linguistics, electronic technology and pattern recognition, and is an important part of the field of pattern recognition research.

[0003] In terms of real-time speech recognition, traditional speech recognition systems often have problems with large recognition delays and inaccurate recognition, making it difficult to meet the needs of real-time interaction. In terms of multilingual speech recognition, traditional methods often require building models for each language separately, which not only increases the complexity and cost of the system, but also limits the flexibility and scalability of the system.

[0004] Therefore, there is an urgent need for a multilingual real-time speech recognition system and method based on deep learning to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a multilingual real-time speech recognition system and method based on deep learning: to solve the technical problems in the prior art of large recognition delay, inaccurate recognition and low flexibility in multilingual real-time speech recognition.

[0006] The purpose of the present invention can be achieved by the following technical solutions: On the one hand, a multi-language real-time speech recognition system based on deep learning, the system includes a speech signal enhancement unit, a speech signal feature extraction unit, and a speech signal recognition unit; A speech signal enhancement unit, used for inputting the speech signal to be recognized into the Wiener filter model, outputting an enhanced speech signal, and sending the enhanced speech signal to the speech signal feature extraction unit; A speech signal feature extraction unit is used to receive the enhanced speech signal, and perform pre-emphasis and windowing on the enhanced speech signal, then perform discrete Fourier transform on each frame of the speech signal to obtain its spectrum, calculate the square of the spectrum amplitude to obtain a discrete energy spectrum, process the discrete energy spectrum using a group of triangular bandpass filters uniformly distributed in frequency, thereby obtaining a corresponding output, perform logarithmic operation on the outputs of all filters to obtain a logarithmic power spectrum, and obtain characteristic parameters of the speech signal based on discrete cosine transform of the logarithmic power spectrum, and extract a speech signal feature vector based on the characteristic parameters; The speech signal recognition unit is used to input the speech signal feature vector into the vector quantizer, output the code vector label formed by quantization, and input the code vector label into the HMM speech recognition model to output the recognition result.

[0007] Furthermore, the speech signal to be recognized is input into the Wiener filter model, and the enhanced speech signal is output, which specifically includes the following process: ; in, is the enhanced speech signal, is the speech signal to be recognized, is the optimal solution of the system function; The calculation process is as follows: ; in, is the mean square error between the speech signal to be recognized and the noise-free speech signal, Indicates the number of frames of the speech signal to be recognized, represents the noise-free speech signal of the i-th frame, represents the i-th frame of speech signal to be recognized; ; in, is the mean square error between the speech signal to be recognized and the noisy speech signal.

[0008] Furthermore, a set of triangular bandpass filters uniformly distributed in frequency are used to process the discrete energy spectrum, so as to obtain the corresponding output, which specifically includes the following process: ; in, is the frequency value filtered out by the triangular bandpass filter, is the frequency peak corresponding to the discrete energy spectrum, is the sampling frequency, is the inverse function of the Mel frequency, is the low frequency of the triangle band filter, is the high frequency of the triangle band filter, is the serial number of the triangular bandpass filter; Remove the filtered frequency value and get the output of the triangular bandpass filter.

[0009] Furthermore, obtaining the characteristic parameters of the speech signal based on the logarithmic power spectrum discrete cosine transform specifically includes the following process: ; in, is the characteristic parameter of the speech signal, is the logarithmic power spectrum, is the total number of triangular bandpass filters.

[0010] Furthermore, the speech signal feature vector is input into a vector quantizer, and the code vector label formed by the quantization is output, which specifically includes the following process: Every 4 speech signal feature vectors are taken as a vector to form 256 vectors. According to the nearest neighbor principle, these 256 vectors are quantized, and the code vectors formed by quantization are numbered from 0 to 127.

[0011] Furthermore, the code vector label is input into the HMM speech recognition model, and the output recognition result specifically includes the following process: Use the code vector labels as the training set: Substitute the code vector label into the HMM speech recognition model for training. The training method adopts the Baum-Welch algorithm. Finally, the speech words corresponding to the code vector label are trained into an HMM model parameter. Use the code vector labels as the test set: Based on the Viterbi algorithm, the probability of the speech word corresponding to the code vector label matching the model parameters formed by DHMM is calculated, and it is determined whether the probability is greater than the preset threshold. If so, the speech word is output and the sequence composed of the speech words is used as the recognition result.

[0012] On the other hand, a multilingual real-time speech recognition method based on deep learning includes: Input the speech signal to be recognized into the Wiener filter model and output the enhanced speech signal; The enhanced speech signal is pre-emphasized, windowed and framed, and then each frame of the speech signal is discretely Fourier transformed to obtain its spectrum, and the square of the spectrum amplitude is calculated to obtain a discrete energy spectrum. The discrete energy spectrum is processed by a set of triangular bandpass filters uniformly distributed in frequency to obtain the corresponding output, and the output of all filters is logarithmically operated to obtain the logarithmic power spectrum, and the characteristic parameters of the speech signal are obtained based on the discrete cosine transform of the logarithmic power spectrum, and the characteristic vector of the speech signal is extracted based on the characteristic parameters; The speech signal feature vector is input into the vector quantizer, the code vector label formed by quantization is output, and the code vector label is input into the HMM speech recognition model to output the recognition result.

[0013] Compared with the existing solutions, the present invention achieves the following beneficial effects: Compared with traditional speech recognition methods, deep learning models can capture more speech details and features, improving the accuracy of speech recognition; By learning the statistical laws in a large amount of corpus data, the key features in the speech are automatically extracted, thereby suppressing background noise and interference to a certain extent, which enables the system to maintain high speech recognition performance even in noisy environments.

[0014] Through efficient algorithm and model optimization, these requirements can be met and stable and reliable real-time speech recognition services can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 is a system block diagram of a multilingual real-time speech recognition system based on deep learning according to an embodiment of the present invention; Figure 2 This is a workflow diagram of a multilingual real-time speech recognition method based on deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] In addition, the described features, structures or characteristics may be combined in one or more example embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the example embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, steps, etc. may be adopted. In other cases, well-known structures, methods, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0019] This embodiment provides a multi-language real-time speech recognition system based on deep learning. Figure 1 is a system block diagram of a multilingual real-time speech recognition system based on deep learning according to an embodiment of the present invention. Figure 1 As shown, the system includes a speech signal enhancement unit, a speech signal feature extraction unit, and a speech signal recognition unit; A speech signal enhancement unit, used for inputting the speech signal to be recognized into the Wiener filter model, outputting an enhanced speech signal, and sending the enhanced speech signal to the speech signal feature extraction unit; A speech signal feature extraction unit is used to receive the enhanced speech signal, and perform pre-emphasis and windowing on the enhanced speech signal, then perform discrete Fourier transform on each frame of the speech signal to obtain its spectrum, calculate the square of the spectrum amplitude to obtain a discrete energy spectrum, process the discrete energy spectrum using a group of triangular bandpass filters uniformly distributed in frequency, thereby obtaining a corresponding output, perform logarithmic operation on the outputs of all filters to obtain a logarithmic power spectrum, and obtain characteristic parameters of the speech signal based on discrete cosine transform of the logarithmic power spectrum, and extract a speech signal feature vector based on the characteristic parameters; The speech signal recognition unit is used to input the speech signal feature vector into the vector quantizer, output the code vector label formed by quantization, and input the code vector label into the HMM speech recognition model to output the recognition result.

[0020] In summary, the present invention inputs the speech signal to be recognized into the Wiener filter model, outputs the enhanced speech signal; pre-emphasizes and window-frames the enhanced speech signal, then performs discrete Fourier transform on each frame of the speech signal to obtain its spectrum, calculates the square of the spectrum amplitude to obtain a discrete energy spectrum, processes the discrete energy spectrum using a set of triangular bandpass filters uniformly distributed in frequency, thereby obtaining a corresponding output, performs logarithmic operations on the outputs of all filters, obtains the logarithmic power spectrum, and obtains the characteristic parameters of the speech signal based on the logarithmic power spectrum discrete cosine transform, extracts the speech signal feature vector based on the feature parameters; inputs the speech signal feature vector into a vector quantizer, outputs the code vector label formed by quantization, and inputs the code vector label into the HMM speech recognition model to output the recognition result. The present invention can improve the accuracy of speech recognition and the stability of real-time speech recognition services.

[0021] In some embodiments, inputting the speech signal to be recognized into the Wiener filter model and outputting the enhanced speech signal specifically includes the following process: ; in, is the enhanced speech signal, is the speech signal to be recognized, is the optimal solution of the system function; The calculation process is as follows: ; in, is the mean square error between the speech signal to be recognized and the noise-free speech signal, Indicates the number of frames of the speech signal to be recognized, represents the noise-free speech signal of the i-th frame, represents the i-th frame of speech signal to be recognized; ; in, is the mean square error between the speech signal to be recognized and the noisy speech signal.

[0022] In some embodiments, the discrete energy spectrum is processed using a set of triangular bandpass filters uniformly distributed in frequency to obtain the corresponding output, which specifically includes the following process: ; in, is the frequency value filtered out by the triangular bandpass filter, is the frequency peak corresponding to the discrete energy spectrum, is the sampling frequency, is the inverse function of the Mel frequency, is the low frequency of the triangle band filter, is the high frequency of the triangle band filter, is the serial number of the triangular bandpass filter; Remove the filtered frequency value and get the output of the triangular bandpass filter.

[0023] In some embodiments, obtaining the characteristic parameters of the speech signal based on the logarithmic power spectrum discrete cosine transform specifically includes the following process: ; in, is the characteristic parameter of the speech signal, is the logarithmic power spectrum, is the total number of triangular bandpass filters.

[0024] In some embodiments, inputting the speech signal feature vector into a vector quantizer and outputting the code vector number formed by quantization specifically includes the following process: Every 4 speech signal feature vectors are taken as a vector to form 256 vectors. According to the nearest neighbor principle, these 256 vectors are quantized, and the code vectors formed by quantization are numbered from 0 to 127.

[0025] Inputting the code vector label into the HMM (Hidden Markov Model) speech recognition model and outputting the recognition result specifically includes the following process: Use the code vector labels as the training set: Substitute the code vector label into the HMM speech recognition model for training. The training method adopts the Baum-Welch algorithm. Finally, the speech words corresponding to the code vector label are trained into an HMM model parameter. Specifically, initialize: Initialize the parameters of HMM randomly or based on domain knowledge, including the state transition probability matrix A, the observation probability matrix B and the initial state probability vector C.

[0026] E step (expectation step): Computes the forward and backward probabilities for each time step of a given observation sequence, using the current model parameters.

[0027] Based on these probabilities, the posterior probability of each state at each time step is calculated, that is, the posterior probability distribution of the hidden state.

[0028] M-step (maximization step): Using the posterior probabilities calculated in step E, re-estimate the parameters of the model.

[0029] The model parameters are updated by maximizing the log-likelihood function to obtain the new state transition probability matrix A, observation probability matrix B and initial state probability vector C.

[0030] Iteration: Repeat the E and M steps until the model parameters converge or the predetermined number of iterations is reached. Convergence is usually determined by checking the changes in model parameters. When the parameter changes are less than a preset threshold, the model can be considered to have converged.

[0031] 3. Algorithm Implementation Details Avoid numerical underflow: When calculating the forward probability and backward probability, numerical underflow may occur due to the small probability value. To avoid this, the log probability and log likelihood functions can be used for calculation.

[0032] Smoothing Techniques: When calculating the observation probability matrix B, it may happen that the probability of a state generating a certain observation value at a certain point in time is zero. To avoid this, smoothing techniques such as additive smoothing or multiplicative smoothing can be used.

[0033] Run the algorithm multiple times: The Baum-Welch algorithm may converge to a local optimal solution rather than a global optimal solution. To avoid this, it is usually necessary to run the algorithm multiple times and select the optimal model parameters.

[0034] Use the code vector labels as the test set: Based on the Viterbi algorithm, the probability of the speech word corresponding to the code vector label matching the model parameters formed by the HMM is calculated, and it is determined whether the probability is greater than the preset threshold. If so, the speech word is output and the sequence composed of the speech words is used as the recognition result.

[0035] Specifically, the following are the steps for calculating the probability of the phonetic word corresponding to the code vector label matching the model parameters formed by the HMM based on the Viterbi algorithm: Preprocessing: First, these feature vectors will be input into the HMM as observation sequences.

[0036] Apply the Viterbi algorithm: For each HMM of a speech word, a dynamic programming table (usually called a Viterbi table) is initialized to store the probability of the optimal path to each state.

[0037] Starting from the initial state, the probability values ​​in the Viterbi table are gradually updated according to the observation sequence and HMM parameters. For each state, the probability of transitioning from all possible previous states is calculated, and the path with the highest probability is selected.

[0038] When the entire observation sequence is processed, the last state in the Viterbi table will contain the probability of the best path to reach that state.

[0039] The optimal path probability is recorded as the probability of HMM model parameter matching.

[0040] In some embodiments, Figure 2 is a flowchart of a multilingual real-time speech recognition method based on deep learning according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps: Step 1: input the speech signal to be recognized into the Wiener filter model and output the enhanced speech signal; Step 2: pre-emphasize and window the enhanced speech signal, then perform discrete Fourier transform on each frame of the speech signal to obtain its spectrum, and calculate the square of the spectrum amplitude to obtain a discrete energy spectrum; Step 3, using a set of triangular bandpass filters evenly distributed in frequency to process the discrete energy spectrum, thereby obtaining the corresponding output; Step 4: Perform logarithmic operation on the outputs of all filters to obtain the logarithmic power spectrum, obtain the characteristic parameters of the speech signal based on the discrete cosine transform of the logarithmic power spectrum, and extract the speech signal feature vector based on the characteristic parameters; Step 5: Input the speech signal feature vector into the vector quantizer, output the code vector label formed by quantization, and input the code vector label into the HMM speech recognition model to output the recognition result.

[0041] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0042] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0043] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0044] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only some logical function divisions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0045] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0046] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multilingual real-time speech recognition system based on deep learning, characterized by: The system includes a speech signal enhancement unit, a speech signal feature extraction unit, and a speech signal recognition unit; A speech signal enhancement unit, used for inputting the speech signal to be recognized into the Wiener filter model, outputting an enhanced speech signal, and sending the enhanced speech signal to the speech signal feature extraction unit; A speech signal feature extraction unit is used to receive the enhanced speech signal, and perform pre-emphasis and windowing on the enhanced speech signal, then perform discrete Fourier transform on each frame of the speech signal to obtain its spectrum, calculate the square of the spectrum amplitude to obtain a discrete energy spectrum, process the discrete energy spectrum using a group of triangular bandpass filters uniformly distributed in frequency, thereby obtaining a corresponding output, perform logarithmic operation on the outputs of all filters to obtain a logarithmic power spectrum, and obtain characteristic parameters of the speech signal based on discrete cosine transform of the logarithmic power spectrum, and extract a speech signal feature vector based on the characteristic parameters; The speech signal recognition unit is used to input the speech signal feature vector into the vector quantizer, output the code vector label formed by quantization, and input the code vector label into the HMM speech recognition model to output the recognition result.

2. The multilingual real-time speech recognition system based on deep learning according to claim 1, characterized in that: The speech signal to be recognized is input into the Wiener filter model, and the enhanced speech signal is output. The process includes: ； in, is the enhanced speech signal, is the speech signal to be recognized, is the optimal solution of the system function; The calculation process is as follows: ； in, is the mean square error between the speech signal to be recognized and the noise-free speech signal, Indicates the number of frames of the speech signal to be recognized, represents the noise-free speech signal of the i-th frame, represents the i-th frame of speech signal to be recognized; ； in, is the mean square error between the speech signal to be recognized and the noisy speech signal.

3. The multilingual real-time speech recognition system based on deep learning according to claim 1, characterized in that: The discrete energy spectrum is processed by a set of triangular bandpass filters uniformly distributed in frequency to obtain the corresponding output, which specifically includes the following process: ； in, is the frequency value filtered out by the triangular bandpass filter, is the frequency peak corresponding to the discrete energy spectrum, is the sampling frequency, is the inverse function of the Mel frequency, is the low frequency of the triangle band filter, is the high frequency of the triangle band filter, is the serial number of the triangular bandpass filter; Remove the filtered frequency value and get the output of the triangular bandpass filter.

4. The multilingual real-time speech recognition system based on deep learning according to claim 1, characterized in that: The characteristic parameters of speech signals are obtained based on logarithmic power spectrum discrete cosine transform. The process includes: ； in, is the characteristic parameter of the speech signal, is the logarithmic power spectrum, is the total number of triangular bandpass filters.

5. The multilingual real-time speech recognition system based on deep learning according to claim 1, characterized in that: The speech signal feature vector is input into the vector quantizer, and the code vector label formed by the output quantization specifically includes the following process: Every 4 speech signal feature vectors are taken as a vector to form 256 vectors. According to the nearest neighbor principle, these 256 vectors are quantized, and the code vectors formed by quantization are numbered from 0 to 127.

6. The multilingual real-time speech recognition system based on deep learning according to claim 1, characterized in that: Input the code vector number into the HMM speech recognition model and output the recognition result. The process includes: Use the code vector labels as the training set: Substitute the code vector label into the HMM speech recognition model for training. The training method adopts the Baum-Welch algorithm. Finally, the speech words corresponding to the code vector label are trained into an HMM model parameter. Use the code vector labels as the test set: Based on the Viterbi algorithm, the probability of the speech word corresponding to the code vector label matching the model parameters formed by DHMM is calculated, and it is determined whether the probability is greater than the preset threshold. If so, the speech word is output and the sequence composed of the speech words is used as the recognition result.

7. A multilingual real-time speech recognition method based on deep learning, characterized in that: The multilingual real-time speech recognition system based on deep learning according to any one of claims 1 to 6, the method comprising: Input the speech signal to be recognized into the Wiener filter model and output the enhanced speech signal; The enhanced speech signal is pre-emphasized, windowed and framed, and then each frame of the speech signal is discretely Fourier transformed to obtain its spectrum, and the square of the spectrum amplitude is calculated to obtain a discrete energy spectrum. The discrete energy spectrum is processed by a set of triangular bandpass filters uniformly distributed in frequency to obtain the corresponding output, and the output of all filters is logarithmically operated to obtain the logarithmic power spectrum, and the characteristic parameters of the speech signal are obtained based on the discrete cosine transform of the logarithmic power spectrum, and the characteristic vector of the speech signal is extracted based on the characteristic parameters; The speech signal feature vector is input into the vector quantizer, the code vector label formed by quantization is output, and the code vector label is input into the HMM speech recognition model to output the recognition result.

Images