Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a speech recognition method and device, electronic equipment and storage medium, wherein the method comprises: obtaining a microphone signal; determining a target beam of the microphone signal based on the correlation between each beam in the microphone signal; performing noise reduction on the target beam based on the noise signal in the microphone signal to obtain an enhanced target signal; and performing speech recognition on the enhanced target signal. The method, device, electronic equipment and storage medium provided by the application determine the target beam of the microphone signal based on the correlation between each beam in the microphone signal, and perform speech recognition using the noise-reduced target beam. Since the commonality and differences between each beam are fully considered during beam forming, a more accurate and reliable target beam is obtained, thereby ensuring the accuracy and reliability of the speech recognition realized based on the target beam.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, speech recognition technology does not achieve ideal accuracy in low signal-to-noise ratio scenarios, such as far-field speech recognition.

[0003] To improve speech signal processing quality and thus increase speech recognition rates in low signal-to-noise ratio scenarios, mainstream far-field speech recognition systems typically consist of a front-end speech enhancement module and a back-end speech recognition acoustic modeling model connected in series. The front-end speech enhancement module can utilize deep neural networks to implement MVDR (Minimum Variance Distortionless Response) or GSC (Generalized Sidelobe Canceller) to achieve beamforming.

[0004] However, the current methods for beamforming using deep neural networks have relatively simple network structures that cannot fully utilize the various types of information contained in the directly acquired microphone signals, resulting in poor optimization of speech recognition. Summary of the Invention

[0005] This invention provides a speech recognition method, apparatus, electronic device, and storage medium to address the shortcomings of poor speech recognition performance in low signal-to-noise ratio scenarios in the prior art.

[0006] This invention provides a speech recognition method, comprising:

[0007] Acquire microphone signal;

[0008] Based on the correlation between beams in the microphone signal, the target beam of the microphone signal is determined, and based on the noise signal in the microphone signal, the target beam is denoised to obtain an enhanced target signal.

[0009] Speech recognition is performed on the enhanced target signal.

[0010] According to a speech recognition method provided by the present invention, determining the target beam of the microphone signal based on the correlation between beams in the microphone signal includes:

[0011] The importance of each beam is determined based on the correlation between the beams.

[0012] Based on the importance of each beam, the beams are weighted and summed to obtain the target beam.

[0013] According to a speech recognition method provided by the present invention, the step of determining the noise signal includes:

[0014] Determine the direction of arrival based on the beam with the highest importance.

[0015] Based on the target beam and the direction of arrival, noise estimation is performed on the microphone signal to obtain the noise signal of the microphone signal.

[0016] According to a speech recognition method provided by the present invention, the speech recognition of the enhanced target signal includes:

[0017] Based on the noise signal, speech recognition is performed on the enhanced target signal.

[0018] According to a speech recognition method provided by the present invention, the step of determining each beam in the microphone signal includes:

[0019] The microphone signal is subjected to time-frequency transformation to obtain a multi-channel frequency domain signal;

[0020] Based on the frequency domain signals of the multi-channel signal, multiple fixed beams in different directions are generated as each beam.

[0021] According to a speech recognition method provided by the present invention, the step of determining a target beam of the microphone signal based on the correlation between beams in the microphone signal, and reducing noise in the target beam based on noise signals in the microphone signal to obtain an enhanced target signal includes:

[0022] Based on the voice enhancement module, the correlation between beams in the microphone signal is applied to determine the target beam of the microphone signal, and the noise signal in the microphone signal is applied to reduce the noise of the target beam to obtain the enhanced target signal.

[0023] The process of performing speech recognition on the enhanced target signal includes:

[0024] Based on the speech recognition module, speech recognition is performed on the enhanced target signal;

[0025] The speech enhancement module and the speech recognition module constitute an integrated model, which is trained based on sample microphone signals and the recognized text of the sample microphone signals.

[0026] According to a speech recognition method provided by the present invention, the integrated model is trained based on sample microphone signals, the recognized text of the sample microphone signals, and the target beam and / or noise signals of the sample microphone signals.

[0027] The present invention also provides a voice recognition device, comprising:

[0028] Acquisition unit, used to acquire microphone signals;

[0029] A beamforming unit is used to determine the target beam of the microphone signal based on the correlation between beams in the microphone signal, and to reduce noise in the target beam based on the noise signal in the microphone signal to obtain an enhanced target signal.

[0030] The recognition unit is used to perform speech recognition on the enhanced target signal.

[0031] The present invention also provides an electronic device, including a microphone, a memory, a processor, and a computer program stored in the memory and executable on the processor;

[0032] The microphone is used to collect microphone signals;

[0033] When the processor executes the program, it determines the target beam of the microphone signal based on the correlation between the beams in the microphone signal, performs noise reduction on the target beam based on the noise signal in the microphone signal to obtain an enhanced target signal, and performs speech recognition on the enhanced target signal.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described above.

[0036] The speech recognition method, apparatus, electronic device, and storage medium provided by this invention determine the target beam of the microphone signal based on the correlation between beams in the microphone signal, and apply the noise-reduced target beam for speech recognition. Since the beamforming process fully considers the commonalities and differences between beams, a more accurate and reliable target beam is obtained, thereby ensuring the accuracy and reliability of the speech recognition achieved based on it. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts of the speech recognition method provided by the present invention;

[0039] Figure 2 This is a schematic diagram of the generalized sidelobe canceller provided by the present invention;

[0040] Figure 3 This is the second flowchart of the speech recognition method provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the speech recognition device provided by the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] Far-field speech recognition is a typical low signal-to-noise ratio (SNR) scenario. When the target sound source is far from the microphone, the received target sound signal gradually attenuates during transmission. Combined with ambient noise and numerous interfering sound sources, this results in a low SNR for the final speech signal recorded by the microphone, leading to poor recognition results. Far-field speech recognition typically refers to a person speaking at a distance of 3 to 5 meters from the microphone. Common scenarios include conference rooms, in-vehicle environments, and smart homes.

[0045] In far-field speech recognition, microphone arrays are generally used as pickups. Microphone array technology is introduced to extract the target speech signal, thereby improving the accuracy of recognition. A microphone array consists of a group of microphones arranged in a specific geometric structure (commonly linear or circular). It performs spatiotemporal processing on the collected sound signals from different spatial directions, achieving functions such as noise suppression, reverberation removal, human voice interference suppression, sound source direction finding, sound source tracking, and array gain adjustment. This improves the quality of speech signal processing, thereby increasing the speech recognition rate in real-world environments.

[0046] Mainstream far-field speech recognition systems typically consist of a front-end speech enhancement module and a back-end speech recognition acoustic modeling model connected in series. The front-end speech enhancement module usually includes Direction of Arrival (DOA) and Beamforming (BF). DOA technology estimates the direction of the target sound source, while BF utilizes the target sound source's azimuth information to enhance the target signal and suppress interference signals. In practical applications, deep neural networks can be used to implement MVDR or GSC to realize beamforming.

[0047] However, current methods for implementing MVDR beamformers using deep neural networks still retain the prior assumptions of traditional MVDR based on digital signal processing. They rely on neural networks to learn the inverse of the spatial correlation covariance matrix and the target signal steering vector required for MVDR calculation. Furthermore, the network structure designed for GSC beamforming using deep neural networks is relatively simple, utilizing only simple fully connected layers and LSTM networks. It fails to leverage the discriminative information between different signals, thus failing to effectively improve speech recognition accuracy.

[0048] In response to the above problems, Figure 1 This is one of the flowcharts illustrating the speech recognition method provided by the present invention, such as... Figure 1 As shown, the method includes:

[0049] Step 110: Obtain the microphone signal.

[0050] Here, the microphone signal refers to the multi-channel signal acquired by the microphone array. The microphone array can be used for signal acquisition in a far-field environment or in a near-field environment. This embodiment of the invention does not specifically limit the specific acquisition method.

[0051] Step 120: Based on the correlation between the beams in the microphone signal, determine the target beam of the microphone signal; based on the noise signal in the microphone signal, reduce the noise of the target beam to obtain an enhanced target signal.

[0052] Specifically, after obtaining the microphone signal, the microphone signal can be converted into a multi-channel signal through time-frequency transformation, such as Fourier transform or short-time Fourier transform. Then, by utilizing the amplitude and phase differences between the signals in each channel of the multi-channel signal, multiple fixed beams in different directions are generated, thus obtaining each beam in the microphone signal.

[0053] After obtaining the beams in the microphone signal, beamforming can be performed to determine the target beam of the microphone signal. To maximize the use of the information contained in the microphone signal and thus improve the reliability of beamforming and subsequent speech recognition, this embodiment of the invention considers the correlation between the beams when determining the target beam. It is understood that the correlation between beams reflects their commonalities and differences. Determining the target beam based on this correlation fully utilizes the information contained in the microphone signal, ensuring the reliability and accuracy of the target beam determination.

[0054] Considering that noise interference still remains in the target beam, after determining the target beam, the noise signal obtained by noise estimation of the microphone signal can be used to denoise the target beam, thereby obtaining the denoised target beam, i.e., the enhanced target beam.

[0055] Step 130: Perform speech recognition on the enhanced target signal.

[0056] Specifically, based on the enhanced target signal obtained in steps 110 and 120, its beamforming process references the correlation between beams in the microphone signal. Based on this, speech recognition can be performed, which can improve the accuracy and reliability of speech recognition.

[0057] The method provided in this invention determines the target beam of the microphone signal based on the correlation between beams in the microphone signal, and applies the noise-reduced target beam for speech recognition. Since the beamforming process fully considers the commonalities and differences between beams, a more accurate and reliable target beam is obtained, thereby ensuring the accuracy and reliability of the speech recognition achieved based on this method.

[0058] Based on the above embodiments, step 120, determining the target beam of the microphone signal based on the correlation between the beams in the microphone signal, includes:

[0059] The importance of each beam is determined based on the correlation between the beams.

[0060] Based on the importance of each beam, the beams are weighted and summed to obtain the target beam.

[0061] Here, the correlation between each beam is calculated pairwise; that is, the correlation reflects the relationship between two beams. It can be understood that, for any beam within a beam, its importance among all beams can be determined based on its correlation with the other beams. For example, the correlations between the beam and the other beams can be summed to obtain the beam's importance; alternatively, the correlations between the beam and the other beams can be summed to obtain the total correlation value for the beam, and then normalized to obtain the importance of each beam. The calculation of correlation can be achieved through an attention mechanism. For instance, fixed beams in different directions can be input as multiple heads into a multi-head attention network to obtain the correlations between beams. Then, each beam can be used as a reference to calculate attention coefficients with other beams, thus obtaining the importance of each beam.

[0062] After obtaining the importance of each beam, the importance can be used as a weight to perform a weighted summation of each beam, thereby obtaining the beam in the direction of the target source, i.e., the target beam.

[0063] Based on any of the above embodiments, the step of determining the noise signal includes:

[0064] Determine the direction of arrival based on the beam with the highest importance.

[0065] Based on the target beam and the direction of arrival, noise estimation is performed on the microphone signal to obtain the noise signal of the microphone signal.

[0066] Specifically, after obtaining the importance of each beam, the beam with the highest importance can be selected, and the direction in which the beam is located can be determined as the direction of arrival (DOA).

[0067] After obtaining the target beam and direction of arrival (DOA), noise estimation of the microphone signal can be performed based on these two parameters. Alternatively, the target beam, DOA, and microphone signal can be fed into a pre-trained noise estimation model, or the target beam, DOA, and a multi-channel signal obtained by time-frequency variation of the microphone signal can be fed into the pre-trained noise estimation model to achieve noise estimation and thus obtain the noise signal contained in the microphone signal.

[0068] Based on any of the above embodiments, step 130 includes:

[0069] Based on the noise signal, speech recognition is performed on the enhanced target signal.

[0070] Specifically, compared to related technologies that directly perform speech recognition on the enhanced target signal, this embodiment of the invention also references a noise signal obtained by noise estimation of the microphone signal during speech recognition. By feeding the noise signal and the enhanced target signal together into the backend speech recognition model, richer information can be provided to the speech recognition model, thereby improving the final recognition performance.

[0071] Based on any of the above embodiments, the steps for determining each beam in the microphone signal include:

[0072] The microphone signal is subjected to time-frequency transformation to obtain a multi-channel frequency domain signal;

[0073] Based on the frequency domain signals of the multi-channel signal, multiple fixed beams in different directions are generated as each beam.

[0074] Specifically, beamforming techniques mainly include fixed beamforming and adaptive beamforming. Due to the mismatch between theoretical and actual noise field assumptions, fixed beamforming has limited noise suppression capabilities. Adaptive beamforming, on the other hand, can improve noise robustness by incorporating higher-order statistical information. However, it typically relies on accurate noise mask estimation and target sound source location estimation. If this information cannot be reliably obtained, adaptive beamforming methods will result in a greater loss of recognition rate than fixed beamforming.

[0075] Therefore, in order to minimize recognition rate loss while ensuring noise suppression capability, this embodiment of the invention obtains each beam through fixed beamforming. Specifically, the microphone signal can be converted into a multi-channel signal by performing time-frequency transformation, such as Fourier transform or short-time Fourier transform. Then, by utilizing the amplitude and phase differences between the channels in the multi-channel signal, multiple fixed beams in different directions are generated, thus obtaining each beam in the microphone signal. Here, the generation of multiple fixed beams in different directions can be achieved by fitting the weight parameters of the fixed beamforming filter (FBF) using a complex convolutional neural network.

[0076] Furthermore, to address the issue of relatively limited noise suppression capability of fixed beamforming, after obtaining each beam and thus determining the target beam and direction of arrival, noise estimation can be performed based on these two factors. The noise signal obtained from the estimation can then be used to reduce the noise of the target beam, thereby ensuring noise suppression capability.

[0077] While related technologies also involve solutions for achieving speech recognition in far-field environments through speech enhancement modules and speech recognition modules, these modules are typically optimized independently. This means that current speech recognition solutions have failed to establish a connection between the front-end and back-end of speech enhancement and speech recognition, resulting in a mismatch between the optimization goals of front-end speech enhancement and the recognition goals of candidate speech recognition.

[0078] For example, the design and testing conditions of conventional microphone array processing filters are not aimed at improving speech recognition accuracy. Furthermore, changes in the parameters of the front-end filters in far-field scenarios can easily lead to incompatibility with the back-end acoustic model, resulting in a decrease in recognition rate. Therefore, how to optimize the front-end algorithm and integrate the acoustic model with the final recognition rate as the goal to maximize the recognition effect has become a current challenge.

[0079] To address this issue, based on any of the above embodiments, step 120, which involves determining the target beam of the microphone signal based on the correlation between beams in the microphone signal, and reducing noise in the target beam based on the noise signal in the microphone signal to obtain an enhanced target signal, includes:

[0080] Based on the voice enhancement module, the correlation between beams in the microphone signal is applied to determine the target beam of the microphone signal, and the noise signal in the microphone signal is applied to reduce the noise of the target beam to obtain the enhanced target signal.

[0081] In step 130, performing speech recognition on the enhanced target signal includes:

[0082] Based on the speech recognition module, speech recognition is performed on the enhanced target signal;

[0083] The speech enhancement module and the speech recognition module constitute an integrated model, which is trained based on sample microphone signals and the recognized text of the sample microphone signals.

[0084] Specifically, in this embodiment of the invention, steps 120 and 130 can be executed through two modules in the integrated model: the speech enhancement module and the speech recognition module. To ensure that the optimization goal of the speech enhancement module and the recognition goal of the speech recognition module remain consistent, the integrated model can be trained using sample microphone signals as training data and the recognized text of the sample microphone signals as training labels. Thus, the consistency between the optimization goal of the speech enhancement module and the recognition goal of the speech recognition module within the integrated model is guaranteed, and the problem of decreased recognition rate due to incompatibility between the speech enhancement module and the speech recognition module in related solutions is also solved.

[0085] The training of the unibody model can include the following steps:

[0086] First, the initial speech enhancement module and initial speech recognition module are obtained to build the initial integrated model;

[0087] Subsequently, the sample microphone signal is input into the initial integrated model. The initial speech enhancement module predicts the target beam of the sample microphone signal based on the correlation between beams in the sample microphone signal, and performs noise reduction on the predicted target beam based on the noise signal in the sample microphone signal to obtain the predicted enhanced target signal. Then, the initial speech recognition module performs speech recognition based on the predicted enhanced target signal and noise information to obtain the sample recognition result of the sample microphone speech.

[0088] Next, the sample recognition results obtained based on the initial ensemble model are compared with the recognized text from the sample microphone signals. A loss function is generated based on the difference between the two, and the parameters of the initial ensemble model are iterated based on the loss function. For example, cross-entropy can be calculated as the loss function. It is understandable that the parameter iteration of the initial ensemble model here achieves joint optimization of the initial speech enhancement module and the initial speech recognition module, ensuring the consistency of the goals of speech enhancement and speech recognition.

[0089] Based on any of the above embodiments, the integrated model is trained based on sample microphone signals, the recognized text of the sample microphone signals, and the target beam and / or noise signals of the sample microphone signals.

[0090] Specifically, when training an integrated model that includes a speech enhancement module and a speech recognition module, the labels of the training samples can include not only the recognized text of the sample microphone signal, but also the target beam and / or noise signal of the sample microphone signal.

[0091] Understandably, using the target beam and / or noise signal of the sample microphone signal as a label for training can further improve the speech enhancement performance of the speech enhancement module and accelerate the training efficiency of the integrated model, while ensuring the consistency of the goals of speech enhancement and speech recognition.

[0092] To reduce the cost of manual annotation, avoid errors caused by manual annotation, and lower the cost of collecting real multi-channel microphone signals, the sample microphone signals here can be obtained by simulating and synthesizing single-channel near-field data.

[0093] Specifically, the room impulse response (RIR) of rooms of different sizes at different distances (3-5 meters) from the target sound source can be simulated first. Then, the mirror source method is used to simulate and generate RIRs containing steering vectors with different azimuth angles. Next, the single-channel near-field data is convolved with the simulated RIRs to simulate multi-microphone array information and generate multi-channel data. Finally, after obtaining the multi-channel data, interference noise is added at a signal-to-noise ratio of, for example, 0-10 dB. The interference noise includes both non-human voice noise and human voice noise, thus obtaining the final far-field multi-channel data, which is the sample microphone signal used as training data in this embodiment of the invention.

[0094] Specifically, during the training process of the integrated model, the target beam and / or noise signal predicted by the initial speech enhancement module can be compared with the target beam and / or noise signal of the sample microphone signal to generate a first loss function for parameter iteration of the initial speech enhancement module itself; in addition, the sample recognition result obtained based on the initial integrated model is compared with the recognized text of the sample microphone signal to generate a second loss function for parameter iteration of the initial integrated model as a whole.

[0095] The method provided in this invention, through multi-task learning including beamforming and speech recognition, effectively improves model training efficiency and ensures model training results while maintaining the consistency of the goals of speech enhancement and speech recognition.

[0096] Based on any of the above embodiments Figure 2 This is a schematic diagram of the generalized sidelobe canceller provided by the present invention, as shown below. Figure 2 As shown, a conventional generalized sidelobe canceller consists of a fixed beamforming (FBF), a blocking matrix (BM), and an adaptive noise canceller (ANC). The upper branch of the generalized sidelobe canceller is composed of a fixed beamforming unit that performs time-delay summation, allowing only the target beam to pass. The lower branch of the generalized sidelobe canceller consists of the blocking matrix and the adaptive noise canceller; the blocking matrix only allows noise interference to pass. The outputs of the fixed beamforming unit, the blocking matrix, and the adaptive noise canceller together form a multi-channel adaptive filtering structure.

[0097] Based on this, in this embodiment of the invention, the generalized sidelobe canceller (GSC) is fitted into a speech enhancement module, and combined with the speech recognition module to achieve speech recognition. Figure 3 This is the second flowchart of the speech recognition method provided by the present invention, as shown below. Figure 3As shown, the generalized sidelobe canceller (GSC) can include complex convolutional neural networks, attention networks, Transformer networks, and noise reduction networks.

[0098] Here, channels 1 to N represent the microphone signals from multiple channels. A short-time Fourier transform is performed on the microphone signals, and the transformed signals are input into a complex convolutional neural network (CNN) used to fit the weight parameters of a fixed beamforming filter. The CNN utilizes the amplitude and phase differences between the multi-channel microphone signals to generate D fixed beams in different directions. Figure 3 Beams 1 to N in the spectrum.

[0099] After obtaining D beams, the D beams can be used as multi-head inputs into the attention network. The attention network is used to learn the cross-correlation coefficients between different beams. That is, each beam is used as a reference to calculate the attention coefficients with other beams in different directions. The learned attention coefficients are used to perform a weighted summation on the D beams in different directions to obtain the beam where the target source direction is located, i.e., the target beam.

[0100] Furthermore, since residual noise signals still exist in the target beam, further noise reduction processing is required. Therefore, a Transformer network is first used to estimate the noise signal, and then a noise reduction network is used to further reduce the noise in the target beam. Here, the maximum value of the attention coefficients learned by the attention network can be taken, that is, by applying Argmax to the attention coefficients of each beam, the estimated direction of arrival is obtained. The direction of arrival, the target beam, and the original microphone signal are then fed into the Transformer network as input to learn the noise signal. Then, the noise signal output by the Transformer and the target beam are fed into the noise reduction network to further reduce the noise in the target beam, obtaining the final enhanced target signal, i.e., the enhanced target signal.

[0101] Finally, the enhanced target signal and the noise signal can be sent together into the speech recognition module to achieve speech recognition.

[0102] The method provided in this invention fully utilizes the discriminative information between each beam and sends both the enhanced target signal and the noise signal to the back-end speech recognition module. Compared to sending only a single enhanced target signal beam, this provides the back-end speech recognition module with richer information, thereby improving the final recognition effect. Furthermore, the introduction of complex convolutional networks, attention networks designed based on the multi-head attention concept, and Transformer networks makes the NN (Neural Network)-GSC modeling more powerful and reasonable.

[0103] Based on any of the above embodiments Figure 4 This is a schematic diagram of the structure of the voice recognition device provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0104] Acquisition unit 410 is used to acquire microphone signals;

[0105] The beamforming unit 420 is used to determine the target beam of the microphone signal based on the correlation between beams in the microphone signal, and to reduce noise in the target beam based on the noise signal in the microphone signal to obtain an enhanced target signal.

[0106] The recognition unit 430 is used to perform speech recognition on the enhanced target signal.

[0107] The apparatus provided in this invention determines the target beam of the microphone signal based on the correlation between beams in the microphone signal, and applies the noise-reduced target beam for speech recognition. Since the beamforming process fully considers the commonalities and differences between beams, a more accurate and reliable target beam is obtained, thereby ensuring the accuracy and reliability of the speech recognition achieved based on it.

[0108] Based on any of the above embodiments, the beamforming unit is specifically used for:

[0109] The importance of each beam is determined based on the correlation between the beams.

[0110] Based on the importance of each beam, the beams are weighted and summed to obtain the target beam.

[0111] Based on any of the above embodiments, the beamforming unit is further used for:

[0112] Determine the direction of arrival based on the beam with the highest importance.

[0113] Based on the target beam and the direction of arrival, noise estimation is performed on the microphone signal to obtain the noise signal of the microphone signal.

[0114] Based on any of the above embodiments, the identification unit is specifically used for:

[0115] Based on the noise signal, speech recognition is performed on the enhanced target signal.

[0116] Based on any of the above embodiments, the beamforming unit is further used for:

[0117] The microphone signal is subjected to time-frequency transformation to obtain a multi-channel frequency domain signal;

[0118] Based on the frequency domain signals of the multi-channel signal, multiple fixed beams in different directions are generated as each beam.

[0119] Based on any of the above embodiments, the beamforming unit is specifically used for:

[0120] Based on the voice enhancement module, the correlation between beams in the microphone signal is applied to determine the target beam of the microphone signal, and the noise signal in the microphone signal is applied to reduce the noise of the target beam to obtain the enhanced target signal.

[0121] The process of performing speech recognition on the enhanced target signal includes:

[0122] Based on the speech recognition module, speech recognition is performed on the enhanced target signal;

[0123] The speech enhancement module and the speech recognition module constitute an integrated model, which is trained based on sample microphone signals and the recognized text of the sample microphone signals.

[0124] Based on any of the above embodiments, the integrated model is trained based on sample microphone signals, the recognized text of the sample microphone signals, and the target beam and / or noise signals of the sample microphone signals.

[0125] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, as well as a microphone 550. The processor 510, communications interface 520, memory 530, and microphone 550 communicate with each other via the communication bus 540. The microphone 550 is used to acquire microphone signals. The processor 510 can call logical instructions in the memory 530 to execute a speech recognition method. This method includes: acquiring the microphone signal; determining the target beam of the microphone signal based on the correlation between beams in the microphone signal; reducing noise in the target beam based on noise signals in the microphone signal to obtain an enhanced target signal; and performing speech recognition on the enhanced target signal.

[0126] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech recognition method provided by the above methods. The method includes: acquiring a microphone signal; determining a target beam of the microphone signal based on the correlation between beams in the microphone signal; reducing noise in the target beam based on noise signals in the microphone signal to obtain an enhanced target signal; and performing speech recognition on the enhanced target signal.

[0128] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method provided by the above methods, the method comprising: acquiring a microphone signal; determining a target beam of the microphone signal based on the correlation between beams in the microphone signal; reducing noise in the target beam based on noise signals in the microphone signal to obtain an enhanced target signal; and performing speech recognition on the enhanced target signal.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Acquire microphone signal; The microphone signal is the multi-channel signal acquired through a microphone array. Based on the correlation between beams in the microphone signal, the target beam of the microphone signal is determined, and based on the noise signal in the microphone signal, the target beam is denoised to obtain an enhanced target signal. The noise signal is obtained by noise prediction of the microphone signal; Speech recognition is performed on the enhanced target signal; Determining the target beam of the microphone signal based on the correlation between beams in the microphone signal includes: For each beam, the importance of the beam among the other beams is determined based on the correlation between the beam and the other beams. The correlation is used to reflect the commonalities and differences among the beams. Based on the importance of each beam, the beams are weighted and summed to obtain the target beam.

2. The speech recognition method according to claim 1, characterized in that, The steps for determining the noise signal include: Determine the direction of arrival based on the beam with the highest importance. Based on the target beam and the direction of arrival, noise estimation is performed on the microphone signal to obtain the noise signal of the microphone signal.

3. The speech recognition method according to claim 1, characterized in that, The process of performing speech recognition on the enhanced target signal includes: Based on the noise signal, speech recognition is performed on the enhanced target signal.

4. The speech recognition method according to claim 1, characterized in that, The steps for determining each beam in the microphone signal include: The microphone signal is subjected to time-frequency transformation to obtain a multi-channel frequency domain signal; Based on the frequency domain signals of the multi-channel signal, multiple fixed beams in different directions are generated as each beam.

5. The speech recognition method according to any one of claims 1 to 4, characterized in that, The process of determining the target beam of the microphone signal based on the correlation between beams in the microphone signal, and performing noise reduction on the target beam based on the noise signal in the microphone signal to obtain an enhanced target signal includes: Based on the voice enhancement module, the correlation between beams in the microphone signal is applied to determine the target beam of the microphone signal, and the noise signal in the microphone signal is applied to reduce the noise of the target beam to obtain the enhanced target signal. The process of performing speech recognition on the enhanced target signal includes: Based on the speech recognition module, speech recognition is performed on the enhanced target signal; The speech enhancement module and the speech recognition module constitute an integrated model, which is trained based on sample microphone signals and the recognized text of the sample microphone signals.

6. The speech recognition method according to claim 5, characterized in that, The integrated model is trained based on sample microphone signals, the recognized text of the sample microphone signals, and the target beam and / or noise signals of the sample microphone signals.

7. A voice recognition device, characterized in that, include: Acquisition unit, used to acquire microphone signals; The microphone signal is the multi-channel signal acquired through a microphone array. A beamforming unit is used to determine the target beam of the microphone signal based on the correlation between beams in the microphone signal, and to reduce noise in the target beam based on the noise signal in the microphone signal to obtain an enhanced target signal. The noise signal is obtained by noise prediction of the microphone signal; A recognition unit is used to perform speech recognition on the enhanced target signal; The beamforming unit is specifically used for: For each beam, the importance of the beam among the other beams is determined based on the correlation between the beam and the other beams. The correlation is used to reflect the commonalities and differences among the beams. Based on the importance of each beam, the beams are weighted and summed to obtain the target beam.

8. An electronic device comprising a microphone, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The microphone is used to collect microphone signals; the microphone signals are multi-channel signals collected by a microphone array. When the processor executes the program, it determines the target beam of the microphone signal based on the correlation between the beams in the microphone signal, and performs noise reduction on the target beam based on the noise signal in the microphone signal to obtain an enhanced target signal. Speech recognition is performed on the enhanced target signal; the noise signal is obtained by noise prediction of the microphone signal; Determining the target beam of the microphone signal based on the correlation between beams in the microphone signal includes: For each beam, the importance of the beam among the other beams is determined based on the correlation between the beam and the other beams. The correlation is used to reflect the commonalities and differences among the beams. Based on the importance of each beam, the beams are weighted and summed to obtain the target beam.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech recognition method as described in any one of claims 1 to 6.

Images