Voice processing method and device, electronic equipment and readable storage medium

By combining multi-beam signal processing and multi-channel speech noise reduction neural network, beam formation and weighting processing are used to solve the problem of poor speech noise reduction effect in high-noise environments, improving the signal-to-noise ratio and fidelity of the speech signal, reducing the Training data and computing power requirements.

CN118918906AActive Publication Date: 2024-11-08MIDEA GRP (SHANGHAI) CO LTD +1

Patent Information

Application Number
CN202411238688.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-08
Estimated Expiration
2044-09-04

AI Technical Summary

Technical Problem

The existing technology has poor voice noise reduction effect in high noise environments, especially the beamforming method of signal processing, the sound source positioning accuracy is low, and the end-to-end voice noise reduction deep neural model requires a large amount of training data and high computing power. The noise reduction performance decreases significantly at the same time between the scene and the training data.

Method used

Combining multi-beam signal processing and multi-channel speech noise reduction neural network, through beamforming and weighting processing, the original speech signals of multiple microphone channels are obtained for beamforming and noise reduction, and the weighting value is adaptively determined by the neural network model for weighting. Achieve noise reduction and fidelity improvement of voice signals.

Benefits of technology

In high-noise environments, the accuracy and signal-to-noise ratio of speech noise reduction are improved, the training data demand is reduced, and the computing power demand is reduced, making the voice processing device more suitable for deployment on small computing power equipment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918906A_ABST
    Figure CN118918906A_ABST
Patent Text Reader

Abstract

The invention provides a voice processing method and device, electronic equipment and a readable storage medium. The voice processing method comprises the following steps: acquiring a plurality of original voice signals of a plurality of microphone channels; performing beam forming on the plurality of original voice signals in N beam directions to obtain N beam signals, N being a positive integer; performing noise reduction processing on a target original voice signal of a target microphone channel in the plurality of microphone channels and the N beam signals to obtain a first voice signal; and weighting the target original voice signal and the first voice signal to obtain a second voice signal. Therefore, voice noise reduction is performed in combination with multi-beam signal processing and the multi-channel voice noise reduction neural network, noise reduction and weighting of the voice signals are performed in combination with the original voice signals, sound source positioning is not needed, a large amount of training data does not need to be used, the accuracy of voice noise reduction in a high-noise environment is improved, and the voice noise reduction efficiency is improved. And the fidelity of the voice signal after noise reduction is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robotics, and in particular to a speech processing method and device, an electronic device and a readable storage medium. Background Art

[0002] At present, there are two major paradigms for conventional multi-channel microphone array speech noise reduction methods: one is to combine sound source localization and use the signal processing beamforming method for noise reduction; the other is to use an end-to-end speech noise reduction deep neural model for noise reduction.

[0003] However, in real-world scenarios, there are application scenarios with high noise and low signal-to-noise ratio. In high-noise environments, the above-mentioned speech noise reduction methods all have defects: for noise reduction methods that use signal processing beamforming methods, in high-noise environments, the accuracy of sound source localization is low, which will lead to poor signal processing beamforming effects; and for noise reduction methods that use end-to-end speech noise reduction deep neural models, in high-noise environments, the model requires a large amount of training data, a larger scale, and higher computing power to achieve better noise reduction results, and the noise reduction performance of the model will be significantly reduced when the application scenarios and training data are different. In this way, the speech noise reduction effect in high-noise environments is reduced. Summary of the invention

[0004] The present invention aims to at least solve the problem of poor speech noise reduction effect in a high noise environment existing in the prior art or related art.

[0005] To this end, a first aspect of the present invention is to provide a speech processing method.

[0006] A second aspect of the present invention is to provide a speech processing device.

[0007] A third aspect of the present invention is to provide an electronic device.

[0008] A fourth aspect of the present invention is to provide a readable storage medium.

[0009] A fifth aspect of the present invention is to provide a computer program product.

[0010] In view of this, according to one aspect of the present invention, a speech processing method is proposed, which includes: obtaining multiple original speech signals from multiple microphone channels; performing beamforming on the multiple original speech signals in N beam directions to obtain N beam signals, where N is a positive integer; performing noise reduction processing on the target original speech signal of the target microphone channel among the multiple microphone channels and the N beam signals to obtain a first speech signal; and performing weighted processing on the target original speech signal and the first speech signal to obtain a second speech signal.

[0011] The execution subject of the technical solution of the speech processing method provided by the present invention can be an electronic device or a speech processing device, and can also be determined according to actual use requirements, which is not specifically limited here. In order to more clearly describe the speech processing method provided by the present invention, the execution subject of the speech processing method is a speech processing device for explanation below.

[0012] Specifically, in the speech processing method provided by the present invention, the speech processing device obtains the original speech signal corresponding to each microphone channel to obtain multiple original speech signals, and then performs beamforming on the obtained multiple original speech signals in N beam directions to enhance the multiple original speech signals in N beam directions to obtain N beam signals. Wherein, N is a positive integer. Further, the speech processing device selects the target original speech signal corresponding to the target microphone channel among the multiple microphone channels, performs noise reduction processing on the above-formed N beam signals and the target original speech signal, and outputs a single-channel noise-reduced speech signal, i.e., the first speech signal. Further, the speech processing device performs weighted processing on the above-mentioned first speech signal and the target original speech signal to obtain a weighted second speech signal. In this way, on the one hand, multi-beam signal processing and multi-channel speech denoising neural network are combined for speech denoising, which does not require sound source localization or the use of a large amount of training data, thereby improving the accuracy of speech denoising in high noise environments and thus improving the signal-to-noise ratio of speech signals; on the other hand, the original speech signal is combined for denoising and weighting the speech signal, thereby ensuring the fidelity of the denoised speech signal.

[0013] The above-mentioned speech processing method according to the present invention may also have the following additional technical features:

[0014] In some technical schemes, optionally, the target original speech signal and the first speech signal are weightedly processed to obtain a second speech signal, including: using a neural network model to adaptively determine a first weight value of the target original speech signal and a second weight value of the first speech signal; and according to the first weight value and the second weight value, weighted processing is performed on the target original speech signal and the first speech signal to obtain a second speech signal.

[0015] In this technical solution, the speech processing device uses a neural network model to adaptively allocate the second weight value of the first speech signal and the first weight value of the target original speech signal. Furthermore, the speech processing device performs weighted processing on the first speech signal and the target original speech signal according to the determined second weight value and the first weight value, thereby obtaining a weighted second speech signal. In this way, the first speech signal obtained after noise reduction and the target original speech signal are adaptively weighted, achieving signal weight adaptation and ensuring the fidelity of the second speech signal obtained after weighting.

[0016] In some technical solutions, optionally, a neural network model is used to adaptively determine a first weight value of a target original speech signal and a second weight value of a first speech signal, including: using a neural network model to adaptively determine the first weight value and the second weight value according to the target original speech signal based on an attention mechanism.

[0017] In this technical solution, the neural network model can be specifically an attention module. In the process of weight adaptation of the target original voice signal and the first voice signal, the neural network model is used to adaptively determine the first weight value and the second weight value according to the target original voice signal based on the attention mechanism. In this way, the weight adaptation of the first voice signal and the target original voice signal is performed based on the situation of the target original voice signal, thereby ensuring the fidelity of the second voice signal obtained after weighting.

[0018] In some technical solutions, optionally, beamforming is performed on multiple original voice signals in N beam directions to obtain N beam signals, including: using a beamforming algorithm to beamform multiple original voice signals in N beam directions within a preset angle range to obtain N beam signals.

[0019] In this technical solution, the speech processing device uses a beamforming algorithm to perform beamforming on multiple acquired original speech signals in N beam directions within a preset angle range, so as to enhance multiple original speech signals in N beam directions within the preset angle range to obtain N beam signals. In this way, interference signals in non-beam directions can be suppressed, and speech signals in beam directions can be enhanced, thereby improving the signal-to-noise ratio.

[0020] In some technical solutions, optionally, the speech processing method is executed by a speech processing device, and the value of N is related to the preset angle range and the directivity of the beamforming algorithm; the value of N is positively correlated with the computing power of the speech processing device.

[0021] In this technical solution, the specific value of the number of beam signals, i.e., N, is related to the directivity of the beamforming algorithm and the preset angle range. Moreover, the value of N is positively correlated with the computing power of the speech processing device that executes the speech processing method. In this way, the computing power, application scenarios, and noise reduction effect can be weighed.

[0022] In some technical solutions, optionally, the directivity indicates the angle value between each two adjacent beam directions; the directivity is negatively correlated with the computing power of the speech processing device; the directivity is related to the noise reduction intensity and the noise information of the application scenario, wherein the noise information includes the noise intensity and noise distribution information of the application scenario.

[0023] In this technical solution, the directivity of the beamforming algorithm indicates the angle value between each two adjacent beam directions, which facilitates the determination of the number of beam signals based on the ratio of the preset angle range to the directivity. Furthermore, the directivity is negatively correlated with the computing power of the speech processing device. Furthermore, the directivity is also related to noise information such as the noise reduction intensity and the noise intensity and noise distribution information of the application scenario. In this way, the beamforming algorithm with the corresponding directivity can be selected by weighing the computing power, application scenario and noise reduction effect, so that the speech processing device can be deployed on electronic devices with less computing power.

[0024] According to a second aspect of the present invention, a speech processing device is proposed, which includes: an acquisition unit, used to acquire multiple original speech signals from multiple microphone channels; a processing unit, used to perform beamforming on the multiple original speech signals in N beam directions to obtain N beam signals, where N is a positive integer; the processing unit, also used to perform noise reduction processing on a target original speech signal of a target microphone channel in the multiple microphone channels and the N beam signals to obtain a first speech signal; the processing unit, also used to perform weighted processing on the target original speech signal and the first speech signal to obtain a second speech signal.

[0025] The speech processing device provided by the present invention includes an acquisition unit and a processing unit. In the process of speech noise reduction, the acquisition unit acquires the original speech signal corresponding to each microphone channel to obtain multiple original speech signals. The processing unit then performs beamforming on the multiple original speech signals acquired by the acquisition unit in N beam directions to enhance the multiple original speech signals in N beam directions to obtain N beam signals. Wherein, N is a positive integer. Further, the processing unit selects the target original speech signal corresponding to the target microphone channel among the multiple microphone channels, performs noise reduction processing on the above-formed N beam signals and the target original speech signal, and outputs a single-channel noise-reduced speech signal, i.e., a first speech signal. Further, the processing unit performs weighted processing on the above-mentioned first speech signal and the target original speech signal to obtain a weighted second speech signal. In this way, on the one hand, multi-beam signal processing and multi-channel speech denoising neural network are combined for speech denoising, which does not require sound source localization or the use of a large amount of training data, thereby improving the accuracy of speech denoising in high noise environments and thus improving the signal-to-noise ratio of speech signals; on the other hand, the original speech signal is combined for denoising and weighting the speech signal, thereby ensuring the fidelity of the denoised speech signal.

[0026] According to the third aspect of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the speech processing method in any of the above technical solutions are implemented. Therefore, the electronic device provided in the third aspect of the present invention has all the beneficial effects of the speech processing method in any of the above technical solutions in the first aspect, which will not be described in detail here.

[0027] According to a fourth aspect of the present invention, a readable storage medium is provided, on which a program or instruction is stored, and when the program or instruction is executed by a processor, a speech processing method as in any of the above technical solutions is implemented. Therefore, the readable storage medium provided in the fourth aspect of the present invention has all the beneficial effects of the speech processing method in any of the technical solutions in the first aspect, and no further description is given here.

[0028] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program, and when the computer program is executed by a processor, the steps of the speech processing method in any of the above technical solutions are implemented. Therefore, the computer program product provided in the fifth aspect of the present invention has all the beneficial effects of the speech processing method in any of the technical solutions in the first aspect, and will not be repeated here.

[0029] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0031] Figure 1 A flowchart of a speech processing method according to an embodiment of the present invention is shown;

[0032] Figure 2 The second flowchart of the speech processing method according to the embodiment of the present invention is shown;

[0033] Figure 3 The third flowchart of the speech processing method according to the embodiment of the present invention is shown;

[0034] Figure 4 A schematic diagram of a speech processing method according to an embodiment of the present invention is shown;

[0035] Figure 5 A structural block diagram of a speech processing device according to an embodiment of the present invention is shown;

[0036] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0039] Combine the following Figures 1 to 6 , the speech processing method and device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0040] In one embodiment of the present invention, Figure 1 As shown, the speech processing method may specifically include the following steps 102 to 108:

[0041] Step 102, obtaining multiple original speech signals from multiple microphone channels;

[0042] Step 104, performing beamforming on the multiple original speech signals in N beam directions to obtain N beam signals;

[0043] Step 106, performing noise reduction processing on a target original speech signal of a target microphone channel among the multiple microphone channels and N beam signals to obtain a first speech signal;

[0044] Step 108, performing weighted processing on the target original speech signal and the first speech signal to obtain a second speech signal;

[0045] Wherein, N is a positive integer.

[0046] The execution subject of the technical solution of the speech processing method provided by the present invention can be an electronic device or a speech processing device, and can also be determined according to actual use requirements, which is not specifically limited here. In order to more clearly describe the speech processing method provided by the present invention, the execution subject of the speech processing method is a speech processing device for explanation below.

[0047] Specifically, in the speech processing method provided by the present invention, the speech processing device obtains the original speech signal corresponding to each microphone channel to obtain multiple original speech signals, and then based on the beamforming algorithm, the spatial information of the multiple original speech signals obtained is used to beamform the multiple original speech signals obtained in N beam directions, so as to enhance the multiple original speech signals in N beam directions and obtain N beam signals. Wherein, N is a positive integer. Further, the speech processing device selects the target original speech signal corresponding to the target microphone channel among the multiple microphone channels, and inputs the above-formed N beam signals and the target original speech signal, a total of N+1 signals, into the multi-beam noise reduction model, so as to perform noise reduction processing on the above-formed N beam signals and the target original speech signal through the multi-beam noise reduction model, and outputs the single-channel noise-reduced speech signal, i.e., the first speech signal. Further, the speech processing device performs weighted processing on the above-mentioned first speech signal and the target original speech signal to obtain a weighted second speech signal. In this way, on the one hand, multi-beam signal processing and multi-channel speech denoising neural network are combined for speech denoising, which does not require sound source localization or the use of a large amount of training data, thereby improving the accuracy of speech denoising in high noise environments and thus improving the signal-to-noise ratio of speech signals; on the other hand, the original speech signal is combined for denoising and weighting the speech signal, thereby ensuring the fidelity of the denoised speech signal.

[0048] The target microphone channel is any one of the multiple microphone channels, and the target original voice signal is the original voice signal corresponding to the target microphone channel.

[0049] Furthermore, for the specific value of the number of the above-mentioned beam signals, that is, N, those skilled in the art can set it according to actual conditions, and no specific limitation is made here.

[0050] Furthermore, for the specific pointing directions of the above-mentioned N beam directions, those skilled in the art may set them according to actual conditions, and no specific limitation is made here.

[0051] Furthermore, a beamforming algorithm is used to combine the original speech signals of the multi-microphone channels, suppress interference signals in non-beam directions, and enhance speech signals in the beam direction.

[0052] In actual application, the above-mentioned beamforming algorithms include but are not limited to: MVDR (Minimum Variance Distortionless Response) beamforming algorithm, GSC (Generalized Sidelobe Canceller) beamforming algorithm, DSB (Delay and Sum Beamforming) algorithm, LCMV (Linearly Constrained Minimum Variance) beamforming algorithm and SBF (Superdirective BeamForming) algorithm, etc. Technical personnel in this field can select the specific type of the above-mentioned beamforming algorithm according to actual conditions, and no specific limitation is made here.

[0053] It is understandable that there are two major paradigms for conventional multi-channel microphone array speech noise reduction methods: one is to combine sound source localization and use the beamforming method of signal processing for noise reduction, and the other is to use an end-to-end speech noise reduction deep neural model for noise reduction. However, in a high-noise environment, for the noise reduction method using the beamforming method of signal processing, the accuracy of sound source localization is low, which will lead to poor signal processing beamforming effect; and for the noise reduction method using the end-to-end speech noise reduction deep neural model, the model requires a large amount of training data (hundreds or even thousands of hours of training data), a larger scale, and higher computing power to achieve better noise reduction results, and the noise reduction performance of the model will be significantly reduced when the application scenario and training data are different. In this way, the speech noise reduction effect in a high-noise environment is reduced.

[0054] Therefore, in the speech processing method provided by the present invention, the traditional speech noise reduction method based on multi-beam signal processing and the speech noise reduction method based on multi-channel speech noise reduction neural network are combined, which does not require sound source localization, nor does it require a large amount of training data to ensure the noise reduction effect. Only a few dozen hours of training data are needed, which reduces the amount of training data used and improves the accuracy of speech noise reduction in high noise environments. In addition, the computing power of BeamForming is much lower than that of the multi-channel speech noise reduction neural network, which can reduce the model scale and greatly reduce the overall computing power of the speech processing device, making it easier to deploy the speech processing device on electronic devices with smaller computing power.

[0055] In some embodiments of the present invention, optionally, Figure 2 As shown, the above step 108 may specifically include the following steps 108a and 108b:

[0056] Step 108a, using a neural network model to adaptively determine a first weight value of the target original speech signal and a second weight value of the first speech signal;

[0057] Step 108b: Perform weighted processing on the target original speech signal and the first speech signal according to the first weight value and the second weight value to obtain a second speech signal.

[0058] In this embodiment, in the process of the speech processing device performing weighted processing on the above-mentioned first speech signal and the target original speech signal to obtain the weighted second speech signal, the speech processing device uses the NN (Neural Network) model to adaptively allocate the second weight value of the above-mentioned first speech signal and the first weight value of the above-mentioned target original speech signal. Furthermore, the speech processing device performs weighted processing on the above-mentioned first speech signal and the target original speech signal according to the determined second weight value and the first weight value, so as to obtain the weighted second speech signal. In this way, the first speech signal obtained after noise reduction and the target original speech signal are adaptively weighted, so as to achieve the weight adaptation of the signal and ensure the fidelity of the second speech signal obtained after weighting.

[0059] In actual application, the above neural network model can specifically be an attention module, which is not specifically limited here.

[0060] In some embodiments of the present invention, optionally, the above step 108a may specifically include the following step 110:

[0061] Step 110, using a neural network model and based on an attention mechanism, adaptively determining a first weight value and a second weight value according to a target original speech signal.

[0062] In this embodiment, the neural network model can be specifically an attention module. In the process of the speech processing device performing weight adaptation on the target original speech signal and the first speech signal, the speech processing device uses the neural network model, based on the attention mechanism, and adaptively determines the first weight value and the second weight value according to the signal condition of the target original speech signal. In this way, the weight adaptation of the first speech signal and the target original speech signal is performed based on the signal condition of the target original speech signal, thereby ensuring the fidelity of the second speech signal obtained after weighting.

[0063] Specifically, the speech processing device uses a neural network model to learn and analyze the signal conditions of the target original speech signal, such as signal quality and signal strength, to determine the signal-to-noise ratio of the target original speech signal. For example, when the signal quality of the target original speech signal is good, the target original speech signal is determined to be a high signal-to-noise ratio signal, and when the signal quality of the target original speech signal is poor, the target original speech signal is determined to be a low signal-to-noise ratio signal. On this basis, the speech processing device uses the neural network model again to adaptively determine the first weight value and the second weight value according to the determined signal-to-noise ratio of the target original speech signal.

[0064] Among them, based on the different signal-to-noise ratios of the target original speech signal, the magnitude relationship between the second weight value and the first weight value is different. Specifically, when the neural network model determines that the above target original speech signal is a high signal-to-noise ratio signal, the neural network model adaptively assigns the above second weight value to be less than the first weight value; and when the neural network model determines that the above target original speech signal is a low signal-to-noise ratio signal, the neural network model adaptively assigns the second weight value to be greater than the first weight value. In this way, by learning and judging the signal-to-noise ratio of the target original speech signal, weight adaptation under different signal-to-noise ratios is achieved, ensuring the fidelity of the second speech signal obtained after weighting.

[0065] In some embodiments of the present invention, optionally, Figure 3 As shown, the above step 104 may specifically include the following step 104a:

[0066] Step 104a: using a beamforming algorithm, beamforming is performed on N beam directions of a plurality of original speech signals within a preset angle range to obtain N beam signals.

[0067] In this embodiment, in the process of the speech processing device performing beamforming on the acquired multiple original speech signals in N beam directions, the speech processing device uses a beamforming algorithm to perform beamforming on the acquired multiple original speech signals in N beam directions within a preset angle range according to the spatial information of the acquired multiple original speech signals, so as to enhance the multiple original speech signals in the N beam directions within the preset angle range to obtain N beam signals. In this way, interference signals in non-beam directions can be suppressed, and speech signals in beam directions can be enhanced, thereby improving the signal-to-noise ratio.

[0068] Wherein, N is a positive integer. As for the specific value of the number of the beam signals, that is, N, those skilled in the art can set it according to actual conditions, and no specific limitation is made here.

[0069] Furthermore, for the specific pointing directions of the above-mentioned N beam directions, those skilled in the art may set them according to actual conditions, and no specific limitation is made here.

[0070] Furthermore, for the specific range of the above-mentioned preset angle range, those skilled in the art can set it according to the actual situation, and no specific limitation is made here. For example, when enhancing the omnidirectional original voice signal, the above-mentioned preset angle range can be specifically an angle range of 360°, and when enhancing the original voice signal within the semicircle range directly in front of the electronic device, the above-mentioned preset angle range can be specifically an angle range of 180° directly in front of the electronic device.

[0071] In actual application, the above-mentioned beamforming algorithms include but are not limited to: MVDR beamforming algorithm, GSC beamforming algorithm, DSB algorithm, LCMV beamforming algorithm and SBF algorithm, etc. Technical personnel in this field can select the specific type of the above-mentioned beamforming algorithm according to actual conditions, and no specific limitation is made here.

[0072] In some embodiments of the present invention, optionally, the speech processing method is executed by a speech processing device, and the value of N is related to the preset angle range and the directivity of the beamforming algorithm; the value of N is positively correlated with the computing power of the speech processing device.

[0073] In this embodiment, the specific value of the number of beam signals, i.e., N, is related to the directivity of the beamforming algorithm and the preset angle range. In addition, the value of N is positively correlated with the computing power of the speech processing device that executes the speech processing method. In this way, the computing power, application scenario, and noise reduction effect can be weighed.

[0074] In some embodiments of the present invention, optionally, the directivity indicates the angle value between each two adjacent beam directions; the directivity is negatively correlated with the computing power of the speech processing device; the directivity is related to the noise reduction intensity and the noise information of the application scenario, wherein the noise information includes the noise intensity and noise distribution information of the application scenario.

[0075] In this embodiment, the directivity of the beamforming algorithm indicates the angle value between every two adjacent beam directions, so as to determine the number of beam signals according to the ratio of the preset angle range to the directivity.

[0076] In actual application, the directivity of the beamforming algorithm may be a fixed value, and the value of N may specifically be a ratio of a preset angle range to the directivity of the beamforming algorithm.

[0077] Exemplarily, when the directivity of the beamforming algorithm is 30° and the preset angle range is 360°, the specific value of N is 12. The speech processing device can specifically select 12 beam directions of 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300° and 330°, and perform beamforming on the multiple original speech signals obtained to enhance the multiple original speech signals obtained in the above 12 beam directions, thereby obtaining 12 beam signals.

[0078] Exemplarily, when the directivity of the beamforming algorithm is 90° and the preset angle range is 360°, the specific value of N is 4. The speech processing device can specifically select 4 beam directions of 0°, 90°, 180° and 270°, and perform beamforming on the multiple original speech signals obtained to enhance the multiple original speech signals obtained in the above 4 beam directions, thereby obtaining 4 beam signals.

[0079] Furthermore, directivity is negatively correlated with the computing power of the speech processing device. In actual applications, when the computing power of the speech processing device is limited and the number of beam signals needs to be reduced, a beamforming algorithm with high directivity can be selected.

[0080] Furthermore, directivity is also related to noise reduction strength and noise information such as noise intensity and noise distribution information of the application scenario. In this way, the beamforming algorithm with corresponding directivity can be selected by weighing computing power, application scenario and noise reduction effect, so that the voice processing device can be deployed on electronic devices with less computing power.

[0081] In actual application, for the key angle range where the original speech signal needs to be enhanced, the number of beam signals and the specific direction of the beam direction can be refined within the key angle range, and no specific limitation is made here.

[0082] In summary, if Figure 4 As shown, the principle architecture of the speech processing method provided by the embodiment of the present invention may specifically include a multi-microphone input module 402, a beamforming module 404, a multi-beam noise reduction model 406 and an attention module 408. Among them, the multi-microphone input module 402 is used to obtain multiple original speech signals corresponding to the multi-microphone channels, and input the multiple original speech signals into the beamforming module 404. Furthermore, the beamforming module 404 performs beamforming on the multiple original speech signals input therein in N beam directions based on the beamforming algorithm to obtain N beam signals, and inputs the N beam signals and the target original speech signal corresponding to the target microphone channel (such as the original speech signal mic1 corresponding to the first microphone channel) into the multi-beam noise reduction model 406, a total of N+1 signals.

[0083] Furthermore, the multi-beam noise reduction model 406 performs noise reduction processing on the N beam signals input therein and the target original speech signal (such as mic1), a total of N+1 signals, and outputs a single-channel first speech signal after noise reduction to the attention module 408. In this way, on the one hand, the signal-to-noise ratio of the speech signal is improved; on the other hand, by combining the target original speech signal (such as mic1) for signal noise reduction, the multi-beam noise reduction model 406 can understand the situation of the original speech signal, so that the fidelity of the first speech signal output by the multi-beam noise reduction model 406 is better, and at the same time, the phase of the multi-beam noise reduction model 406 and the target original speech signal (such as mic1) can be aligned to prepare for the subsequent work of the attention module 408.

[0084] Furthermore, the multi-microphone input module 402 can also input the target original speech signal corresponding to the target microphone channel (such as the original speech signal mic1 corresponding to the first microphone channel) into the attention module 408. On this basis, the attention module 408 uses a neural network model to adaptively weight the target original speech signal (such as mic1) and the first speech signal input therein according to the signal condition of the target original speech signal, instead of outputting the weighted second speech signal. Among them, due to the addition of the target original speech signal (such as mic1), the fidelity of the second speech signal output by the attention module 408 is better.

[0085] In one embodiment of the present invention, a speech processing device is also provided. Figure 5 As shown, Figure 5 The structure block diagram of the speech processing device 200 according to the embodiment of the present invention is shown. The speech processing device 200 may specifically include the following acquisition unit 202 and processing unit 204:

[0086] An acquisition unit 202 is used to acquire multiple original speech signals from multiple microphone channels;

[0087] The processing unit 204 is configured to perform beamforming on the multiple original speech signals in N beam directions to obtain N beam signals, where N is a positive integer;

[0088] The processing unit 204 is further configured to perform noise reduction processing on a target original speech signal of a target microphone channel among the multiple microphone channels and the N beam signals to obtain a first speech signal;

[0089] The processing unit 204 is further configured to perform weighted processing on the target original speech signal and the first speech signal to obtain a second speech signal.

[0090] The speech processing device 200 provided in the embodiment of the present invention includes an acquisition unit 202 and a processing unit 204. In the process of speech denoising, the acquisition unit 202 acquires the original speech signal corresponding to each microphone channel to obtain multiple original speech signals, and the processing unit 204 then uses the spatial information of the multiple original speech signals acquired by the acquisition unit 202 based on the beamforming algorithm to beamform the multiple original speech signals acquired by the acquisition unit 202 in N beam directions, so as to enhance the multiple original speech signals in N beam directions and obtain N beam signals. Wherein, N is a positive integer. Further, the processing unit 204 selects the target original speech signal corresponding to the target microphone channel among the multiple microphone channels, and inputs the above-formed N beam signals and the target original speech signal, a total of N+1 signals, into the multi-beam denoising model, so as to perform denoising on the above-formed N beam signals and the target original speech signal through the multi-beam denoising model, and outputs the denoised speech signal of a single channel, i.e., the first speech signal. Furthermore, the processing unit 204 performs weighted processing on the first speech signal and the target original speech signal through the attention module, i.e., the NN model, to obtain a weighted second speech signal. In this way, on the one hand, by combining multi-beam signal processing and multi-channel speech denoising neural network for speech denoising, there is no need for sound source localization and a large amount of training data, which improves the accuracy of speech denoising in a high noise environment, thereby improving the signal-to-noise ratio of the speech signal; on the other hand, by combining the original speech signal for denoising and weighting the speech signal, the fidelity of the denoised speech signal is ensured.

[0091] The target microphone channel is any one of the multiple microphone channels, and the target original voice signal is the original voice signal corresponding to the target microphone channel.

[0092] Furthermore, for the specific value of the number of the above-mentioned beam signals, that is, N, those skilled in the art can set it according to actual conditions, and no specific limitation is made here.

[0093] Furthermore, for the specific pointing directions of the above-mentioned N beam directions, those skilled in the art may set them according to actual conditions, and no specific limitation is made here.

[0094] Furthermore, a beamforming algorithm is used to combine the original speech signals of the multi-microphone channels, suppress interference signals in non-beam directions, and enhance speech signals in the beam direction.

[0095] In actual application, the above-mentioned beamforming algorithms include but are not limited to: MVDR beamforming algorithm, GSC beamforming algorithm, DSB algorithm, LCMV beamforming algorithm and SBF algorithm, etc. Technical personnel in this field can select the specific type of the above-mentioned beamforming algorithm according to actual conditions, and no specific limitation is made here.

[0096] It is understandable that there are two major paradigms for conventional multi-channel microphone array speech noise reduction methods: one is to combine sound source localization and use the beamforming method of signal processing for noise reduction, and the other is to use an end-to-end speech noise reduction deep neural model for noise reduction. However, in a high-noise environment, for the noise reduction method using the beamforming method of signal processing, the accuracy of sound source localization is low, which will lead to poor signal processing beamforming effect; and for the noise reduction method using the end-to-end speech noise reduction deep neural model, the model requires a large amount of training data (hundreds or even thousands of hours of training data), a larger scale, and higher computing power to achieve better noise reduction results, and the noise reduction performance of the model will be significantly reduced when the application scenario and training data are different. In this way, the speech noise reduction effect in a high-noise environment is reduced.

[0097] Therefore, the speech processing device 200 provided by the present invention combines the traditional speech noise reduction method based on multi-beam signal processing with the speech noise reduction method based on a multi-channel speech noise reduction neural network. It does not require sound source localization, nor does it require a large amount of training data to ensure the noise reduction effect. It only requires dozens of hours of training data, which reduces the amount of training data used and improves the accuracy of speech noise reduction in high-noise environments. In addition, the computing power of BeamForming is much lower than that of the multi-channel speech noise reduction neural network, which can reduce the model scale and greatly reduce the overall computing power of the speech processing device, making it easier to deploy the speech processing device on electronic devices with smaller computing power.

[0098] In some embodiments of the present invention, optionally, the processing unit 204 is specifically used to: use a neural network model to adaptively determine a first weight value of a target original speech signal and a second weight value of a first speech signal; perform weighted processing on the target original speech signal and the first speech signal according to the first weight value and the second weight value to obtain a second speech signal.

[0099] In some embodiments of the present invention, optionally, the processing unit 204 is specifically used to: utilize a neural network model, based on an attention mechanism, to adaptively determine a first weight value and a second weight value according to a target original speech signal.

[0100] In some embodiments of the present invention, optionally, the processing unit 204 is specifically configured to: utilize a beamforming algorithm to perform beamforming on N beam directions of multiple original speech signals within a preset angle range to obtain N beam signals.

[0101] In some embodiments of the present invention, optionally, the value of N is related to a preset angle range and the directivity of a beamforming algorithm; the value of N is positively correlated with the computing power of the speech processing device.

[0102] In some embodiments of the present invention, optionally, the directivity indicates the angle value between each two adjacent beam directions; the directivity is negatively correlated with the computing power of the speech processing device; the directivity is related to the noise reduction intensity and the noise information of the application scenario, wherein the noise information includes the noise intensity and noise distribution information of the application scenario.

[0103] In one embodiment of the present invention, an electronic device is also provided. Figure 6 As shown, Figure 6 FIG. 1 is a block diagram of a structure of an electronic device 300 provided in an embodiment of the present invention. The electronic device 300 includes:

[0104] A memory 302, where programs or instructions are stored;

[0105] Processor 304, when executing the above program or instruction, implements the steps of the speech processing method in any of the above embodiments.

[0106] The electronic device 300 provided in this embodiment includes a memory 302 and a processor 304. When the program or instructions in the memory 302 are executed by the processor 304, the steps of the voice processing method in any of the above embodiments are implemented. Therefore, the electronic device 300 has all the beneficial effects of the voice processing method in any of the above embodiments, which will not be repeated here.

[0107] Specifically, the memory 302 and the processor 304 may be connected via a bus or other means. The processor 304 may include one or more processing units, and the processor 304 may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and other chips.

[0108] In one embodiment of the present invention, a readable storage medium is also provided, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the speech processing method in any of the above embodiments are implemented.

[0109] The readable storage medium provided by the embodiment of the present invention can implement the steps of the speech processing method in any of the above embodiments when the program or instruction stored in the medium is executed by the processor. Therefore, the readable storage medium has all the beneficial effects of the speech processing method in any of the above embodiments, which will not be repeated here.

[0110] Specifically, the above-mentioned readable storage medium may include any medium capable of storing or transmitting information. Examples of readable storage media include electronic circuits, semiconductor memory devices, read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), flash memory, erasable ROM (EROM), magnetic tape, floppy disk, optical disk, hard disk, optical fiber medium, radio frequency (RF) link, optical data storage device, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0111] In one embodiment of the present invention, a computer program product is also proposed, including a computer program, which, when executed by a processor, implements the steps of the speech processing method in any of the above embodiments. Therefore, the computer program product proposed by the present invention has all the beneficial effects of the speech processing method in any of the above embodiments, which will not be described in detail here.

[0112] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance unless otherwise clearly specified and limited; the terms "connection", "installation", "fixation" and the like should be understood in a broad sense, for example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0113] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0114] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A speech processing method, It is characterized in that include: Acquire multiple original speech signals of multiple microphone channels; Performing beamforming on the plurality of original speech signals in N beam directions to obtain N beam signals, where N is a positive integer; Performing noise reduction processing on a target original speech signal of a target microphone channel among the multiple microphone channels and the N beam signals to obtain a first speech signal; The target original speech signal and the first speech signal are weighted to obtain a second speech signal.

2. The speech processing method according to claim 1, It is characterized in that The step of performing weighted processing on the target original speech signal and the first speech signal to obtain a second speech signal includes: Adaptively determining a first weight value of the target original speech signal and a second weight value of the first speech signal using a neural network model; The target original speech signal and the first speech signal are weighted according to the first weight value and the second weight value to obtain the second speech signal.

3. The speech processing method according to claim 2, It is characterized in that The method of adaptively determining a first weight value of the target original speech signal and a second weight value of the first speech signal by using a neural network model includes: The neural network model is used to adaptively determine the first weight value and the second weight value according to the target original speech signal based on an attention mechanism.

4. The speech processing method according to any one of claims 1 to 3, It is characterized in that The beamforming of the plurality of original speech signals in N beam directions to obtain N beam signals includes: By using a beamforming algorithm, beamforming is performed on N beam directions of the plurality of original speech signals within a preset angle range to obtain N beam signals.

5. The speech processing method according to claim 4, It is characterized in that Executed by the speech processing device, the value of N is related to the preset angle range and the directivity of the beamforming algorithm; the value of N is positively correlated with the computing power of the speech processing device.

6. The speech processing method according to claim 5, It is characterized in that The directivity indicates the angle value between each two adjacent beam directions; the directivity is negatively correlated with the computing power of the speech processing device; the directivity is related to the noise reduction intensity and the noise information of the application scenario, wherein the noise information includes the noise intensity and noise distribution information of the application scenario.

7. A speech processing device, It is characterized in that include: An acquisition unit, used for acquiring multiple original voice signals from multiple microphone channels; A processing unit, configured to perform beamforming on the plurality of original speech signals in N beam directions to obtain N beam signals, where N is a positive integer; The processing unit is further configured to perform noise reduction processing on a target original speech signal of a target microphone channel among the plurality of microphone channels and the N beam signals to obtain a first speech signal; The processing unit is further used to perform weighted processing on the target original speech signal and the first speech signal to obtain a second speech signal.

8. An electronic device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the speech processing method according to any one of claims 1 to 6 are implemented.

9. A readable storage medium, It is characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the speech processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the speech processing method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Method and system for eliminating noise

    CN101510426A

  • Voice processing method and device

    CN113744752A

  • Speech recognition method and device, electronic equipment and storage medium

    CN115862632A

  • Voice signal processing method and device, computer equipment and storage medium

    CN116030823A

  • Microphone array-based target voice acquisition method and device

    US20200342887A1

Cited By

  • Voice interaction method, device, equipment, medium and program product

    CN119943071A

  • Voice interaction method, apparatus, device, medium and program product

    CN119943071B