Voice noise reduction method, device, equipment and computer-readable storage medium

Abstract

The present invention discloses a voice noise reduction method, device, equipment and computer-readable storage medium. The method includes: obtaining first audio data collected by a microphone, and obtaining second audio data collected by a bone conduction sensor; splicing the audio data in a first frequency band in the first audio data and the audio data in a second frequency band in the second audio data to obtain third audio data; inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band; and performing gain processing on the audio data in the first frequency band in the third audio data by using the predicted gains to obtain a voice noise reduction result. The present invention realizes a scheme for voice noise reduction by combining the audio data collected by a bone conduction sensor and the audio data collected by a microphone, and improves the voice noise reduction effect.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular, to a method, apparatus, device and computer-readable storage medium for speech noise reduction. Background Art

[0002] Speech noise reduction refers to removing noise from noisy speech data to obtain as pure speech data as possible. Speech noise reduction has a wide range of application fields, such as often being applied in speech call scenarios. Since the signal collected by the bone conduction sensor is the signal generated by the vibration of the skull when a person speaks, and is less affected by external noise compared to a microphone, in current speech noise reduction algorithms, the vibration signal is collected by the bone conduction sensor and then converted into audio data through an algorithm as the noise-reduced speech data. However, the frequency of the audio data collected by the bone conduction sensor is only below 1KHZ, and the sound will sound very dull, with poor listening experience. Therefore, some people consider combining the audio data collected by the bone conduction sensor and the audio data collected by the microphone for speech noise reduction. However, how to combine them to obtain a better noise reduction effect is an urgent problem to be solved at present. Summary of the Invention

[0003] The main purpose of the present invention is to provide a method, apparatus, device and computer-readable storage medium for speech noise reduction, aiming to provide a solution for combining the audio data collected by the bone conduction sensor and the audio data collected by the microphone for speech noise reduction to improve the speech noise reduction effect.

[0004] To achieve the above object, the present invention provides a method for speech noise reduction, the method for speech noise reduction comprising the following steps:

[0005] Obtain first audio data collected by a microphone, and obtain second audio data collected by a bone conduction sensor;

[0006] Splice the audio data in the first frequency band in the first audio data and the audio data in the second frequency band in the second audio data to obtain third audio data, wherein the lower frequency point of the first frequency band is greater than or equal to the upper frequency point of the second frequency band;

[0007] Input the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gain corresponding to each frequency point in the first frequency band, wherein the predicted gain represents the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the frequency point in the third audio data, and the gain prediction model is a neural network model pre-trained with training data;

[0008] Perform gain processing on the audio data in the first frequency band in the third audio data using the predicted gain to obtain a speech noise reduction result.

[0009] Optionally, the step of inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band includes:

[0010] Input the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to a preset number of sub - frequency bands in the first frequency band;

[0011] Use the predicted gain corresponding to the target sub - frequency band as the predicted gains corresponding to each frequency point within the target sub - frequency band, where the target sub - frequency band is any one of the sub - frequency bands.

[0012] Optionally, before the step of inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, the following steps are further included:

[0013] Obtain the first pure speech data collected by a microphone and the second pure speech data collected by a bone conduction sensor in a quiet scene, and obtain the noisy audio data collected by the microphone in a noisy scene;

[0014] Mix the first pure speech data with the noisy audio data to obtain noisy speech data, and splice the audio data within the first frequency band in the noisy speech data and the audio data within the second frequency band in the second pure speech data to obtain the fourth audio data;

[0015] Calculate the label gain corresponding to the target sub - frequency band according to the energy of the pure speech data within the target sub - frequency band in the first pure speech data and the energy of the noisy speech data within the target sub - frequency band in the noisy speech data;

[0016] Use the audio features of the fourth audio data as the model input data, and use the label gains corresponding to each sub - frequency band as the model training labels to train the gain prediction model to be trained, and obtain the trained gain prediction model.

[0017] Optionally, the step of splicing the audio data within the first frequency band in the first audio data and the audio data within the second frequency band in the second audio data to obtain the third audio data includes:

[0018] For N frames of the first audio data and N frames of the second audio data collected synchronously with the N frames of the first audio data, splice the audio data within the first frequency band in the i - th frame of the first audio data and the audio data within the second frequency band in the i - th frame of the second audio data to obtain the i - th frame of the third audio data, where i = 1, 2...N;

[0019] The step of inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band includes:

[0020] Extract the audio features of N frames of the third audio data respectively, and input the audio features of N frames of the third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, where the predicted gain represents the predicted value of the energy proportion of pure speech data in the audio data corresponding to the frequency point within the Nth frame of the third audio data;

[0021] The step of using the predicted gain to perform gain processing on the audio data in the first frequency band of the third audio data to obtain a speech noise reduction result includes:

[0022] Use the predicted gain to perform gain processing on the audio data in the first frequency band of the Nth frame of the third audio data to obtain the speech noise reduction result corresponding to the Nth frame of the third audio data.

[0023] Optionally, before the step of splicing the audio data in the first frequency band of the ith frame of the first audio data and the audio data in the second frequency band of the ith frame of the second audio data to obtain the ith frame of the third audio data for N frames of the first audio data and N frames of the second audio data synchronously collected with the N frames of the first audio data, further includes:

[0024] When the obtained first audio data or second audio data is less than N frames, use preset audio data to fill in the insufficient number of frames to obtain N frames of the first audio data and N frames of the second audio data.

[0025] Optionally, the step of obtaining the first audio data collected by a microphone includes:

[0026] Collect various microphone signals through a microphone array;

[0027] Perform noise reduction processing on the various microphone signals based on a beamforming algorithm to obtain the first audio data.

[0028] Optionally, when the predicted gain is the predicted value of the energy proportion of pure speech data in the audio data corresponding to the frequency point within the third audio data, the step of using the predicted gain to perform gain processing on the audio data in the first frequency band of the third audio data to obtain a speech noise reduction result includes:

[0029] Multiply the audio data of each frequency point in the first frequency band of the third audio data by the predicted gain corresponding to the frequency point to obtain a speech noise reduction result.

[0030] To achieve the above object, the present invention further provides a voice noise reduction device, and the voice noise reduction device includes:

[0031] An acquisition module, configured to acquire first audio data collected by a microphone, and acquire second audio data collected by a bone conduction sensor;

[0032] A splicing module, configured to splice audio data within a first frequency band in the first audio data and audio data within a second frequency band in the second audio data to obtain third audio data;

[0033] A prediction module, configured to input audio features of the third audio data into a gain prediction model for prediction to obtain prediction gains respectively corresponding to each frequency point in the first frequency band, wherein a lower limit frequency point of the first frequency band is greater than or equal to an upper limit frequency point of the second frequency band, the prediction gain represents a predicted value of an energy proportion of pure voice data in audio data corresponding to the frequency point within the third audio data, and the gain prediction model is a neural network model pre-trained with training data;

[0034] A gain module, configured to perform gain processing on the audio data within the first frequency band in the third audio data by using the prediction gain to obtain a voice noise reduction result.

[0035] To achieve the above object, the present invention further provides a voice noise reduction device, and the voice noise reduction device includes: a memory, a processor, and a voice noise reduction program stored on the memory and executable on the processor. When the voice noise reduction program is executed by the processor, the steps of the voice noise reduction method described above are implemented.

[0036] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, on which a voice noise reduction program is stored. When the voice noise reduction program is executed by a processor, the steps of the voice noise reduction method described above are implemented.

[0037] In the present invention, first audio data collected by a microphone is obtained, and second audio data collected by a bone conduction sensor is obtained; audio data within a first frequency band in the first audio data and audio data within a second frequency band in the second audio data are spliced to obtain third audio data, where the lower frequency point of the first frequency band is greater than or equal to the upper frequency point of the second frequency band; the audio features of the third audio data are input into a gain prediction model for prediction to obtain prediction gains respectively corresponding to each frequency point in the first frequency band, where the prediction gain represents a predicted value of the energy proportion of pure speech data in the audio data corresponding to the frequency point within the third audio data, and the gain prediction model is a neural network model pre-trained with training data; the audio data within the first frequency band in the third audio data is subjected to gain processing using the prediction gain to obtain a speech noise reduction result. The present invention realizes a scheme for speech noise reduction by combining audio data collected by a bone conduction sensor and audio data collected by a microphone, combines the advantages of the bone conduction sensor and the microphone, and combines the two types of audio data by using a neural network model for gain prediction, improving the speech noise reduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic structural diagram of the hardware operating environment related to the solution of the embodiment of the present invention;

[0039] Figure 2 It is a schematic flowchart of the first embodiment of the speech noise reduction method of the present invention;

[0040] Figure 3 It is a schematic structural diagram of a gain prediction model related to the embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of the functional modules of the preferred embodiment of the speech noise reduction device of the present invention.

[0042] The realization, functional characteristics and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] As Figure 1 shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment related to the solution of the embodiment of the present invention.

[0045] It should be noted that for the speech noise reduction device in the embodiment of the present invention, the speech noise reduction device may be a device such as an earphone, a smart phone, a personal computer, a server, etc., and no specific limitation is made here.

[0046] As Figure 1As shown in the figure, the voice noise reduction device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0047] Those skilled in the art can understand that Figure 1 the device structure shown in does not constitute a limitation on the voice noise reduction device, and it may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0048] As Figure 1 shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a voice noise reduction program. The operating system is a program for managing and controlling the hardware and software resources of the device, and supports the operation of the voice noise reduction program and other software or programs. In Figure 1 the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 may be used to call the voice noise reduction program stored in the memory 1005 and perform the following operations:

[0049] Obtain first audio data collected by a microphone and second audio data collected by a bone conduction sensor;

[0050] Splice the audio data in the first frequency band in the first audio data and the audio data in the second frequency band in the second audio data to obtain third audio data, where the lower frequency point of the first frequency band is greater than or equal to the upper frequency point of the second frequency band;

[0051] Input the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gain corresponding to each frequency point in the first frequency band, where the predicted gain represents the predicted value of the energy ratio of pure voice data in the audio data corresponding to the frequency point in the third audio data, and the gain prediction model is a neural network model pre-trained with training data;

[0052] Perform gain processing on the audio data within the first frequency band in the third audio data using the predicted gain to obtain a speech noise reduction result.

[0053] Further, the operation of inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains respectively corresponding to each frequency point in the first frequency band includes:

[0054] Input the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains respectively corresponding to a preset number of sub - frequency bands in the first frequency band;

[0055] Use the predicted gain corresponding to the target sub - frequency band as the predicted gains respectively corresponding to each frequency point within the target sub - frequency band, where the target sub - frequency band is any one of the sub - frequency bands.

[0056] Further, before the operation of inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains respectively corresponding to each frequency point in the first frequency band, the processor 1001 can also be used to call the speech noise reduction program stored in the memory 1005 and perform the following operations:

[0057] Obtain the first pure speech data collected by the microphone and the second pure speech data collected by the bone conduction sensor in a quiet scene, and obtain the noisy audio data collected by the microphone in a noisy scene;

[0058] Mix the first pure speech data with the noisy audio data to obtain noisy speech data, and splice the audio data within the first frequency band in the noisy speech data and the audio data within the second frequency band in the second pure speech data to obtain the fourth audio data;

[0059] Calculate the label gain corresponding to the target sub - frequency band according to the energy of the pure speech data within the target sub - frequency band in the first pure speech data and the energy of the noisy speech data within the target sub - frequency band in the noisy speech data;

[0060] Use the audio features of the fourth audio data as model input data, and use the label gains respectively corresponding to each sub - frequency band as model training labels to train the gain prediction model to be trained to obtain the trained gain prediction model.

[0061] Further, the operation of splicing the audio data within the first frequency band in the first audio data and the audio data within the second frequency band in the second audio data to obtain the third audio data includes:

[0062] For N frames of the first audio data and N frames of the second audio data synchronously collected with the N frames of the first audio data, the audio data in the first frequency band in the i-th frame of the first audio data and the audio data in the second frequency band in the i-th frame of the second audio data are spliced to obtain the i-th frame of the third audio data, where i = 1, 2... N;

[0063] The operation of inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band includes:

[0064] Extract the audio features of N frames of the third audio data respectively, and input the audio features of N frames of the third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, where the predicted gain represents the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the frequency point in the N-th frame of the third audio data;

[0065] The operation of using the predicted gain to perform gain processing on the audio data in the first frequency band in the third audio data to obtain a speech noise reduction result includes:

[0066] Use the predicted gain to perform gain processing on the audio data in the first frequency band in the N-th frame of the third audio data to obtain the speech noise reduction result corresponding to the N-th frame of the third audio data.

[0067] Further, before the operation of splicing the audio data in the first frequency band in the i-th frame of the first audio data and the audio data in the second frequency band in the i-th frame of the second audio data to obtain the i-th frame of the third audio data for N frames of the first audio data and N frames of the second audio data synchronously collected with the N frames of the first audio data, the processor 1001 can also be used to call the speech noise reduction program stored in the memory 1005 to perform the following operations:

[0068] When the obtained first audio data or second audio data is less than N frames, use preset audio data to fill in the insufficient number of frames to obtain N frames of the first audio data and N frames of the second audio data.

[0069] Further, the operation of obtaining the first audio data collected by the microphone includes:

[0070] Collect various microphone signals through a microphone array;

[0071] Perform noise reduction processing on the various microphone signals based on a beamforming algorithm to obtain the first audio data.

[0072] Further, when the prediction gain is the predicted value of the proportion of the energy of the pure speech data in the audio data corresponding to the frequency points in the third audio data, the operation of performing gain processing on the audio data in the first frequency band in the third audio data by using the prediction gain to obtain the speech noise reduction result includes:

[0073] Multiplying the audio data of each frequency point in the first frequency band in the third audio data by the prediction gain corresponding to the frequency point to obtain the speech noise reduction result.

[0074] Based on the above structure, various embodiments of the speech noise reduction method are proposed.

[0075] Refer to Figure 2 , Figure 2 which is a schematic flowchart of the first embodiment of the speech noise reduction method of the present invention.

[0076] The embodiments of the present invention provide embodiments of the speech noise reduction method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here. In this embodiment, the execution subject of the speech noise reduction method may be devices such as headphones, personal computers, and smart phones, which are not limited in this embodiment. For the convenience of description below, the execution subject is omitted in the description of each embodiment. In this embodiment, the speech noise reduction method includes:

[0077] Step S10, obtaining first audio data collected by a microphone and second audio data collected by a bone conduction sensor;

[0078] A signal can be collected by a microphone (hereinafter referred to as a microphone signal for distinction), and the microphone signal is converted to obtain audio data (hereinafter referred to as first audio data for distinction); when collecting a signal by a microphone, a signal is synchronously collected by a bone conduction sensor (hereinafter referred to as a bone conduction signal for distinction), and the bone conduction signal is converted to obtain audio data (hereinafter referred to as second audio data for distinction).

[0079] The bone conduction sensor and the microphone sensor can be arranged in the device that executes the speech noise reduction algorithm of this embodiment. For example, they can be arranged in a headphone device or other devices, which are not limited in this embodiment.

[0080] It should be noted that the specific implementation manners of converting the microphone signal to obtain the first audio data and converting the bone conduction signal to obtain the second audio data are not limited in this embodiment and can refer to conventional microphone algorithms or bone conduction algorithms.

[0081] In specific embodiments, voice noise reduction can be performed based on multiple frames of first audio data and multiple frames of second audio data, or on single frames of first audio data and single frames of second audio data. The method of dividing the audio data into frames and the duration of each frame of data are not limited in this embodiment.

[0082] Further, in one embodiment, when converting the microphone signal to obtain the first audio data, other voice noise reduction algorithms for microphone signals can be used for preliminary noise reduction. That is, the obtained first audio data is preliminarily voice noise-reduced, and then the first audio data is combined with the second audio data for further voice noise reduction, thereby further improving the voice noise reduction effect. Among them, the voice noise reduction algorithm used for preliminary noise reduction is not limited.

[0083] For example, in one embodiment, preliminary noise reduction can be performed on the microphone signal based on the beamforming algorithm. Specifically, the step of obtaining the first audio data collected by the microphone in step S10 includes:

[0084] Step S101, collecting various microphone signals through a microphone array;

[0085] Step S102, performing noise reduction processing on the various microphone signals based on the beamforming algorithm to obtain the first audio data.

[0086] The signal can be collected through a set microphone array, and each microphone in the microphone array respectively collects a microphone signal. Based on the beamforming algorithm, noise reduction processing is performed on the various microphone signals to obtain the first audio data. The beamforming algorithm can determine the sound source direction of the microphone signal based on the phase difference of the various microphone signals. According to the direction of the user's mouth relative to the microphone array calibrated in advance, the sound signal coming from the direction of the user's mouth can be extracted from the microphone signal. Furthermore, by enhancing the sound signal in this direction and suppressing the sound signals in other directions, preliminary voice noise reduction processing can be achieved to obtain the first audio data.

[0087] Step S20, splicing the audio data in the first frequency band in the first audio data and the audio data in the second frequency band in the second audio data to obtain the third audio data, where the lower frequency point of the first frequency band is greater than or equal to the upper frequency point of the second frequency band;

[0088] The first frequency band and the second frequency band can be divided within the full frequency band, and the lower frequency point of the first frequency band is greater than or equal to the upper frequency point of the second frequency band. In this embodiment, the division method of the first frequency band and the second frequency band is not limited. In the specific implementation, the frequency band with better sound perception (meeting the set sound perception requirements) of the audio data collected by the bone conduction sensor can be used as the second frequency band, and the part of the full frequency band other than the second frequency band can be used as the first frequency band. For example, the first frequency band can be above 1KHZ (including 1KHZ), and the second frequency band can be below 1KHZ.

[0089] The audio data within the first frequency band in the first audio data and the audio data within the second frequency band in the second audio data are spliced to obtain audio data (hereinafter referred to as the third audio data for distinction). For example, when the first frequency band is above 1KHZ (including 1KHZ) and the second frequency band is below 1KHZ, the audio data above 1KHZ (including 1KHZ) in the first audio data and the audio data below 1KHZ in the second audio data are spliced to obtain the third audio data.

[0090] It should be noted that when performing voice noise reduction based on multiple frames of first audio data and multiple frames of second audio data, for each frame of first audio data, the audio data within the first frequency band in this frame of first audio data is spliced with the audio data within the second frequency band in a frame of second audio data collected synchronously to obtain a frame of third audio data.

[0091] Step S30: Input the audio features of the third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, where the predicted gain represents the predicted value of the energy ratio of the pure voice data in the audio data corresponding to the frequency point within the third audio data, and the gain prediction model is a neural network model pre-trained with training data;

[0092] A neural network model is preset, and its input data is set as the audio features of audio data; the audio features can be obtained by using an audio feature extraction algorithm, which is not limited in this embodiment. For example, MFCC (Mel Frequency Cepstrum Coefficient) features can be extracted; the output data is set as the gains corresponding to each frequency point or each sub-band within the first frequency band, and this gain is used to represent the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the corresponding frequency point in the input audio features. In addition to the settings of the input data and the output data, the internal structure of the neural network model can also be set as needed. Specifically, what kind of structure is adopted is not limited in this embodiment. For the settings of the input data and the output data, corresponding training data is collected in advance to train the neural network model, so that the accuracy of the gain predicted by the neural network model meets the set requirements, and then the neural network model can be deployed. For the sake of distinction, the trained neural network model is called a gain prediction model.

[0093] The audio features of the third audio data can be extracted, and the extracted audio features are input into the gain prediction model for prediction to obtain the gains corresponding to each frequency point within the first frequency band (since it is a prediction result, hereinafter, for the sake of distinction, it is called the predicted gain). It should be noted that when the output of the gain prediction model is the predicted gain corresponding to each sub-band within the first frequency band, the predicted gain of each frequency point within the sub-band can be obtained by interpolation, so as to obtain the predicted gain corresponding to each frequency point within the first frequency band.

[0094] For the predicted gain corresponding to a certain frequency point, this predicted gain represents the predicted value of the energy ratio of the pure speech data in the audio data of this frequency point in the third audio data. It can be understood that the predicted value of the energy ratio is not the energy ratio itself, that is, the gain prediction model can only obtain the prediction result, and this result may be the real energy ratio or may have an error from the real energy ratio. Among them, it should be noted that the meaning of "represent" is that this predicted gain can be but is not limited to the predicted value of the energy ratio of the pure speech data in the audio data of this frequency point in the third audio data to the audio data of this frequency point. For example, it can also be the predicted value of the ratio obtained by dividing the energy of the audio data of this frequency point in the third audio data by the energy of the pure speech data in the audio data of this frequency point.

[0095] Step S40, perform gain processing on the audio data within the first frequency band in the third audio data by using the predicted gain to obtain a speech denoising result.

[0096] After obtaining the predicted gains corresponding to each frequency point within the first frequency band, the predicted gains are used to perform gain processing on the third audio data. Specifically, for the audio data of each frequency point within the first frequency band in the third audio data, the predicted gain of that frequency point is used to perform gain processing on the audio data of that frequency point; for the audio data of each frequency point within the second frequency band in the third audio data, no processing is required. The third audio data after gain processing is used as the speech noise reduction result.

[0097] It should be noted that the specific processing process of performing gain processing on audio data using the predicted gain is related to the definition of the predicted gain. For example, when the predicted gain of a certain frequency point refers to the predicted value of the ratio obtained by dividing the energy of the audio data of that frequency point in the third audio data by the energy of the pure speech data in the audio data of that frequency point, the audio data of that frequency point in the third audio data can be divided by the predicted gain to obtain the speech noise reduction result corresponding to that frequency point.

[0098] For another example, in an embodiment, when the predicted gain is the predicted value of the proportion of the energy of the pure speech data in the audio data of the corresponding frequency point within the third audio data, the step S40 includes:

[0099] Step S401, multiply the audio data of each frequency point within the first frequency band in the third audio data by the corresponding predicted gain of that frequency point to obtain the speech noise reduction result.

[0100] For each frequency point within the first frequency band, multiply the audio data of that frequency point in the third audio data by the corresponding predicted gain of that frequency point to obtain the speech noise reduction result corresponding to that frequency point.

[0101] In this embodiment, the first audio data collected by the microphone and the second audio data collected by the bone conduction sensor are obtained; the audio data within the first frequency band in the first audio data and the audio data within the second frequency band in the second audio data are spliced to obtain the third audio data, where the lower limit frequency point of the first frequency band is greater than or equal to the upper limit frequency point of the second frequency band; the audio features of the third audio data are input into the gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, where the predicted gain represents the predicted value of the proportion of the energy of the pure speech data in the audio data of the corresponding frequency point within the third audio data, and the gain prediction model is a neural network model pre-trained with training data; the predicted gains are used to perform gain processing on the audio data within the first frequency band in the third audio data to obtain the speech noise reduction result. This embodiment realizes a scheme for speech noise reduction by combining the audio data collected by the bone conduction sensor and the audio data collected by the microphone, combines the advantages of the bone conduction sensor and the microphone, and combines the two types of audio data by using the neural network model for gain prediction, improving the speech noise reduction effect.

[0102] Furthermore, based on the above first embodiment, a second embodiment of the voice noise reduction algorithm of the present invention is proposed. In this embodiment, the step S30 includes:

[0103] Step S301, input the audio features of the third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to a preset number of sub-bands in the first frequency band respectively;

[0104] To reduce the model complexity, thereby reducing the requirement of voice noise reduction for device computing power or improving the efficiency of voice noise reduction, in this embodiment, the first frequency band is divided into a preset number of sub-bands, and the output data of the neural network model is set as the gains corresponding to each sub-band respectively. Among them, the preset number can be set as needed and is not limited in this embodiment. For example, it can be set to 20; the spans of each sub-band can be the same or different, and can also be set as needed.

[0105] Input the audio features of the third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to each sub-band in the first frequency band respectively.

[0106] Step S302, use the predicted gain corresponding to the target sub-band as the predicted gains corresponding to each frequency point within the target sub-band, where the target sub-band is any one of the sub-bands.

[0107] For any one sub-band (hereinafter referred to as the target sub-band for distinction), use the predicted gain corresponding to the target sub-band as the predicted gains corresponding to each frequency point within the target sub-band. That is, the interpolation method is used to complement the predicted gains corresponding to each frequency point within the sub-band, so as to obtain the predicted gains corresponding to each frequency point within the first frequency band.

[0108] Furthermore, in an implementation manner, before the step S30, it further includes:

[0109] Step S60, obtain the first pure voice data collected by the microphone and the second pure voice data collected by the bone conduction sensor in a quiet scene, and obtain the noisy audio data collected by the microphone in a noisy scene;

[0110] The quiet scene can be a scene where the noise is lower than a certain standard. The tester can speak in a quiet scene, and at the same time collect signals through the microphone and the bone conduction sensor, convert and process the signals collected by the microphone to obtain audio data (hereinafter referred to as the first pure voice data for distinction), and convert and process the signals collected by the bone conduction sensor to obtain audio data (hereinafter referred to as the second pure voice data for distinction).

[0111] A noise scenario can be a scenario where the noise is higher than a certain standard or has a specific noise, and there is no human voice in the noise scenario. In the noise scenario, signals are collected through a microphone, and the signals collected by the microphone are processed to obtain audio data (hereinafter referred to as noise audio data for distinction).

[0112] Step S70: Mix the first pure speech data with the noise audio data to obtain noisy speech data, and splice the audio data in the first frequency band in the noisy speech data and the audio data in the second frequency band in the second pure speech data to obtain fourth audio data;

[0113] Mix the first pure speech data with the noise audio data, and noisy speech data with noise can be obtained (hereinafter referred to as noisy speech data for distinction).

[0114] It should be noted that when the first audio data is obtained by performing noise reduction processing on the signals collected by the microphone array based on the beamforming algorithm, the signals collected by each microphone in the microphone array in the quiet scenario and the signals collected in the noise scenario can be mixed first, and then the mixed signals of each microphone are subjected to noise reduction processing based on the beamforming algorithm to obtain noisy speech data. At this time, the noise in the noisy speech data is the noise remaining after preliminary noise reduction processing.

[0115] After obtaining the noisy speech data, the audio data in the first frequency band in the noisy speech data and the audio data in the second frequency band in the second pure speech data can be spliced to obtain audio data (hereinafter referred to as fourth audio data for distinction).

[0116] Step S80: Calculate the label gain corresponding to the target sub-band according to the energy of the pure speech data in the target sub-band in the first pure speech data and the energy of the noisy speech data in the target sub-band in the noisy speech data;

[0117] The first pure speech data is equivalent to the result after noise reduction of the noisy speech data. For any one of the sub-bands in the first frequency band (i.e., the target sub-band), the gain corresponding to the target sub-band can be calculated according to the energy of the pure speech data in the target sub-band in the first pure speech data and the energy of the noisy speech data in the target sub-band in the noisy speech data (since it is a real gain and is used as a training label, it is hereinafter referred to as label gain for distinction).

[0118] The specific calculation method of the label gain is related to the definition of the predicted gain. For example, when the predicted gain of a certain frequency point refers to the predicted value of the ratio obtained by dividing the energy of the audio data at this frequency point in the third audio data by the energy of the pure speech data in the audio data at this frequency point, then, for any sub-band, the energy of the noisy speech data in this sub-band in the noisy speech data can be divided by the energy of the pure speech data in this sub-band in the first pure speech data to obtain the label gain corresponding to this sub-band. Another example is that when the predicted gain of a certain frequency point refers to the predicted value of the proportion of the energy of the pure speech data in the audio data at this frequency point in the third audio data, then, for any sub-band, the energy of the pure speech data in this sub-band in the first pure speech data can be divided by the energy of the noisy speech data in this sub-band in the noisy speech data to obtain the label gain corresponding to this sub-band.

[0119] Step S90: Use the audio features of the fourth audio data as the model input data, and use the label gains corresponding to each of the sub-bands as the model training labels to train the gain prediction model to be trained, and obtain the trained gain prediction model.

[0120] For the gain prediction model to be trained (i.e., the pre-set neural network model), the audio features of the fourth audio data can be used as the model input data, the label gains corresponding to each sub-band can be used as the model training labels, and the model can be trained using the supervised training method to obtain the trained gain prediction model. In this embodiment, the training method is not limited.

[0121] Furthermore, based on the above first and / or second embodiments, a third embodiment of the voice noise reduction algorithm of the present invention is proposed. In this embodiment, the step S20 includes:

[0122] Step S201: For N frames of the first audio data and N frames of the second audio data synchronously collected with the N frames of the first audio data, splice the audio data in the first frequency band in the i-th frame of the first audio data and the audio data in the second frequency band in the i-th frame of the second audio data to obtain the i-th frame of the third audio data, where i = 1, 2... N;

[0123] The size of N can be set in advance as needed, for example, set to 5. For each frame of first audio data collected (referred to as the current frame of first audio data) and a frame of second audio data collected synchronously with this frame of first audio data (referred to as the current frame of second audio data), combining the N - 1 frames of first audio data and N - 1 frames of second audio data collected previously, N frames of first audio data and N frames of second audio data are obtained. Using these N frames of first audio data and N frames of second audio data to perform speech noise reduction on the third audio data spliced by the current frame of first audio data (that is, the Nth frame of first audio data) and the current frame of second audio data (that is, the Nth frame of second audio data).

[0124] Specifically, the audio data in the first frequency band of the ith frame of the N frames of first audio data can be spliced with the audio data in the second frequency band of the ith frame of the N frames of second audio data to obtain the ith frame of third audio data. Wherein, i = 1, 2... N. It can be understood that a total of N frames of third audio data can be obtained.

[0125] The step S30 includes:

[0126] Step S303, respectively extract the audio features of the N frames of the third audio data, and input the audio features of the N frames of the third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band. Wherein, the predicted gain represents the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the frequency point in the Nth frame of the third audio data;

[0127] For the N frames of third audio data, extract the audio features respectively, and input the audio features of the N frames of third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band. At this time, the predicted gain represents the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the frequency point in the Nth frame of the third audio data.

[0128] The step S40 includes:

[0129] Step S402, use the predicted gain to perform gain processing on the audio data in the first frequency band of the Nth frame of the third audio data to obtain the speech noise reduction result corresponding to the Nth frame of the third audio data.

[0130] After obtaining the predicted gains corresponding to each frequency point in the first frequency band, use the predicted gain to perform gain processing on the audio data in the first frequency band of the Nth frame of the third audio data. That is, for each frequency point in the first frequency band, use the predicted gain of this frequency point to perform gain processing on the audio data of this frequency point in the Nth frame of the third audio data to obtain the speech noise reduction result corresponding to the Nth frame of the third audio data.

[0131] Further, in one embodiment, the internal structure of the neural network model can be set to be composed of multiple GRUs (gated recurrent neural networks) and a Dense (fully connected layer). The multiple GRUs are connected in a skip connection manner, that is, the multiple GRUs are arranged in sequence, and the result output by each GRU is given to each GRU after it. The fully connected layer is connected after the last GRU. Exemplarily, as Figure 3 shown, the gain prediction model is composed of four GRUs and a fully connected layer. And in the figure, it takes the example of inputting 32-dimensional audio features of 5 frames of the third audio data and outputting the predicted gains corresponding to 20 sub-bands respectively.

[0132] Further, in one embodiment, before the step S201, it further includes:

[0133] Step S202, when the obtained first audio data or the second audio data is less than N frames, use preset audio data to fill in the insufficient number of frames to obtain N frames of the first audio data and N frames of the second audio data.

[0134] In a specific application scenario, the obtained first audio data and second audio data may be less than N frames. For example, in a voice call scenario, when performing voice noise reduction on the first frame of the first audio data and the second audio data collected after the start of the call, the previous N - 1 frames are not available. When the obtained first audio data or the second audio data is less than N frames, preset audio data can be used to fill in the insufficient number of frames, so as to obtain N frames of the first audio data and N frames of the second audio data. The preset audio data can be set according to needs and is not limited in this embodiment; for the first audio data and the second audio data, different preset audio data can be set, or the same preset audio data can be set. For example, both are set to 0, that is, the insufficient number of frames are all filled with zeros.

[0135] In addition, an embodiment of the present invention also proposes a voice noise reduction device. Referring to Figure 4 , the voice noise reduction device includes:

[0136] An acquisition module 10, configured to acquire first audio data collected by a microphone and second audio data collected by a bone conduction sensor;

[0137] A splicing module 20, configured to splice the audio data in the first frequency band in the first audio data and the audio data in the second frequency band in the second audio data to obtain third audio data;

[0138] A prediction module 30, configured to input the audio features of the third audio data into a gain prediction model for prediction, so as to obtain predicted gains corresponding to each frequency point in the first frequency band, where a lower limit frequency point of the first frequency band is greater than or equal to an upper limit frequency point of the second frequency band, the predicted gain represents a predicted value of an energy proportion of pure speech data in the audio data corresponding to the frequency point in the third audio data, and the gain prediction model is a neural network model pre-trained with training data;

[0139] A gain module 40, configured to perform gain processing on the audio data in the first frequency band of the third audio data by using the predicted gain, so as to obtain a speech noise reduction result.

[0140] Further, the prediction module 30 is further configured to:

[0141] Input the audio features of the third audio data into a gain prediction model for prediction, so as to obtain predicted gains corresponding to a preset number of sub-bands in the first frequency band;

[0142] Use the predicted gain corresponding to the target sub-band as the predicted gains corresponding to each frequency point in the target sub-band, where the target sub-band is any one of the sub-bands.

[0143] Further, the acquisition module 10 is further configured to:

[0144] Acquire first pure speech data collected by a microphone and second pure speech data collected by a bone conduction sensor in a quiet scenario, and acquire noisy audio data collected by the microphone in a noisy scenario;

[0145] The splicing module 20 is further configured to:

[0146] Mix the first pure speech data with the noisy audio data to obtain noisy speech data, and splice the audio data in the first frequency band of the noisy speech data and the audio data in a second frequency band of the second pure speech data to obtain fourth audio data;

[0147] The speech noise reduction device further includes:

[0148] A calculation module, configured to calculate a label gain corresponding to the target sub-band according to the energy of the pure speech data in the target sub-band in the first pure speech data and the energy of the noisy speech data in the target sub-band in the noisy speech data;

[0149] A training module, which is used to use the audio features of the fourth audio data as model input data, and use the label gains corresponding to each of the sub-bands as model training labels to train the gain prediction model to be trained, and obtain the trained gain prediction model.

[0150] Further, the splicing module 20 is further used for:

[0151] For N frames of the first audio data and N frames of the second audio data synchronously collected with the N frames of the first audio data, splice the audio data in the first frequency band of the i-th frame of the first audio data and the audio data in the second frequency band of the i-th frame of the second audio data to obtain the i-th frame of the third audio data, where i = 1, 2... N;

[0152] The prediction module 30 is further used for:

[0153] Extract the audio features of N frames of the third audio data respectively, and input the audio features of N frames of the third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, where the predicted gain represents the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the frequency point in the N-th frame of the third audio data;

[0154] The gain module 40 is further used for:

[0155] Perform gain processing on the audio data in the first frequency band of the N-th frame of the third audio data by using the predicted gain to obtain the speech noise reduction result corresponding to the N-th frame of the third audio data.

[0156] Further, the speech noise reduction device further includes:

[0157] A padding module, which is used to pad the insufficient number of frames with preset audio data when the obtained first audio data or second audio data is less than N frames, to obtain N frames of the first audio data and N frames of the second audio data.

[0158] Further, the acquisition module 10 is further used for:

[0159] Collect various microphone signals through a microphone array;

[0160] Perform noise reduction processing on the various microphone signals based on a beamforming algorithm to obtain the first audio data.

[0161] Further, when the predicted gain is the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the frequency point in the third audio data, the gain module 40 is further used for:

[0162] Multiply the audio data of each frequency point in the first frequency band in the third audio data by the predicted gain corresponding to the frequency point to obtain a speech noise reduction result.

[0163] For each embodiment of the speech noise reduction device of the present invention, reference may be made to each embodiment of the speech noise reduction method of the present invention, which will not be elaborated here.

[0164] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a speech noise reduction program is stored. When the speech noise reduction program is executed by a processor, the steps of the speech noise reduction method described below are implemented.

[0165] For each embodiment of the speech noise reduction device and computer-readable storage medium of the present invention, reference may be made to each embodiment of the speech noise reduction method of the present invention, which will not be elaborated here.

[0166] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without more limitations, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or device including the element.

[0167] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0168] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0169] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, are equally included in the patent protection scope of the present invention.

Claims

1. A voice noise reduction method, characterized in that, The voice noise reduction method includes the following steps: Obtain first audio data collected by a microphone and second audio data collected by a bone conduction sensor; Splice the audio data within a first frequency band in the first audio data and the audio data within a second frequency band in the second audio data to obtain third audio data, where the lower frequency point of the first frequency band is greater than or equal to the upper frequency point of the second frequency band; Input the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, where the predicted gain represents the predicted value of the energy ratio of the pure voice data in the audio data corresponding to the frequency point within the third audio data, and the gain prediction model is a neural network model pre-trained with training data; Perform gain processing on the audio data within the first frequency band in the third audio data using the predicted gain to obtain a voice noise reduction result.

2. The voice noise reduction method according to claim 1, wherein, The step of inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band includes: Input the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to a preset number of sub-frequency bands in the first frequency band; Use the predicted gain corresponding to the target sub-frequency band as the predicted gains corresponding to each frequency point within the target sub-frequency band, where the target sub-frequency band is any one of the sub-frequency bands.

3. The voice noise reduction method according to claim 2, wherein, Before the step of inputting the audio features of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, it further includes: Obtain first pure voice data collected by a microphone and second pure voice data collected by a bone conduction sensor in a quiet scene, and obtain noisy audio data collected by a microphone in a noisy scene; Mix the first pure voice data with the noisy audio data to obtain noisy voice data, and splice the audio data within the first frequency band in the noisy voice data and the audio data within a second frequency band in the second pure voice data to obtain fourth audio data; Calculate the label gain corresponding to the target sub-frequency band according to the energy of the pure voice data within the target sub-frequency band in the first pure voice data and the energy of the noisy voice data within the target sub-frequency band in the noisy voice data; Use the audio features of the fourth audio data as model input data, and use the label gains corresponding to each sub-frequency band as model training labels to train the gain prediction model to be trained to obtain the trained gain prediction model.

4. The voice noise reduction method according to claim 1, wherein The step of splicing the audio data within a first frequency band in the first audio data and the audio data within a second frequency band in the second audio data to obtain third audio data includes: For N frames of the first audio data and N frames of the second audio data synchronously collected with the N frames of the first audio data, the audio data in the first frequency band in the i-th frame of the first audio data and the audio data in the second frequency band in the i-th frame of the second audio data are spliced to obtain the i-th frame of the third audio data, where i = 1, 2... N; The step of inputting the audio feature of the third audio data into a gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band includes: Extracting the audio features of N frames of the third audio data respectively, and inputting the audio features of N frames of the third audio data into the gain prediction model for prediction to obtain the predicted gains corresponding to each frequency point in the first frequency band, where the predicted gain represents the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the frequency point in the N-th frame of the third audio data; The step of using the predicted gain to perform gain processing on the audio data in the first frequency band in the third audio data to obtain a speech noise reduction result includes: Using the predicted gain to perform gain processing on the audio data in the first frequency band in the N-th frame of the third audio data to obtain the speech noise reduction result corresponding to the N-th frame of the third audio data.

5. The voice noise reduction method according to claim 4, wherein Before the step of, for N frames of the first audio data and N frames of the second audio data synchronously collected with the N frames of the first audio data, splicing the audio data in the first frequency band in the i-th frame of the first audio data and the audio data in the second frequency band in the i-th frame of the second audio data to obtain the i-th frame of the third audio data, further includes: When the obtained first audio data or second audio data is less than N frames, using preset audio data to fill in the insufficient number of frames to obtain N frames of the first audio data and N frames of the second audio data.

6. The voice noise reduction method according to claim 1, wherein The step of obtaining the first audio data collected by a microphone includes: Collecting various microphone signals through a microphone array; Performing noise reduction processing on the various microphone signals based on a beamforming algorithm to obtain the first audio data.

7. The voice noise reduction method according to any one of claims 1 to 6, characterized in that, When the predicted gain is the predicted value of the energy ratio of the pure speech data in the audio data corresponding to the frequency point in the third audio data, the step of using the predicted gain to perform gain processing on the audio data in the first frequency band in the third audio data to obtain a speech noise reduction result includes: Multiplying the audio data of each frequency point in the first frequency band in the third audio data by the predicted gain corresponding to the frequency point to obtain a speech noise reduction result.

8. A voice noise reduction device, characterized in that, The speech noise reduction device includes: An acquisition module, configured to acquire the first audio data collected by a microphone and acquire the second audio data collected by a bone conduction sensor; A splicing module, configured to splice the audio data in the first frequency band in the first audio data and the audio data in the second frequency band in the second audio data to obtain the third audio data; A prediction module, configured to input audio features of the third audio data into a gain prediction model for prediction, so as to obtain predicted gains corresponding to respective frequency points in the first frequency band, where a lower limit frequency point of the first frequency band is greater than or equal to an upper limit frequency point of the second frequency band, the predicted gain represents a predicted value of an energy proportion of pure speech data in audio data corresponding to the frequency point in the third audio data, and the gain prediction model is a neural network model pre-trained with training data; A gain module, configured to perform gain processing on the audio data in the first frequency band in the third audio data by using the predicted gain to obtain a speech noise reduction result.

9. A voice noise reduction device, characterized in that, The speech noise reduction device includes: a memory, a processor, and a speech noise reduction program stored on the memory and executable on the processor, and when the speech noise reduction program is executed by the processor, the steps of the speech noise reduction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a speech noise reduction program, and when the speech noise reduction program is executed by a processor, the steps of the speech noise reduction method according to any one of claims 1 to 7 are implemented.

Images