Voice noise reduction method, device, equipment and computer-readable storage medium

Abstract

The present invention discloses a method, device, equipment and computer-readable storage medium for speech noise reduction. The speech noise reduction method includes: acquiring first speech data collected by a microphone and second speech data collected by a bone conduction sensor; inputting the speech data in a first frequency band in the first speech data and the speech data in a second frequency band in the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data; wherein the first frequency band is greater than the second frequency band; the speech fusion noise reduction network is pre-trained by using microphone noisy speech data and bone conduction noisy speech data as input data and the microphone clean speech data corresponding to the microphone noisy speech data as training labels. The speech noise reduction solution of the present invention improves the speech noise reduction effect.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and particularly to a speech noise reduction method, device, equipment and computer-readable storage medium. Background Art

[0002] Speech noise reduction refers to the technology of extracting useful speech signals (or clean speech signals) from noisy speech signals as much as possible and suppressing or reducing noise interference when speech signals are interfered or even submerged by various background noises. Speech noise reduction technology is applied to many scenarios, such as for call speech noise reduction. In current speech noise reduction technologies, there are solutions for noise reduction based on speech data collected by single microphones or multiple microphones. However, although the speech data collected by microphones covers a wide frequency range, its noise resistance is almost non-existent. Therefore, the overall noise reduction effect of the speech noise reduction solutions based on speech data collected by microphones cannot be further improved. Summary of the Invention

[0003] The main purpose of the present invention is to provide a speech noise reduction method, device, equipment and computer-readable storage medium, aiming to provide a speech noise reduction solution based on speech data collected by bone conduction sensors and speech data collected by microphones to improve the speech noise reduction effect.

[0004] To achieve the above purpose, the present invention provides a speech noise reduction method, and the speech noise reduction method includes the following steps:

[0005] Obtain first speech data collected by a microphone and second speech data collected by a bone conduction sensor;

[0006] Input the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data;

[0007] Wherein, the first frequency band is greater than the second frequency band; the speech fusion noise reduction network is pre-trained by using microphone noisy speech data and bone conduction noisy speech data as input data and the microphone clean speech data corresponding to the microphone noisy speech data as training labels.

[0008] Optionally, the step of inputting the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data includes:

[0009] Perform time-domain to frequency-domain conversion on a single frame of the first speech data to obtain the first amplitude and the first phase angle value of each frequency point;

[0010] Perform a time-domain to frequency-domain conversion on the single-frame second speech data to obtain the second amplitude and the second phase angle value at each frequency point;

[0011] Generate target input data according to the first amplitude and the first phase angle value corresponding to each frequency point in the first frequency band, and the second amplitude and the second phase angle value corresponding to each frequency point in the second frequency band;

[0012] Input the target input data into the speech fusion noise reduction network for prediction to obtain the third amplitude and the third phase angle value at each frequency point;

[0013] Perform a frequency-domain to time-domain conversion based on the third amplitude and the third phase angle value at each frequency point to obtain the single-frame target noise-reduced speech data.

[0014] Optionally, the step of generating target input data according to the first amplitude and the first phase angle value corresponding to each frequency point in the first frequency band, and the second amplitude and the second phase angle value corresponding to each frequency point in the second frequency band includes:

[0015] Normalize the first amplitude at each frequency point in the first frequency band and the second amplitude at each frequency point in the second frequency band respectively, and then splice them to obtain the first-channel data;

[0016] Normalize the first phase angle value at each frequency point in the first frequency band and the second phase angle value at each frequency point in the second frequency band respectively, and then splice them to obtain the second-channel data;

[0017] Use the first-channel data and the second-channel data as the two-channel target input data.

[0018] Optionally, the step of inputting the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into the speech fusion noise reduction network for prediction to obtain the target noise-reduced speech data includes:

[0019] Input the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into the convolutional layer in the speech fusion noise reduction network for convolutional processing to obtain the convolutional output data;

[0020] Input the convolutional output data into the recurrent neural network layer in the speech fusion noise reduction network for processing to obtain the recurrent network output data;

[0021] Input the convolutional output data and the recurrent network output data into the upsampling convolutional layer in the speech fusion noise reduction network for upsampling convolutional processing, and obtain the target noise-reduced speech data based on the result of the upsampling convolutional processing.

[0022] Optionally, before the step of inputting the speech data in the first frequency band in the first speech data and the speech data in the second frequency band in the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data, the method further includes:

[0023] In one round of training, input the speech data in the first frequency band in the microphone noisy speech data and the speech data in the second frequency band in the bone conduction noisy speech data into the speech fusion noise reduction network to be trained for prediction to obtain predicted noise-reduced speech data;

[0024] Calculate a first loss based on the speech data in the first frequency band in the predicted noise-reduced speech data and the speech data in the first frequency band in the microphone clean speech data;

[0025] Calculate a second loss based on the speech data in the second frequency band in the predicted noise-reduced speech data and the speech data in the second frequency band in the microphone clean speech data;

[0026] Perform a weighted sum of the first loss and the second loss to obtain a target loss, and update the speech fusion noise reduction network to be trained according to the target loss, so as to use the updated speech fusion noise reduction network as the basis for the next round of training;

[0027] After multiple rounds of training, use the updated speech fusion noise reduction network as the trained speech fusion noise reduction network.

[0028] Optionally, the step of performing a weighted sum of the first loss and the second loss to obtain a target loss includes:

[0029] Determine the weighted weight for this round corresponding to the training round number of this round of training, where the weighted weight corresponding to the second loss is larger when the training round number is larger;

[0030] Perform a weighted sum of the first loss and the second loss according to the weighted weight for this round to obtain a target loss.

[0031] Optionally, before the step of inputting the speech data in the first frequency band in the first speech data and the speech data in the second frequency band in the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data, the method further includes:

[0032] Obtain first background noise data collected by a microphone in a background noise environment and first clean speech data collected in a noise isolation environment, and obtain second background noise data collected by a bone conduction sensor in the background noise environment and second clean speech data collected in the noise isolation environment;

[0033] Add the first noise data to the first clean speech data according to a preset signal-to-noise ratio to obtain the microphone noisy speech data;

[0034] Add the second noise data to the second clean speech data according to the noise weight in the microphone noisy speech data to obtain the bone conduction noisy speech data.

[0035] To achieve the above object, the present invention also provides a speech noise reduction device, and the speech noise reduction device includes:

[0036] An acquisition module, configured to acquire first speech data collected by a microphone and acquire second speech data collected by a bone conduction sensor;

[0037] A prediction module, configured to input the speech data in the first frequency band in the first speech data and the speech data in the second frequency band in the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data;

[0038] Wherein, the first frequency band is greater than the second frequency band; the speech fusion noise reduction network is pre-trained by using the microphone noisy speech data and the bone conduction noisy speech data as input data and using the microphone clean speech data corresponding to the microphone noisy speech data as a training label.

[0039] To achieve the above object, the present invention also provides a speech noise reduction device, and the speech noise reduction device includes: a memory, a processor, and a speech noise reduction program stored on the memory and executable on the processor, and when the speech noise reduction program is executed by the processor, the steps of the above-mentioned speech noise reduction method are implemented.

[0040] In addition, to achieve the above object, the present invention also proposes a computer-readable storage medium, on which a speech noise reduction program is stored, and when the speech noise reduction program is executed by a processor, the steps of the above-mentioned speech noise reduction method are implemented.

[0041] In the present invention, by pre - adopting microphone noisy speech data and bone - conduction noisy speech data as input data, and using the corresponding microphone clean speech data of the microphone noisy speech data as training labels, a speech fusion noise reduction network is trained. Then, after obtaining the first speech data collected by the microphone and the second speech data collected by the bone - conduction sensor, the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data are input into the trained speech fusion noise reduction network for prediction to obtain the target noise - reduced speech data. Since the speech fusion noise reduction network learns through training to predict speech data with good speech effects and being clean based on the low - frequency part with less noise in the bone - conduction noisy speech data and the high - frequency part with good speech effects in the microphone noisy speech data, the predicted target noise - reduced speech data not only sounds natural but also shows a better noise reduction effect. That is, compared with only noise - reducing the speech data collected by the microphone, the speech noise reduction scheme of the present invention further improves the speech noise reduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic structural diagram of the hardware operating environment related to the solution of the embodiment of the present invention;

[0043] Figure 2 It is a schematic flowchart of the first embodiment of the speech noise reduction method of the present invention;

[0044] Figure 3 It is a schematic structural diagram of a speech fusion noise reduction network related to the embodiment of the present invention

[0045] Figure 4 It is a schematic diagram of the functional modules of the preferred embodiment of the speech noise reduction device of the present invention.

[0046] The realization, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] As Figure 1 shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment related to the solution of the embodiment of the present invention.

[0049] It should be noted that for the speech noise reduction device in the embodiment of the present invention, the speech noise reduction device can be devices such as headphones, smart phones, personal computers, servers, etc., and no specific limitation is made here.

[0050] As Figure 1As shown in the figure, the voice noise reduction device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0051] Those skilled in the art can understand that Figure 1 the device structure shown in the figure does not constitute a limitation on the voice noise reduction device, and may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0052] As Figure 1 shown in the figure, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a voice noise reduction program. The operating system is a program that manages and controls the hardware and software resources of the device and supports the operation of the voice noise reduction program and other software or programs. In Figure 1 the device shown in the figure, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 may be used to call the voice noise reduction program stored in the memory 1005 and perform the following operations:

[0053] Obtain first voice data collected by a microphone and second voice data collected by a bone conduction sensor;

[0054] Input the voice data in the first frequency band in the first voice data and the voice data in the second frequency band in the second voice data into a voice fusion noise reduction network for prediction to obtain target noise reduction voice data;

[0055] Among them, the first frequency band is greater than the second frequency band; the voice fusion noise reduction network is pre-trained by using the microphone noisy voice data and the bone conduction noisy voice data as input data and the microphone clean voice data corresponding to the microphone noisy voice data as the training label.

[0056] Further, the operation of inputting the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into the speech fusion noise reduction network to obtain the target noise-reduced speech data includes:

[0057] Perform a time-domain to frequency-domain conversion on a single frame of the first speech data to obtain the first amplitude and the first phase angle value at each frequency point;

[0058] Perform a frequency-domain to time-domain conversion on a single frame of the second speech data to obtain the second amplitude and the second phase angle value at each frequency point;

[0059] Generate target input data according to the first amplitude and the first phase angle value corresponding to each frequency point in the first frequency band, and the second amplitude and the second phase angle value corresponding to each frequency point in the second frequency band;

[0060] Input the target input data into the speech fusion noise reduction network for prediction to obtain the third amplitude and the third phase angle value at each frequency point;

[0061] Perform a frequency-domain to time-domain conversion based on the third amplitude and the third phase angle value at each frequency point to obtain a single frame of target noise-reduced speech data.

[0062] Further, the operation of generating target input data according to the first amplitude and the first phase angle value corresponding to each frequency point in the first frequency band, and the second amplitude and the second phase angle value corresponding to each frequency point in the second frequency band includes:

[0063] Normalize the first amplitude at each frequency point in the first frequency band and the second amplitude at each frequency point in the second frequency band respectively, and then splice them to obtain the first channel data;

[0064] Normalize the first phase angle value at each frequency point in the first frequency band and the second phase angle value at each frequency point in the second frequency band respectively, and then splice them to obtain the second channel data;

[0065] Use the first channel data and the second channel data as the two-channel target input data.

[0066] Further, the operation of inputting the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into the speech fusion noise reduction network to obtain the target noise-reduced speech data includes:

[0067] Input the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into the convolutional layer in the speech fusion noise reduction network for convolutional processing to obtain convolutional output data;

[0068] Input the convolutional output data into the recurrent neural network layer in the speech fusion noise reduction network for processing to obtain recurrent network output data;

[0069] Input the convolutional output data and the recurrent network output data into the upsampling convolutional layer in the speech fusion noise reduction network for upsampling convolutional processing, and obtain the target noise-reduced speech data based on the result of the upsampling convolutional processing.

[0070] Further, before the operation of inputting the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into the speech fusion noise reduction network for prediction to obtain the target noise-reduced speech data, the processor 1001 can also be used to call the speech noise reduction program stored in the memory 1005 and perform the following operations:

[0071] In one round of training, input the speech data in the first frequency band of the microphone noisy speech data and the speech data in the second frequency band of the bone conduction noisy speech data into the speech fusion noise reduction network to be trained for prediction to obtain the predicted noise-reduced speech data;

[0072] Calculate the first loss based on the speech data in the first frequency band in the predicted noise-reduced speech data and the speech data in the first frequency band in the microphone clean speech data;

[0073] Calculate the second loss based on the speech data in the second frequency band in the predicted noise-reduced speech data and the speech data in the second frequency band in the microphone clean speech data;

[0074] Perform a weighted sum of the first loss and the second loss to obtain the target loss, and update the speech fusion noise reduction network to be trained according to the target loss, so as to use the updated speech fusion noise reduction network as the basis for the next round of training;

[0075] After multiple rounds of training, use the updated speech fusion noise reduction network as the trained speech fusion noise reduction network.

[0076] Further, the operation of performing a weighted sum of the first loss and the second loss to obtain the target loss includes:

[0077] Determine the weighted weight of this round corresponding to the training round of this round of training, where the weighted weight corresponding to the second loss is larger when the training round is larger;

[0078] Perform a weighted sum of the first loss and the second loss according to the weighted weight of this round to obtain the target loss.

[0079] Further, before the operation of inputting the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into the speech fusion noise reduction network for prediction to obtain the target noise-reduced speech data, the processor 1001 can also be used to call the speech noise reduction program stored in the memory 1005 and perform the following operations:

[0080] Obtain the first background noise data collected by a microphone in a background noise environment and the first clean speech data collected in a noise isolation environment, and obtain the second background noise data collected by a bone conduction sensor in a background noise environment and the second clean speech data collected in a noise isolation environment;

[0081] Add the first noise data to the first clean speech data according to a preset signal-to-noise ratio to obtain microphone noisy speech data;

[0082] Add the second noise data to the second clean speech data according to the noise weight in the microphone noisy speech data to obtain bone conduction noisy speech data.

[0083] Based on the above structure, various embodiments of the speech noise reduction method are proposed.

[0084] Refer to Figure 2 , Figure 2 which is a schematic flowchart of the first embodiment of the speech noise reduction method of the present invention.

[0085] Embodiments of the present invention provide embodiments of the speech noise reduction method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here. In this embodiment, the execution subject of the speech noise reduction method can be devices such as headphones, personal computers, and smart phones, which are not limited in this embodiment. For the convenience of description, the execution subject is omitted in the description of each embodiment. In this embodiment, the speech noise reduction method includes:

[0086] Step S10, obtain the first speech data collected by a microphone and obtain the second speech data collected by a bone conduction sensor;

[0087] In this embodiment, the voice data collected by the bone conduction sensor is used to assist in noise reduction of the voice data collected by the microphone. For the following distinction, the voice data collected by the microphone is referred to as the first voice data, and the voice data collected by the bone conduction sensor is referred to as the second voice data. It can be understood that the first voice data and the second voice data are synchronously collected in the same environment. In a specific application scenario, the microphone and the bone conduction sensor can be set in the product for collecting voice data, for example, set in the earphone, and the specific setting position is designed according to needs. For example, the bone conduction sensor is generally set at a place where it contacts the human skull. In a specific implementation manner, the first voice data and the second voice data can be real-time collected voice data or non-real-time voice data, and specific implementation manners can be selected according to different real-time requirements for voice noise reduction in the application scenario. For example, in the process of call voice noise reduction, the voice data collected by the microphone and the bone conduction sensor can be respectively and real-time framed, and the single-frame first voice data and the single-frame second voice data are used as objects to perform real-time noise reduction processing based on the voice noise reduction scheme in this embodiment.

[0088] Step S20: Input the voice data in the first frequency band of the first voice data and the voice data in the second frequency band of the second voice data into the voice fusion noise reduction network for prediction to obtain the target noise-reduced voice data;

[0089] In this embodiment, a voice fusion noise reduction network is pre-trained. The training process uses the microphone noisy voice data and the bone conduction noisy voice data as the input data of the voice fusion noise reduction network, processes the input data based on the voice fusion noise reduction network to obtain the predicted (or estimated) voice data, and uses the microphone clean voice data corresponding to the microphone noisy voice data as the training label, and adopts the supervised training method for training. That is, the training label is used to supervise the voice data predicted by the voice fusion noise reduction network to continuously update the network parameters in the voice fusion noise reduction network, so that the voice data predicted by the voice fusion noise reduction network after updating the parameters is closer to the microphone clean voice data, and then a voice fusion noise reduction network that can predict the noise-reduced voice data based on the noisy voice data collected by the microphone and the noisy voice data collected by the bone conduction sensor is trained.

[0090] Among them, in this embodiment, the specific network layer structure of the voice fusion noise reduction network is not limited. For example, a convolutional neural network or a recurrent neural network and other network structures can be used to implement it. In the specific implementation, the microphone noisy voice data, bone conduction noisy voice data, and microphone clean voice data used for training can be obtained by playing the same voice in an experimental environment and then collecting it through a microphone and a bone conduction sensor, while the microphone clean voice data can be collected in a noise isolation environment. The number of training samples used can be set as needed and is not limited in this embodiment; it can be understood that one training sample includes one microphone noisy voice data, one bone conduction noisy voice data, and one microphone clean voice data.

[0091] It should be noted that the frequency domain of the data collected by the microphone is relatively complete, but its noise resistance ability is almost zero; while the voice data collected by the bone conduction sensor is mainly concentrated in the low-frequency part. Although it will lose the high-frequency information of the data and result in a not-so-good listening experience for the voice, its noise resistance ability is excellent and can block many types of noise. Therefore, in this embodiment, by taking advantage of the microphone and the bone conduction sensor, when inputting the microphone noisy voice data and the bone conduction noisy voice data into the voice fusion noise reduction network, the voice data in the first frequency band of the microphone noisy voice data and the voice data in the second frequency band of the bone conduction noisy voice data can be input into the voice fusion noise reduction network, and the first frequency band is set to be greater than the second frequency band. Through training, the voice fusion noise reduction network can learn how to use the low-frequency part with less noise in the bone conduction noisy voice data and the high-frequency part with good voice effect in the microphone noisy voice data to predict voice data with good voice effect and being clean. Among them, good voice effect means that the user sounds more natural.

[0092] Among them, the frequency band refers to a frequency range, and a frequency range includes multiple frequency points. The first frequency band being greater than the second frequency band means that the minimum frequency point of the first frequency band is greater than the maximum frequency point of the second frequency band. The demarcation frequency point between the first frequency band and the second frequency band can be set as needed and is not limited in this embodiment. For example, it can be set to 1KHZ. Then the first frequency band includes each frequency point above 1KHZ, and the second frequency band includes each frequency point below 1KHZ (including 1KHZ).

[0093] After obtaining the first speech data that needs to be noise-reduced and the second speech data for assisting in noise reduction, the speech data in the first frequency band in the first speech data is extracted, and the speech data in the second frequency band in the second speech data is extracted. The two types of extracted speech data are input into the trained speech fusion and noise reduction network, and the input speech data is processed through each network layer in the speech fusion and noise reduction network to obtain the noise-reduced speech data (hereinafter referred to as the target noise-reduced speech data for distinction). It can be understood that since the speech data in the first frequency band in the first speech data and the speech data in the second frequency band in the second speech data are input into the already trained speech fusion and noise reduction network for prediction to obtain the target noise-reduced speech data, the obtained target noise-reduced speech data is speech data with good speech effect and clean.

[0094] In this embodiment, by pre-using the microphone noisy speech data and the bone conduction noisy speech data as input data, and using the corresponding microphone clean speech data of the microphone noisy speech data as the training label, the speech fusion and noise reduction network is trained. Then, after obtaining the first speech data collected by the microphone and the second speech data collected by the bone conduction sensor, the speech data in the first frequency band in the first speech data and the speech data in the second frequency band in the second speech data are input into the trained speech fusion and noise reduction network for prediction to obtain the target noise-reduced speech data. Since the speech fusion and noise reduction network learns through training to predict the speech data with good speech effect and clean based on the low-frequency part with less noise in the bone conduction noisy speech data and the high-frequency part with good speech effect in the microphone noisy speech data, the predicted target noise-reduced speech data not only sounds natural but also shows a better noise reduction effect. That is, compared with only noise-reducing the speech data collected by the microphone, the speech noise reduction scheme in this embodiment further improves the speech noise reduction effect.

[0095] Further, in an implementation manner, before step S20, it further includes:

[0096] Step a, obtaining the first background noise data collected by the microphone in the background noise environment and the first clean speech data collected in the noise isolation environment, and obtaining the second background noise data collected by the bone conduction sensor in the background noise environment and the second clean speech data collected in the noise isolation environment;

[0097] In this implementation manner, to improve the noise reduction effect of the noise-reduced speech data predicted by the speech fusion and noise reduction network based on speech data with different signal-to-noise ratios, the clean speech data and the noise data are collected and mixed according to different signal-to-noise ratios to obtain the noisy speech data for training.

[0098] Specifically, background noise data (hereinafter referred to as the first background noise data) can be collected by a microphone in a background noise environment, and clean speech data (hereinafter referred to as the first clean speech data) can be collected by a microphone in a noise isolation environment. Among them, the background noise environment can be an environment where noise is played through a playback device, and the played noise can be selected according to needs to simulate various noises that may occur in a real scenario; the noise isolation environment can be an environment without noise or with very little noise. Therefore, the speech data collected in the noise isolation environment can be considered as speech data without noise, and thus can be called clean speech data. When collecting the first background noise data by a microphone in a background noise environment, background noise data (hereinafter referred to as the second background noise data) can be simultaneously collected by a bone conduction sensor; when collecting the first clean speech data by a microphone in a noise isolation environment, speech data (hereinafter referred to as the second clean speech data) can be simultaneously collected by a bone conduction sensor.

[0099] In the specific implementation, by playing different noises, multiple groups of noise data can be collected. Each group of noise data includes a first background noise data and a second background noise data. By playing different speeches, multiple groups of clean speech data can be collected. Each group of clean speech data includes a first clean speech data and a second clean speech data.

[0100] Step b: Add the first noise data to the first clean speech data according to a preset signal-to-noise ratio to obtain microphone noisy speech data.

[0101] Step c: Add the second noise data to the second clean speech data according to the noise weight in the microphone noisy speech data to obtain bone conduction noisy speech data.

[0102] Adding the first noise data in a group of noise data to the first clean speech data in a group of clean speech data according to a preset signal-to-noise ratio can obtain the microphone noisy speech data in a sample, and this first clean speech data can be used as the microphone clean speech data in this sample, that is, as the training label in this sample. Among them, the preset signal-to-noise ratio can be set according to needs.

[0103] According to the noise weight in the microphone noisy speech data in this sample, adding the second noise data in this group of noise data to the second clean speech data in this group of clean speech data according to this noise weight can obtain the bone conduction noisy speech data in this sample. Among them, the noise weight can be the ratio of the amplitude of the noise signal to the amplitude of the speech signal at the same moment.

[0104] It is understandable that by adding a set of noise data at different signal-to-noise ratios to a set of clean speech data, multiple samples with different signal-to-noise ratios can be obtained. In this embodiment, by mixing the collected clean speech data with noise data at different signal-to-noise ratios to obtain noisy speech data for training the speech fusion noise reduction network, the noise reduction effect of the noise-reduced speech data predicted by the speech fusion noise reduction network based on speech data with different signal-to-noise ratios can be improved. It can also expand the number of training samples and reduce the labor cost of collecting training samples.

[0105] Furthermore, based on the above-mentioned first embodiment, a second embodiment of the speech noise reduction method of the present invention is proposed. In this embodiment, step S20 includes:

[0106] Step S201, converting a single frame of first speech data from the time domain to the frequency domain to obtain a first amplitude and a first phase angle value of each frequency point;

[0107] In this embodiment, a single frame of first voice data can be converted from the time domain to the frequency domain to obtain an amplitude value (hereinafter referred to as the first amplitude value for distinction) and a phase angle value (hereinafter referred to as the first phase angle value for distinction) at each frequency point. The conversion from the time domain to the frequency domain can be achieved through a Fourier transform. Complex numbers can be obtained at each frequency point, and then the amplitude and phase angle values can be calculated based on the complex numbers.

[0108] Step S202: Convert the single frame of second speech data from the time domain to the frequency domain to obtain a second amplitude and a second phase angle value of each frequency point;

[0109] A single frame of second voice data is converted from the time domain to the frequency domain to obtain an amplitude value (hereinafter referred to as the second amplitude value for distinction) and a phase angle value (hereinafter referred to as the second phase angle value for distinction) at each frequency point. The conversion from the time domain to the frequency domain can be achieved through a Fourier transform. Complex numbers can be obtained at each frequency point, and then the amplitude and phase angle values can be calculated based on the complex numbers.

[0110] Step S203, generating target input data according to the first amplitude and the first phase angle value corresponding to each frequency point in the first frequency band, and the second amplitude and the second phase angle value corresponding to each frequency point in the second frequency band;

[0111] After converting the first voice data to obtain the first amplitude and first phase angle value for each frequency point, the first amplitude and first phase angle value for each frequency point within the first frequency band can be extracted. For example, if the first voice data is converted to obtain the first amplitude and first phase angle values for 120 frequency points, and the first frequency band includes the last 113 frequency points of the 120 frequency points, the first amplitude and first phase angle values for the last 113 frequency points are extracted.

[0112] After obtaining the second amplitude and the second phase angle value of each frequency point by converting the second voice data, the second amplitude and the second phase angle value of each frequency point within the second frequency band can be extracted therefrom. For example, after converting the second voice data, the second amplitude and the second phase angle value of 120 frequency points are obtained, and the second frequency band includes the first 7 frequency points among the 120 frequency points. Therefore, the second amplitude and the second phase angle value of the first 7 frequency points are extracted.

[0113] Based on the first amplitude and the first phase angle value corresponding to each frequency point within the first frequency band, and the second amplitude and the second phase angle value corresponding to each frequency point within the second frequency band, input data for the voice fusion noise reduction network (hereinafter referred to as target input data) is generated. Among them, according to the different data structures of the input data of the designed voice fusion noise reduction network, the method for generating the target input data is also different, that is, it is necessary to generate target input data that conforms to the data structure of the input data of the voice fusion noise reduction network.

[0114] Step S204, input the target input data into the voice fusion noise reduction network for prediction to obtain the third amplitude and the third phase angle value of each frequency point;

[0115] Inputting the target input data into the voice fusion noise reduction network for prediction can obtain the amplitude of each frequency point (hereinafter referred to as the third amplitude for distinction) and the phase angle value (hereinafter referred to as the third phase angle value for distinction). For example, the third amplitude and the third phase angle value of 120 frequency points can be obtained.

[0116] Step S205, perform a conversion from the frequency domain to the time domain based on the third amplitude and the third phase angle value of each frequency point to obtain a single-frame target noise-reduced voice data.

[0117] Performing a conversion from the frequency domain to the time domain on the third amplitude and the third phase angle value of each frequency point can obtain a single-frame target noise-reduced voice data. Among them, the conversion from the frequency domain to the time domain can be achieved through an inverse Fourier transform. In a specific implementation manner, when the voice fusion noise reduction network is designed to output values within the range of 0-1, the third amplitude of each frequency point within the first frequency band can be de-normalized and the third amplitude of each frequency point within the second frequency band can be de-normalized to obtain the fourth amplitude of each frequency point. The third phase angle value of each frequency point within the first frequency band can be de-normalized and the third phase angle value of each frequency point within the second frequency band can be de-normalized to obtain the fourth phase angle value of each frequency point. Then, a conversion from the frequency domain to the time domain is performed based on the fourth amplitude and the fourth phase angle value of each frequency point to obtain a single-frame target noise-reduced voice data. Specifically, when performing a conversion from the frequency domain to the time domain based on the amplitude and the phase angle value of each frequency point to obtain the noise-reduced voice data, the complex number of the frequency point can be calculated first according to the amplitude and the phase angle value of the single frequency point, and then an inverse Fourier transform is performed based on the complex numbers of each frequency point to obtain a single-frame noise-reduced voice data.

[0118] In this embodiment, by inputting the amplitude and phase angle values of each frequency point in the first frequency band of the first voice data and the amplitude and phase angle values of each frequency point in the second frequency band of the second voice data into the voice fusion noise reduction network for prediction, the voice fusion noise reduction network can not only predict accurate voice data based on the amplitude of each frequency point, but also predict voice data that sounds more natural to the user based on the phase angle value of each frequency point, thereby further improving the voice noise reduction effect.

[0119] Further, in an implementation manner, step S203 includes:

[0120] Step S2031, respectively performing normalization processing on the first amplitudes of each frequency point in the first frequency band and the second amplitudes of each frequency point in the second frequency band, and then splicing them to obtain first-channel data;

[0121] In this implementation manner, the first amplitudes of each frequency point in the first frequency band can be normalized, the second amplitudes of each frequency point in the second frequency band can be normalized, and then the first amplitudes of each frequency point in the first frequency band after normalization are spliced with the second amplitudes of each frequency point in the second frequency band after normalization to obtain the input data of one channel (hereinafter referred to as first-channel data). Among them, the splicing can specifically be vector splicing. For example, if there are 113 frequency points in the first frequency band and 7 frequency points in the second frequency band, the amplitudes of the 7 frequency points in the second frequency band are vector-spliced with the amplitudes of the 113 frequency points in the first frequency band to obtain a vector including 120 amplitudes.

[0122] Step S2032, respectively performing normalization processing on the first phase angle values of each frequency point in the first frequency band and the second phase angle values of each frequency point in the second frequency band, and then splicing them to obtain second-channel data;

[0123] The first phase angle values of each frequency point in the first frequency band can be normalized, the second phase angle values of each frequency point in the second frequency band can be normalized, and then the first phase angle values of each frequency point in the first frequency band after normalization are spliced with the second phase angle values of each frequency point in the second frequency band after normalization to obtain the input data of one channel (hereinafter referred to as second-channel data). Among them, the splicing can specifically be vector splicing. For example, if there are 113 frequency points in the first frequency band and 7 frequency points in the second frequency band, the phase angle values of the 7 frequency points in the second frequency band are vector-spliced with the phase angle values of the 113 frequency points in the first frequency band to obtain a vector including 120 phase angle values.

[0124] Step S2033, using the first-channel data and the second-channel data as the target input data of two channels.

[0125] Use the first-channel data and the second-channel data as the target input data for the two channels.

[0126] Further, in an embodiment, during the training of the voice fusion noise reduction network, it is also possible to perform a time-domain to frequency-domain conversion on the single-frame microphone noisy voice data to obtain the fifth amplitude and the fifth phase angle value at each frequency point; perform a time-domain to frequency-domain conversion on the single-frame bone conduction noisy voice data to obtain the sixth amplitude and the sixth phase angle value at each frequency point; generate prediction input data according to the fifth amplitude and the fifth phase angle value corresponding to each frequency point in the first frequency band and the sixth amplitude and the sixth phase angle value corresponding to each frequency point in the second frequency band; input the prediction input data into the voice fusion noise reduction network for prediction to obtain the seventh amplitude and the seventh phase angle value at each frequency point; perform a frequency-domain to time-domain conversion based on the seventh amplitude and the seventh phase angle value at each frequency point to obtain the single-frame predicted noise-reduced voice data. Further, in an embodiment, during the training of the voice fusion noise reduction network, it is also possible to normalize the fifth amplitude at each frequency point in the first frequency band and the sixth amplitude at each frequency point in the second frequency band respectively and then splice them to obtain the first-channel data; normalize the fifth phase angle value at each frequency point in the first frequency band and the sixth phase angle value at each frequency point in the second frequency band respectively and then splice them to obtain the second-channel data; use the first-channel data and the second-channel data as the target input data for the two channels.

[0127] Further, based on the above first and / or second embodiments, a third embodiment of the voice noise reduction method of the present invention is proposed. In this embodiment, step S20 includes:

[0128] Step S206, input the voice data in the first frequency band of the first voice data and the voice data in the second frequency band of the second voice data into the convolutional layer in the voice fusion noise reduction network for convolutional processing to obtain convolutional output data;

[0129] In this embodiment, the voice fusion noise reduction network is set to include a convolutional layer, a recurrent neural network layer, and an upsampling convolutional layer. Among them, the convolutional layer is used to distinguish noise and voice features in the spatial range of the input voice data, mainly to solve the learning of the distribution relationship between different frequency points. The recurrent neural network layer is mainly used to perform associative memory on the input voice data in the time range, mainly to retain the information of voice features in terms of time continuity. The upsampling convolutional layer is mainly used to perform restoration on the input voice data in the spatial range, so as to output ideal clean voice data with the same size as the input. The number and size of the convolutional kernels in the convolutional layer and the upsampling convolutional layer can be set as needed and are not limited in this embodiment. The recurrent neural network can be implemented using GRU (gated recurrent neural network), LSTM (Long Short-Term Memory), etc., and is not limited in this embodiment.

[0130] After obtaining the first voice data and the second voice data, the voice data in the first frequency band of the first voice data and the voice data in the second frequency band of the second voice data are first input into the convolutional layer for convolutional processing, and the processed data is called convolutional output data for distinction.

[0131] Step S207: Input the convolutional output data into the recurrent neural network layer in the voice fusion noise reduction network for processing to obtain recurrent network output data;

[0132] Then input the convolutional output data into the recurrent neural network layer for processing, and the processed data is called recurrent network output data for distinction.

[0133] Step S208: Input the convolutional output data and the recurrent network output data into the upsampling convolutional layer in the voice fusion noise reduction network for upsampling convolutional processing, and obtain the target noise reduction voice data based on the result of the upsampling convolutional processing.

[0134] Then input the convolutional output data and the training network output data into the upsampling convolutional layer for upsampling convolutional processing, and the target noise reduction voice data can be obtained according to the processed result. In the specific implementation, when the upsampling convolutional layer is designed to output the amplitude and phase angle values of each frequency point, the target noise reduction voice data can be obtained by performing the conversion from the frequency domain to the time domain based on the amplitude and phase angle values of each frequency point. In other implementation manners, when the upsampling convolutional layer is designed to output other forms of data, the target noise reduction voice data can be obtained by performing corresponding calculations or conversions based on other forms of data.

[0135] Further, in one embodiment, in order to simplify the network size of the voice fusion noise reduction network so that the voice fusion noise reduction network can be deployed on the product side with low computing resources, it can be set that the voice fusion noise reduction network includes 2 convolutional layers, 2 GRU layers, and 2 upsampling convolutional layers. Further, in one embodiment, the voice fusion noise reduction network can be set to the network structure as shown in Figure 3 , where the activation function of each network layer selects Relu.

[0136] Further, based on the above first, second, and / or third embodiments, a fourth embodiment of the voice noise reduction method of the present invention is proposed. In this embodiment, before step S20, it further includes:

[0137] Step S30, in one round of training, input the voice data in the first frequency band in the microphone noisy voice data and the voice data in the second frequency band in the bone conduction noisy voice data into the voice fusion noise reduction network to be trained, and perform prediction to obtain predicted noise-reduced voice data;

[0138] In this embodiment, the voice fusion noise reduction network can be trained with multiple rounds of iteration. In the first round of training, the initialized voice fusion noise reduction network is updated, and in subsequent rounds of training, the voice fusion noise reduction network updated in the previous round of training is used as the basis for updating.

[0139] In one round of training, input the voice data in the first frequency band in the microphone noisy voice data and the voice data in the second frequency band in the bone conduction noisy voice data into the voice fusion noise reduction network to be trained for prediction, and the predicted voice data is called predicted noise-reduced voice data for distinction. The specific implementation manner of this step can refer to the specific implementation manner of step S20 in the above first embodiment, and will not be elaborated here.

[0140] Step S40, calculate the first loss based on the voice data in the first frequency band in the predicted noise-reduced voice data and the voice data in the first frequency band in the microphone clean voice data;

[0141] After obtaining the predicted noise-reduced voice data, the loss can be calculated based on the voice data in the first frequency band in the predicted noise-reduced voice data and the voice data in the first frequency band in the microphone clean voice data (hereinafter referred to as the first loss for distinction).

[0142] In a specific embodiment, when the predicted noise-reduced speech data is the amplitude and phase angle values of each frequency point, the microphone clean speech data can also be converted from the time domain to the frequency domain to obtain the amplitude and phase angle values of each frequency point. Then, calculate the loss between the amplitudes of each frequency point in the first frequency band of the predicted noise-reduced speech data and the amplitudes of each frequency point in the first frequency band of the microphone clean speech data, and calculate the loss between the phase angle values of each frequency point in the first frequency band of the predicted noise-reduced speech data and the phase angle values of each frequency point in the first frequency band of the microphone clean speech data. These two losses are collectively referred to as the first loss.

[0143] Step S50: Calculate a second loss based on the speech data in the second frequency band of the predicted noise-reduced speech data and the speech data in the second frequency band of the microphone clean speech data;

[0144] A loss can be calculated based on the speech data in the second frequency band of the predicted noise-reduced speech data and the speech data in the second frequency band of the microphone clean speech data (hereinafter referred to as the second loss for distinction).

[0145] In a specific embodiment, when the predicted noise-reduced speech data is the amplitude and phase angle values of each frequency point, the microphone clean speech data can also be converted from the time domain to the frequency domain to obtain the amplitude and phase angle values of each frequency point. Then, calculate the loss between the amplitudes of each frequency point in the second frequency band of the predicted noise-reduced speech data and the amplitudes of each frequency point in the second frequency band of the microphone clean speech data, and calculate the loss between the phase angle values of each frequency point in the second frequency band of the predicted noise-reduced speech data and the phase angle values of each frequency point in the second frequency band of the microphone clean speech data. These two losses are collectively referred to as the second loss.

[0146] Step S60: Perform a weighted sum of the first loss and the second loss to obtain a target loss, and update the speech fusion noise reduction network to be trained according to the target loss, so as to use the updated speech fusion noise reduction network as the basis for the next round of training;

[0147] After obtaining the first loss and the second loss, a weighted sum of the first loss and the second loss can be performed to obtain a target loss. Among them, the weighting weights used for the weighted sum can be set in advance according to needs and are not limited in this embodiment. Update the speech fusion noise reduction network to be trained according to the target loss, that is, update each network parameter in the speech fusion noise reduction network.

[0148] Step S70: After multiple rounds of training, use the updated speech fusion noise reduction network as the trained speech fusion noise reduction network.

[0149] Use the voice fusion noise reduction network updated in this round of training as the basis for the next round of training, and conduct the next round of training. After repeating this loop iteration multiple times, use the voice fusion noise reduction network updated in the last round as the trained voice fusion noise reduction network. Herein, the number of training rounds is not limited in this embodiment. For example, it can be set to stop training after reaching a certain number of rounds, or it can be set to stop training after reaching a certain training duration, or it can also be set to stop training after the voice fusion noise reduction network converges.

[0150] In this embodiment, by calculating the target loss through weighted summation of the voice data losses for the first frequency band and the second frequency band, the dominant role of the bone conduction noisy voice data in voice noise reduction during the training process of the voice fusion noise reduction network can be controlled, thereby enhancing the credibility of the low-frequency range in the bone conduction noisy voice data during the voice noise reduction process, and further improving the noise reduction effect of the voice fusion noise reduction network.

[0151] Further, in one implementation manner, the step of weighted summation of the first loss and the second loss to obtain the target loss in step S60 includes:

[0152] Step S601, determine the weighted weight for this round corresponding to the training round of this round of training, wherein the weighted weight corresponding to the second loss is larger when the training round is larger;

[0153] In this implementation manner, it can be set to dynamically adjust the weights corresponding to the first loss and the second loss during the training process.

[0154] Specifically, during a round of training, the weighted weight corresponding to the training round of this round of training can be determined (hereinafter referred to as the weighted weight for this round for distinction). In this implementation manner, there is no limitation on the method for determining the weighted weight for this round. For example, the training round of this round can be substituted into a calculation formula for calculation or substituted into a mapping table for look-up, but the weighted weight determined according to this method conforms to the rule that the weighted weight corresponding to the second loss is larger when the training round is larger. The purpose of such setting is to make the microphone noisy voice data dominant in the training at the beginning of the training, to avoid the training direction of the voice fusion noise reduction network from deviating, and after the general training direction is determined to a certain extent during the training, to make the bone conduction noisy voice data dominant in the training, so that the voice fusion noise reduction network learns how to assist the microphone noisy voice data for voice noise reduction based on the bone conduction noisy voice data, thereby enhancing the credibility of the low-frequency range in the bone conduction noisy voice data during the voice noise reduction process, and further improving the noise reduction effect of the voice fusion noise reduction network.

[0155] Step S602, perform weighted summation of the first loss and the second loss according to the weighted weight for this round to obtain the target loss.

[0156] After determining the weighted weights for this round, the weighted sum of the first loss and the second loss is calculated using the weighted weights for this round to obtain the target loss.

[0157] Further, in one embodiment, when calculating the losses for the amplitude and phase angle values in the microphone clean speech data and the predicted noise-reduced speech data respectively, the losses calculated based on the amplitude and phase angle values can be weighted and summed. The weight for weighting can be such that the weight corresponding to the amplitude is greater than the weight corresponding to the phase angle value, so that while the speech fusion noise reduction network can focus on learning the speech information carried by the amplitude based on frequency points to predict the noise-reduced speech data, it can also learn the phase angle values based on frequency points to predict the noise-reduced speech data, thereby making the finally predicted noise-reduced speech data sound more natural.

[0158] Further, in one embodiment, it is assumed that the predicted noise-reduced speech data predicted by the speech fusion noise reduction network includes the amplitude and phase angle values of 120 frequency points, and the microphone clean speech data also includes the amplitude and phase angle values of 120 frequency points. The loss calculated based on the amplitude can be expressed as:

[0159]

[0160] where L amp is the loss function constructed for the amplitude of the frequency points, preAmp i m is the amplitude of the m-th frequency point in the predicted noise-reduced speech data, i represents the sample serial number, and cleanAmp i m is the amplitude of the m-th frequency point in the microphone clean speech data; u represents the weighted weight corresponding to the second frequency band, and τ represents the weighted weight corresponding to the first frequency band.

[0161] The loss calculated based on the phase angle value can be expressed as:

[0162]

[0163] where L ang is the loss function constructed for the phase angle value of the frequency points, preAng i m is the phase angle value of the m-th frequency point in the predicted noise-reduced speech data, i represents the sample serial number, and cleanAng i m is the phase angle value of the m-th frequency point in the microphone clean speech data; u represents the weighted weight corresponding to the second frequency band, and τ represents the weighted weight corresponding to the first frequency band.

[0164] The target loss can be expressed as:

[0165] L tatal = α * L amp + β * L ang

[0166] Where α represents the weighting weight corresponding to the amplitude value, and β represents the weighting weight corresponding to the phase angle value.

[0167] The voice noise reduction solution of the embodiment of the present invention can complete the real-time fusion processing of the bone conduction voice data frame and the single microphone voice data frame at the Bluetooth chip end, that is, by inputting the frequency point amplitude and phase angle value of the bone conduction voice data frame and the single microphone voice data frame into the voice fusion noise reduction network, the amplitude and phase angle value of the frequency point of the microphone clean voice data frame can be inferred through the voice fusion noise reduction network, and then the data of the sampling points of the microphone clean voice data frame can be output through complex number calculation and inverse Fourier transform; based on the characteristics of the bone conduction voice data, the embodiment of the present invention realizes the frequency point fusion method of the bone conduction voice data frame and the single microphone voice data frame, and finely designs the structure of the voice fusion noise reduction network and its loss function, etc., which improves the real-time noise reduction performance of the bone conduction voice data and the single microphone voice data at the Bluetooth chip end to a certain extent.

[0168] In addition, the embodiment of the present invention also proposes a voice noise reduction device. Referring to Figure 4 , the voice noise reduction device includes:

[0169] An acquisition module 10, configured to acquire first voice data collected by a microphone and second voice data collected by a bone conduction sensor;

[0170] A prediction module 20, configured to input the voice data in the first frequency band in the first voice data and the voice data in the second frequency band in the second voice data into a voice fusion noise reduction network for prediction to obtain target noise reduction voice data;

[0171] Wherein, the first frequency band is greater than the second frequency band; the voice fusion noise reduction network is pre-trained by using the microphone noisy voice data and the bone conduction noisy voice data as input data and the microphone clean voice data corresponding to the microphone noisy voice data as training labels.

[0172] Furthermore, the prediction module 20 is further configured to:

[0173] Perform a time-domain to frequency-domain conversion on a single frame of the first voice data to obtain the first amplitude and the first phase angle value of each frequency point;

[0174] Perform a time-domain to frequency-domain conversion on a single frame of the second voice data to obtain the second amplitude and the second phase angle value of each frequency point;

[0175] Generate target input data according to the first amplitudes and first phase angle values corresponding to the frequency points within the first frequency band, and the second amplitudes and second phase angle values corresponding to the frequency points within the second frequency band;

[0176] Input the target input data into the voice fusion noise reduction network for prediction to obtain the third amplitudes and third phase angle values of each frequency point;

[0177] Perform a frequency-domain to time-domain conversion based on the third amplitudes and third phase angle values of each frequency point to obtain a single-frame target noise-reduced voice data.

[0178] Further, the prediction module 20 is further configured to:

[0179] Normalize the first amplitudes of the frequency points within the first frequency band and the second amplitudes of the frequency points within the second frequency band respectively, and then splice them to obtain the first-channel data;

[0180] Normalize the first phase angle values of the frequency points within the first frequency band and the second phase angle values of the frequency points within the second frequency band respectively, and then splice them to obtain the second-channel data;

[0181] Use the first-channel data and the second-channel data as the two-channel target input data.

[0182] Further, the prediction module 20 is further configured to:

[0183] Input the voice data of the first frequency band in the first voice data and the voice data of the second frequency band in the second voice data into the convolutional layer in the voice fusion noise reduction network for convolutional processing to obtain convolutional output data;

[0184] Input the convolutional output data into the recurrent neural network layer in the voice fusion noise reduction network for processing to obtain recurrent network output data;

[0185] Input the convolutional output data and the recurrent network output data into the upsampling convolutional layer in the voice fusion noise reduction network for upsampling convolutional processing, and obtain the target noise-reduced voice data based on the result of the upsampling convolutional processing.

[0186] Further, the voice noise reduction device further includes:

[0187] A training module, which is configured to input the voice data of the first frequency band in the microphone noisy voice data and the voice data of the second frequency band in the bone conduction noisy voice data into the voice fusion noise reduction network to be trained during one round of training, and perform prediction to obtain predicted noise-reduced voice data;

[0188] Calculate a first loss based on the voice data within the first frequency band in the predicted noise-reduced voice data and the voice data within the first frequency band in the microphone clean voice data;

[0189] Calculate a second loss based on the speech data in the second frequency band of the predicted noise-reduced speech data and the speech data in the second frequency band of the microphone clean speech data;

[0190] Perform a weighted sum of the first loss and the second loss to obtain a target loss, and update the speech fusion noise reduction network to be trained according to the target loss, so as to use the updated speech fusion noise reduction network as the basis for the next round of training;

[0191] After multiple rounds of training, use the updated speech fusion noise reduction network as the trained speech fusion noise reduction network.

[0192] Furthermore, the training module is also used for:

[0193] Determine the weighted weight of this round corresponding to the training round of this round, where the weighted weight corresponding to the second loss is larger when the training round is larger;

[0194] Perform a weighted sum of the first loss and the second loss according to the weighted weight of this round to obtain a target loss.

[0195] Furthermore, the acquisition module 10 is also used for:

[0196] Acquire the first background noise data collected by the microphone in the background noise environment and the first clean speech data collected in the noise isolation environment, and acquire the second background noise data collected by the bone conduction sensor in the background noise environment and the second clean speech data collected in the noise isolation environment;

[0197] Add the first noise data to the first clean speech data according to a preset signal-to-noise ratio to obtain the microphone noisy speech data;

[0198] Add the second noise data to the second clean speech data according to the noise weight in the microphone noisy speech data to obtain the bone conduction noisy speech data.

[0199] For each embodiment of the speech noise reduction device of the present invention, reference can be made to each embodiment of the speech noise reduction method of the present invention, which will not be elaborated here.

[0200] In addition, an embodiment of the present invention also proposes a computer-readable storage medium, on which a speech noise reduction program is stored. When the speech noise reduction program is executed by a processor, the steps of the above speech noise reduction method are implemented.

[0201] For each embodiment of the speech noise reduction device and the computer-readable storage medium of the present invention, reference can be made to each embodiment of the speech noise reduction method of the present invention, which will not be elaborated here.

[0202] It should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising such element.

[0203] The serial numbers of the above embodiments of the present invention are for description only and do not represent the superiority or inferiority of the embodiments.

[0204] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0205] The above are only the preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A voice noise reduction method, characterized in that, The voice noise reduction method includes the following steps: Obtain first voice data collected by a microphone and second voice data collected by a bone conduction sensor; Input the voice data in the first frequency band of the first voice data and the voice data in the second frequency band of the second voice data into a voice fusion noise reduction network for prediction to obtain target noise-reduced voice data; Wherein, the first frequency band is greater than the second frequency band; the voice fusion noise reduction network is pre-trained by using microphone noisy voice data and bone conduction noisy voice data as input data and using the microphone clean voice data corresponding to the microphone noisy voice data as a training label; In one round of training of the voice fusion noise reduction network, input the voice data in the first frequency band of the microphone noisy voice data and the voice data in the second frequency band of the bone conduction noisy voice data into the voice fusion noise reduction network to be trained for prediction to obtain predicted noise-reduced voice data; Calculate a first loss based on the voice data in the first frequency band in the predicted noise-reduced voice data and the voice data in the first frequency band in the microphone clean voice data; Calculate a second loss based on the voice data in the second frequency band in the predicted noise-reduced voice data and the voice data in the second frequency band in the microphone clean voice data; Determine the weighted weight of this round corresponding to the training round of this round of training, wherein the weighted weight corresponding to the second loss is greater when the training round is larger; Perform weighted summation on the first loss and the second loss according to the weighted weight of this round to obtain a target loss, and update the voice fusion noise reduction network to be trained according to the target loss, so as to use the updated voice fusion noise reduction network as the basis for the next round of training; After multiple rounds of training, use the updated voice fusion noise reduction network as the trained voice fusion noise reduction network.

2. The voice noise reduction method according to claim 1, wherein The step of inputting the voice data in the first frequency band of the first voice data and the voice data in the second frequency band of the second voice data into a voice fusion noise reduction network for prediction to obtain target noise-reduced voice data includes: Perform a time-domain to frequency-domain conversion on a single frame of the first voice data to obtain the first amplitude and the first phase angle value of each frequency point; Perform a time-domain to frequency-domain conversion on a single frame of the second voice data to obtain the second amplitude and the second phase angle value of each frequency point; Generate target input data according to the first amplitude and the first phase angle value corresponding to each frequency point in the first frequency band and the second amplitude and the second phase angle value corresponding to each frequency point in the second frequency band; Input the target input data into the voice fusion noise reduction network for prediction to obtain the third amplitude and the third phase angle value of each frequency point; Perform a frequency-domain to time-domain conversion based on the third amplitude and the third phase angle value of each frequency point to obtain a single frame of target noise-reduced voice data.

3. The voice noise reduction method according to claim 2, characterized in that, The step of generating target input data according to the first amplitude and the first phase angle value corresponding to each frequency point in the first frequency band and the second amplitude and the second phase angle value corresponding to each frequency point in the second frequency band includes: Normalize the first amplitudes at each frequency point within the first frequency band and the second amplitudes at each frequency point within the second frequency band respectively, and then splice them to obtain the first channel data; Normalize the first phase angle values at each frequency point within the first frequency band and the second phase angle values at each frequency point within the second frequency band respectively, and then splice them to obtain the second channel data; Use the first channel data and the second channel data as the target input data for two channels.

4. The voice noise reduction method according to claim 1, wherein The step of inputting the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data includes: Input the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into the convolutional layer in the speech fusion noise reduction network for convolutional processing to obtain convolutional output data; Input the convolutional output data into the recurrent neural network layer in the speech fusion noise reduction network for processing to obtain recurrent network output data; Input the convolutional output data and the recurrent network output data into the upsampling convolutional layer in the speech fusion noise reduction network for upsampling convolutional processing, and obtain target noise-reduced speech data based on the result of the upsampling convolutional processing.

5. The voice noise reduction method according to any one of claims 1 to 4, characterized in that, Before the step of inputting the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data, it further includes: Obtain first background noise data collected by a microphone in a background noise environment and first clean speech data collected in a noise isolation environment, and obtain second background noise data collected by a bone conduction sensor in the background noise environment and second clean speech data collected in the noise isolation environment; Add the first background noise data to the first clean speech data according to a preset signal-to-noise ratio to obtain the microphone noisy speech data; Add the second background noise data to the second clean speech data according to the noise weight in the microphone noisy speech data to obtain the bone conduction noisy speech data.

6. A voice noise reduction device, characterized in that, The speech noise reduction device includes: An acquisition module, configured to acquire first speech data collected by a microphone and second speech data collected by a bone conduction sensor; A prediction module, configured to input the speech data in the first frequency band of the first speech data and the speech data in the second frequency band of the second speech data into a speech fusion noise reduction network for prediction to obtain target noise-reduced speech data; Wherein, the first frequency band is greater than the second frequency band; the speech fusion noise reduction network is pre-trained by using the microphone noisy speech data and the bone conduction noisy speech data as input data and using the microphone clean speech data corresponding to the microphone noisy speech data as a training label. In one round of training of the voice fusion noise reduction network, the voice data in the first frequency band in the microphone noisy voice data and the voice data in the second frequency band in the bone conduction noisy voice data are input into the voice fusion noise reduction network to be trained, and predicted to obtain predicted noise-reduced voice data; Calculate a first loss based on the voice data in the first frequency band in the predicted noise-reduced voice data and the voice data in the first frequency band in the microphone clean voice data; Calculate a second loss based on the voice data in the second frequency band in the predicted noise-reduced voice data and the voice data in the second frequency band in the microphone clean voice data; Determine the weighted weight of this round corresponding to the training round of this round of training, wherein the weighted weight corresponding to the second loss is larger when the training round is larger; Perform weighted summation on the first loss and the second loss according to the weighted weight of this round to obtain a target loss, and update the voice fusion noise reduction network to be trained according to the target loss, so as to use the updated voice fusion noise reduction network as the basis for the next round of training; After multiple rounds of training, use the updated voice fusion noise reduction network as the trained voice fusion noise reduction network.

7. A voice noise reduction device, characterized in that, The voice noise reduction device includes: a memory, a processor, and a voice noise reduction program stored on the memory and executable on the processor. When the voice noise reduction program is executed by the processor, the steps of the voice noise reduction method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that, A voice noise reduction program is stored on the computer-readable storage medium. When the voice noise reduction program is executed by a processor, the steps of the voice noise reduction method according to any one of claims 1 to 5 are implemented.

Images