Denoising methods, training methods, devices, equipment, and chips implemented using neural networks

Abstract

This invention discloses a noise reduction method, training method, apparatus, device, and chip implemented using a neural network. The noise reduction method includes: step S100, acquiring the original audio signal collected by an audio device; step S200, determining the Nyquist frequency of the original audio signal; step S300, dividing the original audio signal into several frequency bands according to a preset bandwidth to obtain several frequency band signals of different frequency bands; step S400, feeding the several frequency band signals one-to-one into several sub-networks for noise reduction processing to obtain several noise-reduced signals; step S500, outputting the noise reduction result after noise reduction through the several sub-networks, the noise reduction result being a fused signal obtained by fusing the noise-reduced signal of the highest frequency band with the noise-reduced signal below that frequency band; and step S600, performing an inverse Fourier transform on the noise reduction result to obtain the noise-reduced audio signal in the time domain. This achieves noise reduction processing compatible with multiple sampling rates without significantly increasing resources.

Description

Technical Field

[0001] This invention relates to the field of audio data processing technology, specifically to a noise reduction method and training method, apparatus, device, and chip implemented by a neural network. Background Technology

[0002] In real life, we are surrounded by various kinds of noise, such as traffic noise, machinery noise, and construction noise, which greatly affect our lives. Especially during phone calls, if the environment is noisy, the other party will have difficulty hearing clearly. With the development of voice technology, voice noise reduction has been increasingly widely used, particularly in smart terminals such as Bluetooth headsets, speakers, automotive electronics, and smart home devices. Voice noise reduction algorithms enable people to hear each other clearly even in noisy environments. In particular, with the increasing application of deep learning, deep learning-based noise reduction algorithms have significantly improved noise reduction performance.

[0003] While deep learning-based speech denoising algorithms can achieve excellent noise reduction performance, the fact that many devices do not use a single sampling rate limits their application. Although networks can be trained for different sampling rates to achieve denoising with good results, terminal devices generally have limited resources. Loading different network parameters for different sampling rates requires more memory and causes inconvenience to device applications. Furthermore, model training data is generally scarce, requiring recording in specific environments using specific devices, making data collection for various sampling rates extremely time-consuming and difficult.

[0004] Therefore, traditional deep learning denoising algorithms cannot meet the application requirements of multiple sampling rates on different devices. Furthermore, training multiple sets of network parameters to cope with different sampling rates consumes more resources from the terminal device.

[0005] Therefore, how to achieve noise reduction processing compatible with multiple sampling rates without significantly increasing resources has become an urgent technical problem to be solved. Summary of the Invention

[0006] Based on the above situation, the main objective of this invention is to provide a denoising method and training method, device, equipment, and chip implemented by a neural network, so as to achieve denoising processing compatible with multiple sampling rates without significantly increasing resources.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, embodiments of the present invention disclose a noise reduction method based on a neural network, comprising:

[0009] Step S100: Acquire the raw audio signal collected by the audio device;

[0010] Step S200: Determine the Nyquist frequency of the original audio signal;

[0011] Step S300: When the Nyquist frequency is greater than the preset value, the original audio signal is divided into several frequency band signals with different frequency bands according to the preset frequency band width.

[0012] Step S400: Several frequency band signals of different frequency bands are sent into a neural network. The neural network is constructed from multiple sub-networks, so that several frequency band signals are sent into several sub-networks in a one-to-one correspondence, so that several noise-reduced signals are obtained by performing noise reduction processing on their respective frequency band signals through several sub-networks. Among them, several sub-networks correspond one-to-one with the frequency bands of several frequency band signals.

[0013] Step S500: Output the denoising result after denoising through several sub-networks, wherein the denoising result is a fused signal after fusing the denoised signal of the highest frequency band with the denoised signal below that frequency band;

[0014] Step S600: Perform an inverse Fourier transform on the noise reduction result to obtain the noise-reduced audio signal in the time domain.

[0015] Optionally, in step S300, the number of frequency band signals is determined in the following manner:

[0016] n = round(m max / m min )

[0017] Where round() means rounding up, n is the number of frequency band signals, and m max The maximum Nyquist frequency of the original audio signal, m min It is a preset value, and it is the width value of the preset bandwidth.

[0018] Optionally, in step S400, each sub-network performs noise reduction processing on its respective frequency band signal according to an algorithm that matches the sampling rate of each frequency band signal to obtain several noise-reduced signals.

[0019] Optionally, when the number of sub-networks is i+1, in step S500, the noise reduction signals of the first to the i-th sub-networks are sequentially fused to the noise reduction signal of the i+1-th sub-network to obtain the noise reduction result, where i is an integer greater than or equal to 1, and the frequency band corresponding to the i+1-th sub-network is higher than the frequency band corresponding to the first to the i-th sub-networks.

[0020] Optionally, in step S500, the denoised signals of the first to the i-th sub-networks are fused into the denoised signal of the (i+1)-th sub-network through forward inference.

[0021] Secondly, embodiments of the present invention disclose a neural network training method for audio noise reduction, wherein the neural network includes:

[0022] The noise reduction layer consists of n sub-networks, which are used to perform noise reduction processing on audio signals in n frequency bands respectively. The frequency range of the audio signals in each frequency band is different.

[0023] The fusion layer is connected to the output of each sub-network and is used to fuse the noise reduction signals of each sub-network to obtain the noise reduction result. The noise reduction signals of the 1st to the ith sub-network are fused to the noise reduction signal of the (i+1)th sub-network in sequence to obtain the noise reduction result. Here, i is an integer greater than or equal to 1 and less than or equal to n, and the frequency band corresponding to the (i+1)th sub-network is higher than the frequency band corresponding to the ith sub-network.

[0024] The noise reduction result is the audio noise reduction result of the neural network;

[0025] Training methods include:

[0026] Step S10: Obtain m training audio signals for training the neural network, wherein the m training audio signals are obtained by transforming the original audio signals that need to be compatible with the sampling rate to the frequency domain and dividing them;

[0027] Step S20: Determine the Nyquist frequencies that need to be compatible with audio signals of different sampling rates;

[0028] Step S30: Divide the training audio signal into multiple frequency bands based on each Nyquist frequency.

[0029] Step S40: The training audio signals of each frequency band are sent to the corresponding sub-networks of each frequency band to train each sub-network separately.

[0030] Optionally, step S30 includes:

[0031] Step S31: Determine the maximum Nyquist frequency m among the m training audio signals. max and the minimum Nyquist frequency m min ;

[0032] Step S32, determine the number of frequency bands as follows:

[0033] n = round(m max / m min )

[0034] Where round() means rounding up, and n is the number of frequency bands;

[0035] Step S33: Divide each training audio signal into multiple frequency bands according to the preset bandwidth.

[0036] Thirdly, embodiments of the present invention disclose a noise reduction device based on a neural network, comprising:

[0037] The raw signal acquisition module is used to acquire the raw audio signal collected by the audio device;

[0038] The frequency determination module is used to determine the Nyquist frequency of the original audio signal;

[0039] The frequency band division module is used to divide the original audio signal into several frequency bands with different frequency bands according to the preset frequency band width when the Nyquist frequency is greater than the preset value.

[0040] The noise reduction module is used to send several frequency band signals of different frequency bands into a neural network. The neural network is constructed from multiple sub-networks, so that the several frequency band signals are sent one-to-one into several sub-networks of the multiple sub-networks, so that the noise reduction signals are obtained by the noise reduction of their respective frequency band signals through the several sub-networks. Among them, the several sub-networks correspond one-to-one with the frequency bands of the several frequency band signals.

[0041] The result output module is used to output the noise reduction result after noise reduction through several sub-networks. The noise reduction result is the fused signal after fusing the noise reduction signal of the highest frequency band with the noise reduction signal below that frequency band.

[0042] The time-frequency conversion module is used to perform an inverse Fourier transform on the noise reduction result to obtain the noise-reduced audio signal in the time domain.

[0043] Optionally, in the frequency band allocation module, the number of frequency band signals is determined in the following manner:

[0044] n = round(m max / m min )

[0045] Where round() means rounding up, n is the number of frequency band signals, and m max The maximum Nyquist frequency of the original audio signal, m min It is a preset value, and it is the width value of the preset bandwidth.

[0046] Optionally, in the noise reduction module, each sub-network performs noise reduction processing on its respective frequency band signal according to an algorithm that matches the sampling rate of each frequency band signal to obtain several noise-reduced signals.

[0047] Optionally, when the number of sub-networks is i+1, in the result output module, the noise reduction signals of the 1st to ith sub-networks are sequentially fused to the noise reduction signal of the i+1th sub-network to obtain the noise reduction result, where i is an integer greater than or equal to 1, and the frequency band corresponding to the i+1th sub-network is higher than the frequency band corresponding to the 1st to ith sub-networks.

[0048] Optionally, in the result output module, the denoised signals of the first to the i-th sub-networks are fused into the denoised signal of the (i+1)-th sub-network through forward inference.

[0049] Fourthly, embodiments of the present invention disclose a neural network training device for audio noise reduction, the neural network comprising:

[0050] The noise reduction layer consists of n sub-networks, which are used to perform noise reduction processing on audio signals in n frequency bands respectively, with different sampling rates for each frequency band.

[0051] The fusion layer is connected to the output of each sub-network and is used to fuse the noise reduction signals of each sub-network to obtain the noise reduction result. The noise reduction signals of the 1st to the ith sub-network are fused to the noise reduction signal of the (i+1)th sub-network in sequence to obtain the noise reduction result. Here, i is an integer greater than or equal to 1 and less than or equal to n, and the frequency band corresponding to the (i+1)th sub-network is higher than the frequency band corresponding to the ith sub-network.

[0052] The noise reduction result is the audio noise reduction result of the neural network;

[0053] The training device includes:

[0054] The training signal acquisition module is used to acquire m training audio signals for training the neural network. The m training audio signals are obtained by transforming the original audio signals that need to be compatible with the sampling rate into the frequency domain and dividing them.

[0055] The frequency determination module is used to determine the Nyquist frequencies that need to be compatible with audio signals of different sampling rates.

[0056] The frequency band division module is used to divide the training audio signal into multiple frequency bands based on each Nyquist frequency.

[0057] The training module is used to send the training audio signals of each frequency band into the corresponding sub-network for training each sub-network separately.

[0058] Optionally, the frequency band division module includes:

[0059] The maximum and minimum frequency determination unit is used to determine the maximum Nyquist frequency m among m training audio signals. max and the minimum Nyquist frequency m min ;

[0060] The frequency band number determination unit is used to determine the number of frequency bands in the following manner:

[0061] n = round(m max / m min )

[0062] Where round() means rounding up, and n is the number of frequency bands;

[0063] The frequency band division unit is used to divide each training audio signal into multiple frequency bands with different frequency bands according to a preset frequency band width.

[0064] Fifthly, embodiments of the present invention disclose a Bluetooth audio device that uses the method disclosed in the first aspect to reduce noise in audio signals, or includes the noise reduction device based on a neural network disclosed in the third aspect.

[0065] Alternatively, the Bluetooth audio device may be a Bluetooth headset or a Bluetooth speaker.

[0066] In a sixth aspect, embodiments of the present invention disclose a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, can implement the method disclosed in the first aspect above, or implement the method disclosed in the second aspect above.

[0067] In a seventh aspect, embodiments of the present invention disclose an audio processing chip, including a processor and a memory, wherein the memory stores a computer program, and the processor is capable of executing the computational program to implement the method disclosed in the first aspect above, or to train and solidify parameters via the method disclosed in the second aspect above.

[0068] Eighthly, embodiments of the present invention disclose an integrated circuit, wherein each module in the integrated circuit is designed to implement the method disclosed in the first aspect above, or to train and then solidify parameters via the method disclosed in the second aspect above.

[0069] [Beneficial Effects]

[0070] According to an embodiment of the present invention, a noise reduction method, training method, apparatus, device, and chip implemented using a neural network are disclosed. After acquiring the original audio signal, the Nyquist frequency of the original audio signal is determined. When the Nyquist frequency is greater than a preset value, the original audio signal is divided into several frequency bands according to a preset bandwidth to obtain several frequency band signals of different frequency bands. Then, the several frequency band signals are fed one-to-one into several sub-networks, so that the respective frequency band signals are denoised by the several sub-networks to obtain several denoised signals. The neural network is composed of multiple sub-networks. The system is constructed by assigning several sub-networks to specific frequency bands of signals. When outputting the denoising result, the denoised signal from the low-frequency band is fused with the denoised signal from the highest-frequency band, resulting in a fused signal. This allows each sub-network to perform denoising on a specific frequency band signal. Without significantly increasing computational and memory resources (only adding a data fusion layer at the end of the network), it achieves denoising compatibility with multiple sampling rate audio signals. Furthermore, the denoising effect is more targeted and capable of denoising audio signals with multiple sampling rates compared to single-band audio denoising. Since it eliminates the need to load different network parameters based on different sampling rates, it reduces memory consumption and simplifies device applications.

[0071] Furthermore, training data for training neural networks is generally scarce. The training method of this application can acquire training audio signals for training neural networks at once and train each sub-network based on data at the corresponding sampling rate. This makes data acquisition and training of neural networks extremely convenient. Moreover, since there is no need to load different network parameters based on different sampling rates, this reduces memory consumption to a certain extent and also brings convenience to device applications.

[0072] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0073] The embodiments of the present invention will now be described with reference to the accompanying drawings. In the drawings:

[0074] Figure 1 This is a flowchart of a noise reduction method based on a neural network disclosed in this embodiment;

[0075] Figure 2 This is a schematic diagram of a neural network structure disclosed in this embodiment;

[0076] Figure 3 This is a flowchart of a neural network training method for audio noise reduction disclosed in this embodiment;

[0077] Figure 4 This is a schematic block diagram of a neural network training and testing method disclosed in this embodiment;

[0078] Figure 5 This is a schematic diagram of a noise reduction device based on a neural network disclosed in this embodiment;

[0079] Figure 6 This is a schematic diagram of a neural network training device for audio noise reduction disclosed in this embodiment. Detailed Implementation

[0080] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the following detailed description of the present invention, some specific details are described in detail, but well-known methods, processes, procedures, and elements are not described in detail in order to avoid obscuring the essence of the present invention.

[0081] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0082] Unless the context explicitly requires it, the words "comprising," "including," and similar terms throughout the specification and claims should be interpreted as encompassing rather than being exclusive or exhaustive; that is, meaning "including but not limited to."

[0083] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0084] To achieve noise reduction processing compatible with multiple sampling rates without significantly increasing resources, this embodiment discloses a noise reduction method based on a neural network. Please refer to [link / reference]. Figure 1 This is a flowchart of a noise reduction method based on a neural network disclosed in this embodiment. The method includes steps S100, S200, S300, S400, S500, and S600, wherein:

[0085] Step S100: Acquire the raw audio signal collected by the audio device. In a specific embodiment, the raw audio signal can be a signal collected by the audio device itself; it can also be an audio signal received from another device, such as audio data received by one party in a call from the other, or Bluetooth audio signals received through devices such as Bluetooth. Unless otherwise specified in this application, the term "acquiring the raw audio signal collected by the audio device" includes both of the above embodiments.

[0086] Generally, devices of the same type have different sampling rates, and the required sampling rate varies depending on the application scenario. The sampling rate is usually determined based on actual needs. Taking a call scenario as an example, the main sampling rates for calls are 8kHz and 16kHz. Of course, it is not limited to these two; it can be determined based on the specific product form. Of course, for some products, once the form factor is determined, the sampling rate is also determined. For example, mobile phones usually have two sampling rates: 8kHz and 16kHz. However, for a specific mobile phone, its sampling rate is fixed at 8kHz or 16kHz after leaving the factory.

[0087] Step S200: Determine the Nyquist frequency of the original audio signal. Since the sampling rates of audio devices differ, it is necessary to determine the Nyquist frequency of the original audio signal. In a specific embodiment, when the original audio signal is acquired by the audio device itself, it can determine the Nyquist frequency of the original audio signal; when receiving the original audio signal acquired by other devices, the sampling rate of the original audio signal can be determined through a flag bit during audio data parsing, thereby determining the Nyquist frequency of the original audio signal.

[0088] Taking a noise reduction method compatible with k sampling rates as an example, assuming that the sampling rates of the k signals are M1, M2...Mk, according to the Nyquist theorem, the Nyquist frequencies of these k signals are m1, m2...mk, which are all half of the sampling rate.

[0089] Step S300: When the Nyquist frequency is greater than a preset value, the original audio signal is divided into several frequency band signals of different frequency bands according to a preset bandwidth. In this embodiment, the preset value can be the Nyquist frequency corresponding to the minimum sampling rate among compatible sampling rates; the preset bandwidth can be uniform or non-uniform, preferably the Nyquist frequency range corresponding to the minimum sampling rate as the preset bandwidth. Specifically, in step S300, the number of frequency band signals is determined in the following way:

[0090] n = round(m max / m min )

[0091] Where round() means rounding up, n is the number of frequency band signals, and m max The maximum Nyquist frequency of the original audio signal, m min This is a preset value, and it represents the width of the preset bandwidth. Taking compatibility with k sampling rates as an example, the bandwidth is divided using the minimum value among m1, m2, ..., mk as the unit, and the maximum value among m1, m2, ..., mk as the maximum sampling rate. Therefore, the number of bandwidths is:

[0092] n = round(mmax / m min )

[0093] For the highest frequency band, its actual bandwidth may be smaller than the preset bandwidth.

[0094] Taking call noise reduction as an example, according to the Nyquist theorem, a signal with a sampling rate of 8 kHz corresponds to a Nyquist frequency of 4 kHz, and a signal with a sampling rate of 16 kHz corresponds to a Nyquist frequency of 8 kHz. Following the frequency band division method described above, dividing the frequency band into two bands using 4 kHz units, namely 0-4 kHz and 4-8 kHz, can be achieved. However, the sub-band division can also be adjusted according to actual circumstances, and the method is not limited to the approach described in this scheme.

[0095] It should be noted that high-frequency band signals may also include some low-frequency band signals. That is, the maximum Nyquist frequency in the high-frequency band of 4-8kHz is 8, but the existence of signals in the 0-4kHz range cannot be ruled out.

[0096] Step S400 involves feeding several frequency band signals from different frequency zones into a neural network. The neural network is constructed from multiple sub-networks, ensuring that each frequency band signal is fed into a specific sub-network in a one-to-one correspondence. In this embodiment, the neural network is constructed from multiple sub-networks, with each sub-network corresponding to a frequency band of one of the given signals. Please refer to... Figure 2 The diagram below illustrates a neural network structure disclosed in this embodiment. The noise reduction network is divided into multiple sub-networks (as shown in the dashed box). The frequency band signals 1a and 1b after the frequency band division are fed into the corresponding sub-networks 2a and 2b, respectively, so that the sub-networks 2a and 2b can perform noise reduction processing on their respective frequency band signals to obtain their respective noise-reduced signals.

[0097] To make the noise reduction process more targeted, in a specific embodiment, in step S400, each sub-network performs noise reduction on its respective frequency band signal according to an algorithm matched to the sampling rate of each frequency band signal to obtain several noise-reduced signals. That is, for example, sub-network 2a uses an algorithm more suitable for low frequencies to denoise the 0-4kHz signal, and sub-network 2b uses an algorithm suitable for higher frequencies to denoise the 4-8kHz signal.

[0098] It should be noted that this embodiment does not limit the specific type of neural network. The neural network can be a convolutional neural network, a time series network, and related extended networks, etc.

[0099] Step S500: Output the noise reduction result after noise reduction through several sub-networks. In this embodiment, the noise reduction result is a fused signal obtained by fusing the noise reduction signal of the highest frequency band with the noise reduction signals below that frequency band. That is, all noise reduction signals below the highest frequency band are fused into the noise reduction signal of the highest frequency band, thus obtaining a complete full-band noise reduction signal. Please refer to... Figure 2 For noise reduction signals with only the lowest frequency band, they can be directly output as the noise reduction result. For noise reduction signals with high frequency bands, as analyzed above, since they may also contain low frequency band audio signals, it is necessary to fuse the low frequency band noise reduction signal into the high frequency band noise reduction signal to obtain a fused signal, and use this as the noise reduction result.

[0100] In a preferred embodiment, to preserve as many features as possible, starting from the second sub-band, the features of the previous sub-bands after passing through the sub-network need to be fused together, and the final noise reduction result is output. That is, when the number of sub-networks is i+1, in step S500, the noise reduction signals of the 1st to the ith sub-network are sequentially fused to the noise reduction signal of the i+1th sub-network to obtain the noise reduction result. Here, i is an integer greater than or equal to 1, and the frequency band corresponding to the i+1th sub-network is higher than the frequency bands corresponding to the 1st to the ith sub-networks. In this embodiment, i is a variable value, which is determined based on the number of frequency bands into which the original audio signal is divided. That is, i+1 is the number of frequency band signals obtained by frequency band splitting the original audio signal.

[0101] In a specific embodiment, the fusion method can be forward inference or similar, as long as it can splice the low-frequency band noise-reduced signal to the high-frequency band noise-reduced signal. Specifically, the noise-reduced signals of the 1st to the i-th sub-networks are fused into the noise-reduced signal of the (i+1)-th sub-network through forward inference.

[0102] In this embodiment, by fusing the low-frequency noise-reduced signal into the fused signal after fusing the high-frequency noise-reduced signal, a comparable parameter sampling rate can be achieved, that is, the number of parameters is comparable. This is because although the sub-networks have their own parameters, some parameters are shared by each sub-network, which is equivalent to not adding any parameters.

[0103] Step S600: Perform an inverse Fourier transform on the noise reduction result to obtain the noise-reduced audio signal in the time domain.

[0104] This embodiment also discloses a neural network training method for audio noise reduction; please refer to [reference needed]. Figure 2The neural network includes a noise reduction layer and a fusion layer 3. The noise reduction layer includes n sub-networks 2a, 2b, etc., which are used to perform noise reduction processing on audio signals in n frequency bands respectively. The sampling rate of the audio signals in each frequency band is different. The fusion layer 3 is connected to the output of each sub-network and is used to fuse the noise reduction signals of each sub-network 2a, 2b, etc. to obtain the noise reduction result. The noise reduction signals of the 1st to the ith sub-network are fused to the noise reduction signal of the (i+1)th sub-network in sequence to obtain the noise reduction result. Here, i is an integer greater than or equal to 1 and less than or equal to n, and the frequency band corresponding to the (i+1)th sub-network is higher than the frequency band corresponding to the ith sub-network. The noise reduction result is the audio noise reduction result of the neural network.

[0105] In this embodiment, there is no restriction on the specific type of neural network. The neural network can be a convolutional neural network, a time series network, and related extended networks, etc.

[0106] Please refer to Figure 3 This is a flowchart of a neural network training method for audio noise reduction disclosed in this embodiment. The training method includes steps S10, S20, S30, and S40, wherein:

[0107] Step S10: Obtain m training audio signals for training the neural network. In this embodiment, the sampling rates of the m training audio signals are all different. In a specific embodiment, the training audio signals can be obtained in the time domain or in the frequency domain; when obtaining the training audio signals in the time domain, it is necessary to perform a Fourier transform on the signals in the time domain to convert them into signals in the frequency domain. The m training audio signals are obtained by transforming the original audio signals that need to be compatible with the sampling rate into the frequency domain and dividing them. The compatible sampling rates are M1, M2...Mm, respectively. Common audio sampling rates include 8kHz, 16kHz, 22.05kHz, 32kHz, etc.

[0108] Step S20 involves determining the Nyquist frequencies required to be compatible with audio signals at different sampling rates. For details, please refer to the description above; further elaboration will not be repeated here.

[0109] Step S30 involves dividing the training audio signal into multiple frequency bands based on each Nyquist frequency. In a specific embodiment, step S30 includes:

[0110] Step S31: Determine the maximum Nyquist frequency m among the m training audio signals. max and the minimum Nyquist frequency m min ;

[0111] Step S32, determine the number of frequency bands as follows:

[0112] n = round(mmax / m min )

[0113] Where round() means rounding up, and n is the number of frequency bands;

[0114] Step S33: Divide each training audio signal into multiple frequency bands according to the preset bandwidth.

[0115] For details, please refer to the description above, which will not be repeated here.

[0116] Step S40: The training audio signals of each frequency band are sent to the corresponding sub-networks of each frequency band to train each sub-network separately.

[0117] To facilitate understanding of the neural network disclosed in this embodiment by those skilled in the art, please refer to... Figure 4 This is a schematic block diagram of a neural network training and testing method disclosed in this embodiment, wherein the network training is as follows: Figure 4 As shown in the dashed box marked (a), the input audio data is first subjected to Fourier transform, then frequency bands are divided, and then the divided frequency band data is fed into the network for training. Figure 4 The network model shown in the dashed box in (a) is... Figure 2 In this form, training uses a mask of clean data as the target for iterative regression, ultimately completing the training of the network. Figure 2 and Figure 3 The diagram is provided as an example only and is not limited to the network form shown. The network in the diagram can be a convolutional neural network, a time series network, or related extended networks.

[0118] Speech noise reduction is performed based on the corresponding sub-networks for different frequency band inputs, specifically as follows: Figure 4 As shown in the dashed box in (b), the input audio data is first subjected to a Fourier transform, then frequency band division, and the result of the frequency band division is sent to... Figure 4 In the middle (a) dashed box, the trained network is used for forward inference, and then the output results of each sub-network are fused. The fused data is still in the frequency domain, so an inverse Fourier transform is needed to obtain the time domain data.

[0119] The noise reduction result is obtained by outputting the time-domain data.

[0120] This embodiment also discloses a noise reduction device based on a neural network. Please refer to [link / reference]. Figure 5This is a schematic diagram of a noise reduction device based on a neural network disclosed in this embodiment. The noise reduction device includes: a raw signal acquisition module 100, a frequency determination module 200, a frequency band division module 300, a noise reduction module 400, a result output module 500, and a time-frequency conversion module 600, wherein:

[0121] The raw signal acquisition module 100 is used to acquire the raw audio signal collected by the audio device; the frequency determination module 200 is used to determine the Nyquist frequency of the raw audio signal.

[0122] The frequency band division module 300 is used to divide the original audio signal into several frequency bands with different frequency bands according to a preset frequency band width when the Nyquist frequency is greater than a preset value; the noise reduction module 400 is used to send the several frequency band signals with different frequency bands into a neural network, which is constructed by multiple sub-networks, so that the several frequency band signals are sent one-to-one into several sub-networks of the multiple sub-networks, so that the several sub-networks perform noise reduction processing on their respective frequency band signals to obtain several noise-reduced signals, wherein the several sub-networks correspond one-to-one with the frequency bands of the several frequency band signals; the result output module 500 is used to output the noise reduction result after noise reduction by several sub-networks, wherein the noise reduction result is a fused signal after fusing the noise reduction signal of the highest frequency band with the noise reduction signal below that frequency band; the time-frequency conversion module 600 is used to perform an inverse Fourier transform on the noise reduction result to obtain the noise-reduced audio signal in the time domain.

[0123] In an optional embodiment, the number of frequency band signals is determined in the frequency band division module 300 in the following manner:

[0124] n = round(m max / m min )

[0125] Where round() means rounding up, n is the number of frequency band signals, and m max The maximum Nyquist frequency of the original audio signal, m min It is a preset value, and it is the width value of the preset bandwidth.

[0126] In an optional embodiment, in the noise reduction module 400, each sub-network performs noise reduction processing on its respective frequency band signal according to an algorithm that matches the sampling rate of each frequency band signal to obtain several noise-reduced signals.

[0127] In an optional embodiment, when the number of sub-networks is i+1, in the result output module 500, the noise reduction signals of the 1st to the ith sub-network are sequentially fused to the noise reduction signal of the i+1th sub-network to obtain the noise reduction result, where i is an integer greater than or equal to 1, and the frequency band corresponding to the i+1th sub-network is higher than the frequency band corresponding to the 1st to the ith sub-networks.

[0128] In an optional embodiment, in the result output module 500, the denoised signals of the first to the i-th sub-networks are fused into the denoised signal of the (i+1)-th sub-network through forward inference.

[0129] This embodiment also discloses a neural network training device for audio noise reduction, the neural network including: a noise reduction layer and a fusion layer, wherein:

[0130] The noise reduction layer consists of n sub-networks, which are used to perform noise reduction processing on audio signals in n frequency bands respectively, and the sampling rate of the audio signals in each frequency band is different;

[0131] The fusion layer is connected to the output of each sub-network and is used to fuse the noise reduction signals of each sub-network to obtain the noise reduction result. The noise reduction signals of the 1st to the ith sub-network are fused to the noise reduction signal of the (i+1)th sub-network in sequence to obtain the noise reduction result. Here, i is an integer greater than or equal to 1 and less than or equal to n, and the frequency band corresponding to the (i+1)th sub-network is higher than the frequency band corresponding to the ith sub-network.

[0132] The noise reduction result is the audio noise reduction result of the neural network;

[0133] Please refer to Figure 6 This is a schematic diagram of a neural network training device for audio noise reduction disclosed in this embodiment. The training device includes: a training signal acquisition module 10, a frequency determination module 20, a frequency band division module 30, and a training module 40, wherein:

[0134] The training signal acquisition module 10 is used to acquire m training audio signals for training the neural network, wherein the sampling rates of the m training audio signals are different; the frequency determination module 20 is used to determine the Nyquist frequency of each training audio signal; the frequency band division module 30 is used to divide the training audio signals into multiple frequency bands based on each Nyquist frequency; and the training module 40 is used to send the training audio signals of each frequency band into the corresponding sub-network for training each sub-network.

[0135] In an optional embodiment, the frequency band division module 30 includes:

[0136] Maximum and minimum frequency determination unit 31 is used to determine the maximum Nyquist frequency m among m training audio signals. max and the minimum Nyquist frequency m min ;

[0137] The frequency band number determination unit 32 is used to determine the number of frequency bands in the following manner:

[0138] n = round(m max / mmin )

[0139] Where round() means rounding up, and n is the number of frequency bands;

[0140] The frequency band division unit 33 is used to divide the frequency bands of each training audio signal according to a preset frequency band width to obtain multiple frequency bands with different frequency bands.

[0141] This embodiment also discloses a Bluetooth audio device, which can be a Bluetooth headset or a Bluetooth speaker or other device with audio processing function. In this embodiment, the Bluetooth audio device uses the method disclosed in the above embodiment to reduce noise in the audio signal, or the Bluetooth audio device includes the noise reduction device based on neural network disclosed in the above embodiment.

[0142] This embodiment also discloses a computer-readable storage medium storing a computer program thereon. When the computer program is executed, it can implement the noise reduction method based on the neural network disclosed in the above embodiment, or implement a neural network training method for audio noise reduction.

[0143] This embodiment also discloses an audio processing chip, including a processor and a memory. The memory stores a computer program, and the processor is able to execute the computational program to implement a noise reduction method based on a neural network, or to train and solidify parameters via a neural network training method for audio noise reduction.

[0144] This embodiment also discloses an integrated circuit, in which each module is designed to implement a noise reduction method based on a neural network, or to train and solidify parameters via a neural network training method for audio noise reduction.

[0145] According to an embodiment of the present invention, a noise reduction method, training method, apparatus, device, and chip implemented using a neural network are disclosed. After acquiring the original audio signal, the Nyquist frequency of the original audio signal is determined. When the Nyquist frequency is greater than a preset value, the original audio signal is divided into several frequency bands according to a preset bandwidth to obtain several frequency band signals of different frequency bands. Then, the several frequency band signals are fed one-to-one into several sub-networks, so that the respective frequency band signals are denoised by the several sub-networks to obtain several denoised signals. The neural network is composed of multiple sub-networks. The system is constructed by assigning several sub-networks to specific frequency bands of signals. When outputting the denoising result, the denoised signal from the low-frequency band is fused with the denoised signal from the highest-frequency band, resulting in a fused signal. This allows each sub-network to perform denoising on a specific frequency band signal. Without significantly increasing computational and memory resources (only adding a data fusion layer at the end of the network), it achieves denoising compatibility with multiple sampling rate audio signals. Furthermore, the denoising effect is more targeted and capable of denoising audio signals with multiple sampling rates compared to single-band audio denoising. Since it eliminates the need to load different network parameters based on different sampling rates, it reduces memory consumption and simplifies device applications.

[0146] Furthermore, training data for training neural networks is generally scarce. The training method of this application can acquire training audio signals for training neural networks at once and train each sub-network based on data at the corresponding sampling rate. This makes data acquisition and training of neural networks extremely convenient. Moreover, since there is no need to load different network parameters based on different sampling rates, this reduces memory consumption to a certain extent and also brings convenience to device applications.

[0147] In addition, the present invention also provides a computer-readable storage medium for implementing the above methods, such as a chip, an optical disc, etc., wherein an executable program is stored on the computer-readable storage medium, and when the executable program is executed, it implements the method described in any of the above claims.

[0148] It should be noted that the computer-readable storage medium described in the embodiments of this disclosure is not limited to the embodiments given above. For example, it can also be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the embodiments of this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0149] It will be understood by those skilled in the art that the above-described preferred solutions can be freely combined and superimposed without conflict. The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings; for example, two consecutively indicated blocks may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. The numbering of each step in this document is for ease of explanation and reference only and is not intended to limit the order of execution. The specific execution order is determined by the technology itself, and those skilled in the art can determine various permissible and reasonable orders based on the technology itself.

[0150] It should be noted that the use of step numbers (letters or numbers) to refer to certain specific method steps in this invention is merely for the purpose of convenience and brevity in description, and is by no means intended to restrict the order of these method steps. Those skilled in the art will understand that the order of the relevant method steps should be determined by the technology itself and should not be unduly restricted by the existence of step numbers. Those skilled in the art can determine various permissible and reasonable orderings of steps based on the technology itself.

[0151] Those skilled in the art will understand that, without conflict, the above-mentioned preferred solutions can be freely combined and superimposed.

[0152] It should be understood that the above embodiments are merely exemplary and not restrictive. Various obvious or equivalent modifications or substitutions that can be made by those skilled in the art regarding the above details without departing from the basic principles of the present invention will be included within the scope of the claims of the present invention.

Claims

1. A noise reduction method based on a neural network, characterized in that, include: Step S100: Acquire the raw audio signal collected by the audio device; Step S200: Determine the Nyquist frequency of the original audio signal; Step S300: When the Nyquist frequency is greater than a preset value, the original audio signal is split into several frequency band signals of different frequency bands according to the preset frequency band width. Step S400: The frequency band signals of several different frequency bands are sent into a neural network. The neural network is constructed from multiple sub-networks, so that the several frequency band signals are sent into several sub-networks in a one-to-one correspondence, so that the several sub-networks can perform noise reduction processing on their respective frequency band signals to obtain several noise-reduced signals. The several sub-networks correspond one-to-one with the frequency bands of the several frequency band signals. Step S500: Output the noise reduction result after noise reduction through several sub-networks, wherein the noise reduction result is a fused signal after fusing the noise reduction signal of the highest frequency band with the noise reduction signal below that frequency band; Step S600: Perform an inverse Fourier transform on the noise reduction result to obtain the noise-reduced audio signal in the time domain; When the number of the plurality of sub-networks is i+1, in step S500, the noise reduction signals of the first to the i-th sub-networks are sequentially fused to the noise reduction signal of the i+1-th sub-network to obtain the noise reduction result, wherein i is an integer greater than or equal to 1, and the frequency band corresponding to the i+1-th sub-network is higher than the frequency band corresponding to the first to the i-th sub-networks. In step S500, the denoised signals of the first to the i-th sub-networks are fused into the denoised signal of the (i+1)-th sub-network through forward inference.

2. The noise reduction method based on neural network implementation as described in claim 1, characterized in that, In step S300, the number of frequency band signals is determined in the following manner: Where round() means rounding up, n is the number of frequency band signals, and m max The maximum Nyquist frequency of the original audio signal is m. min It is the preset value, and it is also the width value of the preset bandwidth.

3. The noise reduction method based on neural network implementation as described in claim 1, characterized in that, In step S400, each sub-network performs noise reduction processing on its respective frequency band signal according to an algorithm that matches the sampling rate of each frequency band signal to obtain several noise-reduced signals.

4. A neural network training method for audio noise reduction, characterized in that, The neural network includes: The noise reduction layer consists of n sub-networks, which are used to perform noise reduction processing on audio signals in n frequency bands respectively. The frequency range of the audio signals in each frequency band is different. The fusion layer is connected to the output of each sub-network and is used to fuse the noise reduction signals of each sub-network to obtain the noise reduction result. The noise reduction signals of the 1st to the ith sub-network are fused to the noise reduction signal of the (i+1)th sub-network in sequence to obtain the noise reduction result. Here, i is an integer greater than or equal to 1 and less than or equal to n, and the frequency band corresponding to the (i+1)th sub-network is higher than the frequency band corresponding to the ith sub-network. The noise reduction result is the audio noise reduction result of the neural network; The training method includes: Step S10: Obtain m training audio signals for training the neural network, wherein the m training audio signals are obtained by transforming the original audio signals that need to be compatible with the sampling rate to the frequency domain and dividing them; Step S20: Determine the Nyquist frequencies that need to be compatible with audio signals of different sampling rates; Step S30: Divide the training audio signal into multiple frequency bands based on each Nyquist frequency. Step S40: The training audio signals of each frequency band are sent to the corresponding sub-networks of each frequency band to train each sub-network separately.

5. The neural network training method for audio noise reduction as described in claim 4, characterized in that, Step S30 includes: Step S31: Determine the maximum Nyquist frequency m among the m training audio signals. max and the minimum Nyquist frequency m min ; Step S32, determine the number of frequency bands as follows: Where round() means rounding up, and n is the number of frequency bands; Step S33: Divide each training audio signal into multiple frequency bands according to the preset bandwidth.

6. A noise reduction device based on a neural network, characterized in that, include: The raw signal acquisition module (100) is used to acquire the raw audio signal collected by the audio device; A frequency determination module (200) is used to determine the Nyquist frequency of the original audio signal; The frequency band division module (300) is used to divide the original audio signal into several frequency band signals with different frequency bands according to the preset frequency band width when the Nyquist frequency is greater than the preset value. The noise reduction module (400) is used to send the frequency band signals of the several different frequency bands into the neural network. The neural network is constructed from multiple sub-networks, so that the several frequency band signals are sent one-to-one into several sub-networks of the multiple sub-networks, so that the noise reduction signals of their respective frequency band signals are processed by the several sub-networks respectively to obtain several noise-reduced signals. The several sub-networks correspond one-to-one with the frequency bands of the several frequency band signals. The result output module (500) is used to output the noise reduction result after noise reduction through several sub-networks, wherein the noise reduction result is a fused signal after fusing the noise reduction signal of the highest frequency band with the noise reduction signal below that frequency band; The time-frequency conversion module (600) is used to perform an inverse Fourier transform on the noise reduction result to obtain the noise-reduced audio signal in the time domain; When the number of the plurality of sub-networks is i+1, in the result output module (500), the noise reduction signals of the 1st to the ith sub-network are sequentially fused to the noise reduction signal of the i+1th sub-network to obtain the noise reduction result, wherein i is an integer greater than or equal to 1, and the frequency band corresponding to the i+1th sub-network is higher than the frequency band corresponding to the 1st to the ith sub-networks; In the result output module (500), the denoised signals of the first to the i-th sub-networks are fused into the denoised signal of the (i+1)-th sub-network through forward inference.

7. The noise reduction device based on a neural network as described in claim 6, characterized in that, In the frequency band division module (300), the number of frequency band signals is determined in the following manner: Where round() means rounding up, n is the number of frequency bands, and m max The maximum Nyquist frequency of the original audio signal is m. min It is the preset value, and it is also the width value of the preset bandwidth.

8. The noise reduction device based on a neural network as described in claim 6, characterized in that, In the noise reduction module (400), each sub-network performs noise reduction processing on its respective frequency band signal according to an algorithm that matches the sampling rate of each frequency band signal to obtain several noise-reduced signals.

9. A neural network training device for audio noise reduction, characterized in that, The neural network includes: The noise reduction layer consists of n sub-networks, which are used to perform noise reduction processing on audio signals in n frequency bands respectively. The frequency range of the audio signals in each frequency band is different. The fusion layer is connected to the output of each sub-network and is used to fuse the noise reduction signals of each sub-network to obtain the noise reduction result. The noise reduction signals of the 1st to the ith sub-network are fused to the noise reduction signal of the (i+1)th sub-network in sequence to obtain the noise reduction result. Here, i is an integer greater than or equal to 1 and less than or equal to n, and the frequency band corresponding to the (i+1)th sub-network is higher than the frequency band corresponding to the ith sub-network. The noise reduction result is the audio noise reduction result of the neural network; The training device includes: The training signal acquisition module (10) is used to acquire m training audio signals for training the neural network, wherein the m training audio signals are obtained by transforming the original audio signals that need to be compatible with the sampling rate into the frequency domain and dividing them; The frequency determination module (20) is used to determine the frequency of audio signals that need to be compatible with different sampling rates; The frequency band division module (30) is used to divide the training audio signal into multiple frequency bands based on each Nyquist frequency. The training module (40) is used to send the training audio signals of each frequency band into the corresponding sub-network of the frequency band, so as to train each sub-network separately.

10. The neural network training apparatus for audio noise reduction as described in claim 9, characterized in that, The frequency band division module (30) includes: The maximum and minimum frequency determination unit (31) is used to determine the maximum Nyquist frequency m among m training audio signals. max and the minimum Nyquist frequency m min ; The frequency band number determination unit (32) is used to determine the number of frequency bands in the following manner: Where round() means rounding up, and n is the number of frequency bands; The frequency band division unit (33) is used to divide the frequency bands of each training audio signal according to the preset frequency band width to obtain multiple frequency bands with different frequency bands.

11. A Bluetooth audio device, characterized in that, Including the noise reduction device based on neural network implementation as described in any one of claims 6-8.

12. The Bluetooth audio device as described in claim 11, characterized in that, The Bluetooth audio device is a Bluetooth headset or a Bluetooth speaker.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it can implement the method as described in any one of claims 1-3, or the method as described in claim 4 or 5.

14. An audio processing chip, comprising a processor and a memory, characterized in that, The memory stores a computer program, and the processor is capable of executing the computer program to implement the method as described in any one of claims 1-3, or to train and solidify parameters via the method as described in claim 4 or 5.

15. An integrated circuit, characterized in that, Each module in the integrated circuit is designed to implement the method as described in any one of claims 1-3, or to train and then solidify parameters via the method as described in claim 4 or 5.

Images