Noise removal method and device

Abstract

To provide a noise removal method and device (computer system) that exhibit superior performance even in a very noisy environment.SOLUTION: A computer system includes: a machine learning model 120 for restoring a phase of a voice signal in a very noisy environment to perform primary removal of noise; a machine learning model 150 which inputs an amplitude signal output from the machine learning model 120, and generates a mask-applied amplitude; and an IFFT 160 which restores a FFT coefficient from phase signals output from the machine learning model 150 and the machine learning model 120.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments relate to a method and apparatus for removing noise from a noisy input audio signal, and more particularly to a method and apparatus for removing noise by simultaneously utilizing information in the time domain and the frequency domain. [Background technology]

[0002] In recent years, there has been growing interest in the development and provision of Internet telephony such as VoIP and other content that utilizes voice or sound signals, and as a result, there has also been growing interest in technologies for removing noise from voice signals.

[0003] Existing deep learning noise reduction technologies have been researched using methods to remove noise components in the frequency domain or to infer clean speech in the time domain, but the performance of these existing methods has limitations.

[0004] First, there are two methods for removing noise in the frequency domain: one that adjusts only the frequency amplitude, and one that adjusts both the amplitude and phase. The method of adjusting only the frequency amplitude performs well in low-noise environments, but in noisy environments, the phase information of the noise becomes dominant over the phase information of the voice, so even if the amplitude noise component is removed, the phase noise component remains, making it impossible to restore a clean sound.

[0005] To solve this problem, a method of adjusting the amplitude and phase simultaneously was tried, but due to the random nature of the phase, phase learning did not work properly, and the overall performance was not good.

[0006] To overcome the drawbacks of frequency-domain techniques, a method for inferring clean speech in the time domain was developed. This method automatically adjusts the phase and amplitude of frequency components to generate clean raw speech signals. However, because high-frequency components exhibit random characteristics in the time domain, learning of high-frequency signals does not work properly, resulting in the problem of white noise remaining after noise removal.

[0007] The information provided above is merely intended to aid in the understanding of the present invention and may include material that does not form part of the prior art, or that the prior art would suggest to one of ordinary skill in the art. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Korean Patent Publication No. 10-2018-0067608 Summary of the Invention [Problem to be solved by the invention]

[0009] To solve the problems of the method of removing noise in the frequency domain and the method of inferring clean voice in the time domain, we provide a noise removal method and device that can demonstrate excellent performance even in a noisy environment by configuring networks in the frequency domain and the time domain respectively so that information in the frequency domain and information in the time domain can be simultaneously utilized and adjusted, and by training the two networks to demonstrate excellent performance in a complementary manner. [Means for solving the problem]

[0010] A method for removing noise using a computer system is provided, comprising: inputting a noisy input audio signal to a first network of a U-Net structure trained to infer noise-removed audio in the time domain, thereby restoring the phase and generating a first audio signal from which noise has been primarily removed; applying a first window to the first audio signal; performing a Fast Fourier Transform (FFT) on the first windowed first audio signal to obtain an amplitude signal and a phase signal; inputting the amplitude signal to a second network of a U-Net structure trained to estimate a mask to be applied to the amplitude signal, thereby obtaining the mask to be applied to the amplitude signal as an output of the second network; applying the obtained mask to the amplitude signal; performing an Inverse Fast Fourier Transform (IFFT) on the first windowed first audio signal using the masked amplitude signal and the phase signal to generate a second audio signal from which noise has been primarily removed; and applying a second window to the second audio signal.

[0011] A computer system for noise removal includes at least one processor configured to execute computer-readable instructions, wherein the at least one processor inputs a noisy input audio signal to a first network of a U-Net structure trained to infer noise-removed audio in the time domain, thereby generating a first audio signal with its phase restored and noise removed to a first order; applies a first window to the first audio signal; performs a Fast Fourier Transform (FFT) on the first windowed first audio signal to obtain an amplitude signal and a phase signal; inputs the amplitude signal to a second network of a U-Net structure trained to estimate a mask to be applied to the amplitude signal; obtains the mask to be applied to the amplitude signal as an output of the second network; applies the obtained mask to the amplitude signal; and performs an Inverse Fast Fourier Transform on the first windowed first audio signal using the masked amplitude signal and the phase signal to generate a second audio signal with noise removed to a second order; and applies a second window to the second audio signal. [Effects of the Invention]

[0012] To solve the problems of the method of removing noise in the frequency domain and the method of inferring clean voice in the time domain, we constructed networks in the frequency domain and the time domain so that information in the frequency domain and time domain can be simultaneously utilized and adjusted. By training the two networks to complement each other and perform well, we can achieve excellent performance even in noisy environments. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 illustrates a method for processing a noisy audio signal to recover a denoised audio signal in one embodiment. [Figure 2] FIG. 1 illustrates the architecture of a computer system for processing a noisy audio signal to remove the noise in one embodiment. [Figure 3] 1 is a flowchart illustrating a method for removing noise, according to one embodiment. [Figure 4] 1 is a flowchart illustrating a method for utilizing a machine learning model to obtain a mask to be applied to an amplitude signal, in one example. [Figure 5] 1 is a flowchart illustrating a method for determining parameters to be input into a machine learning model when using the machine learning model to generate a mask to be applied to a first amplitude signal in a first frequency band of the amplitude signal, in one example. [Figure 6] 1 is a flowchart illustrating a method for restoring a denoised speech signal using an input speech signal and a mask from a machine learning model, in one example. [Figure 7] FIG. 1 illustrates a mask estimated by a machine learning model in one example. [Figure 8] FIG. 2 illustrates an example of a first network in one example. [Figure 9] FIG. 10 illustrates an example of a second network in one example. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments will be described in detail with reference to the accompanying drawings, in which the same reference numerals in the various drawings denote the same elements.

[0015] FIG. 1 illustrates a method for processing a noisy audio signal to recover a denoised audio signal, in one embodiment.

[0016] Referring to FIG. 1, a method for obtaining a noise-removed restored speech signal 180 from an input speech signal 110 containing noise and a speech signal will be described.

[0017] To apply the technique in real time in a mobile environment, the operation may be performed on the input audio signal 110 in units of frames. For example, the input audio signal 110 may be a signal having a sampling rate of 16 kHz, and the length of one frame may be 480 samples. The hop size may be 240 samples, and the operation may be performed by a 50% overlap add method.

[0018] The method for removing noise from an input audio signal 110 according to an embodiment may be performed by a computer system 100 described below.

[0019] To recover the phase of a speech signal in a noisy environment and perform primary noise removal, the low waveform of the input speech signal 110 may be used as input to the machine learning model Time NS net 120. The Time NS net 120 may be a network designed with a U-Net structure. The Time NS net 120 may be implemented as a neural network (e.g., CNN, DNN, or Dense). For example, the size of each of the 12 layers may be half that of the previous layer, such as 512-256-128-64-32-16-16-32-64-128-256-512. The only important thing about the Time NS net 120 is that it uses a U-Net; the details may vary depending on tuning. The output of the Time NS net 120 is a primary estimated low speech signal, with a weak white noise component mixed into the first speech signal.

[0020] Before performing a Fast Fourier Transform (FFT) 140, a window may be applied to the first speech signal to improve the output characteristics of the FFT 140. A window must also be applied to remove noise caused by discontinuities in the overlap-add section between the previous and current frames. FIG. 1 shows that a first window 130 is applied before performing the FFT 140, and a second window 170 is applied after the IFFT 160. Many types of windows can be applied, but one example is a Kaiser-Bessel-derived (KBD) window used for time-domain aliasing cancellation (TDAC) in a Modified Discrete Cosine Transform (MDCT). Although not intended for its original purpose, the KBD window ensures that the sum of squares of the window in the overlap section satisfies 1, and it is possible to simultaneously apply a window (KBD window 130, 170) before the FFT 140 and after the IFFT 160. Applying windows in two sections in this way effectively removes noise caused by discontinuities between frames. Of the technologies that have been implemented, the best performance was recorded when applying the KBD window.

[0021] Meanwhile, an FFT 140 may be performed on the first audio signal to which the first window 130 has been applied, and an amplitude signal indicating the amplitude and a phase signal indicating the phase of the input audio signal 110 may be obtained from the first audio signal to which the FFT 140 has been performed.

[0022] As shown in the figure, in an embodiment, amplitude signals belonging to a low band (e.g., a bandwidth less than 8 kHz) may be restored using a machine learning model, Frequency NS net 150. On the other hand, amplitude signals belonging to a high band (e.g., a bandwidth equal to or greater than 8 kHz) may be segmented in Bark scale units, and the average energy of the segmented amplitude signals may be input to Frequency NS net 150 for restoration. In addition, phase signals may be used to perform IFFT 160 without any special processing. In addition, Mel Frequency Cepstral Coefficients (MFCCs) generated based on the amplitude signals belonging to the low band may be input to Frequency NS net 150 as parameters, along with the amplitude signals belonging to the low band.

[0023] The Frequency NS net 150 may be realized, for example, by a neural network (e.g., CNN, DNN, or Dense) with a U-Net structure.

[0024] The output from the Frequency NS net 150 may be a mask to be applied to the amplitude as an FFT coefficient output from the FFT 140. Noise may be secondarily removed using the amplitude to which the mask has been applied and the phase signal as an FFT coefficient output from the FFT 140. For example, the FFT coefficients of the first audio signal may be restored using the amplitude and phase signal to which the mask has been applied, and the IFFT 160 may be performed to secondarily remove noise and output the restored audio signal 120.

[0025] At this time, as described above, the second window 170 may be applied to the output of the IFFT 160 to minimize noise due to frame discontinuities.

[0026] In the embodiment, by using amplitude signals belonging to a high frequency band (compared to, for example, inputting amplitude signals belonging to a high frequency band directly to Frequency NS net 150), the amount of calculation required to remove noise can be significantly reduced.

[0027] In this way, in the embodiment, by constructing networks for the frequency domain and the time domain, and training the two networks to complement each other and provide excellent performance, it is possible to provide excellent noise removal performance even in noisy environments.

[0028] A more specific method for processing the noisy input speech signal 110 to generate the noise-removed restored speech signal 180 will be described in detail with reference to FIGS.

[0029] FIG. 2 illustrates the architecture of a computer system for processing a noisy audio signal to remove the noise, in one embodiment.

[0030] The computer system 100 shown in the figure may correspond to the computer system 100 described in FIG. 1. The computer system 100 may be an electronic device equipped with a lightweight inference model (e.g., machine learning models (Time NS net 120 and Frequency NS net 150)) for removing noise from an input audio signal 110. Alternatively, unlike the one shown in the figure, the computer system 100 may be a device for obtaining a restored audio signal 180 in which noise has been removed from the input audio signal 110 using machine learning models 120, 150 existing in an electronic device or server external to the computer system 100. In this case, the computer system 100 may obtain the input audio signal 110 by communicating with the external electronic device or server.

[0031] Computer system 100 may include, for example, a personal computer (PC), a laptop computer, a smartphone, a tablet, a wearable computer, an Internet of Things device, etc. As an example, computer system 100 may be a device such as a mobile terminal, and may not correspond to a computer device such as a high-performance PC or server.

[0032] The computer system 100 may include a communication unit 210 and a processor 220. The computer system 100 may include a microphone 230 for receiving an input audio signal 110 from a user, and may include a speaker 240 for outputting a restored audio signal 180 from which noise has been removed. The microphone 230 may generate an audio signal from audio input from the user or from the outside, and the speaker 240 may be configured to output the audio signal.

[0033] Although not shown, computer system 100 may further include a display for displaying information input by a user and / or information or content provided at the user's request.

[0034] The communication unit 210 may be a device that allows the computer system 100 to communicate with other servers or other devices. In other words, the communication unit 210 may be a hardware module, such as a network interface card, a network interface chip, or a networking interface port of the computer system 100, and / or a software module, such as a network device driver or a networking program, that transmits or receives data and / or information from other servers or other devices.

[0035] The processor 220 may manage the components of the computer system 100 and may execute programs or applications used by the computer system 100. For example, the processor 220 may acquire an audio signal 105 input to the microphone 230 or input in advance, process the input audio signal 110 using the machine learning models 120, 150, and generate a restored audio signal 180 in which noise has been removed from the input audio signal 110. The processor 220 may process calculations necessary to execute programs or applications and process data required to perform the operations described above. The processor 220 may be at least one processor of the computer system 100 or at least one core within a processor.

[0036] Although not shown, the computer system 100 may include memory. The memory is a computer-readable recording medium and may include random access memory (RAM), read-only memory (ROM), and a persistent mass storage device such as a disk drive. Here, the ROM and the persistent mass storage device may be included as separate persistent storage devices distinct from the memory. The memory may also store an operating system and at least one program code. Such software components may be loaded from a computer-readable recording medium separate from the memory. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD or CD-ROM drive, or a memory card. In other embodiments, the software components may be loaded into the memory through a communication unit 210 that is not a computer-readable recording medium. Here, the machine learning models 120 and 150 may be substantially software modules driven by the processor 220 according to program code loaded into the memory.

[0037] The processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory or the communication unit 210. For example, the processor 220 may be configured to execute instructions received according to program code loaded into the memory. Through such operation by the processor 220, the computer system 100 may generate a restored audio signal 180 in which noise has been removed from the input audio signal 110.

[0038] For example, processor 220 may input a noisy input audio signal to a first network having a U-Net structure trained to infer clean audio in the time domain to generate a first audio signal with its phase restored and noise removed, apply a first window to the first audio signal, and perform an FFT on the first windowed audio signal to obtain an amplitude signal and a phase signal. Processor 220 may also input the amplitude signal to a second network having a U-Net structure trained to estimate a mask to be applied to the amplitude signal, obtain the mask to be applied to the amplitude signal as an output of the second network, apply the obtained mask to the amplitude signal, and perform an IFFT on the first windowed audio signal using the masked amplitude signal and phase signal to generate a second audio signal with noise removed, and apply a second window to the second audio signal to generate a restored input signal with noise removed.

[0039] The Time NS net 120 may be a machine learning model pre-trained to recover the phase and perform first-order noise removal to infer clean speech in the time domain for the input speech signal 110, and the Frequency NS net 150 may be a model pre-trained to estimate a mask to be applied to the input speech signal 110 to remove noise contained in the input speech signal 110. The Time NS net 120 and the Frequency NS net 150 may be trained using a set of multiple training input speech signals for which the correct answer is known. Meanwhile, each of the Time NS net 120 and the Frequency NS net 150 may be realized with a U-Net structure based on a neural network (e.g., CNN, DNN, or Dense). The machine learning model may include multiple layers of the U-Net structure that constitute a neural network.

[0040] The method of using computer system 100 to process noisy input speech signal 110 to produce noise-removed reconstructed speech signal 180 is described in more detail below.

[0041] In the detailed description that follows, for convenience of explanation, operations performed by the configuration of computer system 100 or processor 220, or operations performed by an application or program executed by computer system 100 or processor 220, will be described as operations performed by computer system 100.

[0042] FIG. 3 is a flow chart illustrating a method for removing noise, in one embodiment.

[0043] In step 310, the computer system 100 may generate a first audio signal from which the phase has been restored and the noise has been primarily removed by inputting the noisy input audio signal to a first network having a U-Net structure trained to infer clean audio (e.g., audio from which noise has been removed) in the time domain. Here, the input audio signal may correspond to the input audio signal 110 described in FIG. 1 and may be an audio signal input to the computer system 100 via the microphone 230 described in FIG. 2, an audio signal pre-stored in the computer system 100, or an audio signal received by the computer system 100 from an external source via a network. The noise contained in the input audio signal may be noise contained in the amplitude signal and / or phase signal and may represent noise other than the audio signal that the user wants to hear.

[0044] As described above, the first network may correspond to a machine learning model such as Time NS net 120 and may be pre-trained to restore the phase of the input audio signal and perform primary noise removal using a U-Net structure. The first audio signal output in step 310 may include a weak white noise component.

[0045] In step 320, the computer system 100 may apply a first window to the first audio signal. As mentioned above, the first window may include, but is not limited to, a KBD window used for TDAC in MDCT.

[0046] In step 330, the computer system 100 may perform an FFT (Fast Fourier Transform) on the first windowed first audio signal to obtain an amplitude signal and a phase signal. As an example, the computer system 100 may also perform a 512 FFT on the first windowed first audio signal.

[0047] In step 340, the computer system 100 may input the amplitude signal to a second network having a U-Net structure trained to estimate a mask to be applied to the amplitude signal, and obtain the mask to be applied to the amplitude signal as the output of the second network. For example, the computer system 100 may extract the amplitude component from the output of the 512 FFT and use the 256 amplitude data as input to the second network. Here, the second network may correspond to a machine learning model such as a Frequency NS net 150 and may be trained to estimate a mask to be applied to the input amplitude signal. A specific method for obtaining the mask to be applied to the amplitude signal will be described in more detail with reference to FIGS. 4 and 5.

[0048] The amplitude signal of the input audio signal (and / or parameters derived from the amplitude signal) may serve as input parameters for performing inference in the second network.

[0049] In step 350, the computer system 100 may apply the obtained mask to the amplitude signal. For example, the computer system 100 may obtain a frequency-domain noise-removed amplitude signal by applying (e.g., multiplying) the mask output by the second network to the amplitude signal of the first windowed first audio signal.

[0050] In step 360, the computer system 100 may perform an IFFT on the first windowed first audio signal using the masked amplitude and phase signals to generate a second audio signal from which noise has been secondarily removed. A method for obtaining the second audio signal will be described in more detail with reference to FIG. 6.

[0051] In step 370, the computer system 100 may apply a second window to the second speech signal. As described above, the first window may include, but is not limited to, a KBD window used for TDAC in MDCT. The first and second windows may be used to minimize noise caused by frame discontinuities that occur when frame-by-frame operations are performed to apply the technique to the input speech signal in real time in a mobile environment.

[0052] FIG. 4 is a flow chart illustrating a method for obtaining a mask to be applied to an amplitude signal using a machine learning model, in one example.

[0053] A specific method for obtaining a mask to be applied to an amplitude signal will now be described with reference to steps 410 to 440 of FIG.

[0054] In step 410, the computer system 100 may input a first amplitude signal of a first frequency band among the amplitude signals of the first audio signal to which the first window has been applied to the second network, and may obtain a first mask to be applied to the first amplitude signal as an output of the second network. The first amplitude signal of the first frequency band may represent an amplitude signal corresponding to a low band among the amplitude signals of the first audio signal to which the first window has been applied. For example, the second frequency band may represent a band from 0 to less than 8 kHz, and the first amplitude signal may represent an amplitude signal of such a band from 0 to less than 8 kHz.

[0055] The first amplitude signal (and / or parameters derived from the first amplitude signal) may be input parameters for performing inference in a second network, and a first mask output by the second network may be applied to (e.g., multiplied with) the first amplitude signal to obtain a noise-removed amplitude signal (i.e., a noise-removed first amplitude signal).

[0056] Here, the first mask may be an ideal ratio mask (IRM) for the first amplitude signal. As described above, such a first mask may be applied to the first amplitude signal by being multiplied by the first amplitude signal.

[0057] In step 420, the computer system 100 may classify a second amplitude signal of a second frequency band, which is a frequency band higher than the first frequency band, among the amplitude signals of the first audio signal to which the first window has been applied, as amplitude signals of a plurality of bandwidth sections. The second amplitude signal of the second frequency band may represent an amplitude signal corresponding to a high frequency band among the amplitude signals of the first audio signal to which the first window has been applied. For example, the first frequency band may represent a band between 8 kHz and 16 kHz, and the first amplitude signal may represent an amplitude signal of such a band between 0 kHz and 8 kHz.

[0058] For example, the computer system 100 may divide the second frequency band of the second amplitude signal by at least one bark scale unit, thereby dividing the second amplitude signal into amplitude signals of multiple bandwidth sections.

[0059] The Bark scale may be a psychoacoustic scale that relates to sound characteristics such as amplitude, pitch, duration, and timbre that humans can distinguish with their hearing organs, and may be a scale for distinguishing between different sounds to specifically represent sound characteristics.

[0060] According to psychoacoustic theory, a user perceives amplitude signals in a high frequency band (e.g., a bandwidth of 8 to 16 kHz) as having low resolution, and therefore amplitude signals corresponding to such high frequency audio signals are divided into amplitude signals of a plurality of bandwidth sections by dividing them into Bark scale units. The plurality of bandwidth sections may be, for example, 8000 to 9600 Hz, 9600 to 12000 Hz, and 12000 to 15600 Hz according to the Bark scale, and the amplitude signals corresponding to the high frequency audio signals may be divided into amplitude signals of the above-mentioned respective bandwidth sections.

[0061] In step 430, the computer system 100 may calculate the average energy of each of the amplitude signals divided in step 420. For each divided amplitude signal, the computer system 100 may calculate the average energy (i.e., the average frequency energy) in the bandwidth section to which each amplitude signal belongs.

[0062] In step 440, the computer system 100 may input the average energy calculated in step 430 to a second network and obtain as an output of the second network a second mask to be applied to the second amplitude signal.

[0063] In this manner, the average energy calculated in step 430 may be an input parameter for performing inference in the second network. Applying (e.g., multiplying) the second mask output by the second network to the second amplitude signal results in a noise-removed amplitude signal (i.e., a noise-removed second amplitude signal).

[0064] The second mask may be an IRM for the average energy calculated in step 430. As described above, such a second mask may be applied to the second magnitude signal by being multiplied by the second magnitude signal.

[0065] In an embodiment, as in step 410, the low-band amplitude signal is input to a second network (i.e., the low-band amplitude signal becomes an input parameter for the second network), and noise removal is performed by inference by the second network.

[0066] However, as in steps 420 to 440, for the high-band amplitude signal, the average energy of each of the amplitude signals divided into multiple bandwidth sections is calculated, and the calculated average energy is used as an input parameter for the second network, thereby reducing the amount of calculation of the second network for the high-band amplitude signal.

[0067] FIG. 5 is a flowchart illustrating an example method for determining parameters to be input into a machine learning model when using the machine learning model to generate a mask to be applied to a first amplitude signal in a first frequency band of the amplitude signal.

[0068] A method for determining parameters to be input to the second network based on a first amplitude signal of a first frequency band corresponding to a low band among amplitude signals will be described with reference to steps 510-1 to 520-2 of FIG.

[0069] At step 510-1, the computer system 100 may calculate a predetermined number of MFCCs based on a first amplitude signal in a first frequency band.

[0070] At step 520-1, the computer system 100 may input the calculated MFCCs into the machine learning model 110 to obtain a first mask to be applied to the first amplitude signal.

[0071] That is, the MFCCs for the first amplitude signal, along with the first amplitude signal, may be input parameters for performing inference in the second network. A predetermined number (e.g., 20) of coefficients for the first amplitude signal may be calculated and input to the second network. The MFCCs may provide information about the shape of the first amplitude signal across frequencies.

[0072] The MFCCs may be coefficients necessary for feature vectorizing an audio signal. For example, the MFCCs may be features of a first amplitude signal.

[0073] The MFCCs may be calculated (extracted) from the first amplitude signal based on the Mel-scale, which takes into account the characteristics of the cochlea, which recognizes audio signals in a relatively low frequency band well but not in a high frequency band well. The first amplitude signal may be divided into a plurality of sections using the Mel-scale, and the MFCCs may be calculated for each section.

[0074] At step 510-2, the computer system 100 may calculate a ZCR (zero crossing rate) based on the first amplitude signal in the first frequency band.

[0075] At step 520-2, the computer system 100 may input the calculated ZCR into a second network to obtain a first mask to be applied to the first amplitude signal.

[0076] That is, the ZCR for the first amplitude signal, together with the first amplitude signal, may be an input parameter for performing inference in the second network. The ZCR may be calculated by analyzing the first amplitude signal in a time domain. The ZCR may provide information about noise contained in the time domain component of the first amplitude signal. The ZCR may indicate the sign change rate of the (audio) signal, i.e., the rate at which the signal changes sign. That is, the ZCR may indicate the rate at which the signal changes sign as it passes through zero.

[0077] The computer system 100 may input the first amplitude signal, the MFCC for the first amplitude signal, and the ZCR for the first amplitude signal into a second network, and may obtain a first mask applied to the first amplitude signal as an output of the second network.

[0078] FIG. 6 is a flow chart illustrating a method for restoring a denoised speech signal using an input speech signal and a mask from a machine learning model, in one example.

[0079] A method for restoring a noise-removed audio signal will now be described in detail with reference to steps 610 to 640 of FIG.

[0080] In step 610, the computer system 100 may estimate a noise-removed amplitude signal by multiplying the amplitude signal of the first audio signal to which the first window has been applied by the mask applied to the amplitude signal. For example, as described above, the computer system 100 may estimate, as the noise-removed amplitude signal, an amplitude signal obtained by multiplying the first amplitude signal by the first mask obtained in step 410 and by multiplying the second amplitude signal by the second mask obtained in step 440. The noise-removed first amplitude signal may be estimated by multiplying the first mask by the first amplitude signal, and the noise-removed second amplitude signal may be estimated by multiplying the second mask by the second amplitude signal.

[0081] In step 620, the computer system 100 may restore FFT coefficients of the restored audio signal from which noise has been removed, based on the noise-removed amplitude signal and the phase signal obtained in step 330. In other words, the computer system 100 may restore FFT coefficients of the restored audio signal from which noise has been removed, based on the noise-removed amplitude signal obtained in step 610 and the noise-removed phase signal obtained in step 330 (the phase signal from which noise has been removed by the first network in step 310), for the first audio signal to which the first window has been applied.

[0082] In step 630, the computer system 100 may generate a restored audio signal from which noise has been removed by performing an IFFT based on the restored FFT coefficients. The restored audio signal generated in step 630 may be a second audio signal before the second window is applied compared to the restored audio signal 180 in Fig. 1, or may become the restored audio signal 180 in Fig. 1 by applying the second window in step 370.

[0083] The reconstructed audio signal 180 may be output from, for example, a speaker 240 in the computer system 100 .

[0084] FIG. 7 illustrates a mask estimated by a machine learning model in one example.

[0085] As shown in Figure 7 ~ <d> Each of the figures shows an example of a mask (IRM or CIRM) estimated by inference using the second network.< / d> ~ <d>may represent the optimal value estimated by the second network.

[0086] The amplitude signal of the first audio signal to which the first window has been applied is multiplied by the mask, so that noise contained in the first audio signal to which the first window has been applied is suppressed.

[0087] As shown in the figure< / d> ~ <d>In the above, for example, the x-axis may represent frequency (or time) and the y-axis may represent a value by which the first windowed first audio signal is multiplied.

[0088] The shape and amplitude of the values ​​of the mask estimated by the second network may differ from those indicated by the input parameters to the second network and the results of the estimation by the second network described above.

[0089] FIG. 8 illustrates an example of a first network, and FIG. 9 illustrates an example of a second network. As described above, the first network and the second network may each be a machine learning model with a U-Net structure. As shown in FIG. 8, the U-Net structure is characterized in that the size of each layer is half that of the previous layer, e.g., 512-256-128-64-32-16-16-32-64-128-256-512, and the shape of such layers represents a "U." FIG. 8 illustrates an example in which the Time NS net 120 described in FIG. 1 is realized using a total of 12 CNN layers, and FIG. 9 illustrates an example in which the Frequency NS net 150 described in FIG. 1 is realized using a total of six Dense layers. This is merely one embodiment of configuring the first network and the second network, and the present invention is not limited thereto.

[0090] The above-described devices may be realized using hardware components or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the OS. The processing device may also access, record, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, a single processing device may be described. However, those skilled in the art will understand that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0091] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may independently or collectively instruct the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored and executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0092] Methods according to embodiments may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. Here, the medium may continuously record a computer-executable program or temporarily record it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single piece of hardware or multiple pieces of hardware combined together. It is not limited to media directly connected to a computer system, but may also be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROMs, RAMs, flash memories, and the like, configured to record program instructions. Other examples of media include recording media or storage media managed by application stores that distribute applications, or by sites, servers, and the like that provide or distribute various software.

[0093] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments, and various modifications and variations are possible. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be replaced or substituted by other components or equivalents to achieve suitable results. [Explanation of symbols]

[0094] 100: Computer Systems 120, 150: Machine learning models 210: Communications Department 220: Processor 230:Mike 240:Speaker< / d>

Claims

1. 1. A computer system for removing noise, comprising: At least one processor implemented to execute computer-readable instructions Including, by the at least one processor The input speech signal including noise is input to a first network having a U-Net structure trained to infer speech with noise removed in the time domain, thereby generating a first speech signal with phase restored and noise removed to a first degree; applying a first window to the first audio signal; performing a fast Fourier transform (FFT) on the first windowed first audio signal to obtain an amplitude signal and a phase signal; inputting the amplitude signal into a second network of a U-Net structure trained to estimate a mask to be applied to the amplitude signal, and obtaining as an output of the second network the mask to be applied to the amplitude signal; applying the obtained mask to the amplitude signal; performing an inverse fast Fourier transform (IFFT) on the first windowed audio signal using the masked amplitude signal and the phase signal to generate a second audio signal from which noise has been secondarily removed; applying a second window to the second audio signal; Computer system.

2. 2. The computer system of claim 1, wherein at least one of the first window and the second window comprises a Kaiser-Bessel-derived (KBD) window used for time-domain aliasing cancellation (TDAC) in a modified discrete cosine transform (MDCT).

3. Obtaining, by the at least one processor, a mask to be applied to the magnitude signal, comprises: by the at least one processor a first amplitude signal in a first frequency band among the amplitude signals is input to the second network, and a first mask is obtained as an output of the second network to be applied to the first amplitude signal; dividing a second amplitude signal in a second frequency band, which is a frequency band greater than the first frequency band, into amplitude signals of a plurality of bandwidth sections; calculating an average energy of each of the divided amplitude signals; applying the calculated average energy as an input to the second network to obtain as an output of the second network a second mask to be applied to the second amplitude signal.

3. The computer system according to claim 1, further comprising:

4. 4. The computer system of claim 3, wherein the second amplitude signal is divided into amplitude signals of the plurality of bandwidth sections by dividing the second frequency band into Bark scale units.

5. the first mask is an IRM (Ideal Ratio Mask) for the first amplitude signal, the second mask is an IRM for the calculated average energy; When the at least one processor applies the obtained mask to the amplitude signal, the at least one processor multiplies the first amplitude signal by the first mask and the second amplitude signal by the second mask; 5. A computer system according to claim 3 or 4.

6. Obtaining the first mask by the at least one processor includes: by the at least one processor calculating a predetermined number of Mel Frequency cepstral coefficients (MFCCs) based on the first amplitude signal; inputting the calculated MFCC into the second network to obtain the first mask; 4. The computer system of claim 3, comprising:

7. Obtaining the first mask by the at least one processor includes: by the at least one processor Calculating a ZCR (zero crossing rate) based on the first amplitude signal; inputting the calculated ZCR into the second network to obtain the first mask; The computer system according to any one of claims 3 to 6, comprising:

8. generating, by the at least one processor, the second audio signal, by the at least one processor estimating a denoised magnitude signal by multiplying the magnitude signal with a mask applied to the magnitude signal; Reconstructing FFT coefficients of the noise-removed audio signal based on the noise-removed amplitude signal and the phase signal; performing IFFT on the restored FFT coefficients to restore the second audio signal from which noise has been secondarily removed; The computer system according to any one of claims 1 to 7, comprising:

9. 9. The computer system according to claim 1, wherein the input audio signal input to the first network is made up of a plurality of frames.

10. 1. A method for removing noise, comprising: generating a phase-restored, noise-removed first speech signal by inputting the noisy input speech signal to a first network having a U-Net structure trained to infer noise-removed speech in the time domain; applying a first window to the first audio signal; performing a fast Fourier transform (FFT) on the first windowed first audio signal to obtain an amplitude signal and a phase signal; inputting the amplitude signal into a second network of a U-Net structure trained to estimate a mask to be applied to the amplitude signal, and obtaining the mask to be applied to the amplitude signal as an output of the second network; applying the obtained mask to the amplitude signal; generating a second audio signal from which noise has been secondarily removed by performing an IFFT (Inverse Fast Fourier Transform) on the first audio signal to which the first window has been applied, using the amplitude signal to which the mask has been applied and the phase signal; applying a second window to the second audio signal. A noise removal method comprising:

11. A computer program product for causing a computer system to perform the method of claim 10.

12. A computer-readable recording medium storing the computer program according to claim 11.

Images