Beamforming method and system using neural network
The described beamforming method and system optimize neural networks for reduced calculations by using Fourier transforms and phase differences, effectively separating desired sounds in noisy environments without requiring steering vectors, thus enhancing efficiency and performance.
Patent Information
- Application Number
- JP2023551942
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-11-04
- Filing Date
- 2021-09-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-09-29
AI Technical Summary
Neural networks used for beamforming require a large amount of calculations and are not optimized for minimizing parameters, leading to inefficiencies in spatial information acquisition.
A beamforming method and system utilizing a neural network that minimizes calculations by designing the neural network structure, employing Fourier transforms, phase differences, and Hadamard products to process sound signals from multiple microphones, with optional masking and gain control.
Enables efficient beamforming by restoring sound using phase differences without calculating steering vectors, reducing weight and size, and improving separation of desired sounds in noisy environments.
Smart Images

Figure 0007591848000026 
Figure 0007591848000027 
Figure 0007591848000028
Abstract
Description
[Technical field]
[0001] The present invention relates to a beamforming method and a beamforming system using a neural network. [Background technology]
[0002] The cocktail party effect is a phenomenon whereby partygoers selectively focus on and are good at listening to their interlocutor, even when they are in a room with a lot of background noise. Realizing this ability, i.e. beamforming, in a machine has been known as the cocktail party problem, and recently there have been increasing attempts to use neural networks to solve the cocktail party problem. Improving the performance of beamforming techniques is closely related to the performance of audio-related electronic products, especially hearing aids, and therefore can be a very important social issue.
[0003] Beamforming can refer to the process of reconstructing a target signal by analyzing sounds captured using two or more microphones. For this purpose, techniques that minimize the loudness of incoming sounds while satisfying given constraints, such as creating an artificial delay time for the signals coming from each microphone, have been used for some time. Recently, there has been active research into improving the performance of MVDR (minimum variance distortionless response) beamformers using neural networks, and methods for training neural network models that embody beamformers. Summary of the Invention [Problem to be solved by the invention]
[0004] The problem to be solved by the present invention is to provide a beamforming method and system using a neural network, which uses a neural network to overcome the limitation of requiring a large amount of calculations to obtain spatial information in rule-based beamforming, but can minimize the amount of parameters and calculations by designing the structure of the neural network to be optimized for beamforming. [Means for solving the problem]
[0005] A beamforming method according to an embodiment of the present invention may include the steps of receiving a first sound signal and a second sound signal using a first microphone and a second microphone disposed at a predetermined distance from the first microphone, respectively; acquiring Fourier transform results for the first sound signal and the second sound signal; acquiring a phase difference between the first sound signal and the second sound signal from the Fourier transform results; inputting the phase difference into a beamforming model using a neural processor and performing a calculation; performing a Hadamard product on a calculation result of the neural processor and a Fourier transform result for the first sound signal; and outputting the result of the Hadamard product.
[0006] The step of performing the Hadamard product may further include applying a mask to the operation result before performing the Hadamard product.
[0007] The step of performing the Hadamard product may further include performing a gain control after performing the Hadamard product.
[0008] The predetermined distance may be between 10cm and 14cm.
[0009] The beamforming method may further include training the beamforming model using the phase difference.
[0010] A beamforming system according to an embodiment of the present invention may include a first microphone for receiving a first sound signal, a second microphone disposed a predetermined distance away from the first microphone for receiving a second sound signal, a first STFT module for acquiring a Fourier transform result for the first sound signal, a second STFT module for acquiring a Fourier transform result for the second sound signal, a phase difference acquisition module for acquiring a phase difference between the first sound signal and the second sound signal from the Fourier transform result, a neural processor for receiving the phase difference and performing a neural network operation using a beamforming model, a Hadamard product module for performing a Hadamard product on a calculation result of the neural processor and a Fourier transform result for the first sound signal, and an output module for outputting the result of the Hadamard product.
[0011] The beamforming system may further include a masking module that applies a mask to the operation result before performing the Hadamard product.
[0012] The beamforming system may further include a gain control module that performs gain control after performing the Hadamard product.
[0013] The predetermined distance may be between 10cm and 14cm.
[0014] The beamforming system may further include a learning model that uses the phase difference to train the beamforming model. Effect of the Invention
[0015] According to an embodiment of the present invention, it is possible to efficiently realize beamforming by restoring a voice received by a microphone using only a phase difference without the need to calculate a steering vector and a spatial correlation matrix for various noise environments. In addition, it is possible to apply a lightweight autoencoder, which enables weight reduction and miniaturization. [Brief description of the drawings]
[0016] [Figure 1] FIG. 1 is a diagram illustrating a beamforming system according to an embodiment of the present invention.
[0017] [Diagram 2] FIG. 2 is a diagram illustrating a computing device for implementing a beamforming device according to an embodiment of the present invention.
[0018] [Diagram 3] FIG. 3 is a diagram illustrating a beamforming method according to an embodiment of the present invention.
[0019] [Figure 4] FIG. 4 is a diagram illustrating a beamforming method according to an embodiment of the present invention.
[0020] [Diagram 5] FIG. 5 is a diagram illustrating a specific example of a beamforming method according to an embodiment of the present invention.
[0021] [Figure 6] FIG. 6 is a diagram illustrating a beamforming system according to an embodiment of the present invention.
[0022] [Figure 7] FIG. 7 is a diagram illustrating a beamforming system according to an embodiment of the present invention.
[0023] [Figure 8] FIG. 8 is a diagram illustrating a beamforming system according to an embodiment of the present invention.
[0024] [Figure 9-10] 9 and 10 are diagrams illustrating advantageous effects of a beamforming method and a beamforming system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains can easily carry out the embodiments. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. In the drawings, parts that are not related to the description are omitted in order to clearly explain the present invention, and similar parts are designated by similar reference numerals throughout the specification.
[0026] Throughout the specification and claims, when a part "includes" a certain component, this does not mean excluding other components, but may further include other components, unless otherwise specified to the contrary. In addition, terms such as "unit", "device", and "module" described in the specification mean a unit that processes at least one function or operation, which may be embodied in hardware, software, or a combination of hardware and software. In addition, at least a part of the beamforming method, beamforming device, and beamforming system according to the embodiments described below may be embodied in a program or software, and the program or software may be stored in a computer-readable medium.
[0027] FIG. 1 is a diagram illustrating a beamforming system according to an embodiment of the present invention.
[0028] Referring to FIG. 1, a beamforming system 1 according to an embodiment of the present invention may include a first microphone M1, a second microphone M2, a beamforming device 10 including a connection terminal T, and a monitor 20.
[0029] The beamforming device 10 is attached to the monitor 20 and can receive sound using the microphones M1 and M2. For example, the beamforming device 10 can receive the voice of a person participating in a video conference in front of the monitor 20 using the microphones M1 and M2. In particular, the beamforming device 10 can receive the voice of a person participating in a video conference in a noisy environment.
[0030] The beamforming device 10 may perform beamforming on the sound signals received using the microphones M1 and M2, and then output the beamformed sound signals obtained as a result. For example, the beamforming device 10 may distinguish the voice of a person participating in a video conference in a noisy environment and provide the voice to another computing device (e.g., a personal computer connected to a monitor 20). Then, the computing device may provide the distinguished voice of the person to, for example, other participants in the video conference. A connection terminal T may be used for the beamforming device 10 to provide an output signal obtained by distinguishing the voice of the person to another computing device, and in this embodiment, the connection terminal T may be a USB (Universal Serial Bus) terminal, but the scope of the present invention is not limited thereto.
[0031] In this embodiment, the first microphone M1 and the second microphone M2 may be disposed at a predetermined distance D apart. For example, the first microphone M1 can receive the voice of a person participating in a video conference on a first side (e.g., the left side) and surrounding noise (i.e., a first sound signal), and the second microphone M2 can receive the voice of a person participating in a video conference on a second side (e.g., the right side) that is separated from the first microphone M1 by the predetermined distance D and surrounding noise (i.e., a second sound signal).
[0032] In this embodiment, the predetermined distance D between the first microphone M1 and the second microphone M2 may be 10 cm to 14 cm, preferably 12 cm, but the scope of the present invention is not limited thereto.
[0033] FIG. 2 is a diagram illustrating a computing device for implementing a beamforming device according to an embodiment of the present invention.
[0034] 2, a computing device for implementing the beamforming device 10 according to an embodiment of the present invention may include a processor 100, a neural processor 110, a memory 120, an output module 130, a first microphone M1, and a second microphone M2, and the computing device may also operate to perform the beamforming method according to an embodiment of the present invention. The processor 100, the neural processor 110, the memory 120, the output module 130, the first microphone M1, and the second microphone M2 may exchange data via a bus 190.
[0035] The processor 100 performs overall control over the beamforming device 10 and can perform the functions and methods described herein together with or independently of the neural processor 110. The processor 100 may be embodied in various types of processors, such as an Application Processor (AP), a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), etc., and the scope of the present invention is not limited to a particular processor.
[0036] The neural processor 110 may perform the functions and methods described herein, particularly a neural network operation. For example, the neural processor 110 may perform an operation using a beamforming model described herein. Here, the neural network may include a convolutional neural network (CNN), but the scope of the present invention is not limited thereto.
[0037] The memory 120 can store data required for the processor 100 or the neural processor 110 to perform calculations or applications to be run. It can include various types of volatile memory, such as dynamic random access memory (DRAM) and static random access memory (SRAM), but can also include non-volatile memory, such as flash memory, or a combination of volatile and non-volatile memory.
[0038] The output module 130 may include any input / output interface device for outputting the beamformed sound signal obtained as a result after the beamforming device 10 performs beamforming on the sound signals received using the microphones M1 and M2.
[0039] In addition, at least some of the beamforming method, beamforming device, and beamforming system according to the embodiments of the present invention, at least some of the key control method, key control device, and user equipment may be embodied as a program or software executed in a computing device, and the program or software may be stored in a computer-readable medium. In addition, at least some of the beamforming method, beamforming device, and beamforming system according to the embodiments of the present invention may be embodied as hardware that can be electrically connected to a computing device.
[0040] The beamforming device 10 as described in Fig. 1 may be embodied to be attached to a monitor 20 and may be coupled to another computing device to provide an output signal for distinguishing human voices. It is particularly noted that the beamforming device 10 has its own neural processor 110 and can perform neural network operations to distinguish the voices of people participating in a video conference in a noisy environment using its own neural processor 110 without using the computing resources of another computing device.
[0041] Reference is now made to Figures 3 to 5, where Figure 3 illustrates a beamforming method according to an embodiment of the present invention.
[0042] FIG. 3 is a diagram illustrating a beamforming method according to an embodiment of the present invention.
[0043] 3, in a beamforming method 30 according to an embodiment of the present invention, a first microphone M1 can receive a first sound signal S1 from a first side. For example, the first microphone M1 can receive a first sound signal S1 including a voice of a person participating in a video conference and surrounding noise from the first side and transmit the first sound signal S1 to a first STFT module 300 (represented as an L_STFT module in the drawing).
[0044] The first STFT module 300 may perform a Fourier transform operation on the first sound signal S1 received from the first microphone M1 to obtain a Fourier transform result P1 for the first sound signal S1.
[0045] Meanwhile, the second microphone M2, which is disposed at a predetermined distance D from the first microphone M1, can receive a second sound signal S2 from the second side. For example, the second microphone M2 can receive the second sound signal S2 including the voice of a person participating in the video conference and surrounding noise from the second side and transmit the second sound signal S2 to a second STFT module 301 (shown as an R_STFT module in the drawings).
[0046] The second STFT module 301 may perform a Fourier transform operation on the second sound signal S2 received from the second microphone M2 to obtain a Fourier transform result P2 on the second sound signal S2.
[0047] The phase difference acquisition module 302 can acquire the phase difference dP between the first sound signal S1 and the second sound signal S2 from the Fourier transform result P1 provided from the first STFT module 300 and the Fourier transform result P2 provided from the second STFT module 301.
[0048] The learning module 303 can train the beamforming model 304 using the phase difference dP between the first sound signal S1 and the second sound signal S2. Thus, the beamforming model 304 can be trained to perform beamforming using only the phase difference between the two sound signals received via the first microphone M1 and the second microphone M2.
[0049] During learning, the predetermined distance D between the first microphone M1 and the second microphone M2 may be 10 cm to 14 cm, preferably 12 cm, but the scope of the present invention is not limited thereto. When the predetermined distance D is 12 cm, the performance of the trained beamforming model 304 can be maintained even when the distance between the first microphone M1 and the second microphone M2 during inference is 10 cm to 14 cm.
[0050] FIG. 4 is a diagram illustrating a beamforming method according to an embodiment of the present invention.
[0051] 4, in a beamforming method 31 according to an embodiment of the present invention, a first microphone M1 can receive a first sound signal S1 from a first side. For example, the first microphone M1 can receive a first sound signal S1 including a voice of a person participating in a video conference and surrounding noise from the first side and transmit the first sound signal S1 to a first STFT module 310 (represented as an L_STFT module in the drawing).
[0052] The first STFT module 310 may perform a Fourier transform operation on the first sound signal S1 received from the first microphone M1 to obtain a Fourier transform result P1 for the first sound signal S1.
[0053] Meanwhile, the second microphone M2, which is disposed at a predetermined distance D from the first microphone M1, can receive a second sound signal S2 from the second side. For example, the second microphone M2 can receive the second sound signal S2 including the voices of people participating in the video conference and surrounding noise from the second side and transmit the second sound signal S2 to a second STFT module 311 (represented as an R_STFT module in the drawing).
[0054] The second STFT module 311 may perform a Fourier transform operation on the second sound signal S2 received from the second microphone M2 to obtain a Fourier transform result P2 on the second sound signal S2.
[0055] The phase difference acquisition module 312 can acquire the phase difference dP between the first sound signal S1 and the second sound signal S2 from the Fourier transform result P1 provided from the first STFT module 310 and the Fourier transform result P2 provided from the second STFT module 311.
[0056] As described in FIG. 3, the trained beamforming model 314 can receive the phase difference dP between the first sound signal S1 and the second sound signal S2 as an input and perform a neural network operation (ie, an inference operation).
[0057] The masking module 315 may apply a mask to the inference operation result, and then the Hadamard product module 316 may perform a Hadamard product between the inference operation result (or the result to which the mask has been applied) and the Fourier transform result P2 for the second sound signal S2 provided from the second STFT module 311. Here, the Hadamard product may be an operation of multiplying each element of two matrices of the same size.
[0058] The output module 317 may output the Hadamard product result S3 provided by the Hadamard product module 316. For example, the output module 317 may output a beamformed sound signal S3 obtained as a result of performing beamforming on sound signals received by the microphones M1 and M2 using the beamforming model 314. For example, the beamformed sound signal S3 may be a separation of a voice of a person participating in a video conference in a noisy surrounding environment, and may be provided to another computing device (e.g., a personal computer coupled to a monitor 20) and subsequently provided to other video conference participants.
[0059] FIG. 5 is a diagram illustrating a specific example of a beamforming method according to an embodiment of the present invention.
[0060] For directional hearing, two or more microphones are basically required. The geometry of the microphone array has spatial properties that align the signals received by each microphone. The process of obtaining a mask for beamforming can be formulated as follows. First, it is assumed that a spectrogram can be obtained by performing STFT (short time Fourier transform) on each signal received from multiple microphones. TIFF0007591848000001.tif6150 is the audio to be obtained, and is represented as follows.
[0061] TIFF0007591848000002.tif6150
[0062] TIFF0007591848000003.tif11150
[0063] Also, TIFF0007591848000004.tif6150 is noise, and the input from the microphone array where voice and noise come in at the same time is It can be represented as TIFF0007591848000005.tif6150.
[0064] TIFF0007591848000006.tif6150 shows the characteristics of the microphone array depending on the angle. At this time, the input spectrogram is filtered. If you take TIFF0007591848000007.tif6150, the result is TIFF0007591848000008.tif6150. Therefore, the magnitude of the input signal is minimized, but the signal to be obtained is left intact. You can design TIFF0007591848000009.tif6150. If the bold text is a vector, When TIFF0007591848000010.tif6150 is used,
[0065] TIFF0007591848000011.tif6150
[0066] At this time, the solution is
[0067] TIFF0007591848000012.tif11150
[0068] Here, the superscript H is the matrix that is prepended after taking the complex complex number. Therefore, the spectrogram of the speech that we want to obtain using the filter we have obtained is The most important part of implementing beamforming using this method is to correctly set the steering vector. TIFF0007591848000014.tif6150 and spatial correlation matrix The goal is to find TIFF0007591848000015.tif6150 accurately.
[0069] In this way, in the MVDR method that does not use a neural network, in order to accurately satisfy the MVDR, a steering vector is required that mathematically models the path of the sound generated from the sound source until it reaches each microphone. However, in the case of video conversation, when the microphone and the user's mouth are located close to each other, for example, about 1m, and the distance between the microphones is also close, for example, about several centimeters or several tens of centimeters, mathematical modeling is very difficult. In addition, when the user moves irregularly as in the actual video conversation environment, there is a limit that the inconvenience to the user increases if the steering vector is kept at a fixed value. In order to overcome this limit, a neural network can be used instead of a steering vector.
[0070] In addition, the phase difference matrix used as an input of the neural network in this method is easier to obtain than the spatial correlation matrix used to obtain the location information of the sound source in the existing method, and the location information of the sound source can be obtained more easily. In the beamforming method according to an embodiment of the present invention, a neural network for spatial information is trained simply by two microphones, and sound can be obtained in a predetermined direction through this. Therefore, the following advantages are provided.
[0071] 1. In practical situations, when measuring with a fixed microphone, it is not necessary to precisely define the steering vector for a moving target.
[0072] 2. The divergence of singular points that occurs in the process of calculating linear filters disappears. This is because This automatically reduces the side lobes, which is the power of TIFF0007591848000016.tif6150, resulting in side lobe cancelation.
[0073] 3. When looking at the STOI value of audio restored from angles other than the front, when audio signals come in from various directions, the side STOI of MVDR appears larger than the STOI of the neural network, i.e., when using a neural network, the frontal audio signal is more accurately classified and output.
[0074] When there are no obstacles between the sound source and the microphone, the sound path is affected only by the angle of the sound source. Since the distance between the microphones and the sound source on both sides is the same, the sounds coming from the front are the same. In other words, the time difference of arrival (TDOA) of the front source is close to 0. By taking advantage of this, the sounds coming from the front can be left as they are.
[0075] Sound generated from a sound source at any position and reaching two microphones TIFF0007591848000017.tif6150, TIFF0007591848000018.tif6150, two phase matrices are obtained from each signal. TIFF0007591848000019.tif6150, TIFF0007591848000020.tif6150 can be obtained. The closer the difference between these phase matrices is to 0, the closer the sound is to a sound source that is closer to the front. In order to distinguish this, if we consider a function that outputs closer to 0 the further the input is from 0, and outputs 1 the closer the input is to 0, it is possible to create a mask that leaves only signals coming from the front. Therefore, the mask that minimizes the error L below can be By searching for TIFF0007591848000021.tif6150, you can recover the signal coming from the front.
[0076] TIFF0007591848000022.tif10150
[0077] When using Fourier transform, the signal can be considered as a sum of simple pure tones of each frequency, but when two pure tones of the same frequency are combined through a phase delay, they create an entirely new pure tone signal. The problem is that it can create a situation where the phase difference between the two microphones is not exactly proportional to the angle of the sound source, so creating a mask based on a rule that simply depends on the phase difference is inaccurate. Since obtaining sound requires a filter that processes a wide range of frequencies, this problem must be solved. As a solution to overcome this, an embodiment of the present invention provides a method of recognizing the phase difference through a neural network and a spectrogram pattern of a reference microphone.
[0078] Referring to FIG. 6, the mask to be designed is a mask between 0 and 1 defined by real numbers, so it can perform a function similar to an IBM (Ideal Binary mask). However, since there are values in between, it can be an SBM (Soft Binary mask). Looking at existing research on neural networks using IBM, a noisy aspect was used as it is. Thus, when reconstructing speech from a noisy spectrogram, it is possible to reconstruct the speech of a signal by only leaving a frequency domain related to the structure of the speech. Therefore, the mask obtained by the SBM-based neural network creates a directional auditory mask that is applied to the magnitude from the phase difference, which performs a function of leaving a signal even if there is an element that has a phase difference. Meanwhile, there is a method that proposes optimizing the assumed time delay to obtain directional information more clearly from TDOA, and in various embodiments of the present invention, a CNN is used. This is because CNN is more efficient for 2D matrices such as images and does not require much computational power. In addition, the convolution filter is optimized for the reduction width due to the phase difference of the ideal mask. On the other hand, in case where only phase difference is input to the neural network, the target voice pattern cannot be learned. Inverse wave can be used to prevent this problem. The mean squared error (MSE) calculated in the time domain is used as a loss function to train a mask with phase information, which is similar to the method mentioned above. By restoring the masked spectrogram with the inverse STFT (ISTFT), it can be compared with the clean target sound in the time domain using the loss function. In this case, the gradient value to be updated may include voice pattern information for the reference microphone.
[0079] In order to train the model with data from various situations, a given data set was generated using a simulation method, the target voice was reconstructed using a neural network, and a stereo channel sound source was generated through a spatial simulation of 10 x 10 x 10 m. The microphone height was 2 m and two positions (9, 5.06, 2) and (9, 4.94, 2) were specified with 6 cm intervals on the left and right. The sound source was located on a semicircle with a diameter of 1 meter, and the center of the semicircle was the same as the center of the microphone. The position of the sound source to be obtained was located at 90°, (7, 5, 2) in front. Then, the semicircle was divided into four sections [0°, 40°), [40°, 80°), [100°, 140°), and [140°, 180°), and a noise sound source was placed at a random position for each section. Part of the voice data set was used as training data, and the other part was used as test data. The noise dataset used two artificial noises (speech shape noise and babble noise) and the DEMAND database, which contains 13 recorded noises. The training data consisted of two artificial noises and eight recorded noises (cafe, car, kitchen, meeting, metro, restaurant, station, traffic). The test data consisted of five recorded noises (bus, cafeteria, living room, office, and public square). To generate the noise, noise was extracted from four random points of the same noise signal for the length of the voice. After that, a simulation was performed by emitting sound from four sound sources, and a method of containing it with two microphones was adopted.
[0080] For the training data, 40 conditions were used, i.e., 10 noise conditions and 4 SNRs (0 dB, 5 dB, 10 dB, 15 dB). For the test data, 20 other conditions were used, i.e., 5 noise conditions and 4 SNRs (2.5 dB, 7.5 dB, 12.5 dB, 17.5 dB). In this case, only the energy reduction due to distance was simulated using the Room Impulse Response of the image-source method. Reverberation was not taken into account, so the reverberation time TIFF0007591848000023.tif6150 was set to 0.
[0081] Each signal received by the two microphones was transformed into the Fourier domain via STFT. STFT uses a 256-point Hamming window for signals with a 16kHz sample rate. The window shift uses 128 points (128-point overlap). After the neural network operation is completed, the same conditions are used when performing the ISTFT operation. The model structure follows the structure in Table 1.
[0082] The input value is the STFT result that contains only the lowest 128 frequencies. The mask obtained as a result of the neural network is multiplied by the spectrogram of the input 128 frequencies, and the 129th frequency is filled with 0 and ISTFT is performed to obtain the restored signal.
[0083] [Table 1]
[0084] Here, the input consists of (batch, frequency, time step, channel). And the convolution layer consists of (filter height, filter width), (stride height, stride width), (padding height, padding width). The output consists of (batch, frequency, time step, channel). PReLU was used for all activation functions. The activation function of the last layer uses a sigmoid function to use channels 1 and 2 as the real and imaginary parts of the mask, respectively.
[0085] Advantageous effects of embodiments of the present invention are described below in conjunction with FIGS.
[0086] FIG. 6 is a diagram for explaining a beamforming system according to an embodiment of the present invention, FIG. 7 is a diagram for explaining a beamforming system according to an embodiment of the present invention, and FIG. 8 is a diagram for explaining a beamforming system according to an embodiment of the present invention.
[0087] Referring to FIG. 6, a beamforming system 2 according to an embodiment of the present invention may be implemented in a monitor comprising a first microphone M1 and a second microphone M2.
[0088] 7, a beamforming system 2 according to an embodiment of the present invention may be embodied to include a large stationary device 14 on which a portable computing device 22, such as a smartphone, can be placed. After performing beamforming on the sound signals received using the microphones M1 and M2, the resulting beamformed sound signals may be provided to the portable computing device 22 via any connection means. The computing device 22 may then provide the separated human voices to, for example, other participants in a video conference.
[0089] 8, a beamforming system 4 according to an embodiment of the present invention may be embodied as a mountable device 16 that may be attached to a portable computing device 22, such as a smartphone. After performing beamforming on the sound signals received using the microphones M1 and M2, the resulting beamformed sound signals may be provided to the portable computing device 22 via any connection means. The computing device 22 may then provide the separated human voices to, for example, other participants in a video conference.
[0090] 9 and 10 are diagrams illustrating advantageous effects of a beamforming method and a beamforming system according to an embodiment of the present invention.
[0091] The beamforming algorithm was used to measure the sound loudness (top row) and STOI (short-time objective intelligibility) score (bottom row) according to the angle of beamforming using a classical method (MVDR), when the predetermined distance D between the first microphone M1 and the second microphone M2 was 12 cm. FIG. 9 shows the loudness (top row) and STOI (short-time objective intelligibility) score (bottom row) according to the angle of beamforming using a neural network according to various embodiments of the present invention.
[0092] STOI is an index related to the degree to which a sound is restored, regardless of the loudness of the sound. In the case of a traditional method, the STOI is indicated as being high from directions other than the direction of the intended listening, whereas in the case of the neural network method according to various embodiments of the present invention, the STOI is indicated as being low. Through this, it can be seen that the neural network method according to various embodiments of the present invention can more reliably separate voices spoken by speakers in directions other than the desired direction, and is less susceptible to interference.
[0093] According to the embodiment of the present invention described above, it is possible to restore the sound received by the microphone using only the phase difference without the need to calculate steering vectors for various noise environments, and it is possible to efficiently implement beamforming.
[0094] In the case of existing MVDR, it is difficult to obtain a steering vector in an environment with a lot of reverberation, and even in an environment without reverberation, in the case of video conversation, it is difficult to obtain a simple steering vector because it is not possible to obtain a form assuming a distant sound source. Also, in real situations, the speaker often moves, and the more accurately the steering vector is predicted, the more difficult it is to use in such an environment. When calculating the spatial correlation matrix expressed in TIFF0007591848000025.tif6150, it uses sounds mixed with noise, so accurate prediction is impossible. However, this method uses a neural network to learn spatial information from a large amount of data, so accurate learning is possible because it does not calculate steering vectors and uses a method of learning by using clean voice signals as target voices. The average SSNR (Segmented Signal to Noise Ratio) index of the frontal sound obtained by calculating 824 test data using the artificial intelligence model trained through this method and the existing MVDR method is -1.557 for sounds containing noise, -0.120 for sounds using MVDR, and 2.165 for the model that passed through the neural network, showing that the neural network model is the highest.
[0095] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those having ordinary skill in the technical field to which the present invention pertains, using the basic concept of the present invention defined in the following claims, also fall within the scope of the present invention.
Claims
1. receiving a first sound signal and a second sound signal using a first microphone and a second microphone disposed at a predetermined distance from the first microphone; obtaining a Fourier transform result for each of the first sound signal and the second sound signal; obtaining a phase difference between the first sound signal and the second sound signal from the Fourier transform result; applying the phase difference to an autoencoder; inputting the phase difference applied to the autoencoder into a beamforming model and performing a calculation using a neural processor; performing a Hadamard product on a calculation result of the beamforming model of the neural processor and a Fourier transform result of the first sound signal; performing an inverse STFT on the Hadamard product result to recover a masked spectrogram; outputting a beamformed sound signal; A beamforming method comprising:
2. The step of performing the Hadamard product comprises: The beamforming method of claim 1 , further comprising applying a mask to the operation result before performing the Hadamard product.
3. The step of performing the Hadamard product comprises: The beamforming method of claim 1 , further comprising the step of performing a gain control after performing the Hadamard product.
4. The beamforming method of claim 1 , wherein the predetermined distance is between 10 cm and 14 cm.
5. The beamforming method of claim 1 , further comprising training the beamforming model using the phase difference.
6. A first microphone for receiving a first sound signal; a second microphone disposed a predetermined distance away from the first microphone to receive a second sound signal; a first STFT module for obtaining a Fourier transform result for the first sound signal; a second STFT module for obtaining a Fourier transform result for the second sound signal; a phase difference acquisition module for acquiring a phase difference between the first sound signal and the second sound signal from the Fourier transform result; a neural processor that receives the phase difference and performs a neural network operation using a beamforming model; a Hadamard product module that performs a Hadamard product on a calculation result of the beamforming model of the neural processor and a Fourier transform result of the first sound signal; an output module for outputting the result of the Hadamard product; Including, The neural processor applies the phase difference to an autoencoder, and inputs the phase difference applied to the autoencoder to the beamforming model for calculation; The output module outputs a beamformed sound signal; The masked spectrogram is restored by performing an inverse STFT on the Hadamard product result. Beamforming system.
7. The beamforming system of claim 6 , further comprising a masking module that applies a mask to the result before performing the Hadamard product.
8. The beamforming system of claim 6 , further comprising a gain control module that performs gain control after performing the Hadamard product.
9. The beamforming system of claim 6 , wherein the predetermined distance is between 10 cm and 14 cm.
10. The beamforming system according to claim 6, further comprising a learning model for learning the beamforming model using the phase difference.
Citation Information
Patent Citations
Signal generation device, signal generation system, signal generation method, and program
JP2020034624A
Voice Recognition System
JP2020503570A
