Signal processing device and method, and program

Abstract

The present technology pertains to a signal processing device, method, and program, which make it possible to effectively reduce noise. The signal processing device reduces at least one of noise of a first speech signal outputted from a first microphone and noise of a second speech signal outputted from a second microphone provided to a position different from that of the first microphone, and reduces noise in at least a part of a frequency band of the first speech signal by using the second speech signal. The present technology can be applied to an imaging device that captures a moving image.

Description

Signal processing device, method, and program

[0001] The present technology relates to a signal processing device, method, and program, and more particularly to a signal processing device, method, and program that are capable of effectively reducing noise.

[0002] Conventionally, a technology has been proposed that estimates and reduces (suppresses) noise in an audio signal output from a microphone unit (hereinafter referred to as a target sound collection microphone) built into a camera to collect surrounding sounds (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2016-208109

[0004] However, no technology has been proposed for reducing noise in audio signals output from a microphone unit (hereinafter referred to as an external microphone) that is attached externally to a camera and collects sounds from the surrounding area or a specific direction.

[0005] The present technology has been made in view of such circumstances, and makes it possible to effectively reduce noise.

[0006] A signal processing device according to one aspect of the present technology includes a noise reduction processing unit that reduces at least one of noise in a first audio signal output from a first microphone and noise in a second audio signal output from a second microphone provided at a position different from the first microphone, and a power spectrum replacement unit that uses the second audio signal to reduce noise in at least a portion of a frequency band of the first audio signal.

[0007] In one aspect of the present technology, at least one of noise in a first audio signal output from a first microphone and noise in a second audio signal output from a second microphone provided at a different position from the first microphone is reduced, and noise in at least a portion of the frequency band of the first audio signal is reduced using the second audio signal.

[0008] FIG. 1 is a diagram illustrating an overview of unmixing matrix estimation. FIG. 2 is a block diagram illustrating a configuration example of an imaging device as a signal processing device according to a first embodiment of the present technology. FIG. 3 is a flowchart illustrating lens noise reduction processing. FIG. 4 is a flowchart illustrating wind noise reduction processing. FIG. 4 is a block diagram illustrating a configuration example of an imaging device as a signal processing device according to a second embodiment of the present technology. FIG. 5 is a schematic diagram illustrating a flow of noise reduction processing using a noise reference microphone. FIG. 6 is a flowchart illustrating lens noise reduction processing. FIG. 7 is a flowchart illustrating another example of lens noise reduction processing. FIG. 8 is a flowchart illustrating wind noise reduction processing. FIG. 9 is a block diagram illustrating a configuration example of a computer.

[0009] Hereinafter, embodiments of the present technology will be described in the following order: 1. Conventional noise reduction 2. First embodiment 3. Second embodiment 4. Others

[0010] 1. Conventional Noise Reduction Technology Conventional Noise Reduction Technology Compared to the Present Technology The present technology is a technology for reducing noise (static) mixed into an external microphone that is detachable from an imaging device (camera).

[0011] For example, in addition to the external microphones described above, the housing of an imaging device may also include a built-in target sound collection microphone that collects ambient sounds. When capturing video (moving images), lens noise and camera shake noise may be mixed into these microphones. Here, lens noise is noise generated by mechanisms operating within the lens, such as the electric zoom, focus, and aperture mechanisms. Camera shake noise is noise generated when the camera shake correction mechanism within the imaging device operates.

[0012] Lens noise and camera shake noise include airborne noise and vibration noise.

[0013] Both the target sound collection microphone and the external microphone are affected by lens noise and camera shake noise. In such cases, due to a resonance phenomenon of the housing of the imaging device, the vibration noise contained in the lens noise and camera shake noise mixed into the external microphone may be louder than the vibration noise contained in the lens noise and camera shake noise mixed into the target sound collection microphone.

[0014] Known technologies for reducing lens noise and camera shake noise include the spectral subtraction method (hereinafter referred to as the SS method), beamforming (hereinafter referred to as BF) processing, and noise suppression using deep learning (hereinafter referred to as DNN (Deep Neural Network)-NR (Noise Reduction)).

[0015] The SS method reduces noise by estimating the average frequency characteristics of noise and subtracting the noise components from the recorded sound. However, lens noise has a short duration, making it difficult to estimate the average frequency characteristics. In response to this, the SS method could record the average frequency characteristics for each lens in the imaging device and switch the average frequency characteristics depending on the lens attached. However, as the number of lenses increases, the number of recorded parameters (average frequency characteristics) increases, which can lead to storage space limitations within the imaging device.

[0016] Furthermore, camera shake noise has frequency characteristics that vary depending on the situation and the individual, and is non-stationary noise whose noise characteristics fluctuate over time. Therefore, the SS method may not be able to completely eliminate camera shake noise.

[0017] BF processing is a technology that combines multiple microphone signals to collect sound from a specific direction. By directing the main axis of directivity away from the noise source, BF processing can reduce noise components that enter the microphones. However, because vibration noise enters multiple microphones irregularly, BF processing cannot reduce vibration noise.

[0018] DNN-NR is a method of removing noise by using a DNN to learn in advance the process by which the target sound is output from a signal containing a mixture of target sound and noise. Recently, DNN-NR, which extracts only the target sound from a signal containing a mixture of target sound and noise, has become popular.

[0019] However, in DNN-NR, the sounds that need to be collected when shooting video with an imaging device are not just voices, but also ambient sounds, so the amount of training data becomes enormous. When considering training to remove only specific noises, it is necessary to collect a large amount of noise data for each model, which increases the cost of design and development.

[0020] Furthermore, unmixing matrix estimation is a method for effectively suppressing lens noise and camera shake noise while maintaining the quality of the target sound.

[0021] FIG. 1 is a diagram showing an overview of unmixing matrix estimation.

[0022] First, in unmixing matrix estimation, microphone 1, which is primarily intended to collect (record) a target sound, such as an external microphone or a target sound collection microphone, and microphone 2, which is primarily intended to collect noise, such as a noise reference microphone, are used.

[0023] In unmixing matrix estimation, the transfer functions (how the sounds are mixed and the characteristics of the propagation paths) of the target sound and noise arriving at microphone 1 and microphone 2 are learned, and the inverse characteristics of the transfer functions are estimated, making it possible to separate the target sound from the signal of microphone 1.

[0024] For example, let S(ω) be the frequency spectrum of the target sound (voice or environmental sound), N(ω) be the frequency spectrum of the noise, and X be the frequency spectrum of the signal from microphone 1. １ (ω), the frequency spectrum of the signal from microphone 2 is X ２ (ω), the transfer function between the target sound source and microphone 1 is H １１ (ω), the transfer function between the target sound source and microphone 2 is H ２１ (ω), the transfer function between the noise source and microphone 1 is H １２ (ω), the transfer function between the noise source and microphone 2 is H ２２ (ω), where ω represents the frequency bin of the frequency spectrum obtained from the FFT (Fast Fourier Transform).

[0025] The frequency spectrum of the signals from microphone 1 and microphone 2 is expressed by the following equation (1).

[0026] The frequency spectrum of the target sound (desired sound excluding noise) collected by microphone 1 is H １１ (ω)S １ (ω). H １１ (ω)S １ When (ω) is expressed in matrix form, it is expressed as the following equation (2).

[0027] Equation (2) can be transformed into the following equation (3).

[0028] This formula (3) represents [frequency spectrum of sound collected by each microphone] = [matrix expressed by four transfer functions] × [frequency spectrum of target sound alone and noise alone collected by each microphone].

[0029] Here, as shown in the following equation (4), a matrix expressed by four transfer functions is defined as a mixing matrix G(ω).

[0030] The frequency spectrum of the target sound alone collected by the microphone 1 is calculated from the following equation (5).

[0031] As described above, by estimating the unmixing matrix, it is possible to reduce the airborne sound component of the lens noise and camera shake noise that are mixed into the sound collected by the target sound collecting microphone and the external microphone.

[0032] However, as described above, an external microphone may contain a large amount of vibration sound components, and noise cannot be reduced sufficiently by unmixing matrix estimation alone.

[0033] For example, when noise is reduced by applying unmixing matrix estimation using two signals from an external microphone and a noise reference microphone, it is necessary that the relationship of external microphone > noise reference microphone holds for the target sound frequency component, and the relationship of external microphone < noise reference microphone holds for the noise frequency component.

[0034] Therefore, even in the low frequency band where vibration noise appears among noise frequency components, if the external microphone is greater than the noise reference microphone, noise reduction in that frequency band becomes theoretically impossible. In this specification, the low frequencies where vibration noise appears are assumed to be approximately 1000 Hz or lower.

[0035] As described above, with conventional techniques, it has been difficult to effectively reduce lens noise and camera shake noise while maintaining the quality of sound other than noise in an external microphone.

[0036] 2. First Embodiment <Device Configuration> FIG. 2 is a block diagram showing an example configuration of an imaging device as a signal processing device according to a first embodiment of the present technology.

[0037] FIG. 2 shows the configuration of blocks that process audio signals in an image pickup device that captures video.

[0038] The imaging device 10 in FIG. 2 reduces noise such as lens noise that gets mixed into the external microphone (first microphone) 11-1 and the target sound collection microphone (second microphone) 11-2.

[0039] 2 is configured to include an external microphone 11-1, a target sound collection microphone 11-2, an A / D (Analog to Digital) converter 12-1, an A / D converter 12-2, a signal processing unit 13, a control unit 14, and a wind noise detection unit 15. When there is no need to distinguish between the external microphone 11-1 and the target sound collection microphone 11-2, they will be referred to as microphones 11. When there is no need to distinguish between the A / D converters 12-1 and 12-2, they will be referred to as A / D converters 12.

[0040] Each microphone housing of the external microphone 11-1 and the target sound collection microphone 11-2 is equipped with standard microphone units for N channels. Although Fig. 2 shows a case where each microphone unit is one, the number of each microphone unit may be one or more.

[0041] As described above, the external microphone 11-1 is a microphone unit that is detachable from the housing of the imaging device 10 and is attached to the housing to collect sounds from the surrounding area or a specific direction. The external microphone 11-1 generates an audio signal corresponding to the collected sound and outputs the generated audio signal to the A / D converter 12-1 via an amplifier (not shown) for amplifying the signal.

[0042] As described above, the target sound collection microphone 11-2 is a microphone unit built into the housing of the imaging device 10 to collect ambient sounds. In the housing of the imaging device 10, the target sound collection microphone 11-2 is installed in a position physically closer to the shutter mechanism and the like than the external microphone 11-1. The target sound collection microphone 11-2 generates an audio signal corresponding to the collected sound and outputs the generated audio signal to the A / D converter 12-2 via an amplifier (not shown).

[0043] When it is necessary to distinguish between the audio signal output from the external microphone 11-1 and the audio signal output from the target sound collection microphone 11-2, they are referred to as the external microphone signal and the target sound collection microphone signal, respectively.

[0044] The A / D converter 12 converts the analog audio signal supplied from the microphone 11 via the amplifier into a digital audio signal, and outputs the converted digital audio signal to the signal processing unit 13 .

[0045] The signal processing unit 13 is composed of a DSP (Digital Signal Processor) and a CPU (Central Processing Unit), and outputs an audio signal from which unnecessary noise has been removed.

[0046] The signal processing unit 13 is configured to include an STFT (Short Term Fourier Transform) unit 21-1, an STFT unit 21-2, a ratio calculation unit 22, a noise reduction processing unit 23, a power spectrum permutation unit 24, and an ISTFT (Inverse Short Term Fourier Transform)-Overlap processing unit 25. When there is no need to distinguish between the STFT unit 21-1 and the STFT unit 21-2, they will be referred to as the STFT unit 21.

[0047] The STFT unit 21 performs a Fourier transform on the audio data supplied from the A / D converter 12 using STFT processing while shifting the window function, converting the data into an audio signal in the frequency domain. Instead of STFT, band division processing such as a QMF (Quadrature Mirror Filter) or a DFT (Discrete Fourier Transform) filter bank may be used. The STFT unit 21 outputs the audio signal in the frequency domain to the wind noise detection unit 15 and the ratio calculation unit 22.

[0048] The ratio calculation unit 22 calculates the ratio of power spectra (hereinafter referred to as power spectrum ratio) for the audio signals divided into each frequency band supplied from the STFT unit 21, depending on the determination result of whether or not the lens is being controlled supplied from the control unit 14 or the determination result of whether or not wind noise has been detected supplied from the wind noise detection unit 15. The ratio calculation unit 22 calculates the power spectrum ratio of the external microphone signal and the target sound collection microphone signal.

[0049] The ratio calculation unit 22 outputs the audio signals divided into each frequency band supplied from the STFT unit 21 and the power spectrum ratio to the noise reduction processing unit 23 .

[0050] The noise reduction processing unit 23 performs noise reduction processing on the audio signal divided into each frequency band, depending on the determination result of whether or not lens control is in progress supplied from the control unit 14 or the determination result of whether or not wind noise has been detected supplied from the wind noise detection unit 15, thereby reducing unnecessary noise.

[0051] The noise reduction processing unit 23 performs noise reduction processing only when it is determined that the control unit 14 is controlling the lens or when it is determined that the wind noise detection unit 15 has detected wind noise.

[0052] As described above, since the distance between the target sound collection microphone 11-2 and the shutter mechanism, etc., and the distance between the external microphone 11-1 and the shutter mechanism, etc., are different, the unmixing matrix estimation method can be used for the noise reduction process. For example, when the external microphone 11-1 is used as microphone 1 in Fig. 1, the target sound collection microphone 11-2 is used as microphone 2 in Fig. 1. Furthermore, when the target sound collection microphone 11-2 is used as microphone 1 in Fig. 1, the external microphone 11-1 is used as microphone 2 in Fig. 1.

[0053] When the unmixing matrix estimation method is used for noise reduction processing, the noise reduction processing unit 23 uses the power spectrum ratios calculated during lens control and during lens non-control to perform noise reduction on the external microphone signal and the target sound collection microphone signal according to the above-described equation (5).

[0054] The noise reduction method is not limited to unmixing matrix estimation, and any of the above-mentioned methods such as beamforming and spectral subtraction may be used. For example, low-pass filtering or filtering that blocks a frequency band corresponding to the noise may be used for wind noise.

[0055] Furthermore, noise reduction processing may be performed using different noise reduction methods on the external microphone signal of the external microphone 11-1 and the target sound collection microphone signal of the target sound collection microphone 11-2. Furthermore, if necessary, noise reduction processing may be performed on only at least one of the external microphone signal of the external microphone 11-1 and the target sound collection microphone signal of the target sound collection microphone 11-2.

[0056] Specifically, for example, there may be cases where only the audio signal from either the target sound collection microphone 11-2 or the external microphone 11-1 is recorded. In such cases, the amount of calculation can be reduced by not performing noise reduction on the audio signal that is not recorded.

[0057] For example, when vibrations of a specific frequency are generated from a lens or the like, a resonance phenomenon may cause noise only on the external microphone 11-1 side. In such a case, it is desirable to reduce noise only on the external microphone signal.

[0058] For example, if a wind jammer is attached to the external microphone 11-1, wind noise may occur only on the target sound collecting microphone 11-2 side. In such a case, it is desirable to reduce noise only on the target sound collecting microphone signal.

[0059] The two noise-reduced signals (the external microphone signal and the target sound collection microphone signal) obtained by the noise reduction process are output to the power spectrum replacement unit 24. If the noise has not been reduced, the signals are output as they are.

[0060] The power spectrum replacement unit 24 calculates the power spectrum of the external microphone signal and the target sound collection microphone signal after noise reduction processing, and replaces the frequency components in which noise remains in the external microphone signal with the frequency components of the target sound collection microphone signal using the target sound collection microphone signal after noise reduction processing.

[0061] Specifically, the power spectrum replacement unit 24 generates a power spectrum of an approximate microphone signal that approximates the external microphone signal by multiplying the power spectrum of the target sound collection microphone signal after noise reduction processing by the power spectrum ratio between the target sound collection microphone signal and the external microphone signal, which was calculated while the lens was not being controlled.

[0062] Then, the power spectrum replacement unit 24 replaces frequency components in which noise remains in the external microphone signal with corresponding frequency components in the approximate microphone signal.

[0063] The power spectrum substitution unit 24 outputs the external microphone signal after substitution to the ISTFT-Overlap processing unit 25 .

[0064] The ISTFT-Overlap processing unit 25 inversely converts the replaced external microphone signal back into a time-axis audio signal by ISTFT and Overlap&Add processing, and outputs the inversely converted audio signal to the subsequent stage.

[0065] The target sound collection microphone signal is output as is from the power spectrum substitution unit 24 to the ISTFT-Overlap processing unit 25, where it is converted back into a time-axis audio signal by ISTFT and Overlap & Add processing and then output.

[0066] The control unit 14 controls each unit of the imaging device 10. In particular, the control unit 14 acquires lens control information from an optical system driving unit (not shown) of the imaging device 10, determines whether or not lens control is being performed, and outputs the determination result to the ratio calculation unit 22 and the noise reduction processing unit 23.

[0067] The lens control information is general operating information inside the lens, such as "whether the zoom drive motor of the electric zoom lens is operating," "the zoom drive speed of the electric zoom lens," "whether autofocus is being performed," and "whether the aperture value is being changed."

[0068] It should be noted that the noise reduction processing of the present technology is not limited to noise caused by vibration of the lens noise, but can also be applied to other noises such as wind noise, which is noise in the same low frequency band.

[0069] The wind noise detection unit 15 detects wind noise using the audio signal divided into each frequency band supplied from the STFT unit 21, and outputs the determination result of whether wind noise has been detected to the ratio calculation unit 22 and the noise reduction processing unit 23.

[0070] Methods for detecting wind noise include, for example, a first detection method that detects wind noise from the correlation value of signals between multiple microphones 11, and a second detection method that detects wind noise using machine learning, but any method may be used to detect wind noise.

[0071] Note that because wind noise is random noise, when wind noise is collected by multiple microphones 11, the frequency characteristics of the microphone signals vary, resulting in a low correlation value between the microphones 11. The first detection method focuses on this characteristic and determines that "if the correlation between the multiple microphones 11 is low, wind noise is present."

[0072] The second detection method is a method in which wind noise is detected by learning wind noise and other sounds using a neural network model or the like.

[0073] Although FIG. 2 shows an example of an imaging device including the signal processing unit 13, the signal processing unit 13 may be configured as a server or the like.

[0074] <Lens Noise Reduction Processing> FIG. 3 is a flowchart illustrating the lens noise reduction processing performed by the ratio calculation unit 22, the noise reduction processing unit 23, and the power spectrum substitution unit 24 in FIG.

[0075] For example, when the sampling frequency is 48 kHz and the FFT sampling width is 256 samples, the process in Fig. 3 is performed every 256 / 48000 = approximately 5.3 ms. Note that the noise reduction processes in the subsequent flowcharts are also performed in the same manner.

[0076] In step S11, the control unit 14 acquires lens control information of the imaging device 10 and determines whether or not the lens is being controlled. If it is determined in step S11 that the lens is not being controlled, the process proceeds to step S12.

[0077] In step S12, ratio calculation unit 22 calculates the power spectral ratio of the external microphone signal and the target sound collection microphone signal while the lens is not controlled, i.e., while no lens noise is present. The power spectral ratio calculated while the lens is not controlled is used for unmixing matrix estimation when unmixing matrix estimation is used for noise reduction in step S14, which will be described later, and for approximating the external microphone signal in step S15. After that, the lens noise reduction process in FIG. 3 ends.

[0078] If it is determined in step S11 that lens control is in progress, the process proceeds to step S13.

[0079] In step S13, ratio calculation unit 22 calculates the power spectral ratio of the external microphone signal and the target sound collection microphone signal during lens control, and outputs the calculated power spectral ratio to noise reduction processing unit 23. At that time, the external microphone signal and the target sound collection microphone signal are also output to noise reduction processing unit 23. The power spectral ratio calculated during lens control is used for unmixing matrix estimation when unmixing matrix estimation is used for noise reduction in step S14.

[0080] In step S14, the noise reduction processing unit 23 performs noise reduction processing to reduce noise in at least one of the external microphone signal and the target sound collection microphone signal. As described above, any method may be used for the noise reduction processing, such as unmixing matrix estimation, beamforming, or spectral subtraction. Note that, depending on the noise reduction method used, the power spectral ratio during lens control calculated in step S13 may not be used.

[0081] In step S15, after the noise reduction process, the power spectrum replacement unit 24 uses the power spectrum ratio calculated while the lens was not being controlled to replace the power spectrum for frequency bands where noise remains in the external microphone signal and the amount of residual noise is greater than the amount of residual noise in the target sound collection microphone signal.

[0082] That is, the power spectrum substitution unit 24 multiplies the target sound collection microphone signal by the power spectrum ratio calculated while the lens is not controlled to generate an approximate microphone signal that approximates the external microphone signal. The power spectrum substitution unit 24 substitutes frequency bands in which noise remains in the external microphone signal and in which the amount of residual noise is greater than the amount of residual noise in the target sound collection microphone signal with the frequency bands of the generated approximate microphone signal.

[0083] When the noise reduction method is unmixing matrix estimation, the frequency band where noise remains in the external microphone signal and where the amount of remaining noise is greater than the amount of remaining noise in the target sound collection microphone corresponds to the low frequency band (low range) where vibration-induced noise remains. As described above, in this specification, the low frequency where vibration-induced noise appears, i.e., the low frequency band where vibration-induced noise remains, is assumed to be approximately 1000 Hz or less. However, this method is also effective when vibration noise occurs in a band higher than 1000 Hz. Note that the power spectrum is replaced, i.e., the frequency band where noise remains in the external microphone signal is determined and set in advance by learning or other means depending on the noise reduction method. After this, the lens noise reduction process in FIG. 3 ends.

[0084] The above processing can reduce noise in the low frequency band where noise due to vibration remains, and therefore lens noise can be effectively reduced.

[0085] <Wind Noise Reduction Processing> FIG. 4 is a flowchart illustrating the wind noise reduction processing performed by the ratio calculation unit 22, the noise reduction processing unit 23, and the power spectrum substitution unit 24 in FIG.

[0086] Note that unmixing matrix estimation is not suitable for wind noise reduction, and noise reduction is not performed using unmixing matrix estimation. Therefore, in the case of FIG. 4, the power spectrum ratio is not calculated during wind noise detection.

[0087] In step S31, the wind noise detection unit 15 performs wind noise detection and determines whether wind noise has been detected. If it is determined in step S31 that wind noise has not been detected, the process proceeds to step S32.

[0088] In step S32, the ratio calculation unit 22 calculates the power spectrum ratio between the external microphone signal and the target sound collection microphone signal when no wind noise is detected, after which the lens noise reduction process in FIG.

[0089] If it is determined in step S31 that wind noise has been detected, the process proceeds to step S33.

[0090] In step S33, the noise reduction processing unit 23 performs noise reduction processing to reduce noise in at least one of the external microphone signal and the target sound collection microphone signal. Any method for noise reduction processing, such as beamforming or spectral subtraction, may be used.

[0091] In step S34, similar to step S15 in FIG. 3, the power spectrum replacement unit 24 performs power spectrum replacement for frequency bands where noise remains in the external microphone signal after the noise reduction process and the amount of remaining noise is greater than the amount of remaining noise in the target sound collection microphone signal.

[0092] That is, in step S32, the power spectrum ratio calculated while wind noise was not being controlled is multiplied by the target sound collection microphone signal to generate an approximate microphone signal, and the frequency band in which noise remains in the external microphone signal is replaced with the frequency band of the approximate microphone signal.

[0093] After that, the wind noise reduction process of FIG. 4 ends.

[0094] The above processing can effectively reduce wind noise.

[0095] 3. Second Embodiment <Device Configuration> FIG. 5 is a block diagram showing an example configuration of an imaging device as a signal processing device according to a second embodiment of the present technology.

[0096] 5, similar to FIG. 2, shows the configuration of blocks that process audio signals in an image pickup device that captures video.

[0097] In Figure 5, the imaging device 101 uses audio signals supplied from an external microphone 11-1, a target sound collection microphone 11-2, and a noise reference microphone (third microphone) 11-3 to reduce noise such as lens noise in the external microphone 11-1 and the target sound collection microphone 11-2.

[0098] The imaging device 101 in FIG. 5 differs from the imaging device 10 in FIG. 2 in that a noise reference microphone 11-3 and an A / D converter 12-3 are added, and that the signal processing unit 13 is replaced with a signal processing unit 111.

[0099] The imaging device 101 is configured to include an external microphone 11-1, a target sound collection microphone 11-2, a noise reference microphone 11-3, A / D converters 12-1 to 12-3, and a signal processing unit 111. When there is no need to distinguish between the external microphone 11-1 and the target sound collection microphone 11-2, the noise reference microphone 11-3 will also be referred to as the microphone 11. When there is no need to distinguish between the A / D converters 12-1 and 12-2, the A / D converter 12-3 will also be referred to as the A / D converter 12.

[0100] The microphone housing of the noise reference microphone 11-3 is equipped with standard microphone units for N channels. Although Fig. 5 shows the case where there is one microphone unit, the number of microphone units may be one or more.

[0101] The noise reference microphone 11-3 is a microphone that is mounted inside the housing of the imaging device 10 and actively collects noise inside the housing. The noise reference microphone 11-3 generates an audio signal corresponding to the collected sound and outputs the generated audio signal to an A / D converter 12-3 via an amplifier (not shown).

[0102] In addition, when it is necessary to distinguish the audio signal output from the noise reference microphone 11-3 from the audio signal output from the external microphone 11-1 and the audio signal output from the target sound collection microphone 11-2, it is referred to as the noise reference microphone signal.

[0103] The signal processing unit 111 differs from the signal processing unit 13 in FIG. 2 in that an STFT unit 21-3 is added and the noise reduction processing unit 23 is replaced with a noise reduction processing unit 121.

[0104] It should be noted that the STFT unit 21-3 will be referred to as the STFT unit 21 when there is no need to distinguish it from the STFT unit 21-1 and the STFT unit 21-2.

[0105] The noise reduction processing unit 121 performs noise reduction processing on the audio signal divided into each frequency band, using the noise reference microphone signal as well, in accordance with the determination result of whether or not lens control is in progress supplied from the control unit 14 or the determination result of whether or not wind noise has been detected supplied from the wind noise detection unit 15, thereby reducing unnecessary noise.

[0106] When the unmixing matrix estimation method described above with reference to FIG. 1 is used as the noise reduction method, for example, the external microphone 11-1 and the target sound collection microphone 11-2 are used as microphone 1 in FIG. 1, and the noise reference microphone 11-3 is used as microphone 2 in FIG. 1.

[0107] In this case, the noise reduction method is not limited to unmixing matrix estimation, and any method may be used. In this case, depending on the noise reduction method, all of the external microphone signal, the target sound collection microphone signal, and the noise reference microphone signal may be used, or any combination of the signals may be used. Furthermore, if necessary, noise reduction processing may be performed on at least one of the external microphone signal and the target sound collection microphone signal.

[0108] The two noise-reduced signals (the external microphone signal and the target sound collection microphone signal) obtained by the noise reduction process are output to the power spectrum replacement unit 24. If the noise has not been reduced, the signals are output as they are.

[0109] <Noise Reduction Processing Using Noise Reference Microphone> FIG. 6 is a schematic diagram showing the flow of noise reduction processing using the noise reference microphone 11-3.

[0110] FIG. 6 shows an example in which unmixing matrix estimation is performed as noise reduction processing.

[0111] The ratio calculation unit 22 is supplied with an external microphone signal from the external microphone 11-1 and a target sound collection microphone signal from the target sound collection microphone 11-2.

[0112] The ratio calculation unit 22 performs a calculation process A101 for the MPR (Microphone Power Ratio: power ratio between microphones) between the target sound collection microphone and the external microphone using the external microphone signal and the target sound collection microphone signal when the lens is not controlled, and obtains the MPR between the target sound collection microphone and the external microphone.

[0113] During lens control, the noise reduction processing unit 121 is supplied with an external microphone signal from the external microphone 11-1, a target sound collection microphone signal from the target sound collection microphone 11-2, and a noise reference microphone signal from the noise reference microphone 11-3.

[0114] 6 , for the sake of convenience of explanation, it is assumed that each signal is supplied, but in reality, each signal is supplied to ratio calculation unit 22 and supplied via ratio calculation unit 22. Alternatively, each signal may be supplied to ratio calculation unit 22, which may calculate the MPR between each microphone used for unmixing matrix estimation, and the calculated MPR between each microphone may be supplied to noise reduction processing unit 121.

[0115] The noise reduction processing unit 121 performs unmixing matrix estimation A102 using the external microphone signal and the noise reference microphone signal during lens non-control and lens control to obtain a noise-reduced external microphone signal. The noise reduction processing unit 121 also performs unmixing matrix estimation A103 using the target sound collection microphone signal and the noise reference microphone signal during lens non-control and lens control to obtain a noise-reduced target sound collection microphone signal.

[0116] The power spectrum replacement unit 24 performs approximation A104 of the target sound collection microphone signal to the external microphone signal by multiplying the MPR between the target sound collection microphone and the external microphone obtained by the ratio calculation unit 22 by the target sound collection microphone signal after noise reduction obtained by the noise reduction processing unit 121, thereby obtaining an approximate microphone signal.

[0117] The power spectrum substitution unit 24 performs substitution A105 of the low frequency range of the approximation microphone signal with the low frequency range (low frequency band) of the external microphone signal after noise reduction obtained by the noise reduction processing unit 121 .

[0118] By performing noise reduction in the manner described above, it is possible to obtain an external microphone signal in which low-frequency noise has been reduced, which was previously difficult to achieve.

[0119] <Lens Noise Reduction Processing (Unmixing Matrix Estimation)> FIG. 7 is a flowchart illustrating lens noise reduction processing performed by the ratio calculation unit 22, noise reduction processing unit 121, and power spectrum substitution unit 24 in FIG.

[0120] FIG. 7 shows an example in which unmixing matrix estimation is performed as noise reduction processing.

[0121] In step S111, the control unit 14 acquires lens control information of the imaging device 101 and determines whether or not the lens is under control. If it is determined in step S111 that the lens is not under control, the process proceeds to step S112.

[0122] In step S112, the ratio calculation unit 22 calculates the "power spectral ratio between the external microphone signal and the target sound collection microphone signal," "power spectral ratio between the external microphone signal and the noise reference microphone signal," and "power spectral ratio between the target sound collection microphone signal and the noise reference microphone signal" when the lens is not controlled, i.e., when there is no lens noise.

[0123] The power spectrum ratio calculated in step S112 is used in the noise reduction process in step S114, which will be described later. The "power spectrum ratio between the external microphone signal and the target sound collection microphone signal" is used in the process in step S115, which will be described later. After that, the lens noise reduction process in FIG. 7 ends.

[0124] If it is determined in step S111 that lens control is being performed, the process proceeds to step S113.

[0125] In step S113, the ratio calculation unit 22 calculates the "power spectrum ratio between the external microphone signal and the target sound collection microphone signal," "power spectrum ratio between the external microphone signal and the noise reference microphone signal," and "power spectrum ratio between the target sound collection microphone signal and the noise reference microphone signal" of the external microphone signal and the target sound collection microphone signal during lens control. The calculated power spectrum ratios are output to the noise reduction processing unit 23. The power spectrum ratios calculated in step S113 are used in the noise reduction processing in step S114, which will be described later.

[0126] In step S114, the noise reduction processing unit 23 performs noise reduction processing to reduce noise in at least one of the external microphone signal and the target sound collection microphone signal by unmixing matrix estimation using the power spectral ratios during lens non-control and lens control calculated in steps S112 and S113.

[0127] In step S115, similar to step S15 in FIG. 3, the power spectrum replacement unit 24 uses the power spectrum ratio calculated after the noise reduction process while the lens is not being controlled to replace the power spectrum for frequency bands where noise remains in the external microphone signal and the amount of residual noise is greater than the amount of residual noise in the target sound collection microphone.

[0128] In the case of unmixing matrix estimation, the band in which noise remains in the external microphone signal and the amount of remaining noise is greater than the amount of remaining noise in the target sound collecting microphone corresponds to the low frequency concentration band in which noise due to vibration remains. After that, the lens noise reduction process in FIG. 7 ends.

[0129] As described above, noise in the low frequency band where noise due to vibration remains can be reduced, so lens noise can be effectively reduced.

[0130] <Lens Noise Reduction Process (Noise Reduction Method Other Than Unmixing Matrix Estimation)> FIG. 8 is a flowchart illustrating another example of lens noise reduction process performed by ratio calculation unit 22, noise reduction processing unit 121, and power spectrum substitution unit 24 in FIG. 2 .

[0131] Fig. 8 shows an example in which a noise reduction method other than unmixing matrix estimation is used as the noise reduction process. Since the noise reduction method is other than unmixing matrix estimation, the power spectrum ratio is not calculated during lens control in the case of Fig. 8.

[0132] In step S131, the control unit 14 acquires lens control information of the imaging device 101 and determines whether the lens is under control. If it is determined in step S131 that the lens is not under control, the process proceeds to step S132.

[0133] In step S132, the ratio calculation unit 22 calculates the power spectrum ratio between the external microphone signal and the target sound collection microphone signal when the lens is not controlled, i.e., when there is no lens noise. The power spectrum ratio calculated in step S132 is used in the process of step S134, which will be described later. After that, the lens noise reduction process of FIG. 8 ends.

[0134] If it is determined in step S131 that lens control is being performed, the process proceeds to step S133.

[0135] In step S133, the noise reduction processing unit 121 performs noise reduction processing to reduce noise in at least one of the external microphone signal and the target sound collection microphone signal using the noise reference microphone signal. Any method, such as beamforming or spectral subtraction, may be used for the noise reduction processing.

[0136] In step S134, the power spectrum replacement unit 24 uses the power spectrum ratio calculated while the lens is not being controlled after the noise reduction process to replace the power spectrum for frequency bands where noise remains in the external microphone signal and where the amount of remaining noise is greater than the amount of remaining noise in the target sound collecting microphone, similar to step S15 in Fig. 3. Thereafter, the lens noise reduction process in Fig. 8 ends.

[0137] As described above, lens noise in a frequency band where noise remains even after conventional noise reduction processing can be reduced, so lens noise can be effectively reduced.

[0138] <Wind Noise Reduction Processing> FIG. 9 is a flowchart illustrating the wind noise reduction processing performed by the ratio calculation unit 22, the noise reduction processing unit 121, and the power spectrum substitution unit 24 in FIG.

[0139] The processing in steps S151 to S154 in FIG. 9 is the same as that in steps S11 to S14 in FIG. 4 except for the noise reduction processing using the noise reference microphone signal in step S153, and therefore a description thereof will be omitted.

[0140] The process in step S153 is the same as that in step S133 in FIG. 8, and therefore a description thereof will be omitted.

[0141] <4. Other> <Effects of the Present Technology> With the present technology, at least one of noise in a first audio signal output from a first microphone and noise in a second audio signal output from a second microphone provided in a position different from the first microphone is reduced, and noise in at least a part of the frequency band of the first audio signal is reduced using the second audio signal.

[0142] As a result, noise can be effectively reduced.

[0143] In addition, in the present technology, the first microphone is an external microphone that is detachably attached to the housing of the imaging device and collects surrounding sounds, and the second microphone is an internal microphone that is built into the housing and collects surrounding sounds.

[0144] Therefore, noise can be effectively reduced, particularly in the external microphone signal from the first microphone.

[0145] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0146] FIG. 10 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0147] A CPU 301 , a ROM (Read Only Memory) 302 , and a RAM (Random Access Memory) 303 are interconnected by a bus 304 .

[0148] An input / output interface 305 is also connected to the bus 304. An input unit 306 including a keyboard, a mouse, etc., and an output unit 307 including a display, a speaker, etc. are connected to the input / output interface 305. In addition, a storage unit 308 including a hard disk, a nonvolatile memory, etc., a communication unit 309 including a network interface, etc., and a drive 310 that drives removable media 311 are also connected to the input / output interface 305.

[0149] In a computer configured as described above, the CPU 301 performs the above-described series of processes by, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program.

[0150] The program executed by the CPU 301 is provided, for example, by being recorded on a removable medium 311 or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 308 .

[0151] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0152] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0153] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0154] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0155] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0156] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0157] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0158] <Examples of Combinations of Configurations> The present technology may also be configured as follows. (1) A signal processing device comprising: a noise reduction processing unit that reduces at least one of noise in a first audio signal output from a first microphone and noise in a second audio signal output from a second microphone provided at a position different from the first microphone; and a power spectrum replacement unit that reduces noise in at least a part of a frequency band of the first audio signal by using the second audio signal. (2) The signal processing device described in (1), wherein the first microphone is an external microphone that is detachably attached to a housing of an imaging device and collects ambient sound, and the second microphone is an internal microphone that is built into the housing and collects ambient sound. (3) The signal processing device described in (1) or (2), wherein the power spectrum replacement unit reduces noise in at least a part of a frequency band of the first audio signal by replacing the first audio signal with a signal obtained by multiplying the second audio signal by a ratio of the power spectra of the first audio signal and the second audio signal, in at least a part of the frequency band. (4) The signal processing device according to (3), wherein the at least a portion of the frequency band is a low frequency portion. (5) The signal processing device according to (3) or (4), further comprising a ratio calculation unit that calculates a ratio of power spectra of the first audio signal and the second audio signal. (6) The signal processing device according to (1), wherein the noise reduction processing unit reduces noise in the first audio signal by using the first audio signal and a third audio signal output from a third microphone provided at a position different from the first microphone and the second microphone, and reduces noise in the second audio signal by using the second audio signal and the third audio signal, and the power spectrum replacement unit reduces noise remaining in the first audio signal by using the first audio signal after noise reduction and the second audio signal after noise reduction.(7) The signal processing device according to (6), wherein the first microphone is a detachable external microphone attached to the housing of an imaging device, the second microphone is a built-in microphone built into the housing and collecting ambient sound, and the third microphone is a noise reference microphone provided inside the housing and collecting noise inside the housing. (8) The signal processing device according to (6) or (7), wherein the power spectrum substitution unit reduces noise remaining in the first audio signal by replacing the noise-reduced first audio signal with a signal obtained by multiplying the noise-reduced second audio signal by a ratio of the power spectra of the first audio signal and the second audio signal, in at least a part of the frequency band. (9) The signal processing device according to (8), wherein the at least part of the frequency band is a low-frequency portion. (10) The signal processing device according to (8) or (9), further comprising a ratio calculation unit that calculates a ratio of the power spectra of the first audio signal and the second audio signal. (11) The signal processing device according to (10), wherein the ratio calculation unit also calculates a ratio of power spectra between the first audio signal and the third audio signal and a ratio of power spectra between the second audio signal and the third audio signal. (12) The signal processing device according to (7), wherein the signal processing device is the imaging device. (13) The signal processing device according to any of (1) to (12), wherein the noise reduction processing unit reduces noise by estimation using an unmixing matrix. (14) A signal processing method, wherein a signal processing device reduces at least one of noise in a first audio signal output from a first microphone and noise in a second audio signal output from a second microphone provided at a position different from the first microphone, and noise in at least a part of a frequency band of the first audio signal is reduced by using the second audio signal.(15) A program that causes a computer to function as: a noise reduction processing unit that reduces at least one of noise in a first audio signal output from a first microphone and noise in a second audio signal output from a second microphone provided at a position different from the first microphone; and a power spectrum replacement unit that uses the second audio signal to reduce noise in at least a portion of the frequency band of the first audio signal.

[0159] 1, 2 Microphone, 10 Imaging device, 11 Microphone, 11-1 External microphone, 11-2 Target sound collection microphone, 11-3 Noise reference microphone, 12, 12-1 to 12-3 A / D converter, 13 Signal processing unit, 14 Wind noise detection unit, 21, 21-1 to 21-3 STFT unit, 22 Ratio calculation unit, 23 Noise reduction processing unit, 24 Power spectrum substitution unit, 25 ISTFT-Overlap processing unit, 101 Imaging device, 111 Signal processing unit, 121 Noise reduction processing unit

Claims

1. A signal processing device comprising: a noise reduction processing unit that reduces at least one of noise in a first audio signal output from a first microphone and noise in a second audio signal output from a second microphone provided at a position different from the first microphone; and a power spectrum replacement unit that uses the second audio signal to reduce noise in at least a portion of the frequency band of the first audio signal.

2. The signal processing device according to claim 1, wherein the first microphone is an external microphone that is detachably attached to the housing of the imaging device and collects surrounding sounds, and the second microphone is an internal microphone that is built into the housing and collects surrounding sounds.

3. The signal processing device according to claim 2, wherein the power spectrum replacement unit reduces noise in at least a portion of the frequency band of the first audio signal by replacing the first audio signal with a signal obtained by multiplying the second audio signal by a ratio of the power spectra of the first audio signal and the second audio signal in at least a portion of the frequency band.

4. The signal processing device according to claim 3, wherein the at least a portion of the frequency band is a low frequency portion.

5. The signal processing device according to claim 3, further comprising a ratio calculation unit that calculates a ratio between the power spectra of the first audio signal and the second audio signal.

6. The signal processing device according to claim 1, wherein the noise reduction processing unit reduces noise in the first audio signal using the first audio signal and a third audio signal output from a third microphone provided at a position different from the first microphone and the second microphone, and reduces noise in the second audio signal using the second audio signal and the third audio signal, and the power spectrum replacement unit reduces noise remaining in the first audio signal using the first audio signal after noise reduction and the second audio signal after noise reduction.

7. The signal processing device according to claim 6, wherein the first microphone is a detachable external microphone attached to the outside of the housing of the imaging device, the second microphone is a built-in microphone built into the housing and collects ambient sound, and the third microphone is a noise reference microphone provided inside the housing and collects noise inside the housing.

8. The signal processing device according to claim 7, wherein the power spectrum replacement unit reduces noise remaining in the first audio signal by replacing the first audio signal after noise reduction with a signal obtained by multiplying the second audio signal after noise reduction by a ratio of the power spectra of the first audio signal and the second audio signal, in at least a portion of the frequency bands.

9. The signal processing device according to claim 8, wherein the at least a portion of the frequency band is a low frequency portion.

10. The signal processing device according to claim 8, further comprising a ratio calculation unit that calculates a ratio between the power spectra of the first audio signal and the second audio signal.

11. The signal processing device according to claim 10, wherein the ratio calculation unit also calculates a ratio between the power spectra of the first audio signal and the third audio signal, and a ratio between the power spectra of the second audio signal and the third audio signal.

12. The signal processing device according to claim 7, wherein the signal processing device is the imaging device.

13. The signal processing device according to claim 1, wherein the noise reduction processing unit reduces noise by estimation using an unmixing matrix.

14. A signal processing method in which a signal processing device reduces at least one of noise in a first audio signal output from a first microphone and noise in a second audio signal output from a second microphone provided at a position different from the first microphone, and reduces noise in at least a part of the frequency band of the first audio signal by using the second audio signal.

15. A program that causes a computer to function as a noise reduction processing unit that reduces at least one of the noise of a first audio signal output from a first microphone and the noise of a second audio signal output from a second microphone provided at a position different from the first microphone, and a power spectrum replacement unit that uses the second audio signal to reduce noise in at least a portion of the frequency band of the first audio signal.