Imaging device, control method, and program
Patent Information
- Application Number
- JP2021191538
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-11-25
- Publication Date
- 2024-11-27
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Conventional noise reduction methods using a noise reference microphone fail to adequately suppress noise components other than lens drive noise, leading to insufficient or excessive reduction of audio signals due to self-noise and external sound leakage.
An imaging device with an external sound microphone and a noise reference microphone that detects and updates background noise, allowing for accurate extraction and reduction of lens drive noise by generating an audio signal related to driving noise and applying noise reduction means during non-noise periods.
The solution effectively suppresses the influence of temporary noise mixtures, enhancing noise reduction performance by accurately isolating and reducing lens drive noise while minimizing interference from other noise sources.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to an imaging device capable of reducing noise contained in an audio signal. [Background technology]
[0002] Conventionally, a process for reducing lens driving noise using a noise reference microphone installed inside a camera body has been known. Patent Document 1 discloses a process for sequentially estimating the impulse response of the transfer system of the main microphone from the noise reference microphone so as to minimize the result of subtracting the impulse response from the signal of the noise reference microphone, and then subtracting the signal obtained by convolving the impulse response into the signal of the noise reference microphone from the signal of the main microphone. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 6-253387 Summary of the Invention [Problem to be solved by the invention]
[0004] However, in addition to lens drive noise, noise reference microphones can also be contaminated with noise thought to be generated from factors other than lens drive, such as self-noise, such as electrical noise in the microphone, and external sound leakage. As a result, there are cases where noise reduction is not performed appropriately, such as when the audio signal input from the main microphone is excessively reduced or when noise reduction is insufficient.
[0005] Therefore, an object of the present invention is to suppress the influence of temporary noise on noise reduction processing. [Means for solving the problem]
[0006] an external audio microphone for collecting external audio from the imaging device; a noise reference microphone for acquiring drive noise from a drive unit of the imaging device or an external device connected to the imaging device; a detection unit for detecting a period during which drive noise is occurring; an update unit for updating background noise based on an audio signal acquired by the noise reference microphone during a period during which the drive noise detected by the detection unit is not occurring; a generation unit for generating an audio signal related to the drive noise from the audio signal acquired by the noise reference microphone and the background noise updated by the update unit; and a noise reduction unit for reducing the audio signal related to the drive noise generated by the generation unit from the audio signal acquired by the external audio microphone during a period during which the drive noise detected by the detection unit is occurring. [Effects of the Invention]
[0007] According to the present invention, it is possible to suppress the influence of temporary noise contamination on noise reduction processing. [Brief explanation of the drawings]
[0008] [Figure 1] FIG. 2 is an example of a block diagram of an imaging device according to the first embodiment. [Figure 2] FIG. 2 is an example of a block diagram of an audio processing unit and an audio collection unit according to the first embodiment. [Figure 3] 4 is a flowchart of an audio processing unit according to the first embodiment. [Figure 4] 10A and 10B are diagrams illustrating an example of a change in influence of a background noise spectrum acquired in the first embodiment. [Figure 5] FIG. 10 is an example of a block diagram of an audio processing unit and an audio collection unit according to a second embodiment. [Figure 6] 10 is a flowchart of an audio processing unit in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. Note that the same components will be described with the same reference numerals.
[0010] [First embodiment] FIG. 1 is a block diagram showing an example of the configuration of an image capturing apparatus 100 according to the first embodiment.
[0011] The image capturing device 100 comprises a lens unit 101, a lens control unit 102, an image capturing unit 103, an image processing unit 104, a control unit 105, an operation unit 106, a display / playback unit 107, a recording unit 108, an audio processing unit 200, and an audio pickup unit 300.
[0012] The lens unit 101 performs operations such as autofocus and zooming based on signals from the lens control unit 102. The lens unit 101 has a drive unit for performing operations such as autofocus and zooming. The drive unit is, for example, a motor such as a stepping motor or an ultrasonic motor. Here, the lens unit 101 may be configured to be detachable from the image capturing device 100. In this embodiment, the lens unit 101 is described as being connected to the image capturing device 100. The image capturing unit 103 captures an optical image of a formed subject using an image capturing element such as a CMOS sensor and outputs a digitized image signal to the image processing unit 104. The image processing unit 104 adjusts the image quality of the image signal input from the image capturing unit 103 and outputs the result. The control unit 105 is composed of a processor or the like and sends control signals to each block of the image capturing device 100. The operation unit 106 inputs user instructions to start and stop capturing, or capture settings, using a touch panel, dial, or the like. The display / playback unit 107 displays captured images and videos or plays audio signals accompanying the videos. The recording unit 108 records the captured images and moving images.
[0013] Next, the sound processing unit 200 and the sound collection unit 300 will be described with reference to FIG.
[0014] <Description of Audio Processing Unit 200> The audio processing unit 200 comprises an A / D conversion unit 201 , a waveform extraction unit 202 , a time-frequency conversion unit 203 , a noise period detection unit 204 , a background noise update unit 205 , a noise calculation unit 206 , a switching unit 207 , a noise reduction unit 208 , and a frequency-time conversion unit 209 .
[0015] A / D conversion unit 201 samples analog signals from multiple microphones input from sound collection unit 300 at the same timing and converts them into digital signals. Although it is represented as one block in FIG. 2, it has multiple A / D converters corresponding to the number of input channels from sound collection unit 300. The converted digital microphone signals are output to waveform extraction unit 202.
[0016] The waveform cutout unit 202 cuts out the digital audio signal input from the A / D conversion unit 201 to a predetermined length for each channel, performs windowing processing, and outputs the result to the time-frequency conversion unit 203. The series of processes performed by the waveform cutout unit 202 is, for example, half-overlap processing used in general audio processing. In this embodiment, the waveform cutout unit 202 cuts out 1024 samples while shifting the time by 512 samples, performs windowing processing using a sine window or Hann window, and outputs the result. Hereinafter, this cutout unit of 1024 samples is referred to as one frame. Thereafter, while shooting is being performed, signal processing is performed in the above cutout units.
[0017] The time-frequency conversion unit 203 converts the audio signal input from the waveform extraction unit 202 from a time-domain audio signal into a frequency-domain audio spectrum by processing such as Fourier transform. The audio spectrum generated from the audio signal input from the external audio microphone 301 is output to either the noise reduction unit 208 or the frequency-time conversion unit 209 via the switching unit 207 based on the detection result by the noise period detection unit 204. Hereinafter, the audio to be picked up by the external audio microphone 301 will be referred to as external audio or environmental sound. On the other hand, the noise reference sound spectrum, which is the audio spectrum generated from the audio signal input from the noise reference microphone 302, is output to the background noise update unit 205 and the noise calculation unit 206.
[0018] The noise period detection unit 204 acquires drive information from the lens control unit 102 when driving the lens unit 101, which is the source of noise, and detects a noise period based on the acquired drive information. The drive information indicates, for example, the lens drive speed by the drive unit, the lens drive direction, the position of the drive member used for drive, etc. The drive information also includes, for example, control information (instruction information) instructing the lens control unit 102 to start or end lens drive to the lens unit 101.
[0019] The drive information may be internal information used by the lens control unit 102 to determine whether to drive the lens, such as detection of an out-of-focus state or an in-focus state. The drive information may be transmitted as a signal to the noise period detection unit 204. The noise period detection unit 204 may perform noise detection based on the noise reference sound spectrum input from the time-frequency conversion unit 203. In this case, the noise period detection unit 204 may determine whether the input noise reference sound spectrum is noise for each frame.
[0020] The noise period detection unit 204 may further detect multiple types of noise based on the drive information or the noise reference sound spectrum. For example, the noise period detection unit 204 may detect long-term noise that continues for a certain period of time due to the operation of the drive unit, which is a noise source, and short-term noise that occurs before and after the long-term noise.
[0021] The noise period detected here is output to the background noise update unit 205 and the switching unit 207. Here, background noise refers to noise other than lens driving noise, such as noise caused by contact between the camera body or lens and the user's body (touch noise), self-noise such as electrical noise from a microphone, and leakage of external sound.
[0022] The background noise update unit 205 has a storage unit that stores a background noise spectrum included in a noise reference sound spectrum (not shown), and updates the stored background noise spectrum as appropriate. It is desirable that the background noise spectrum does not include a lens driving noise component among the noise reference sound spectrum. Therefore, when the noise period detection unit 204 determines that the period is outside the noise period, the background noise update unit 205 calculates a new background noise spectrum using Equation 1 to reduce the lens driving noise component included in the background noise spectrum.
[0023]
number
[0024] where S nrefc (ω,t) is the noise reference sound spectrum, S nrbkg (ω,t) is the background noise spectrum, ω is the frequency, t is the processing frame, and α is a predetermined coefficient. nrbkg (ω, t) is the background noise spectrum after updating. In this way, the new external environmental sound spectrum after updating is the average value obtained by weighting the external environmental sound spectrum. The period during which the weighted average is performed for the processing frames is a fixed period (predetermined period). Note that at t=0, that is, at the start of the noise reduction processing, the initial value of the background noise spectrum is the noise reference sound spectrum |S nref (ω,0)| or a pre-recorded background noise spectrum.
[0025] According to Equation 1, the influence of the background noise spectrum of the previous frame on the current processing frame t on the next frame is 1 / (α+1). For example, as shown in Figure 4, when α = 1, the influence of a given frame is halved with each frame. In this way, the influence of the previous frame is stronger, and the influence of past frames rapidly weakens over time. This makes it possible to reduce the influence of noise other than lens driving noise, such as touch noise, mixed into the audio signal input from the noise reference microphone within an appropriate period of time.
[0026] The noise calculation unit 206 calculates the noise component spectrum contained in the audio signal input from the noise reference microphone 302 by subtracting the background noise spectrum from the noise reference sound spectrum. The noise component is an audio signal of noise contained in the audio signal. The noise calculation unit 206 performs correction to bring the noise component of the audio signal input from the noise reference microphone closer to the noise component contained in the audio signal input from the external audio microphone 301. Here, the noise calculation unit 206 is provided with a correction coefficient table for this correction, and obtains the corrected noise spectrum by multiplying the correction coefficient according to the audio spectrum input from the noise reference microphone. The noise calculation unit 206 outputs the corrected noise component spectrum to the noise reduction unit 208.
[0027] The switching unit 207 outputs the audio spectrum of a frame determined to be within a noise period by the noise period detection unit 204, out of the audio spectra of the signal input from the external audio microphone 301, to the noise reduction unit 208. The audio spectrum of a frame determined to be outside a noise period is output to the frequency-time conversion unit 209 (without passing through the noise reduction unit 208).
[0028] The noise reduction unit 208 reduces noise from the audio spectrum of the external audio microphone 301 input from the time-frequency conversion unit 203 by using the corrected noise component spectrum input from the noise calculation unit 206, thereby obtaining a noise-reduced audio spectrum. Hereinafter, the audio spectrum of the external audio microphone 301 is also referred to as the external audio spectrum. The noise reduction unit 208 uses, for example, a Wiener filter as noise reduction means.
[0029] The frequency-time transform unit 209 transforms the external speech spectrum input from the switching unit 207 or the noise-reduced speech spectrum input from the noise reduction unit 208 into a waveform signal in the time domain by processing such as inverse Fourier transform. In this embodiment, the frequency-time transform unit 209 outputs frames while adding them together using half overlap.
[0030] The output signal is recorded in the recording unit 108 together with the image signal from the image processing unit 104 .
[0031] <Explanation of the sound pickup unit 300> The sound collection unit 300 comprises an external sound microphone 301 and a noise reference microphone 302 .
[0032] The external audio microphone 301 consists of two microphones and is installed to mainly capture the audio of the subject. In this embodiment, the two microphones capture audio corresponding to the audio signals of the Rch and Lch of the stereo audio.
[0033] The noise reference microphone 302 is installed so as to mainly capture driving noise inside the camera housing. For example, the noise reference microphone 302 is installed without an opening to the outside and shielded by an exterior so that external sound is not input. The noise reference microphone is also installed near the external sound microphone 301 to detect noise that is closer to the noise input to the external sound microphone 301. Alternatively, the noise reference microphone may be installed near the noise source to capture noise more accurately.
[0034] The functions of the above-described components allow the lens driving noise, which is the target of reduction, to be extracted with high accuracy, and the influence of other temporary noise contamination is suppressed, thereby improving noise reduction performance.
[0035] 3 is a flowchart of the audio processing of this embodiment. The processing of this flowchart is executed when the start of video recording is triggered. The following processing is realized by a control unit such as a processor included in the image capturing device 100 controlling each unit of the image capturing device 100, such as the audio processing unit 200 and the sound collection unit 300. The control unit also realizes the processing of this flowchart by expanding software recorded in the image capturing device 100 into memory and executing it.
[0036] In step S101, the waveform cutting unit 202 cuts out a waveform from the digital signal that is the output of the A / D conversion unit 201. The cut-out signal is output to the time-frequency conversion unit 203.
[0037] In step S102, a fast Fourier transform (FFT) process is performed on the digital signal input to the time-frequency conversion unit 203. The signal obtained by FFT processing the signal from the external audio microphone 301 is output to the switching unit 207, and the signal obtained by FFT processing the signal from the noise reference microphone 302 is output to the background noise update unit 205 and the noise calculation unit 206.
[0038] In step S103, the noise period detection unit 204 performs a noise period detection process. The noise period detection result is output to the background noise update unit 205 and the switching unit 207.
[0039] If the noise period detection unit 204 determines in step S104 that the period is outside the noise period, the background noise spectrum is updated for the processed frames in that period in step S105. The background noise update unit 205 calculates a new background noise spectrum according to Equation 1 and updates the stored background noise spectrum.
[0040] On the other hand, if it is determined in step S104 that the period is within a noise period, in step S106, the noise calculation unit 206 calculates the noise component spectrum contained in the speech signal input from the noise reference microphone 302 for the processing frame of that period. Here, in this step, the noise calculation unit 206 subtracts the background noise spectrum from the noise reference sound spectrum. The noise spectrum generated in step S106 is output to the noise reduction unit 208.
[0041] In step S107, the noise reduction unit 208 performs processing to reduce noise components contained in the audio signal input from the external audio microphone 301. Here, noise is reduced from the external audio spectrum calculated in step S102 using the noise component spectrum calculated in step S106. The processing in step S107 generates an audio spectrum after noise reduction. In step S107, a Wiener filter, for example, is used as noise reduction means. Alternatively, in step S107, waveform subtraction in the frequency domain, for example, may be performed as noise reduction means. The audio spectrum after noise reduction is output to the frequency-time transform unit 209.
[0042] In step S108, regardless of whether the process of step S107 has been performed, the frequency-time transform unit 209 performs an inverse fast Fourier transform (IFFT) process on the input audio spectrum. The converted signals are sequentially output to the recording unit 108 and recorded together with the image (moving image).
[0043] Then, the processes of steps S101 to S108 are repeated until it is determined in step S109 that the shooting has ended. The shooting is determined to have ended when, for example, the user performs an operation to end the recording of the moving image.
[0044] By carrying out the processing control described above, the lens driving noise to be reduced can be extracted with high accuracy, and the influence of other temporary noise contamination is suppressed, thereby improving noise reduction performance.
[0045] In this embodiment, the recording medium of the recording unit 108 is, for example, a semiconductor memory such as an SD card or a CFExpress card.
[0046] The image capturing device 100 may further include a data compression unit to compress the data of images and moving images to be recorded.
[0047] In this embodiment, the drive noise is lens drive noise, but noise generated by other drive units on the image capturing device body side may also be reduced in the same manner.
[0048] In this embodiment, each of the units constituting the audio processing unit 200, except for the A / D conversion unit 201, may be processed by a program using a CPU. Furthermore, each of the units constituting the audio processing unit 200, except for the A / D conversion unit 201, may be processed by hardware, such as a DSP, a dedicated LSI, or other electronic circuit.
[0049] The noise calculation unit 206 may perform different noise estimation processes depending on the type of noise detected by the noise period detection unit 204 .
[0050] In this embodiment, the noise reduction unit 208 uses a Wiener filter as noise reduction means, but other methods may be used. For example, the noise reduction unit 208 may use spectral subtraction or perform waveform subtraction in the time domain. The noise reduction unit 208 may also perform processing to further reduce the level of signals in frequency bins whose levels are equal to or lower than a predetermined threshold. The noise reduction unit 208 may also perform different noise reduction processing depending on the type of noise detected by the noise period detection unit 204.
[0051] In this embodiment, the external audio microphone 301 is two microphones (stereo microphones), but similar processing is possible even if the number of channels is different, such as monaural, surround, or Ambisonics.
[0052] In this embodiment, the audio processing unit 200 performs only noise reduction processing for simplicity, but it may also perform other processing. For example, it may be configured to perform spectral correction processing such as equalization to make audio easier to hear, or processing to emphasize the stereo effect of the reproduced audio. It may also be configured to perform encoding using various audio codecs such as MP3 and AAC.
[0053] In this embodiment, the noise reference microphone 302 is configured to consist of one microphone, but more microphones may be provided. Furthermore, some of the noise reference microphones 302 may be configured to be attached to the lens side. Furthermore, some or all of the noise reference microphones 302 may be vibration sensors that detect vibrations of an object, rather than microphones that capture vibrations in the air.
[0054] Furthermore, although the noise reference microphone 302 is set up near the external audio microphone 301 or near the noise source, the setting location is not important as long as noise can be acquired that can be used to estimate the noise components input to the external audio microphone 301.
[0055] [Second embodiment] 5 and 6 are diagrams illustrating a second embodiment of the present invention.
[0056] In the second embodiment, the noise period detection unit 204 may compare the noise reference sound spectrum of the frame in question with the stored background noise spectrum in addition to the excitation information, and determine whether or not it is a noise period based on the ratio.
[0057] Furthermore, in the first embodiment, the audio spectrum of a frame determined by the noise period detection unit 204 to be outside a noise period is output by the switching unit 207 to the frequency-time conversion unit 209 without passing through the noise reduction unit 208. On the other hand, in the second embodiment, as shown in Figures 5 and 6, a second noise reduction unit 210 is provided before the frequency-time conversion unit 209. In this case, the audio signal is subjected to noise reduction processing (second noise reduction processing) in step S110 regardless of whether it passes through the noise reduction unit 208 or not.
[0058] The second noise reduction processing may be, for example, processing that reduces background noise, such as pre-recorded background noise of the external audio microphone element, from the audio spectrum of the signal input from the external audio microphone 301 or the audio spectrum after noise reduction processing.
[0059] Furthermore, when the external audio microphone 301 and the noise reference microphone 302 have similar frequency characteristics, the second noise reduction process may be, for example, a process of subtracting a stored background noise spectrum from the audio spectrum. The audio spectrum here is either an audio spectrum generated from the audio signal input from the external audio microphone 301 or an audio spectrum after noise reduction processing.
[0060] The second noise reduction unit 210 uses a Wiener filter as noise reduction means, and may also use, for example, spectral subtraction or waveform subtraction in the time domain to perform noise reduction.
Claims
1. An imaging device, an external sound microphone for acquiring external sound from the imaging device; a noise reference microphone for acquiring driving noise from a driving unit of the image capture device or from an external device connected to the image capture device; a detection means for detecting a noise period during which the drive noise is generated; an update means for storing background noise based on the audio signal acquired by the noise reference microphone, and updating the stored background noise based on the audio signal acquired by the noise reference microphone in a period other than the noise period detected by the detection means; a generating means for generating an audio signal of the driving noise based on the audio signal acquired by the noise reference microphone and the background noise updated by the updating means; a noise reduction unit that performs a process of reducing the driving noise from the audio signal acquired by the external audio microphone based on the audio signal of the driving noise generated by the generation unit during the noise period detected by the detection unit; 13. An imaging device comprising:
2. The update means calculates an average value of the voice signal acquired by the noise reference microphone during a period other than the noise period and the stored background noise, and updates the stored background noise based on the average value.
2. The imaging device according to claim 1 .
3. The update means performs a weighted average of a voice signal for a predetermined period acquired by the noise reference microphone during a period other than the noise period and the stored background noise, and updates the stored background noise based on an average value of the weighted average.
3. The imaging device according to claim 2.
4. The audio signal from the noise reduction means is further provided with a second noise reduction means for performing a noise reduction process different from that performed by the noise reduction means.
4. The photographing apparatus according to claim 1, wherein the first and second lenses are arranged in a first direction.
5. 5. The image capturing apparatus according to claim 1, wherein the detection means detects the noise period based on control information for a drive unit of the image capturing apparatus or control information for the external device.
6. The detection means detects the noise period based on a ratio of the audio signal acquired by the noise reference microphone to the stored background noise.
6. The imaging device according to claim 1, wherein the imaging device is a single lens.
7. A conversion means for converting the audio signal acquired by the external audio microphone and the audio signal acquired by the noise reference microphone from an audio signal in a time domain to an audio spectrum in a frequency domain, The generating means generates a speech spectrum of the driving noise in a frequency domain; The imaging device according to any one of claims 1 to 6, characterized in that the noise reduction means performs a process of reducing the drive noise from the audio spectrum of the audio signal acquired by the external audio microphone using the audio spectrum of the drive noise generated by the generation means.
8. A control method for an image capture device having an external sound microphone for acquiring external sound from the image capture device, and a noise reference microphone for acquiring driving noise from a driving unit of the image capture device or an external device connected to the image capture device, comprising: a detection step of detecting a noise period during which the driving noise occurs; an updating step of storing background noise based on the audio signal acquired by the noise reference microphone, and updating the stored background noise based on the audio signal acquired by the noise reference microphone during the noise period detected by the detection step; a generation step of generating an audio signal of the driving noise based on the audio signal acquired by the noise reference microphone and the background noise updated by the update step; a noise reduction step of performing a process of reducing the driving noise from the audio signal acquired by the external audio microphone based on the audio signal of the driving noise generated by the generation step during the noise period detected by the detection step; A control method comprising the steps of:
9. A program for causing a computer of an image capturing apparatus having an external sound microphone for acquiring external sound of the image capturing apparatus and a noise reference microphone for acquiring driving noise of a driving unit of the image capturing apparatus or an external apparatus connected to the image capturing apparatus to execute a control method, the control method comprising the steps of: a detection step of detecting a noise period during which the driving noise occurs; an updating step of storing background noise based on the audio signal acquired by the noise reference microphone, and updating the stored background noise based on the audio signal acquired by the noise reference microphone during the noise period detected by the detection step; a generation step of generating an audio signal of the driving noise based on the audio signal acquired by the noise reference microphone and the background noise updated by the update step; and a noise reduction process for performing processing to reduce the drive noise from the audio signal acquired by the external audio microphone based on the audio signal of the drive noise generated by the generation process during the noise period detected by the detection process.
