Imaging device

Abstract

To provide an imaging device that can easily acquire audio that aligns with the user's intentions when using the imaging device. [Solution] The imaging device comprises an imaging unit that captures an image of a subject and generates image data, a communication unit that communicates with a sound receiver, an audio processing unit that forms directionality in the sound picked up by the sound receiver, and a control unit that controls sound pickup by the sound receiver. The sound receiver is attached to the user's head and comprises a plurality of microphones that are each positioned at predetermined locations on the head to which it is attached. The control unit controls the communication unit to receive audio data from the sound receiver indicating the sound picked up by the imaging unit during imaging. The control unit controls the directionality of the received audio data by the audio processing unit according to the arrangement of the plurality of microphones on the head to which the sound receiver is attached, and outputs audio data that is recorded in association with the image data generated by the imaging unit.

Description

Technical Field

[0001] The present disclosure relates to an imaging device that acquires audio, for example, during video shooting.

Background Art

[0002] Patent Document 1 discloses an imaging device including a microphone and an imaging unit capable of shooting a video. The imaging device records by associating audio data generated by the microphone with moving image data shot by the imaging unit. When shooting a moving image, the audio captured by the microphone of the imaging device may include noises such as lens driving sound and wind cut sound. In addition to the microphone, the imaging device of Patent Document 1 uses a smartphone as an external recording device to obtain audio with reduced noise.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides an imaging device that can easily acquire audio in accordance with the intention of a user using the imaging device.

Means for Solving the Problems

[0005] An imaging device in one aspect of this disclosure includes an imaging unit that captures an image of a subject and generates image data, a communication unit that communicates with a sound receiver, an audio processing unit that forms directionality in the sound picked up by the sound receiver, and a control unit that controls sound pickup by the sound receiver. The sound receiver is attached to the user's head and includes a plurality of microphones that are each positioned at predetermined locations on the head to which it is attached. The control unit controls the communication unit to receive audio data from the sound receiver indicating the sound picked up by the imaging unit during imaging. The control unit controls the directionality of the received audio data by the audio processing unit according to the arrangement of the plurality of microphones on the head to which the sound receiver is attached, and outputs audio data that is recorded in association with the image data generated by the imaging unit. [Effects of the Invention]

[0006] According to the imaging device described herein, it is possible to easily acquire audio that aligns with the user's intentions when using the imaging device. [Brief explanation of the drawing]

[0007] [Figure 1] Figure illustrating the imaging system according to Embodiment 1 of this disclosure. [Figure 2] A diagram illustrating the configuration of a digital camera in an imaging system. [Figure 3] A diagram illustrating the earphones worn by the user in an imaging system. [Figure 4] A diagram illustrating the configuration of earphones in an imaging system. [Figure 5] A flowchart illustrating the operation when connecting earphones to a digital camera. [Figure 6] A flowchart illustrating the device setup process in a digital camera. [Figure 7] A diagram illustrating the calculation of the distance between earphones in a digital camera. [Figure 8] A flowchart illustrating the operation of a digital camera during video recording. [Figure 9] Diagram illustrating the beamforming section in a digital camera. [Figure 10] A diagram illustrating the recording modes in digital cameras. [Figure 11] A diagram illustrating an example of changing the recording mode during video recording. [Figure 12] A flowchart illustrating the playback process after recording video with a digital camera. [Figure 13] A diagram illustrating an imaging system according to a modified example of Embodiment 1. [Modes for carrying out the invention]

[0008] The embodiments will be described in detail below, with reference to the drawings as appropriate. However, unnecessary details may be omitted. For example, detailed explanations of already well-known matters or redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding by those skilled in the art. The inventors provide the accompanying drawings and the following explanation so that those skilled in the art can fully understand this disclosure, and do not intend to limit the subject matter described in the claims by means of these.

[0009] (Embodiment 1) Embodiment 1 describes an imaging system that acquires sound during video recording.

[0010] 1. Configuration of the imaging system The imaging system in Embodiment 1 of this disclosure will be described with reference to Figure 1.

[0011] The imaging system 1 of this embodiment includes a digital camera 100 and two earphones 20L and 20R, as shown in Figure 1. In the imaging system 1 of this embodiment, each earphone 20L and 20R has a built-in microphone and is connected to the digital camera 100 for data communication. Hereinafter, earphones 20L and 20R will be collectively referred to as earphone 20. The digital camera 100 and earphones 20 of this system 1 are examples of an imaging device and a sound receiver, respectively, in this embodiment.

[0012] In this system 1, when the digital camera 100 takes a moving picture with the earphone 20 worn on the ear of a user such as a photographer, voice data etc. collected by the microphone from the earphone 20 is transmitted to the digital camera 100. Further, for example, after taking a moving picture, voice data etc. in the reproduction of the taken moving picture is transmitted from the digital camera 100 to the earphone 20.

[0013] <00000�8>The configurations of the digital camera 100 and the earphone 20 in this system 1 as described above will be explained below.

[0014] 1-1. Configuration of Digital Camera FIG. 2 is a diagram illustrating the configuration of the digital camera 100 in this system 1. The digital camera 100 includes an image sensor 115, an image processing unit 120, a display monitor 130, and a controller 135. Further, the digital camera 100 includes a buffer memory 125, a card slot 140, a flash memory​​​​​​​​​​​The lens drive unit 112 drives the focus lens and other components in the optical system 110. The lens drive unit 112 includes a motor and moves the focus lens along the optical axis of the optical system 110 based on the control of the controller 135. The configuration for driving the focus lens in the lens drive unit 112 can be implemented using a DC motor, stepping motor, servo motor, or ultrasonic motor, etc.

[0018] The image sensor 115 captures an image of a subject formed through the optical system 110 and generates imaging data. The imaging data constitutes image data representing the image captured by the image sensor 115. The image sensor 115 generates image data of a new frame at a predetermined frame rate (e.g., 30 frames / second). The timing of image data generation and the operation of the electronic shutter in the image sensor 115 are controlled by the controller 135. The image sensor 115 can use various image sensors, such as a CMOS image sensor, a CCD image sensor, or an NMOS image sensor.

[0019] The image sensor 115 performs actions such as capturing still images, moving images, or through images. Through images are mainly moving images and are displayed on the display monitor 130, for example, to allow the user to determine the composition for capturing still and moving images for recording. The image sensor 115 is an example of an imaging unit in this embodiment.

[0020] The image processing unit 120 performs various processes on the imaging data output from the image sensor 115 to generate image data, and also performs various processes on the image data to generate an image for display on the display monitor 130. Examples of these processes include, but are not limited to, white balance correction, gamma correction, YC conversion, electronic zoom, compression, and decompression. The image processing unit 120 may be composed of hardwired electronic circuits, or it may be composed of a microcomputer or processor using a program.

[0021] The display monitor 130 is an example of a display unit that displays various information. For example, the display monitor 130 displays an image (through image) represented by image data captured by the image sensor 115 and processed by the image processing unit 120. The display monitor 130 also displays a menu screen or the like for the user to make various settings for the digital camera 100. The display monitor 130 can be made of, for example, a liquid crystal display device or an organic EL device.

[0022] The operation unit 150 is a general term for the user interface that accepts user input. The operation unit 150 includes, for example, physical buttons such as release buttons and cursor buttons, levers, dials, switches, joysticks, and touch panels. Furthermore, the operation unit 150 also includes virtual buttons and icons displayed on the display monitor 130. When the operation unit 150 receives user input, it transmits an operation signal corresponding to that user input to the controller 135.

[0023] The controller 135 provides overall control over the operation of the digital camera 100. The controller 135 includes a CPU, and the CPU executes programs (software) to realize predetermined functions. Instead of a CPU, the controller 135 may include a processor consisting of dedicated electronic circuits designed to realize predetermined functions. In other words, the controller 135 can be implemented with various processors such as a CPU, MPU, GPU, DSU, FPGA, and ASIC. The controller 135 may consist of one or more processors. Alternatively, the controller 135 may be configured on a single semiconductor chip together with the image processing unit 120, etc.

[0024] The buffer memory 125 is a recording medium that functions as work memory for the image processing unit 120 and the controller 135. The buffer memory 125 is implemented using DRAM (Dynamic Random Access Memory) or the like. The flash memory 145 is a non-volatile recording medium. The controller 135 may also have various internal memories, such as built-in ROM. The ROM stores various programs that the controller 135 executes. The controller 135 may also have built-in RAM that functions as a work area for the CPU.

[0025] The card slot 140 is a means for inserting a removable memory card 142. The card slot 140 can electrically and mechanically connect to the memory card 142. The memory card 142 is an external memory equipped with recording elements such as flash memory. The memory card 142 can store data such as image data generated by the image processing unit 120.

[0026] The communication module 155 is a module (circuit) that connects to an external device according to a predetermined communication standard in wired or wireless communication. The predetermined communication standard includes, for example, Bluetooth®, IEEE 802.11, Wi-Fi®, USB, and HDMI®. The digital camera 100 can communicate with other devices via the communication module 155, and may communicate directly with other devices via the communication module 155, or it may communicate via an access point. The communication module 155 may be connectable to a communication network such as the Internet. The communication module 155 includes various antennas that transmit and receive various signals, such as wireless signals.

[0027] The audio processing unit 170 performs various audio processing on the input audio data. In the digital camera 100, for example, audio data received from the earphone 20 via the communication module 155 is input to the audio processing unit 170 under control from the controller 135. Various audio processing may include, for example, gain adjustment processing to amplify or attenuate the sound. The audio processing unit 170 may be composed of hardwired electronic circuits, or it may be composed of a microcomputer, processor, etc. using a program.

[0028] In the digital camera 100 of this embodiment, the audio processing unit 170 includes a beamforming unit 172 and an output selector 174, as shown in Figure 2, for example. The beamforming unit 172 implements a function to control the directivity of sound. Details of the beamforming unit 172 will be described later. The output selector 174 switches the source of audio data to select the audio data to be output from the audio processing unit 170 among audio data input from multiple sources. For example, the output selector 174 switches the audio data that is output to the controller 135 and transmitted from the communication module 155 to the earphone 20.

[0029] 1-2. Earphone Configuration Figure 3 is a diagram illustrating the earphones 20 worn by user 3 in this system 1. For example, earphone 20L is worn on user 3's left ear and earphone 20R is worn on user 3's right ear.

[0030] For example, as shown in Figure 3, the earphone 20L includes an outer microphone 261L positioned to pick up external sounds for noise cancellation, an inner microphone 262L positioned to pick up the voice of the user 3, and a speaker 280L. In the earphone 20L, the outer microphone 261L, the inner microphone 262L, and the speaker 280L are arranged in order of furthest from the user 3 in the straight line direction connecting the user 3's ears. The inner microphone 262L includes, for example, two adjacent microphone elements. The earphone 20R is similar to the earphone 20L, and includes an outer microphone 261R, an inner microphone 262R, and a speaker 280R.

[0031] Figure 4 illustrates the configuration of the earphone 20 in this system 1. In addition to an external microphone 261, an internal microphone 262, and a speaker 280, the earphone 20 includes an audio processing unit 220, a controller 235, a buffer memory 225, a flash memory 245, an operation unit 250, and a communication module 255. Furthermore, the earphone 20 includes A / D converters 265 and 266 for the microphones. The earphone 20 also includes, for example, a proximity sensor 215.

[0032] The external microphone 261 includes, for example, one microphone element. The external microphone 261 is equipped with a microphone element that has a relatively wide bandwidth and a high signal-to-noise ratio, for example, to ensure accurate noise cancellation even in quiet environments.

[0033] The internal microphone 262 includes multiple microphone elements to form directionality in capturing the voice of user 3's speech. In this embodiment, an example in which two microphone elements are installed is described, but the internal microphone 262 may include three or more microphone elements.

[0034] The external microphone 261 and the internal microphone 262 convert the sound picked up by their respective microphone elements into an analog signal, which is an electrical signal, and output it.

[0035] The A / D converter 265 converts the analog signal output from the external microphone 261 into digital audio data. The A / D converter 266 converts the analog signal from the internal microphone 262 into audio data. The A / D converters 265 and 266 may be integrated with the microphone elements of the external microphone 261 and the internal microphone 262, respectively. In other words, the external microphone 261 and the internal microphone 262 may be digital microphones that include A / D converters, etc.

[0036] The audio processing unit 220 receives audio data output from each A / D converter 265, 266 and performs various audio processing on the received audio data. For example, the various audio processing may include gain adjustment processing. The audio processing unit 220 may be configured as a hardwired electronic circuit, or as a microcomputer or processor using a program.

[0037] In the earphone 20 of this embodiment, the audio processing unit 220 includes, for example, a beamforming unit 222, an input selector 224, and a noise canceller 226, as shown in Figure 4.

[0038] The beamforming unit 222 implements a function to control the directivity of the audio data from the inner microphone 262 and outputs audio data D0 with directionality formed. The input selector 224 switches the source of the audio data to select the audio data to input to the controller 235 from between the audio data D0 processed by the beamforming unit 222 from the audio data from the inner microphone 262 and the audio data from the outer microphone 261. The noise canceller 226 outputs audio data that is less audible to the user 3 by, for example, canceling out noise components such as external ambient sounds in the audio data output from the speaker 280, based on the audio data from the outer microphone 261.

[0039] The speaker 280 includes, for example, one or more speaker elements and outputs sound to the outside of the earphone 20 from audio data input by control from the controller 235.

[0040] The controller 235 provides overall control over the operation of the earphone 20. The controller 235 includes a CPU, and the CPU executes programs (software) to realize predetermined functions. The controller 235 can be implemented with various processors, similar to the controller 135 of the digital camera 100, for example. The controller 235 may consist of one or more processors. Alternatively, the controller 135 may be configured on a single semiconductor chip together with the audio processing unit 220, etc.

[0041] The buffer memory 225 is a recording medium that functions as work memory for the audio processing unit 220 and the controller 235. The buffer memory 225 is implemented using DRAM (Dynamic Random Access Memory) or the like. Various audio data, for example, is stored in the buffer memory 225. The flash memory 245 is a non-volatile recording medium. The controller 235 may also have various internal memories, for example, it may have built-in ROM. Various programs executed by the controller 235 are stored in the ROM. The controller 235 may also have built-in RAM that functions as a work area for the CPU.

[0042] The operation unit 250 is a general term for the user interface that accepts user operations. The operation unit 250 includes, for example, buttons, switches, and / or touch panels. When the operation unit 250 receives a user operation, it transmits an operation signal corresponding to that user operation to the controller 235.

[0043] The communication module 255 is a module (circuit) that connects to an external device according to a predetermined communication standard in wired or wireless communication, similar to the communication module 155 of a digital camera 100, for example. In this embodiment, the communication module 255 of the earphone 20 includes, for example, an antenna array that transmits wireless signals used for detecting the direction of the earphone 20 relative to the external device to which it is connected.

[0044] The proximity sensor 215 is configured, for example, as a photoelectric sensor, and detects the presence or absence of an object close to it based on the amount of light it receives. One application of the proximity sensor 215 is to detect whether or not the earphones 20 are being worn in the ears of user 3.

[0045] 2.Operation The operation of the imaging system 1, configured as described above, will now be explained.

[0046] In this system 1, the digital camera 100 establishes a communication connection for data communication with the earphone 20, for example as shown in Figure 1, and then performs video recording operations in response to user 3's input. During video recording, the digital camera 100 sequentially generates image data by capturing through images or moving images. In parallel with the imaging operation, the digital camera 100 of this system 1 also captures sound using the external microphone 261 or internal microphone 262 of the earphone 20 attached to user 3, for example as shown in Figure 3.

[0047] This system 1 uses the microphones 261 and 262 of the earphone 20 for sound pickup during video recording with the digital camera 100, thereby obtaining audio that is less affected by noise generated from various lenses or cooling fans in the digital camera 100. Furthermore, by using the earphone 20, which is located further away from the digital camera 100 than, for example, the built-in microphone of the digital camera 100 or an external microphone attached to the accessory shoe, sound quality degradation caused by noise generated from vibrations of the digital camera 100 can be suppressed in the picked-up audio.

[0048] In addition to suppressing sound quality degradation due to noise from the digital camera 100 as described above, this system 1 can capture sound with microphones positioned close to both of the user's ears in the earphones 20 worn by the user 3. As a result, this system 1 can acquire audio with enhanced realism, as if the user 3 were listening to it.

[0049] When the digital camera 100 receives audio data from the earphone 20 indicating sound picked up by the microphones 261 and 262, the audio processing unit 170 (see Figure 2) processes the audio data. For example, in this embodiment, the digital camera 100 controls the directivity of the audio data from the outer microphone 261 using the beam forming unit 172. Through this audio processing, the digital camera 100 of this embodiment can, for example, direct the directivity towards the front of the user 3 (for example, in the direction of the digital camera 100) or create different directivity to the left and right through stereo recording, depending on the shooting scene in which video is being recorded.

[0050] Furthermore, System 1 allows the audio recorded during video recording by the digital camera 100 to be played back while being picked up by the earphones 20. In this way, sound pickup and monitoring of the picked-up audio can be achieved simultaneously with the earphones 20. This allows User 3 to check the recorded audio while shooting video, making it easier to obtain audio that matches User 3's intentions. In addition, by using the earphones 20 for both sound pickup and monitoring, the number of pieces of equipment that User 3 uses for video recording can be reduced, making it easier for them to concentrate on shooting. The operation of the digital camera 100 in System 1 is described below.

[0051] 2-1. Connecting and setting up earphones First, using Figures 5 to 7, we will explain how the digital camera 100 of this system 1 initiates communication with the earphones 20 and performs settings according to the device information of the connected earphones 20. Below, we will explain an example in which the digital camera 100 and the earphones 20 are connected via Bluetooth.

[0052] Figure 5 is a flowchart illustrating the operation of the digital camera 100 when connected to the earphones 20. The process shown in Figure 5 starts, for example, when the digital camera 100 is powered on, and each process is executed by the controller 135 of the digital camera 100.

[0053] The controller 135 determines whether an external device, such as the earphone 20, has been connected to the communication network based on information such as a connection request received by the communication module 155 (S1). If the external device is not connected to the communication network (NO in S1), the controller 135 terminates the processing of this flowchart. After a predetermined period of time has elapsed since the termination of this processing, the processing from step S1 onward may be repeated.

[0054] If an external device is connected via communication (YES in S1), the controller 135 recognizes the connected device and obtains device information from it via the communication module 155 (S2). The device information includes, for example, identification information to identify the device, as well as the Bluetooth version, audio standard and codec available on the device.

[0055] The controller 135 determines, based on the device information, whether the connected external device is the earphone 20 with a built-in microphone (S3). If the connected external device is not the earphone 20 (NO in S3), the controller 135 terminates the process in this flowchart, assuming that the earphone 20 is not connected.

[0056] If the connected external device is the earphone 20 (YES in S3), the controller 135 configures the digital camera 100 to record and play back audio using the earphone 20 (S4). In this device configuration process (S4), the distance between the earphones 20L and 20R is set, for example, from the viewpoint of accurately beamforming the audio data from the external microphones 261L and 261R of the earphones 20L and 20R, which are attached to the left and right ears of user 3, respectively. Details of the device configuration process (S4) will be described later.

[0057] If a device other than the earphones 20 is connected to the digital camera 100, settings may be made according to that device. For example, if earphones without a microphone are connected, settings may be made to play audio using those earphones. After executing the device setting process (S4), the controller 135 terminates the processing of this flowchart.

[0058] According to the above process, the digital camera 100 acquires device information from an externally connected device (S2), and if the earphone 20 with a microphone is connected (YES in S3), it can set up recording and playback of audio using the earphone 20 (S4).

[0059] 2-1-1. Equipment Setup Process The device configuration process in step S4 of Figure 5 will be explained using Figures 6 and 7.

[0060] Figure 6 is a flowchart illustrating the device setup process (S4) in the digital camera 100. The process shown in Figure 6 starts when the earphones 20 are connected to the digital camera 100 (YES in S3), with the device information acquired in step S2 of Figure 5 being held in the buffer memory 125, etc.

[0061] First, the controller 135 determines, based on the device information, whether or not it is possible to measure the distance between the left and right earphones 20L and 20R in the connected earphones 20 (S11). For example, the controller 135 makes the determination in step S11 based on the Bluetooth version and audio standard in the device information. For example, if the Bluetooth version corresponds to a function that detects the direction of the earphones 20 from the connected device, and the audio standard is LE Audio which allows communication between the device and each of the earphones 20L and 20R, it is determined that distance measurement is possible (YES in S11).

[0062] If distance measurement is possible (YES in S11), the controller 135 calculates the distance between the earphones 20R and 20L by sending and receiving wireless signals to, for example, each earphone 20R and 20L (S12). Figure 7 is a diagram illustrating the calculation of the distance between the earphones 20R and 20L in the digital camera 100.

[0063] Figure 7(A) illustrates the positional relationship between the earphones 20L and 20R, which are attached to the left and right ears of user 3, and the communication module 155 of the digital camera 100. Figure 7(B) illustrates a different positional relationship than that shown in Figure 7(A).

[0064] First, the controller 135 determines the distances d1 and d2 between the digital camera 100 and the earphones 20L and 20R, respectively, by channel sounding, for example, using phase-based ranging (PBR). For example, two or more signals with different frequencies, such as a sweep signal, are transmitted and received between the controller and earphone 20L, and the phase change between the transmitted and received signals is calculated for each frequency. Then, from the phase change at each frequency, the phase difference due to the change in frequency is calculated, and the distance d1 can be calculated from the relationship that this phase difference occurs over the round trip path of distance d1 due to the transmission and reception of the signals. The distance d2 between earphone 20R and the digital camera 100 is calculated in the same way as distance d1.

[0065] Furthermore, in the imaging system 1 of this embodiment, a wireless signal for direction detection is transmitted from the earphone 20 whose direction is specified, and the digital camera 100 receives this signal via the communication module 155, thereby performing direction detection based on the angle of departure (AoD). Below, we will describe an example in which the multiple antennas included in the antenna arrays of each earphone 20L, 20R are arranged on a straight line connecting both ears of the user 3, as shown by the dashed line in Figure 3. For example, as shown in Figure 7(A), a signal is transmitted from the antenna array of earphone 20L to the digital camera 100 at a radiation angle θ1. The direction of earphone 20L relative to the digital camera 100 can be represented by this radiation angle θ1.

[0066] For example, the radiation angle θ1 can be calculated from the phase difference of the signals from each antenna in the antenna array of the earphone 20L. The phase difference between antennas can be calculated, for example, from the intensity of the cosine and sine components in the IQ data calculated based on the signal. The radiation angle θ2, which represents the direction of the earphone 20R relative to the digital camera 100, is calculated in the same way as θ1.

[0067] The distance D12 between earphones 20R and 20L can be calculated using the following formula, based on the trigonometric relationship shown in Figure 7(A), for example. D12 = d1 cosθ1 + d2 cosθ2

[0068] The above formula for calculating distance D12 also holds true when the positional relationship between the digital camera 100 and the earphones 20R and 20L is different from the example in Figure 7(A), as shown in Figure 7(B).

[0069] The controller 135 calculates the distance D12 between the earphones 20R and 20L, as described above (S12), and then sets the calculated distance D12 to the beamforming unit 172 (S14).

[0070] On the other hand, if the controller 135 determines that it is not possible to measure the distance between the connected earphones 20R and 20L (NO in S11), it accepts a user input, for example, via the operation unit 150, to input the distance D12 between the earphones 20R and 20L (S13). In step S13, the controller 135 may display a predetermined setting menu screen or the like on the display monitor 130. The distance D12 may be directly input as a numerical value in a predetermined unit, or it may be selected from a plurality of options pre-recorded in the flash memory 145 or the like. The controller 135 sets the input distance D12 in the beamforming unit 172 (S14).

[0071] After setting the distance D12 between earphones 20R and 20L (S14), the controller 135 terminates the processing of this flowchart.

[0072] According to the above device setting process (S4), if the digital camera 100 can measure the distance between the connected earphones 20R and 20L (YES in S11), it calculates the distance D12 (S12) and sets it in the beamforming unit 172 (S14). If distance measurement is not possible (NO in S11), the distance D12 input in the operation unit 150 is set (S13, S14). As a result, the beamforming unit 172 can accurately process the audio data, such as beamforming that reflects the arrangement of the external microphones 261L and 261R when the earphones 20 connected to the digital camera 100 are worn on the user 3's head.

[0073] Furthermore, by calculating the distance D12 in the digital camera 100 (S12), the influence of the user's head interfering with the signal can be suppressed, and the distance D12 can be calculated with greater accuracy than, for example, calculating it by wireless signal communication between earphones 20L and 20R.

[0074] In the above, an example was described in which, in the calculation of distance D12 (S12), multiple antennas in the antenna array of each earphone 20 are arranged on a straight line connecting the two ears of user 3. The multiple antennas do not have to be arranged on that straight line. For example, an offset indicating the difference between the straight line connecting the multiple antennas and the straight line connecting the two ears of user 3 may be included in the equipment information, such as the angle between the two straight lines, and in step S12, distance D12 may be calculated based on this offset in the equipment information.

[0075] In the above, an example was described in which, in the calculation of distance D12 (S12), the digital camera 100 performs direction detection according to the angle of arrival (AoD) using an antenna array mounted on the earphone 20. Such an antenna array for direction detection may be mounted on the communication module 155 of the digital camera 100, and in step S12, direction detection may be performed using the angle of arrival (AoA) instead of AoD.

[0076] The above describes an example in which, when distance measurement between earphones 20R and 20L is not possible (NO in S11), the user inputs the distance D12 (S13). For example, the controller 135 may determine from the identification information of the earphone 20 whether the earphone 20 has been connected in the past, and for earphones 20 that have been connected in the past, it may set the distance D12 that was set at the time of that connection again. Such distance D12 from past connections is stored in the flash memory 145 as a registered distance D12, for example, associated with the identification information. Even when distance measurement is possible (YES in S11), for earphones 20 that have been connected in the past, the setting in step S14 may be performed by reading the registered distance D12.

[0077] Furthermore, in the above example, user 3 may be able to select whether or not to use a registered distance D12 on the operation unit 150, and a message prompting user 3 to make such a selection may be displayed on the display monitor 130. Also, for example, if in step S12 any of the distances d1, d2 and / or radiation angles θ1, θ2 are not within a predetermined range, a message prompting user 3 to change the wearing position of the earphones 20 and repeat step S12 may be displayed, or a message prompting user 3 to perform the same user operation as in step S13, after notifying user 3 of an error in the calculation of distance D12 may be displayed. The predetermined range may be set considering, for example, the positional relationship between the earphones 20R, 20L and user 3's head, from the viewpoint of accurately calculating distance D12.

[0078] 2-2. Operation during video recording As described above, the operation of the digital camera 100 when recording video after setting the distance D12 between the connected earphones 20R and 20L will be explained using Figures 8 to 11.

[0079] Figure 8 is a flowchart illustrating the operation of a digital camera during video recording. The process shown in Figure 8 is initiated, for example, after setting the distance D12, when the digital camera 100 enters a video recording standby state. In the standby state, before the digital camera 100 starts the shooting operation in which it records image data and audio data in association, a pass-through image is displayed on the display monitor 130. Each process in Figure 8 is executed by the controller 135 in parallel with, for example, the operations in the standby state before video recording and the shooting operations during video recording.

[0080] 2-2-1. Beamforming The digital camera 100 of this embodiment has multiple recording modes for capturing sound during video recording and recording it in the video data. The multiple recording modes are pre-set, for example, so that sound with various directional characteristics is captured by the earphone 20.

[0081] For example, in this embodiment, the digital camera 100 performs beamforming using a beamforming unit 172 to control the directivity of the sound picked up by the external microphone 261 for noise cancellation in the earphone 20. The beamforming unit 172 controls the direction and range of the directivity of the sound picked up by each external microphone 261L, 261R, thereby setting the physical range in which the sound is picked up.

[0082] Figure 9 is a diagram illustrating the beamforming unit 172 in the digital camera 100. The controller 135 inputs audio data received from the external microphones 261L and 261R of the earphones 20L and 20R via the communication module 155 to the beamforming unit 172.

[0083] The beamforming unit 172 includes, for example, delay devices P1-P2, filters F1-F5, and adders A1-A4 as a functional configuration as shown in Figure 9. Each of these functional components of the beamforming unit 172 may be implemented by dedicated hardware circuits. The beamforming unit 172 adjusts the delay period of the sound picked up by the external microphones 261L and 261R, respectively, using delay devices P1 and P2, and then adjusts the frequency band of the sound with the adjusted delay period using filters F1-F5. The beamforming unit 172 outputs audio data D1-D3, which represents the signed sum of the sound before or after such adjustment, using adders A1-A4.

[0084] For example, delay unit P1 adjusts the delay period for sound arriving at the external microphone 261R from the direction of user 3's right ear by a phase difference calculated to compensate for the propagation delay corresponding to the distance D12 between earphones 20R and 20L. Delay unit P2 adjusts the delay period for sound arriving at the external microphone 261L from the direction of user 3's left ear by a phase difference corresponding to the distance D12, similar to delay unit P1. Filters F1 to F5 are, for example, bandpass filters, and their respective passbands and other filter characteristics are set from the viewpoint of improving the S / N ratio of the audio signal by suppressing noise.

[0085] 2-2-2. Recording Mode Figure 10 is a diagram illustrating the recording modes in the digital camera 100. The digital camera 100 has recording modes such as "Stereo," "Front," and "Narration," as shown in Figure 10. The recording mode table T1 shown in Figure 10(A) manages the audio data recorded as sound output from the left and right earphones 20L and 20R, respectively, 280L and 280R, for each recording mode. For example, the recording mode table T1 is pre-stored in the flash memory 145. In Figure 10, "Lch" and "Rch" correspond to the output sound from these speakers 280L and 280R, respectively, and indicate each channel in the stereo audio data.

[0086] For example, when the recording mode of the digital camera 100 is stereo mode, the audio data D2 and D3 generated by the beamforming unit 172 are recorded in channels Lch and Rch, respectively, as shown in Figure 9. Figure 10(B) illustrates the polar patterns corresponding to the directivity formed in the output audio of each channel Lch and Rch for each recording mode. For example, as shown in Figures 10(A) and (B), in stereo mode, audio data D2, which emphasizes sound from the left ear, is recorded in channel Lch, and audio data D3, which emphasizes sound from the right ear, is recorded in channel Rch.

[0087] Furthermore, when the recording mode is set to front mode, the digital camera 100 records the audio data D1 generated by the beamforming unit 172 to both channels Lch and Rch, as shown in Figure 9. The audio data D1 is generated to emphasize the sound in front of the user 3, for example, as shown in Figure 10(B). The filter F5 of the beamforming unit 172 shown in Figure 9 may have filter characteristics set to allow the frequency band included in human voices to pass through from the audio data D2 and D3. The beamforming unit 172 is not limited to the example in Figure 9 and may not include the filter F5.

[0088] Furthermore, in this embodiment, when the recording mode is narration mode, the digital camera 100 records audio data D0 (see Figure 4), which is generated in the earphone 20 based on the audio data from the internal microphone 262 for speech, to both channels Lch and Rch. The audio data D0 is generated by the beamforming unit 222 of the earphone 20 in a manner that emphasizes the sound coming from near the user's mouth, for example, as shown in Figure 10(B).

[0089] 2-2-3. Recording Operation The digital camera 100 records the audio acquired from the earphone 20 during video recording according to the recording mode described above. Returning to Figure 8, this recording operation will be explained.

[0090] First, the controller 135 determines whether the recording mode set in the digital camera 100 is (i) narration mode or (ii) stereo or front mode (S21). In the digital camera 100 of this embodiment, the recording mode is set by a user operation in which the user selects one mode from, for example, the three modes shown in Figure 10 using a setting menu displayed on the display monitor 130, etc., at the operation unit 150.

[0091] If the recording mode is narration mode (S21(i)), the controller 135 instructs the earphone 20 to switch the source of the input selector 224 to the audio data D0 from the internal microphone 262 (S22A). For example, as such an instruction, the controller 135 sends a control signal to the controller 235 of the earphone 20 via the communication modules 155 and 255 to change the source of the input selector 224.

[0092] The controller 135 receives the audio data D0 from the earphone 20, which has been picked up by the internal microphone 262 and output by the beamforming unit 222, via the communication module 155 (S23A).

[0093] On the other hand, if the recording mode is stereo mode or front mode (S21(ii)), the controller 135 instructs the earphone 20 to switch the source of the input selector 224 to audio data from the external microphone 261 (S22B). For example, as in step S22A, a control signal is sent to the controller 235 of the earphone 20.

[0094] The controller 135 receives audio data picked up by the external microphone 261 from the earphone 20 via the communication module 155 (S23B).

[0095] The controller 135 uses the beamforming unit 172 to perform beamforming on the audio data received from the external microphone 261, according to the set recording mode, either stereo mode or front mode (S24). For example, in step S24, the controller 135 calculates the phase difference to be set in the delay units P1 and P2 based on the distance D12 set in the beamforming unit 172. The phase difference δ is calculated, for example, by the following formula, where f represents the frequency of the audio signal represented by the audio data, and c represents the speed of sound. δ = D12 / 2πfc

[0096] After receiving audio data D0 from the sound pickup result by the internal microphone 262 (S22B), or after performing beamforming on the audio data from the external microphone 261 (S24), the controller 135 determines whether or not audio monitoring is enabled (S26). In the digital camera 100 of this embodiment, whether or not to play back the recording audio recorded during video shooting can be set by enabling or disabling monitoring. For example, this setting can be made by user operation using a setting menu or the like before the execution of the process in this flowchart.

[0097] If monitoring is enabled (YES in S26), the controller 135 switches the source of the output selector 174 in the audio processing unit 170 to the audio data for recording obtained in step S23A or step S24 (S27). This allows the audio data output from the audio processing unit 170 to be used to play back the audio recorded along with the shooting operation, for example, instead of the previously recorded audio already recorded in the video data.

[0098] The controller 135 transmits the audio data output from the audio processing unit 170 as audio for recording to the earphone 20 via the communication module 155 (S28).

[0099] After transmitting audio data to the earphone 20 (S28), the controller 135 determines whether or not video recording has been started by, for example, a user operation on the control unit 150 (S29).

[0100] If monitoring is not enabled (NO in S26), the controller 135 skips steps S27 and S28 and proceeds to step S29.

[0101] If the shooting operation has not started (NO in S29), the controller 135 repeats the processing from step S21 onwards. For example, when the video recording standby state receives a user operation to change the recording mode via the operation unit 150, the controller 135 acquires audio data for recording from the audio data picked up by the earphone 20 according to the changed recording mode (S21-S24). This makes it easier to acquire the audio intended by the user 3, even if the shooting scene changes.

[0102] Figure 11 illustrates an example of changing the recording mode during video recording. For example, as shown in Figure 11(A), the recording mode may be set to stereo mode to record ambient sounds in a landscape shooting scene, and then, as shown in Figure 11(B), the shooting scene may change to a selfie of user 3. In this case, the recording mode can be changed to narration mode, and the voice of user 3 can be recorded by the internal microphone 262 of the earphone 20.

[0103] Furthermore, if audio monitoring is enabled (YES in S26), user 3 can check the recording audio played from the speaker 280 of the earphone 20 and make adjustments such as changing the recording mode. This makes it easier to avoid unintended recordings due to user 3's mistakes, such as incorrectly setting the recording mode or incorrectly setting the digital camera 100 to pick up sound from the built-in microphone if the camera has one. Also, for example, monitoring before shooting can make it easier to obtain the intended audio by adjusting the recording mode if the recording differs from the intended audio due to noise in the environment surrounding user 3 and the volume of the audio to be recorded.

[0104] Returning to Figure 8, if the video recording operation has started (YES in S29), the controller 135 stores the audio data for recording in the buffer memory 125 or the like (S30).

[0105] The controller 135 determines, for example, whether the video recording operation has ended based on user operation at the operation unit 150 (S31).

[0106] If the shooting operation is not yet complete (NO in S31), the controller 135 repeats the processing from step S21 onwards, for example.

[0107] When the shooting operation is completed (YES in S31), the controller 135 stores the audio data held in the buffer memory 125 into video data (S32) and records the video data to the memory card 142 or the like. After that, the controller 135 terminates the processing of this flowchart.

[0108] According to the above process, when shooting video with the digital camera 100, audio data picked up by the external microphone 261 or internal microphone 262 of the earphone 20 can be acquired according to the set recording mode (S21~S23A, S23B). The audio data from the earphone 20 can then be recorded as audio in the video data shot by the digital camera 100 (S29~S32). Furthermore, the digital camera 100 of this embodiment performs beamforming on the audio data from the external microphone 261 based on the phase difference calculated from the distance D12 between the earphones 20L and 20R (S24). This makes it possible to accurately form the directivity of sound pickup, such as the stereo mode and front mode shown in Figure 10(B).

[0109] Furthermore, when audio monitoring is enabled during video recording (YES in S26), the recording audio is played back from the speaker 280 of the earphone 20 (S27-S29). This allows, for example, user 3 to confirm whether the audio they intended is being recorded.

[0110] The above describes an example in which, after video recording has started (YES in S29), if video recording has not finished (NO in S31), the processing from step S21 onwards is repeated. In this example, even during video recording, recording audio can be played back according to the monitoring settings (S26-S28), and actions such as changing the recording mode may be performed. Also, if video recording has not started (NO in S29), the controller 135 repeats the processing from steps S21-S29 until video recording starts (YES in S29), and can perform actions such as monitoring the recording audio and changing the recording mode.

[0111] 2-3. Playback Operation In this embodiment, the digital camera 100 records video data containing audio data through the recording operation during video shooting as described above, and then outputs the audio from the earphone 20 during playback of the recorded video data. This playback operation will be explained with reference to Figure 12.

[0112] Figure 12 is a flowchart illustrating the playback operation after video recording in the digital camera 100. The process shown in Figure 12 starts, for example, when video data has been recorded on the memory card 142. Each process in Figure 12 is executed by the controller 135.

[0113] First, the controller 135 switches the source of the output selector 174 in the audio processing unit 170 to recorded audio data, such as audio data stored in video data (S41). This allows the recorded audio to be played back, for example, instead of the audio being recorded, based on the audio data output from the audio processing unit 170.

[0114] Next, the controller 135 reads audio data from the recorded video data to play back the recorded audio (S42).

[0115] The controller 135 transmits the read and recorded audio data to the earphone 20 via the communication module 155 (S43). This allows the speaker 280 of the earphone 20 to play the audio data.

[0116] According to the above process, the digital camera 100 can not only play back the audio captured by the earphone 20 for monitoring during video recording, but can also output audio from the earphone 20 when playing back the video after recording.

[0117] 3. Summary As described above, in this embodiment, the digital camera 100, which is an example of an imaging device, comprises an image sensor 115, which is an example of an imaging unit, a communication module 155, which is an example of a communication unit, an audio processing unit 170, and a controller 135, which is an example of a control unit. The image sensor 115 captures an image of a subject and generates image data. The communication module 155 communicates data with an earphone 20, which is an example of a sound receiver. The audio processing unit 170 gives directionality to the sound picked up by the earphone 20. The controller 135 controls the sound pickup by the earphone 20. The earphone 20 is worn on the head of the user 3 and comprises a plurality of microphones 261, 262, each positioned at a predetermined location on the head where it is worn. The controller 135 controls the communication module 155 to receive audio data from the earphone 20 indicating the sound picked up by the image sensor 115 during imaging (S22A~S23B). The controller 135 controls the directivity of the received audio data using the audio processing unit 170, according to the arrangement of the multiple microphones 261, 262 on the head where the earphones 20 are attached, and outputs audio data that is recorded in association with the image data generated by the image sensor 115 (S24, S30, S32).

[0118] With the digital camera 100 described above, for example, when shooting video, sound picked up during image capture can be acquired from multiple microphones 261, 262 in the earphones 20 worn by user 3. Then, sound with directionality controlled according to the arrangement of microphones 261, 262 in the earphones 20 worn on user 3's head can be obtained. This makes it possible to pick up sound as user 3 hears it, control the directionality and make it easier to obtain sound that matches user 3's intentions using the earphones 20.

[0119] In this embodiment, the sound receiver consists of two earphones 20L and 20R, each having one or more microphones from a plurality of microphones 261 and 262. The communication module 155 communicates data with both of the two earphones 20L and 20R. For example, earphone 20L has an outer microphone 261L and an inner microphone 262L, and earphone 20R has an outer microphone 261R and an inner microphone 262R (see Figure 3). In this way, the digital camera 100 can receive audio data of the sound picked up by each microphone from each earphone 20L and 20R.

[0120] In this embodiment, the audio processing unit 170 forms a directivity pattern in the received audio data based on the phase difference between the audio signals output by the external microphones 261L and 261R, which are an example of two microphones in the earphone 20 (S24). This makes it possible to form different directivity patterns for the left and right as stereo sound, or to form a directivity pattern that emphasizes sound from a specific direction, based on the audio data from the external microphones 261L and 261R of the earphones 20L and 20R (see Figures 9 and 10).

[0121] In this embodiment, the multiple microphones in the earphone 20 include an external microphone 261, which is an example of a first microphone positioned to pick up sounds from the surrounding area of ​​the earphone 20 (see Figure 3). The controller 135 instructs the audio processing unit 170 to create directionality in the sound from the external microphone 261 (S24). As a result, for example, the external microphone 261, which is provided for noise cancellation in the earphone 20 and has characteristics such as a relatively wide bandwidth and high signal-to-noise ratio, can pick up sound and accurately obtain sound with controlled directionality.

[0122] In this embodiment, the digital camera 100 has multiple recording modes as an example of multiple sound pickup modes for capturing sound during image capture (see Figure 10). The multiple recording modes include a first mode (e.g., stereo mode and front mode) in which sound is captured by the external microphone 261. The first mode is not limited to the stereo mode and front mode shown in Figure 10, but may be a recording mode that captures sound with a different directivity than the example in Figure 10.

[0123] In this embodiment, the multiple microphones in the earphone 20 further include an internal microphone 262, which is an example of a second microphone positioned to pick up sound from the user 3. The multiple recording modes in the digital camera 100 further include a second mode (e.g., narration mode) in which sound is picked up by the internal microphone 262. With such recording modes, it is possible to accurately acquire sound, for example, from the user 3's speech.

[0124] In this embodiment, the digital camera 100 further includes an operation unit 150 that accepts user input to select one recording mode from a plurality of recording modes. The controller 135 controls the communication module 155 to receive audio data from the internal microphone 262 (S22A, S23A) when the selected recording mode is narration mode (S21(i)), and to receive audio data from the external microphone 261 (S22A~S23B) when the selected recording mode is stereo mode or front mode (S21(ii)). This allows the earphone 20 to acquire audio from different microphones depending on the recording mode selected by user input.

[0125] In this embodiment, the earphone 20 has a speaker 280. The controller 135 controls video recording, for example, by associating audio data output from the audio processing unit 170 as recording audio with image data. The controller 135 controls the communication module 155 to transmit the recording audio data to the earphone 20 before video recording, such as in standby mode, and during video recording (S27, S28~S31). This allows the recording audio data to be received by the earphone 20 and played back from the speaker 280 during standby before video recording, etc. In this way, the user 3 can monitor the recording audio and easily set the recording mode according to their intentions.

[0126] In this embodiment, the digital camera 100 further includes an operation unit 150 that accepts a user operation to select a recording mode, as an example of a user operation to select a type of directivity formed in the sound from the earphone 20. In response to the user operation on the operation unit 150 (S21), the digital camera 100 controls the audio processing unit 170 to form directivity in the audio data received from the earphone 20 (S24). This makes it possible to change the type of directivity in sound capture by selecting a recording mode according to the shooting scene etc. with the digital camera 100, making it easier to obtain sound that matches the user's intentions.

[0127] In this embodiment, the imaging system 1 comprises a digital camera 100 and an earphone 20 (see Figure 1). For example, as shown in Figure 2, the earphone 20 comprises a communication module 255, which is an example of a device communication unit that communicates data with the digital camera 100, and a controller 235, which is an example of a device control unit that transmits audio data indicating the sound picked up by one or more of the multiple microphones 261, 262 to the digital camera 100 via the communication module 255. With this system 1, the digital camera 100 can acquire the sound picked up by the multiple microphones 261, 262 in the earphone 20 through data communication with the earphone 20.

[0128] (Other embodiments) As described above, Embodiment 1 has been explained as an example of the technology disclosed in this application. However, the technology in this disclosure is not limited to this and can be applied to embodiments that have been modified, substituted, added, or omitted as appropriate. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiment. Therefore, other embodiments are described below as examples.

[0129] In the above embodiment 1, as shown in Figure 1, for example, an example was described in which the digital camera 100 in the imaging system 1 performs data communication with both earphones 20L and 20R. The digital camera 100 may perform data communication with only one of the earphones 20L or 20R. A modified example of this embodiment 1 will be explained using Figure 13.

[0130] Hereinafter, descriptions of the configuration and operation similar to that of the imaging system 1 according to Embodiment 1 will be omitted as appropriate, and the imaging system 1A according to this modified example will be described.

[0131] Figure 13 is a diagram illustrating an imaging system 1A according to a modified embodiment of Embodiment 1. In this system 1A, earphones 20L and 20R communicate with each other to send and receive audio data, etc. The digital camera 100 in this example communicates with earphone 20L to send and receive audio data, etc. System 1A may perform such transmission and reception according to an audio standard such as Bluetooth Classic Audio. With this system 1A, for example, in the same way as imaging system 1 described above, sound can be captured by the earphone 20 when shooting video with the digital camera 100.

[0132] As described above, in this modified example, the communication module 155 of the digital camera 100 communicates data with one of the two earphones 20L and 20R. In other words, in this embodiment, the communication module 155 communicates data with both or one of the two earphones 20L and 20R.

[0133] In the above embodiment 1, an example was described in which the distance D12 between earphones 20L and 20R is calculated in the device setting process (S4) by measuring the distance between the digital camera 100 and the earphones 20 and detecting their direction (S12 in Figure 6). In this embodiment, instead of step S12, the digital camera 100 may, for example, acquire a selfie image of user 3 wearing the earphones 20 and calculate the distance D12 based on that image. In this embodiment, it is not necessary to make a determination (S11) as to whether or not it is possible to measure the distance D12.

[0134] In the above embodiment 1, an example was described in which, in the device setting process (S4), if it is not possible to measure the distance between earphones 20L and 20R (NO in S11), the distance D12 is input by user operation on the operation unit 150, etc. (S13). In the digital camera 100 of this embodiment, even if distance measurement is possible (YES in S11), the distance D12 may be input by user operation in the same way as in step S13. Alternatively, the distance D12 may be input in the same way as in step S13 regardless of whether distance measurement is possible or not, and the judgment in step S11 may not be performed.

[0135] In each of the embodiments described above, an example was shown in which audio data from either the outer microphone 261 or the inner microphone 262 of the earphone 20 is transmitted to the digital camera 100 (S22A to S23B). In this embodiment, audio data from both the outer microphone 261 and the inner microphone 262 may be transmitted to the digital camera 100 simultaneously. For example, if the earphone 20 can transmit two or more channels of audio data simultaneously, the digital camera 100 may acquire audio data from both microphones 261 and 262 instead of following steps S22A to S23B.

[0136] In the above embodiment, for example, audio data acquired from both microphones 261 and 262 may be stored in a buffer memory 125 or the like. In this embodiment, the digital camera 100 may accept a user operation to change the recording mode after shooting a video, and may update the audio data stored in the video data based on the stored audio data according to the changed recording mode.

[0137] In each of the embodiments described above, an example was explained in which audio data for recording audio is transmitted to the earphone 20 according to the monitoring settings both in the standby state before the start of video recording and during the recording operation after the start of recording (S26-S28). In this embodiment, the controller 135 may perform the processing in steps S26-S28 only in either before video recording or during video recording. As described above, in this embodiment, the controller 135 controls the communication module 155 (an example of a communication unit) to transmit audio data to the earphone 20 (an example of a sound receiver) in at least one of the states before video recording or during video recording (S26-S28).

[0138] In each of the embodiments described above, an example was explained in which a phase difference is calculated to adjust the delay period in the audio data from the external microphones 261L and 261R, and beamforming is performed by the beamforming unit 172 of the audio processing unit 170 (S24). For example, in step S24, a transfer function that outputs audio data D1 to D3 from the audio data input to the beamforming unit 172 may be calculated based on the phase difference and the device information acquired in step S2, and applied to the input audio data. For example, a filter that corrects the sound according to the characteristics such as the sensitivity of the external microphone 261 may be pre-stored in the flash memory 145 for each model of earphone 20, and a transfer function including the corresponding filter may be calculated from the identification information of the earphone 20 in the device information.

[0139] In each of the embodiments described above, an example was explained in which the controller 135 of the digital camera 100 holds the audio data acquired from the earphone 20 in the buffer memory 125 and stores it in the video data recorded on the memory card 142 (S30, S32). In this embodiment, the controller 135 may output the audio data to various internal or external memories and / or recording media of the digital camera 100, or it may transmit it externally via the communication module 155.

[0140] In each of the embodiments described above, an example was described in which the earphones 20L and 20R each have one external microphone 261L and 261R. In this embodiment, the earphones 20L and 20R may each have two or more external microphones 261. The digital camera 100 of this embodiment may further have a recording mode in which sound is picked up to form surround sound or directivity in an arbitrary direction, as an example of a first mode in which sound is picked up by the external microphones 261.

[0141] In each of the embodiments described above, an earphone 20 having an external microphone 261 and an internal microphone 262 was described as an example of a sound pickup. The sound pickup in this embodiment may be, for example, two earphones, each having one microphone. For example, each earphone may not have an internal microphone 262, but may have only an external microphone 261.

[0142] In each of the embodiments described above, an example was shown in which the digital camera 100 performs beamforming on the audio data from the external microphones 261L and 261R (S24). In this embodiment, the digital camera 100 does not need to perform beamforming.

[0143] In each of the embodiments described above, earphones 20 were described as an example of a sound receiver. The sound receiver in this embodiment may be, for example, a headset having a microphone and a speaker and worn on the user's head 3.

[0144] In the embodiments described above, an earphone with a microphone was described as an example of a sound receiver. The sound receiver of this disclosure does not need to have a speaker, and may be an earphone-type microphone that is worn on the ear of user 3, for example. Even with such a microphone, it is possible to acquire sound that enhances the sense of presence as if user 3 were listening.

[0145] In each of the embodiments described above, an example was explained in which a sound receiver, such as earphones 20, is attached to a user 3, such as a photographer using a digital camera 100. In this embodiment, the sound receiver is not limited to the photographer, but may also be attached to, for example, the subject being photographed by the digital camera 100.

[0146] In each of the embodiments described above, a digital camera 100 that acquires sound using a sound receiver was described. In this embodiment, instead of the digital camera 100, various sound processing devices may acquire sound using a sound receiver. In this embodiment, such a sound processing device may receive sound data from a sound receiver such as an earphone 20, similar to the digital camera 100 in Embodiment 1, and control the directionality of the sound data. The sound processing device in this embodiment does not necessarily have an image capture function, and may be, for example, a voice recorder.

[0147] In other words, the audio processing device of this embodiment comprises a communication unit that communicates data with a sound receiver, an audio processing unit that forms directionality in the sound picked up by the sound receiver, and a control unit that controls the sound pickup by the sound receiver. The sound receiver is attached to the user's head and comprises a plurality of microphones that are each positioned at predetermined locations on the head where it is attached. The control unit controls the communication unit to receive audio data indicating the sound picked up from the sound receiver. The control unit controls the directionality of the received audio data by the audio processing unit according to the arrangement of the plurality of microphones on the head where the sound receiver is attached, and outputs the audio data to be recorded. This makes it easier to acquire audio that matches the intentions of the user using the audio processing device.

[0148] In each of the above embodiments, a digital camera 100 comprising an optical system 110 and a lens drive unit 112 was illustrated. The imaging device of this embodiment does not necessarily have to include an optical system 110 and a lens drive unit 112; for example, it may be a camera with interchangeable lenses.

[0149] In the embodiments described above, a digital camera was used as an example of an imaging device, but the invention is not limited to this. The imaging device of this disclosure may be any electronic device having an image capture function (e.g., a video camera, smartphone, tablet terminal, etc.).

[0150] (Summary of characteristics) The various aspects of this disclosure are listed below.

[0151] A first aspect of the present disclosure is an imaging device comprising: an imaging unit that captures an image of a subject and generates image data; a communication unit that communicates data with a sound receiver; an audio processing unit that forms directionality in the sound picked up by the sound receiver; and a control unit that controls sound pickup by the sound receiver. The sound receiver is attached to the user's head and comprises a plurality of microphones, each positioned at a predetermined location on the head to which it is attached. The control unit controls the communication unit to receive audio data from the sound receiver indicating the sound picked up by the imaging unit during imaging. The control unit controls the directionality of the received audio data by the audio processing unit according to the arrangement of the plurality of microphones on the head to which the sound receiver is attached, and outputs audio data to be recorded in association with the image data generated by the imaging unit.

[0152] According to the second embodiment, in the imaging device of the first embodiment, the sound receiver is two earphones, each having one or more microphones from a plurality of microphones, and the communication unit performs data communication with both or one of the two earphones.

[0153] According to the third embodiment, in the imaging device of the first or second embodiment, the audio processing unit forms directivity in the received audio data based on the phase difference between the audio signals output by the two microphones in the sound receiver.

[0154] According to the fourth embodiment, in the imaging device of any of the first to third embodiments, the plurality of microphones in the sound receiver include a first microphone arranged to pick up sound around the sound receiver, and the control unit causes the sound processing unit to form directionality in the sound from the first microphone.

[0155] According to the fifth aspect, the imaging device of the fourth aspect has a plurality of sound acquisition modes for acquiring sound during imaging. The plurality of sound acquisition modes include a first mode in which sound is acquired by a first microphone.

[0156] According to the sixth aspect, in the imaging device of the fifth aspect, the plurality of microphones in the sound receiver further include a second microphone arranged to pick up voice from the user, and the plurality of sound pickup modes further include a second mode in which voice is picked up by the second microphone.

[0157] According to the seventh aspect, the imaging device of the sixth aspect further includes an operation unit that accepts a user operation to select one sound pickup mode from a plurality of sound pickup modes. The control unit controls the communication unit to receive audio data from the first microphone when the selected sound pickup mode is the first mode, and to receive audio data from the second microphone when the selected sound pickup mode is the second mode.

[0158] According to the eighth aspect, in the imaging device of any of the first to seventh aspects, the sound receiver has a speaker. The control unit controls video recording, which records audio data output from the audio processing unit in association with image data. The control unit controls the communication unit to transmit the audio data to the sound receiver at least one of the following times: before video recording and during video recording.

[0159] According to the ninth aspect, the imaging device of the eighth aspect further includes an operation unit that accepts user input to select from a plurality of types of directivity formed in the sound from a sound receiver. The control unit controls the sound processing unit to form directivity in the sound data received from the sound receiver in response to the user input in the operation unit.

[0160] The tenth embodiment comprises an imaging device according to any of the first to ninth embodiments and a sound receiver. The sound receiver includes a device communication unit that communicates data with the imaging device and a device control unit that transmits audio data indicating sound picked up by one or more of the multiple microphones to the imaging device via the device communication unit.

[0161] As described above, embodiments have been explained as examples of the technology in this disclosure. For this purpose, accompanying drawings and a detailed description have been provided.

[0162] Therefore, the components described in the attached drawings and detailed descriptions may include not only components essential for solving the problem, but also components that are not essential for solving the problem, provided that they illustrate the technology described above. For this reason, the mere presence of these non-essential components in the attached drawings and detailed descriptions should not be immediately assumed to mean that they are essential.

[0163] Furthermore, since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents. [Industrial applicability]

[0164] The concept of this disclosure is applicable to various electronic devices that acquire sound during image capture, such as video recording (e.g., imaging devices such as digital cameras, video cameras, and camcorders, as well as smartphones, tablet devices, etc.). [Explanation of Symbols]

[0165] 1.1A Imaging System 100 Digital Cameras 115 Image Sensor 120 Image Processing Unit 125 buffer memory 130 Display Monitor 135 Controller 145 Flash Memory 150 Operation section 155 Communication Module 170 Audio Processing Unit 20 (20L, 20R) Earphones 261 (261L, 261R) External microphone 262 (262L, 262R) Inner Microphone 280 (280L, 280R) Speakers

Claims

1. An imaging unit that captures an image of the subject and generates image data, A sound collector and a communication unit that performs data communication, A sound processing unit that forms directionality in the sound picked up by the sound receiver, The system includes a control unit that controls sound collection by the aforementioned sound collector, The sound pickup is attached to the user's head and comprises a plurality of microphones, each positioned at a predetermined location on the head where it is attached. The control unit, The communication unit is controlled to receive audio data from the sound receiver indicating the sound picked up by the imaging unit during imaging, Depending on the arrangement of the multiple microphones on the head to which the sound receiver is attached, the audio processing unit controls the directivity of the received audio data and outputs audio data that is recorded in association with the image data generated by the imaging unit. Imaging device.

2. The sound pickup consists of two earphones, each having one or more microphones from the plurality of microphones. The communication unit communicates data with both or one of the two earphones. The imaging apparatus according to claim 1.

3. The audio processing unit forms directivity in the received audio data based on the phase difference between the audio signals output by the two microphones in the sound receiver. The imaging apparatus according to claim 1.

4. The plurality of microphones in the sound receiver include a first microphone arranged to pick up sound from the surrounding area of ​​the sound receiver. The control unit instructs the audio processing unit to give directionality to the sound from the first microphone. The imaging apparatus according to claim 1.

5. The imaging device has multiple sound acquisition modes for acquiring sound during imaging, The plurality of sound pickup modes include a first mode in which sound is picked up by the first microphone. The imaging apparatus according to claim 4.

6. The plurality of microphones in the sound pickup device further include a second microphone arranged to pick up sound from the user, The plurality of sound pickup modes further include a second mode in which sound is picked up by the second microphone. The imaging apparatus according to claim 5.

7. The system further includes an operating unit that accepts user input to select one sound pickup mode from the aforementioned plurality of sound pickup modes. The control unit, When the selected sound pickup mode is the first mode, audio data is received from the first microphone. When the selected sound pickup mode is the second mode, the communication unit is controlled to receive audio data from the second microphone. The imaging device according to claim 6.

8. The sound receiver has a speaker, The control unit controls video recording, which records the audio data output from the audio processing unit in association with the image data. The control unit controls the communication unit to transmit the audio data to the sound receiver at least one of the following times: before video recording and during video recording. The imaging apparatus according to claim 1.

9. The imaging device further includes an operating unit that accepts user input to select from a plurality of types of directivity formed in the sound from the sound receiver, The control unit, In response to user operation on the control unit, the audio processing unit is controlled to form directionality in the audio data received from the sound receiver. The imaging apparatus according to claim 8.

10. An imaging device according to any one of claims 1 to 9, The aforementioned sound receiver, Equipped with, The aforementioned sound receiver is The aforementioned imaging device and the equipment communication unit that performs data communication with the imaging device, The system comprises a device communication unit and a device control unit that transmits audio data indicating sound picked up by one or more of the plurality of microphones to the imaging device. Imaging system.

Images