Imaging apparatus

Abstract

To provide an imaging apparatus that takes an image while acquiring voice, the apparatus making it easier to collect voice of an object according to an intention of a user.SOLUTION: The imaging apparatus includes: an imaging unit for imaging an object and generating image data; a voice acquisition unit for acquiring a voice signal showing voice collected during imaging of the imaging unit; a setting unit for setting an own device in an auto-mode as an operation mode for automatically changing the directivity of the voice acquisition unit in response to an instruction of a user; and a control unit for controlling a voice collecting area for collecting voice from an object in the voice signal. The control unit controls the voice collecting area so that the area will include the object, by changing the directivity of the voice acquisition unit in association with the imaging state of the own device when the setting unit sets the own device in the auto-mode.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an imaging device that captures an image while acquiring sound. [Background technology]

[0002] Patent Document 1 discloses a video camera with a face detection function. The video camera of Patent Document 1 changes the directional angle of the microphone according to the zoom ratio and the size of the face of the person in the captured image. In this way, the video camera controls the directional angle of the microphone in association with the distance between the video camera and the subject image, thereby achieving control to change the directional angle of the microphone so as to more reliably capture the voice of the subject while aligning the image and the sound. In this case, the video camera detects the position and size of the face of the person (subject), displays the detected face with a frame (face detection frame), and uses information on the size of the face detection frame (face size). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-283706 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an imaging device that captures images while acquiring sound, and that can easily pick up the sound of a subject in accordance with a user's intention. [Means for solving the problem]

[0005] In the present disclosure, an imaging device includes an imaging unit that images a subject and generates image data, an audio acquisition unit that acquires an audio signal indicating audio picked up during imaging by the imaging unit, a setting unit that sets the device to an auto mode, which is an operating mode in which the directivity of the audio acquisition unit is automatically changed in response to a user's instruction, and a control unit that controls a sound collection area that picks up audio from the subject in the audio signal, and when the setting unit is set to the auto mode, the control unit controls the sound collection area to include the subject by changing the directivity of the audio acquisition unit in conjunction with the shooting state of the device. Effect of the Invention

[0006] According to the imaging device according to the present disclosure, in an imaging device that captures images while acquiring sound, it is possible to easily pick up the sound of a subject in accordance with the user's intention. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 shows a configuration of a digital camera 100 according to a first embodiment of the present disclosure. [Diagram 2] FIG. 1 is a diagram illustrating the rear view of a digital camera 100. [Diagram 3] FIG. 1 is a diagram illustrating a state of the digital camera 100 when taking a selfie; [Figure 4] FIG. 1 is a diagram illustrating a state of the digital camera 100 when taking a vertical shot; [Diagram 5] FIG. 1 is a diagram illustrating a configuration example of a beam forming section 172 in a digital camera 100. [Figure 6] FIG. 1 is a diagram illustrating an example of a sound collection area in a digital camera 100. [Figure 7] FIG. 1 shows an example of a setting menu display in the digital camera 100. [Figure 8] FIG. 13 is a diagram illustrating an example of a further sound collection area in the digital camera 100. [Figure 9] FIG. 1 is a diagram for explaining an overview of the operation of the auto mode of the digital camera 100; [Figure 10] A flowchart illustrating the operation of the digital camera 100 in the focus mode according to the first embodiment. [Figure 11] FIG. 1 is a diagram for explaining an overview of the operation of the focus mode of the digital camera 100; [Figure 12] 10 is a flowchart illustrating a process of selecting sound collection targets (S3 in FIG. 10) of the digital camera 100 according to the first embodiment. [Figure 13] FIG. 1 is a diagram for explaining a process for selecting sound collection targets in the digital camera 100. [Figure 14] 10 is a flowchart illustrating a process for determining a sound collection area in the digital camera 100 (S4 in FIG. 10). [Figure 15] FIG. 1 is a diagram for explaining a process for determining a sound collection area in the digital camera 100; [Figure 16] A flowchart illustrating sound collection control (S5 in FIG. 10) using face recognition in the digital camera 100. [Figure 17] FIG. 13 is a diagram for explaining management information obtained by a process for determining a sound collection area; [Figure 18] FIG. 13 is a diagram illustrating a relationship for calculating gain from the horizontal angle of view and the focal distance in the digital camera 100; [Figure 19] A flowchart illustrating sound collection control (S6 in FIG. 10) without using face recognition in the digital camera 100. [Figure 20] A flowchart illustrating the operation of the digital camera 100 in the auto mode according to the first embodiment. [Figure 21] FIG. 1 is a diagram for explaining sound collection control during horizontal and vertical shooting in auto mode according to the first embodiment; [Figure 22] FIG. 13 shows a display example of a digital camera 100 according to a second embodiment. [Figure 23] FIG. 11 is a diagram for explaining manual operations in the digital camera 100 according to the second embodiment. [Figure 24] 11 is a flowchart illustrating an example of an operation during manual operation of the digital camera 100 according to the second embodiment. [Diagram 25] FIG. 13 is a diagram illustrating an example of the arrangement of a microphone 161A in a digital camera 100A according to a modified example. [Figure 26]FIG. 13 is a diagram for explaining sound collection control during vertical shooting in auto mode according to a modified example; [Figure 27] FIG. 1 is a diagram for explaining an example of the operation of sound collection control linked to face recognition in the digital camera 100. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the inventor(s) provide the attached drawings and the following explanation so that those skilled in the art can fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0009] (Embodiment 1) In embodiment 1, as an example of an imaging device according to the present disclosure, a digital camera is described that detects a subject based on image recognition technology, controls the sound collection area according to the size of the detected subject, and controls the sound collection gain to emphasize the sound to be collected.

[0010] [1-1. Structure]

[0011] FIG. 1 is a diagram showing the configuration of a digital camera 100 according to this embodiment. The digital camera 100 of this embodiment includes an image sensor 115, an image processing engine 120, a display monitor 130, and a controller 135. The digital camera 100 also includes a buffer memory 125, a card slot 140, a flash memory 145, an operation unit 150, and a communication module 160. The digital camera 100 also includes a microphone 161, an analog / digital (A / D) converter 165 for the microphone, and an audio processing engine 170. The digital camera 100 also includes, for example, an optical system 110 and a lens driving unit 112. The digital camera 100 also includes, for example, a magnetic sensor 132 and an acceleration sensor 137.

[0012] Fig. 2 illustrates the rear view of digital camera 100. Fig. 2 illustrates three axial directions X, Y, and Z of digital camera 100, as well as the direction of gravity G. The X, Y, and Z axes correspond to the horizontal and vertical angle of view directions of digital camera 100, and the optical axis direction of the lens in optical system 110, respectively. In the example of Fig. 2, the Y-axis direction of digital camera 100 is oriented along the direction of gravity G, i.e., horizontally.

[0013] The digital camera 100 of this embodiment can be used by the user to take a selfie or to take a vertical shot by using the digital camera 100 in a vertical orientation. Fig. 3 shows an example of the state of the digital camera 100 when taking a selfie. Fig. 4 shows an example of the state of the digital camera 100 when taking a vertical shot.

[0014] Returning to FIG. 1, the optical system 110 includes a focus lens, a zoom lens, an optical image stabilization lens (OIS), an aperture, a shutter, etc. The focus lens is a lens for changing the focus state of a subject image formed on the image sensor 115. The zoom lens is a lens for changing the magnification of a subject image formed by the optical system. The focus lens, etc. are each composed of one or more lenses.

[0015] Lens driving unit 112 drives a focus lens and the like in optical system 110. Lens driving unit 112 includes a motor, and moves the focus lens along the optical axis of optical system 110 under the control of controller 135. The configuration for driving the focus lens in lens driving unit 112 can be realized by a DC motor, a stepping motor, a servo motor, an ultrasonic motor, or the like.

[0016] The image sensor 115 captures an image of a subject formed via the optical system 110 to generate imaging data. The imaging data constitutes image data indicative of an image captured by the image sensor 115. The image sensor 115 generates new frame image data at a predetermined frame rate (e.g., 30 frames / second). The timing of generating imaging data and electronic shutter operation in the image sensor 115 are controlled by the controller 135. The image sensor 115 can be any of various image sensors, such as a CMOS image sensor, a CCD image sensor, or an NMOS image sensor.

[0017] The image sensor 115 performs operations for capturing moving images, still images, through images, etc. Through images are mainly moving images, and are displayed on the display monitor 130 so that the user can determine a composition for capturing a still image, for example. The through image, the moving image, and the still image are each an example of a captured image in this embodiment. The image sensor 115 is an example of an imaging section in this embodiment.

[0018] The image processing engine 120 performs various processes on the imaging data output from the image sensor 115 to generate image data, and performs various processes on the image data to generate an image to be displayed on the display monitor 130. The various processes include, but are not limited to, white balance correction, gamma correction, YC conversion processing, electronic zoom processing, compression processing, and expansion processing. The image processing engine 120 may be configured with a hardwired electronic circuit, or may be configured with a microcomputer, processor, etc. using a program.

[0019] In this embodiment, the image processing engine 120 includes a face recognition unit 122 that realizes a function of detecting a subject such as a human face by image recognition of a captured image. The face recognition unit 122 performs face detection, for example, by rule-based image recognition processing, and outputs detection information. Face detection may be performed by various image recognition algorithms. The detection information includes position information corresponding to the detection result of the subject. The position information is defined, for example, by a horizontal position and a vertical position on the image Im to be processed, and indicates, for example, a rectangular area surrounding a human face as a detected subject (see FIG. 11).

[0020] The display monitor 130 is an example of a display unit that displays various information. For example, the display monitor 130 displays an image (through image) represented by image data captured by the image sensor 115 and processed by the image processing engine 120. The display monitor 130 also displays a menu screen or the like that allows the user to perform various settings on the digital camera 100. The display monitor 130 can be configured, for example, by a liquid crystal display device or an organic EL device.

[0021] The digital camera 100 of this embodiment is configured to be movable so that the position of the display monitor 130 can be changed, for example, as shown in Figures 2 and 3. In the example of Figure 2, the display monitor 130 is in a position where the display surface faces the rear side (-Z side) of the digital camera 100. Such a position of the display monitor 130 is hereinafter referred to as the "normal position." In the example of Figure 3, the display monitor 130 is in a position where the display surface faces the front side (+Z side) of the digital camera 100, i.e., the subject side. Such a position of the display monitor 130 is hereinafter referred to as the "selfie position."

[0022] The magnetic sensor 132 is an example of a detection unit that detects whether the display monitor 130 is in a normal position or a selfie position. The magnetic sensor 132 outputs a detection signal indicating the detection result of the position of the display monitor 130 to the controller 135, for example.

[0023] For example, a variable angle type or a tilt type can be adopted as the movable display monitor 130. For example, a hinge 131 is provided to rotatably connect the display monitor 130 to the body of the digital camera 100. The magnetic sensor 132 is provided, for example, inside the hinge 131, and is composed of a switch or the like having two states corresponding to FIGS.

[0024] Acceleration sensor 137 detects acceleration in one or more of three axial directions, for example, X, Y, and Z, and outputs a detection signal to controller 135. Acceleration sensor 137 is an example of an orientation detection unit that detects whether the orientation of digital camera 100 is landscape as illustrated in Fig. 2 or portrait as illustrated in Fig. 4 based on the detected state of gravitational acceleration.

[0025] Operation unit 150 is a general term for hard keys such as operation buttons and operation levers provided on the exterior of digital camera 100, and accepts operations by the user. Operation unit 150 includes, for example, a release button, a mode dial, a touch panel, a cursor button, and a joystick. When operation unit 150 accepts an operation by the user, it transmits an operation signal corresponding to the user operation to controller 135. Operation unit 150 includes, for example, as shown in FIG. 2, a release button 151, a selection button 152, a decision button 153, function buttons 154, a touch panel 155, and the like.

[0026] The controller 135 centralizes and controls the overall operation of the digital camera 100. The controller 135 includes a CPU and the like, and the CPU executes a program (software) to realize a predetermined function. Instead of a CPU, the controller 135 may include a processor configured with a dedicated electronic circuit designed to realize a predetermined function. That is, the controller 135 can be realized by various processors such as a CPU, MPU, GPU, DSU, FPGA, and ASIC. The controller 135 may be configured with one or more processors. Furthermore, the controller 135 may be configured as a single semiconductor chip together with the image processing engine 120 and the like.

[0027] The buffer memory 125 is a recording medium that functions as a work memory for the image processing engine 120 and the controller 135. The buffer memory 125 is realized by a dynamic random access memory (DRAM) or the like. The flash memory 145 is a non-volatile recording medium. Although not shown, the controller 135 may have various types of internal memory, for example, a built-in ROM. The ROM stores various programs executed by the controller 135. The controller 135 may also have a built-in RAM that functions as a work area for the CPU.

[0028] The card slot 140 is a means for inserting a removable memory card 142. The card slot 140 is capable of electrically and mechanically connecting the memory card 142. The memory card 142 is an external memory equipped with an internal recording element such as a flash memory. The memory card 142 can store data such as image data generated by the image processing engine 120.

[0029] The communication module 160 is a communication module (circuit) that performs communication in compliance with the communication standard IEEE802.11 or the Wi-Fi standard. The digital camera 100 can communicate with other devices via the communication module 160. The digital camera 100 may communicate with other devices directly via the communication module 160, or may communicate with other devices via an access point. The communication module 160 may be connectable to a communication network such as the Internet.

[0030] The microphone 161 is an example of a sound collection unit that collects sound. The microphone 161 converts the collected sound into an analog signal, which is an electrical signal, and outputs the analog signal. The microphone 161 of this embodiment includes three microphone elements 161L, 161C, and 161R. The microphone 161 may be composed of two or four or more microphone elements.

[0031] The microphone A / D converter 165 converts an analog signal from the microphone 161 into audio data of a digital signal. The microphone A / D converter 165 is an example of an audio acquisition unit in this embodiment. The microphone 161 may include a microphone element that is external to the digital camera 100. In this case, the digital camera 100 includes an interface circuit for the external microphone 161 as the audio acquisition unit.

[0032] The voice processing engine 170 receives voice data output from a voice acquisition unit such as the A / D converter 165 for a microphone, and performs various voice processes on the received voice data. The voice processing engine 170 is an example of a voice processing unit in this embodiment.

[0033] The voice processing engine 170 of this embodiment includes a beam forming unit 172 and a gain adjustment unit 174, as shown in FIG. 1 for example. The beam forming unit 172 realizes a function of controlling the directivity of the voice. The beam forming unit 172 will be described in detail later. The gain adjustment unit 174 performs a multiplication process of multiplying the input voice data by a sound collection gain set by the controller 135, for example, to amplify the voice. The gain adjustment unit 174 may perform a process of suppressing the voice by multiplying the input voice data by a negative gain. The sound collection gain adjustment unit 14 may further have a function of changing the frequency characteristics and stereo characteristics of the input voice data. The setting of the sound collection gain will be described in detail later.

[0034] [1-1-1. Beam forming section] The beam forming section 172 in this embodiment will be described in detail below.

[0035] The beam forming section 172 performs beam forming to control the directivity of the sound picked up by the microphone 161. An example of the configuration of the beam forming section 172 in this embodiment is shown in FIG.

[0036] 5, the beam forming unit 172 includes, for example, filters D1 to D3 and an adder 173, adjusts the delay period of the sound collected by each of the microphone elements 161L, 161C, and 161R, and outputs the weighted sum. The beam forming unit 172 controls the direction and range of the sound collection directivity of the microphone 161, and can set the physical range in which the microphone 161 collects sound.

[0037] In the figure, the beam forming unit 172 outputs one channel using one adder 173, but it may have two or more adders and output differently on each channel, such as a stereo output. Also, a subtractor may be used in addition to the adder 173 to form a directivity having a blind spot in a specific direction, which is a direction with particularly low sensitivity, or adaptive beamforming may be performed to change processing according to the environment. Different processing may be applied depending on the frequency band of the audio signal.

[0038] 5 shows an example in which the microphone elements 161L, 161C, and 161R are linearly arranged, but the arrangement of the microphone elements is not limited to this. For example, even if they are arranged in a triangular shape, the delay period and weight of the filters D1 to D3 can be appropriately adjusted to control the sound collection directivity of the microphone 161. The beam forming unit 172 may apply a known method to control the sound collection directivity. For example, a sound processing technology such as OZO Audio may be used to perform a process of forming directivity and a process of suppressing noise in the sound.

[0039] The sound collection area of ​​the digital camera 100 that can be set by the beam forming section 172 as described above will be described.

[0040] [1-1-2. About the recording area] Fig. 6 shows an example of a sound collection area defined in digital camera 100. Fig. 6 illustrates the sound collection area as a sector of a circle centered on digital camera 100. In digital camera 100 of this embodiment, the horizontal angle of view direction coincides with the direction in which microphone elements 161R, 161C, and 161R are arranged.

[0041] FIG. 6(A) shows a "front center sound collection area" 41 that faces the sound collection area in front of the digital camera 100 (i.e., the shooting direction) in an angle range 401 (e.g., 70°). FIG. 6(B) shows a "left half sound collection area" 42 that faces the sound collection area to the left of the digital camera 100 in the angle range 401. FIG. 6(C) shows a "right half sound collection area" 43 that faces the sound collection area to the right of the digital camera 100 in the angle range 401. FIG. 6(D) shows a "front sound collection area" 44 that faces the sound collection area in front of the digital camera 100 in an angle range 402 (e.g., 160°) larger than the angle range 401. These sound collection areas are examples of a plurality of predetermined areas in this embodiment, and the angle ranges 401 and 402 are examples of the first angle range and the second angle range.

[0042] When the subject is located in the center of the captured image, the digital camera 100 of this embodiment uses the front center sound collection area 41 in Fig. 6(A). When the subject is located in the left half of the captured image, the left half sound collection area 42 in Fig. 6(B) is used, and when the subject is located in the right half of the captured image, the right half sound collection area 43 in Fig. 6(C) is used. When the subject is located over the entire captured image, the front sound collection area 44 in Fig. 6(D) is mainly used.

[0043] In the example of Fig. 11(B), subjects R1 and R3 to be collected are located in the center of the captured image, so a front center sound collection area 41 is used. In the example of Fig. 11(C), subjects R1 and R2 to be collected are located in the left half of the captured image, so a left half sound collection area 42 is used.

[0044] In the sound collection section of an imaging device such as the microphone 161 of the digital camera 100, the number and arrangement of microphone elements are restricted by factors such as the space available for mounting the elements. For example, in a shooting scene where a user wants to record the sounds of multiple subjects, the sound collection directivity may not be narrowed sufficiently due to restrictions on the number of microphone elements. Even in such a case, the digital camera 100 of this embodiment can provide a sound collection area according to the user's intention by defining a sound collection area assuming the shooting scene of the user and determining the sound collection area using face recognition.

[0045] [1-1-3. Microphone settings] The settings relating to the sound collection area of ​​microphone 161 in digital camera 100 will be described with reference to FIGS.

[0046] Fig. 7 shows an example of a setting menu display in digital camera 100. Digital camera 100 of this embodiment has, for example, as shown in Fig. 7, "auto", "surround", "front", "focus", and "narration" as operation modes (i.e., sound collection modes) for controlling the sound collection area of ​​microphone 161.

[0047] The focus mode is an operation mode that automatically changes the directivity of the microphone 161 and adjusts the sound collection area in conjunction with face recognition and the angle of view by the digital camera 100. For example, the focus mode can be realized by switching between the various sound collection areas 41 to 44 described above. By intentionally roughly realizing the focus mode using the four sound collection areas 41 to 44, it is possible to avoid a situation in which the sound collection directivity frequently changes due to slight movements of the subject, and to reduce the annoyance to the user's hearing. Further sound collection areas in the digital camera 100 are shown in FIG. 8.

[0048] 8A illustrates an example of a sound collection area 45 in surround mode. The surround mode is an operating mode for collecting sounds over a wide range across the left, right, front and rear of the digital camera 100. The sound collection area 45 in surround mode has an angular range of, for example, 360° all around the XZ plane.

[0049] 8(B) illustrates the sound collection area 46 in the front mode. The front mode is an operating mode for collecting sounds in front of the digital camera 100. The sound collection area 46 in the front mode faces the +Z side from the digital camera 100, and has an angle range equal to or larger than that of the front sound collection area 44 described above, for example.

[0050] 8(C) illustrates sound collection area 47 in narration mode. Narration mode is an operation mode for collecting sounds behind digital camera 100. Sound collection area 47 in narration mode is formed toward the -Z side from digital camera 100. When narration mode is set and it is detected that display monitor 130 is in the selfie position, digital camera 100 may execute the operation of focus mode.

[0051] The auto mode is an operation mode that automatically changes the directivity of the microphone 161 and adjusts the sound collection area in conjunction with the shooting state of the digital camera 100. The shooting state of the digital camera 100 includes, for example, whether or not the image is a self-portrait and whether the image is taken vertically or horizontally, in addition to face recognition and the like that are taken into account in the focus mode.

[0052] The microphone settings for setting the various sound collection modes as described above are provided, for example, as one of the video menus in the setting menu of the digital camera 100. The user can select a desired sound collection mode from the setting menu by touching the touch panel 155 or pressing the various buttons 152, 153. The microphone settings may be assigned in advance to the function buttons 154, etc. A setting for a specific sound collection mode such as an auto mode may be assigned to the function buttons 154, etc.

[0053] [1-2. Operation] The following describes the operation of digital camera 100 configured as above. The following describes the operation of digital camera 100 when shooting moving images.

[0054] Digital camera 100 sequentially captures the subject image formed via optical system 110 with image sensor 115 to generate imaging data. Image processing engine 120 performs various processes on the imaging data generated by image sensor 115 to generate image data and records it in buffer memory 125. In addition, face recognition unit 122 of image processing engine 120 detects the subject area based on the image indicated by the imaging data, and outputs detection information to controller 135, for example.

[0055] The digital camera 100 of this embodiment has a face recognition mode, which is an operating mode in which face detection is performed by image recognition processing in a captured image input to the face recognition unit 122, and a subject to be targeted for autofocus (AF) control is identified based on the detection information.

[0056] Simultaneously with the above imaging operations, the digital camera 100 collects sound with the microphone 161. Sound data resulting from the sound collection is output from the microphone A / D converter 165 and processed by the sound processing engine 170. The sound processing engine 170 records the processed sound data Aout in the buffer memory 125.

[0057] The controller 135 records moving images in the memory card 142 by synchronizing image data received from the image processing engine 120 and audio data received from the audio processing engine 170 via the buffer memory 125. The controller 135 also sequentially causes the display monitor 130 to display a through image. The user can check the composition of the shot at any time by viewing the through image on the display monitor 130. The operation of shooting moving images is started / ended in response to the user's operation on the operation unit 150.

[0058] In the above-described video shooting with the digital camera 100, there are various situations in which the user may want to record sound in various shooting conditions. For example, there may be a case where the user focuses on a group of subjects having a conversation among themselves, such as the photographer and his / her companion. In this case, there may be a need to clearly record the vocalizations of the group of subjects.

[0059] In the digital camera 100 of this embodiment, for example, as an operation of the above-mentioned focus mode, a subject is detected by detection information from the face recognition unit 122 in the image processing engine 120, and when the subject to be AF-targeted is determined, a process is executed in the sound processing engine 170 to emphasize the sound picked up for the subject and for the subjects around the subject in the shooting space. In this way, the face recognition of the image processing engine 120 is linked with the sound emphasis, etc. of the sound processing engine 170 to accurately realize the pick-up of sound with emphasis on the sound of a group of subjects having a conversation as described above.

[0060] Furthermore, in the auto mode operation of the digital camera 100 of this embodiment, in addition to the focus mode operation described above, appropriate sound collection control is also realized in accordance with various shooting conditions. An overview of the auto mode operation will be described with reference to FIG.

[0061] 9 shows an example of the correspondence between the auto mode and various sound collection modes of the digital camera 100. When face recognition is performed in a landscape shooting state, for example, the digital camera 100 in the auto mode performs the same operation as in the focus mode.

[0062] On the other hand, when face recognition is not performed, for example when not taking a selfie (see FIG. 2), digital camera 100 operates in the same way as in surround mode, that is, it employs surround mode sound pickup area 45. When face recognition is not performed and a selfie is being taken (see FIG. 3), digital camera 100 operates in the same way as in front mode.

[0063] In addition, in the case of vertical shooting, the operation when face recognition is not performed is the same as in the case of horizontal shooting described above. On the other hand, when face recognition is performed in vertical shooting, the digital camera 100 of this embodiment performs the same operation as in the front mode instead of the focus mode.

[0064] According to the above-described auto mode operation, by combining various sound collection mode operations according to various shooting conditions, as shown in FIG. 9, it is possible to easily realize appropriate sound collection control in each shooting condition.

[0065] [1-2-1. Focus mode operation] An overview of the operation of the focus mode of the digital camera 100 according to this embodiment will be described with reference to FIGS.

[0066] Fig. 10 is a flowchart illustrating the operation of the focus mode of the digital camera 100 according to this embodiment. Each process shown in the flowchart of Fig. 10 is repeatedly executed at a predetermined cycle, for example, when the digital camera 100 is set to the focus mode. The predetermined cycle is, for example, the frame cycle of a moving image. Fig. 11 is a diagram for explaining an overview of the operation of the focus mode of the digital camera 100 according to this embodiment.

[0067] The controller 135 identifies an AF target based on the detection information by the face recognition unit 122, and executes AF control (S1). The AF target indicates an area on an image of a subject that is the target of AF control. FIG. 11(A) illustrates a captured image Im including face areas R1, R2, and R3 that indicate areas where a subject is detected in the detection information from the face recognition unit 122. The face areas R1, R2, and R3 are examples of subject areas in this embodiment. For example, the face area R1 is identified as the face area 60 of the AF target.

[0068] Next, the controller 135 determines whether or not a face area specified as an AF target is present (S2). Specifically, the controller 135 determines whether or not a face area has been detected and the AF target is the face area.

[0069] If there is a face area 60 that is an AF target (YES in S2), the controller 135 executes a process of selecting a sound collection target of the microphone 161 from the subjects in the detection information (S3). The sound collection target is a subject whose sound is to be emphasized and collected by the microphone 161. The face area R1 (60) identified as the AF target becomes the sound collection target. FIG. 11(B) shows an example in which the face areas R1 and R3 are determined to be sound collection targets based on the detection information shown in FIG. 11(A), while the face area R2 is not determined to be a sound collection target.

[0070] In the digital camera 100 of this embodiment, in the process of selecting a sound collection target (S3), in addition to the face area R1 (60) that is the AF target, a face R3 that shows a face size similar to that of the face area R1 in the captured image Im is determined as a further sound collection target. On the other hand, a face area R2 that is a different size from the face area R1 is excluded from the sound collection target. This allows, for example, a group of subjects having a conversation among themselves to be set as a sound collection target, reflecting that the person 21 and the person 23 are at approximately the same distance from the digital camera 100 (i.e., the difference in distance in the Z axis direction is small) and the person 22 is at a different distance. The process of selecting a sound collection target (S3) will be described in detail later.

[0071] Next, the controller 135 performs a process of determining a sound collection area based on the determined sound collection targets (S4). The sound collection area determination process (S4) determines a sound collection area that includes all the determined sound collection targets. In the example of FIG. 11(B), the sound collection area is determined to be the front center sound collection area 41 (FIG. 6(A)) so as to include the face areas R1 and R3 of the sound collection targets. The sound collection area determination process (S4) will be described in detail later.

[0072] Next, the controller 135 controls the sound collection using face recognition based on the determined sound collection target and sound collection area (S5). The sound collection control using face recognition (S5) is performed by setting sound collection parameters including the sound collection target, sound collection area, and sound collection gain determined by the controller 135 in the sound processing engine 170. The sound processing engine 170 realizes the sound collection directivity and sound collection gain according to the sound collection parameters.

[0073] On the other hand, if there is no face area 60 to be an AF target (NO in S2), for example, if no face area is detected during operation in the face recognition mode, the controller 135 performs sound collection control (S6) without using face recognition. Details of sound collection control (S5, S6) with or without using face recognition will be described later.

[0074] After executing the sound collection control in step S5 or S6, the controller 135 repeats the processes in step S1 and thereafter.

[0075] According to the above process, the digital camera 100 of this embodiment selects sound collection targets from the subjects detected by face recognition, determines a sound collection area that includes all of the sound collection targets, and performs sound collection control using face recognition. This makes it possible to collect sound with emphasis on the voice of a group of subjects having a conversation among themselves, for example.

[0076] In the AF control by face recognition (S1), the AF target can be identified based on the detection information, for example, by displaying a frame indicating the face area on the through image displayed on the display monitor 130, and receiving an operation by the user via the operation unit 150 to select the frame display.

[0077] FIG. 11C shows an example of a captured image Im in which people 21 to 23 are located at positions different from those in FIG. 11A and FIG. 11B. As in the example of FIG. 11B, the digital camera 100 first specifies, for example, face area R1 as face area 60 to be AF-targeted (S1) and determines it as a sound collection target. In the example of FIG. 11C, the selection process of the sound collection target (S3) determines face area R2, which has a face size similar to that of face area R1 on the captured image Im, as a sound collection target, and excludes face area R3 from the sound collection target. The process of determining the sound collection area (S4) determines the left half sound collection area 42 (FIG. 6B) including the face areas R1 and R2 determined as sound collection targets as the sound collection area. The sound collection control using face recognition (S5) is performed by setting sound collection parameters so as to control the directivity of the left half sound collection area 42 to clearly collect the voices of people 21 and 22.

[0078] [1-2-2. Sound recording selection process] The details of the selection process of the sound collection target in step S3 of FIG. 10 will be described with reference to FIG. 12 and FIG.

[0079] 12 is a flowchart illustrating the process (S3) of selecting sound collection targets of the digital camera 100. Each process in the flowchart shown in FIG 12 is executed by, for example, the controller 135 of the digital camera 100 when the process proceeds to YES in step S11 of FIG 10.

[0080] 13 is a diagram for explaining the process of selecting a sound collection target (S3) in the digital camera 100. The operation of determining a sound collection target will be explained below using the examples in FIGS.

[0081] In the flowchart of Fig. 12, the controller 135 determines the subject corresponding to the face area of ​​the AF target identified in step S1 of Fig. 10 as the sound collection target (S10). At this time, the controller 135 sets the size of the face area of ​​the AF target (i.e., face width W) as a criterion for selecting the sound collection target from other subjects based on the detection information acquired from the face recognition unit 122.

[0082] Fig. 13(A) illustrates a case where a sound collection target is selected in the example of Fig. 11(A) and (B). Face widths W1, W2, and W3 indicate the size of face areas R1, R2, and R3 in the captured image Im by the width in the X-axis direction. In the example of Fig. 13(A), the controller 135 sets the face width W1 of the face area R1 of the AF target to the reference face width W (S10). The set face width W is held in, for example, the RAM of the controller 135.

[0083] Next, the controller 135 determines whether or not there is a detected subject other than the AF target (S11). Specifically, the controller 135 determines whether or not the detection information of the face recognition unit 122 includes a face area other than the face area of ​​the AF target.

[0084] If there is a detected subject other than the AF target (YES in S11), the controller 135 selects one subject i as a sound collection candidate that is a candidate for a sound collection target (S12). In the example of Fig. 13(A), the detection information is selected in association with the subject i of the sound collection candidate in sequence for each step S12, in other face areas R2 and R3 than the face area R1 of the AF target.

[0085] The controller 135 performs a calculation to compare the face width Wi of the selected subject i with the reference face width W (S13). Specifically, the controller 135 calculates the ratio Wi / W of the face width Wi of the subject i to the reference face width W. In the example of Fig. 13(A), when the face area R2 is selected as a sound collection candidate (S12), the ratio W2 / W for the face width W2 is calculated (S13).

[0086] The controller 135 judges whether the ratio Wi / W between the face width Wi of the sound collection candidate and the reference face width W is within a predetermined range (S14). The predetermined range is specified by an upper limit value greater than "1" and a lower limit value less than "1" from the viewpoint of specifying a range in which the face width Wi of the sound collection candidate is considered to be relatively similar to the reference face width Wi. Note that a user interface for setting the predetermined range may be provided, and the predetermined range set by the user via the operation unit 150 may be stored in the buffer memory 125 or the like.

[0087] When the controller 135 determines that the face width ratio Wi / W is within a predetermined range (YES in S14), it determines that the subject i is to be the sound collection target (S15).

[0088] On the other hand, when the controller 135 determines that the face width ratio W i / W is not within the predetermined range (NO in S14), the controller 135 determines that the subject i is not to be included in the sound collection target (S16). In the example of Fig. 13(A), the ratio W2 / W is below the lower limit of the predetermined range, and it is determined that the face area R2 is not to be included in the sound collection target.

[0089] When the controller 135 determines whether or not to collect sound from the subject i (S15 or S16), the controller 135 records the result of the determination for the subject i in the buffer memory 125 (S17). Next, the controller 135 performs the process from step S11 onward again for a subject other than the subject already selected as a sound collection candidate.

[0090] In the example of FIG. 13(A), in addition to face region R2, face region R3 is included in the detection information (YES in S11). When controller 135 selects a subject corresponding to face region R3 (S12), it calculates a ratio W3 / W of face width W3 to the reference face width W, similar to the case of face region R2 (S13). In the example of FIG. 13(A), the ratio W3 / W is calculated to be close to "1". Controller 135 determines that the calculated face width ratio W3 / W is within a predetermined range of the sound collection target (YES in S14), and determines the subject corresponding to face region R3 as the sound collection target (S15).

[0091] The controller 135 repeats the processes of steps S11 to S17 until there is no subject that has not been selected as a sound collection candidate (NO in step S11). After that, the controller 135 ends the selection process (S3) of the sound collection target, and proceeds to step S4 in FIG.

[0092] According to the above process, for subjects detected by face recognition, the relative sizes of face areas R2 and R3 are compared based on face area R1 specified as the AF target. This makes it possible to select subjects whose relative face area R3 is about the same size as the AF target face area R1 and determine them as sound collection targets.

[0093] Fig. 13(B) illustrates a case where a sound collection target is selected in the example of Fig. 11(C). In the example of Fig. 13(B), the face area R1 is specified as an AF target, similar to the example of Fig. 13(A). Therefore, the controller 135 determines the face area R1 as a sound collection target, and sets the face width W1 as a reference face width W (S10).

[0094] In the example of FIG. 13B, the face width W2 of the face region R2 is approximately the same as the reference face width W (=W1). On the other hand, the face width W3 of the face region R3 is larger than the other face widths W1 and W2. In this example, the controller 135 judges that the ratio W2 / W is within a predetermined range (YES in S14), and determines the subject of the face region R2 as the sound collection target (S15). On the other hand, since the ratio W3 / W exceeds the upper limit of the predetermined range (NO in S14), it is determined that the subject of the face region R3 is not the sound collection target (S16). Therefore, the sound collection targets in this example are determined to be two subjects corresponding to the face regions R1 and R2 (see FIG. 11C).

[0095] FIG. 13C illustrates an example in which a face region R3 is specified as the face region 60 of the AF target in a photographed image Im similar to that in FIG. 11C (S1 in FIG. 10). The controller 135 determines the face region R3 as the sound collection target, and sets the face width W3 as the reference face width W (S10). In the example of FIG. 13C, since the ratios W2 / W and W1 / W are below the lower limit of the predetermined range (NO in S14), it is determined that the subjects corresponding to the face regions R1 and R2 are not to be the sound collection target (S16). Therefore, the sound collection target in this example is determined to be one subject corresponding to the face region R3.

[0096] As described above, the digital camera 100 of this embodiment can determine, from among multiple subjects detected by image recognition, a subject that is approximately the same size as the AF target as the sound collection target, and can be used to determine the sound collection area in line with the user's intention, which will be described later.

[0097] [1-2-3. Sound collection area determination process] The sound collection area determination process in step S4 in FIG. 10 will be described in detail with reference to FIGS.

[0098] Fig. 14 is a flowchart illustrating the process (S4) of determining the sound collection area in the digital camera 100 of this embodiment. Each process in the flowchart shown in Fig. 14 is executed by, for example, the controller 135 of the digital camera 100 after executing step S3 in Fig. 10.

[0099] FIG. 15 is a diagram for explaining the sound collection area determination process (S4) in the digital camera 100. FIGS. 15(A) and (B) respectively illustrate a case where a sound collection area is determined following the example of FIGS. 13(A) and (B). FIG. 15(C) illustrates a case further different from that of FIGS. 15(A) and (B). In FIGS. 15(A) to (C), the center position x0 indicates the center position of the captured image Im in the X-axis direction, and the image width Wh indicates the width of the captured image Im in the X-axis direction. The image range is defined as the range x0±xh from the X coordinate −xh to xh on the captured image Im with the center position x0 as the reference. The X coordinate xh is defined as xh=Wh / 2 (>0).

[0100] 14, the controller 135 determines whether the center of the face region or the like of each sound collection target is within the central range of the captured image Im (S20). The central range is a range associated with the front central sound collection area 41 in the captured image Im.

[0101] The central range is defined as a range x0±xe from X coordinate -xe to xe on the captured image Im with respect to a central position x0 as shown in Fig. 15(A), for example. The X coordinate xe is defined as, for example, xe=xh×θe / θh (>0) based on a predetermined angle of view θe and a horizontal angle of view θh corresponding to the image width Wh. The predetermined angle of view θe is set in advance from the viewpoint of including one person, for example, 30°. The controller 135 obtains the current horizontal angle of view θh from, for example, the zoom magnification of the zoom lens of the optical system 110, and calculates the central range x0±xe.

[0102] In wide-angle shooting with a large horizontal angle of view θh, the X coordinate xe is small and the central range x0±xe is narrow. On the other hand, in telephoto shooting with a small horizontal angle of view θh, the X coordinate xe is large and the central range x0±xe is wide. This makes it easier to determine the sound collection area that corresponds to the physical range and distance to be captured.

[0103] If the positions of the face areas of all sound collection targets are within the central range (YES in S20), the controller 135 determines the sound collection area to be the front central sound collection area 41 (S21). In the example of FIG. 15(A), the sound collection targets correspond to face areas R1 and R3. The center positions x1 and x3 of the face areas R1 and R3 are both within the range of x0±xe (YES in S20). Therefore, the sound collection area is determined to be the front central sound collection area 41 (S21, see FIG. 11(B)).

[0104] On the other hand, if the position of the face area of ​​at least one sound collection target is not within the central range (NO in S20), a sound collection area other than the front central sound collection area 41 is used. In this case, the controller 135 determines, for example, for all sound collection targets, whether the position of the face area is only in the left or right half of the captured image Im (S22). The left half range is a range whose X coordinate is smaller than the central position x0 in the X-axis direction, and the right half range is a range whose X coordinate is larger than the central position x0.

[0105] If the positions of the face areas of all the sound collection targets are within only the left or right half of the captured image Im (YES in S22), the controller 135 further determines whether the positions of the face areas of all the sound collection targets are within the left half of the captured image Im (S23).

[0106] If the positions of the face areas of all sound collection targets are within the left half of the captured image Im (YES in S23), the controller 135 determines the sound collection area to be the left half sound collection area 42 (S24). In the example of FIG. 15(B), the sound collection targets correspond to face areas R1 and R2. Since the position x1 of face area R1 and the position x2 of face area R2 are on the left side (i.e., the X coordinate is smaller) of the center position x0 in the X-axis direction (YES in S23), the sound collection area is determined to be the left half sound collection area 42 (S24, see FIG. 11(C)).

[0107] On the other hand, if the positions of the face regions of all sound collection targets are within the right half of the captured image Im, and not within the left half (NO in S23), the controller 135 determines the sound collection area to be the right half sound collection area 43 (S25).

[0108] Furthermore, if the positions of the face regions of all the sound collection targets are not located in only the left or right half of the captured image Im (NO in S22), the controller 135 determines the sound collection area to be the front sound collection area 44 (S26). As shown in Figs. 6(D) and (A), the front sound collection area 44 has an angle range 402 wider than the angle range 401 of the front central sound collection area 41. In other words, the front sound collection area 44 includes subjects of the sound collection targets located in a wide range in the X-axis direction in the captured image Im.

[0109] In the example of Fig. 15(C), the sound collection targets correspond to face areas R1, R2, and R3. The center positions x1, x2, and x3 of the face areas R1 to R3 include positions x1 and x2 outside the center range x0±xe (NO in S20), and also include position x1 within the left half range and positions x2 and x3 within the right half range (NO in S22 and S23). Therefore, in this example, the sound collection area is determined to be the front sound collection area 44 (S26).

[0110] When the controller 135 determines the sound collection area (S21, S24 to S26), it records the determined sound collection area in the buffer memory 125 or the like as management information (S27). This ends the sound collection area determination process (S4), and the process proceeds to step S5 in FIG.

[0111] According to the above process, a collection area is determined from a plurality of predefined collection areas so as to include all of the collection targets according to the position of the subject determined as the collection target on the captured image. This allows the collection area to be determined so as to include the subject of the collection target according to the user's intention in shooting a video.

[0112] Fig. 17 is a diagram for explaining management information obtained by the sound collection area determination process (S4). Fig. 17(A) illustrates management information obtained at the stage of executing the sound collection target selection process (S3) and the sound collection area determination process (S4) in the examples of Fig. 13(A) and Fig. 15(A). Fig. 17(B) illustrates management information in the examples of Fig. 13(B) and Fig. 15(B).

[0113] The management information manages, for example, a "sound collection target" determined by a process of selecting a sound collection target (S3), a "sound collection area" determined by a process of determining a sound collection area (S4), a "horizontal angle of view," and a "focus distance" in association with each other. The focus distance is acquired, for example, when AF control (S1) is performed using face recognition. For example, the controller 135 may acquire the corresponding focus distance based on the positions or focal lengths of various lenses of the optical system 110 at the time of focusing. The digital camera 100 may also detect the focus distance by using DFD (Depth from Defocus) technology or measurement using a distance measuring sensor.

[0114] In addition, the digital camera 100 of this embodiment can set the angle of view θe of the central range used in determining the front central sound collection area (S20), and this is recorded, for example, in the ROM of the controller 135. Also, a user interface for setting the angle of view θe is provided, and the value set by the user via the operation unit 150 may be held in the buffer memory 125, for example.

[0115] [1-2-4. Sound pickup control] (1) Step S5 in Figure 10 The details of the sound collection control using face recognition in step S5 of FIG. 10 will be described with reference to FIGS.

[0116] In the sound collection control by the sound collection parameter setting, the digital camera 100 of the present embodiment sets the sound collection gain so as to emphasize the moving image sound of the subject corresponding to the face area of ​​the AF target, for example. The sound collection gain has, for example, a frequency filter characteristic and a stereo separation characteristic. The digital camera 100 calculates the sound collection gain based on the horizontal angle of view and the focusing distance when the digital camera 100 focuses on the face area of ​​the AF target while shooting a moving image, for example. The sound collection gain is specified so that, for example, the larger the calculated value, the more the sound collection zoom effect is generated by suppressing frequency bands other than human voices and controlling the stereo effect.

[0117] Fig. 16 is a flowchart illustrating sound collection control (S5) using face recognition. Each process shown in the flowchart of Fig. 16 is executed by, for example, the controller 135 of the digital camera 100 after executing step S4 of Fig. 10.

[0118] Digital camera 100 starts the process of step S5 in a state in which the management information shown in FIG. 17 is held.

[0119] The controller 135 acquires the horizontal angle of view from the buffer memory 125, for example, and calculates the gain Gh based on the horizontal angle of view (S30). Fig. 18(A) illustrates an example of the relationship for calculating the gain Gh from the horizontal angle of view. In the example of Fig. 18(A), the gain Gh increases as the horizontal angle of view becomes smaller between the predetermined maximum value Gmax and minimum value Gmin of the gain. This allows the gain to be increased during sound collection as the horizontal angle of view becomes smaller due to zooming, etc., thereby emphasizing the sound of the subject captured on the telephoto side.

[0120] The controller 135 acquires the focal distance as in step S30, and calculates the gain Gd based on the focal distance (S31). Fig. 18(B) illustrates an example of the relationship for calculating the gain Gd from the focal distance. In the example of Fig. 18(B), the gain Gd increases as the focal distance increases, between a predetermined maximum value Gmax and minimum value Gmin of the gain. This allows the gain to be increased when focusing on an object farther from the digital camera 100, and the sound can be emphasized as the object is farther away.

[0121] The controller 135 compares the calculated sound collection gain Gh based on the horizontal angle of view with the sound collection gain Gd based on the focal distance, and sets the larger gain as the sound collection gain G (S32). This allows the sound collection gain G to be calculated so as to emphasize the sound of the subject in accordance with the intention of the user who is shooting with, for example, a telephoto horizontal angle of view or a long focal distance.

[0122] The controller 135 judges whether the sound collection gain G calculated a predetermined number of times in the past (for example, five times) and the determined sound collection area are the same (S33). For example, the sound collection gain G is stored together with the above management information each time it is calculated within a range of a predetermined number of times in the execution cycle of steps S1 to S5 in Fig. 10. If the controller 135 judges that the sound collection gain G and sound collection area calculated a predetermined number of times in the past are the same (YES in S33), the controller 135 proceeds to step S34.

[0123] The controller 135 sets the sound collection target determined by the sound collection target selection process in step S3, the sound collection area determined by the sound collection area determination process in step S4, and the sound collection gain G calculated in step S32 as sound collection parameters in the sound processing engine 170 (S34). The sound processing engine 170 realizes the sound collection area and sound collection gain according to the set sound collection parameters by the beam forming unit 172 and the gain adjustment unit 174.

[0124] After setting the sound collection parameters (S34), the controller 135 ends the process of sound collection control using face recognition (S5). If the controller 135 determines that the sound collection gain G and sound collection area are not the same for a predetermined number of times in the past (NO in S33), it ends the process of step S5 in Fig. 10 without performing the process of step S34. Then, the process of step S1 and subsequent steps in Fig. 10 is repeated.

[0125] According to the above processing, the calculated sound collection gain and the sound collection target and sound collection area determined based on face recognition can be set as sound collection parameters, thereby realizing a sound collection area and sound collection gain that makes it easier to clearly pick up the sound of the subject of the sound collection target, including the AF target.

[0126] The order of execution of steps S30 and S31 is not limited to that of this flowchart. For example, the gain Gd may be calculated in step S31 and then the gain Gh may be calculated in step S30, or steps S30 and S31 may be executed in parallel.

[0127] Also, according to the above step S33, only when the sound collection area and the sound collection gain G do not change a predetermined number of times (for example, five times), the process (S34) of setting the sound collection parameters is executed. This prevents the sound collection area and the sound collection gain G from being changed too frequently due to the movement of the subject, etc., and enables the sound collection control (S5) using face recognition to be realized with high accuracy according to the user's intention.

[0128] (2) Step S6 in Figure 10 The details of the sound collection control (S6) without using face recognition in step S6 of FIG. 10 will be described with reference to FIG.

[0129] Fig. 19 is a flowchart illustrating sound collection control (S6) without face recognition. Each process shown in the flowchart of Fig. 19 is executed by, for example, the controller 135 of the digital camera 100 when there is no face area to be the AF target in step S2 of Fig. 10 (NO in S2), such as when no face area is detected.

[0130] First, the controller 135 determines the sound collection area to be, for example, the front sound collection area 44 (S40).

[0131] Next, the controller 135 calculates the gain Gh based on the horizontal angle of view in the same manner as in step S30, and sets the calculated gain Gh as the sound collection gain G (S41). Furthermore, the controller 135 determines whether the sound collection gain G calculated a predetermined number of times in the past and the determined sound collection area are the same as in step S33 (S42).

[0132] When the controller 135 determines that the sound collection gain G and sound collection area are the same for the predetermined number of times in the past (YES in S42), it sets the sound collection area and sound collection gain G as sound collection parameters (S43) and ends the sound collection control without face recognition (S6). When the controller 135 determines that the sound collection gain G and sound collection area are not the same for the predetermined number of times in the past (NO in S42), it ends step S6 in Fig. 10 without performing the process of step S43. After the end of step S6, the processes from step S1 onwards are repeated.

[0133] According to the above processing, even if there is no face area to be AF-targeted, sound from a wide range in front of the digital camera 100 can be picked up, and the smaller the horizontal angle of view due to zooming, etc., the larger the sound pickup gain is, making it easier to clearly pick up sound from within the range being captured.

[0134] Note that a total sound collection area having an angular range of 360° around the digital camera 100 may be defined according to the operation mode of the digital camera 100, and may be determined as the total sound collection area in step S40. At this time, for example, only the total sound collection area may be set as the sound collection parameter.

[0135] [1-2-5. Auto mode operation] The operation of the auto mode of the digital camera 100 according to this embodiment will be described in detail with reference to FIGS.

[0136] Fig. 20 is a flowchart illustrating an example of the operation of the auto mode of digital camera 100 according to embodiment 1. Each process shown in the flowchart of Fig. 20 is executed in a state in which digital camera 100 is set to the auto mode by controller 135, for example, in the same manner as in Fig. 10.

[0137] 20, in the digital camera 100 in the auto mode, the controller 135 determines whether or not the display monitor 130 is in the selfie position based on, for example, a detection signal from the magnetic sensor 132 (S51). If the controller 135 determines that the display monitor 130 is not in the selfie position (NO in S51), it sets the sound collection area for non-face recognition to the sound collection area 45 in the surround mode (S52). On the other hand, if the controller 135 determines that the display monitor 130 is in the selfie position (YES in S51), it sets the sound collection area for non-face recognition to the sound collection area 46 in the front mode (S53).

[0138] The controller 135 performs face recognition processing (S1, S2) in the same manner as in the focus mode described above. For example, when a face area of ​​the AF target is detected (YES in S2), the controller 135 judges whether or not the digital camera 100 is in a vertical shooting position based on a detection signal from the acceleration sensor 137 (S54). When the controller 135 judges that the shooting is not vertical (NO in S54), it performs processing of steps S3 to S5 similar to the focus mode and executes sound collection control. On the other hand, when the controller 135 judges that the shooting is vertical (YES in S54), it employs the sound collection area 46 of the front mode and executes sound collection control (S55).

[0139] If the face area of ​​the AF target is not detected (NO in S2), the controller 135 performs sound collection control without face recognition based on the setting results of steps S52 and S53 (S6A). The sound collection control of step S6A is performed in the same manner as in step S6 described above, using the sound collection area set as the sound collection area when no face is recognized.

[0140] The above process realizes an auto mode operation that adjusts the directivity of the microphone 161 in conjunction with various shooting conditions. Sound collection control in the auto mode during horizontal and vertical shooting will be further described with reference to FIG.

[0141] Fig. 21(A) illustrates an example of the relationship between a captured image Im and sound collection areas 41 to 43 when taken horizontally. Fig. 21(B) illustrates an example of a case where step S55 is not performed when taken vertically. Fig. 21(C) illustrates an example of a case where step S55 is performed when taken vertically.

[0142] As shown in Fig. 21(A), during side-by-side capture, the sound collection areas, face areas R1-R3, and sound collection areas 41-43 are in a coaxial relationship, and by switching the sound collection areas 41-43, it is possible to perform intended sound collection control to match the positions of the desired face areas R1-R3. However, during vertical shooting, as shown in Fig. 21(B), the relationship between the sound collection areas 41-43 and the face areas R1-R3 does not match. For this reason, it is possible that sound collection control cannot be performed as intended for the positions of the desired face areas R1-R3, and instead unintended sound collection control may occur.

[0143] Therefore, in this embodiment, during vertical shooting, the sound collection area is fixed to the sound collection area 46 in the front mode as shown in Fig. 21(C). This ensures a range in which sound can be collected throughout the range being imaged, and makes it possible to avoid situations in which unintended sound collection control occurs.

[0144] [1-3. Effects, etc.] In this embodiment, the digital camera 100 includes an image sensor 115 as an example of an imaging unit, a microphone 161 as an example of an audio acquisition unit, an operation unit 150 as an example of a setting unit, and a controller 135 as an example of a control unit. The image sensor 115 captures an image of a subject to generate image data. The microphone 161 acquires an audio signal indicating audio picked up during imaging by the imaging unit. The operation unit 150 receives an instruction from a user and sets the device to an auto mode, which is an operation mode in which the directivity of the audio acquisition unit is automatically changed. The controller 135 controls a sound collection area in which audio from the subject is collected in the audio signal. When the controller 135 is set to the auto mode, the controller 135 controls the sound collection area to include the subject by changing the directivity of the microphone 161 in conjunction with the imaging state of the device. This allows appropriate sound collection control to be realized according to various imaging states, and makes it easier to collect the audio of the subject according to the user's intention when imaging while acquiring audio.

[0145] The digital camera 100 of this embodiment includes a face recognition unit 122, which is an example of a face detection unit that detects the face area of ​​a subject in image data. When the controller 135 is set to the auto mode, the controller 135 determines the subject to be collected in the audio signal based on the face area detected by the face recognition unit 122, and controls the sound collection area so that the determined subject is included in the sound collection target. This makes it possible to perform sound collection control according to the shooting conditions, such as various face recognition of the subject, and to easily collect sound according to the user's intention.

[0146] In this embodiment, when the controller 135 is set to the auto mode, the controller 135 controls the sound collection area so as to change the directivity of the sound acquisition unit in conjunction with the shooting state of the device itself, whether the device is in portrait or landscape orientation. This makes it possible to perform sound collection control according to the shooting state, such as portrait or landscape, and to easily collect sound according to the user's intention.

[0147] In this embodiment, when the controller 135 is set to the auto mode, the controller 135 controls the sound collection area so as to change the directivity of the sound acquisition unit in conjunction with the shooting state of whether or not the photographer is shooting himself / herself. This makes it possible to perform sound collection control according to the shooting state of whether or not the photographer is shooting a self-portrait, making it easier to collect sound according to the user's intention.

[0148] The digital camera 100 of this embodiment further includes a display monitor 130 as an example of a display unit, and a magnetic sensor 132 as an example of a detection unit. The display monitor 130 has a display surface for displaying an image of a subject, and is configured to be able to displace the display surface toward the subject. The magnetic sensor 132 detects whether the display monitor 130 has displaced the display surface toward the subject. The setting unit of this embodiment may set the digital camera 100 to the auto mode when the magnetic sensor 132 detects that the display monitor 130 has displaced the display surface toward the subject. For example, the controller 135 may automatically set the digital camera 100 to the auto mode when the display monitor 130 is in the selfie position in response to a detection signal from the magnetic sensor 132.

[0149] In the digital camera 100 of this embodiment, the setting unit can set the sound acquisition unit to at least one of a plurality of operation modes having different directivities in addition to the auto mode in response to a user's instruction. For example, the operation mode can be set to a surround mode, a front mode, or a navigation mode, and may also be set to a focus mode.

[0150] In the digital camera 100 of this embodiment, when the setting unit is set to the auto mode and imaging is started by the image sensor 115, the controller 135 may cause the display monitor 130 to display information indicating that the auto mode is set along with the subject. For example, the controller 135 may cause the display monitor 130 to display an icon or the like dedicated to the auto mode.

[0151] (Embodiment 2) Hereinafter, the second embodiment will be described with reference to the drawings. In the first embodiment, the digital camera 100 that selects and determines a sound collection target when shooting a moving image or the like has been described. In the second embodiment, the digital camera 100 that visualizes information related to the determined sound collection target to the user during the operation as in the first embodiment will be described.

[0152] Hereinafter, the description of the configuration and operation similar to those of the digital camera 100 according to the first embodiment will be omitted as appropriate and the digital camera 100 according to this embodiment will be described.

[0153] [2-1. Overview] An overview of the operation of the digital camera 100 according to this embodiment for displaying various types of information will be described with reference to FIG.

[0154] Fig. 22 shows a display example of the digital camera 100 according to this embodiment. The display example of Fig. 22 shows an example displayed in real time on the display unit 130 when the digital camera 100 has determined a sound collection target as shown in Fig. 11(B) as an example. In this display example, the digital camera 100 displays on the display monitor 130 an AF frame 11 indicating the subject to be AF-targeted and a detection frame 13 indicating a detected subject other than the AF target, as well as a sound collection icon 12 indicating the subject to be collected, superimposed on the captured image Im.

[0155] The digital camera 100 of this embodiment uses a sound collection icon 12 in combination with the AF frame 11 and detection frame 13 to visualize to the user whether a main subject such as an AF target and other detected subjects have been determined as AF targets and / or sound collection targets.

[0156] For example, in the display example of Fig. 22, the digital camera 100 has determined that the subject corresponding to face area R1 (60) in the example of Fig. 11(B) is an AF target and a sound collection target, and therefore displays the AF frame 11 and the sound collection icon 12 for the person 21. Also, the digital camera 100 has determined that the subject corresponding to face area R3 in the example of Fig. 11(B) is a sound collection target other than the AF target, and therefore displays the detection frame 13 and the sound collection icon 12 for the person 23. Furthermore, by displaying the detection frame 13 without the sound collection icon 12, the digital camera 100 visualizes to the user that the subject other than the AF target corresponding to face area R2 in the example of Fig. 11(B) has been determined not to be a sound collection target.

[0157] In the digital camera 100 of this embodiment, the user can check whether or not the detected subject is an AF target by the display of either the AF frame 11 or the detection frame 13. The user can also check whether or not the subject is a sound collection target by the presence or absence of the sound collection icon 12. The combination of the AF frame 11 and the sound collection icon 12 is an example of first identification information in this embodiment. The combination of the detection frame 13 and the sound collection icon 12 is an example of second identification information in this embodiment. The detection frame 13 is an example of third identification information.

[0158] As described above, the digital camera 100 according to this embodiment displays a distinction between the sound collection target and the AF target determined from the subject included in the detection information. This allows the user to know which subjects are sound collection targets among the subjects detected by the digital camera 100, and to check, for example, whether or not the intended subject has been determined as the sound collection target.

[0159] 23(A) and (B) are diagrams for explaining manual operations in the digital camera 100 of this embodiment. FIG. 23(A) illustrates a state in which a specific person 22 has not been detected by the face recognition unit 122. For example, even if the photographer wants to collect the voice of the person 22, the face of the person 22 may be facing sideways or backwards with respect to the digital camera 100, and thus face recognition may not be performed. In addition, the sound source that the photographer wants to collect may not be a person. In consideration of such cases, the digital camera 100 of this embodiment operates to allow input of a manual operation that allows the sound collection area to be manually set.

[0160] Fig. 23(B) illustrates an example of manual operation of the sound collection area in digital camera 100. For example, the manual operation of the sound collection area is implemented as a touch operation. Fig. 24 is a flowchart illustrating an example of an operation during manual operation in digital camera 100. The processing shown in this flowchart may be executed independently of the processing in the auto mode or focus mode described above, or may be executed by interrupting the processing in the auto mode or focus mode.

[0161] First, the controller 135 of the digital camera 100 accepts a manual operation by a user such as a photographer (S61). During manual operation, the controller 135 displays a designated range 48 of the sound collection area on the display monitor 130, as shown in Fig. 23(B), for example. This example is a case where a manual operation is performed by interrupting processing in the auto mode or focus mode.

[0162] In the example of Fig. 23(B), the photographer inputs a manual operation by touch operation to adjust the designated range 48 of the sound collection area so as to include the person 22 whose face was not recognized. The controller 135 determines the designated range 48 of the sound collection area based on the input manual operation (S62). If the start point and end point of the sound collection range are input by touch operation as the manual operation, a sound collection range 48 of a predetermined size including the start point and end point is displayed. In this state, if the confirmation button displayed on the display monitor 130 is touched, the sound collection range 48 is confirmed.

[0163] The controller 135 reflects the determined designated range 48 of the sound collection area in the sound collection control of the microphone 161 (S63). As a result, the sound collection control is executed so as to emphasize the sound from the sound collection area corresponding to the designated range 48.

[0164] [2-3. Effects, etc.] As described above, the digital camera 100 of this embodiment includes the image sensor 115 that captures an image of a subject and generates image data, the microphone 161 that acquires an audio signal indicating audio picked up during imaging by the image sensor 115, and the display monitor 130 that displays an image of the subject. The digital camera 100 of this embodiment includes an input unit such as an operation unit 150 that inputs a user operation to set the subject displayed on the display monitor 130 as a sound collection area that collects audio from the subject, and a controller 135 that controls the sound collection area in the audio signal. When a user operation to set the sound collection area is input, the controller 135 controls the sound collection area so as to include the subject by changing the directivity of the microphone 161 based on the user operation. Such manual operation allows the sound collection area to be controlled, making it easier to collect audio according to the user's intention.

[0165] In the digital camera 100 of this embodiment, the display monitor 130 may be configured to be able to displace the display surface toward the subject, similarly to the first embodiment. When the display surface of the display monitor 130 is displaced toward the subject, a user operation may be performed. That is, when taking a selfie with the digital camera 100, the above-mentioned manual operation may be input.

[0166] (Other embodiments) As described above, the above-mentioned embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are appropriately made. In addition, it is also possible to combine the components described in the above-mentioned embodiments to create new embodiments.

[0167] In the first embodiment, an example has been described in which three microphone elements 161L, 161C, and 161R are used in the microphone 161. A modified example in which four microphone elements are used will be described with reference to Figs. 25 and 26.

[0168] 25 illustrates an example of the arrangement of microphone 161A in digital camera 100A of this modification. In this modification, microphone 161A of digital camera 100A includes a fourth microphone element 161B in addition to three microphone elements 161L, 161C, and 161R that are mutually on the XZ plane. The fourth microphone element 161B is arranged so that its position in the Y direction is different from the other microphone elements 161L to 161R.

[0169] FIG. 26 illustrates the relationship between the captured image Im and the sound collection areas 41A to 43A during vertical shooting in this modified example. According to the configuration of the microphone 161A described above, the sound collection area can also be changed in the Y-axis direction of the digital camera 100. Therefore, during sound collection control during vertical shooting, by using the sound collection area using the fourth microphone element 161B, sound collection control that follows face recognition even during vertical shooting can be realized, for example, as shown in FIG. 26. For example, the controller 135 of the digital camera 100A of this modified example performs sound collection control using the fourth microphone element 161B as described above, instead of step S55, in the same process as in FIG. 20. This allows for intended sound collection control to be performed using the sound collection areas 41A to 43A to match the positions of the desired face areas R1 to R3, even during vertical shooting.

[0170] In addition, in the sound collection control described in each of the above embodiments, by providing speed and variability in the period in which the sound collection area is transitioned in conjunction with face recognition, it is possible to further suppress the discomfort felt by the listener. An example of this operation will be described with reference to FIG.

[0171] 27 shows an example of operation in which the width of sound collection directivity (i.e., the angular range of the sound collection area) is changed in conjunction with the presence or absence of face recognition in digital camera 100. In this example of operation, the face of a subject is detected by face recognition unit 122 of digital camera 100 at time t1. Then, erroneous detection prevention control (i.e., chattering) is performed (see S33 in FIG. 16).

[0172] For example, if face recognition is performed only for an instant, or if the subject whose face has been recognized turns to the side, changing the sound pickup directivity may cause a user listening to the collected sound to feel uncomfortable. In response to this, the above-mentioned erroneous detection prevention control constantly monitors the position of the subject's face recognition, and changes the sound pickup directivity when the subject is in the sound collection area for a certain period of time. This makes it possible to avoid the above-mentioned uncomfortable feeling.

[0173] In addition, the controller 135 of the digital camera 100 transitions the sound collection area so as to narrow the sound collection directivity after chattering from time t1. The transition period when narrowing the sound collection directivity is set to be relatively short, for example. This allows the sound collection directivity to be quickly changed so as to be focused on the detected face when the face of the subject is detected in the image recognition, and a good impression can be given to the user listening to the collected sound results. In addition, the transition of sound collection areas having the same angle range, such as the front center sound collection area 41 and the left half sound collection area 42, is also performed relatively quickly in the same manner as above.

[0174] In the example of FIG. 27, for example, the subject moves or turns his / her face sideways, and thus the face of the subject is no longer detected by the face recognition unit 122 at time t2. In this case, the digital camera 100 also transitions the sound collection directivity after chattering control. Here, the transition period when the range of the sound collection directivity is widened is set longer than the transition period when narrowing the range. This makes it possible to avoid giving the user an uncomfortable feeling in hearing that may occur when a situation occurs suddenly where sounds from a wide range start to be heard. Rather, by executing the transition when the range of the sound collection directivity is widened slowly, it is possible to suppress the uncomfortable feeling in hearing of the user.

[0175] Furthermore, in this example, face recognition is performed again at time t3 during the transition period in which the sound collection directivity is widened. In this case, the digital camera 100 switches to narrowing control again before the sound collection directivity is fully widened as interruption control. This allows the control of directing the sound collection directivity to the subject whose face has been recognized to be quickly performed when the subject's face is recognized intermittently, thereby further suppressing the discomfort felt by the user.

[0176] In each of the above embodiments, the face recognition unit 122 is used to detect the sound collection target. In this embodiment, the detection of the sound collection target is not limited to the face recognition unit 122, and for example, instead of or in addition to the face recognition unit 122, human body recognition that performs image recognition of the whole or at least a part of a human may be used. Furthermore, the sound collection target does not necessarily have to be a person, and may be, for example, various animals. In this case, the sound collection target may be detected by image recognition of a part or the whole of the animal.

[0177] Moreover, the first identification information, the second identification information, and the third identification information according to the second embodiment identify whether or not the subject is a main subject based on the presence or absence of the AF frame 11, and identify whether or not the subject is a sound collection target based on the presence or absence of the sound collection icon 12. In this embodiment, the first to third identification information are not particularly limited to this, and may be, for example, three types of frame displays. FIG. 22 illustrates three types of frame displays according to this embodiment. In the example of FIG. 22, the display of the AF target and other subjects and the display of the sound collection target are integrated with the display of the sound collection target by the frame display 11A indicating a subject that is an AF target and a sound collection target, the frame display 13A indicating a subject that is not an AF target and a sound collection target, and the frame display 13B indicating a subject that is not a sound collection target.

[0178] In the above first and second embodiments, the operation in the auto mode and the manual operation have been described, but these may be combined. That is, the digital camera 100 of this embodiment may include a display monitor 135 as a display unit that displays an image of a subject, and an operation unit 150 as an input unit that inputs a user operation to set the subject displayed on the display monitor 135 as a sound collection area that collects sound from the subject. When a user operation to set the sound collection area is input, the controller 135 may control the sound collection area so as to include the subject by changing the directivity of the microphone 161 based on the user operation. In such a case, the display monitor 135 does not need to be particularly movable, and may be a fixed type that is fixed, for example, to the normal position described above.

[0179] In the first and second embodiments, an operation example of performing sound collection control (S5 or S6) using or not using face recognition for the microphone 161 built into the digital camera 100 has been described in the flowchart of FIG. 10. The digital camera 100 of this embodiment may be provided with an external microphone (hereinafter referred to as "microphone 161a") instead of the built-in microphone 161. The microphone 161a includes a microphone element outside the digital camera 100 and has three or more microphone elements. In this embodiment, the controller 135 can execute step S5 or S6 as in the first embodiment by storing information on the arrangement of the microphone elements for the microphone 161a in advance in the buffer memory 125 or the like. In this case, too, it is possible to easily obtain the sound of the subject clearly according to the sound collection target and / or sound collection area determined as in the first embodiment.

[0180] In the first and second embodiments, the operation example of calculating (S30) the gain Gh based on the horizontal angle of view corresponding to the imaging range of the digital camera 100 has been described in the flowchart of FIG. 16. The horizontal angle of view in this case is the same as the horizontal angle of view θh used in determining the front center sound collection area (S20) in the flowchart of FIG. 14. In this embodiment, a horizontal angle of view different from the horizontal angle of view θh in step S20 may be used to calculate the gain Gh. For example, the angle range corresponding to the width in the X-axis direction including all the sound collection target subjects on the captured image is set as the horizontal angle of view in step S30. This allows the gain Gh to be calculated according to the angle of view at which the sound collection target is captured so as to collect the voice of a distant subject more clearly.

[0181] In the first and second embodiments, the face recognition unit 122 detects a human face. In the present embodiment, the face recognition unit 122 may detect, for example, an animal's face. It is considered that the size of an animal's face varies depending on the type of animal. Even in this case, the sound collection target can be selected as in the first embodiment, for example, by expanding the predetermined range (see S14) for selecting the sound collection target. Furthermore, the face recognition unit 122 may detect a face for each type of animal, and set the predetermined range in step S14 according to the type.

[0182] In the first and second embodiments, the digital camera 100 is described as including the face recognition unit 122. In the present embodiment, the face recognition unit 122 may be provided in an external server. In this case, the digital camera 100 may transmit image data of a captured image to the external server via the communication module 160, and receive detection information of the processing result by the face recognition unit 122 from the external server. In such a digital camera 100, the communication module 160 functions as a detection unit.

[0183] In addition, in the first and second embodiments, the digital camera 100 is illustrated as including the optical system 110 and the lens driving unit 112. The imaging device of the present embodiment does not need to include the optical system 110 and the lens driving unit 112, and may be, for example, an interchangeable lens type camera.

[0184] In addition, in the first and second embodiments, a digital camera has been described as an example of an imaging device, but the imaging device is not limited to this. The imaging device of the present disclosure may be any electronic device having an image capturing function (for example, a video camera, a smartphone, a tablet terminal, etc.).

[0185] As described above, the embodiments have been described as examples of the technology in the present disclosure, and the accompanying drawings and detailed description have been provided for this purpose.

[0186] Therefore, among the components described in the attached drawings and the detailed description, not only are there components essential for solving the problem, but there may also be components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that such non-essential components are described in the attached drawings or the detailed description should not be interpreted as immediately indicating that such non-essential components are essential.

[0187] Furthermore, since the above-described embodiments are intended to illustrate the technology in the present disclosure, various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0188] The present disclosure is applicable to imaging devices that capture images while acquiring audio. [Explanation of symbols]

[0189] 100 Digital Camera 115 Image Sensor 120 Image Processing Engine 122 Face Recognition Unit 125 Buffer Memory 130 Display Monitor 132 Magnetic Sensor 135 Controller 137 Acceleration Sensor 145 Flash Memory 150 Operation section 11 AF frame 12 Audio recording icon 13 Detection Frame

Claims

1. an imaging unit that captures an image of a subject and generates image data; an image recognition unit that recognizes the subject based on the image data; a sound acquisition unit that acquires a sound signal indicating a sound picked up during imaging by the imaging unit; A display unit that displays an image of the subject; an input unit for inputting a user operation for setting the subject displayed on the display unit as a sound collection area for collecting sound from the subject; A control unit for controlling the sound collection area, the control unit determines a first object to be a sound collection target in the audio signal based on a recognition result of the object by the image recognition unit, and controls the sound collection area to include the first object; When a user operation is input to set a second object not recognized by the image recognition unit to be included in the sound collection area, the control unit controls a range of the sound collection area to include the second object by changing a directivity of the sound acquisition unit based on the user operation. Imaging device.

2. the display unit is configured to be able to displace a display surface toward the subject, The imaging device according to claim 1 , wherein the user performs the operation on the display unit while the display surface is displaced toward the subject.

Images