Image pickup apparatus, method of controlling image pickup apparatus, and program

Abstract

To provide an image pickup apparatus capable of properly detecting an object in a case where a lens unit having two optical systems disposed in parallel with each other is used.SOLUTION: An image pickup apparatus (100) includes: imaging means (211) configured to photoelectrically convert a first optical image formed by a first optical system (301L) to output first image data, and to photoelectrically convert a second optical image formed by a second optical system (301R) arranged in parallel with the first optical system to output second image data; and control means (50) configured to determine a main object based on first object information in the first image data and second object information in the second image data.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an imaging device, a control method for an imaging device, and a program. [Background technology]

[0002] Patent Document 1 discloses a camera equipped with a lens unit having two optical systems, capable of capturing two images with parallax at once. If the two optical systems are arranged to capture images in the same direction, it is possible to create an image capturing a range of 180 degrees (an image of a hemisphere) or an image that can be viewed stereoscopically from the two captured images with parallax.

[0003] Patent document 2 discloses an imaging device that, when a lens unit having two optical systems is attached, performs automatic focus adjustment (autofocus: AF) by changing the photometry and distance measurement area to an area that includes the center of at least one of two images with parallax. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-141052 A [Patent Document 2] JP 2001-222083 A Summary of the Invention [Problem to be solved by the invention]

[0005] To solve the problem that, in detecting a subject when performing automatic focus adjustment, it may be impossible to detect the subject due to a change in the subject's direction, angle, or sudden position, or an area unrelated to the subject may be detected, making it impossible to appropriately detect the subject while accurately following the movement of the subject.

[0006] In particular, when using a lens unit with two optical systems arranged in parallel to each other, two optical images are formed on one image sensor, which is equivalent to taking an image at half the size of the original image sensor. This means that the accuracy of subject detection may be reduced compared to when a general monocular lens is used.

[0007] SUMMARY OF THE PRESENT DISCLOSURE In view of the above, an object of the present invention is to provide an imaging device that is capable of more appropriately detecting a subject when using a lens unit having two optical systems arranged in parallel with each other. [Means for solving the problem]

[0008] An imaging device according to one aspect of the present invention has imaging means for photoelectrically converting a first optical image formed by a first optical system to output first image data, and for photoelectrically converting a second optical image formed by a second optical system arranged in parallel to the first optical system to output second image data, and control means for determining a main subject based on first subject information in the first image data and second subject information in the second image data.

[0009] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0010] According to the present invention, it is possible to provide an imaging device that is capable of more appropriately detecting a subject when using a lens unit having two optical systems arranged in parallel with each other. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram showing an example of an overall configuration of an imaging system in each embodiment. [Diagram 2] FIG. 2 is an external view of an imaging device according to each embodiment. [Diagram 3] FIG. 2 is a block diagram of an imaging device in each embodiment. [Figure 4]FIG. 2 is a configuration diagram of a lens device in each embodiment. [Diagram 5] FIG. 4 is an explanatory diagram of a captured image in each embodiment. [Figure 6] 5 is a flowchart showing a shooting process in each embodiment. [Figure 7] 5 is a flowchart showing a main subject determination process in the first embodiment. [Figure 8] 10 is a flowchart showing a main subject determination process in the second embodiment. [Figure 9] 10 is a flowchart showing a noise determination process in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0013] <First embodiment> First, a first embodiment of the present invention will be described. Fig. 1 is a schematic diagram showing an example of the overall configuration of an imaging system 10 in this embodiment. The imaging system 10 has a camera (imaging device) 100 and a lens unit (lens device) 300 that is detachable from the camera 100. Although details of the lens unit 300 will be described later, by attaching the lens unit 300, the camera 100 can simultaneously capture two images (still images or video) having a predetermined parallax.

[0014] 2(a) and (b) are external views showing an example of the camera 100. Fig. 2(a) is a perspective view of the camera 100 seen from the front side, and Fig. 2(b) is a perspective view of the camera 100 seen from the rear side.

[0015] The camera 100 has on its top surface a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an outside viewfinder display unit 107. The shutter button 101 is an operating member for issuing a shooting preparation instruction or a shooting instruction. The power switch 102 is an operating member for switching the power of the camera 100 on and off.

[0016] The mode change switch 103 is an operating member for switching between various modes. The main electronic dial 104 is a rotary operating member for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operating member for moving a selection frame (cursor) and forwarding images. The video button 106 is an operating member for issuing instructions to start and stop video shooting (recording). The outside viewfinder display unit 107 displays various settings such as shutter speed and aperture.

[0017] Camera 100 has, on the rear surface, a display unit 108, a touch panel 109, direction keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, a menu button 115, an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar 119. The display unit 108 is a display unit that displays images and various information.

[0018] The touch panel 109 is an operation member that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are an operation unit made up of keys (four-way keys) that can be pressed up, down, left, and right. Processing can be performed according to the position where the directional keys 110 are pressed. The SET button 111 is an operation member that is pressed mainly to confirm a selection item. The AE lock button 112 is an operation member that is pressed to fix the exposure state in a shooting standby state.

[0019] The enlargement button 113 is an operating member for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is also used to enlarge the playback image or increase the magnification ratio in the playback mode.

[0020] The playback button 114 is an operating member for switching between a shooting mode and a playback mode. When the playback button 114 is pressed in the shooting mode, the mode changes to the playback mode, and the latest image among the images recorded in the recording medium 227 described later can be displayed on the display unit 108.

[0021] Menu button 115 is an operating member that is pressed to display a menu screen on which various settings can be made on display unit 108. A user can intuitively make various settings using the menu screen displayed on display unit 108, direction keys 110, and SET button 111. Eyepiece unit 116 is a portion into which an eyepiece finder (a peer-in type finder) 117 is placed close to the eye and peered into.

[0022] A user can view an image displayed on an EVF 217 (Electronic View Finder) (described later) inside the camera 100 through the eyepiece unit 116. The eyepiece detection unit 118 is a sensor that detects whether or not the user places his / her eye close to the eyepiece unit 116 (eyepiece finder 117).

[0023] Touch bar 119 is a line-shaped touch operation member (line touch sensor) capable of receiving a touch operation. Touch bar 119 is disposed at a position where it can be touched by the thumb of the right hand when grip portion 120 is held in the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that shutter button 101 can be pressed with the index finger of the right hand. In other words, touch bar 119 can be operated in a state (shooting posture) where the eye is placed close to eyepiece viewfinder 117, the eye is looked into eyepiece portion 116, and the camera is held so that shutter button 101 can be pressed at any time.

[0024] Touch bar 119 can accept tap operations (operation of touching and then releasing the touched position within a predetermined period of time without moving it) and left and right slide operations (operation of touching and then moving the touched position while still touching) on ​​touch bar 119. Touch bar 119 is an operating member different from touch panel 109 and does not have a display function. Touch bar 119 functions as, for example, a multi-function bar (M-Fn bar) to which various functions can be assigned.

[0025] The camera 100 also has a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section formed in a shape that is easy to hold with the right hand when the user holds the camera 100. In a state in which the camera 100 is held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are disposed in positions that can be operated with the index finger of the right hand. In a similar state, the sub electronic dial 105 and the touch bar 119 are disposed in positions that can be operated with the thumb of the right hand.

[0026] Thumb rest section 121 (thumb standby position) is a grip section provided on the rear side of camera 100 at a position where it is easy to place the thumb of the right hand holding grip section 120 when none of the operation members are being operated. Thumb rest section 121 is made of a rubber member or the like for enhancing holding power (grip feeling). Terminal cover 122 protects connectors such as a connection cable that connects camera 100 to an external device (external apparatus).

[0027] Lid 123 protects recording medium 227 and the slot by closing the slot for storing recording medium 227 described later. Communication terminal 124 is a terminal for communicating with a lens unit (such as lens unit 200 described later or lens unit 300 shown in FIG. 1) that is detachable from camera 100.

[0028] Fig. 3 is a block diagram showing an example of the configuration of camera 100. Note that the same components as those in Fig. 2 are given the same reference numerals as those in Fig. 2, and descriptions of those components will be omitted as appropriate. Fig. 3 shows a state in which lens unit (lens device) 200 is attached to camera 100.

[0029] First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that is detachable from the camera 100. The lens unit 200 is a single-lens lens, and is an example of a normal lens. The lens unit 200 has an aperture (aperture diaphragm) 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0030] The aperture 201 is configured so that the aperture diameter can be adjusted. The lens 202 is composed of a plurality of lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit (control means) 50, which will be described later.

[0031] The lens system control circuit 205 controls the aperture 201 via an aperture drive circuit 203, and adjusts the focus by changing the position of the lens 202 via an AF drive circuit 204. The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, communication is performed via a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100.

[0032] Next, a description will be given of the camera 100. The camera 100 has a shutter 210, an imaging section (imaging means) 211, an A / D converter 212, a memory control section 213, an image processing section 214, a memory 215, a D / A converter 216, an EVF 217, a display section 108, and a system control section 50.

[0033] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on an instruction from the system control unit 50. The imaging unit 211 is an imaging element (image sensor) composed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 211 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts an analog signal output from the imaging unit 211 into a digital signal. The A / D converter 212 may be configured to be built into the imaging unit 211.

[0034] The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained arithmetic results. This processing allows TTL (through-the-lens) AF processing, AE (auto exposure) processing, EF (flash pre-flash) processing, etc. to be performed. Furthermore, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs TTL AWB (auto white balance) processing based on the obtained arithmetic results.

[0035] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, and image data to be displayed on the display unit 108 and the EVF 217. The memory (storage means) 215 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 215 also serves as a memory for displaying images (video memory).

[0036] The D / A converter 216 converts the image data for display stored in the memory 215 into an analog signal and supplies it to the display unit 108 or the EVF 217. Therefore, the image data for display written in the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display according to the analog signal from the D / A converter 216. The display unit 108 or the EVF 217 is, for example, an LCD or an organic EL display. A digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216, and the analog signal is sequentially transferred to and displayed on the display unit 108 or the EVF 217, thereby performing live view display.

[0037] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 executes a program recorded in the non-volatile memory 219 to realize each process of the flowchart described later. The system control unit 50 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, etc.

[0038] The camera 100 also has a system memory 218 , a non-volatile memory 219 , a system timer 220 , a communication unit 221 , an attitude detection unit 222 , and an eye proximity detection unit 118 .

[0039] As the system memory 218, for example, a RAM is used. In the system memory 218, constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are expanded. The nonvolatile memory 219 is an electrically erasable and recordable memory, and for example, an EEPROM is used as the nonvolatile memory 219. In the nonvolatile memory 219, constants and programs for the operation of the system control unit 50 are recorded. The programs here are programs for executing the flowcharts described later. The system timer 220 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0040] The communication unit 221 transmits and receives video signals and audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 221 can also connect to a wireless LAN (Local Area Network) or the Internet. The communication unit 221 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth (registered trademark) Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded in a recording medium 227, and can receive images and various other information from external devices.

[0041] The attitude detection unit 222 detects the attitude of the camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 is an image captured by holding the camera 100 horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image according to the detected attitude. For example, an acceleration sensor or a gyro sensor can be used for the attitude detection unit 222. It is also possible to detect the movement of the camera 100 (panning, tilting, lifting, whether it is stationary, etc.) using the attitude detection unit 222.

[0042] Eyepiece detection unit 118 can detect the approach of some object to eyepiece unit 116 (eyepiece finder 117). Eyepiece detection unit 118 can be, for example, an infrared proximity sensor. When an object approaches, infrared light projected from a light-projecting unit of eyepiece detection unit 118 is reflected by the object and received by a light-receiving unit of the infrared proximity sensor. The distance from eyepiece unit 116 to the object can be determined based on the amount of infrared light received. In this way, eyepiece detection unit 118 performs eyepiece detection, which detects the proximity of an object to eyepiece unit 116.

[0043] The eye contact detection unit 118 is an eye contact detection sensor that detects the approach (eye contact) and departure (eye separation) of an eye (object) to the eye piece unit 116. When an object is detected approaching within a predetermined distance from the eye piece unit 116 from a non-eye contact state (non-approaching state), it detects that the eye has been placed near the object. On the other hand, when an object that was detected as approaching from the eye contact state (approaching state) moves away by more than a predetermined distance, it detects that the eye has been removed. The threshold value for detecting the eye contact and the threshold value for detecting the eye separation may be different, for example, by providing a hysteresis. Furthermore, after detecting the eye contact, it is assumed that the eye contact state remains until the eye separation is detected. After detecting the eye separation, it is assumed that the eye contact state remains until the eye contact is detected.

[0044] The system control unit 50 switches between display (display state) / non-display (non-display state) of the display unit 108 and the EVF 217 depending on the state detected by the eyepiece detection unit 118. Specifically, when at least the shooting standby state is present and the display destination switching setting is automatic switching, the display destination is set to the display unit 108 when the eye is not placed in contact with the subject, the display is turned on, and the EVF 217 is hidden. Also, when the eye is placed in contact with the subject, the display destination is set to the EVF 217, the display is turned on, and the display unit 108 is hidden. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor, and other sensors may be used for the eyepiece detection unit 118 as long as they can detect a state that can be regarded as being in contact with the eyepiece.

[0045] The camera 100 also has an outside-finder display unit 107, an outside-finder display drive circuit 223, a power supply control unit 224, a power supply unit 225, a recording medium I / F 226, an operation unit 228, and the like.

[0046] The outside viewfinder display unit 107 is driven by an outside viewfinder display drive circuit 223, and displays various settings of the camera 100 such as the shutter speed and aperture. The power supply control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery level, etc. The power supply control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage to each unit including the recording medium 227 for the required period of time.

[0047] Power supply unit 225 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery or a Li battery, an AC adapter, etc. Recording medium I / F 226 is an interface with recording medium 227 such as a memory card or a hard disk. Recording medium 227 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, etc. Recording medium 227 may be detachable from camera 100, or may be built into camera 100.

[0048] The operation unit 228 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode switching switch 103, the touch panel 109, and other operation units 229. The other operation units 229 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction keys 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, and the touch bar 119.

[0049] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 is turned on when the shutter button 101 is pressed halfway (a shooting preparation instruction) during operation, and outputs a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing.

[0050] The second shutter switch 231 is turned on when the operation of the shutter button 101 is completed, that is, when the shutter button 101 is fully pressed (shooting instruction), and outputs a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processes, from reading out a signal from the imaging unit 211 to generating an image file including a shot image and writing it to the recording medium 227.

[0051] The mode changeover switch 103 switches the operation mode of the system control unit 50 to one of still image shooting mode, video shooting mode, playback mode, etc. Modes included in the still image shooting mode include auto shooting mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide shooting settings for different shooting scenes.

[0052] A user can directly switch to any one of the above-mentioned shooting modes by using the mode changeover switch 103. Alternatively, a user can once switch to a list screen of shooting modes by using the mode changeover switch 103, and then selectively switch to any one of the displayed modes by using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0053] Touch panel 109 is a touch sensor that detects various touch operations on the display surface of display unit 108 (operation surface of touch panel 109). Touch panel 109 and display unit 108 can be configured as one unit. For example, touch panel 109 is attached to the upper layer of the display surface of display unit 108 so that the light transmittance does not interfere with the display of display unit 108. Then, by associating input coordinates on touch panel 109 with display coordinates on the display surface of display unit 108, a GUI (Graphical User Interface) can be configured as if the user could directly operate the screen displayed on display unit 108.

[0054] The touch panel 109 can use any of various types such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, an optical sensor type, etc. Depending on the type, there are types that detect a touch by contact with the touch panel 109, and types that detect a touch by the approach of a finger or a pen to the touch panel 109, but any type may be used.

[0055] The system control unit 50 can detect the following operations or states on the touch panel 109.

[0056] A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down).

[0057] A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On).

[0058] The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move).

[0059] The finger or pen that has been touching the touch panel 109 is removed (released) from the touch panel 109, that is, the touch ends (hereinafter, referred to as "touch-up").

[0060] A state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0061] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on will also be detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. After it is detected that all fingers or pens that were touching have touched up, touch-off will occur.

[0062] These operations and states, as well as the position coordinates of the touch panel 109 touched by a finger or pen, are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding touch-move, the moving direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical component and horizontal component on the touch panel 109 based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed.

[0063] An operation in which a finger is touched on the touch panel 109, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 109 as if flicking. When a touch-move is detected over a predetermined distance or more at a predetermined speed or more, and a touch-up is detected immediately after that, it is determined that a flick has been performed (it can be determined that a flick has occurred following a sliding operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer to each other is called a pinch-in, and a touch operation in which the touch positions are moved away from each other is called a pinch-out. Pinch-out and pinch-in are collectively called a pinch operation (or simply pinch).

[0064] Fig. 4 is a configuration diagram of lens unit 300. Fig. 4 shows a state in which lens unit 300 is attached to camera 100. Note that, among the camera 100 shown in Fig. 4, the same components as those described with reference to Fig. 3 are denoted by the same reference numerals as in Fig. 3, and descriptions of those components will be omitted as appropriate.

[0065] The lens unit 300 is a type of interchangeable lens that can be attached to and detached from the camera 100. The lens unit 300 is a twin lens that can capture two images, a right image and a left image, which have a parallax between them. In this embodiment, the lens unit 300 has two optical systems, and each of the two optical systems can capture an image over a wide viewing angle range of approximately 180 degrees. Specifically, each of the two optical systems of the lens unit 300 can capture an image of a subject over a viewing field (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle). In other words, each of the two optical systems can capture an image over the range of the front hemisphere.

[0066] The lens unit 300 has a right-eye optical system 301R having a plurality of lenses and a reflecting mirror, a left-eye optical system 301L having a plurality of lenses and a reflecting mirror, a lens system control circuit 303, and an AF drive circuit 304. The left-eye optical system 301L is an example of a first optical system, and the right-eye optical system 301R is an example of a second optical system. However, this embodiment is not limited to this, and the right-eye optical system 301R may be the first optical system, and the left-eye optical system 301L may be the second optical system. The right-eye optical system 301R has a lens 302R arranged on the subject side, and the left-eye optical system 301L has a lens 302L arranged on the subject side. The lenses 302R and 302L face in the same direction, and their optical axes are approximately parallel.

[0067] The lens system control circuit 303 controls the AF drive circuit 304 and the like. The lens system control circuit 303 focuses on the subject by changing the positions of the right eye optical system 301R and the left eye optical system 301L via the AF drive circuit 304. In this embodiment, the AF drive circuit 304 controls the right eye optical system 301R and the left eye optical system 301L so that they operate in conjunction with each other. That is, the focus adjustment is adjusted for the entire lens unit 300, and the focus adjustment does not cause a defocus between the right image and the left image. Note that the defocus between the right image and the left image can be finely adjusted by an adjustment unit not shown in FIG. 4, and can be adjusted in advance before shooting begins.

[0068] The lens unit 300 is a twin lens (VR180 lens) for obtaining a VR180 image, which is one of the formats of VR (Virtual Reality) images that allow twin-eye stereoscopic vision. In this embodiment, the lens unit 300 has fisheye lenses capable of capturing a range of approximately 180 degrees in each of the right-eye optical system 301R and the left-eye optical system 301L. The range that can be captured by the lenses of each of the right-eye optical system 301R and the left-eye optical system 301L may be approximately 160 degrees, which is narrower than the range of 180 degrees. The lens unit 300 can form a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L on one or two image pickup elements of a camera to which the lens unit 300 is attached.

[0069] The lens unit 300 is attached to the camera 100 via the lens mount portion 305 and the camera mount portion 306 of the camera 100. In this manner, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 307 of the lens unit 300.

[0070] 5 is an explanatory diagram of a captured image 500 acquired by the camera 100 of this embodiment. In this embodiment, a right image formed via the right eye optical system 301R and a left image formed via the left eye optical system 301L are simultaneously (as a set) formed on the imaging unit 211 of the camera 100. That is, two optical images formed by the right eye optical system 301R and the left eye optical system 301L are formed as an image 501R and a left image 501L, respectively, on an imaging surface 502 of one imaging element.

[0071] The imaging unit 211 converts the formed subject image (optical signal) into an electrical signal. By using the lens unit 300 in this way, two images having a parallax from each other can be simultaneously acquired (as a set) from two locations (optical systems) of the right eye optical system 301R and the left eye optical system 301L. By dividing the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image in a range of approximately 180 degrees.

[0072] As described above, the imaging unit 211 photoelectrically converts the first optical image formed by the left-eye optical system (first optical system) 301L and outputs first image data (image for the left eye, left image). The imaging unit 211 also photoelectrically converts the second optical image formed by the right-eye optical system (second optical system) 301R and outputs second image data (image for the right eye, right image).

[0073] Next, the photographing process by the imaging system 10 of this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the photographing process. First, in step S601, the system control unit 50 of the camera 100 acquires a photographed image captured with the lens unit 300 attached. The image acquired in step S601 is like the photographed image 500 shown in Fig. 5, in which two images (left image 501L, right image 501R) having two parallaxes for the same field of view are formed on one imaging element.

[0074] Next, in step S602, the system control unit 50 determines a main subject by a main subject determination process based on the captured image 500 acquired in step S601. In this embodiment, the system control unit 50 determines the main subject based on the subject information (first subject information) in the left image 501L and the subject information (second subject information) in the right image 501R.

[0075] The first subject information includes, for example, information on whether or not a subject (first subject) is detected in the first image data, and information on a first evaluation value of the first subject. Similarly, the second subject information includes, for example, information on whether or not a subject (second subject) is detected in the second image data, and information on a second evaluation value of the second subject. Details of the main subject determination process will be described later.

[0076] Next, in step S603, the system control unit 50 displays a subject frame for the main subject determined in step S602 in the captured image 500 displayed on the display unit 108. This allows the user to recognize which subject in the captured image 500 is the subject (main subject) that is the target of AF processing.

[0077] For example, the system control unit 50 superimposes a rectangular frame indicating the main subject on the subject area in the captured image 500 displayed on the display unit 108. Alternatively, for example, assuming that the subject exists in both the left and right images (two images), a rectangular frame may be superimposed on the image in which the subject is not detected (one of the two images) at the same coordinate position as the coordinate position in the image in which the main subject is detected.

[0078] Next, in step S604, system control unit 50 executes autofocus processing (AF processing) on ​​the main subject determined in step S602. Note that in this embodiment, steps S603 and S604 are executed in series, but this is not limited to this. If the main subject has been determined in step S602, the processing of step S604 can be executed, so steps S603 and S604 may be executed in parallel.

[0079] Next, the main subject determination process (step S602 in FIG. 6) will be described in detail with reference to FIG. 7. FIG. 7 is a flowchart showing the main subject determination process. In the main subject determination process, one of right image 501R and left image 501L in captured image 500 shown in FIG. 5 is preferentially set as the main subject, and only when a subject cannot be detected in the priority image, a subject detected in the non-priority image is set as the main subject.

[0080] By taking advantage of the fact that two images, one on the left and one on the right, of the same field of view can be acquired, even if detection is not possible in one image, the other image can be used to increase the probability of detecting the same subject. In addition, the subject detection process can be made efficient by normally prioritizing the subject detected in the priority image, and only when detection fails, using the non-priority image. In addition, even if there is a slight focus position shift between the left and right lenses, one image is always prioritized, so fluctuations in the focus position when focusing on the same subject can be suppressed, and AF processing accuracy can be stabilized.

[0081] In this embodiment, a process will be described in which the left image 501L is a priority image and the right image 501R is a non-priority image, but the present invention is not limited to this. The right image 501R may be a priority image and the left image 501L may be a non-priority image, and similar processes can be applied.

[0082] First, in step S701, the system control unit 50 detects a subject (subject area) from the captured image acquired in step S601 in Fig. 6. At this point, subject detection processing is performed on both the left and right images regardless of whether the image is a priority image or a non-priority image. In addition, the number of subjects detected is not limited to one, and multiple subjects may be detected.

[0083] Next, in step S702, the system control unit 50 calculates an evaluation value for at least one object region detected in step S701. The evaluation value can be set (calculated) based on the degree of likelihood that the object region is considered to be a comparison object, for example, by comparing the object region with a comparison object registered in advance in the camera 100. Specifically, a higher evaluation value is set for an object region with a higher likelihood. However, the method of calculating the evaluation value is not limited to this, and various methods are applicable. In addition, the comparison object may be one or more objects automatically selected from people, animals, vehicles, etc., or may be selected in advance by the user. In addition, for example, the comparison object may be a subject detected in a previous frame during shooting.

[0084] In the following step S703 and thereafter, a process is performed to determine whether or not the subject detected in step S701 is a main subject, based on the evaluation value obtained in step S702 and a predetermined threshold value (threshold value TH1) that is set in advance in the camera 100.

[0085] First, in step S703, the system control unit 50 determines whether or not a subject (first subject) is detected in the left image (first image data), which is the priority image, and whether or not an evaluation value (first evaluation value) for the subject exceeds a threshold value (first threshold value) TH1.

[0086] If multiple subjects are detected in the left image, the evaluation value of the subject with the highest evaluation value obtained in step S702 is compared with a threshold value TH1. If it is determined that the evaluation value exceeds the threshold value TH1, the process proceeds to step S704. On the other hand, if it is determined that the evaluation value does not exceed the threshold value TH1, it is determined that no main subject is present in the left image, and the process proceeds to step S705.

[0087] In step S705, the system control unit 50 determines whether a subject (second subject) is detected in the right image (second image data) that is not the priority image, and whether the evaluation value (second evaluation value) for the subject exceeds a threshold value TH1. If multiple subjects are detected in the right image, the evaluation value of the subject with the highest evaluation value obtained in step S702 is compared with the threshold value TH1. If it is determined that the evaluation value exceeds the threshold value TH1, the process proceeds to step S704. On the other hand, if it is determined that the evaluation value does not exceed the threshold value TH1, it is determined that no main subject is present in the right image, and the process proceeds to step S706.

[0088] In step S704, the system control unit 50 determines the subject (first subject or second subject) selected in step S703 or step S704 as the main subject. The system control unit 50 then stores information such as the coordinates of the subject in the image and whether the subject is detected as the left image or the right image as main subject information in a storage unit such as memory 215, and uses this information to record in metadata in association with the image when saving the captured image. This makes it possible to use the subject information that has been subjected to AF processing during retouching after image capture, etc.

[0089] In step S706, since the subject is not detected in either the left image or the right image, the system control unit 50 determines that the main subject is not detected (main subject not detected).

[0090] According to this embodiment, even if a subject cannot be detected by detection processing using one of two left and right images acquired by imaging the same field of view, failure of detection processing of the same subject can be suppressed by performing detection processing using the other image.

[0091] <Second embodiment> Next, a second embodiment of the present invention will be described. In this embodiment, it is assumed that the same subject is to be tracked in previous and next frames during shooting, and a case will be described in which the determination method is changed in the main subject determination process according to the distance from the main subject detection coordinates in the previous frame. In this embodiment, the same reference numerals as in the first embodiment perform the same operation or process as in the first embodiment, and the description thereof will be omitted.

[0092] The main subject determination process in this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the main subject determination process in this embodiment.

[0093] In this embodiment, when a subject having an evaluation value greater than a threshold (first threshold) TH1 is detected in step S703 in the main subject determination process, a noise determination process in step S801 described later is added before determining the subject as the main subject. That is, when the system control unit 50 determines in step S703 that the first evaluation value for the first subject is greater than the threshold TH, the system control unit 50 determines in step S801 whether the first subject is noise or not.

[0094] In step S801, the system controller 50 performs a noise determination process. The noise determination process will be described in detail later.

[0095] Next, in step S802, system control unit 50 determines whether or not the subject determined in step S703 to have an evaluation value greater than threshold TH1 is a main subject based on the determination result of the noise determination process in step S801. That is, in step S802, the process proceeds to step S704 only if the subject is determined to be a main subject based on the determination result of the noise determination process in step S801.

[0096] On the other hand, if the subject is determined to be noise in step S801, the system control unit 50 excludes the subject determined to be noise from candidates for the main subject in step S802. If multiple other subjects are detected in the left image, the process is repeated from step S703 for the subject with the highest evaluation value among the remaining multiple subjects.

[0097] This type of processing can reduce cases where a subject that should not actually be tracked is mistakenly detected as the main subject, allowing for more accurate subject detection.

[0098] Next, the noise determination process in step S801 in Fig. 8 will be described in detail with reference to Fig. 9. Fig. 9 is a flowchart showing the noise determination process.

[0099] First, in step S901, the system control unit 50 determines how far the position (coordinates) on the image of the subject having an evaluation value greater than the threshold value TH1 in step S703 of Fig. 8 is detected from the position of the main subject detected in the previous frame. Then, the system control unit 50 determines whether the distance between the positions of the subject detected in each of the previous and next frames is within a threshold value (second threshold value) TH2. That is, the system control unit 50 determines whether the subject is noise or not based on the distance between the first position of the subject in the first frame (previous frame) of the left image and the second position of the subject in the second frame (following frame) after the first frame.

[0100] If it is determined that the distance is within the threshold TH2, the process proceeds to step S902. In step S902, the system control unit 50 determines that the subject (first subject) detected in step S703 is the main subject. On the other hand, if it is determined that the distance exceeds the threshold TH2, the process proceeds to steps S903 and S904 to verify whether the subject is noise.

[0101] In step S903, the system control unit 50 refers to the position (coordinates) in the left image of the subject detected in the left image, and acquires an image area of ​​the same position (same target) in the right image.

[0102] Next, in step S904, the system control unit 50 determines whether or not the subject detected in the left image exists in the image area in the right image acquired in step S903 (at coordinates corresponding to the coordinates of the subject in the left image). If it is determined in step S904 that the same subject exists at the same coordinates in the right image, it is determined that the subject detected in the left image is not noise, and the process proceeds to step S902.

[0103] On the other hand, if it is determined in step S904 that the same subject does not exist at the same coordinates in the right image, the process proceeds to step S905. In step S905, the system control unit 50 determines that the subject detected in the left image is not a subject to be tracked but is noise that has been detected by chance.

[0104] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0105] In each embodiment, when shooting using two optical systems arranged in parallel with each other, the results of two images, the left image and the right image, are used together to reduce the influence of sudden changes in position due to the subject's direction, angle, or quick movement. This makes it possible to suppress failures in subject detection and tracking or false detections. Therefore, according to each embodiment, it is possible to provide an imaging device, a control method for an imaging device, and a program that can perform more appropriate subject detection when using a lens unit having two optical systems arranged in parallel with each other.

[0106] The disclosure of each embodiment includes the following configurations and methods. (Configuration 1) an imaging means for photoelectrically converting a first optical image formed by a first optical system to output first image data, and for photoelectrically converting a second optical image formed by a second optical system arranged in parallel with the first optical system to output second image data; and a control means for determining a main subject based on first subject information in the first image data and second subject information in the second image data. (Configuration 2) 2. The imaging apparatus according to claim 1, wherein the control means performs autofocus processing on the main subject. (Configuration 3) the first object information includes information on whether a first object has been detected in the first image data and information on a first evaluation value of the first object; The imaging device described in configuration 1 or 2, characterized in that the second subject information includes information regarding whether a second subject is detected in the second image data, and information regarding a second evaluation value of the second subject. (Configuration 4) The control means If it is determined that the first evaluation value is greater than a first threshold value, it is determined that the first subject is present in the first image data; 4. The imaging device according to configuration 3, wherein if it is determined that the second evaluation value is greater than the first threshold value, it is determined that the second subject is present in the second image data. (Configuration 5) 5. The imaging apparatus according to claim 3, wherein the control means determines the first subject as the main subject when it is determined that the first evaluation value is greater than a first threshold value. (Configuration 6) The imaging device described in any one of configurations 3 to 5, characterized in that when the control means determines that the first evaluation value is smaller than a first threshold value, it determines whether or not the second evaluation value is larger than the first threshold value. (Configuration 7) 7. The imaging apparatus according to configuration 6, wherein the control means determines the second subject as the main subject when it is determined that the second evaluation value is greater than the first threshold value. (Configuration 8) The imaging device according to configuration 6 or 7, wherein the control means determines that the main subject has not been detected when it determines that the second evaluation value is smaller than the first threshold value. (Configuration 9) The imaging device according to configuration 3 or 4, wherein the control means determines whether the first subject is noise or not when it determines that the first evaluation value is greater than a first threshold value. (Configuration 10) The imaging device described in configuration 9, characterized in that the control means determines whether the first subject is the noise based on a distance between a first position of the first subject in a first frame of the first image data and the second position of the first subject in a second frame following the first frame. (Configuration 11) 11. The imaging apparatus according to configuration 10, wherein the control means determines that the first subject is the main subject when it is determined that the distance is smaller than the second threshold value. (Configuration 12) The imaging device described in configuration 10 or 11, characterized in that when the control means determines that the distance is greater than the second threshold, it determines whether the second subject is present at a position corresponding to the second position in the second image data. (Configuration 13) The control means If it is determined that the second object is present at the position corresponding to the second position in the second image data, the second object is determined to be the main object; An imaging device according to any one of configurations 10 to 12, characterized in that if it is determined that the second object does not exist at the position corresponding to the second position in the second image data, it is determined that the first object is the noise. (Configuration 14) 14. The imaging device according to any one of configurations 1 to 13, wherein the first image data and the second image data have a parallax with each other. (Configuration 15) The control means When it is determined that a plurality of subjects are present in the first image data, the subject having the highest evaluation value among the plurality of subjects is determined to be the first subject; An imaging device according to any one of configurations 1 to 14, characterized in that when it is determined that multiple subjects are present in the second image data, the subject having the highest evaluation value among the multiple subjects is determined to be the second subject. (Configuration 16) 16. The imaging apparatus according to any one of configurations 1 to 15, wherein the control means displays an area including the main subject that is to be subjected to autofocus processing on a display means. (Configuration 17) 17. The imaging apparatus according to any one of configurations 1 to 16, wherein the control means causes information about the main subject to be stored in the storage means in association with the first image data and the second image data. (Method 1) a step of photoelectrically converting a first optical image formed by a first optical system to output first image data, and photoelectrically converting a second optical image formed by a second optical system arranged in parallel with the first optical system to output second image data; determining a main subject based on first subject information in the first image data and second subject information in the second image data. (Configuration 18) A program for causing a computer to execute the imaging device control method according to Method 1.

[0107] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined. [Explanation of symbols]

[0108] 50 System control unit (control means) 100 Camera (imaging device) 211 Imaging unit (imaging means) 301L Left eye optical system (first optical system) 301R Right eye optical system (second optical system)

Claims

1. an imaging means for photoelectrically converting a first optical image formed by a first optical system to output first image data, and for photoelectrically converting a second optical image formed by a second optical system arranged in parallel with the first optical system to output second image data; and a control means for determining a main subject based on first subject information in the first image data and second subject information in the second image data.

2. 2. The image pickup apparatus according to claim 1, wherein the control means performs an autofocus process on the main subject.

3. the first object information includes information on whether a first object has been detected in the first image data and information on a first evaluation value of the first object; 2. The imaging device according to claim 1 , wherein the second subject information includes information regarding whether a second subject is detected in the second image data, and information regarding a second evaluation value of the second subject.

4. The control means If it is determined that the first evaluation value is greater than a first threshold value, it is determined that the first subject is present in the first image data; 4. The imaging apparatus according to claim 3, wherein when it is determined that the second evaluation value is greater than the first threshold value, it is determined that the second subject is present in the second image data.

5. 4. The imaging apparatus according to claim 3, wherein the control means determines the first subject as the main subject when it is determined that the first evaluation value is greater than a first threshold value.

6. 4. The imaging apparatus according to claim 3, wherein the control means, when determining that the first evaluation value is smaller than a first threshold value, determines whether or not the second evaluation value is larger than the first threshold value.

7. 7. The imaging apparatus according to claim 6, wherein the control means determines the second subject as the main subject when it is determined that the second evaluation value is greater than the first threshold value.

8. 7. The imaging apparatus according to claim 6, wherein the control means determines that the main subject has not been detected when it is determined that the second evaluation value is smaller than the first threshold value.

9. 4. The imaging apparatus according to claim 3, wherein the control means determines whether the first subject is noise or not when it is determined that the first evaluation value is greater than a first threshold value.

10. 10. The imaging device according to claim 9, wherein the control means determines whether the first subject is the noise based on a distance between a first position of the first subject in a first frame of the first image data and the second position of the first subject in a second frame following the first frame.

11. 11. The image pickup apparatus according to claim 10, wherein the control means determines the first object to be the main object when it is determined that the distance is smaller than the second threshold value.

12. The imaging device according to claim 10, characterized in that, when the control means determines that the distance is greater than the second threshold, it determines whether or not the second subject is present at a position corresponding to the second position in the second image data.

13. The control means If it is determined that the second object is present at the position corresponding to the second position in the second image data, the second object is determined to be the main object; 11. The imaging device according to claim 10, wherein when it is determined that the second object is not present at the position corresponding to the second position in the second image data, it is determined that the first object is the noise.

14. 2. The imaging device according to claim 1, wherein the first image data and the second image data have a parallax with each other.

15. The control means When it is determined that a plurality of subjects are present in the first image data, the subject having the highest evaluation value among the plurality of subjects is determined to be the first subject; 2. The imaging apparatus according to claim 1, wherein, when it is determined that a plurality of subjects are present in the second image data, the subject having the highest evaluation value among the plurality of subjects is determined to be the second subject.

16. 16. The imaging apparatus according to claim 1, wherein the control means displays an area including the main subject that is to be subjected to autofocus processing on a display means.

17. 16. The imaging apparatus according to claim 1, wherein the control means causes information about the main subject to be stored in a storage means in association with the first image data and the second image data.

18. a step of photoelectrically converting a first optical image formed by a first optical system to output first image data, and photoelectrically converting a second optical image formed by a second optical system arranged in parallel with the first optical system to output second image data; determining a main subject based on first subject information in the first image data and second subject information in the second image data.

19. A program causing a computer to execute the method for controlling an imaging apparatus according to claim 18.

Images