Control device, control method, and program

Abstract

To provide a control device, control method and program which can correctly and quickly perform an autofocus operation while, for example, reducing a lateral difference of autofocus accuracy suitable to imaging of an image that can be three-dimensionally viewed.SOLUTION: A control device 1000 comprises: first focus detection means 241 which detects a focus of a right eye optical system 301R; second focus detection means 242 which detects a focus of a left eye optical system 301L; first focus control means 311 which performs focus adjustment of the right eye optical system 301R; and second focus control means 312 which performs focus adjustment of the left eye optical system 301L. The first focus control means 311 and the second focus control means 312 switch between coordination control that performs the focus adjustment of the right eye optical system 301R and the focus adjustment of the left eye optical system 301L by a common focus adjustment amount and independent control that performs the focus adjustment of the right eye optical system 301R by the first focus adjustment amount and performs the focus adjustment of the left eye optical system 301L by the second focus adjustment amount.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] In recent years, imaging devices capable of capturing stereoscopic images have become widespread. For example, VR180 is a video format that enables binocular stereoscopic viewing. Devices such as VR180-compatible head-mounted displays (HMDs) can display images so that they can be viewed separately by the left and right eyes. This allows for the perception of a stereoscopic image. Furthermore, the range of the image displayed on the display (display range) can be changed in response to changes in the device's orientation. This allows for a highly immersive and realistic image display. Phase-difference focus detection (hereinafter referred to as "phase-difference AF") is known as an autofocus detection (AF) method for imaging devices. Phase-difference AF is an AF method often used in digital still cameras. Some phase-difference AF-compatible devices use an image sensor as a focus detection sensor. In a split-pupil phase-difference detection focus detection device, a pair of images is formed by a light beam passing through the exit pupil of an imaging optical system. Focus detection can then be performed by calculating the amount of defocus from the image shift amount between the images and a conversion coefficient. In the imaging device described in Patent Document 1, when the range of the angle of view is equal to or less than the threshold range for the base length between the first imaging optical system and the second imaging optical system, focus adjustment is performed for each image, whereas when the range of the angle of view is greater than the threshold range for the base length, the focus position of one image is made to follow the focus position of the other image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-191927 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the imaging device described in Patent Document 1 has limited conditions under which focus adjustment is possible using the phase difference method, making it difficult to achieve both accurate and rapid focus adjustment from a large blur state (a state with large defocus) to an in-focus state and reduction of the difference in autofocus accuracy between the left and right sides, which are required for the recent trend toward higher resolution.

[0005] The present invention aims to provide a control device, control method, and program that can perform autofocus operations accurately and quickly while reducing the difference in autofocus accuracy between the left and right sides, suitable for capturing stereoscopically visible images, for example. [Means for solving the problem]

[0006] In order to achieve the above object, a control device of the present invention comprises: a first focus detection means for detecting a focus of a first optical system based on first phase difference information corresponding to a defocus state of the first optical system through which light passing toward an image sensor passes; a second focus detection means, which is disposed at a position different from the first optical system, for detecting a focus of the second optical system based on second phase difference information corresponding to a defocus state of the second optical system through which light passing toward the image sensor passes; a first focus control means for adjusting the focus of the first optical system; and a second focus control means for adjusting the focus of the second optical system. the first focus detection means is capable of detecting a defocus direction of the first optical system, and the second focus detection means is capable of detecting a defocus direction of the second optical system, The first focus control means and the second focus control means switch between cooperative control in which a focus adjustment amount based on at least one of the first phase difference information and the second phase difference information is used as a common focus adjustment amount, and focus adjustment of the first optical system is performed using the common focus adjustment amount, and focus adjustment of the second optical system is performed using the common focus adjustment amount, and independent control in which focus adjustment of the first optical system is performed using a first focus adjustment amount based on the first phase difference information, and focus adjustment of the second optical system is performed using a second focus adjustment amount based on the second phase difference information, independently of the common focus adjustment amount. When the defocus direction of the first optical system and the defocus direction of the second optical system are different from each other and the defocus difference between the first optical system and the second optical system is equal to or greater than a predetermined value, the independent control is performed, and when the defocus difference between the first optical system and the second optical system is less than the predetermined value, the cooperative control is performed. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to perform an autofocus operation accurately and quickly while reducing the difference between the left and right in autofocus accuracy suitable for capturing, for example, stereoscopically viewable images. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the external configuration of a digital camera. [Figure 2] FIG. 1 is a diagram illustrating an example of the internal configuration of a digital camera. [Figure 3] FIG. 2 is a schematic diagram illustrating an example of the configuration of a lens unit. [Figure 4] FIG. 2 is a schematic diagram illustrating an example of a pixel array of an imaging element. [Figure 5] FIG. 10 is a schematic diagram showing the relationship between the defocus amount and the image shift amount. [Figure 6] 10 is a flowchart of a focus detection process (program) executed by the system control unit. [Figure 7] FIG. 2 is a schematic diagram showing the relationship between an imaging element and an optical system having a plurality of optical axes. [Figure 8] 10 is a graph showing the signal intensities (changes in light amount) of a first focus detection signal and a second focus detection signal, and a graph showing correction values ​​for matching the intensities of the first focus detection signal and the second focus detection signal. [Figure 9] 10 is a flowchart illustrating a process for performing autofocus driving under cooperative control. [Figure 10] 10 is a flowchart for switching autofocus driving between cooperative control and independent control. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments. Furthermore, in this embodiment, a case will be described in which the control device of the present invention is built into a digital camera (imaging device).

[0010] <Overall structure> FIG. 1 shows an example of the external configuration of a digital camera, with FIG. 1(a) being a perspective view of the digital camera as seen from the front side, and FIG. 1(b) being a perspective view of the digital camera as seen from the back side. As shown in FIG. 1(a), a digital camera (hereinafter referred to as "camera") 100 has a shutter button 101, a power switch 102, and a mode selector switch 103 arranged on the upper side. The camera 100 also has a main electronic dial 104, a sub electronic dial 105, a video button 106, and an extra-viewfinder display 107. The shutter button 101 is an operation unit used to prepare for shooting or to issue shooting instructions. The power switch 102 is an operation unit used to turn the camera 100 on and off. The mode selector switch 103 is an operation unit used to switch between various modes, such as a shooting mode. The main electronic dial 104 is a rotary operation unit used to change settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation unit for moving the selection frame (cursor), advancing images, etc. The video button 106 is an operation unit for issuing instructions to start or stop video shooting (recording). The outside viewfinder display unit 107 displays various setting values ​​such as shutter speed and aperture.

[0011] As shown in FIG. 1B, the camera 100 has a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, and a menu button 115, which are arranged on the rear side. The camera 100 also has an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar 119. The display unit 108 displays images and various information. The touch panel 109 is an operation unit (touch sensor) that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are an operation unit consisting of keys (four-way keys) that can be pressed up, down, left, and right. The directional keys 110 can perform operations depending on the position where the directional keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed to confirm a selection item. The AE lock button 112 is an operation unit that is pressed to fix the exposure state in a shooting standby state. The enlargement button 113 is an operation unit for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. Operating the main electronic dial 104 while the enlargement mode is on enlarges or reduces the live view image (LV image). The enlargement button 113 is also used to enlarge a playback image or increase the magnification in playback mode. The playback button 114 is an operation unit for switching between the shooting mode and playback mode. Pressing the playback button 114 in the shooting mode switches to playback mode, and the most recent image recorded on the recording medium 228 can be displayed on the display unit 108. The menu button 115 is an operation unit that is pressed to display a menu screen on the display unit 108 that allows various settings to be made. The user can intuitively make various settings using the menu screen displayed on the display unit 108, the direction keys 110, and the SET button 111. The eyepiece 116 is a portion for placing the eye on an eyepiece finder (a peer-type finder) 117. The user can view an image displayed on an internal EVF (Electronic View Finder) 217 ​​(described later) through the eyepiece 116. The eyepiece detection unit 118 is a sensor that detects whether the user places his / her eye close to the eyepiece 116.The touch bar 119 is a line-shaped touch operation unit (line touch sensor) that can accept touch operations. The touch bar 119 is positioned so that the user can place the thumb of their right hand on it when they hold the grip unit 120 with the little finger, ring finger, and middle finger of their right hand and place the index finger on the shutter button 101. This allows the touch bar 119 to be operated while the user places their eye on the eyepiece unit 116, looks through the eyepiece finder 117, and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can also accept tap operations (touching and then releasing without moving within a predetermined period of time) and slide operations (touching and then moving the touched position), etc. The touch bar 119 is an operation unit different from the touch panel 109 and does not have a display function. The touch bar 119 in this embodiment is a multi-function bar and functions as, for example, an M-Fn bar.

[0012] As shown in FIGS. 1(a) and 1(b), the camera 100 has a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, and a communication terminal 124. The grip section 120 is a holding section shaped to be easily gripped with the user's right hand when holding the camera 100. When the user holds the camera 100 by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned so that they can be operated with the index finger of the right hand. In addition, when holding the camera 100 in a similar manner, the sub electronic dial 105 and the touch bar 119 are positioned so that they can be operated with the thumb of the right hand. The thumb rest section 121 is located in a thumb standby position on the back side of the camera 100, where the thumb of the right hand holding the grip section 120 can be easily placed when not operating any of the operation sections. The thumb rest section 121 is made of a rubber member or the like to enhance holding strength (grip feeling). Terminal cover 122 protects connectors such as connection cables that connect camera 100 to external devices. Lid 123 protects recording medium 228 and the slot by closing the slot for storing recording medium 228. Communication terminal 124 is a terminal for communicating with lens unit 200, which is detachable from camera 100 and will be described later.

[0013] <Internal structure of the camera> FIG. 2 is a diagram showing an example of the internal configuration of a digital camera. As shown in FIG. 2, a lens unit 200 is attached to the camera 100. First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that can be attached to and detached from the camera 100, and in the configuration shown in FIG. 2, it is a single lens. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, and a communication terminal 206. The aperture 201 is configured so that its aperture diameter is adjustable. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit 218, which will be described later. The lens system control circuit 205 controls the aperture 201 via an aperture drive circuit 203, and adjusts the focus by displacing the position of the lens 202 via an AF drive circuit 204. The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, the lens system control circuit 205 is capable of communicating with the camera 100 via a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100.

[0014] Next, the camera 100 will be described. As shown in FIG. 2, the camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, and a system control unit 218. The shutter 210 is a focal plane shutter that can control the exposure time of the imaging unit 211 based on instructions from the system control unit 218. The imaging unit 211 is an imaging element (image sensor) formed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. In this embodiment, the imaging unit 211 preferably includes an imaging surface phase difference sensor that outputs information about the defocus amount to the system control unit 218. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined arithmetic processing using the captured image data, and the system control unit 218 performs exposure control and distance measurement control based on the obtained arithmetic results. This processing results in TTL (through-the-lens) AF processing, AE (auto-exposure) processing, EF (pre-flash) processing, and other processing. The image processing unit 214 also performs predetermined arithmetic processing using the captured image data, and performs TTL AWB (auto-white balance) processing based on the obtained arithmetic results. Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without passing through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data to be displayed on the display unit 108 and EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of video and audio. The memory 215 also serves as a memory for image display (video memory).

[0015] The D / A converter 216 converts image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 or the EVF 217. As a result, the display image data written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The EVF 217 performs display according to the analog signal from the D / A converter 216 (the same applies to the display unit 108). The EVF 217 is, for example, an LCD or organic EL display. In the camera 1, the digital signal that is A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216. This signal is then sequentially transferred to and displayed on the EVF 217, thereby performing a live view display on the EVF 217 (the same applies to the display unit 108). The system control unit 218 is a computer including at least one processor and / or at least one circuit. That is, the system control unit 218 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 218 constitutes a part of the control device 1000 and controls the entire camera 100. The system control unit 218 executes various programs recorded in the nonvolatile memory 220. Examples of these programs include a program for causing the system control unit 218 to execute the operation of each unit and each means of the control device 1000 (the control method of the present invention). The system control unit 218 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0016] The camera 100 also includes a system memory 219, a nonvolatile memory 220, a system timer 221, a communication unit 222, and an orientation detection unit 223. The system memory 219 may be, for example, a RAM. Constants and variables for the operation of the system control unit 218, programs read from the nonvolatile memory 220, and the like are loaded into the system memory 219. The nonvolatile memory 220 is an electrically erasable and recordable memory, for example, an EEPROM. Constants for the operation of the system control unit 218, the above-mentioned programs, and the like are stored in the nonvolatile memory 220. The system timer 221 is a timing unit that measures the time used for various controls and the time of an internal clock. The communication unit 222 transmits and receives video signals and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 222 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 222 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 222 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 228, and can receive image data and various other information from external devices. The orientation detection unit 223 detects the orientation of the camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 223, it is possible to determine whether an image captured by the imaging unit 211 was captured with the camera 100 held horizontally or vertically. The system control unit 218 can add orientation information corresponding to the orientation detected by the orientation detection unit 223 to the image file of the image captured by the imaging unit 211, or rotate and record the image. For example, an acceleration sensor or a gyro sensor can be used as the orientation detection unit 223. The orientation detection unit 223 can also detect movement of the camera 100 (panning, tilting, lifting, whether the camera is stationary, etc.).

[0017] As described above, the camera 100 includes an eyepiece detection unit 118. The eyepiece detection unit 118 can detect the approach of an object, such as the user's eye, to the eyepiece 116 of the eyepiece finder 117, which incorporates the EVF 217. The eyepiece detection unit 118 can be, for example, an infrared proximity sensor or other sensors. If the eyepiece detection unit 118 is configured as an infrared proximity sensor, when an object approaches, infrared light emitted from the light-emitting unit of the eyepiece detection unit 118 is reflected by the object and received by the light-receiving unit. The distance from the eyepiece 116 to the object is determined based on the amount of received infrared light. In this way, the eyepiece detection unit 118 performs eyepiece detection, detecting the proximity of the object to the eyepiece 116. Therefore, the eyepiece detection unit 118 can detect the approach (approach) and departure (away) of the eye (object) from the eyepiece 116 of the eyepiece finder 117. The eye-contact detection unit 118 detects that an eye has been placed in contact with the eyepiece 116 when an object approaching within a predetermined distance from the eyepiece 116 is detected from a non-eye-contact state (non-approach state). The eye-contact detection unit 118 also detects that an eye has been moved away from the eyepiece 116 when the object that was detected as being close to the eyepiece 116 moves away from the eyepiece 116 by a predetermined distance or more from the eyepiece 116. The threshold for detecting eye-contact and the threshold for detecting eye-movement may be different, for example, using hysteresis. After detecting eye-contact, the eye-contact detection unit 118 remains in the eye-contact state until it detects eye-movement. After detecting eye-movement, the system control unit 218 remains in the non-eye-contact state until it detects eye-contact again. The system control unit 218 switches the display unit 108 and the EVF 217 between on (display state) and off (non-display state) depending on the state detected by the eye-contact detection unit 118. Specifically, when the camera is at least in a standby state for shooting and the display destination switching setting is automatic switching, when the camera is not in eye contact, the display unit 108 is set as the display destination and the display is turned on, and the EVF 217 is not displayed. When the camera is in eye contact, the display unit 108 is set as the display destination and the display is turned on, and the display unit 108 is not displayed.

[0018] The camera 100 also includes an extra-viewfinder display unit 107, an extra-viewfinder display drive circuit 224, a power supply control unit 225, a power supply unit 226, a recording medium I / F 227, a recording medium 228, and an operation unit 229. The extra-viewfinder display unit 107 displays settings for imaging conditions such as shutter speed and aperture via the extra-viewfinder display drive circuit 224. The power supply control unit 225 is composed of a battery detection circuit, a DC-DC converter, a switch circuit, etc., and detects whether a battery is installed, the battery type, and the remaining battery power. Based on the detection results of the power supply control unit 225 and instructions from the system control unit 218, the power supply control unit 225 controls the DC-DC converter to supply the required voltage for the required period to each unit, including the recording medium 228. The power supply unit 226 may be a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The recording medium I / F 227 is an interface with the recording medium 228, such as a memory card or a hard disk. The recording medium 228 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 228 may be detachable from or fixed to the camera 1. The operation unit 229 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 218. The operation unit 229 includes the shutter button 101, the power switch 102, the mode selector switch 103, the touch panel 109, and other operation units 230. The other operation units 230 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction keys 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, the touch bar 119, and the like.

[0019] As described above, the camera 100 has a mode selector switch 103. The mode selector switch 103 switches the operation mode of the system control unit 218 to one of still image capture mode, video capture mode, playback mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). Other modes include various scene modes and custom modes that provide capture settings for different capture scenes. The user can directly switch to one of the above-mentioned capture modes using the mode selector switch 103. After using the mode selector switch 103 to switch to a list screen of capture modes, the user can selectively switch to one of the displayed modes using the operation unit 229.

[0020] The shutter button 101 also has a first shutter switch 231 and a second shutter switch 232. The first shutter switch 231 is turned on when the shutter button 101 is pressed halfway (a shooting preparation instruction) during operation, and generates a first shutter switch signal SW1. The system control unit 218 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 232 is turned on when the shutter button 101 is pressed fully (a shooting instruction) after completion of operation, and generates a second shutter switch signal SW2. The system control unit 218 starts a series of shooting processing, from reading out a signal from the imaging unit 211 to generating an image file including the captured image and writing it to the recording medium 228, in response to the second shutter switch signal SW2.

[0021] As described above, the camera 100 has a touch panel 109. In this embodiment, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that, for example, its light transmittance does not interfere with the display of the display unit 108. As a result, the touch panel 109 is configured integrally with the display unit 108. Furthermore, input coordinates on the touch panel 109 are associated with display coordinates on the display surface of the display unit 108. This configures a GUI (Graphical User Interface) that makes it appear as if the user can directly operate the screen displayed on the display unit 108. The touch panel 109 may be of various types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, or an optical sensor type. Furthermore, some types detect a touch on the touch panel 109 when a contact is made with the touch panel 109, and others detect a touch on the touch panel 109 when a finger or a pen approaches the touch panel 109. The system control unit 218 can detect the following operations and states on the touch panel 109. A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as "touch-down"). A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as "Touch-On"). The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as "Touch-Move"). The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the end of the touch (hereinafter referred to as "touch-up"). A state in which nothing is touching the touch panel 109 (hereinafter referred to as "Touch-Off").

[0022] On the touch panel 109, when a touch-down is detected, a touch-on is also detected at the same time. After a touch-down, a touch-on is normally detected unless a touch-up is detected. When a touch-move is detected, a touch-on is also detected at the same time. Even if a touch-on is detected, a touch-move is not detected unless the touch position moves. After it is detected that all fingers or pens that were touching the panel have touched up, a touch-off occurs. These operations and states, as well as the position coordinates of the fingers or pens touching the touch panel 109, are notified to the system control unit 218 via the internal bus. The system control unit 218 determines what type of operation (touch operation) was performed on the touch panel 109 based on the notified information. In a touch-move, the direction of movement of the finger or pen moving on the touch panel 109 is also determined for each vertical and horizontal component on the touch panel 109 based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation was performed. Furthermore, an operation in which a finger is touched to the touch panel 109, quickly moved a certain distance, and then released is called a "flick." In other words, a flick is an operation in which a finger is quickly traced across the touch panel 109 as if flicking. When a touch-move is detected over a predetermined distance or more at a predetermined speed or more, and a touch-up is then detected, it is determined that a flick has been performed. In other words, it is determined that a flick has occurred following a slide operation. Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called a "pinch in," and a touch operation in which the touch positions are moved farther apart is called a "pinch out." Pinch in and pinch out are collectively called a "pinch operation (or simply pinch)."

[0023] <Lens unit configuration> FIG. 3 is a schematic diagram showing an example of the configuration of a lens unit. Lens unit 300 shown in FIG. 3 is attached to camera 100. Lens unit 300 is a type of interchangeable lens that can be attached to and detached from camera 100. Unlike single-lens lens unit 200, lens unit 300 is a twin-lens lens that can capture images with parallax between left and right images. In this embodiment, lens unit 300 has two optical systems, each with a wide viewing angle of approximately 180 degrees, and can capture images of the front hemisphere. Specifically, the two optical systems of lens unit 300 can each capture an object within a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation / depression angle, pitch angle).

[0024] The lens unit 300 includes a right-eye optical system (first optical system) 301R, a left-eye optical system (second optical system) 301L arranged at a different position (adjacent to) the right-eye optical system 301R, a lens system control circuit 303, and a focus ring (not shown). The right-eye optical system 301R and the left-eye optical system 301L each include at least one lens and a reflecting mirror, through which light traveling toward the imaging unit 211 passes. In this embodiment, the right-eye optical system 301R and the left-eye optical system 301L each include a plurality of lenses. The lens 302R located on the subject side of the right-eye optical system 301R and the lens 302L located on the subject side of the left-eye optical system 301L face in the same direction, and their optical axes are approximately parallel. The lens unit 300 in this embodiment is a lens for capturing images in a so-called "VR180" format, which is a VR image format that allows binocular stereoscopic viewing. In this case, the lens unit 300 has a fisheye lens that enables the right-eye optical system 301R and the left-eye optical system 301L to each capture an image in a range of approximately 180 degrees. Note that the lens unit 300 only needs to be a lens that can capture an image in a wide viewing angle range of approximately 160 degrees, which is narrower than the 180-degree range, as long as the right-eye optical system 301R and the left-eye optical system 301L can each capture an image that can be displayed in two-eye VR as VR180. The lens unit 300 can also form, on the imaging element of the imaging section 211, a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L, which has parallax from the right image.

[0025] Although not shown, a first focus control unit 311 that performs focus adjustment of the right-eye optical system 301R (first focus control step) and a second focus control unit 312 that performs focus adjustment of the left-eye optical system 301L (second focus control step) are also provided. The first focus control unit 311 and the second focus control unit 312 constitute part of the control device 1000, and in this embodiment, for example, the lens system control circuit 303 functions as each focus control unit. Note that the system control unit 218 may also function as each focus control unit. The first focus control unit 311 controls the driving of the lens of the right-eye optical system 301R based on the focus detection result calculated by the system control unit 218. This allows the focus adjustment of the right-eye optical system 301R. Similarly, the second focus control unit 312 controls the driving of the lens of the left-eye optical system 301L based on the focus detection result calculated by the system control unit 218. This allows the focus adjustment of the left-eye optical system 301L.

[0026] Furthermore, the lens unit 300 has a lens mount section 304. The lens mount section 304 is a section that is connected to a camera mount section 305 of the camera 100. This connection places the lens unit 300 in an attached state, in which it is attached to the camera 100. Furthermore, in the attached state, the system control section 218 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0027] As described above, in this embodiment, the right image formed via the right-eye optical system 301R and the left image formed via the left-eye optical system 301L have parallax, and are imaged side by side on the imaging unit 211. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on one imaging element. The imaging unit 211 converts the imaged subject image (optical signal) into an analog electrical signal. In this way, by using the lens unit 300, two images with parallax can be simultaneously (collectively) acquired from two locations (optical systems), the right-eye optical system 301R and the left-eye optical system 301L. Furthermore, by dividing the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image with a range of approximately 180 degrees, known as VR180.

[0028] Here, a "VR image" refers to an image that can be displayed in VR. VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera) and panoramic images having a video range (effective video range) wider than the maximum display range of the display unit 108. VR images also include still images, videos, and live images (images acquired from the camera 1 in almost real time). VR images have a video range (effective video range) of up to a field of view of 360 degrees horizontally and 360 degrees vertically. VR images also include images that have a wider angle of view than the angle of view that can be captured by the camera 1 or a video range wider than the maximum display range of the display unit 108, even if the field of view is less than 360 degrees horizontally or vertically. An image captured by the camera 100 using the lens unit 300 is a type of VR image. VR images can be displayed in VR by, for example, setting the display mode of a display device (a display device that can display VR images), such as an HMD (head-mounted display), to "VR view." By displaying a VR image with a 360-degree angle of view and changing the orientation of the display device left and right (horizontal rotation direction), the user can view seamless, omnidirectional images left and right.

[0029] Here, "VR display (VR view)" refers to a display method (display mode) that displays a VR image with a field of view that corresponds to the orientation of the display device, allowing for a change in display range. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by mapping a VR image onto a virtual sphere (deformation that corrects distortion). VR display also includes "two-eye VR display (two-eye VR view)," which displays a VR image for the left eye and a VR image for the right eye side by side on the left and right sides of a virtual sphere by mapping each image onto the virtual sphere. Stereoscopic viewing is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax. Regardless of the VR display, for example, when a user wears an HMD (head-mounted display) as a display device, an image with a field of view that corresponds to the orientation of the user's face is displayed. For example, a VR image with a field of view centered at 0 degrees left and right (a specific direction, e.g., north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time can be displayed. Then, if the orientation of the display device is flipped from this state (for example, by changing the display surface from facing south to facing north), the display range of the same VR image is changed to an image with a field of view centered at 180 degrees left and right (the opposite direction, e.g., south) and 90 degrees up and down. In other words, when the user wears the HMD and turns their face from north to south (i.e., turns around), the image displayed on the HMD also changes from a north image to a south image. By displaying the VR image in this way, the user visually feels as if they are inside the VR image (in the VR space). Note that the method of displaying the VR image is not limited to changing the orientation of the display device. For example, the display range may be moved (scrolled) in response to operations such as a touch panel or directional keys. Furthermore, during VR display (in the "VR view" display mode), in addition to changing the display range due to changes in orientation, the display range may also be changed in response to touch-move on the touch panel, dragging with a mouse, pressing directional keys, etc.

[0030] The VR image captured using the lens unit 300 of this embodiment is a VR180 image capturing a range of approximately 180 degrees forward, and does not contain any image in a range of approximately 180 degrees backward. If such a VR180 image is displayed in VR and the position of the display device is changed to the side where no image exists, a blank area will be displayed.

[0031] <Configuration of the image sensor in the imaging unit> FIG. 4 is a schematic diagram showing an example of a pixel array of an image sensor. In FIG. 4, the pixel array of a two-dimensional CMOS sensor used as the image sensor in the image capturing unit 211 is shown as a 4-column by 4-row area of ​​imaging pixels (an 8-column by 4-row area of ​​focus detection pixel array). In this embodiment, the pixel group 400 is assumed to be composed of 2 columns by 2 rows of pixels and covered with color filters in a Bayer array. As shown in FIG. 4, the pixel group 400 has a pixel 400R having a spectral sensitivity of R (red) at the upper left position, a pixel 400G having a spectral sensitivity of G (green) at the upper right and lower left positions, and a pixel 400B having a spectral sensitivity of B (blue) at the lower right position. Furthermore, since the image sensor performs focus detection using an image-surface phase difference method, each of the pixels 400R, 400G, and 400B has a plurality of photodiodes (photoelectric conversion units) for one microlens 401. In this embodiment, pixel 400R, pixel 400G, and pixel 400B are each configured with a first photodiode 402 and a second photodiode 403 arranged in 2 columns and 1 row. The image sensor (image capturing unit 211) can acquire an image capturing signal and a focusing signal by arranging a large number of pixel groups 400, each of which is made up of 2 columns and 2 rows of pixels (4 columns and 2 rows of photodiodes) shown in FIG. 4, on the imaging surface.

[0032] Furthermore, in pixel 400R (similarly for pixel 400G and pixel 400B), a light beam is separated by a microlens 401 and focused on a first photodiode 402 and a second photodiode 403. A signal (signal A+B) obtained by adding together the signals from the first photodiode 402 and the second photodiode 403 is used as an imaging signal. A pair of focus detection signals (image signals A and B) read out from each of the first photodiode 402 and the second photodiode 403 are used as a focusing signal. Note that the imaging signal and the focusing signal may be read out separately, but in consideration of the processing load, the following may also be used, for example. The imaging signal (signal A+B) and a focusing signal (e.g., signal A) from either the first photodiode 402 or the second photodiode 403 may be read out, and the other focusing signal (e.g., signal B) may be obtained by reading out the imaging signal (signal A+B) and a focusing signal (e.g., signal A) from either the first photodiode 402 or the second photodiode 403 and calculating the difference.

[0033] In this embodiment, the first photodiode 402 and the second photodiode 403 are provided for each microlens 401. However, the number of photodiodes is not limited to two and may be, for example, two or more. The image sensor may also have a plurality of pixels with different aperture positions for the light receiving portion relative to the microlens 401. That is, the image sensor may be configured to obtain two signals for phase difference detection, such as an A image signal and a B image signal. In the configuration shown in FIG. 4, the pixel 400R, the pixel 400G, and the pixel 400B each have the first photodiode 402 and the second photodiode 403, but this is not limiting. For example, focus detection pixels may be provided discretely within the normal pixels that make up the image sensor.

[0034] <Relationship between defocus amount and image shift amount> FIG. 5 is a schematic diagram showing the relationship between the defocus amount and the image shift amount. Hereinafter, with reference to FIG. 5, the relationship between the defocus amount calculated from a pair of signals (image A signal and image B signal) acquired by an image sensor and the image shift amount will be described. Here, as an example, an optical system in which the imaging center and the optical axis center coincide will be described. As shown in FIG. 5, an image sensor (not shown) is disposed on an image sensor plane 500, and the exit pupil of the imaging optical system is divided into a first pupil partial region 503 and a second pupil partial region 504. The defocus amount d is the distance from the imaging position of the subject to the image sensor plane 500, and its magnitude is represented by |d|. Furthermore, the defocus amount d is represented by a negative sign (d<0) in a front-focus state in which the imaging position of the subject is closer to the subject than the image sensor plane 500, and a positive sign (d>0) in a back-focus state in which the imaging position of the subject is on the opposite side of the subject than the image sensor plane 500. The front-focus state (d<0) and the back-focus state (d>0) are combined to form a defocused state (|d|>0). The defocus amount d is expressed as d=0 in a focused state where the image of the subject is located at the imaging plane 500 (focus position). In the example shown in FIG. 5 , the subject 501 is in a focused state (d=0), and the subject 502 is in a front-focused state (d<0). In the front-focused state, the light beams from the subject 502 that pass through the first pupil partial region 503 are first collected and then spread over a width Γ2 centered on the center of gravity G2 of the light beams on the imaging plane 500. Similarly, the light beams from the subject 502 that pass through the second pupil partial region 504 are first collected and then spread over a width Γ1 centered on the center of gravity G1 of the light beams on the imaging plane 500. Due to this spread, a blurred image appears on the imaging plane 500 in the front-focused state. The blurred image is then received by the first photodiode 402 of the image sensor to generate an A image signal (first focus detection signal), and is also received by the second photodiode 403 to generate a B image signal (second focus detection signal). The A and B image signals are recorded as subject images of the subject 502 on the image sensing surface 500 that are blurred by widths Γ1 and Γ2. The widths Γ1 and Γ2 tend to increase as the magnitude |d| of the defocus amount d increases. Furthermore, the magnitude |p| of the image shift amount p, i.e., the difference between the center of gravity positions G1 and G2, also tends to increase as the magnitude |d| of the defocus amount d increases.The same tendency occurs in the back-focus state. Camera 1 can calculate the defocus amount d based on the image shift amount p and a conversion coefficient K for converting the image shift amount p, which has been calculated in advance, into the defocus amount d. The conversion coefficient K is a value that depends on the angle of incidence, F-number, and optical axis position of the imaging optical system.

[0035] <Defocus calculation flow> FIG. 6 is a flowchart of the focus detection process (program) executed by the system control unit. The right-eye optical system 301R and the left-eye optical system 301L have different focus positions due to, for example, mechanical factors, but the focus detection process for each optical system is the same. Therefore, the focus detection process will be described below without distinguishing between the right-eye optical system 301R and the left-eye optical system 301L. In this embodiment, the first photodiode 402 collects light reception signals to generate a first focus detection signal, and the second photodiode 403 collects light reception signals to generate a second focus detection signal. Therefore, the imaging unit 211 functions as a first focus detection unit 241 that detects the focus of the right-eye optical system 301R (first focus detection step) and as a second focus detection unit 242 that detects the focus of the left-eye optical system 301L (second focus detection step). The imaging unit 211, which functions as such a focus detection unit, constitutes a part of the control device 1000. Furthermore, both the first and second focus detection signals use a signal Y calculated by adding together the outputs of four pixels: green (G), red (R), blue (b), and green (G). Phase-difference AF detects the amount of defocus d from the amount of image shift p of the two focus detection signals Y.

[0036] In step S601, the system control unit 218 operates the imaging unit 211 to generate a first focus detection signal A from the light reception signal of the first focus detection pixel in the focus detection area and a second focus detection signal B from the light reception signal of the second focus detection pixel in the focus detection area. In step S602, the system control unit 218 performs pixel addition processing in the column direction to reduce the signal data amount for each focus detection signal, and then performs addition processing of RGB signals to generate a Y signal. Here, these two addition processes are collectively referred to as "pixel addition processing." When the number of pixel additions is two, the pixel pitch is doubled, so the Nyquist frequency is half that when no additions are performed. When the number of pixel additions is three, the pixel pitch is tripled, so the Nyquist frequency is one-third that when no additions are performed. In step S603, the system control unit 218 performs shading correction processing (optical correction processing) on ​​the first focus detection signal and the second focus detection signal to equalize the signal intensities of both signals. The shading correction value is a value that depends on the angle of incidence, F-number, and optical axis position of the imaging optical system. In step S604, the system control unit 218 performs band-pass filtering having a specific pass frequency band on the first focus detection signal and the second focus detection signal to improve correlation (degree of signal coincidence) and increase focus detection accuracy. The band-pass filter is not particularly limited, and examples include a differential filter such as (1, 4, 4, 4, 0, -4, -4, -4, -1) that cuts out DC components and extracts edges, and an additive filter such as (1 / 2, 1) that suppresses high-frequency noise components.

[0037] In step S605, the system control unit 218 performs shift processing to relatively shift the first and second focus detection signals after band-pass filtering in the pupil division direction, and calculates a correlation amount that represents the degree of match between the signals. Here, the k-th first focus detection signal A after band-pass filtering is defined as the first focus detection signal A(k), the second focus detection signal B is defined as the second focus detection signal B(k), and the range of number k corresponding to the focus detection area is defined as W. Furthermore, where the shift amount due to the shift processing is defined as s and the shift range of shift amount s is defined as Γ, the correlation amount COR is calculated using the following equation (1):

[0038]

number

[0039] Through shift processing, the kth first focus detection signal A(k) and the ksth second focus detection signal B(ks) are matched and subtracted to generate a shift subtraction signal. The absolute value of this shift subtraction signal is then calculated, and the sum of the number k is taken within the range W corresponding to the focus detection area to calculate the correlation amount COR(s). If necessary, the correlation amounts COR calculated for each row may be added across multiple rows for each shift amount s. Furthermore, when the correlation amount COR(s) is calculated, the reliability of the defocus amount d can be evaluated by checking the amount of change, peak-bottom values, etc.

[0040] In step S606, the system control unit 218 performs sub-pixel calculations based on the correlation amount COR to calculate the real-valued shift amount s that minimizes the correlation amount COR, and sets this as the image shift amount p. Then, the image shift amount p is multiplied by a conversion coefficient K to detect the defocus amount d. Note that the reliability of the defocus amount d can also be evaluated based on the magnitude of the conversion coefficient K. Furthermore, in the process of calculating the defocus amount d in phase difference autofocus, signal correction and conversion coefficients according to the lens unit 300 are used. The adjustment value used in the process of calculating the focus detection is set as the focus detection adjustment value, but the signal correction amount and conversion coefficient of the focus detection adjustment value depend on the angle of incidence on the image sensor and the state of vignetting.

[0041] In an interchangeable lens camera, an imaging optical system having a single optical axis centered on one image sensor is typically attached to one image sensor. However, when a twin lens is attached, it may be difficult to calculate appropriate signal correction amounts and conversion coefficients, potentially reducing the accuracy of phase-difference AF. Therefore, camera 1 is configured to reduce such problems. This configuration and operation are described below. While this embodiment describes two focus adjustments, the number of focus adjustment targets is not limited to two and may be, for example, three or more.

[0042] <About light intensity correction> Figure 7 is a schematic diagram showing the relationship between an image sensor and an optical system with multiple optical axes. In Figure 7, the imaging optical system has one optical axis on each side of the center of the image sensor. The image circle for each optical axis is approximately half that of the image sensor.

[0043] FIG. 8(a) is a graph showing the signal intensities (changes in light intensity) of the first focus detection signal and the second focus detection signal. FIG. 8(b) is a graph showing correction values ​​for equalizing the intensities of the first focus detection signal and the second focus detection signal. As described above, because there are optical axes on both the left and right sides of the center of the image sensor, a discontinuous, i.e., abrupt, change in light intensity occurs at the center of the image sensor, as shown in FIG. 8(a). To accommodate (suppress) this discontinuous change in light intensity, the correction values ​​also exhibit discontinuous changes at the image circle boundary between the two optical axes, as shown in FIG. 8(b). These correction values ​​are stored, for example, in the memory 215. Note that in the camera 1, correction values ​​for an imaging optical system having one optical axis at the center of the image sensor are also stored in the memory 215. The correction values ​​are switched depending on whether the imaging optical system attached to the camera 1 has a single or multiple optical axes. This switching is possible, for example, by the camera 1 receiving information (such as an ID) regarding the number of optical axes from the imaging optical system.

[0044] Since focus adjustment depends on the angle of incidence on the image sensor and the vignetting state, the position of the optical axis on the image sensor is important. For example, for optical information with the center of the optical axis as the origin, the position of the optical axis on the image sensor must be considered as the origin, which is particularly important in imaging optical systems where the optical axis is different from the center of the image sensor. When the center of the image sensor and the optical axis are different, optical information expressed with the optical axis as the origin is expanded to coordinates on the image sensor by acquiring optical axis position information on the image sensor coordinate system. Therefore, camera 1 converts the optical information expressed with the optical axis as the origin into coordinates on the image sensor by acquiring optical axis position information (Lx, Ly) on the image sensor coordinate system. For example, as shown in Table 1 below, assume that optical information is obtained in which the angle of incidence at a position 10 mm away from the optical axis is 5 degrees and the optical axis position is Lx = 5 mm, Ly = 0 mm. This indicates that the optical axis of the optical system is located at x = 5 mm, y = 0 mm on the image sensor, so the angle of incidence at (x = 15 mm, y = 0 mm) or (x = -5 mm, y = 0 mm), which are 10 mm away from the optical axis, is 5 degrees. By acquiring the optical axis position information (Lx, Ly) in this way, it is possible to convert and calculate optical information expressed with the optical axis as the origin into coordinates on the image sensor. Furthermore, the optical information calculated by converting it into coordinates on the image sensor can be used to calculate a correction value.

[0045] [Table 1]

[0046] Furthermore, in this embodiment, with regard to the conversion coefficients for converting the image shift amount p into the defocus amount d, conversion coefficients for when there is one optical axis that coincides with the center of the image sensor, and conversion coefficients for when there are multiple optical axes, are both stored in memory 215. The method of switching between conversion coefficients is the same as that for the correction values ​​described above. Furthermore, when there are multiple optical axes, it is preferable to refer to the optical information for the optical axis that is closest to the coordinates on the image sensor.

[0047] <Coordinated and independent control of left and right optical focus lenses> 9 is a flowchart showing how autofocus driving is performed under cooperative control. Autofocus driving is performed by servo AF control in the lens system control circuit 303. Servo AF control is a control that adjusts an out-of-focus state where the defocus amount is equal to or greater than a predetermined threshold to an in-focus state where the defocus amount is equal to or less than the predetermined threshold. Note that, although an example of servo AF control is described in this embodiment, the AF control method may be other than servo AF control, such as one-shot AF control or video AF control.

[0048] First, a case where cooperative control, which is a common focus lens control in a twin-lens camera, is performed to control focus (focus adjustment) using a common focus control value (focus adjustment amount) for the right-eye optical system 301R and the left-eye optical system 301L will be described with reference to FIG. 9 . The common focus control value is a focus control value based on at least one of a first defocus amount (first phase difference information) and a second defocus amount (second phase difference information). The first defocus amount corresponds to the defocus state of the right-eye optical system 301R, and the second defocus amount corresponds to the defocus state of the left-eye optical system 301L. In this case, for example, the first defocus amount can be used as the common focus control value. Other examples of the common focus control value include the second defocus amount and the average of the first and second defocus amounts. Here, as an example, when performing cooperative control, the first defocus amount is used, i.e., the right-eye optical system 301R is used as a reference. In the camera 1, the focus of the right-eye optical system 301R can be detected based on the first defocus amount, and the focus of the left-eye optical system 301L can be detected based on the second defocus amount.

[0049] In the cooperative control, the focus of the right-eye optical system 301R is adjusted using a common (same) focus control value, and the focus of the left-eye optical system 301L is also adjusted. This makes it possible to maintain the defocus difference between the right-eye optical system 301R and the left-eye optical system 301L. This makes the defocus difference between the right-eye optical system 301R and the left-eye optical system 301L less noticeable except near the in-focus point, allowing the user to view LV images and moving images being captured without feeling uncomfortable. On the other hand, in the cooperative control, the defocus difference between the right-eye optical system 301R and the left-eye optical system 301L tends to be more noticeable near the in-focus point. In recent years, with the spread of 4K and 8K standards in display devices such as HMDs and the progress of increasing resolution, there is a risk that the defocus difference between the left and right near the in-focus point will be more noticeable.

[0050] In step S901, the system control unit 218 acquires a focus detection signal generated by the imaging unit 211 when light (subject image) passing through the right-eye optical system 301R is captured by the imaging unit 211. In step S902, the system control unit 218 calculates a first defocus amount, which is first phase difference information corresponding to the defocus state of the right-eye optical system 301R, according to the flowchart (defocus amount calculation) shown in FIG. 6. In step S903, the system control unit 218 determines whether the first defocus amount is less than a predetermined threshold. If the system control unit 218 determines in step S903 that the first defocus amount is less than the predetermined threshold, the system control unit 218 considers the image to be in focus and ends the process. On the other hand, if the system control unit 218 determines in step S903 that the first defocus amount is not less than the predetermined threshold but is equal to or greater than the predetermined threshold, the process proceeds to step S904. In step S904, the system control unit 218 activates the lens system control circuit 303. At this time, the system control unit 218 drives the right-eye optical system 301R and the left-eye optical system 301L by cooperative control using the first defocus amount as a common focus control value, using the lens system control circuit 303. As a result, the right-eye optical system 301R and the left-eye optical system 301L are both driven by the first defocus amount.

[0051] In conventional twin-lens imaging devices (cameras), the resolution of display devices such as HMDs was relatively low. Therefore, degradation of image quality (degradation of image quality) due to the difference in defocus between the left and right lenses near the focus point during cooperative control was not very noticeable to users. However, as the resolution of display devices increases, degradation of image quality near the focus point, which was previously not very noticeable (tolerable), is becoming more noticeable. In order to ensure the AF accuracy required for high-resolution display devices, it is preferable to perform AF control independently for the right-eye optical system 301R and the left-eye optical system 301L.

[0052] Next, we will explain the case where the right-eye optical system 301R and the left-eye optical system 301L are independently focused, i.e., independent control in which different focus control values ​​are used for the right-eye optical system 301R and the left-eye optical system 301L. In independent control, the focus of the right-eye optical system 301R is adjusted using a first focus adjustment amount based on a first defocus amount, and the focus of the left-eye optical system 301L is adjusted using a second focus adjustment amount based on a second defocus amount, independent of the first focus adjustment. This makes it possible to change the defocus difference between the right-eye optical system 301R and the left-eye optical system 301L, i.e., to vary the magnitude of the defocus difference. This makes it possible to match the defocus accuracy of the right-eye optical system 301R and the left-eye optical system 301L even near the in-focus point. Furthermore, in independent control, when the defocus state of the right-eye optical system 301R and the left-eye optical system 301L is outside the near-focus state, differences in how the left and right subjects appear can cause differences in the defocus direction (defocus drive direction) and focus adjustment amount (focus drive amount) between the optical systems. In this case, when a user observes an LV image or moving image being captured, the image quality may be reduced. Therefore, to prevent differences in the defocus drive of the left and right optical systems outside the near-focus state and to maintain high defocus accuracy of the left and right optical systems within the near-focus state, it is preferable to switch between cooperative control and independent control. In this embodiment, switching between cooperative control and independent control is performed by the lens system control circuit 303 (first focus control unit 311 and second focus control unit 312).

[0053] FIG. 10 is a flowchart showing how to switch the autofocus drive between cooperative control and independent control.

[0054] In step S1001, the system control unit 218 acquires a focus detection signal generated by the imaging unit 211 when light (subject image) passing through the right-eye optical system 301R and the left-eye optical system 301L is captured by the imaging unit 211. If there is no history of previous lens control determination, cooperative control is first tentatively determined in this embodiment, and in step S1002, the lens system control circuit 303 performs a predetermined AF frame adjustment suitable for the right-eye optical system 301R in accordance with an AF frame position designated by a user or the like. The reason for performing the AF frame adjustment is that parallax exists between the right-eye optical system 301R and the left-eye optical system 301L, resulting in different visually recognized areas for the right-eye optical system 301R and the left-eye optical system 301L. Furthermore, if it is determined in subsequent step S1005 that there is a history of lens control determination, the AF frame adjustment is performed based on the lens control determination result. In the case of cooperative control, the process is the same as when there is no lens control history as described above, and in the case of independent control, the AF frame is adjusted according to the AF frame position specified by the user using pattern matching or the like so that the right eye optical system 301R and the left eye optical system 301L focus on the same subject.

[0055] In step S1003, if the lens control determination result is cooperative control, the system control unit 218 calculates the defocus amount of the same optical system as the optical system for which the AF frame was set in step S1002. If the lens control determination result is independent control, the system control unit 218 calculates a first defocus amount for the right-eye optical system 301R and a second defocus amount for the left-eye optical system 301L. In step S1004, the system control unit 218 determines whether the absolute value of the first defocus amount is equal to or greater than a first predetermined threshold. If the system control unit 218 determines in step S1004 that the absolute value of the first defocus amount is equal to or greater than the first predetermined threshold, cooperative control is performed by the lens system control circuit 303, and the process proceeds to step S1005. On the other hand, if the system control unit 218 determines that the absolute value of the first defocus amount is not equal to or greater than the first predetermined threshold, i.e., is less than the first predetermined threshold, independent control is performed by the lens system control circuit 303, and the process proceeds to steps S1007 and S1008, respectively.

[0056] In step S1005, the system control unit 218 determines whether the absolute value of the first defocus amount is equal to or greater than a second predetermined threshold. If the system control unit 218 determines in step S1005 that the absolute value of the first defocus amount is equal to or greater than the second predetermined threshold, the process proceeds to step S1006. On the other hand, if the system control unit 218 determines in step S1005 that the absolute value of the first defocus amount is not equal to or greater than the second predetermined threshold, that is, is less than the second predetermined threshold, the system control unit 218 considers the image to be in focus and ends the process.

[0057] In step S1007, the system control unit 218 determines whether the absolute value of the first defocus amount is equal to or greater than a second predetermined threshold. If the system control unit 218 determines in step S1007 that the absolute value of the first defocus amount is equal to or greater than the second predetermined threshold, the process proceeds to step S1006. On the other hand, if the system control unit 218 determines in step S1007 that the absolute value of the first defocus amount is not equal to or greater than the second predetermined threshold, the system control unit 218 determines that the image is in focus and ends the process. In addition, in step S1008, the system control unit 218 determines whether the absolute value of the second defocus amount is equal to or greater than the second predetermined threshold. If the system control unit 218 determines in step S1008 that the absolute value of the second defocus amount is equal to or greater than the second predetermined threshold, the process proceeds to step S1006. On the other hand, if the system control unit 218 determines in step S1008 that the absolute value of the second defocus amount is not equal to or greater than the second predetermined threshold, the system control unit 218 determines that the image is in focus and ends the process.

[0058] In step S1006, the system control unit 218 operates the lens system control circuit 303. At this time, if it is determined in step S1004 that cooperative control is to be performed, the first defocus amount is set as a common focus control value, and the right-eye optical system 301R and the left-eye optical system 301L are driven by the lens system control circuit 303 by the focus control value. On the other hand, if it is determined in step S1004 that independent control is to be performed, the right-eye optical system 301R is driven by the first defocus amount, and the left-eye optical system 301L is driven by the second defocus amount.

[0059] This switching allows cooperative control and independent control to complement each other. In other words, there is no difference in the defocus drive of the left and right optical systems outside the vicinity of the in-focus point, and the defocus accuracy of the left and right optical systems can be maintained with high accuracy within the vicinity of the in-focus point. This reduces the difference in autofocus accuracy between the left and right optical systems, suitable for capturing stereoscopic images, for example, and allows the user to always enjoy LV images and moving images without any discomfort. Furthermore, autofocus operation can be performed accurately and quickly.

[0060] In addition, when switching between cooperative control and independent control, cooperative control can also be performed if the difference between the first defocus amount and the second defocus amount, i.e., the defocus difference between the right eye optical system 301R and the left eye optical system 301L (hereinafter referred to as the "defocus difference"), is greater than or equal to a predetermined value.

[0061] In the camera 1, the first focus detection means 241 can detect the defocus direction (lens drive direction) of the right eye optical system 301R, and the second focus detection means 242 can detect the defocus direction (lens drive direction) of the left eye optical system 301L. When the first focus detection means 241 and the second focus detection means 242 detect a state in which the defocus directions of the right eye optical system 301R and the left eye optical system 301L are different from each other and the defocus difference is less than a predetermined value, cooperative control can also be performed.

[0062] In addition, when the first focus detection means 241 and the second focus detection means 242 detect a state in which the defocus directions of the right eye optical system 301R and the left eye optical system 301L are different from each other, and the defocus difference is equal to or greater than a predetermined value, independent control can be performed.

[0063] As described above, even by using cooperative control and independent control that can be switched under different conditions than those in the flowchart shown in Figure 10, it is possible to perform autofocus operations accurately and quickly while reducing the difference in autofocus accuracy between the left and right sides, which is suitable for capturing stereoscopic images.

[0064] Furthermore, when there is a difference in reliability between the first defocus amount and the second defocus amount, cooperative control can be performed. Here, "reliability" refers to the degree of visibility (sharpness, brightness, etc.) of the image observed (visually recognized) by the user. When the reliability is high, the degree of image visibility is high, and when the reliability is low, the degree of image visibility is low. Furthermore, when cooperative control is performed, the defocus amount with higher reliability between the first defocus amount and the second defocus amount may be used. Such cooperative control based on reliability enables more accurate and faster autofocus operation, i.e., the quality of lens drive can be further improved.

[0065] Furthermore, when performing independent control, it is preferable to keep the defocus difference within a predetermined value. Since the change in resolution due to the change in defocus is large near the in-focus point, keeping the defocus difference within a predetermined value can make the defocus difference between the left and right lenses less noticeable. Note that, to keep the defocus difference within a predetermined value, for example, at least one of the following first and second settings can be performed. The first setting is to set an upper limit on the difference between the drive speed of the lens of the right-eye optical system 301R and the drive speed of the lens of the left-eye optical system 301L. The second setting is to set an upper limit on the drive speed of the lens of the right-eye optical system 301R and the drive speed of the lens of the left-eye optical system 301L.

[0066] Furthermore, if a ghost is detected in one of the left and right optical systems, or if the K value difference is equal to or greater than a predetermined threshold, coordinated driving may be performed.

[0067] Furthermore, in the camera 1, the phase (amount of focus adjustment) in the right-eye optical system 301R can be divided into an amount used for fine adjustment (hereinafter referred to as a "first fine adjustment amount") and an amount used for coarse adjustment (hereinafter referred to as a "first coarse adjustment amount"). Similarly, the phase (amount of focus adjustment) in the left-eye optical system 301L can be divided into an amount used for fine adjustment (hereinafter referred to as a "second fine adjustment amount") and an amount used for coarse adjustment (hereinafter referred to as a "second coarse adjustment amount"). When the first coarse adjustment amount is larger than the first fine adjustment amount and the second coarse adjustment amount is larger than the second fine adjustment amount, cooperative driving may be performed. On the other hand, when the first fine adjustment amount is larger than the first coarse adjustment amount and the second fine adjustment amount is larger than the second coarse adjustment amount, independent control may be performed.

[0068] Furthermore, in this embodiment, switching between cooperative control and independent control when two lenses are used has been described, but this switching can also be applied to three or more lenses.

[0069] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and variations are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. [Explanation of symbols]

[0070] 100 Digital Camera (Camera) 218 System Control Unit 241 first focus detection means 242 second focus detection means 300 Lens Unit 301R Right eye optical system (1st optical system) 301L Left eye optical system (second optical system) 303 Lens system control circuit 311 first focus control means 312 second focus control means

Claims

1. a first focus detection means for detecting a focus of the first optical system based on first phase difference information corresponding to a defocus state of the first optical system through which light passing toward the image sensor passes; a second focus detection unit that is disposed at a position different from the first optical system and detects a focus of the second optical system based on second phase difference information corresponding to a defocus state of the second optical system through which light traveling toward the image sensor passes; a first focus control unit for adjusting the focus of the first optical system; a second focus control unit that adjusts the focus of the second optical system, the first focus detection means is capable of detecting a defocus direction of the first optical system, the second focus detection means is capable of detecting a defocus direction of the second optical system, The first focus control means and the second focus control means a coordinated control in which a focus adjustment amount based on at least one of the first phase difference information and the second phase difference information is set as a common focus adjustment amount, and focus adjustment of the first optical system and focus adjustment of the second optical system are performed using the common focus adjustment amount; and switching to an independent control in which focus adjustment of the first optical system is performed by a first focus adjustment amount based on the first phase difference information, and focus adjustment of the second optical system is performed by a second focus adjustment amount based on the second phase difference information, independently of the focus adjustment. A control device characterized in that when the defocus direction of the first optical system and the defocus direction of the second optical system are different from each other and the defocus difference between the first optical system and the second optical system is equal to or greater than a predetermined value, the independent control is performed, and when the defocus difference between the first optical system and the second optical system is less than the predetermined value, the cooperative control is performed.

2. the cooperative control can maintain a defocus difference between the first optical system and the second optical system, 2. The control device according to claim 1, wherein the independent control allows a defocus difference between the first optical system and the second optical system to be changed.

3. a first focus detection means for detecting a focus of the first optical system based on first phase difference information corresponding to a defocus state of the first optical system through which light passing toward the image sensor passes; a second focus detection unit that is disposed at a position different from the first optical system and detects a focus of the second optical system based on second phase difference information corresponding to a defocus state of the second optical system through which light traveling toward the image sensor passes; a first focus control unit for adjusting the focus of the first optical system; a second focus control unit that adjusts the focus of the second optical system, the first focus detection means is capable of detecting a defocus direction of the first optical system, the second focus detection means is capable of detecting a defocus direction of the second optical system, The first focus control means and the second focus control means cooperative control capable of maintaining a defocus difference between the first optical system and the second optical system through common focus adjustment based on at least one of the first phase difference information and the second phase difference information; and a first focus adjustment based on the first phase difference information and a second focus adjustment based on the second phase difference information are independently performed, and a defocus difference between the first optical system and the second optical system is variable; A control device characterized in that when the defocus direction of the first optical system and the defocus direction of the second optical system are different from each other and the defocus difference between the first optical system and the second optical system is equal to or greater than a predetermined value, the independent control is performed, and when the defocus difference between the first optical system and the second optical system is less than the predetermined value, the cooperative control is performed.

4. 4. The control device according to claim 1, wherein the first focus control means and the second focus control means, when performing the independent control, keep the defocus difference within a predetermined value.

5. the first optical system and the second optical system each have at least one lens; the first focus control means drives a lens of the first optical system, the second focus control means drives a lens of the second optical system, The control device described in claim 4, characterized in that when performing the independent control, the first focus control means and the second focus control means keep the defocus difference within a predetermined value by at least one of setting an upper limit on the difference between the drive speed of the lens of the first optical system and the drive speed of the lens of the second optical system, and setting an upper limit on the drive speed of the lens of the first optical system and the drive speed of the lens of the second optical system.

6. the first phase difference information is a first defocus amount of the first optical system, 6. The control device according to claim 1, wherein the second phase difference information is a second defocus amount of the second optical system.

7. The first focus control means and the second focus control means perform the cooperative control when there is a difference in reliability between the first phase difference information and the second phase difference information.

7. The control device according to claim 1, wherein:

8. 8. The control device according to claim 7, wherein the first focus control means and the second focus control means use the more reliable phase difference information of the first phase difference information and the second phase difference information when performing the cooperative control.

9. a first focus detection step of detecting a focus of a first optical system based on first phase difference information corresponding to a defocus state of the first optical system through which light traveling toward the image sensor passes; a second focus detection step of detecting a focus of the first optical system based on second phase difference information corresponding to a defocus state of a second optical system that is disposed at a position different from the first optical system and through which light traveling toward the image sensor passes; a first focus control step of adjusting the focus of the first optical system; a second focus control step of adjusting the focus of the second optical system, the first focus detection step is capable of detecting a defocus direction of the first optical system, the second focus detection step can detect a defocus direction of the second optical system, The first focus control step and the second focus control step include: a coordinated control in which a focus adjustment amount based on at least one of the first phase difference information and the second phase difference information is set as a common focus adjustment amount, and focus adjustment of the first optical system and focus adjustment of the second optical system are performed using the common focus adjustment amount; and switching to an independent control in which focus adjustment of the first optical system is performed by a first focus adjustment amount based on the first phase difference information, and focus adjustment of the second optical system is performed by a second focus adjustment amount based on the second phase difference information, independently of the focus adjustment. A control method characterized by performing the independent control when the defocus direction of the first optical system and the defocus direction of the second optical system are different from each other and the defocus difference between the first optical system and the second optical system is equal to or greater than a predetermined value, and performing the cooperative control when the defocus difference between the first optical system and the second optical system is less than the predetermined value.

10. A first focus detection step of detecting a focus of a first optical system based on first phase difference information corresponding to a defocus state of the first optical system through which light passing toward an imaging element passes; a second focus detection step of detecting a focus of the first optical system based on second phase difference information corresponding to a defocus state of a second optical system that is disposed at a position different from the first optical system and through which light traveling toward the image sensor passes; a first focus control step of adjusting the focus of the first optical system; a second focus control step of adjusting the focus of the second optical system, the first focus detection step is capable of detecting a defocus direction of the first optical system, the second focus detection step can detect a defocus direction of the second optical system, The first focus control step and the second focus control step include: cooperative control capable of maintaining a defocus difference between the first optical system and the second optical system through common focus adjustment based on at least one of the first phase difference information and the second phase difference information; and a first focus adjustment based on the first phase difference information and a second focus adjustment based on the second phase difference information are independently performed, and a defocus difference between the first optical system and the second optical system is variable; A control method characterized by performing the independent control when the defocus direction of the first optical system and the defocus direction of the second optical system are different from each other and the defocus difference between the first optical system and the second optical system is equal to or greater than a predetermined value, and performing the cooperative control when the defocus difference between the first optical system and the second optical system is less than the predetermined value.

11. A program for causing a computer to execute each means of the control device described in any one of claims 1 to 8.

Images