Imaging device and method

Abstract

To provide an imaging apparatus that can perform accurate focus detection of an imaging surface phase difference detection system even when a lens unit having a plurality of optical axes is attached to the apparatus.SOLUTION: An imaging apparatus comprises: an image pickup device that can generate a signal pair used for focus detection of a phase difference detection system; calculation means that calculates the focal distance of a lens unit attached to the apparatus based on a defocus amount obtained by using the signal pair; and adjustment means that adjusts the focal distance of the lens unit based on the focal distance. When the lens unit is a multi-lens unit that has a plurality of imaging optical systems having different optical axes, the calculation means calculates the focal distance by using an adjustment value obtained based on optical axis positions being positions on the image pickup device through which the optical axes of the multi-lens unit pass.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to an imaging device and method, and more particularly to an imaging device and method capable of using a lens unit having a plurality of optical axes. [Background technology]

[0002] Conventionally, a stereoscopic camera that is equipped with multiple imaging optical systems and can capture stereoscopic images with a single imaging element is known (Patent Document 1). Meanwhile, in recent years, due to factors such as the reduction in the price of VR goggles, there is a demand for a more convenient method of capturing stereoscopic images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-205558 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, by incorporating two imaging optical systems in one lens barrel and preparing it as an interchangeable lens unit, it is possible to capture a stereoscopic image with a general lens-interchangeable imaging device.

[0005] However, the image plane phase difference focus detection method currently used mainly in mirrorless cameras is based on the premise that the lens unit has one optical axis, so when a lens unit with multiple optical axes is attached, such as a lens unit with two imaging optical systems built into one lens barrel, the accuracy of focus detection can decrease.

[0006] An object of the present invention is to provide an imaging device and method that are capable of performing accurate focus detection using an imaging surface phase difference detection method even when a lens unit having multiple optical axes is attached. [Means for solving the problem]

[0007] The object of the present invention is to provide an image sensor capable of generating a signal pair used in focus detection using a phase difference detection method, and Based on Defocus Amount A detection means for detecting and, Defocus Amount Based on focus adjustment Control means for performing control and Detection Method teeth, A multi-lens unit having a plurality of imaging optical systems with different optical axes Based on the beam position Ku Using the adjustment value, Detecting the amount of defocus The object is achieved by an imaging device characterized by: Effect of the Invention

[0008] According to the present invention, it is possible to provide an imaging device and method capable of performing accurate focus detection using an imaging surface phase difference detection method even when a lens unit having a plurality of optical axes is attached. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a perspective view showing an example of the appearance of a camera 100 which is an example of an imaging device according to an embodiment. [Diagram 2] Block diagram showing an example of the functional configuration of a camera system [Diagram 3] FIG. 13 is a block diagram showing another example of the functional configuration of a camera system. [Figure 4] FIG. 1 shows an example of a pixel array of an image sensor of a camera 100. [Diagram 5] Schematic diagram of the relationship between the defocus amount and the image shift amount based on a pair of focus detection signals [Figure 6] Flowchart for focus detection processing in the first embodiment [Figure 7] Schematic cross-sectional view of an optical system having an image sensor and multiple optical axes [Figure 8] FIG. 13 is a diagram showing an example of a change in the light amount of a focus detection signal when an optical system having a plurality of optical axes is attached; [Figure 9] FIG. 13 is a diagram showing a display example of a live view image in the second embodiment. [Figure 10] FIG. 13 is a diagram showing an example of a display form of a focus guide; [Figure 11] Flowchart for focus guide display processing [Figure 12] Flowchart regarding live view display operation in the second embodiment [Figure 13] FIG. 13 is a diagram showing an example of a display form of an index that indicates a difference in the degree of focus between left and right images in the third embodiment; [Figure 14] Flowchart for indicator display control processing in the third embodiment [Figure 15] Flowchart regarding live view display operation in the third embodiment [Figure 16] FIG. 13 is a diagram showing a display example of a live view image in the third embodiment. [Figure 17] FIG. 13 is a diagram showing an example of a display form of an index that indicates a difference in the degree of focus between left and right images in the fourth embodiment; [Figure 18] FIG. 13 is a diagram showing a display example of a live view image in the fourth embodiment. [Figure 19] Flowchart regarding deviation adjustment processing in the fifth embodiment [Figure 20] FIG. 13 is a diagram showing an example of a display form of an indicator of the fourth embodiment corresponding to the sixth embodiment; [Figure 21] FIG. 23 is a diagram showing an example of a display form of a calibration guide in the sixth embodiment; [Figure 22] FIG. 23 is a diagram for explaining a calibration method in the sixth embodiment. [Diagram 23] FIG. 13 is a diagram showing XR goggles used in the seventh embodiment. [Figure 24] FIG. 23 is a diagram showing an example of a recorded image in the seventh embodiment. [Diagram 25] FIG. 23 is a diagram showing an example of an operation for changing a focused subject of a recorded image in the seventh embodiment. [Figure 26] FIG. 13 is a block diagram showing an example of the configuration of a computer capable of implementing the seventh embodiment. [Figure 27] FIG. 23 is a diagram for explaining a calibration method in the seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0011] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in an interchangeable lens digital camera. However, the present invention can also be implemented in any electronic device that can have a camera equipped with a focus detection function using an imaging surface phase difference detection method. Such electronic devices include the following: general imaging devices (video cameras, surveillance cameras, etc.), computer devices (personal computers, tablets, media players, PDAs, etc.), communication devices (mobile phones, smartphones, IoT devices, etc.), game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] ●(First embodiment) [Overall configuration] Fig. 1 is a perspective view showing an example of the exterior of a body 100 of a lens-interchangeable mirrorless digital camera (hereinafter, referred to as camera 100) as an example of an imaging device according to a first embodiment of the present invention. Fig. 1(a) is a perspective view of the camera 100 as seen obliquely from above the front, and Fig. 1(b) is a perspective view of the camera 100 as seen obliquely from above the rear.

[0013] The camera 100 has a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub electronic dial 105, a movie button 106, and a viewfinder display 107 on the top surface. The shutter button 101 is an operation unit for preparing for shooting or issuing a shooting instruction. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode switch 103 is an operation unit for switching between various modes. The main electronic dial 104 is a rotary operation unit for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation unit for moving a selection frame (cursor), forwarding images, etc. The movie button 106 is an operation unit for issuing an instruction to start or stop movie shooting (recording). The viewfinder display 107 displays various settings such as shutter speed and aperture.

[0014] Camera 100 has, on the rear surface, a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, a menu button 115, an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar 119. Display unit 108 displays images and various information. Touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of display unit 108.

[0015] The directional key 110 is an operation section consisting of keys (four-way keys) that can be pressed up, down, left and right. An operation can be performed according to the position of the directional key 110 pressed. The SET button 111 is an operation section that is pressed mainly to confirm a selection item. The AE lock button 112 is an operation section that is pressed to fix an exposure state in a shooting standby state. The enlargement button 113 is an operation section for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is also used to enlarge a playback image or increase the magnification ratio in the playback mode.

[0016] The playback button 114 is an operation unit for switching between the shooting mode and the playback mode. In the shooting mode, pressing the playback button 114 causes a transition to the playback mode, and the latest image among the images recorded in the recording medium 228 described later can be displayed on the display unit 108. The menu button 115 is an operation unit that is pressed when displaying a menu screen on which various settings can be made on the display unit 108. The user can make various settings of the camera 100 by operating the menu screen displayed on the display unit 108 using the direction keys 110 and the SET button 111. Note that the menu screen may be operated using the touch panel 109 instead of or in combination with the buttons.

[0017] Eyepiece 116 is a window for looking into eyepiece finder (peek-in type finder) 117. A user can view an image displayed on an internal EVF (Electronic View Finder) 217 ​​(described later) through eyepiece 116. Eyepiece detection unit 118 is a sensor that detects whether an object is close to eyepiece 116.

[0018] The touch bar 119 is a line-shaped touch operation unit (line touch sensor) capable of receiving a touch operation. The touch bar 119 is disposed at a position where it can be touched (touched) by the thumb of the right hand when the grip unit 120 is held in the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. That is, the touch bar 119 can be operated in a state (shooting posture) where the user looks into the eyepiece finder 117 through the eyepiece unit 116 and is ready to press the shutter button 101 at any time. The touch bar 119 can receive a tap operation (operation of touching and releasing the touch position without moving it within a predetermined period of time) on the touch bar 119, a slide operation to the left and right (operation of moving the touch position while touching), and the like. The touch bar 119 is an operation unit different from the touch panel 109, and does not have a display function. The touch bar 119 of this embodiment functions as a multi-function bar (M-Fn bar).

[0019] The camera 100 also has a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section formed in a shape that is easy to hold with the right hand when the user holds the camera 100. In a state in which the camera 100 is held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are disposed in positions that can be operated with the index finger of the right hand. In a similar state, the sub electronic dial 105 and the touch bar 119 are disposed in positions that can be operated with the thumb of the right hand.

[0020] The thumb rest section 121 (thumb standby position) is a grip section provided on the rear side of the camera 100 at a position where it is easy to place the thumb of the right hand gripping the grip section 120 when none of the operation sections are being operated. The thumb rest section 121 is made of a rubber member or the like for enhancing holding power (grip feeling). The terminal cover 122 protects connectors such as a connection cable that connects the camera 100 to an external device. The lid 123 protects the recording medium 228 and the slot by closing the slot for storing the recording medium 228 described later. The communication terminal 124 is a terminal for communicating with the lens unit 200 described later which is detachable from the camera 100.

[0021] <Internal configuration of camera 100> Fig. 2 is a block diagram showing an example of the internal configuration (functional configuration) of a camera system in which an interchangeable lens unit 200 is attached to a camera 100. Note that in Fig. 2, the components shown in Fig. 1 are given the same reference numerals as in Fig. 1. Descriptions of the components already explained with reference to Fig. 1 will be omitted as appropriate.

[0022] First, the lens unit 200 will be described. Lens unit 200 is an example of an interchangeable lens that can be attached to and detached from camera 100. Lens unit 200 is a typical single lens (a lens with one optical axis). Lens unit 200 includes aperture 201, lens 202, aperture drive circuit 203, AF (autofocus) drive circuit 204, lens system control circuit 205, communication terminal 206, and the like.

[0023] The aperture 201 is configured so that the aperture diameter can be adjusted. The lens 202 is composed of a plurality of lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the focus lens included in the lens 202 to adjust the distance at which the lens unit 200 is in focus.

[0024] The lens system control circuit 205 has, for example, a CPU, a ROM, and a RAM, and controls the operation of each part of the lens unit 200 by loading a program stored in the ROM into the RAM and executing the program with the CPU. The lens unit 200 and the camera 100 are electrically connected via communication terminals 206 and 124, and the lens system control circuit 205 and a system control unit 218 of the camera 100 can communicate with each other. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from the system control unit 218.

[0025] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 218.

[0026] The shutter 210 is a focal plane shutter that operates based on instructions from the system control unit 218 and controls the exposure time of the imaging unit 211. The imaging unit 211 is an imaging element (image sensor) configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal. In this embodiment, the imaging unit 211 is an imaging element that supports focus detection using an imaging surface phase difference detection method. Specifically, the imaging unit 211 is capable of outputting a focus detection signal pair for realizing focus detection using a phase difference detection method.

[0027] The A / D converter 212 converts an analog signal output from the imaging unit 211 into a digital signal (image data). The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on data input through the A / D converter 212 or the memory control unit 213. The image processing unit 214 also performs predetermined arithmetic processing using the captured image data to calculate evaluation values ​​used for AF and AE. The system control unit 218 performs exposure control and focus detection control based on the obtained arithmetic results. The image processing unit 214 also calculates a defocus amount based on a focus detection signal pair obtained from the imaging unit 211 as one of the evaluation values. Furthermore, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and performs AWB (auto white balance) processing on the image data based on the obtained arithmetic results.

[0028] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without via the image processing unit 214. The memory 215 stores image data output by the A / D converter 212, image data generated by the image processing unit 214, and the like. The image data generated by the image processing unit 214 includes display image data to be displayed on the display unit 108 and the EVF 217, and recording image data to be recorded on the recording medium 228. The memory 215 has a storage capacity sufficient to store a predetermined number of still image data, and a predetermined amount of moving image data and audio data. A portion of the memory 215 is used as a video memory for the display unit 108.

[0029] The D / A converter 216 converts the image data stored in the memory 215 into an analog signal suitable for display on the display unit 108 or the EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display according to the analog signal from the D / A converter 216. The display unit 108 or the EVF 217 is, for example, an LCD, an organic EL display, or the like.

[0030] While the imaging unit 211 is capturing a moving image, image data stored in a memory 215 via an A / D converter 212 is converted into an analog signal by a D / A converter 216, and the analog signal is sequentially transferred to and displayed on the display unit 108 or the EVF 217. This allows a live view display on the display unit 108 or the EVF 217.

[0031] The system control unit 218 is a control unit that includes at least one processor (CPU) and / or at least one circuit. That is, the system control unit 218 may be a processor (CPU), may be a circuit, or may be a combination of a processor and a circuit. For example, when the system control unit 218 has a processor (CPU), the system control unit 218 controls the entire camera 100 by loading a program stored in the nonvolatile memory 220 into the system memory 219 and executing it on the processor. The system control unit 218 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, and the like.

[0032] The camera 100 also includes a system memory 219 , a non-volatile memory 220 , a system timer 221 , a communication unit 222 , an attitude detection unit 223 , and an eye proximity detection unit 118 . For example, a RAM is used as the system memory 219. Constants and variables for the operation of the system control unit 218, programs read from the non-volatile memory 220, and the like are loaded into the system memory 219. The non-volatile memory 220 may be, for example, an electrically erasable and recordable EEPROM. Constants, programs, and the like for the operation of the system control unit 218 are recorded in the non-volatile memory 220.

[0033] The system timer 221 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock. The communication unit 222 transmits and receives image signals and audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 222 can communicate with external devices that comply with a wireless LAN (Local Area Network) and devices on the Internet. The communication unit 222 can also communicate with external devices that comply with Bluetooth (registered trademark). The communication unit 222 can transmit images (including live images) captured by the imaging unit 211 and images recorded in a recording medium 228, and can receive image data and various other information from external devices.

[0034] The attitude detection unit 223 outputs a signal indicating the attitude of the camera 100 with respect to the direction of gravity. Based on the signal output by the attitude detection unit 223, it is possible to determine whether the image captured by the imaging unit 211 is an image captured by holding the camera 100 horizontally or vertically. The system control unit 218 can add orientation information corresponding to the signal output by the attitude detection unit 223 to the image file of the image captured by the imaging unit 211, or rotate and record the image. The attitude detection unit 223 can use, for example, an acceleration sensor or a gyro sensor. Based on the output signal from the attitude detection unit 223, the system control unit 218 can also detect the movement of the camera 100 (panning, tilting, lifting, whether or not it is stationary, etc.).

[0035] Eyepiece detection unit 118 can detect the approach of some object to eyepiece unit 116 of eyepiece finder 117 incorporating EVF 217. Eyepiece detection unit 118 can use, for example, an infrared proximity sensor. When an object approaches, infrared light projected from a light projecting unit of eyepiece detection unit 118 is reflected by the object and received by a light receiving unit of the infrared proximity sensor. The presence or absence of an object approaching eyepiece unit 116 can be determined based on the amount of infrared light received.

[0036] The system control unit 218 switches between display (display state) / non-display (non-display state) of the display unit 108 and the EVF 217 depending on the presence or absence of a nearby object detected by the eye proximity detection unit 118. Specifically, when at least in a shooting standby state and the display destination switching setting is automatic switching, if an approaching object is not detected, the display of the display unit 108 is turned on and the display of the EVF 217 is turned off. Also, if an approaching object is detected, the display of the EVF 217 is turned on and the display of the display unit 108 is turned off. Note that the eye proximity detection unit 118 is not limited to an infrared proximity sensor, and other sensors may be used as long as they can detect a state that can be regarded as an eye proximity.

[0037] The camera 100 also has an outside-finder display unit 107, an outside-finder display drive circuit 224, a power supply control unit 225, a power supply unit 226, a recording medium I / F 227, an operation unit 229, and the like. The outside-finder display unit 107 displays various settings of the camera 100, such as the shutter speed and aperture, via an outside-finder display drive circuit 224. The power supply control unit 225 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and the like, and detects whether a battery is attached, the type of battery, and the remaining battery level. The power supply control unit 225 also controls the DC-DC converter based on the detection result and an instruction from the system control unit 218, and supplies the required voltage to each unit including the recording medium 228 for the required period. The power supply unit 226 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. The recording medium I / F 227 is an interface with a recording medium 228, such as a memory card or a hard disk. The recording medium 228 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 228 may be removable or may be built-in.

[0038] The operation unit 229 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 218. The operation unit 229 includes the shutter button 101, the power switch 102, the mode changeover switch 103, the touch panel 109, and other operation members 230. The other operation members 230 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction key 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, the touch bar 119, and the like.

[0039] The shutter button 101 has a first shutter switch 231 and a second shutter switch 232. The first shutter switch 231 is turned on when the shutter button 101 is pressed halfway, generating a first shutter switch signal SW1. The system control unit 218 interprets the first shutter switch signal SW1 as a shooting preparation instruction and starts shooting preparation processing. The shooting preparation processing includes AF processing, AE processing, AWB processing, and flash pre-emission processing.

[0040] The second shutter switch 232 is turned on when the shutter button 101 is fully pressed, and generates a second shutter switch signal SW2. The system control unit 218 interprets the second shutter switch signal SW2 as an instruction to shoot a still image, and starts a still image shooting operation based on the exposure conditions determined by the AE processing. Then, it controls each unit to execute a series of shooting processes from reading a signal from the imaging unit 211 to generating an image file including the captured image data and writing it to the recording medium 228.

[0041] The mode changeover switch 103 changes the operation mode of the system control unit 218 to one of a still image shooting mode, a video shooting mode, a playback mode, etc. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). There are also various scene modes and custom modes that are shooting settings according to shooting scenes. The user can directly switch to one of the above-mentioned shooting modes using the mode changeover switch 103. Alternatively, the user can selectively switch to one of the displayed modes using the operation unit 229 after once switching to a list screen of shooting modes using the mode changeover switch 103. Similarly, the video shooting mode may also include a plurality of modes.

[0042] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as one unit. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108. Then, by associating input coordinates on the touch panel 109 with display coordinates on the display surface of the display unit 108, a GUI can be configured as if the user could directly operate the screen displayed on the display unit 108. GUI is an abbreviation for Graphical User Interface. The touch panel 109 can use any of various methods such as a resistive film method, a capacitive method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, an image recognition method, and an optical sensor method. Depending on the method, there is a method that detects a touch by contact with the touch panel 109, and a method that detects a touch by approaching a finger or a pen to the touch panel 109, but either method may be used.

[0043] The system control unit 218 can detect the following operations or states on the touch panel 109. A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down). A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On). The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 109 is removed (released) from the touch panel 109, that is, the touch ends (hereinafter, referred to as "touch-up"). A state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0044] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on will also be detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. After it is detected that all fingers or pens that were touching have touched up, touch-off will occur.

[0045] These operations and states, and the position coordinates of the touch panel 109 touched by a finger or pen are notified to the system control unit 218. The system control unit 218 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. For touch-move, the moving direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical component and horizontal component on the touch panel 109 based on the change in the position coordinates. When a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 109 as if flicking it. When a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is detected immediately, it is determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called a pinch in, and a touch operation in which the touch positions are moved farther apart is called a pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply pinch).

[0046] <Configuration of multi-lens unit> 3 is a schematic diagram showing an example of the configuration of a twin lens unit 300 as an example of a multi-lens unit. In this specification, a "multi-lens" refers to a lens unit having a configuration in which multiple imaging optical systems are provided in one lens mount (or lens barrel), and has multiple optical axes. FIG. 3 shows the twin lens unit 300 attached to the camera 100. FIG. 3 shows only a part of the configuration of the camera 100 shown in FIG.

[0047] The twin lens unit 300 is a type of interchangeable lens that is detachable from the camera 100. The twin lens unit 300 has two imaging optical systems 301L and 301R in one lens barrel, and therefore has two optical axes.

[0048] Here, it is assumed that when the twin lens unit 300 is attached to the camera 100, the two imaging optical systems 301L and 301R are arranged so that the two optical axes are aligned on a horizontal line. The two imaging optical systems 301L and 301R have a viewing angle of approximately 180 degrees and can capture the range of the front hemisphere. Specifically, the two imaging optical systems 301L and 301R can each capture a field of view of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle). The two imaging optical systems 301L and 301R are A pair of parallax images having left and right parallax is formed on the imaging surface of the imaging unit 211. In the following description, the imaging optical system 301L is referred to as the left eye optical system 301L, and the imaging optical system 301R is referred to as the right eye optical system 301R.

[0049] Each of the right-eye optical system 301R and the left-eye optical system 301L has a plurality of lenses and a reflecting mirror, etc. The plurality of lenses includes at least a focus lens for adjusting the focal distance. The twin lens unit 300 also has a lens system control circuit 303. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the lenses 302R and 302L located on the subject side face in the same direction, and the optical axes are approximately parallel.

[0050] 3, the twin lens unit 300 has a configuration similar to that of the AF drive circuit 204. In this case, the twin lens unit 300 may have one or more of an AF drive circuit that drives the focus lenses of the right-eye optical system 301R and the left-eye optical system 301L in conjunction with each other, and an AF drive circuit that independently drives the focus lens of at least one of the right-eye optical system 301R and the left-eye optical system 301L. The focus lens is driven by the lens system control circuit 303 under the control of the system control unit 218 (adjustment means).

[0051] The twin lens unit 300 also has an encoder that detects the amount and direction of rotation of a focus ring provided on the lens barrel. The lens system control circuit 303 provides a so-called by-wire manual focus function by controlling the AF drive circuit in response to the focus lens operation detected by the encoder. In this case, the twin lens unit 300 may have a switch that allows the user to switch the focus lens driven by the focus ring operation.

[0052] The twin lens unit 300 is a lens for VR180 that allows the camera 100 to capture an image in the VR180 format, which is a format for VR images that allows binocular stereoscopic viewing. The lens for VR180 has fisheye lenses in which the right eye optical system 301R and the left eye optical system 301L each have a viewing angle of approximately 180 degrees. The right eye optical system 301R and the left eye optical system 301L only need to be able to obtain an image that allows binocular VR display as VR180, and the viewing angle may be approximately 160 degrees. The lens for VR180 can form a right image (first image) by the right eye optical system 301R and a left image (second image) by the left eye optical system 301L on the same imaging surface. Here, it is assumed that the imaging unit 211 of the camera 100 has one imaging element, and the twin lens unit 300 forms a right image and a left image on the imaging surface of the one imaging element. However, camera 100 may have two image sensors arranged in parallel, and twin lens unit 300 may form a right image on the imaging surface of one image sensor and a left image on the imaging surface of the other image sensor.

[0053] The twin lens unit 300 includes a focus ring for adjusting the focus of the right-eye optical system 301R and a focus ring for adjusting the focus of the left-eye optical system 301L. Alternatively, the twin lens unit 300 includes a focus ring for simultaneously adjusting the focus of the right-eye optical system 301R and the left-eye optical system 301L, and a focus ring for adjusting the focus of one of the right-eye optical system 301R and the left-eye optical system 301L. By operating these focus rings, the user can manually adjust the focal distance of the right-eye optical system 301R and the left-eye optical system 301L. These focus rings may be provided separately, or may be realized by switching the function of one focus ring in the case of a by-wire system.

[0054] Like the (single) lens unit 200, the twin lens unit 300 is attached to the camera 100 via a mount section. The mount section is made up of a lens mount section 304 and a camera mount section 305. When the twin lens unit 300 is attached to the camera 100, the communication terminal 124 of the camera 100 and the communication terminal 306 of the twin lens unit 300 are electrically connected. This enables the system control section 218 of the camera 100 and the lens system control circuit 303 of the twin lens unit 300 to communicate with each other.

[0055] In this embodiment, the right and left images are formed on the imaging surface of the imaging unit 211, separated in the left-right direction. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on one imaging element. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. In this way, by attaching the twin lens unit 300, a pair of parallax images (right and left images) formed by the right-eye optical system 301R and the left-eye optical system 301L can be acquired in one shooting. In addition, by VR displaying the acquired right and left images as images for the right eye and the left eye, the user can observe a stereoscopic VR image in a range of approximately 180 degrees, a so-called VR180 image.

[0056] Here, a VR image is an image that can be displayed in VR, which will be described later. VR images include omnidirectional images (spherical images) taken by an omnidirectional camera (spherical camera), panoramic images with an image range (effective image range) wider than the display range that can be displayed at one time on a display unit, and the like. In addition, a VR image may be either a still image or a video. A video may be a recorded video or a live image (an image obtained from a camera in almost real time).

[0057] A VR image has an image range (effective image range) of a maximum field of view of 360 degrees in the left-right direction and 360 degrees in the up-down direction. In addition, the VR image also includes an image having a wider angle of view than the angle of view that can be captured by a normal camera, or a wider image range than the display range that can be displayed at one time on a display unit, even if the field of view is less than 360 degrees in the left-right direction or less than 360 degrees in the up-down direction. An image captured by the camera 100 using the above-mentioned twin lens unit 300 is a type of VR image. A VR image can be displayed in VR by, for example, setting the display mode of a display device (a display device that can display a VR image) to "VR view." By displaying a VR image having a 360-degree angle of view in VR and changing the attitude of the display device in the left-right direction (horizontal rotation direction), a user can view an omnidirectional image without seams in the left-right direction.

[0058] Here, VR display (VR view) is a display mode that displays an image of a predetermined range of visual fields according to the attitude of the display device, among the visual fields captured in the VR image. VR display includes "single-eye VR display (single-eye VR view)" that displays one image by performing a transformation (transformation that performs distortion correction) that maps the VR image onto a virtual sphere. VR display also includes "two-eye VR display (two-eye VR view)" that displays a VR image for the left eye and a VR image for the right eye side by side in left and right regions by performing a transformation that maps each onto a virtual sphere.

[0059] Stereoscopic vision is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye, which have a parallax between them. In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), an image with a field of view corresponding to the direction of the user's face is displayed. For example, assume that a VR image is displayed with a field of view centered on 0 degrees left and right (a specific direction, for example, north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is flipped from this state (for example, the display surface is changed from facing south to facing north), the display range is changed to an image with a field of view centered on 180 degrees left and right (the opposite direction, for example, south) and 90 degrees up and down in the same VR image. In other words, when a user faces from north to south (i.e., turns around) while wearing an HMD, the image displayed on the HMD is also changed from a north image to a south image.

[0060] The VR image captured using the twin lens unit 300 of this embodiment is a VR180 format image capturing an image of a range of approximately 180 degrees forward, and no image exists in a range of approximately 180 degrees backward. When such an image in VR180 format is displayed in VR and the attitude of the display device is changed to a side where no image exists, for example, a blank area is displayed.

[0061] By displaying the VR image in this way, the user visually feels as if he or she is inside the VR image (in the VR space). The method of displaying the VR image is not limited to changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operation via a touch panel or directional buttons. Furthermore, during VR display (when in the "VR view" display mode), in addition to changing the display range due to posture changes, the display range may also be changed in response to touch moves on a touch panel, dragging operations with a mouse, pressing directional buttons, etc. A configuration in which a display device such as a smartphone is attached to VR goggles (head-mounted adapter) is a type of HMD.

[0062] <Configuration of the imaging unit 211 (imaging element)> Fig. 4 is a diagram showing a schematic example of a pixel array of the imaging section 211 (imaging element) in this embodiment. The imaging element constituting the imaging section 211 in this embodiment is capable of generating signal pairs used for focus detection by a phase difference detection method. Fig. 4 shows the pixel array of the imaging element (two-dimensional CMOS sensor) in a range of 4 columns x 4 rows of imaging pixels (a range of 8 columns x 4 rows as the array of focus detection pixels). Hereinafter, when simply referred to as "pixel", it means an imaging pixel.

[0063] The imaging unit 211 is provided with color filters in a primary color Bayer array. The pixel group 400 indicates 2 columns x 2 rows of pixels, which are the repeating units of the color filters. The pixel group 400 has a pixel 400R having a spectral sensitivity of R (red), pixels 400Gr and 400Gb having a spectral sensitivity of G (green), and a pixel 400B having a spectral sensitivity of B (blue). In addition, each pixel is provided with a microlens 401.

[0064] In order to enable focus detection by the imaging surface phase difference method, the imaging unit 211 has two photodiodes (photoelectric conversion units) 402 and 403 that share a microlens 401 in each of a plurality of pixels arranged two-dimensionally in the imaging unit 211. The first photodiode 402 and the second photodiode 403 each function as a subpixel or a focus detection pixel. That is, one pixel functions as two focus detection pixels. In addition, by treating the first photodiode 402 and the second photodiode 403 as one photodiode together, one pixel functions as one imaging pixel. Hereinafter, the signal obtained by the first photodiode 402 is called the A signal, the signal obtained by the second photodiode 403 is called the B signal, and the signal obtained by adding the A signal and the B signal obtained by the same pixel is called the A+B signal. In addition, the A signal and the B signal are called the focus detection signal, and the A+B signal is called the imaging signal. The A signal (B signal) may be obtained by subtracting the B signal (A signal) from the A+B signal.

[0065] In this embodiment, each pixel is configured to have two photodiodes sharing the microlens 401, but the number of photodiodes provided in each pixel may be three or more. Also, a configuration may be provided in which a pixel dedicated to focus detection that can essentially output only the A signal or the B signal is provided. There is no restriction on the configuration of pixels provided in the image sensor as long as it is possible to output a signal that can realize focus detection using the phase difference detection method. Also, in this embodiment, all pixels are configured to have multiple photodiodes, but only some pixels may have multiple photodiodes.

[0066] <Relationship between defocus amount and image shift amount> The relationship between the defocus amount and the image shift amount obtained from the A signal and the B signal obtainable by the image sensor shown in Fig. 4 will be described with reference to Fig. 5. The defocus amount is calculated using a pair of image signals consisting of an A image signal obtained by concatenating multiple A signals and a B image signal obtained by concatenating multiple B signals. Each of the A image signal and the B image signal is also called a focus detection signal. Here, it is assumed that the imaging center (the center of the pixel area used for imaging in the image sensor) coincides with the optical axis center.

[0067] 5 is a schematic diagram showing the relationship between the defocus amount d and the image shift amount between a pair of focus detection signals (an A-image signal and a B-image signal). Reference numeral 1300 denotes the imaging surface of the image sensor. The exit pupil of the imaging optical system is divided into a first pupil partial region 1303 and a second pupil partial region 1304 by a first photodiode 402 and a second photodiode 403 that share a single microlens.

[0068] The magnitude |d| of the defocus amount d represents the distance from the imaging position of the subject image to the imaging plane 1300. In a front-focus state where the imaging position of the subject image is closer to the subject than the imaging plane 1300, the direction is defined as a negative sign (d<0), and in the opposite back-focus state, the direction is defined as a positive sign (d>0). In a focused state where the imaging position of the subject image is on the imaging plane 1300 (i.e., the in-focus position), d=0. For example, the subject 1301 is in-focus because it is imaged in a focused state (d=0). Also, the imaging position of the subject 1302 is closer to the subject than the imaging plane 1300 (d<0), so it is in front-focus. In the following, the front-focus state (d<0) and the back-focus state (d>0) are collectively referred to as a defocus state (|d|>0).

[0069] In a front-focus state (d<0), of the light beams received from the subject 1302, those that pass through the first pupil partial region 1303 (or the second pupil partial region 1304) are collected and then spread to a width Γ1 (or Γ2) centered on the center of gravity position G1 (or G2) of the light beams. In this case, the image of the subject 1302 is blurred on the imaging surface 1300. The blurred image is received by the first photodiode 402 (or the second photodiode 403) provided in each pixel arranged in the imaging element, and an A signal (or a B signal) is generated.

[0070] Therefore, a pair of focus detection signals (A image signal and B image signal) are stored in memory as image data of a (blurred) subject image having width Γ1 (or Γ2) at center of gravity position G1 (or G2) on the imaging surface 1300. The subject image width Γ1 (or Γ2) increases roughly in proportion to an increase in the magnitude |d| of the defocus amount d. Similarly, if the image shift amount between the first focus detection signal and the second focus detection signal is "p," the magnitude |p| of the image shift amount increases as the magnitude |d| of the defocus amount d increases.

[0071] 5, the image shift amount p can be defined as the difference between the center of gravity positions of the light beams, "G1-G2," and its magnitude |p| increases roughly in proportion to the increase in the defocus amount magnitude |d|. Note that in a back-focus state (d>0), the direction of image shift between a pair of focus detection signals (image A signal and image B signal) is opposite to that in the front-focus state, but the defocus amount magnitude |p| is proportional to the defocus amount magnitude |d|.

[0072] Therefore, phase difference AF can be achieved by detecting the amount of image shift p between a pair of focus detection signals (image signal A and image signal B) and converting the amount of image shift p into a defocus amount using a conversion coefficient K. The amount of image shift p between a pair of focus detection signals (image signal A and image signal B) can be found by relatively shifting the image signal A and the image signal B to calculate the amount of correlation, and from the shift amount at which good correlation (degree of signal agreement) is obtained. Note that the conversion coefficient K has a value that depends on the angle of incidence, F-number, and optical axis position of the imaging optical system. Therefore, a conversion coefficient K that corresponds to the lens unit is used.

[0073] The conversion coefficient K is stored in, for example, a non-volatile memory included in the lens system control circuit 205 of the lens unit, and the system control unit 218 can acquire it from the attached lens unit. Of course, the conversion coefficient K may be acquired in other ways, such as by storing the conversion coefficient K in association with the identification information of the lens unit in the non-volatile memory 220 and acquiring the conversion coefficient K from the non-volatile memory 220 based on the identification information of the attached lens unit. A pair of focus detection signals is usually generated based on the signals of pixels in the focus detection area. Therefore, when a focus detection area is set for each imaging optical system, a defocus amount is calculated for each focus detection area. When the focus lens of the imaging optical system can be driven individually, the focal distance can be adjusted for each imaging optical system. Note that one defocus amount based on the defocus amount calculated for each focus detection area may be used to adjust the focal distance of multiple imaging optical systems. One defocus amount may be, for example, an average value or a representative value.

[0074] <Defocus amount calculation process> The defocus amount calculation process will be described with reference to the flowchart shown in FIG. In this embodiment, the image processing unit 214 generates a first focus detection signal (A image signal) by combining A signals obtained by the first photodiodes 402 of multiple pixels in the image sensor, and generates a second focus detection signal (B image signal) by combining B signals obtained by the second photodiodes 403 of each pixel used to generate the A image signal.

[0075] In S1401, the image processing unit 214 (calculation means) acquires an A signal and a B signal from each of a plurality of pixels included in an area of ​​the image sensor corresponding to the focus detection area. As described above, the A signal (or the B signal) may be acquired by subtracting the B signal (or the A signal) from the A+B signal.

[0076] In S1402, the image processing unit 214 adds the A signals of pixels of the same color that are located at the same position in the horizontal direction (row direction) in the vertical direction (column direction) to reduce the data amount of the A and B image signals. This compresses the signals to two rows. Furthermore, the image processing unit 214 generates a luminance signal Y by adding green (Gr), red (R), blue (B), and green (Gb) signals to the A signals added in the column direction. These multiple luminance signals Y arranged in the row direction are considered to be the A image signal. The image processing unit 214 also applies a similar addition process to the B signal to generate the B image signal. By adding the signals, the Nyquist frequency in the addition direction becomes 1 / n of that when not added, where n is the number of pixels to be added.

[0077] In S1403, the image processing unit 214 applies shading correction processing (optical correction processing) to the A image signal and the B image signal to make the signal intensities uniform (suppress the difference in signal intensity). The shading correction value has a value that depends on the incidence angle, F-number, and optical axis position of the imaging optical system. Like the conversion coefficient K, the shading correction value can also be acquired from the lens unit or the non-volatile memory 220.

[0078] In S1404, the image processing unit 214 applies spatial band-pass filter processing having a specific pass frequency band to the A and B image signals in order to improve the correlation (degree of signal agreement) between the A and B image signals and thereby improve focus detection accuracy. Examples of band-pass filters include differential filters such as {1, 4, 4, 4, 0, -4, -4, -4, -1} that cut off DC components and extract edges, and additive filters such as {1, 2, 1} that suppress high-frequency noise components.

[0079] In S1405, the image processing unit 214 calculates the amount of correlation between the A and B image signals after the filter processing is applied. The amount of correlation is calculated for each shift amount while changing the relative shift amount of the A and B image signals in the pupil division direction.

[0080] The number of signals constituting the A and B image signals after band-pass filtering is W (>2), and the kth (1≦k≦W) signals are A(k) and B(k). If the shift amount is s and the range of the shift amount s is Γ, the correlation amount COR is calculated by formula (1). COR(s)=Σ(k∈W)|A(k)-B(ks)|, s∈Γ (1)

[0081] The absolute difference values ​​between the kth image A signal A(k) and the ksth B image signal B(ks) are accumulated for k within the range of the number of signals W, and a correlation amount COR(s) for a shift amount s is calculated. The shift amount is, for example, in units of one pixel. When there are multiple A and B image signals in the vertical direction, the correlation amounts calculated for each pair of A and B image signals for the same shift amount may be added together.

[0082] In S1406, the image processing unit 214 calculates the shift amount at which the correlation amount becomes the minimum in units of less than one pixel, based on the correlation amount COR(s) calculated for the shift amount in units of one pixel. Then, the image processing unit 214 sets the calculated shift amount as the image shift amount p between the A image signal and the B image signal. Furthermore, the image processing unit 214 applies (for example, multiplies) the image shift amount p by the above-mentioned conversion coefficient K to convert it into a defocus amount d. Through the above processing, the defocus amount d is calculated.

[0083] In this way, the correction value used for the signal correction performed in the process of calculating the defocus amount d and the conversion coefficient K for converting the image shift amount p to the defocus amount d have values ​​that depend on the characteristics of the lens unit. Usually, the focus detection adjustment value is calculated assuming a lens unit with one optical axis that passes through the center of the image sensor. Note that the optical axis passing through the center of the image sensor here indicates a design or ideal state, and may have deviations due to manufacturing errors. For example, even if the aperture of the lens frame is the same, the vignetting of incident light by the lens frame differs between the case where there is one optical axis and the case where there are multiple optical axes. Therefore, even if the focus detection adjustment value is calculated from the lens information for a lens unit with two optical axes in the same way as for a lens unit with one optical axis, the focus detection adjustment value will not be appropriate, and as a result, the accuracy of the defocus amount may decrease.

[0084] In this embodiment, even when an imaging optical system having multiple optical axes is attached to a main body having one image sensor, the deterioration of the accuracy of the defocus amount is suppressed by using an appropriate focus detection adjustment value. In this embodiment, the focus detection adjustment value for a lens unit having multiple optical axes is as follows: Shading correction coefficient for matching the strength of a pair of focus detection signals Conversion coefficient K for converting the image shift amount p to the defocus amount d -Best focus compensation value that corrects the focus distance based on the defocus amount However, the concept of this embodiment can be similarly applied to other focus detection adjustment values. The focus detection adjustment values ​​are basically used for each focus detection area, that is, for each imaging optical system.

[0085] <Shading correction coefficient> Fig. 7 shows a schematic cross-sectional view of an optical system when a twin lens unit 300 having two optical axes is attached to a camera 100 having one image sensor. In this embodiment, the twin lens unit 300 has two imaging optical systems, and the optical axes of the respective imaging optical systems pass on a straight line in the horizontal direction (direction parallel to the long side of the image sensor) that passes through the center (imaging center) of the image sensor and at a position equidistant from the imaging center. Therefore, Fig. 7 shows a horizontal cross section including the center of the image sensor and the two optical axes.

[0086] The diameter of the image circle of each imaging optical system is approximately 1 / 2 the length of the long side of the effective pixel area of ​​the image sensor. FIG. 8(a) shows an example of the signal intensity of the A image signal and the B image signal obtained when the twin lens unit 300 is attached. As shown in FIG. 8, since there are optical axes on both sides of the center of the image sensor, the signal intensity shows a discontinuous change at the center of the image sensor. When an imaging optical system having one optical axis passing through the center of the image sensor is attached, the intensity change of the A image signal and the B image signal is continuous. Therefore, the discontinuous intensity change of the A image signal and the B image signal is a characteristic phenomenon when an imaging optical system having multiple optical axes is attached to one image sensor.

[0087] Fig. 8(b) shows an example of shading correction values ​​for aligning the intensities of the image signals A and B having the intensity changes shown in Fig. 8(a). In order to correct the image signals A and B whose signal intensities change discontinuously, the shading correction values ​​also change discontinuously at the boundary between the two image circles.

[0088] In this embodiment, the shading correction value for the twin lens unit 300 having two optical axes is stored in advance in, for example, the non-volatile memory 220 of the camera 100, similar to the shading correction value for the lens unit 200 having one optical axis. The system control unit 218 selects a correction value to be used for shading correction according to the number of optical axes of the attached lens unit. The shading correction value may be calculated in advance for each model of lens unit and stored in the non-volatile memory 220 in association with the identification information of the lens unit. Also, a calculation formula for the correction value according to the number of optical axes may be stored in the non-volatile memory 220, and information required for calculating the shading correction value may be obtained (for example, from the attached lens unit) and applied to the calculation formula to calculate the shading correction value.

[0089] When calculating the shading correction value, the number of optical axes and the positions on the image sensor through which the optical axes pass (hereinafter referred to as optical axis positions) are necessary. Optical information with the optical axis center as the origin needs to be handled with the optical axis position as the origin. Therefore, for lens units whose optical axes do not pass through the center of the image sensor, such as twin lens unit 300 having multiple optical axes, the optical axis positions are important in calculating the shading correction value.

[0090] In this embodiment, the optical axis position (Lx, Ly) is obtained, for example, from the twin lens unit 300, and the optical information of the imaging optical system expressed with the optical axis as the origin is converted to information in the coordinate system of the image sensor. For example, as shown in Table 1, assume that the incident angle of light at a position 10 mm away from the optical axis is 5 degrees (with respect to the optical axis), and the optical axis position (Lx, Ly) = (5 mm, 0 mm) is obtained as optical information of one imaging optical system.

[0091] The optical axis position (5mm, 0mm) indicates that the optical axis passes through the position x = 5mm, y = 0mm in an orthogonal coordinate system with the center of the imaging element as the origin. Therefore, from the optical information, it can be seen that the angle of incidence of light at a point on the imaging device 10mm away from the optical axis (for example, (15mm, 0mm) or (-5mm, 0mm)) is 5 degrees.

[0092] In this way, by acquiring the optical axis position (Lx, Ly) expressed in the coordinate system of the image sensor, the optical information expressed with the optical axis as the origin can be converted into optical information in the coordinate system of the image sensor. Then, by using the optical information in the coordinate system of the image sensor, an appropriate shading correction value can be calculated. [Table 1]

[0093] As with the shading correction values, the conversion coefficients for converting the image shift amount into the defocus amount are stored in advance in, for example, the non-volatile memory 220 of the camera 100 for the twin lens unit 300 having two optical axes and the lens unit 200 having one optical axis. Note that for lens units with multiple optical axes, it is advantageous in terms of conversion accuracy to use a conversion coefficient related to the optical axis position closest to the coordinates (or focus detection area) to which correction is applied.

[0094] <Best focus correction value> An image captured by adjusting the focus of the imaging optical system to a focus distance based on the defocus amount obtained by focus detection using the image plane phase difference method may not match the image that humans perceive as being most in focus. One of the reasons for this is thought to be that the spatial frequency band used to calculate the defocus amount does not match the spatial frequency band observed by the human eye. For this reason, it is known to correct the focus distance detected in the imaging device to a focus distance that produces an image that humans perceive as being in the best focus state. The correction value used to correct this focus distance is called the best focus correction value.

[0095] In addition, there are cases where the best focus correction value has a value according to the distance from the optical axis position, for example, the best focus correction value is expressed by the following formula according to relative coordinates (x, y) with the optical axis position as the origin. Best focus correction value = a00 + a10 * x + a01 * y + a11 * xy Here, a00, a10, a01, and a11 are coefficients. When the optical axis position coincides with the center of the image sensor, the best focus correction value at any relative coordinate (x, y) on the image sensor can be calculated using the above formula. The best focus correction value is obtained for the focus detection area, and the focus distance detected for that focus detection area is corrected using the best focus correction value. By adjusting the focus distance of the imaging optical system based on the corrected focus distance, it becomes possible to obtain an image that humans perceive as being in the best focus state.

[0096] However, if the optical axis position does not coincide with the center of the image sensor, the above formula cannot express the best focus correction value. In this embodiment, by correcting the above formula based on the above-mentioned optical axis position (Lx, Ly), it is possible to calculate the best focus correction value at any relative coordinate (x, y) with the center of the image sensor as the origin, even for an imaging optical system whose optical axis position does not coincide with the center of the image sensor.

[0097] Specifically, by modifying the above equation using the optical axis position (Lx, Ly) as follows, it becomes possible to calculate the best focus correction value at any relative coordinate (x, y) with the center of the image sensor as the origin for an imaging optical system with the optical axis position (Lx, Ly). Best focus correction value = a00 + a10 (x-Lx) + a01 (y-Ly) + a11 (x-Lx) (y-Ly)

[0098] By correcting the best focus correction value expressed as a function using coordinates with the optical axis as the origin based on the optical axis position, an appropriate correction value can be calculated for a lens unit having an imaging optical system in which the optical axis position differs from the center of the image sensor, such as a lens unit having multiple optical axes. The optical axis position (Lx, Ly) expressed in the coordinate system of the image sensor can be acquired from the lens unit through communication. Alternatively, the optical axis position (Lx, Ly) previously stored in the image capture device in association with the identification information of the lens unit may be acquired by referring to the identification information of the attached lens unit. For a lens unit having multiple optical axes, accurate correction is possible by using the best focus correction value corrected using the optical axis position closest to the coordinates (x, y) to be corrected.

[0099] In this embodiment, the shading correction value, conversion coefficient, and best focus correction value are described as focus detection adjustment values ​​that are based on the assumption that the optical axis position is the center of the image sensor. However, the technical idea of ​​this embodiment is to correct the focus detection adjustment value that is based on the assumption that the optical axis position is the center of the image sensor based on the information on the optical axis position, and is applicable to any focus detection adjustment value that is based on the assumption that the optical axis position is the center of the image sensor.

[0100] In addition, for a lens unit having multiple imaging optical systems with different optical axes, in principle, it is necessary to acquire or store optical information (optical axis position, incident angle, etc.) for each imaging optical system. However, for imaging optical systems in which optical information other than the optical axis position is common, the optical information other than the optical axis position is acquired or stored for only one imaging optical system, thereby making it possible to reduce the storage capacity of the lens unit or the imaging device body.

[0101] As described above, according to this embodiment, when a lens unit having multiple optical axes is attached to an imaging device having one imaging element, the correction value used in the calculation process of the defocus amount is calculated using position information through which each optical axis passes in the imaging element. Therefore, even for correction values ​​for which an appropriate value cannot be obtained by a calculation method assuming a general lens unit having one optical axis passing through the center of the imaging element, an appropriate value can be obtained. As a result, even when a lens unit having multiple optical axes is attached to an imaging device having one imaging element, accurate focus detection using the imaging surface phase difference detection method can be realized.

[0102] ●(Second embodiment) Next, a second embodiment of the present invention will be described. The first embodiment has been described as being capable of realizing accurate focus detection using an image plane phase difference detection method even when a lens unit having multiple optical axes is attached to an imaging device having one image sensor. This embodiment relates to a configuration (focus guide function) that supports manual focus operation when a lens unit having multiple optical axes is attached to an imaging device having one image sensor.

[0103] The focus guide function is a function that presents the position of the focus detection area and the degree of focus of the focus detection area to the user. For example, the position of the focus detection area and the degree of focus can be presented to the user by superimposing a GUI such as an index or mark indicating the position of the focus detection area and the degree of focus on the live view display.

[0104] When a normal lens unit 200 that is assumed to have one optical axis passing through the center of the imaging element is attached, one image is formed on the imaging element. Therefore, the focus guide function only needs to be provided for one image. However, when a lens unit having multiple optical axes is attached, multiple images are formed on the imaging element. For example, when the twin lens unit 300 is attached to the camera 100, two images are formed on the imaging element (imaging section 211).

[0105] In this case, if a focus guide function is provided for only one image, it becomes difficult to perform accurate manual focusing for the image for which the focus guide is not provided. In addition, it is not possible to grasp the focus degree of each image at once. Therefore, it is necessary to provide a focus guide function suitable for the case where multiple images are formed on one image sensor.

[0106] An example of the focus guide function provided in this embodiment is shown in FIG. 9. In FIG. 9, the focus guide function is provided for each of the right image 800R and the left image 800L of a live view image captured using the right eye optical system 301R and the left eye optical system 301L. Specifically, indices 801R and 801L indicating information regarding the position of the focus detection area and the degree of focus are superimposed on the right image 800R and the left image 800L, respectively. The indices 801R and 801L are displayed at positions having the same image height. FIG. 9 shows an example in which the indices 801R and 801L are displayed at the same relative coordinates with the optical axis position of each imaging optical system as the origin.

[0107] In this embodiment, the focus guide function is provided by the image processing unit 214 under the control of the system control unit 218. Specifically, in parallel with the live view display process, the image processing unit 214 calculates the defocus amount of the imaging optical system for the focus detection area notified by the system control unit 218, and outputs it to the system control unit 218 together with the reliability. Then, based on the display form instructed by the system control unit 218, the image processing unit 214 generates an image of an index, and writes the image of the index in an address area corresponding to the focus detection area in a video memory area in the system memory 219. This causes the index to be superimposed on the live view image, and it is possible to provide the user with information regarding the position and the degree of focus of the focus detection area. Note that calculating the defocus amount for the focus detection area means calculating the defocus amount based on the A image signal and the B image signal obtained from pixels in the focus detection area.

[0108] Next, specific examples of indicators provided by the focus guide function will be described with reference to Figs. 10(A) to 10(D). The indices include a frame-shaped first index 500 displayed on the outer periphery of the focus detection area, and a third index 502 displayed at a position tangent to a virtual circle 510 that encompasses the focus detection area and has a common center with the focus detection area. The indices also include a second index 501 displayed at a position tangent to a virtual circle 511 whose radius is larger than that of the virtual circle 510 by the length of the third index 502.

[0109] The first index 500 indicates the position and size of the focus detection area, and also indicates whether the focus detection area is in-focus or out-of-focus depending on the display form. The second index 501 and the third index 502 indicate the degree of focus in the focus detection area depending on the display form and positional relationship between them. Specifically, in addition to the in-focus state and out-of-focus state, for the out-of-focus state, a distinction is made between a state in which the focus is closer to the subject (front focus) and a state in which the focus is infinity farther from the subject (back focus), and the amount of deviation from the in-focus state is presented.

[0110] FIG. 10A shows an example of the display form of the indices when the focus detection area is in the in-focus state. In the in-focus state, the first index 500 is displayed as a continuous frame. The second index 501 is a wedge-shaped one pointed downward, and the third index 502 is a wedge-shaped one pointed upward. The second index 501 and the third index 502 are displayed on a vertical line passing through the center of the first index 500, facing each other so that their tips are in contact. In the in-focus state, the second index 501 and the third index 502 have a display form in which the insides are filled. The display forms of the first to third indexes 500 to 502 may differ in attributes other than the shape and the presence or absence of filling, such as color, brightness, and the presence or absence of blinking, as long as the in-focus state and the out-of-focus state can be visually distinguished. For example, the second index 501 and the third index 502 can be displayed in green in the in-focus state and in white in the out-of-focus state.

[0111] 10B and 10C show examples of the display form of the indicator when the reliability of the defocus amount is high in an out-of-focus state. FIG. 10B shows an example of the display form of the index in the front focus state. In the out-of-focus state, the first index 500 is displayed as a frame with a gap. The display position of the second index 501 is the same as in the in-focus state, but the display form is different. Here, an example is shown in which the second index 501 is filled in in the in-focus state, but is hollow in the out-of-focus state. On the other hand, the third index 502 is displayed as two indexes 502A and 502B, each of which is displayed at a position to the left (right) away from the display position in the in-focus state by a distance according to the magnitude of the defocus amount. The second index 501 faces downward, indicating that it is focused closer to the subject, and is displayed in the same display position as in the in-focus state, making it easier to grasp the magnitude of the defocus amount indicated by the index 502A (or index 502B).

[0112] FIG. 10C shows an example of the display form of the index in the back focus state. Since it is an out-of-focus state, the first index 500 is displayed as a frame with a gap. Also, the display position of the third index 502 is the same as in the in-focus state, but the display form is different. Here, an example is shown in which the third index 502 is filled in in the in-focus state, but is hollow in the out-of-focus state. On the other hand, the second index 501 becomes two indexes 501A and 501B, and is displayed at positions to the left and right from the display position in the in-focus state by a distance according to the magnitude of the defocus amount. The third index 502 faces upward, indicating that it is focused farther than the subject, and is displayed in the same display position as in the in-focus state, making it easy to grasp the magnitude of the defocus amount indicated by the index 501A (or index 501B).

[0113] In this embodiment, in the case of the front focus state, the third index 502 is set to two, but the number does not need to be increased. Simply, one of the indexes 502A and 502B in Fig. 10(B) may be displayed (i.e., the display form and display position of the third index 502 may be changed). Similarly, in the case of the back focus state, one of the indexes 501A and 501B in Fig. 10(C) may be displayed (i.e., the display form and display position of the second index 502 may be changed).

[0114] Fig. 10(D) shows an example of the display form of the indicator when the defocus amount is large and the reliability of the focus detection result is low (for example, when the image is significantly blurred). In this case, neither the front focus / back focus state (defocus direction) nor the magnitude of the defocus amount is presented. The indicator is displayed in a form that notifies the user that focus detection sufficient to provide the focus guide function is not possible.

[0115] Here, the first index 500 is displayed in a non-focused state, and the second index 501 and the third index 502 are displayed in a different form from the focused state and the non-focused state in which the defocus direction and the defocus amount can be presented. Specifically, the shapes of the second index 501 and the third index 502 are changed from a wedge shape to a bar or line shape with a constant thickness, and are displayed in a color (for example, gray) different from the other focused states. In addition, the display positions of the second index 501 and the third index 502 are fixed positions determined in advance. In the example shown in FIG. 10(D), the third index 502 is divided into two indexes 502A and 502B as in the front focus state, but it is not necessary to increase the number of indexes. The reliability of the defocus amount can be calculated by the image processing unit 214 using any known method. For example, if the maximum value of the correlation degree does not reach a threshold value, the reliability of the defocus amount may be considered to be low.

[0116] In this embodiment, the focus guide function is provided based on the defocus amount and direction obtained by using the focus detection configuration of the image plane phase difference method. However, the basic technical idea of ​​this embodiment does not depend on the method of acquiring the focus degree of the focus detection area. Therefore, the focus guide function may be provided for each imaging optical system based on other evaluation values ​​that depend on the focus degree of the image formed by each imaging optical system, such as a contrast evaluation value.

[0117] <Display control process for focus guide> Next, the display control process of the focus guide executed by the system control unit 218 will be described with reference to the flowchart shown in Fig. 11. This process is realized by the system control unit 218 expanding a program recorded in the non-volatile memory 220 into the system memory 219 and executing the program. Note that the display control process of the focus guide is executed in parallel with the live view display process.

[0118] The focus guide display control process described here does not depend on the number of optical axes (the number of imaging optical systems) of the attached lens unit. It is sufficient to execute the focus guide display control process for each image formed by the imaging optical system.

[0119] In S601, the system control unit 218 notifies the image processing unit 214 of the position and size of the focus detection area. There is no particular limitation on the method of determining the position and size of the focus detection area when the twin lens unit 300 having multiple optical axes is attached. As in the case where the lens unit 200 with one optical axis is attached, the position and size may be specified by the user, may be set based on a characteristic area such as a face, or may be set at a predetermined position and size. However, it is assumed that the same position and size of the focus detection area is set for the image formed by each imaging optical system. The image processing unit 214 calculates the defocus amount for the focus detection area notified from the system control unit 218 as described in the first embodiment.

[0120] In S602, the system control unit 218 acquires the defocus amount and its reliability of the imaging optical system in the focus detection area from the image processing unit 214. The reliability may be, for example, a correlation amount corresponding to the defocus amount.

[0121] In S604, the system control unit 218 determines whether the reliability of the defocus amount acquired in S602 is high. For example, if the acquired reliability is equal to or higher than a predetermined threshold, the system control unit 218 can determine that the reliability of the defocus amount is high.

[0122] If the system control unit 218 determines that the reliability of the defocus amount is high, it executes S605, and if not, it executes S610. In S610, the system control unit 218 determines to display the index in a display form in a large blur state (fourth display form), and executes S611. The fourth display form is the display form shown in FIG.

[0123] In S605, the system control unit 218 determines whether the focus detection area is in focus or out of focus. The system control unit 218 can determine that a focus detection area whose absolute value of the defocus amount is equal to or less than a threshold is in focus, and that a focus detection area whose absolute value of the defocus amount exceeds the threshold is out of focus. If the system control unit 218 determines that the focus detection area is in focus, it executes S607, and if the system control unit 218 determines that the focus detection area is out of focus, it executes S606.

[0124] In S607, the system control unit 218 determines to display the index in the in-focus state display form (first display form), and executes S611. The first display form is the display form shown in FIG.

[0125] In S606, the system control unit 218 determines whether the state is front focus or back focus. The system control unit 218 can determine whether the state is front focus or back focus based on the sign of the defocus amount (defocus direction). If the state is determined to be front focus, the system control unit 218 executes S608, and if the state is determined to be back focus, the system control unit 218 executes S609.

[0126] In S608, the system control unit 218 determines to display the index in the front focus state display mode (second display mode), and executes S611. The second display mode is the display mode shown in FIG. In S609, the system control unit 218 determines to display the index in the back focus state display mode (third display mode), and executes S611. The third display mode is the display mode shown in FIG.

[0127] In S611, the system control unit 218 determines the display position of the index and notifies the image processing unit 214 of the display mode and the position together with the display mode. In accordance with the notification, the image processing unit 214 generates an image of the index according to the display mode and writes the image of the index to an address in the video memory area according to the display position. As a result, the index is superimposed on the live view image and displayed on the EVF 217 or the display unit 108.

[0128] Next, the live view display operation will be described with reference to the flowchart shown in Fig. 12. This process is realized by loading a program recorded in the non-volatile memory 220 into the system memory 219 and executing it by the system control unit 218. This process can be executed when performing live view display, for example, in a shooting standby state or during video shooting.

[0129] In S701, the system control unit 218 starts live view display on the EVF 217 or the display unit 108. Specifically, the system control unit 218 controls the image processing unit 214 to generate images for display and sequentially write them to the video memory area of ​​the system memory 219 while continuing to capture moving images with the imaging unit 211. As a result, a live view image is displayed on the EVF 217 or the display unit 108. The processes from S702 onwards are executed in parallel with the live view display.

[0130] In S702, the system control unit 218 confirms the type (number of optical axes) of the attached lens unit. The system control unit 218 communicates with the lens system control circuit 205 or 303 via the communication terminal 124 and the communication terminal 206 or 306 to confirm the lens type information. The lens type information includes information capable of identifying the model of the lens unit and the number of optical axes. Note that the lens type information is included in the lens information transmitted from the lens unit to the camera 100 when the camera 100 is powered on or when the lens unit is replaced. Therefore, in S702, the system control unit may refer to already acquired lens information without communicating with the lens unit.

[0131] The system control unit 218 executes S703 if it is determined that a lens unit with one optical axis is attached, and executes S704 if it is determined that a lens unit with multiple optical axes (two in this case) is attached.

[0132] In S703, the system control unit 218 executes focus guide display processing for a lens unit with one optical axis (a normal lens unit). The focus guide display processing may be the focus guide display control processing described using the flowchart in Fig. 11. As a result, one focus guide is displayed in a live view image formed by one imaging optical system.

[0133] In S704 and S705, the system control unit 218 performs focus guide display processing for each of the images (left image and right image) formed by the two imaging optical systems. Specifically, in S704, the system control unit 218 performs focus guide display processing for the left image, and in S705, performs focus guide display processing for the right image. The focus guide display processing for each image may be the focus guide display control processing described using the flowchart in Fig. 11. As a result, an index 801L is displayed in the left image 800L of the live view image 800 shown in Fig. 9, and an index 801R is displayed in the right image 800R. The indexes 801L and 801R are displayed at the same position within the image.

[0134] In S706, the system control unit 218 determines whether or not an operation to end the live view display has been detected. If the system control unit 218 determines that an operation to end the live view display has been detected, the system control unit 218 ends the live view display, and if the system control unit 218 does not determine that an operation to end the live view display has been detected, the system control unit 218 repeats the process from S702 to continue the live view display. Note that although the process from S702 is repeated here, after S702 is processed once, S703 and S706, or S704 to S706 may be repeatedly executed according to the determination result of S702 until the lens unit is removed.

[0135] As described above, in this embodiment, a focus guide is displayed for each live view image formed by the imaging optical system according to the number of optical axes of the attached lens unit. Therefore, when a twin lens unit 300 having two optical axes (imaging optical systems) is attached, a focus guide is displayed for each of the left and right images of the live view image.

[0136] According to this embodiment, for example, when a twin lens unit 300 for taking an image for VR180 is attached to the camera 100, a focus guide is displayed on each of the two live view images on the screen. Therefore, the degree of focus of each live view image can be easily grasped. In addition, when adjusting the focus distance of the imaging optical system, if the focus distances of all the imaging optical systems are adjusted in conjunction with each other, an image to be focused on can be selected as a priority. In addition, if the focus distance can be adjusted for each imaging optical system, the focus distance can be accurately adjusted for each imaging optical system. In either case, manual focus operation when a multi-lens is attached can be effectively supported.

[0137] In this embodiment, the defocus amount obtained as a result of focus detection is used to display the focus guide, but the defocus amount may be calculated using the correction value described in the first embodiment. This makes it possible to realize a focus guide display that shows the focus state with higher accuracy.

[0138] ●(Third embodiment) Next, a third embodiment of the present invention will be described. This embodiment relates to a guide function for manually adjusting the difference in focus degree between the imaging optical systems.

[0139] 13 is a diagram showing an example of an index provided by the focus guide function during live view display when the twin lens unit 300 is attached in this embodiment. In this embodiment, an index is provided that shows the difference in focus level between the focus detection areas of the left and right images.

[0140] Specifically, an index indicating the difference in the degree of focus is displayed based on the defocus amounts of the left eye optical system 301L and the right eye optical system 301R acquired in the focus guide display process in S704 and S705 in Fig. 12. As shown in Fig. 13, the index has an axis 900, an indicator 901 indicating the origin (difference = 0), positive and negative indicators 902 and 903 indicating the direction of the difference, and a mark 904 indicating the difference in the actual degree of focus.

[0141] The difference in the degree of focus is expressed by a relative value and direction based on the defocus amount for one of the imaging optical systems. For example, when the defocus amount of the left eye optical system 301L is used as a reference, the difference in the degree of focus can be obtained by the following formula. Difference in degree of focus = Defocus amount of the right eye optical system 301R - Defocus amount of the left eye optical system 301L The sign of the difference in the degree of focus is positive for front focus and negative for rear focus. Therefore, the difference in the degree of focus is a positive value if the other image (right image) is more front-focused than the reference image (left image), and a negative value if it is rear-focused.

[0142] 13A is an example of a first display form of the index when it is determined that the focus degrees of the left and right images (defocus amounts of the left eye optical system 301L and the right eye optical system 301R) are the same (no difference). When the difference in focus degrees is determined to be 0, the mark 904 is arranged so as to point to the position (reference position) indicated by the indicator 901 on the axis 900. When the difference in focus degrees is determined to be 0, the color and visual effect (such as blinking) of the index may be different from other cases. For example, when the difference in focus degrees is determined to be 0, the index may be green, and otherwise the index may be white.

[0143] 13B and 13C are diagrams showing examples of the second and third display forms, respectively. These display forms are used to show the magnitude and direction of the difference in the focus degree based on the difference in the defocus amount when the difference in the focus degree is not 0 but the reliability of the defocus amount is high.

[0144] 13B shows an example of the second display form of the index when the focus distance of the other image (right image) is shifted to the close side (positive direction) with respect to the reference image (left image), that is, when the right image is in front of the left image. In this case, the mark 904 is positioned to the right (positive direction) of the indicator 901 so as to point to a position moved by a distance according to the difference in the degree of focus.

[0145] 13C shows an example of a third display form of the indicator when the focus distance of the other image (right image) is shifted toward infinity (negative direction) with respect to the reference image (left image), that is, when the right image is in back focus with respect to the left image. In this case, the mark 904 is positioned to the left (negative direction) of the indicator 901 so as to point to a position moved by a distance according to the difference in focus degree.

[0146] In the second and third display forms, the magnitude and direction of the difference in the relative focus degree between the left and right images can be shown to the user depending on the position of the mark 904. Specifically, the magnitude of the difference in the focus degree can be shown depending on the distance of the mark 904 from the position on the axis 900 indicated by the indicator 901 indicating the reference point where the difference is 0. Also, depending on whether the mark 904 is located to the right or left of the position on the axis 900 indicated by the indicator 901, it can be shown whether the focus distance of the other image is shifted to the close side or the infinity side relative to the reference image.

[0147] 13D shows an example of the fourth display form of the index when the reliability of the defocus amount for at least one of the imaging optical systems is low, such as when the left image, the right image, or both are significantly blurred. In this case, the reliability of the difference in the degree of focus obtained by the above formula is also low, so the mark 904 indicating the magnitude and direction of the difference in the degree of focus is not displayed. Also, of the indexes, the axis 900 and the indicators 901 to 903, which are always displayed, may have different colors or visual effects (such as blinking) in the fourth display form from those in the first to third display forms. For example, the display color may be gray.

[0148] Note that the display formats shown in Figures 13(A) to 13(D) are merely examples, and the difference in focus degree may be displayed using other types of indicators, such as using numerical values ​​or using marks or indicators of different shapes.

[0149] Next, a display control process of the difference in the focus degree between the left and right images executed by the system control unit 218 will be described with reference to the flowchart shown in Fig. 14. This process is executed when the twin lens unit 300 is attached to the camera 100. This process is realized by the system control unit 218 expanding a program recorded in the non-volatile memory 220 into the system memory 219 and executing it. The display control process of the difference in the focus degree between the left and right images is executed in parallel with the live view display process. It can also be executed in parallel with the display control process of the focus guide described in the second embodiment.

[0150] In S1001, the system control unit 218 acquires the defocus amount and reliability of the right eye optical system 301R and the left eye optical system 301L from the image processing unit 214. When performing display control processing of the focus guide, the defocus amount and reliability acquired in S602 of Fig. 11 may be referred to. Then, the system control unit 218 determines whether or not the reliability of at least one of the defocus amounts is low. If it is determined that the reliability of at least one of the defocus amounts is low, the system control unit 218 executes S1008, and if not, executes S1002.

[0151] In S1002, the system control unit 218 calculates the difference in the degree of focus based on one of the left and right images. When the left image is used as the reference, the system control unit 218 can calculate the difference in the degree of focus using the above-mentioned calculation formula. After calculating the difference in the degree of focus, the system control unit 218 executes S1003.

[0152] In S1003, the system control unit 218 determines whether the difference in the degree of focus calculated in S1002 is 0 or not. If it is determined that the difference is 0, the system control unit 218 executes S1005, and if it is not determined that the difference is 0, the system control unit 218 executes S1004.

[0153] In S1004, the system control unit 218 determines whether the sign of the difference in the degree of focus calculated in S1002 is positive or negative. If the system control unit 218 determines that the sign of the difference is positive, it executes S1006, and if the system control unit 218 determines that the sign of the difference is negative, it executes S1007.

[0154] In S1005, the system control unit 218 determines to display the index in the first display form (FIG. 13(A)), and executes S1009. In S1006, the system control unit 218 determines to display the index in the second display form (FIG. 13(B)), and executes S1009. In S1007, the system control unit 218 determines to display the index in the third display mode (FIG. 13(C)), and executes S1009. In S1008, the system control unit 218 determines to display the index in the fourth display mode (FIG. 13(D)), and executes S1009.

[0155] In S1009, the system control unit 218 notifies the image processing unit 214 of the display form of the index determined in S1005 to S1008 and the display position of the mark 904 according to the difference in the degree of focus. In accordance with the notification, the image processing unit 214 generates an image of the index according to the display form and writes the image of the index to an address in the video memory area according to the predetermined display position of the index. As a result, the index indicating the difference in the degree of focus between the left and right images is superimposed on the live view image and displayed on the EVF 217 or the display unit 108.

[0156] Next, the live view display operation in this embodiment will be described with reference to the flowchart shown in Fig. 15. In Fig. 15, the same operations as in the second embodiment are denoted by the same reference numerals as in Fig. 12. This process is realized by loading a program recorded in the non-volatile memory 220 into the system memory 219 and executing it by the system control unit 218. This process can be executed when performing live view display, for example, in a shooting standby state or during video shooting.

[0157] The processes in S701 to S705 are the same as those in the second embodiment, and therefore the description will be omitted. When the focus guide display process for the right image in S705 ends, in S1106, the system control unit 218 determines whether or not to display the difference in the focus degree between the left and right images.

[0158] Whether or not to display the difference in the degree of focus between the left and right images may be, for example, one of the user setting items, or may be determined by the system control unit 218 depending on the operation mode of the camera 100. For example, when the video mode of the camera 100 is set to an adjustment mode in which the user adjusts the difference in the degree of focus between the imaging optical systems of the twin lens unit 300, the system control unit 218 determines to display the difference in the degree of focus between the left and right images. In the adjustment mode, the user can adjust the focal distances of the right eye optical system 301R and the left eye optical system 301L by, for example, operating individual focus rings so that there is no difference in the focal distance between the right eye optical system 301R and the left eye optical system 301L.

[0159] If it is determined that the difference in the degree of focus between the left and right images is to be displayed, the system control unit 218 executes S1107, and if not, executes S706. In S1107, the system control unit 218 executes the display control process described using the flowchart in Fig. 14. While viewing the displayed index, the user can adjust the focusing distance of the imaging optical system (here, the right eye optical system 301R) that forms an image other than the reference image so that there is no difference in the focusing degree.

[0160] 16 is a diagram showing an example of the live view display in S1107. As in the second embodiment, one focus guide 1201L and one focus guide 1201R are displayed for the left image 1200L and the right image 1200R in one frame of the live view image 1200 obtained by the imaging element constituting the imaging unit 211. In this embodiment, an index 1202 indicating the difference in the degree of focus between the left and right images is further displayed between the left image 1200L and the right image 1200R.

[0161] The process of S706 is the same as that of the second embodiment, and therefore the description will be omitted. In this embodiment, when the twin lens unit 300 is attached to the camera 100, an index is displayed that indicates the magnitude and direction of the difference between the two imaging optical systems or the degree of focus of the images. This allows the user to easily grasp the shift in the focal distance of the imaging optical systems that has occurred due to aging or the like. The user can adjust the focal distance shift between the imaging optical systems to zero while looking at the index.

[0162] In the second and third embodiments, the focus guide is displayed to assist in manual focus operation. However, the basic technical idea is to display some kind of image to assist in shooting images formed by each imaging optical system when a multi-lens unit is attached, and the display content is not limited to the focus guide indicating the degree of focus. For example, a peaking pattern indicating an overexposed or underexposed area may be superimposed on the live view image.

[0163] In this embodiment, the defocus amount used to display the focus guide may be calculated using the correction value described in the first embodiment. This makes it possible to realize a focus guide display that shows the focus state with higher accuracy.

[0164] ●(Fourth embodiment) Next, a fourth embodiment of the present invention will be described. This embodiment relates to an index that collectively indicates the degree of focus and the difference between the degrees of focus for each image.

[0165] 17(A) to 17(D) show examples of first to fourth display forms of the index according to this embodiment. The index has an axis 1700, an indicator 1701 that is arranged near the center of the axis 1700 and indicates the in-focus position, an indicator 1702 that indicates a back focus state and an indicator 1703 that indicates a front focus state, which are arranged near both ends of the axis 1700. The index also has a mark 1704 that indicates the degree of focus of the right image and a mark 1705 that indicates the degree of focus of the left image, which are arranged near the top and bottom of the axis 1700 so as to indicate a position on the axis 1700.

[0166] 17A is an example of the first display form when both the right and left images are in focus (the defocus amount is 0 for both the right eye optical system 301R and the left eye optical system 301L). In this case, since there is no difference in the degree of focus between the right and left images, the marks 1704 and 1705 are arranged to point to the same position on the axis 1700. Also, since the defocus amount is 0, the marks 1704 and 1705 are arranged to point to the position on the axis 1700 indicated by the indicator 1701. In this case, as with the first display form used when in focus in the second embodiment, at least a part of the indicator may be displayed using a color or visual effect different from that of the other display forms.

[0167] FIG. 17B is an example of the second display form in the case where there is no difference in the degree of focus between the right and left images but the image is in a front focus state. In this case, since there is no difference in the degree of focus between the right and left images, the marks 1704 and 1705 are arranged to point to the same position on the axis 1700 as in the first display form. Also, since the image is in a front focus state, the marks 1704 and 1705 are arranged to point to a position on the axis 1700 that is moved to the end of the indicator 1703 side by a distance corresponding to the magnitude of the defocus amount from the position on the axis 1700 pointed to by the indicator 1701. In order to bring the left and right images into focus while there is no difference in the degree of focus between the left and right images, a display may be displayed that prompts the user to drive a focus ring that simultaneously adjusts the focus distance of the right eye optical system 301R and the left eye optical system 301L.

[0168] FIG. 17C is an example of the third display form in which there is a difference in the degree of focus between the right image and the left image, and neither of them is in focus. Here, it is assumed that both the right image and the left image are in front focus. In this case, the mark 1704 is arranged to point to a position on the axis 1700 that is moved toward the end of the indicator 1703 side by a distance corresponding to the magnitude of the defocus amount of the right eye optical system 301R from the position on the axis 1700 pointed to by the indicator 1701. Also, the mark 1705 is arranged to point to a position on the axis 1700 that is moved toward the end of the indicator 1703 side by a distance corresponding to the magnitude of the defocus amount of the left eye optical system 301L from the position on the axis 1700 pointed to by the indicator 1701. In the example shown in FIG. 17C, since the defocus amount of the left eye optical system 301L is larger, the mark 1705 is arranged to point to a position closer to the end of the indicator 1703 side on the axis 1700 than the mark 1704.

[0169] While looking at the index, the user adjusts the focal distance of the left-eye optical system 301L so that the marks 1704 and 1705 point to the same position on the axis 1700. After that, the left and right images can be brought into focus by simultaneously adjusting the focal distances of the right-eye optical system 301R and the left-eye optical system 301L.

[0170] In this embodiment, the positions on axis 1700 indicated by marks 1704 and 1705 can present to the user the degree of focus of the right and left images, the difference therebetween, and whether the image is in front focus or back focus relative to the focus state.

[0171] In this embodiment, the marks 1704 and 1705 independently indicate the magnitude and direction of the defocus amount of the right-eye optical system 301R and the left-eye optical system 301L, so there is no need to calculate the difference in the degree of focus calculated in the previous embodiment. The system control unit 218 can determine the positions on the axis 1700 to which the marks 1704 and 1705 should point (i.e., the display positions of the marks 1704 and 1705) based on the defocus amount and its sign of the right-eye optical system 301R and the left-eye optical system 301L.

[0172] 17D shows an example of the fourth display form of the index when the reliability of the defocus amount of the right eye optical system 301R and the left eye optical system 301L is low, such as when both the right image and the left image are significantly blurred. In this case, the marks 1704 and 1705 are not displayed because the reliability of the defocus amount is low. Also, of the indexes, the axis 1700 and the indicators 1701 to 1703 that are always displayed may have different colors or visual effects (such as blinking) in the fourth display form from those in the first to third display forms. For example, the display color may be gray.

[0173] Although not shown here, if the reliability of the defocus amount of one of the right eye optical system 301R and the left eye optical system 301L is low and the reliability of the defocus amount of the other is high, mark 1704 or 1705 based on the defocus amount with the higher reliability may be displayed.

[0174] 17(A) to 17(D) are merely examples. The degree of focus and the difference therebetween may be shown using other indicators, such as by showing the magnitude and direction of the defocus amount of the right-eye optical system 301R and the left-eye optical system 301L as values, or by using marks or indicators of different shapes.

[0175] The system control unit 218 can display the indicator of this embodiment in S1107 in a live view display operation in which the processes of S704 and S705 are deleted from the flowchart of FIG. 15 described in the third embodiment. In this case, in S1107, the system control unit 218 acquires the defocus amount and its reliability for each imaging optical system from the image processing unit 214. If the reliability is low, the system control unit 218 determines the fourth display form. If the reliability of the defocus amount is high, the system control unit 218 determines the first display form if all the defocus amounts are 0, the second display form if all the defocus amounts are not 0 and there is no difference, and the third display form if there is a difference in the defocus amounts. In addition, when the system control unit 218 determines the first to third display forms, the system control unit 218 also determines the display positions of the marks 1704 and 1705. Then, the system control unit 218 notifies the image processing unit 214 of the determined display form and the display positions of the marks 1704 and 1705 if they are to be displayed.

[0176] 18 is a diagram showing an example of a live view display in this embodiment. Focus guides 1801L and 1801R are displayed on a left image 1800L and a right image 1800R, respectively, in one frame of a live view image 1800 obtained by an imaging element constituting the imaging unit 211. In this embodiment, an index 1802 indicating the difference in focus between the left and right images, as well as the magnitude and direction of the defocus amount for each imaging optical system, is further displayed between the left image 1800L and the right image 1800R.

[0177] In this embodiment, in addition to the effects of the third embodiment, the magnitude and direction of the defocus amount for each imaging optical system are presented in a different way than in the second embodiment, so that the user can obtain the necessary information by referring to indicators that are easy for the user to understand.

[0178] In this embodiment, too, the basic technical idea is to display some kind of information to assist in shooting images formed by each imaging optical system when a multi-lens unit is attached, and the content of the display is not limited to the degree of focus or the difference therebetween. For example, a peaking pattern indicating overexposed or underexposed areas may be displayed superimposed on the live view image.

[0179] ●(Fifth embodiment) Next, a fifth embodiment of the present invention will be described. This embodiment relates to a technique for automatically adjusting the difference in the degree of focus between the imaging optical systems when a multi-lens unit is attached.

[0180] In the third embodiment, the difference in the degree of focus between the right-eye optical system 301R and the left-eye optical system 301L (the difference in the amount of defocus obtained for the right image and the left image) is obtained. By using this difference to drive the focus lens of the right-eye optical system 301R or the left-eye optical system 301L, the difference in the degree of focus between the imaging optical systems (focus distance deviation) can be automatically adjusted.

[0181] 19 is a flowchart of the process of adjusting the misalignment between the imaging optical systems, which is executed by the system control unit 218. This process is realized by the system control unit 218 expanding a program recorded in the non-volatile memory 220 into the system memory 219 and executing it. The misalignment adjustment process may be executed in response to a user's instruction, or may be executed automatically at a predetermined timing. An example of the predetermined timing may be when the system control unit 218 determines that the attached lens unit is a multi-lens unit (for example, when replacing the lens unit, when starting up the camera 100, etc.).

[0182] When the operation mode of the camera 100 is the adjustment mode described in the third embodiment, the user adjusts the misalignment between the imaging optical systems by manual operation. Therefore, the misalignment adjustment process that is automatically performed by the system control unit 218 in this embodiment may be performed when the operation mode is not the adjustment mode. Note that the misalignment adjustment process is performed in parallel with the live view display operation.

[0183] In S1901, the system control unit 218 sets one of the right eye optical system 301R and the left eye optical system 301L of the twin lens unit 300 as a reference imaging optical system. For example, the twin lens unit 300 is capable of driving the right eye optical system 301R and the left eye optical system 301L in conjunction with each other and driving only the left eye optical system 301L by operating the focus ring. In this case, the system control unit 218 sets the right eye optical system 301R, which cannot be driven independently, as the reference.

[0184] The system control unit 218 may set the imaging optical system corresponding to the eye that is set in advance as the dominant eye of the user as the reference imaging optical system. Alternatively, when there is a difference in subject detection accuracy or focus detection accuracy between the right eye optical system 301R and the left eye optical system 301L, the imaging optical system with the better accuracy may be set as the reference.

[0185] In S1902, the system control unit 218 acquires information on the current focus detection areas for each of the right eye optical system 301R and the left eye optical system 301L, which is stored in the system memory 219. When the focus detection areas for the right eye optical system 301R and the left eye optical system 301L are set at the same position based on the optical axis, it is sufficient to read out information on either one of the focus detection areas. The system control unit 218 notifies the image processing unit 214 of the information on the focus detection areas and instructs it to calculate the defocus amount. The image processing unit 214 calculates the defocus amount and its reliability based on the signals of the focus detection areas for each of the right image and the left image, as described above.

[0186] In S1903, the system control unit 218 acquires from the image processing unit 214 the defocus amount DEF_L of the left eye optical system 301L and the defocus amount DEF_R of the right eye optical system 301R.

[0187] In S1904, the system control unit 218 calculates the difference in the defocus amount of the other imaging optical systems with respect to the defocus amount of the imaging optical system set as the reference in S1901. For example, when the right eye optical system 301R is set as the reference in S1901, the system control unit 218 calculates the difference in the defocus amount DEF_dif by the following formula. DEF_dif = DEF_L - DEF_R

[0188] Next, in S1905, the system control unit 218 drives the focus lens of the non-reference imaging optical system (here, the left eye optical system 301L) in the optical axis direction by PLS_dif, which is the drive amount and drive direction corresponding to DEF_dif detected in S1904. The lens drive amount PLS_dif is, for example, PLS_dif = DEF_dif / SENS_L Ask for more.

[0189] Here, SENS_L is a conversion coefficient for converting the defocus amount of a non-reference (i.e., the imaging optical system to be adjusted) (here, the left-eye optical system 301L) into a lens drive amount, and is stored in advance in the lens unit. The twin lens unit 300 holds the focus sensitivity for each of the right-eye optical system 301R and the left-eye optical system 301L in a non-volatile memory inside the lens system control circuit 303. Note that when there are multiple imaging optical systems with the same configuration like the twin lens unit 300, one focus sensitivity may be used in common for each imaging optical system.

[0190] By the above processing, it is possible to automatically adjust the difference in the degree of focus between the imaging optical systems (deviation in the focal distance) to eliminate it. According to this embodiment, the difference in the degree of focus between the imaging optical systems (deviation in the focal distance) that was manually adjusted in the third embodiment can be automatically adjusted, so that it is possible to eliminate the user's trouble of making adjustments and improve usability.

[0191] ●(Sixth embodiment) Next, a sixth embodiment of the present invention will be described. In this embodiment, a focusing calibration function is provided when a multi-lens unit is attached.

[0192] Fig. 20 shows the indexes described in the fourth embodiment with reference to Fig. 17. Fig. 20(A) shows the third display form, in which marks 1704 and 1705 indicate that the right image is in focus, but the left image is in front focus. From this state, it is assumed that the user operates the focus ring to adjust the focal distance of the left eye optical system 301L so that the mark 1704 indicating the degree of focus of the left image comes to the position indicated by the indicator 1701 (so that the index is in the state shown in Fig. 20(B)).

[0193] As a result, for example, both the right image 1800R and the left image 1800L of the live view image 1800 displayed in Fig. 18 should be in focus. However, due to factors such as manufacturing errors and aging of components such as multiple lenses and reflecting mirrors included in the right eye optical system 301R and the left eye optical system 301L, and environmental factors, a discrepancy may occur between the calculated defocus amount and the degree of focus of the image.

[0194] Therefore, even if the marks 1704 and 1705 indicate an in-focus state, the user may feel that the right and / or left image observed through the display unit 108 or EVF 217 is not in focus. In particular, when the degree of focus of one image is lower than that of the other image, the two images are displayed adjacent to each other, making it easy for the user to notice the difference in the degree of focus. For this reason, in this embodiment, a function (focus calibration function) is provided to correct the difference between the state in which an image that the user feels is in focus is obtained and the state in which the camera 100 determines that the image is in focus.

[0195] In this embodiment, a correction value for correcting the difference between a state in which an image that the user feels is in focus and a state in which the camera 100 determines the image to be in focus is called a calibration value. The calibration value can be set and held independently for each of the right eye optical system 301R and the left eye optical system 301L.

[0196] 21 is a diagram showing an example of a calibration lane guide displayed on the display unit 108 or the EVF 217 when the focusing calibration function is executed. Here, for ease of explanation and understanding, it is assumed that the focus lens of the left eye optical system 301L and the focus lens of the right eye optical system 301R can be driven independently. However, the focusing calibration function can be realized even if the focus lenses of both imaging optical systems are driven in conjunction with each other and the focus lens of one imaging optical system can be driven independently. In this case, the calibration of the imaging optical system whose focus lens cannot be driven independently is performed first, and then the calibration of the remaining imaging optical systems is performed.

[0197] FIG. 21(A) shows an example of a first display form of a calibration guide 2110. The calibration guide 2110 has an axis 2100 corresponding to a range of calibration values. A scale is provided on the axis 2100, and values ​​are indicated on some scales. Here, an indicator 2101 indicating a calibration value of 0, an indicator 2102 indicating the negative maximum value (here, -20), and an indicator 2103 indicating the positive maximum value (here, +20) are displayed near the lower part of the axis 2100. The sign of the calibration value is set so that positive indicates the front focus direction and negative indicates the back focus direction so that the positional relationship between the mark and the reference is the same as that in FIG. 17. Note that indicators indicating values ​​other than these may be added. In the example of FIG. 21(A), indicators indicating -10 and +10 are added.

[0198] Further, a mark 2104 indicating the calibration value of the right-eye optical system 301R and a mark 2105 indicating the calibration value of the left-eye optical system 301L are disposed near the axis 2100. Furthermore, marks 1704 and 1705 indicating the degree of focus of the right-eye optical system 301R and the left-eye optical system 301L are disposed on the axis 2100. Here, the display positions of the marks 1704 and 1705 are determined by setting the position of the calibration value 0 as the defocus amount 0. Further, an indicator 2106 indicating the difference in the calibration values ​​of the right-eye optical system 301R and the left-eye optical system 301L as a numerical value is disposed near the left end of the axis 2100. Here, the relative value of the calibration value of the right-eye optical system 301R when the calibration value of the left-eye optical system 301L is set to 0 is shown as the difference in the calibration values.

[0199] 21A shows a state in which the defocus amount of the right-eye optical system 301R and the left-eye optical system 301L are both 0, but the focus lens position is corrected by the calibration value. Specifically, it shows a state in which the focus lens position corresponding to the defocus amount of 0 is corrected by 2 in the front focus direction (+) for the right-eye optical system 301R, and by 4 in the back focus direction (-) for the left-eye optical system 301L. Here, the unit of the calibration value is predetermined by the camera 100. For example, it may be the number of pulses when driving the focus lens.

[0200] With the marks 1704 and 1705 indicating the in-focus state, the user adjusts the focus lens positions of the right eye optical system 301R and the left eye optical system 301L, for example, while observing the right image 1800R and the left image 1800L of the live view image 1800. Then, when the user feels that the degree of in-focus in the focus detection areas of the right image 1800R and the left image 1800L is highest, the user issues an instruction to the camera 100 via the operation member 230. When the system control unit 218 detects this instruction, it stores the calibration values ​​of the right eye optical system 301R and the left eye optical system 301L at that time in the non-volatile memory 220.

[0201] The system control unit 218 corrects the lens driving amount based on the defocus amount based on the calibration value stored in the nonvolatile memory 220, and then transmits it to the lens system control circuit 303. This makes it possible to obtain an image that the user feels is most in-focus when the defocus amount is 0. In other words, it is possible to correct the discrepancy between the degree of focus felt by the user and the degree of focus determined by the camera 100.

[0202] FIG. 21B shows an example of the second display form of the calibration guide 2110. In the second display form, the information presented in the first display form is shown for each imaging optical system. Specifically, axes 2100R and 2100L are provided for each imaging optical system, and a mark 1704 (1705) indicating the degree of focus and a mark 2104 (2105) indicating the calibration value are arranged on the corresponding axis. Also, instead of an indicator 2106 indicating the difference in the calibration value, indicators 2106R and 2106L indicating the calibration value for each imaging optical system are arranged on the corresponding axis 2100R and 2100L. In the second display form, since information is presented for each imaging optical system, the display area is large, but the user can easily grasp the information. Note that information regarding one imaging optical system according to a user's instruction may be displayed.

[0203] FIG. 21C shows an example of the third display form of the calibration guide 2110. This display form is the same as the first display form shown in FIG. 21A, but both the marks 1704 and 1705 are at +4 and in a front focus state. The marks 2104 and 2105 indicating the calibration value indicate the calibration value at the position when the marks 1704 and 1705 are at the position of the defocus amount 0 (FIG. 21A). When the marks 1704 and 1705 are not at the position of the defocus amount 0, the marks 2104 and 2105 indicate the calibration value by the difference in position with the marks 1704 and 1705. Therefore, in FIG. 21C, the calibration value of the right eye optical system 301R is -2, and the calibration value of the left eye optical system 301L is 0. Note that the calibration value may be indicated by the display positions of the marks 2104 and 2105, regardless of the defocus amount of the right eye optical system 301R and the left eye optical system 301L.

[0204] A method of focus calibration in a state in which the defocus amounts of the right-eye optical system 301R and the left-eye optical system 301L are not 0 will be described with reference to FIG. Fig. 22(A) shows an example of a sample image corresponding to the focus degree in Fig. 21(C). When focus calibration is performed with the defocus amount not 0, the system control unit 218 displays a sample image corresponding to the current focus degree and a live view image (Fig. 22(B)) on the display unit 108 or the EVF 217.

[0205] When calibrating the focus state as in Fig. 21(A), the sample image in Fig. 22(A) is not necessary. The sample image in Fig. 22(A) is, for example, an image corresponding to the current focus degree predicted from the design state of the imaging optical system by using a line image previously recorded in the non-volatile memory 220.

[0206] The user compares the image in Fig. 22(A) with the live view image in Fig. 22(B) (the left image 2201L and the right image 2201R). For example, if only the right image 2201R in the live view image in Fig. 22(B) appears to have a large amount of blur, the user adjusts the focus lens position of the right eye optical system 301R to a position where the right image 2201R is felt to have the same degree of focus as the left image 2201L. This sets the calibration value of the right eye optical system 301R.

[0207] 21C corresponds to a display example of the calibration guide after the calibration value of the right-eye optical system 301R is set in this way. By changing the calibration amount of the right-eye optical system 301R from the original value (0) to -2, the blur states of the left and right images match, and a desired image can be obtained. Note that, without displaying a sample image, the focus lens position of the imaging optical system that forms the other image may be adjusted so that the left and right images in the live view image have the same degree of focus as one that is perceived to have a higher degree of focus.

[0208] The calibration values ​​may be stored in at least one of the camera 100 and the twin lens unit 300. This allows the calibration values ​​to be obtained from the twin lens unit 300 and used in a camera other than the camera that performed the calibration of the twin lens unit 300.

[0209] According to this embodiment, it is possible to set a calibration value for correcting the focus lens position for the imaging optical system of the twin lens unit, which makes it possible to correct the difference between the focus state determined by the camera and the focus state perceived by the user for the twin lens unit.

[0210] ●(Seventh embodiment) Next, a seventh embodiment of the present invention will be described. This embodiment relates to a focusing calibration function in the case where a pair of parallax images such as a left image and a right image captured by a twin lens unit 300 are recorded with a refocusable lock k. A refocusable image is an image in which the subject distance at which the image is focused can be changed after shooting (recording). For example, it may be an image captured by a light field camera, or an image in which a group of images captured of the same scene with different focus distances are associated and recorded.

[0211] The focus calibration function provided by this embodiment can be used, for example, to set a calibration value for correcting the difference in the focus degree (focus state) between the right and left images when a user views a recorded pair of parallax images. Here, it is assumed that the user views the pair of parallax images by wearing XR goggles, which are display devices having a display unit for the left eye and a display unit for the right eye. Here, XR is a general term for VR (virtual reality), AR (augmented reality), and MR (mixed reality).

[0212] The left diagram of FIG. 23(a) is a perspective view showing an example of the appearance of the XR goggles 2300. As shown in the right diagram of FIG. 23(a), the XR goggles 2300 are generally worn on the eye area SO of the head. FIG. 23(b) is a perspective view showing an example of the appearance of the XR goggles 2300 seen from the wearing surface side. Also, FIG. 23(c) is a diagram showing a schematic positional relationship between the eyepiece lenses 2301R and 2301L, the right eye display unit 2308R and the left eye display unit 2308L, and the user's right eye 501R and left eye 501L when the XR goggles 2300 are worn.

[0213] The XR goggles 2300 displays, for example, a right image obtained by the twin lens unit 300 on the right eye display unit 2308R and a left image on the left eye display unit 2308L. Since the right and left images are a pair of parallax images, the user recognizes the right image with the right eye 501R through the eyepiece 302R and the left image with the left eye 501L through the eyepiece 302L, thereby recognizing the right and left images as 3D images. Note that the pair of parallax images displayed on the XR goggles 2300 is not limited to those captured by the twin lens unit 300. For example, stereoscopic vision is possible even if the right and left images captured by a stereo camera are displayed on the right eye display unit 2308R and the left eye display unit 2308L of the XR goggles 2300.

[0214] This embodiment assumes that the focal distances of the right and left images constituting the pair of parallax images displayed on the XR goggles 2300 can be changed independently. For example, video data in which the right and left images are recorded in a refocusable format may be played back on a computing device and viewed on the XR goggles 2300 connected to the computing device.

[0215] Fig. 24 is a diagram showing a schematic diagram of a change in the focused subject by the refocusing process. Fig. 24(a) shows a state where the subject 2403 is in focus, but the subjects 2402 and 2404 are not in focus. Note that the images of the subjects 2402 and 2404 are not in focus because they are not blurred and are outside the depth of field.

[0216] If the image shown in Figure 24(a) (either a still image or a frame of a video) was recorded in a format that allows for refocusing, it can be altered to focus on object 2402 or object 2404. Figures 24(b) and 24(c) show the image altered to focus on objects 2402 and 2404, respectively.

[0217] For example, when a position to be focused is specified by a user, the system control unit 218 changes the image so that the specified position is focused. Any known method can be used to change the focused subject by the refocus process. When a refocusable image is captured by the camera 100, the number of photodiodes sharing the microlens 401 may be increased in both the horizontal and vertical directions.

[0218] Fig. 25 shows an example of a right eye image 2501R and a left eye image 2501L corresponding to the image shown in Fig. 24(a). By displaying the right eye image 2501R and the left eye image 2501L on the right eye display unit 2308R and the left eye display unit 2308L of the XR goggles 2300, a user wearing the XR goggles 2300 can get a sense of being in the scene shown in Fig. 24(a). By detecting the movement of the user's hand and associating it with a position in the image, the user can virtually touch the subject 2403 to specify the subject 2403 as the subject to be focused on.

[0219] FIG. 26 is a block diagram showing an example of the functional configuration of a computer 2600 that can be used as a computing device in this embodiment.

[0220] The display 2701 displays information on data being processed by an application program, various message menus, etc., and is composed of an LCD (Liquid Crystal Display) or the like. The display 2701 may be a touch display. A display controller 2702 controls the screen display on the display 2701. A keyboard 2703 and a pointing device 2704 are used to input characters, etc., and to point to icons and buttons in a GUI (Graphical User Interface). A CPU 2705 controls the entire computer 2600.

[0221] A ROM 2706 (Read Only Memory) stores programs and parameters executed by the CPU 2705. A RAM (Random Access Memory) 2707 is used as a work area when the CPU 2705 executes various programs, a temporary save area during error processing, and the like.

[0222] A hard disk drive (HDD) 2708 and a removable media drive (RMD) 2709 function as external storage devices. The removable media drive is a device that reads and writes data from or to a removable recording medium, and may be a flexible disk drive, an optical disk drive, a magneto-optical disk drive, a memory card reader, or a removable HDD. In addition to or instead of the HDD 2708, an SSD (Solid State Drive) may be provided.

[0223] Programs for implementing various functions of the computer 2600 described in this embodiment, an OS, application programs such as a browser, data, libraries, and the like are stored in one or more of a ROM 2706, a HDD 2708, and an RMD 2709.

[0224] The expansion slot 2710 is a slot for mounting an expansion card that complies with, for example, the PCI (Periferal Component Interconnect) bus standard. Various expansion boards such as a video capture board, a sound board, and a GPIB board can be mounted in the expansion slot 2710.

[0225] The external IF 2711 is an interface for connecting the computer 2600 and an external device so that they can communicate with each other, and complies with one or more wired and / or wireless communication standards. The external IF 2711 may have an interface that complies with one or more of, for example, wireless LAN, USB (Universal Serial Bus), HDMI (registered trademark), Bluetooth (registered trademark), 4G (LTE), 5G, and the like.

[0226] The bus 2712 comprises an address bus, a data bus, and a control bus, and connects the above-mentioned units.

[0227] Next, the focusing calibration function in this embodiment will be described. Here, the focusing calibration function for the pair of parallax images displayed on the XR goggles 2300 is provided by executing an application program that provides the focusing calibration function in the above-mentioned computer 2600. However, the focusing calibration function can also be provided by the system control unit 218 of the camera 100 executing a similar program.

[0228] Here, it is assumed that data of a pair of parallax images (right and left images) is recorded in a refocusable format in a storage device (e.g., HDD 2708) of the computer 2600. The data of the right and left images may be acquired from the camera 100 or other external devices via an external IF 2711. Also, as shown in Fig. 27(a), XR goggles 2300 are connected to the external IF 2711 of the computer 2600, and the right and left images can be displayed.

[0229] The focus calibration function provided by the computer 2600 displays the parallax image pair 2602 and the calibration guide 2601 on the display 2701 .

[0230] 27(b) shows an example of a calibration guide 2601. The calibration guide 2601 has an axis 2610 indicating a distance range from the closest end to infinity. A scale is provided on the axis 2610, and values ​​are indicated at some of the scale marks. Here, an indicator 2611 indicating a minimum value (close end) and an indicator 2612 indicating a maximum value (infinity) are displayed near the bottom of the axis 2610. Indicators indicating values ​​for some distances between the minimum and maximum values ​​are also displayed.

[0231] Also, a mark 2613 indicates the position of the axis 2610 corresponding to the focal distance when the currently displayed disparity image pair was captured. The focal distance when capturing is recorded together with the image data as one piece of information when capturing.

[0232] In the example shown in FIG. 27(b), the mark 2613 indicates 1 m. The image at this time is assumed to be, for example, an image focused on the subject 2403 shown in FIG. 24(a). By moving the mark 2613 to change the distance indicated by the mark 2613, the user can indicate the changed focus distance to the CPU 2705. The user can move the mark 2613 to a desired position by operating the keyboard 2703 or the pointing device 2704, or by touching the display 2701.

[0233] For example, as shown in FIG. 27(c), it is assumed that the user moves the mark 2613 to a position where the axis 2610 indicates 2 m. The CPU 2705 changes the focal distance of the image data according to the changed position of the mark 2613. The CPU 2705 can change the focal distance by a method according to the recording method. When a light field image is recorded, the focal distance can be changed by a shift calculation. Also, when a group of images with different focal distances are recorded, it is sufficient to extract an image focused at a specified distance (an image whose depth of field includes the specified distance). Here, it is assumed that the image shown in FIG. 24(b) is an image with a focal distance of 2 m, and the image shown in FIG. 24(c) is an image with a focal distance of 3 m.

[0234] There should be no difference in the degree of focus between the right and left images because they are images focused at the distance indicated by mark 2613. Differences in the degree of focus between the right and left images can occur due to factors such as manufacturing errors, deterioration over time of components such as multiple lenses and reflecting mirrors included in the two imaging optical systems that form the right and left images, and environmental factors.

[0235] Therefore, the user may feel that the focus levels of the right and left images observed through the display 2701 or the XR goggles 2300 are different. In particular, when the focus level of one image is lower than that of the other image, the two images are displayed adjacent to each other, so that the user is likely to notice the difference in focus level. Therefore, in this embodiment, a focus calibration function is provided to correct the difference between the focus level expected by the user for the distance specified by the user and the focus level of the displayed image.

[0236] Fig. 27(d) is a diagram showing an example of the display form of the calibration guide 2601 when setting the calibration values. In this embodiment, as shown in Fig. 27(a), the calibration values ​​of the left and right images are set by finely adjusting the focus distance while checking the focus degree of the parallax image pair 2602 displayed on the display 2701.

[0237] Fig. 27(d) shows a state where the CPU 2705 changes the display form of the calibration guide 2601 in response to a user's instruction to transition from the state of Fig. 27(c) to a mode for setting calibration values ​​(calibration mode). Therefore, the currently displayed disparity image pair 2602 is an image presented by the CPU 2705 as an image focused at a subject distance of 2 m.

[0238] Here, when transitioning to the calibration mode, the indicator given to the scale of the axis 2610 is changed from a state showing distance to a state showing a numerical value with the current setting value set to 0. This is to make it easier to grasp the magnitude and direction of the calibration value. Note that the state showing distance may be maintained as shown in FIG. 26(c).

[0239] Furthermore, when the CPU 2705 transitions to the calibration mode, it displays an indicator 2624 that indicates the current calibration value numerically. Furthermore, the CPU 2705 displays a mark 2623 for setting the calibration value of the right image and a mark 2622 for setting the calibration value of the left image. The user can move the marks 2622 and 2623 by touching the keyboard 2703, the pointing device 2704, the display 2701, or the like.

[0240] When the CPU 2705 detects a movement operation of the mark 2623, the CPU 2705 changes the focal distance of the right image of the parallax image pair 2602 according to the movement direction and the movement amount from the initial position (0). For example, when the CPU 2705 detects a movement of the mark 2623 in the + direction, the CPU 2705 changes the focal distance toward infinity, and when the CPU 2705 detects a movement of the mark 2623 in the - direction, the CPU 2705 changes the focal distance toward the closest end. Note that the amount of change in distance per scale may be a predetermined constant value. When the CPU 2705 detects a movement operation of the mark 2622, the CPU 2705 changes the focal distance of the left image in the same way as the right image. The user operates the marks 2622 and 2623 so that the right and left images feel as if they are focused at the specified distance.

[0241] 27(d) shows a state in which the calibration value of the left image is 0 and the calibration value of the right image is -4. This shows that there is no need to correct the set distance for the left image, and by adjusting the set distance for the right image to the close end by the calibration value of -4, the in-focus area of ​​the right image can be felt to be the same as the in-focus area of ​​the left image.

[0242] When the calibration is completed, the user instructs the computer 2600 to end the calibration by touching the keyboard 2703, the pointing device 2704, the display 2701, or the like.

[0243] When the CPU 2705 detects this instruction, it stores the calibration value set at that time as a setting value of the application, for example, in the HDD 2708. When updating the parallax image pair 2602, the CPU 2705 reflects the calibration value in the setting value of the distance. Note that the calibration value can be applied not only when playing back the image data used for the setting, but also to other image data captured by the same device as the device that captured the image data. In addition, when a calibration value is set for one frame of video data, the calibration can be applied to other frames. The set calibration value may be applied automatically, or may be applied when the user instructs application.

[0244] According to this embodiment, it is possible to provide a function for calibrating the focal distance for a pair of parallax images recorded in a refocusable format.

[0245] (Modification) An indicator showing the hyperfocal distance may be added to the calibration guide 2601. This allows the user to easily obtain a deep focus image. The hyperfocal distance can be calculated from the focal length and F-number of the lens unit recorded together with the image data as information at the time of shooting, and the allowable circle of confusion diameter, which is, for example, the pixel pitch of the image sensor.

[0246] (Other embodiments) In the above-described embodiment, the display positions of the marks or indicators relating to the right and left images may be reversed. In addition, the form of the GUI (Graphical User Interface) constituting the various indicators and guides is not limited to that shown in the drawings. Any form of GUI that can present the same information as the indicators and guides exemplified in the embodiment to the user may be used.

[0247] Furthermore, the various controls described above as being performed by the system control unit 218 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0248] In addition, although the present invention has been described in detail based on the exemplary embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0249] In the above-mentioned embodiment, the present invention is applied to a digital camera (imaging device), but the present invention is not limited to this example and can be applied to any display control device that can display display items related to focus. In other words, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, printer devices equipped with displays, digital photo frames, music players, game consoles, electronic book readers, etc.

[0250] The present invention is also applicable not only to the imaging device itself, but also to a control device that communicates with an imaging device (including a network camera) via wired or wireless communication and remotely controls the imaging device. Examples of devices that remotely control an imaging device include a smartphone, a tablet PC, and a desktop PC. The imaging device can be remotely controlled by notifying the control device of commands for performing various operations and settings based on operations performed on the control device side or processes performed on the control device side. Also, a live view image captured by the imaging device can be received via wired or wireless communication and displayed on the control device side.

[0251] Examples of embodiments disclosed herein are listed below. [Embodiment 1] an image sensor having a plurality of focus detection pixels that receive light beams passing through different pupil partial regions of an imaging optical system; a focus detection unit that obtains a defocus amount of the imaging optical system based on a pair of signals from the focus detection pixels; A focus detection adjustment value used when acquiring the defocus amount; In an imaging device having an imaging device characterized in that the imaging optical system has a plurality of optical axes, the focus detection adjustment values ​​correspond to the plurality of optical axes, optical information for each optical axis of the imaging optical system is converted based on the position of each optical axis on the imaging device, and a focus detection adjustment value in the imaging optical system is calculated from the converted optical information. [Embodiment 2] The imaging device described in embodiment 1, characterized in that the focus detection adjustment value includes at least one of a conversion coefficient for calculating a defocus amount or a correction value used in a correction means for suppressing the difference in intensity between a pair of focus detection signals. [Embodiment 3] The imaging device described in embodiment 1 or 2, characterized in that the imaging device has focus detection adjustment values ​​when an imaging optical system having one optical axis at the center of the imaging element is attached, and focus detection adjustment values ​​when an imaging optical system having multiple optical axes is attached at a position different from the center of the imaging element. [Embodiment 4] An imaging device as described in embodiment 3, characterized in that the focus detection adjustment value to be used is switched based on information determining whether the imaging optical system is an imaging optical system having one optical axis at the center of the image sensor, or an imaging optical system having multiple optical axes at a position different from the center of the image sensor. [Embodiment 5] An imaging device as described in embodiment 4, characterized in that the information is stored in an imaging optical system, and when attached to the imaging device, the imaging device acquires the information via communication. [Embodiment 6] An imaging device as described in embodiment 5, characterized in that the information includes an optical axis position on the imaging element. [Embodiment 7] an image sensor having a plurality of focus detection pixels that receive light beams passing through different pupil partial regions of an imaging optical system; a focus detection unit that obtains a defocus amount of the imaging optical system based on a pair of signals from the focus detection pixels; A calculation means for calculating a correction value for correcting the defocus amount; In an imaging device having an imaging device characterized in that the correction value is composed of coefficients centered on an optical axis of the imaging optical system, the imaging optical system having a plurality of optical axes, and the calculation means calculates the correction value by using the coordinates of the optical axis position of each optical axis on an imaging element and the coefficients. [Embodiment 8] 8. The imaging device according to claim 7, wherein the correction value is a correction value for correcting the focus detection result to a more suitable in-focus position for imaging. [Embodiment 9] An imaging device as described in embodiment 7 or 8, characterized in that the coordinates of the optical axis position are stored in the imaging optical system, and when attached to the imaging device, the imaging device acquires information on the optical axis position via communication. [Embodiment 10] An electronic device that displays a captured image in live view, an acquisition means for acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same direction as the first optical system and having a parallax with respect to the first live view image; a display control means for controlling the display item indicating information related to focus to be superimposed on the first live view image and the second live view image so that the first display item and the second display item are displayed at the same image height; 1. An electronic device comprising: [Embodiment 11] An electronic device that displays a captured image in live view, a first live view image captured via a first optical system; an acquisition means for acquiring a second live view image that is captured through a second optical system having the same orientation as the first optical system and has parallax with respect to the first live view image; a display control means for controlling a third display item indicating a difference in information regarding a focus between the first live view image and the second live view image to be superimposed on the live view image; 1. An electronic device comprising: [Embodiment 12] 12. The electronic device according to claim 10 or 11, wherein the focus information is information regarding focus for a focus detection area based on a display position. [Embodiment 13] The electronic device described in embodiment 10, characterized in that the display control means moves in tandem the position of a display item indicating information regarding focus, which is displayed superimposed on the first live view image and the second live view image. [Embodiment 14] The electronic device described in embodiment 10 or 13, characterized in that the display control means displays the display positions of the first display item and the second display item at the same image height based on the optical axis position of the first optical system and the optical axis position of the second optical system, respectively. [Embodiment 15] The electronic device described in embodiment 10, characterized in that the display control means selectively displays a display item indicating information regarding focus in either the first live view image and the second live view image, only the first live view image, or only the second live view image. [Embodiment 16] The electronic device described in embodiment 10 or 11, characterized in that the display control means controls when a lens having a first focus ring for adjusting the focus of the first live view image and a second focus ring for adjusting the focus of the second live view image is attached. [Embodiment 17] An electronic device as described in embodiment 16, characterized in that the display color of the first display item is the same as the color of the first focus ring, and the display color of the second display item is the same as the color of the second focus ring, thereby making the display colors of the first display item and the second display item different. [Embodiment 18] The electronic device described in embodiment 10 or 11, characterized in that the display control means controls when a lens is attached, the lens having a third focus ring that adjusts the focus of the first live view image and the second live view image simultaneously and a fourth focus ring that adjusts the focus of either the first live view image or the second live view image. [Embodiment 19] An electronic device as described in embodiment 18, characterized in that the display color of a display item whose focus can be adjusted using only the third focus ring is different from the display color of a display item whose focus can be adjusted using the third and fourth focus rings. [Embodiment 20] The electronic device according to embodiment 10 or 11, characterized in that the display control means displays the first display item and the second display item when displaying the third display item. [Embodiment 21] The electronic device described in embodiment 10 or 11, characterized in that the display control means displays the third display item indicating the difference between the information regarding the first focus displayed in the first display item and the information regarding the second focus displayed in the second display item as difference information based on the information regarding the first focus or based on the information regarding the second focus. [Embodiment 22] The electronic device described in embodiment 10 or 11, characterized in that the display control means displays the third display item only when performing an adjustment mode for adjusting the difference between information regarding the first focus displayed on the first display item and information regarding the second focus displayed on the second display item. [Embodiment 23] an acquiring step of acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system having the same orientation as the first optical system and having a parallax with respect to the first live view image; a display control step of superimposing a display item indicating information related to focus on the first live view image and the second live view image, and controlling the first display item and the second display item to be displayed at the same image height; 13. A method for controlling an electronic device comprising: [Embodiment 24] an acquiring step of acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system having the same orientation as the first optical system and having a parallax with respect to the first live view image; a display control step of controlling to display a third display item indicating a difference in information regarding focus between the first live view image and the second live view image so as to be superimposed on the live view image; 13. A method for controlling an electronic device comprising: [Embodiment 25] A program for causing a computer to function as each of the means of an electronic device described in any one of embodiments 10 to 22. [Embodiment 26] A computer-readable storage medium storing a program for causing a computer to function as each of the means of an electronic device described in any one of embodiments 10 to 22. [Embodiment 27] An electronic device that displays a captured image in live view, an acquisition means for acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same direction as the first optical system and having a parallax with respect to the first live view image; a first display item showing information related to a focus of the first live view image; and a second display item showing information related to a focus of the second live view image. a display control means for controlling the display so as to be superimposed on the live view display; 1. An electronic device comprising: [Embodiment 28] An electronic device as described in embodiment 27, characterized in that the display control means displays a plurality of the first display items and the second display items superimposed on the live view display. [Embodiment 29] The electronic device described in embodiment 27, characterized in that the first display item and the second display item are displayed at positions based on the optical axis positions of the first optical system and the second optical system. [Embodiment 30] An electronic device as described in embodiment 27 or 29, characterized in that when the display position of the first or second display item in the live view display is changed, the first or second display item also moves according to the optical axis positions of the first optical system and the second optical system. [Embodiment 31] an acquisition means for acquiring the first defocus amount detected by phase difference detection using a pair of light beams passing through different exit pupils of the first optical system including a first focus ring, and the second defocus amount detected by phase difference detection using a pair of light beams passing through different exit pupils of the second optical system including a second focus ring; an adjustment unit that automatically adjusts a drive amount of the first focus ring or the second focus ring based on a difference between the first defocus amount and the second defocus amount; An imaging device comprising: [Embodiment 32] An imaging device that displays a captured image in live view, an acquisition means for acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same direction as the first optical system and having a parallax with respect to the first live view image; an adjustment means for adjusting the focus states of the first and second live view images in the same manner; A storage means for storing the calibration value set by the adjustment means, An imaging device, characterized in that the calibration value can be applied at any timing. [Embodiment 33] A display device that displays a pair of parallax images including a first image and a second image, an adjustment means for adjusting the focus states of the first image and the second image in the same manner; a storage means for storing the calibration value set by the adjustment means; having A display device capable of changing a focus state by applying the calibration value to an image. [Embodiment 34] The display device according to embodiment 33, characterized in that the change in focus state by the adjustment means is performed by selecting from a plurality of images having different focus states. [Embodiment 35] 35. The display device according to claim 34, wherein the change in focus state by the adjustment means is performed by a refocus process. [Embodiment 36] A display device described in any one of embodiments 33 to 35, characterized in that the calibration value is set in one frame and applied to other frames when the image is a video.

[0252] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0253] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0254] 100: camera body, 108: display unit, 200: single lens unit, 211: imaging unit, 214: image processing unit, 218: system control unit, 300: twin lens unit

Claims

1. an image sensor capable of generating a signal pair used for focus detection using a phase difference detection method; a detection means for detecting a defocus amount based on the signal pair; a control means for performing focus adjustment control based on the defocus amount, the detection means detects the defocus amount using adjustment values ​​based on a plurality of optical axis positions of a multi-lens unit including a plurality of imaging optical systems having different optical axes.

1. An imaging device comprising:

2. 2. The imaging device according to claim 1, wherein the adjustment value is a shading correction value used to make the intensities of the signal pairs uniform.

3. 3. The image pickup apparatus according to claim 1, wherein the adjustment value is a conversion coefficient for converting an amount of deviation of the signal pair into an amount of defocus.

4. 4. The image pickup apparatus according to claim 3, wherein the conversion coefficient is based on the position of the optical axis that is closest to a focus detection area among the optical axes of the plurality of image pickup optical systems.

5. 5. The imaging apparatus according to claim 1, wherein the adjustment value is a correction value for correcting a focus distance based on the defocus amount.

6. 6. The imaging apparatus according to claim 1, further comprising an acquisition unit that acquires the optical axis positions from the multi-lens unit.

7. 6. The imaging apparatus according to claim 1, further comprising a storage unit for storing the positions of the optical axes of the multi-lens unit.

8. 8. The imaging device according to claim 1, wherein the multi-lens unit is a lens unit in which the plurality of imaging optical systems are provided within a single lens barrel.

9. 9. The imaging device according to claim 1, wherein at least a portion of the pixels of the imaging element have a plurality of photodiodes sharing a microlens provided in the pixel, and the signal pair is generated from a signal of the pixel having the plurality of photodiodes.

10. 9. The imaging device according to claim 1, wherein the imaging element has a plurality of pixels, and at least some of the plurality of pixels are dedicated pixels for generating the signal pairs.

11. A method performed by an imaging device, comprising: a detection step of detecting a defocus amount based on a signal pair used for focus detection using a phase difference detection method; A control step of performing focus adjustment control based on the defocus amount, In the detection step, the defocus amount is detected using an adjustment value based on a plurality of optical axis positions of a multi-lens unit including a plurality of imaging optical systems having different optical axes. A method comprising:

12. A program for causing a computer to execute each step of the method according to claim 11.

13. A computer-readable storage medium storing a program for causing a computer to execute each step of the method according to claim 11.

Images