Imaging apparatus and method

Abstract

To provide an imaging apparatus capable of focus detection of an imaging surface phase difference detection system even when a lens unit having a plurality of optical axes is mounted thereon.SOLUTION: An imaging apparatus has: an image pick-up device that can generate a signal pair used for focus detection of a phase difference detection system; calculation means that calculates the focal length of a lens unit mounted on the apparatus on the basis of a defocus amount obtained by using the signal pair; and adjustment means that adjusts the focal length of the lens unit on the basis of the focal length. When the lens unit is a multiple-lens unit having a plurality of image forming optical systems with different optical axes, the calculation means calculates the focal length by using an adjustment value obtained on the basis of an optical axis position that is a position on the image pick-up device through which the optical axes of the multiple-lens unit pass.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an imaging device and method, and more particularly to an imaging device and method that can use a lens unit having a plurality of optical axes. [Background technology]

[0002] Conventionally, a stereoscopic camera equipped with multiple imaging optical systems and capable of capturing stereoscopic images with a single imaging element has been known (Patent Document 1). Meanwhile, in recent years, due to factors such as the falling prices of VR goggles, there has been a demand for a more convenient method of capturing stereoscopic images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-205558 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, by incorporating two imaging optical systems into one lens barrel and providing them as an interchangeable lens unit, it is possible to capture a stereoscopic image using a general interchangeable lens imaging device.

[0005] However, the image-plane phase-difference focus detection method currently used primarily in mirrorless cameras is based on the premise that the lens unit has a single optical axis, and therefore, when a lens unit with multiple optical axes is attached, such as a lens unit with two imaging optical systems built into a single lens barrel, the accuracy of focus detection can decrease.

[0006] One of the objects of the present invention is to provide an imaging device and method that are capable of performing highly accurate focus detection using an imaging surface phase difference detection method even when a lens unit having multiple optical axes is attached. [Means for solving the problem]

[0007] The above-mentioned object can be achieved by an imaging device comprising: an image sensor capable of generating a signal pair used for focus detection using a phase difference detection method; a calculation means for calculating the focal length of the attached lens unit based on the defocus amount obtained using the signal pair; and an adjustment means for adjusting the focal length of the lens unit based on the focal length, wherein when the lens unit is a multi-lens unit having multiple imaging optical systems with different optical axes, the calculation means calculates the focal length using an adjustment value obtained based on the optical axis position, which is the position on the image sensor through which the optical axis of the multi-lens unit passes. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an imaging device and method that are capable of performing focus detection using an imaging surface phase difference detection method with high accuracy even when a lens unit having multiple optical axes is attached. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a perspective view showing an example of the appearance of a camera 100, which is an example of an imaging device according to an embodiment. [Figure 2] Block diagram showing an example of the functional configuration of a camera system [Figure 3] Block diagram showing another example of the functional configuration of a camera system [Figure 4] FIG. 1 is a diagram showing an example of a pixel array of an image sensor of a camera 100. [Figure 5] Schematic diagram of the relationship between the defocus amount and the image shift amount based on a pair of focus detection signals [Figure 6] Flowchart for focus detection processing in the first embodiment [Figure 7] Schematic cross-sectional view of an optical system having an image sensor and multiple optical axes [Figure 8] FIG. 10 is a diagram showing an example of a change in the light amount of a focus detection signal when an optical system having multiple optical axes is attached. [Figure 9] FIG. 10 is a diagram showing a display example of a live view image in the second embodiment. [Figure 10]FIG. 10 is a diagram showing an example of a display form of a focus guide; [Figure 11] Flowchart for focus guide display processing [Figure 12] Flowchart for live view display operation in the second embodiment [Figure 13] FIG. 11 is a diagram showing an example of a display form of an index that indicates the difference in focus degree between left and right images in the third embodiment; [Figure 14] Flowchart for indicator display control processing in the third embodiment [Figure 15] Flowchart for live view display operation in the third embodiment [Figure 16] FIG. 10 is a diagram showing a display example of a live view image in the third embodiment. [Figure 17] FIG. 13 is a diagram showing an example of a display form of an index that indicates the difference in focus degree between left and right images in the fourth embodiment; [Figure 18] FIG. 13 is a diagram showing a display example of a live view image in the fourth embodiment. [Figure 19] Flowchart for misalignment adjustment processing in the fifth embodiment [Figure 20] FIG. 13 is a diagram showing an example of a display form of an indicator of the fourth embodiment corresponding to the sixth embodiment. [Figure 21] FIG. 23 is a diagram showing an example of a display form of a calibration guide in the sixth embodiment. [Figure 22] FIG. 20 is a diagram for explaining a calibration method in the sixth embodiment. [Figure 23] FIG. 10 is a diagram showing XR goggles used in the seventh embodiment. [Figure 24] FIG. 20 is a diagram showing an example of a recorded image in the seventh embodiment. [Figure 25] FIG. 20 is a diagram showing an example of an operation for changing a focused subject of a recorded image in the seventh embodiment. [Figure 26] FIG. 13 is a block diagram showing an example of the configuration of a computer capable of implementing the seventh embodiment. [Figure 27] FIG. 13 is a diagram illustrating a calibration method according to the seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following embodiments, the present invention will be described with reference to an interchangeable lens digital camera. However, the present invention can also be implemented in any electronic device that has a camera equipped with a focus detection function using an image plane phase difference detection method. Such electronic devices include the following: general imaging devices (video cameras, surveillance cameras, etc.), computer devices (personal computers, tablets, media players, PDAs, etc.), communication devices (mobile phones, smartphones, IoT devices, etc.), game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] ●(First embodiment) [Overall configuration] 1 is a perspective view showing an example of the appearance of a body 100 of a lens-interchangeable mirrorless digital camera (hereinafter referred to as camera 100) as an example of an imaging device according to a first embodiment of the present invention. Fig. 1(a) is a perspective view of the camera 100 as seen obliquely from above the front, and Fig. 1(b) is a perspective view of the camera 100 as seen obliquely from above the back.

[0013] The camera 100 has, on its top surface, a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an extra-viewfinder display 107. The shutter button 101 is an operation unit used to prepare for shooting or to give instructions to shoot. The power switch 102 is an operation unit used to switch the power of the camera 100 on and off. The mode switch 103 is an operation unit used to switch between various modes. The main electronic dial 104 is a rotary operation unit used to change settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation unit used to move the selection frame (cursor), advance images, etc. The video button 106 is an operation unit used to give instructions to start and stop video shooting (recording). The extra-viewfinder display 107 displays various settings such as shutter speed and aperture.

[0014] Camera 100 has, on its rear surface, a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119. Display unit 108 displays images and various information. Touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of display unit 108.

[0015] The directional keys 110 are an operation unit consisting of keys that can be pressed up, down, left, and right (four-way keys). Operations can be performed according to the position of the directional keys 110 that is pressed. The SET button 111 is an operation unit that is mainly pressed to confirm a selection item. The AE lock button 112 is an operation unit that is pressed to fix the exposure state in a shooting standby state. The enlargement button 113 is an operation unit for switching the enlargement mode on and off in the live view display (LV display) in shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is also used to enlarge a playback image or increase the magnification ratio in playback mode.

[0016] The playback button 114 is an operation unit for switching between the shooting mode and the playback mode. Pressing the playback button 114 in the shooting mode switches to the playback mode, and the most recent image recorded on the recording medium 228 (described later) can be displayed on the display unit 108. The menu button 115 is an operation unit that is pressed to display a menu screen on the display unit 108 that allows various settings to be made. The user can make various settings for the camera 100 by operating the menu screen displayed on the display unit 108 using the direction keys 110 and the SET button 111. Note that the menu screen may be operated using the touch panel 109 instead of or in addition to the buttons.

[0017] Eyepiece 116 is a window for looking into eyepiece finder (peek-in type finder) 117. A user can view an image displayed on an internal EVF (Electronic View Finder) 217 (described later) through eyepiece 116. Eyepiece detection unit 118 is a sensor that detects whether an object is approaching eyepiece 116.

[0018] The touch bar 119 is a line-shaped touch operation unit (line touch sensor) capable of receiving touch operations. The touch bar 119 is positioned so that it can be touched by the thumb of the right hand when the grip unit 120 is held in the right hand (holding the grip unit 120 with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. In other words, the touch bar 119 can be operated while looking into the eyepiece finder 117 through the eyepiece unit 116 and in a state (shooting posture) in which the shutter button 101 can be pressed at any time. The touch bar 119 can receive tap operations (operations in which the user touches the touch bar 119 and then releases the touched position without moving it within a predetermined period of time), slide operations to the left or right (operations in which the user moves the touched position while still touching it), and the like. The touch bar 119 is an operation unit different from the touch panel 109 and does not have a display function. The touch bar 119 in this embodiment functions as a multi-function bar (M-Fn bar).

[0019] The camera 100 also has a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section formed in a shape that is easy to hold in the user's right hand when holding the camera 100. When the user holds the camera 100 by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and main electronic dial 104 are located in positions that can be operated with the index finger of the right hand. In a similar state, the sub electronic dial 105 and touch bar 119 are located in positions that can be operated with the thumb of the right hand.

[0020] The thumb rest 121 (thumb standby position) is a grip section provided on the rear side of the camera 100 in a position where it is easy to place the thumb of the right hand gripping the grip section 120 when none of the operation sections are being operated. The thumb rest 121 is made of a rubber member or the like to enhance holding power (grip feeling). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 228 and the slot by closing the slot for storing the recording medium 228, which will be described later. The communication terminal 124 is a terminal for communicating with the lens unit 200, which will be described later and is detachable from the camera 100.

[0021] <Internal configuration of camera 100> Fig. 2 is a block diagram showing an example of the internal configuration (functional configuration) of a camera system in which an interchangeable lens unit 200 is attached to a camera 100. Note that in Fig. 2, the components shown in Fig. 1 are assigned the same reference numerals as in Fig. 1. Explanation of the components already explained with reference to Fig. 1 will be omitted as appropriate.

[0022] First, the lens unit 200 will be described. The lens unit 200 is an example of an interchangeable lens that can be attached to and detached from the camera 100. The lens unit 200 is a typical single lens (a lens with one optical axis). The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0023] The aperture 201 is configured so that its aperture diameter is adjustable. The lens 202 is made up of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the focus lens included in the lens 202 to adjust the distance at which the lens unit 200 focuses.

[0024] The lens system control circuit 205 has, for example, a CPU, ROM, and RAM, and controls the operation of each part of the lens unit 200 by loading a program stored in the ROM into the RAM and executing it with the CPU. The lens unit 200 and camera 100 are electrically connected via communication terminals 206 and 124, and the lens system control circuit 205 and a system control unit 218 of the camera 100 can communicate with each other. The lens system control circuit 205 controls the aperture drive circuit 203, AF drive circuit 204, etc. based on instructions from the system control unit 218.

[0025] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 218.

[0026] The shutter 210 is a focal plane shutter that operates based on instructions from the system control unit 218 and controls the exposure time of the imaging unit 211. The imaging unit 211 is an imaging element (image sensor) configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal. In this embodiment, the imaging unit 211 is an imaging element that supports focus detection using an imaging surface phase difference detection method. Specifically, the imaging unit 211 can output a focus detection signal pair for realizing focus detection using the phase difference detection method.

[0027] The A / D converter 212 converts analog signals output from the imaging unit 211 into digital signals (image data). The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on data input via the A / D converter 212 or the memory control unit 213. The image processing unit 214 also performs predetermined arithmetic processing using the captured image data to calculate evaluation values used for AF and AE, etc. The system control unit 218 performs exposure control and focus detection control based on the obtained arithmetic results. The image processing unit 214 also calculates a defocus amount based on a focus detection signal pair obtained from the imaging unit 211 as one of the evaluation values. The image processing unit 214 also performs predetermined arithmetic processing using the captured image data and performs AWB (auto white balance) processing on the image data based on the obtained arithmetic results.

[0028] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data output by the A / D converter 212, image data generated by the image processing unit 214, and the like. The image data generated by the image processing unit 214 includes display image data to be displayed on the display unit 108 and the EVF 217, and recording image data to be recorded on the recording medium 228. The memory 215 has a storage capacity sufficient to store a predetermined number of still image data, and a predetermined amount of moving image data and audio data. In addition, a portion of the memory 215 is used as a video memory for the display unit 108.

[0029] The D / A converter 216 converts the image data stored in the memory 215 into an analog signal suitable for display on the display unit 108 or the EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display in accordance with the analog signal from the D / A converter 216. The display unit 108 or the EVF 217 is, for example, an LCD, an organic EL display, or the like.

[0030] While the imaging unit 211 is capturing moving images, image data stored in a memory 215 via an A / D converter 212 is converted into an analog signal by a D / A converter 216, and the signal is sequentially transferred to and displayed on the display unit 108 or the EVF 217. This allows a live view display on the display unit 108 or the EVF 217.

[0031] The system control unit 218 is a control unit including at least one processor (CPU) and / or at least one circuit. That is, the system control unit 218 may be a processor (CPU), a circuit, or a combination of a processor and a circuit. For example, if the system control unit 218 has a processor (CPU), the system control unit 218 controls the entire camera 100 by loading a program stored in the nonvolatile memory 220 into the system memory 219 and executing it on the processor. The system control unit 218 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, etc.

[0032] The camera 100 also includes a system memory 219 , a nonvolatile memory 220 , a system timer 221 , a communication unit 222 , an attitude detection unit 223 , and an eye proximity detection unit 118 . For example, a RAM is used as the system memory 219. Constants and variables for the operation of the system control unit 218, programs read from the nonvolatile memory 220, and the like are loaded into the system memory 219. The nonvolatile memory 220 may be, for example, an electrically erasable and programmable read-only memory (EEPROM), and stores constants, programs, and the like for the operation of the system control unit 218.

[0033] The system timer 221 is a timing unit that measures the time used for various controls and the time of a built-in clock. The communication unit 222 transmits and receives image signals and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 222 can communicate with external devices that comply with a wireless LAN (Local Area Network) and devices on the Internet. The communication unit 222 can also communicate with external devices that comply with Bluetooth (registered trademark). The communication unit 222 can transmit images (including live images) captured by the imaging unit 211 and images recorded on a recording medium 228, and can receive image data and various other information from external devices.

[0034] The orientation detection unit 223 outputs a signal indicating the orientation of the camera 100 with respect to the direction of gravity. Based on the signal output by the orientation detection unit 223, it is possible to determine whether an image captured by the imaging unit 211 was captured with the camera 100 held horizontally or vertically. The system control unit 218 can add orientation information corresponding to the signal output by the orientation detection unit 223 to the image file of the image captured by the imaging unit 211, or rotate and record the image. The orientation detection unit 223 can use, for example, an acceleration sensor or a gyro sensor. Based on the output signal from the orientation detection unit 223, the system control unit 218 can also detect the movement of the camera 100 (panning, tilting, lifting, whether or not the camera is stationary, etc.).

[0035] The eyepiece detection unit 118 can detect the approach of an object to the eyepiece unit 116 of the eyepiece finder 117 that incorporates the EVF 217. The eyepiece detection unit 118 can be, for example, an infrared proximity sensor. When an object approaches, infrared light is projected from a light-projecting unit of the eyepiece detection unit 118, reflected by the object, and received by a light-receiving unit of the infrared proximity sensor. The presence or absence of an object approaching the eyepiece unit 116 can be determined based on the amount of infrared light received.

[0036] The system control unit 218 switches the display unit 108 and the EVF 217 between display (display state) and non-display (non-display state) depending on the presence or absence of a nearby object detected by the eye proximity detection unit 118. Specifically, at least in a shooting standby state and when the display destination switching setting is automatic switching, if no approaching object is detected, the display unit 108 is turned on and the EVF 217 is turned off. Also, if an approaching object is detected, the EVF 217 is turned on and the display unit 108 is turned off. Note that the eye proximity detection unit 118 is not limited to an infrared proximity sensor, and any other sensor that can detect a state that can be considered as eye proximity may be used.

[0037] The camera 100 also has an outside-viewfinder display unit 107, an outside-viewfinder display drive circuit 224, a power supply control unit 225, a power supply unit 226, a recording medium I / F 227, an operation unit 229, and the like. The viewfinder display unit 107 displays various settings of the camera 100, such as shutter speed and aperture, via an viewfinder display drive circuit 224. The power supply control unit 225 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, and other components, and detects whether a battery is installed, the battery type, and the remaining battery charge. The power supply control unit 225 also controls the DC-DC converter based on the detection results and instructions from the system control unit 218 to supply the required voltage for the required period to various components, including the recording medium 228. The power supply unit 226 may be a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The recording medium I / F 227 is an interface with a recording medium 228, such as a memory card or a hard disk. The recording medium 228 is a memory card or the like for recording captured images, and may be composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 228 may be removable or built-in.

[0038] The operation unit 229 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 218. The operation unit 229 includes the shutter button 101, the power switch 102, the mode selector switch 103, the touch panel 109, and other operation members 230. The other operation members 230 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction keys 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, the touch bar 119, and the like.

[0039] The shutter button 101 has a first shutter switch 231 and a second shutter switch 232. The first shutter switch 231 is turned on when the shutter button 101 is pressed halfway, generating a first shutter switch signal SW1. The system control unit 218 interprets the first shutter switch signal SW1 as a shooting preparation instruction and starts shooting preparation processing. The shooting preparation processing includes AF processing, AE processing, AWB processing, and flash pre-flash processing.

[0040] The second shutter switch 232 is turned on when the shutter button 101 is fully pressed, and generates a second shutter switch signal SW2. The system control unit 218 interprets the second shutter switch signal SW2 as an instruction to capture a still image, and starts a still image capture operation based on the exposure conditions determined by the AE processing. Then, it controls each unit to execute a series of capture processes, from reading a signal from the imaging unit 211 to generating an image file containing the still image data obtained by the capture and writing it to the recording medium 228.

[0041] The mode selector switch 103 switches the operation mode of the system control unit 218 to one of still image capture mode, video capture mode, playback mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide capture settings for different capture scenes. The user can directly switch to one of the above-mentioned capture modes using the mode selector switch 103. Alternatively, the user can first switch to a list screen of capture modes using the mode selector switch 103, and then selectively switch to one of the displayed modes using the operation unit 229. Similarly, the video capture mode may also include multiple modes.

[0042] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108. Then, by associating input coordinates on the touch panel 109 with display coordinates on the display surface of the display unit 108, a GUI can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. GUI is an abbreviation for Graphical User Interface. The touch panel 109 can use any of various types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, or an optical sensor type. Depending on the type, there are types that detect a touch by contact with the touch panel 109, and types that detect a touch by the approach of a finger or a pen to the touch panel 109, but either type is acceptable.

[0043] The system control unit 218 can detect the following operations or states on the touch panel 109. A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down). A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On). The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the end of the touch (hereinafter referred to as "touch-up"). A state in which nothing is touching the touch panel 109 (hereinafter referred to as Touch-Off).

[0044] When a touch down is detected, a touch on is also detected at the same time. After a touch down, a touch on is usually continued to be detected unless a touch up is detected. If a touch move is detected, a touch on is also detected at the same time. Even if a touch on is detected, a touch move is not detected unless the touch position moves. Once it is detected that all fingers or pens that were touching have touched up, a touch off occurs.

[0045] These operation states and the position coordinates of the finger or pen touching the touch panel 109 are notified to the system control unit 218. The system control unit 218 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding touch-move, the movement direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced across the touch panel 109 as if flicking. If a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is then detected, it is determined that a flick has been performed (it can be determined that a flick occurred following a slide operation). Furthermore, when multiple points (for example, two points) are touched together (multi-touch), the touch operation of bringing the touched positions closer together is called pinch in, and when the touch operation of moving the touched positions farther apart is called pinch out. Pinch out and pinch in are collectively called pinch operations (or simply pinch).

[0046] <Configuration of multi-lens unit> FIG. 3 is a schematic diagram showing an example of the configuration of a twin lens unit 300 as an example of a multi-lens unit. In this specification, a "multi-lens" refers to a lens unit configured with multiple imaging optical systems provided within a single lens mount (or lens barrel), and has multiple optical axes. FIG. 3 shows the twin lens unit 300 attached to the camera 100. Note that FIG. 3 shows only a portion of the configuration of the camera 100 shown in FIG.

[0047] Twin lens unit 300 is a type of interchangeable lens that can be attached to and detached from camera 100. Twin lens unit 300 has two imaging optical systems 301L and 301R within a single lens barrel, and therefore has two optical axes.

[0048] Here, when the twin lens unit 300 is attached to the camera 100, the two imaging optical systems 301L and 301R are arranged so that the two optical axes are aligned on a horizontal line. The two imaging optical systems 301L and 301R have a viewing angle of approximately 180 degrees, and can capture the range of the forward hemisphere. Specifically, the two imaging optical systems 301L and 301R can each capture a field of view of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle). The two imaging optical systems 301L and 301R are A pair of parallax images having left and right parallax is formed on the imaging surface of the imaging unit 211. In the following description, the imaging optical system 301L will be referred to as the left eye optical system 301L, and the imaging optical system 301R will be referred to as the right eye optical system 301R.

[0049] The right-eye optical system 301R and the left-eye optical system 301L each have a plurality of lenses, a reflecting mirror, etc. The plurality of lenses includes at least a focus lens for adjusting the focal distance. The twin lens unit 300 also has a lens system control circuit 303. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the lenses 302R and 302L, which are located on the subject side, face in the same direction, and their optical axes are approximately parallel.

[0050] 3, the twin lens unit 300 has a configuration similar to that of the AF drive circuit 204. In this case, the twin lens unit 300 may have one or more AF drive circuits: an AF drive circuit that drives the focus lenses of the right eye optical system 301R and the left eye optical system 301L in conjunction with each other; and an AF drive circuit that independently drives the focus lens of at least one of the right eye optical system 301R and the left eye optical system 301L. The focus lens is driven by the lens system control circuit 303 under the control of the system control unit 218 (adjustment means).

[0051] The twin lens unit 300 also includes an encoder that detects the amount and direction of rotation of a focus ring mounted on the lens barrel. The lens system control circuit 303 controls the AF drive circuit in response to the focus lens operation detected by the encoder, thereby providing a so-called by-wire manual focus function. In this case, the twin lens unit 300 may also include a switch that allows the user to select the focus lens driven by the focus ring operation.

[0052] The twin lens unit 300 is a VR180 lens for using the camera 100 to capture images in VR180 format, a VR image format that enables binocular stereoscopic viewing. The VR180 lens has fisheye lenses, each with a field of view of approximately 180 degrees, for the right-eye optical system 301R and the left-eye optical system 301L. Note that the right-eye optical system 301R and the left-eye optical system 301L only need to be able to acquire images that enable binocular VR display as VR180, and the field of view may be approximately 160 degrees. The VR180 lens can form a right image (first image) using the right-eye optical system 301R and a left image (second image) using the left-eye optical system 301L on the same imaging surface. Here, it is assumed that the imaging unit 211 of the camera 100 has one imaging element, and the twin lens unit 300 forms the right image and the left image on the imaging surface of the single imaging element. However, camera 100 may have two image sensors arranged side by side, and twin lens unit 300 may form a right image on the imaging surface of one image sensor and a left image on the imaging surface of the other image sensor.

[0053] The twin lens unit 300 includes a focus ring for adjusting the focus of the right-eye optical system 301R and a focus ring for adjusting the focus of the left-eye optical system 301L. Alternatively, the twin lens unit 300 includes a focus ring for simultaneously adjusting the focus of the right-eye optical system 301R and the left-eye optical system 301L, and a focus ring for adjusting the focus of either the right-eye optical system 301R or the left-eye optical system 301L. By operating these focus rings, the user can manually adjust the focal distance of the right-eye optical system 301R and the left-eye optical system 301L. These focus rings may be provided separately, or in the case of a by-wire system, they may be realized by switching the function of a single focus ring.

[0054] Like the (single) lens unit 200, the twin lens unit 300 is attached to the camera 100 via a mount. The mount is made up of a lens mount 304 and a camera mount 305. When the twin lens unit 300 is attached to the camera 100, the communication terminal 124 of the camera 100 and the communication terminal 306 of the twin lens unit 300 are electrically connected. This enables communication between the system control unit 218 of the camera 100 and the lens system control circuit 303 of the twin lens unit 300.

[0055] In this embodiment, the right and left images are formed on the imaging surface of the imaging unit 211, spaced apart in the left-right direction. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single imaging element. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. In this way, by attaching the twin lens unit 300, a pair of parallax images (right and left images) formed by the right-eye optical system 301R and the left-eye optical system 301L can be acquired in a single capture. Furthermore, by VR displaying the acquired right and left images as images for the right eye and the left eye, the user can observe a stereoscopic VR image with a range of approximately 180 degrees, a so-called VR180 image.

[0056] Here, VR images are images that can be displayed in VR, as described below. VR images include omnidirectional images (spherical images) taken with an omnidirectional camera (spherical camera), panoramic images with an image range (effective image range) wider than the display range that can be displayed at one time on a display unit, and the like. VR images may be either still images or videos. Videos may be recorded videos or live images (images acquired from a camera in almost real time).

[0057] A VR image has an image range (effective image range) of up to 360 degrees horizontally and vertically. VR images also include images with a wider field of view than that which can be captured by a normal camera, or an image range wider than the display range that can be displayed at one time on a display unit, even if the field of view is less than 360 degrees horizontally or vertically. Images captured by camera 100 using the above-described twin lens unit 300 are a type of VR image. VR images can be displayed in VR, for example, by setting the display mode of a display device (a display device capable of displaying VR images) to "VR view." By displaying a VR image with a 360-degree field of view and changing the orientation of the display device left and right (horizontal rotation direction), a user can view a seamless, omnidirectional image in the horizontal direction.

[0058] Here, VR display (VR view) is a display mode that displays an image of a predetermined range of the field of view captured in the VR image according to the attitude of the display device. VR display includes "single-eye VR display (single-eye VR view)" that displays a single image by mapping the VR image onto a virtual sphere (deformation that performs distortion correction). VR display also includes "two-eye VR display (two-eye VR view)" that displays a VR image for the left eye and a VR image for the right eye side by side in left and right regions by mapping them onto a virtual sphere.

[0059] Stereoscopic viewing is possible through "two-eye VR display" using a VR image for the left eye and a VR image for the right eye, which have a parallax between them. Regardless of the VR display, when a user wears a display device such as an HMD (head-mounted display), an image with a field of view corresponding to the orientation of the user's face is displayed. For example, suppose a VR image is displayed with a field of view centered at 0 degrees left and right (a specific direction, e.g., north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is flipped from this state (e.g., the display surface is changed from facing south to facing north), the display range changes to the same VR image with a field of view centered at 180 degrees left and right (the opposite direction, e.g., south) and 90 degrees up and down. In other words, when a user wearing an HMD turns their face from north to south (i.e., turns around), the image displayed on the HMD also changes from a north image to a south image.

[0060] The VR image captured using the twin lens unit 300 of this embodiment is a VR180 format image capturing a range of approximately 180 degrees forward, and does not contain an image of a range of approximately 180 degrees backward. If such an image in VR180 format is displayed in VR and the orientation of the display device is changed to the side where no image exists, for example, a blank area will be displayed.

[0061] By displaying VR images in this way, the user visually feels as if they are inside the VR image (in the VR space). Note that the method of displaying VR images is not limited to changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operation via a touch panel or directional buttons. Furthermore, during VR display (in the "VR view" display mode), in addition to changing the display range due to posture changes, the display range may also be changed in response to touch-move on the touch panel, dragging with a mouse, pressing directional buttons, etc. Note that a configuration in which a display device such as a smartphone is attached to VR goggles (head-mounted adapter) is a type of HMD.

[0062] <Configuration of the imaging unit 211 (imaging element)> FIG. 4 is a diagram schematically illustrating an example of a pixel array of the imaging unit 211 (image sensor) in this embodiment. The image sensor constituting the imaging unit 211 in this embodiment is capable of generating signal pairs used for focus detection using a phase difference detection method. FIG. 4 shows the pixel array of the image sensor (two-dimensional CMOS sensor) in an area of 4 columns x 4 rows of imaging pixels (an area of 8 columns x 4 rows for the focus detection pixel array). Hereinafter, when simply referred to as "pixel," it means an imaging pixel.

[0063] The imaging unit 211 is provided with color filters in a primary color Bayer array. The pixel group 400 indicates 2 columns x 2 rows of pixels, which are the repeating unit of the color filters. The pixel group 400 includes a pixel 400R having a spectral sensitivity of R (red), pixels 400Gr and 400Gb having a spectral sensitivity of G (green), and a pixel 400B having a spectral sensitivity of B (blue). Each pixel is provided with a microlens 401.

[0064] To enable focus detection using an imaging surface phase difference method, the imaging unit 211 has two photodiodes (photoelectric conversion units) 402 and 403, each of which has a plurality of pixels arranged two-dimensionally in the imaging unit 211 and which share a microlens 401. The first photodiode 402 and the second photodiode 403 each function as a subpixel or a focus detection pixel. That is, one pixel functions as two focus detection pixels. Furthermore, by treating the first photodiode 402 and the second photodiode 403 collectively as a single photodiode, one pixel functions as a single imaging pixel. Hereinafter, the signal obtained by the first photodiode 402 will be referred to as signal A, the signal obtained by the second photodiode 403 as signal B, and the signal obtained by adding the A and B signals obtained by the same pixel as signal A+B. Furthermore, the A and B signals will be referred to as focus detection signals, and the A+B signal will be referred to as an imaging signal. Note that signal A (signal B) may be obtained by subtracting signal B (signal A) from signal A+B.

[0065] In this embodiment, each pixel is configured to have two photodiodes that share the microlens 401, but the number of photodiodes provided in each pixel may be three or more. Also, a pixel dedicated to focus detection that can essentially output only the A signal or the B signal may be provided. There are no limitations on the configuration of pixels provided in the image sensor as long as they can output a signal that can realize focus detection using the phase difference detection method. In this embodiment, all pixels are configured to have multiple photodiodes, but only some pixels may have multiple photodiodes.

[0066] <Relationship between defocus amount and image shift amount> Using Figure 5, we will explain the relationship between the defocus amount and the image shift amount obtained from the A signal and B signal obtainable by the image sensor shown in Figure 4. The defocus amount is calculated using a pair of image signals consisting of an A image signal obtained by concatenating multiple A signals and a B image signal obtained by concatenating multiple B signals. The A image signal and the B image signal are also called focus detection signals. Here, we assume that the imaging center (the center of the pixel area used for imaging in the image sensor) and the optical axis center coincide with each other.

[0067] 5 schematically shows the relationship between the defocus amount d and the amount of image shift between a pair of focus detection signals (image A signal and image B signal). Reference numeral 1300 denotes the imaging surface of the image sensor. The first photodiode 402 and second photodiode 403, which share a single microlens, divide the exit pupil of the imaging optical system into a first pupil partial region 1303 and a second pupil partial region 1304.

[0068] The magnitude |d| of the defocus amount d represents the distance from the imaging position of the subject image to the imaging plane 1300. In a front-focus state, where the imaging position of the subject image is closer to the subject than the imaging plane 1300, the direction is defined as a negative sign (d<0), and in the opposite back-focus state, the direction is defined as a positive sign (d>0). In a focused state, where the imaging position of the subject image is on the imaging plane 1300 (i.e., the in-focus position), d=0. For example, subject 1301 is in-focus because it is imaged in a focused state (d=0). Similarly, subject 1302 is in front-focus because its imaging position is closer to the subject than the imaging plane 1300 (d<0). Hereinafter, the front-focus state (d<0) and the back-focus state (d>0) will be collectively referred to as the defocus state (|d|>0).

[0069] In a front-focus state (d<0), the light beam received from the subject 1302 that passes through the first pupil partial region 1303 (or the second pupil partial region 1304) is collected and then spreads to a width Γ1 (or Γ2) centered on the center of gravity position G1 (or G2) of the light beam. In this case, the image of the subject 1302 is blurred on the imaging surface 1300. The blurred image is received by the first photodiode 402 (or the second photodiode 403) provided in each pixel arrayed on the imaging element, and an A signal (or a B signal) is generated.

[0070] Therefore, the pair of focus detection signals (image signal A and image signal B) are stored in memory as image data of a (blurred) subject image having width Γ1 (or Γ2) at center of gravity position G1 (or G2) on the imaging surface 1300. The width Γ1 (or Γ2) of the subject image increases roughly in proportion to the magnitude |d| of the defocus amount d. Similarly, if the image shift amount between the first focus detection signal and the second focus detection signal is "p," the magnitude of the image shift amount |p| increases as the magnitude |d| of the defocus amount d increases.

[0071] 5, the image shift amount p can be defined as the difference between the center of gravity positions of the light beams, "G1 - G2," and its magnitude |p| increases roughly in proportion to the increase in the defocus amount |d|. Note that in a back-focus state (d>0), the direction of image shift between a pair of focus detection signals (image A signal and image B signal) is opposite to that in a front-focus state, but the magnitude of the image shift amount |p| is proportional to the defocus amount |d|.

[0072] Therefore, phase-difference AF can be achieved by detecting the amount of image shift p between a pair of focus detection signals (image A signal and image B signal) and converting the amount of image shift p into a defocus amount using a conversion coefficient K. The amount of image shift p between a pair of focus detection signals (image A signal and image B signal) can be determined by shifting the image A signal and the image B signal relatively to calculate the amount of correlation and then finding the shift amount that results in good correlation (degree of signal agreement). Note that the conversion coefficient K has a value that depends on the angle of incidence, F-number, and optical axis position of the imaging optical system. Therefore, a conversion coefficient K appropriate for the lens unit is used.

[0073] The conversion coefficient K is stored in, for example, a nonvolatile memory included in the lens system control circuit 205 of the lens unit, and can be acquired by the system control unit 218 from the attached lens unit. Of course, the conversion coefficient K may also be acquired in other ways, such as by storing the conversion coefficient K in association with the identification information of the lens unit in the nonvolatile memory 220 and acquiring the conversion coefficient K from the nonvolatile memory 220 based on the identification information of the attached lens unit.

[0074] A pair of focus detection signals is usually generated based on signals from pixels within a focus detection area. Therefore, if a focus detection area is set for each imaging optical system, a defocus amount is calculated for each focus detection area. If the focus lenses of the imaging optical systems can be driven individually, the focus distance can be adjusted for each imaging optical system. Note that a single defocus amount based on the defocus amount calculated for each focus detection area may be used to adjust the focus distances of multiple imaging optical systems. The single defocus amount may be, for example, an average value or a representative value.

[0075] <Defocus amount calculation process> The defocus amount calculation process will be described with reference to the flowchart shown in FIG. In this embodiment, the image processing unit 214 generates a first focus detection signal (A image signal) by combining A signals obtained by the first photodiodes 402 of multiple pixels in the image sensor, and generates a second focus detection signal (B image signal) by combining B signals obtained by the second photodiodes 403 of each pixel used to generate the A image signal.

[0076] In S1401, the image processing unit 214 (calculation means) acquires an A signal and a B signal from each of a plurality of pixels included in an area of the image sensor corresponding to the focus detection area. As described above, the A signal (or the B signal) may be acquired by subtracting the B signal (or the A signal) from the A+B signal.

[0077] In S1402, the image processing unit 214 adds the A signals of pixels of the same color that are located at the same horizontal position (row direction) in the vertical direction (column direction) to reduce the data volume of the A and B image signals. This compresses the signals to two rows. The image processing unit 214 then adds green (Gr), red (R), blue (B), and green (Gb) signals to the A signals added in the column direction to generate a luminance signal Y. These multiple luminance signals Y arranged in the row direction constitute the A image signal. The image processing unit 214 also applies a similar addition process to the B signals to generate the B image signal. By adding the signals, the Nyquist frequency in the addition direction becomes 1 / n of that when no addition is performed, where n is the number of pixels added.

[0078] In S1403, the image processing unit 214 applies shading correction processing (optical correction processing) to the image A signal and the image B signal to make the signal intensities uniform (to suppress the difference in signal intensity). The shading correction value has a value that depends on the incident angle, F-number, and optical axis position of the imaging optical system. Like the conversion coefficient K, the shading correction value can also be acquired from the lens unit or the nonvolatile memory 220.

[0079] In S1404, the image processing unit 214 applies spatial band-pass filtering having a specific pass frequency band to the image signals A and B in order to improve the correlation (degree of signal agreement) between the image signals A and B and thereby enhance focus detection accuracy. Examples of band-pass filters include differential filters such as {1, 4, 4, 4, 0, -4, -4, -4, -1} that cut off DC components and extract edges, and additive filters such as {1, 2, 1} that suppress high-frequency noise components.

[0080] In S1405, the image processing unit 214 calculates the amount of correlation between the image signal A and the image signal B after the filter processing has been applied. The amount of correlation is calculated for each shift amount while changing the relative shift amount between the image signal A and the image signal B in the pupil division direction.

[0081] The number of signals constituting the A and B image signals after band-pass filtering is W (>2), and the kth (1≦k≦W) signals are A(k) and B(k). If the shift amount is s and the range of the shift amount s is Γ, the correlation amount COR is calculated by equation (1). COR(s)=Σ(k∈W)|A(k)-B(ks)|, s∈Γ (1)

[0082] The absolute difference between the kth image A signal A(k) and the ksth image B signal B(ks) is accumulated over k within the range of the number of signals W, and the correlation amount COR(s) for the shift amount s is calculated. The shift amount is, for example, in units of one pixel. If there are multiple image A signals and multiple B image signals in the vertical direction, the correlation amounts calculated for each pair of image A and B signals for the same shift amount may be added together.

[0083] In S1406, the image processing unit 214 calculates the shift amount in units of less than one pixel, at which the correlation amount becomes the minimum, based on the correlation amount COR(s) calculated for the shift amount in units of one pixel. The image processing unit 214 then sets the calculated shift amount as the image shift amount p between the A and B image signals. Furthermore, the image processing unit 214 applies (for example, multiplies) the image shift amount p by the above-mentioned conversion coefficient K to convert it into a defocus amount d. The defocus amount d is calculated through the above processing.

[0084] As such, the correction value used in the signal correction performed in the process of calculating the defocus amount d and the conversion coefficient K used to convert the image shift amount p to the defocus amount d have values that depend on the characteristics of the lens unit. Furthermore, focus detection adjustment values are typically calculated assuming a lens unit with a single optical axis passing through the center of the image sensor. Note that the optical axis passing through the center of the image sensor here represents a design or ideal state, and deviations due to manufacturing errors and the like are acceptable. For example, vignetting of incident light by the lens frame differs between a single-optical axis lens and a multiple-optical axis lens, even if the lens frame diameter is the same. Therefore, even if the focus detection adjustment value is calculated from lens information for a lens unit with two optical axes in the same way as for a lens unit with one optical axis, the focus detection adjustment value will not be appropriate, and as a result, the accuracy of the defocus amount may be reduced.

[0085] In this embodiment, even when an imaging optical system having multiple optical axes is attached to a main body having one image sensor, the use of an appropriate focus detection adjustment value suppresses a decrease in the accuracy of the defocus amount. In this embodiment, examples of focus detection adjustment values for a lens unit having multiple optical axes are as follows: Shading correction coefficient for matching the intensity of a pair of focus detection signals Conversion coefficient K for converting the image shift amount p to the defocus amount d Best focus correction value that corrects the focus distance based on the amount of defocus However, the concept of this embodiment can be similarly applied to other focus detection adjustment values. The focus detection adjustment values are basically used for each focus detection area, that is, for each imaging optical system.

[0086] <Shading correction coefficient> 7 shows a schematic cross-sectional view of an optical system when a twin lens unit 300 having two optical axes is attached to a camera 100 having one image sensor. In this embodiment, twin lens unit 300 has two imaging optical systems, and the optical axes of each imaging optical system are on a straight line in the horizontal direction (parallel to the long side of the image sensor) that passes through the center of the image sensor (image capture center) and at positions equidistant from the image capture center. Therefore, FIG. 7 shows a horizontal cross section that includes the center of the image sensor and the two optical axes.

[0087] The diameter of the image circle of each imaging optical system is approximately half the length of the long side of the effective pixel area of the image sensor. Figure 8(a) shows an example of the signal intensities of the image A and B signals obtained when the twin lens unit 300 is attached. As shown in Figure 8, there are optical axes on both sides of the center of the image sensor, so the signal intensities show discontinuous changes at the center of the image sensor. When an imaging optical system with a single optical axis passing through the center of the image sensor is attached, the intensity changes of the image A and B signals are continuous. Therefore, discontinuous changes in the intensity of the image A and B signals are a characteristic phenomenon when an imaging optical system with multiple optical axes is attached to a single image sensor.

[0088] Fig. 8(b) shows an example of shading correction values for aligning the intensities of the image signals A and B, which have the intensity variations shown in Fig. 8(a). To correct the image signals A and B, whose signal intensities vary discontinuously, the shading correction values also vary discontinuously at the boundary between the two image circles.

[0089] In this embodiment, shading correction values for the twin lens unit 300 having two optical axes are stored in advance in the camera 100, for example, in the nonvolatile memory 220, similar to the shading correction values for the lens unit 200 having one optical axis. The system control unit 218 selects a correction value to be used for shading correction depending on the number of optical axes of the attached lens unit. Note that the shading correction values may be calculated in advance for each lens unit model and stored in the nonvolatile memory 220 in association with the lens unit identification information. Alternatively, a calculation formula for the correction value corresponding to the number of optical axes may be stored in the nonvolatile memory 220, and information required to calculate the shading correction value may be obtained (for example, from the attached lens unit) and applied to the calculation formula to calculate the shading correction value.

[0090] When calculating the shading correction value, the number of optical axes and the positions on the image sensor through which the optical axes pass (hereinafter referred to as optical axis positions) are required. Optical information with the center of the optical axis as the origin must be treated as the optical axis position. Therefore, for lens units whose optical axes do not pass through the center of the image sensor, such as twin lens unit 300 with multiple optical axes, the optical axis positions are important in calculating the shading correction value.

[0091] In this embodiment, the optical axis position (Lx, Ly) is acquired from, for example, the twin lens unit 300, and the optical information of the imaging optical system expressed with the optical axis as the origin is converted into information in the coordinate system of the image sensor. For example, as shown in Table 1, assume that the optical information of one imaging optical system is that the angle of incidence of light at a position 10 mm away from the optical axis is 5 degrees (with respect to the optical axis), and the optical axis position (Lx, Ly) = (5 mm, 0 mm).

[0092] The optical axis position (5mm, 0mm) indicates that the optical axis passes through the position x = 5mm, y = 0mm in a Cartesian coordinate system with the center of the image sensor as the origin. Therefore, from the optical information, we can see that the angle of incidence of light at a point on the image sensor 10mm away from the optical axis (for example, (15mm, 0mm) or (-5mm, 0mm)) is 5 degrees.

[0093] In this way, by acquiring the optical axis position (Lx, Ly) expressed in the coordinate system of the image sensor, it is possible to convert the optical information expressed with the optical axis as the origin into optical information in the coordinate system of the image sensor. Then, by using the optical information in the coordinate system of the image sensor, it is possible to calculate an appropriate shading correction value. [Table 1]

[0094] As with the shading correction values, the conversion coefficients for converting the image shift amount into the defocus amount are stored in advance in, for example, nonvolatile memory 220 of camera 100 for twin lens unit 300 having two optical axes and lens unit 200 having one optical axis. Note that for lens units with multiple optical axes, it is advantageous in terms of conversion accuracy to use the conversion coefficient related to the optical axis position closest to the coordinates (or focus detection area) to which correction is applied.

[0095] <Best focus correction value> An image captured by adjusting the focus of the imaging optical system to a focal distance based on the defocus amount obtained by image-plane phase-difference focus detection may not match the image perceived as most in-focus when viewed by humans. This is thought to be due in part to the fact that the spatial frequency band used to calculate the defocus amount does not match the spatial frequency band observed by the human eye. For this reason, it is known to correct the focal distance detected by the imaging device to a focal distance that produces an image perceived as best in focus by humans. The correction value used to correct this focal distance is called the best focus correction value.

[0096] In some cases, the best focus correction value has a value that corresponds to the distance from the optical axis position. For example, the best focus correction value may be expressed by the following formula according to relative coordinates (x, y) with the optical axis position as the origin. Best focus correction value = a00 + a10 * x + a01 * y + a11 * xy Here, a00, a10, a01, and a11 are coefficients. When the optical axis position coincides with the center of the image sensor, the best focus correction value at any relative coordinate (x, y) on the image sensor can be calculated using the above formula. The best focus correction value is determined for the focus detection area, and the focus distance detected for that focus detection area is corrected using the best focus correction value. By adjusting the focus distance of the imaging optical system based on the corrected focus distance, it becomes possible to obtain an image that humans perceive as being in the best focus state.

[0097] However, if the optical axis position does not coincide with the center of the image sensor, the above formula cannot express the best focus correction value. In this embodiment, by modifying the above formula based on the above-mentioned optical axis position (Lx, Ly), it becomes possible to calculate the best focus correction value at any relative coordinate (x, y) with the center of the image sensor as the origin, even for an imaging optical system whose optical axis position does not coincide with the center of the image sensor.

[0098] Specifically, by modifying the above equation using the optical axis position (Lx, Ly) as follows, it becomes possible to calculate the best focus correction value at any relative coordinate (x, y) with the center of the image sensor as the origin for an imaging optical system with the optical axis position (Lx, Ly). Best focus correction value = a00 + a10(x-Lx) + a01(y-Ly) + a11(x-Lx)(y-Ly)

[0099] By correcting the best focus correction value, which is expressed as a function using coordinates with the optical axis as the origin, based on the optical axis position, an appropriate correction value can be calculated for a lens unit having an imaging optical system whose optical axis position is different from the center of the image sensor, such as a lens unit with multiple optical axes. Note that the optical axis position (Lx, Ly) expressed in the coordinate system of the image sensor can be obtained from the lens unit via communication. Alternatively, the optical axis position (Lx, Ly) pre-stored in the image capture device in association with the lens unit's identification information can be obtained by referencing the attached lens unit's identification information. For a lens unit with multiple optical axes, accurate correction is possible by using the best focus correction value corrected using the optical axis position closest to the coordinate (x, y) to be corrected.

[0100] In this embodiment, the shading correction value, conversion coefficient, and best focus correction value have been described as focus detection adjustment values that are based on the assumption that the optical axis position is the center of the image sensor. However, the technical idea of this embodiment is to correct the focus detection adjustment values that are based on the assumption that the optical axis position is the center of the image sensor based on information about the optical axis position, and this can be applied to any focus detection adjustment value that is based on the assumption that the optical axis position is the center of the image sensor.

[0101] In addition, for a lens unit having multiple imaging optical systems with different optical axes, it is necessary in principle to acquire or store optical information (optical axis position, angle of incidence, etc.) for each imaging optical system. However, for imaging optical systems that share optical information other than the optical axis position, the optical information other than the optical axis position can be acquired or stored for only one imaging optical system, thereby reducing the storage capacity of the lens unit or the imaging device body.

[0102] As described above, according to this embodiment, when a lens unit having multiple optical axes is attached to an imaging device having a single image sensor, the correction values used in the calculation process for the defocus amount are calculated using position information on the image sensor through which each optical axis passes. Therefore, appropriate correction values can be obtained even for correction values that would not be appropriate using a calculation method assuming a general lens unit having a single optical axis passing through the center of the image sensor. As a result, accurate focus detection using the image sensor plane phase difference detection method can be achieved even when a lens unit having multiple optical axes is attached to an imaging device having a single image sensor.

[0103] ●(Second embodiment) Next, a second embodiment of the present invention will be described. The first embodiment has explained that accurate focus detection using the image plane phase difference detection method can be achieved even when a lens unit with multiple optical axes is attached to an imaging device with a single image sensor. This embodiment relates to a configuration (focus guide function) that supports manual focus operation when a lens unit with multiple optical axes is attached to an imaging device with a single image sensor.

[0104] The focus guide function is a function that presents the position of the focus detection area and the degree of focus in the focus detection area to the user. For example, the position of the focus detection area and the degree of focus can be presented to the user by superimposing a GUI such as an index or mark indicating the position and degree of focus in the focus detection area on the live view display.

[0105] When a normal lens unit 200, which is assumed to have one optical axis passing through the center of the image sensor, is attached, only one image is formed on the image sensor. Therefore, the focus guide function only needs to be provided for one image. However, when a lens unit with multiple optical axes is attached, multiple images are formed on the image sensor. For example, when a twin lens unit 300 is attached to the camera 100, two images are formed on the image sensor (image capturing unit 211).

[0106] In this case, if a focus guide function is provided for only one image, it becomes difficult to perform accurate manual focusing on the image for which the focus guide function is not provided. Furthermore, it becomes impossible to grasp the focus level of each image at once. Therefore, it is necessary to provide a focus guide function that is suitable for cases where multiple images are formed on a single image sensor.

[0107] FIG. 9 shows an example of the focus guide function provided in this embodiment. In FIG. 9, the focus guide function is provided for each of a right image 800R and a left image 800L of a live view image captured using a right-eye optical system 301R and a left-eye optical system 301L. Specifically, indices 801R and 801L indicating information about the position of the focus detection area and the degree of focus are superimposed on the right image 800R and the left image 800L, respectively. The indices 801R and 801L are displayed at positions having the same image height. FIG. 9 shows an example in which the indices 801R and 801L are displayed at the same relative coordinates with the optical axis position of each imaging optical system as the origin.

[0108] In this embodiment, the focus guide function is provided by the image processing unit 214 under the control of the system control unit 218. Specifically, in parallel with the live view display processing, the image processing unit 214 calculates the defocus amount of the imaging optical system for the focus detection area notified by the system control unit 218 and outputs this amount together with the reliability to the system control unit 218. Then, the image processing unit 214 generates an index image based on the display format instructed by the system control unit 218 and writes the index image to an address area in the video memory area in the system memory 219 that corresponds to the focus detection area. This superimposes the index on the live view image, making it possible to provide the user with information regarding the position and focus degree of the focus detection area. Note that calculating the defocus amount for the focus detection area means calculating the defocus amount based on the image A signals and image B signals obtained from pixels in the focus detection area.

[0109] Next, specific examples of indicators provided by the focus guide function will be described with reference to FIGS. 10(A) to 10(D). The indices include a frame-shaped first index 500 displayed on the periphery of the focus detection area, and a third index 502 displayed at a position tangent to an imaginary circle 510 that encompasses the focus detection area and has a common center with the focus detection area. The indices also include a second index 501 displayed at a position tangent to an imaginary circle 511 whose radius is larger than that of the imaginary circle 510 by the length of the third index 502.

[0110] The first index 500 indicates the position and size of the focus detection area, and also indicates whether the focus detection area is in focus or out of focus depending on the display format. The second index 501 and the third index 502 indicate the degree of focus of the focus detection area depending on the display format and positional relationship between them. Specifically, in addition to the in-focus state and out-of-focus state, for the out-of-focus state, a distinction is made between a state in which the subject is in focus closer to the subject (front focus) and a state in which the subject is in focus at infinity closer to the subject (back focus), and the amount of deviation from the in-focus state is presented.

[0111] FIG. 10A shows an example of the display form of the indices when the focus detection area is in an in-focus state. In the in-focus state, the first index 500 is displayed as a continuous frame. The second index 501 is a wedge-shaped inclined downward, and the third index 502 is a wedge-shaped inclined upward. The second index 501 and the third index 502 are displayed on a vertical line passing through the center of the first index 500, facing each other with their tips touching. In the in-focus state, the second index 501 and the third index 502 are displayed with their interiors filled in. The display forms of the first to third indices 500 to 502 may differ in attributes other than shape and whether or not they are filled in, such as color, brightness, and whether or not they blink, as long as the in-focus state and the out-of-focus state can be visually distinguished. For example, the second index 501 and the third index 502 can be displayed in green when in-focus and in white when out-of-focus.

[0112] 10B and 10C show examples of the display form of the indicator when the reliability of the defocus amount is high in an out-of-focus state. FIG. 10B shows an example of the display format of the index in the front-focus state. In the out-of-focus state, the first index 500 is displayed as a frame with gaps. The display position of the second index 501 is the same as in the in-focus state, but the display format is different. Here, the second index 501 is displayed as a solid frame in the in-focus state, but is displayed as a hollow frame in the out-of-focus state. On the other hand, the third index 502 is displayed as two indexes 502A and 502B, each displayed at a position to the left (right) of the display position in the in-focus state by a distance corresponding to the amount of defocus. The second index 501 faces downward, indicating that it is in focus closer than the subject, and its display position is the same as in the in-focus state, making it easy to grasp the amount of defocus indicated by index 502A (or index 502B).

[0113] FIG. 10C shows an example of the display format of the index in the back-focus state. Because the image is out of focus, the first index 500 is displayed as a frame with gaps. The display position of the third index 502 is the same as in the in-focus state, but the display format is different. Here, the third index 502 is displayed as a solid frame in the in-focus state, but is hollow in the out-of-focus state. On the other hand, the second index 501 is displayed as two indexes 501A and 501B, each displayed at a distance to the left and right from the display position in the in-focus state corresponding to the amount of defocus. The third index 502 is facing upward, indicating that it is in focus farther than the subject, and its display position is the same as in the in-focus state, making it easier to grasp the amount of defocus indicated by index 501A (or index 501B).

[0114] Here, in the case of a front focus state, two third indexes 502 are used, but the number does not need to be increased. Simply, one of indexes 502A or 502B in Fig. 10(B) may be displayed (i.e., the display form and display position of the third index 502 may be changed). Similarly, in the case of a back focus state, one of indexes 501A or 501B in Fig. 10(C) may be displayed (i.e., the display form and display position of the second index 502 may be changed).

[0115] 10(D) shows an example of the indicator display format when the defocus amount is large and the reliability of the focus detection result is low (for example, when the image is significantly blurred). In this case, neither the front focus / back focus state (defocus direction) nor the magnitude of the defocus amount is presented. The indicator is displayed in a format that notifies the user that focus detection sufficient to provide the focus guide function is not possible.

[0116] Here, the first index 500 is displayed in a non-focus state, and the second index 501 and the third index 502 are displayed in a different form from both the focused state and the non-focus state in which the defocus direction and defocus amount can be displayed. Specifically, the shapes of the second index 501 and the third index 502 are changed from a wedge shape to a bar or line of a constant thickness, and are displayed in a color (e.g., gray) different from that of the other focused states. Furthermore, the display positions of the second index 501 and the third index 502 are fixed in advance. In the example shown in FIG. 10(D), the third index 502 is divided into two indexes 502A and 502B, as in the front focus state, but the number of indexes does not necessarily need to be increased. The image processing unit 214 can calculate the reliability of the defocus amount using any known method. For example, if the maximum correlation value is less than a threshold, the reliability of the defocus amount may be considered low.

[0117] In this embodiment, the focus guide function is provided based on the defocus amount and direction obtained using a focus detection configuration based on the image plane phase difference method. However, the basic technical concept of this embodiment does not depend on the method for obtaining the focus level of the focus detection area. Therefore, the focus guide function may be provided for each imaging optical system based on other evaluation values that depend on the focus level of the image formed by each imaging optical system, such as a contrast evaluation value.

[0118] <Focus guide display control processing> Next, the focus guide display control process executed by the system control unit 218 will be described using the flowchart shown in Fig. 11. This process is realized by the system control unit 218 loading a program recorded in the nonvolatile memory 220 into the system memory 219 and executing the program. Note that the focus guide display control process is executed in parallel with the live view display process.

[0119] The focus guide display control process described here does not depend on the number of optical axes (number of imaging optical systems) of the attached lens unit. It is sufficient to execute the focus guide display control process for each image formed by the imaging optical system.

[0120] In S601, the system control unit 218 notifies the image processing unit 214 of the position and size of the focus detection area. There are no particular limitations on the method for determining the position and size of the focus detection area when a twin lens unit 300 with multiple optical axes is attached. As with the case when a lens unit 200 with a single optical axis is attached, the position and size may be specified by the user, may be set based on a characteristic area such as a face, or may be set at a predetermined position and size. However, it is assumed that focus detection areas of the same position and size are set for images formed by each imaging optical system. The image processing unit 214 calculates the defocus amount for the focus detection area notified by the system control unit 218, as described in the first embodiment.

[0121] In S602, the system control unit 218 acquires the defocus amount of the imaging optical system in the focus detection area and its reliability from the image processing unit 214. The reliability may be, for example, a correlation amount corresponding to the defocus amount.

[0122] In S604, the system control unit 218 determines whether the reliability of the defocus amount acquired in S602 is high. For example, if the acquired reliability is equal to or greater than a predetermined threshold, the system control unit 218 can determine that the reliability of the defocus amount is high.

[0123] If the system control unit 218 determines that the reliability of the defocus amount is high, it executes S605, and if not, it executes S610. In S610, the system control unit 218 determines to display the index in a highly blurred display mode (fourth display mode), and executes S611. The fourth display mode is the display mode shown in FIG.

[0124] In S605, the system control unit 218 determines whether the focus detection area is in focus or out of focus. The system control unit 218 can determine that a focus detection area whose absolute value of the defocus amount is equal to or less than a threshold is in focus, and that a focus detection area whose absolute value of the defocus amount exceeds the threshold is out of focus. If the system control unit 218 determines that the focus detection area is in focus, it executes S607, and if it determines that the focus detection area is out of focus, it executes S606.

[0125] In S607, the system control unit 218 determines to display the index in the in-focus state display mode (first display mode), and executes S611. The first display mode is the display mode shown in FIG.

[0126] In S606, the system control unit 218 determines whether the state is front focus or back focus. The system control unit 218 can determine whether the state is front focus or back focus based on the sign of the defocus amount (defocus direction). If the state is determined to be front focus, the system control unit 218 executes S608, and if the state is determined to be back focus, the system control unit 218 executes S609.

[0127] In S608, the system control unit 218 determines to display the index in a front-focus state display mode (second display mode), and executes S611. The second display mode is the display mode shown in FIG. In S609, the system control unit 218 determines to display the index in the display mode of the back focus state (third display mode), and executes S611. The third display mode is the display mode shown in FIG.

[0128] In S611, the system control unit 218 determines the display position of the index and notifies the image processing unit 214 of this and the display mode. In accordance with the notification, the image processing unit 214 generates an image of the index according to the display mode and writes the image of the index to an address in the video memory area according to the display position. As a result, the index is superimposed on the live view image and displayed on the EVF 217 or the display unit 108.

[0129] Next, the live view display operation will be described using the flowchart shown in Fig. 12. This process is realized by loading a program recorded in nonvolatile memory 220 into system memory 219 and executing it by system control unit 218. This process can be executed when performing live view display, for example, in shooting standby mode or during video shooting.

[0130] In S701, the system control unit 218 starts live view display on the EVF 217 or the display unit 108. Specifically, while the imaging unit 211 continues to capture moving images, the system control unit 218 controls the image processing unit 214 to generate display images and write them sequentially to the video memory area of the system memory 219. As a result, a live view image is displayed on the EVF 217 or the display unit 108. The processing from S702 onwards is executed in parallel with the live view display.

[0131] In S702, the system control unit 218 checks the type (number of optical axes) of the attached lens unit. The system control unit 218 communicates with the lens system control circuit 205 or 303 via the communication terminal 124 and the communication terminal 206 or 306 to check the lens type information. The lens type information includes information that can identify the model of the lens unit and the number of optical axes. Note that the lens type information is included in the lens information that is transmitted from the lens unit to the camera 100 when the camera 100 is powered on or when the lens unit is replaced. Therefore, in S702, the system control unit may refer to already acquired lens information without communicating with the lens unit.

[0132] The system control unit 218 executes S703 if it is determined that a lens unit with one optical axis is attached, and executes S704 if it is determined that a lens unit with multiple optical axes (here, two) is attached.

[0133] In S703, the system control unit 218 executes focus guide display processing for a lens unit with one optical axis (a normal lens unit). The focus guide display processing may be the focus guide display control processing described using the flowchart in Fig. 11. As a result, one focus guide is displayed on a live view image formed by one imaging optical system.

[0134] In S704 and S705, the system control unit 218 performs focus guide display processing for each of the images (left image and right image) formed by the two imaging optical systems. Specifically, in S704, the system control unit 218 performs focus guide display processing for the left image, and in S705, performs focus guide display processing for the right image. The focus guide display processing for each image may be the focus guide display control processing described using the flowchart in FIG. 11. As a result, an index 801L is displayed in the left image 800L of the live view image 800 shown in FIG. 9, and an index 801R is displayed in the right image 800R. The indexes 801L and 801R are displayed at the same position within the image.

[0135] In S706, the system control unit 218 determines whether an operation to end the live view display has been detected. If it is determined that an operation to end the live view display has been detected, the system control unit 218 ends the live view display. If it is determined that an operation to end the live view display has not been detected, the system control unit 218 repeats the processing from S702 to continue the live view display. Note that although the processing from S702 is repeated here, after S702 is processed once, S703 and S706, or S704 to S706, may be repeatedly executed in accordance with the determination result of S702 until the lens unit is removed.

[0136] As described above, in this embodiment, a focus guide is displayed for each live view image formed by the imaging optical system according to the number of optical axes of the attached lens unit. Therefore, when twin lens unit 300 having two optical axes (imaging optical systems) is attached, focus guides are displayed for both the left and right images of the live view image.

[0137] According to this embodiment, when a twin-lens unit 300 for capturing images for VR180 is attached to the camera 100, a focus guide is displayed for each of the two live view images on the screen. This allows the degree of focus of each live view image to be easily grasped. Furthermore, when adjusting the focus distance of the imaging optical system, if the focus distances of all imaging optical systems are adjusted in conjunction with each other, the image to be focused on first can be selected. Furthermore, if the focus distance can be adjusted for each imaging optical system, the focus distance can be accurately adjusted for each imaging optical system. In either case, manual focus operation when a multi-lens unit is attached can be effectively supported.

[0138] In this embodiment, the defocus amount obtained as a result of focus detection is used to display the focus guide, but the defocus amount may be calculated using the correction value described in the first embodiment, which makes it possible to display a focus guide that shows the focus state with higher accuracy.

[0139] ●(Third embodiment) Next, a third embodiment of the present invention will be described, which relates to a guide function for manually adjusting the difference in focus degree between the imaging optical systems.

[0140] 13 is a diagram showing an example of an indicator provided by the focus guide function during live view display when the twin lens unit 300 is attached in this embodiment. In this embodiment, an indicator is provided that shows the difference in focus level between the focus detection areas of the left and right images.

[0141] Specifically, an index indicating the difference in the degree of focus is displayed based on the defocus amounts of the left eye optical system 301L and the right eye optical system 301R acquired in the focus guide display process in S704 and S705 of Fig. 12. As shown in Fig. 13, the index has an axis 900, an indicator 901 indicating the origin (difference = 0), positive and negative indicators 902 and 903 indicating the direction of the difference, and a mark 904 indicating the difference in the actual degree of focus.

[0142] The difference in the degree of focus is expressed as a relative value and direction based on the defocus amount for one of the imaging optical systems. For example, when the defocus amount of the left eye optical system 301L is used as the reference, the difference in the degree of focus can be obtained by the following formula: Difference in degree of focus = Defocus amount of the right eye optical system 301R - Defocus amount of the left eye optical system 301L The sign of the difference in focus level is positive for front focus and negative for back focus. Therefore, the difference in focus level will be a positive value if the other image (right image) is more front-focused than the reference image (left image), and a negative value if it is more back-focused.

[0143] 13A shows an example of a first display form of the index when it is determined that the focus degrees of the left and right images (the defocus amounts of the left eye optical system 301L and the right eye optical system 301R) are the same (no difference). When it is determined that the difference in focus degrees is 0, the mark 904 is arranged so as to point to the position (reference position) indicated by the indicator 901 on the axis 900. When it is determined that the difference in focus degrees is 0, the color or visual effect (whether or not it blinks) of the index may be different from other cases. For example, when it is determined that the difference in focus degrees is 0, the index may be green, and otherwise the index may be white.

[0144] 13(B) and 13(C) are diagrams showing examples of the second and third display modes, respectively. These display modes are used to indicate the magnitude and direction of the difference in focus degree based on the difference in defocus amount when the difference in focus degree is not zero but the reliability of the defocus amount is high.

[0145] 13B shows an example of a second display form of the indicator when the focus distance of the other image (right image) is shifted to the closest side (positive direction) relative to the reference image (left image), that is, when the right image is in front of the left image. In this case, the mark 904 is positioned to the right (positive direction) of the indicator 901 so as to point to a position moved by a distance corresponding to the difference in focus level.

[0146] 13C shows an example of a third display mode of the indicator when the focus distance of the other image (right image) is shifted toward infinity (negative direction) relative to the reference image (left image), that is, when the right image is in back focus relative to the left image. In this case, the mark 904 is positioned to the left (negative direction) of the indicator 901 so as to point to a position moved by a distance corresponding to the difference in focus level.

[0147] In the second and third display forms, the magnitude and direction of the difference in the relative focus degree between the left and right images can be shown to the user by the position of mark 904. Specifically, the magnitude of the difference in focus degree can be shown by the distance of mark 904 from the position on axis 900 indicated by indicator 901, which indicates the reference point where the difference is 0. Also, whether mark 904 is located to the right or left of the position on axis 900 indicated by indicator 901 can indicate whether the focus distance of the other image is shifted to the close side or to the infinity side relative to the reference image.

[0148] 13(D) shows an example of a fourth display form of the index when the reliability of the defocus amount for at least one of the imaging optical systems is low, such as when the left image, the right image, or both, are significantly blurred. In this case, the reliability of the difference in focus degree obtained by the above formula is also low, so the mark 904 indicating the magnitude and direction of the difference in focus degree is not displayed. Furthermore, the axis 900 and indicators 901 to 903, which are always displayed among the indexes, may have different colors or visual effects (such as whether they blink) in the fourth display form from those in the first to third display forms. For example, the display color may be gray.

[0149] Note that the display formats shown in Figures 13(A) to 13(D) are merely examples, and the difference in focus degree may be displayed using other types of indicators, such as using numerical values or marks or indicators of different shapes.

[0150] Next, the display control process for the difference in the focus degree between the left and right images, which is executed by the system control unit 218, will be described using the flowchart shown in Fig. 14. This process is executed when the twin lens unit 300 is attached to the camera 100. This process is realized by the system control unit 218 loading a program recorded in the non-volatile memory 220 into the system memory 219 and executing it. The display control process for the difference in the focus degree between the left and right images is executed in parallel with the live view display process. It can also be executed in parallel with the display control process for the focus guide described in the second embodiment.

[0151] In S1001, the system control unit 218 acquires the defocus amounts and reliability of the right-eye optical system 301R and the left-eye optical system 301L from the image processing unit 214. When executing focus guide display control processing, the system control unit 218 may refer to the defocus amounts and reliability acquired in S602 of FIG. 11. Then, the system control unit 218 determines whether the reliability of at least one of the defocus amounts is low. If the system control unit 218 determines that the reliability of at least one of the defocus amounts is low, it executes S1008, and if not, it executes S1002.

[0152] In S1002, the system control unit 218 calculates the difference in the degree of focus using one of the left and right images as a reference. When the left image is used as a reference, the system control unit 218 can calculate the difference in the degree of focus using the calculation formula described above. After calculating the difference in the degree of focus, the system control unit 218 executes S1003.

[0153] In S1003, the system control unit 218 determines whether the difference in the degree of focus calculated in S1002 is 0 or not, and if it is determined that the difference is 0, executes S1005, and if it is not determined that the difference is 0, executes S1004.

[0154] In S1004, the system control unit 218 determines whether the sign of the difference in the degree of focus calculated in S1002 is positive or negative. If the system control unit 218 determines that the sign of the difference is positive, it executes S1006, and if the sign of the difference is negative, it executes S1007.

[0155] In S1005, the system control unit 218 determines to display the indicator in the first display mode (FIG. 13(A)), and executes S1009. In S1006, the system control unit 218 determines to display the indicator in the second display format (FIG. 13(B)), and executes S1009. In S1007, the system control unit 218 determines to display the indicator in the third display mode (FIG. 13(C)), and executes S1009. In S1008, the system control unit 218 determines to display the indicator in the fourth display mode (FIG. 13(D)), and executes S1009.

[0156] In S1009, the system control unit 218 notifies the image processing unit 214 of the display format of the index determined in S1005 to S1008 and the display position of the mark 904 corresponding to the difference in focus degree. In accordance with the notification, the image processing unit 214 generates an image of the index corresponding to the display format and writes the image of the index to an address in the video memory area corresponding to the predetermined display position of the index. As a result, the index indicating the difference in focus degree between the left and right images is superimposed on the live view image and displayed on the EVF 217 or the display unit 108.

[0157] Next, the live view display operation in this embodiment will be described using the flowchart shown in Fig. 15. In Fig. 15, the same operations as in the second embodiment are denoted by the same reference numerals as in Fig. 12. This processing is realized by loading a program recorded in non-volatile memory 220 into system memory 219 and executing it by the system control unit 218. This processing can be executed when performing live view display, for example, in a shooting standby state or during video shooting.

[0158] The processes in S701 to S705 are the same as those in the second embodiment, and therefore description thereof will be omitted. When the focus guide display process for the right image in S705 ends, in S1106 the system control unit 218 determines whether or not to display the difference in focus degree between the left and right images.

[0159] Whether or not to display the difference in focus degree between the left and right images may be, for example, one of the user setting items, or may be determined by the system control unit 218 depending on the operation mode of the camera 100. For example, when the video mode of the camera 100 is set to an adjustment mode in which the user adjusts the difference in focus degree between the imaging optical systems of the twin lens unit 300, the system control unit 218 determines to display the difference in focus degree between the left and right images. In the adjustment mode, the user can adjust the focal distances of the right eye optical system 301R and the left eye optical system 301L, for example, by operating individual focus rings, so that the difference in focal distance between the right eye optical system 301R and the left eye optical system 301L is eliminated.

[0160] If it is determined that the difference in the degree of focus between the left and right images is to be displayed, the system control unit 218 executes S1107, and if not, executes S706. In S1107, the system control unit 218 executes the display control process described using the flowchart in Fig. 14. While viewing the displayed index, the user can adjust the focus distance of the imaging optical system (here, the right eye optical system 301R) that forms an image other than the reference image so that there is no difference in the degree of focus.

[0161] 16 is a diagram showing an example of the live view display in S1107. As in the second embodiment, one focus guide 1201L and one focus guide 1201R are displayed for the left image 1200L and the right image 1200R in one frame of live view image 1200 obtained by the image sensor that constitutes the image capture unit 211. In this embodiment, an index 1202 that indicates the difference in focus between the left and right images 1200L and 1200R is also displayed between the left and right images 1200L and 1200R.

[0162] The processing of S706 is the same as in the second embodiment, and therefore a description thereof will be omitted. In this embodiment, when the twin lens unit 300 is attached to the camera 100, an indicator is displayed that indicates the magnitude and direction of the difference between the two imaging optical systems or the degree of focus of the images. This allows the user to easily grasp the deviation in the focal distance of the imaging optical systems that has occurred due to aging or other reasons. While viewing the indicator, the user can make adjustments so that the deviation in the focal distance between the imaging optical systems becomes zero.

[0163] In the second and third embodiments, the display of a focus guide was described as supporting manual focus operation. However, the basic technical idea is to display some kind of display that supports shooting of images formed by each imaging optical system when a multi-lens unit is attached, and the display content is not limited to a focus guide that indicates the degree of focus. For example, a peaking pattern that indicates overexposed or underexposed areas may be superimposed on the live view image.

[0164] In this embodiment, the defocus amount used to display the focus guide may also be calculated using the correction value described in the first embodiment, thereby realizing a focus guide display that shows the focus state with higher accuracy.

[0165] ●(Fourth embodiment) Next, a fourth embodiment of the present invention will be described. This embodiment relates to an index that collectively indicates the degree of focus and the difference between the degrees of focus for each image.

[0166] 17(A) to 17(D) show examples of first to fourth display forms of the index according to this embodiment. The index has an axis 1700, an indicator 1701 that is arranged near the center of the axis 1700 and indicates the in-focus position, an indicator 1702 that indicates a back-focus state and an indicator 1703 that indicates a front-focus state, both of which are arranged near both ends of the axis 1700. The index also has a mark 1704 that indicates the degree of focus of the right image and a mark 1705 that indicates the degree of focus of the left image, which are arranged near the top and bottom of the axis 1700 to indicate the position on the axis 1700.

[0167] 17A is an example of the first display form when both the right and left images are in focus (the defocus amount is 0 for both the right-eye optical system 301R and the left-eye optical system 301L). In this case, there is no difference in the degree of focus between the right and left images, so the marks 1704 and 1705 are arranged to point to the same position on the axis 1700. Also, because the defocus amount is 0, the marks 1704 and 1705 are arranged to point to the position on the axis 1700 indicated by the indicator 1701. In this case, as with the first display form used when in focus in the second embodiment, at least a part of the indicators may be displayed using a color or visual effect that differs from that of the other display forms.

[0168] 17B shows an example of the second display mode in a case where there is no difference in the degree of focus between the right and left images but the image is in a front-focus state. In this case, because there is no difference in the degree of focus between the right and left images, the marks 1704 and 1705 are arranged to point to the same position on the axis 1700, as in the first display mode. Furthermore, because the image is in a front-focus state, the marks 1704 and 1705 are arranged to point to a position on the axis 1700 that is moved a distance corresponding to the magnitude of the defocus amount toward the end of the indicator 1703 from the position on the axis 1700 indicated by the indicator 1701. In order to bring the left and right images into focus while maintaining the same difference in the degree of focus between the left and right images, a display may be displayed that prompts the user to drive a focus ring that simultaneously adjusts the focal distances of the right-eye optical system 301R and the left-eye optical system 301L.

[0169] FIG. 17C shows an example of a third display mode in which the right and left images are different in focus and neither is in focus. Here, both the right and left images are assumed to be in front focus. In this case, the mark 1704 is positioned to point to a position on the axis 1700 that is shifted from the position on the axis 1700 indicated by the indicator 1701 toward the end of the indicator 1703 by a distance corresponding to the magnitude of the defocus amount of the right-eye optical system 301R. The mark 1705 is positioned to point to a position on the axis 1700 that is shifted from the position on the axis 1700 indicated by the indicator 1701 toward the end of the indicator 1703 by a distance corresponding to the magnitude of the defocus amount of the left-eye optical system 301L. In the example shown in FIG. 17C, because the defocus amount of the left-eye optical system 301L is larger, the mark 1705 is positioned to point to a position on the axis 1700 closer to the end of the indicator 1703 than the mark 1704.

[0170] While looking at the index, the user adjusts the focal length of the left eye optical system 301L so that the marks 1704 and 1705 point to the same position on the axis 1700. After that, the left and right images can be brought into focus by simultaneously adjusting the focal lengths of the right eye optical system 301R and the left eye optical system 301L.

[0171] In this embodiment, the positions on axis 1700 indicated by marks 1704 and 1705 can present to the user the degree of focus of the right and left images, the difference between them, and whether the image is in front focus or back focus.

[0172] In this embodiment, the marks 1704 and 1705 independently indicate the magnitude and direction of the defocus amounts of the right-eye optical system 301R and the left-eye optical system 301L, so there is no need to calculate the difference in the degree of focus calculated in the previous embodiment. The system control unit 218 can determine the positions on the axis 1700 to which the marks 1704 and 1705 should point (i.e., the display positions of the marks 1704 and 1705) based on the defocus amounts of the right-eye optical system 301R and the left-eye optical system 301L and their signs.

[0173] 17(D) shows an example of a fourth display mode of the index when the reliability of the defocus amount of the right-eye optical system 301R and the left-eye optical system 301L is low, such as when both the right and left images are significantly blurred. In this case, the marks 1704 and 1705 are not displayed because the reliability of the defocus amount is low. Furthermore, among the indexes, the axis 1700 and indicators 1701 to 1703, which are always displayed, may have different colors or visual effects (such as whether they blink) in the fourth display mode from those in the first to third display modes. For example, the display color may be gray.

[0174] Although not shown here, if the reliability of the defocus amount of one of the right eye optical system 301R and the left eye optical system 301L is low and the reliability of the defocus amount of the other is high, mark 1704 or 1705 based on the defocus amount with high reliability may be displayed.

[0175] 17(A) to 17(D) are merely examples. The degree of focus and the difference therebetween may be displayed using other indicators, such as by indicating the magnitude and direction of the defocus amount of the right-eye optical system 301R and the left-eye optical system 301L as values, or by using marks or indicators of different shapes.

[0176] The system control unit 218 can display the indicators of this embodiment in S1107 in a live view display operation in which, for example, S704 and S705 are deleted from the flowchart of FIG. 15 described in the third embodiment. In this case, in S1107, the system control unit 218 acquires the defocus amount and its reliability for each imaging optical system from the image processing unit 214. If all the reliability levels are low, the system control unit 218 determines the fourth display mode. If all the defocus amount reliability levels are high, the system control unit 218 determines the first display mode if all the defocus amounts are 0, the second display mode if all the defocus amounts are not 0 and there is no difference, and the third display mode if there is a difference in the defocus amounts. Furthermore, if the system control unit 218 determines one of the first to third display modes, the system control unit 218 also determines the display positions of the marks 1704 and 1705. Then, the system control unit 218 notifies the image processing unit 214 of the determined display mode and, if the marks 1704 and 1705 are to be displayed, the display positions of the marks 1704 and 1705.

[0177] 18 is a diagram showing an example of a live view display in this embodiment. Focus guides 1801L and 1801R are displayed on a left image 1800L and a right image 1800R, respectively, in one frame of a live view image 1800 obtained by an image sensor constituting the imaging unit 211. In this embodiment, an indicator 1802 is also displayed between the left image 1800L and the right image 1800R, indicating the difference in focus between the left and right images, as well as the magnitude and direction of the defocus amount for each imaging optical system.

[0178] In this embodiment, in addition to the effects of the third embodiment, the magnitude and direction of the defocus amount for each imaging optical system are presented in a different way than in the second embodiment, so that the user can obtain the necessary information by referring to indicators that are easy for the user to understand.

[0179] In this embodiment, too, the basic technical idea is to display some kind of image to assist in photography regarding the images formed by each imaging optical system when a multi-lens unit is attached, and the displayed content is not limited to the degree of focus or its difference. For example, a peaking pattern indicating overexposed or underexposed areas may be displayed superimposed on the live view image.

[0180] ●(Fifth embodiment) Next, a fifth embodiment of the present invention will be described. This embodiment relates to a technique for automatically adjusting the difference in focus level between the imaging optical systems when a multi-lens unit is attached.

[0181] In the third embodiment, the difference in the degree of focus between the right-eye optical system 301R and the left-eye optical system 301L (the difference in the amount of defocus obtained for the right image and the left image) is obtained. By using this difference to drive the focus lens of the right-eye optical system 301R or the left-eye optical system 301L, the difference in the degree of focus between the imaging optical systems (deviation in focal distance) can be automatically adjusted.

[0182] 19 is a flowchart of the process for adjusting misalignment between the imaging optical systems, which is executed by the system control unit 218. This process is realized by the system control unit 218 loading a program recorded in the nonvolatile memory 220 into the system memory 219 and executing it. The misalignment adjustment process may be executed in response to a user instruction, or may be executed automatically at a predetermined timing. An example of the predetermined timing may be when the system control unit 218 determines that the attached lens unit is a multi-lens unit (for example, when the lens unit is replaced, when the camera 100 is started, etc.).

[0183] When the operation mode of the camera 100 is the adjustment mode described in the third embodiment, the user manually adjusts the misalignment between the imaging optical systems. Therefore, the misalignment adjustment process automatically performed by the system control unit 218 in this embodiment may be performed when the operation mode is not the adjustment mode. Note that the misalignment adjustment process is performed in parallel with the live view display operation.

[0184] In S1901, the system control unit 218 sets one of the right-eye optical system 301R and the left-eye optical system 301L of the twin lens unit 300 as a reference imaging optical system. For example, assume that the twin lens unit 300 is capable of driving the right-eye optical system 301R and the left-eye optical system 301L in conjunction with each other and of driving only the left-eye optical system 301L by operating the focus ring. In this case, the system control unit 218 sets the right-eye optical system 301R, which cannot be driven independently, as the reference.

[0185] The system control unit 218 may also set the imaging optical system corresponding to the eye that is previously set as the user's dominant eye as the reference imaging optical system. Alternatively, if there is a difference in subject detection accuracy or focus detection accuracy between the right eye optical system 301R and the left eye optical system 301L, the imaging optical system with the better accuracy may be set as the reference.

[0186] In S1902, the system control unit 218 acquires information about the current focus detection areas for each of the right-eye optical system 301R and the left-eye optical system 301L, which is stored in the system memory 219. Note that if the focus detection areas for the right-eye optical system 301R and the left-eye optical system 301L are set at the same position relative to the optical axis, it is sufficient to read out information about either one of the focus detection areas. The system control unit 218 notifies the image processing unit 214 of the focus detection area information and instructs it to calculate a defocus amount. As described above, the image processing unit 214 calculates the defocus amount and its reliability based on the focus detection area signals for each of the right and left images.

[0187] In S1903, the system control unit 218 acquires from the image processing unit 214 the defocus amount DEF_L of the left eye optical system 301L and the defocus amount DEF_R of the right eye optical system 301R.

[0188] In S1904, the system control unit 218 calculates the difference in the defocus amount of the other imaging optical systems with respect to the defocus amount of the imaging optical system set as the reference in S1901. For example, if the right eye optical system 301R is set as the reference in S1901, the system control unit 218 calculates the difference in the defocus amount DEF_dif using the following formula. DEF_dif = DEF_L - DEF_R

[0189] Next, in S1905, the system control unit 218 drives the focus lens of the non-reference imaging optical system (here, the left eye optical system 301L) in the optical axis direction by PLS_dif, which is the drive amount and drive direction corresponding to DEF_dif detected in S1904. The lens drive amount PLS_dif is, for example, PLS_dif = DEF_dif / SENS_L Ask for more.

[0190] Here, SENS_L is a conversion coefficient for converting the defocus amount of the non-reference (i.e., the imaging optical system to be adjusted) (here, the left-eye optical system 301L) imaging optical system into a lens drive amount, and is stored in advance in the lens unit. The twin lens unit 300 stores the focus sensitivities of the right-eye optical system 301R and the left-eye optical system 301L in a non-volatile memory inside the lens system control circuit 303. Note that when there are multiple imaging optical systems with the same configuration, such as the twin lens unit 300, one focus sensitivity may be used commonly for each imaging optical system.

[0191] The above processing allows automatic adjustment to eliminate the difference in focus degree (deviation in focal distance) between the imaging optical systems. According to this embodiment, the difference in focus degree (deviation in focal distance) between the imaging optical systems that was manually adjusted in the third embodiment can be automatically adjusted, thereby saving the user the trouble of making adjustments and improving usability.

[0192] ●(Sixth embodiment) Next, a sixth embodiment of the present invention will be described. This embodiment provides a focus calibration function when a multi-lens unit is attached.

[0193] Fig. 20 shows the indicators described with reference to Fig. 17 in the fourth embodiment. Fig. 20(A) shows a third display mode, in which marks 1704 and 1705 indicate that the right image is in focus, but the left image is in front-focus. From this state, it is assumed that the user operates the focus ring to adjust the focal distance of the left-eye optical system 301L so that mark 1704, which indicates the degree of focus of the left image, comes to the position indicated by indicator 1701 (so that the indicator is in the state shown in Fig. 20(B)).

[0194] As a result, for example, both the right image 1800R and the left image 1800L in the live view image 1800 displayed in Fig. 18 should be in focus. However, due to factors such as manufacturing errors, deterioration over time, and the environment in the components such as the multiple lenses and reflecting mirrors included in the right eye optical system 301R and the left eye optical system 301L, a discrepancy may occur between the calculated defocus amount and the degree of focus of the image.

[0195] Therefore, even if marks 1704 and 1705 indicate an in-focus state, the user may feel that the right and / or left image observed through the display unit 108 or EVF 217 is not in focus. In particular, if the degree of focus of one image is lower than the degree of focus of the other image, the two images are displayed adjacent to each other, making it easy for the user to notice the difference in focus. For this reason, this embodiment provides a function (focus calibration function) for correcting the difference between the state in which an image that the user feels is in focus is obtained and the state in which the camera 100 determines that the image is in focus.

[0196] In this embodiment, a correction value for correcting the difference between the state in which an image perceived by the user as being in focus and the state determined as being in focus by the camera 100 is called a calibration value. The calibration value can be set and stored independently for each of the right-eye optical system 301R and the left-eye optical system 301L.

[0197] 21 is a diagram showing an example of a calibration line guide displayed on the display unit 108 or the EVF 217 when the focus calibration function is being executed. Here, for ease of explanation and understanding, it is assumed that the focus lens of the left-eye optical system 301L and the focus lens of the right-eye optical system 301R can be driven independently. However, the focus calibration function can also be implemented by driving the focus lenses of both imaging optical systems in conjunction with each other, and by configuring the focus lens of one imaging optical system to be independently driven. In this case, calibration of the imaging optical system whose focus lens cannot be driven independently can be performed first, and then calibration of the remaining imaging optical systems can be performed.

[0198] FIG. 21(A) shows an example of a first display form of a calibration guide 2110. The calibration guide 2110 has an axis 2100 corresponding to the range of calibration values. A scale is provided on the axis 2100, and values are indicated on some of the scale marks. Here, an indicator 2101 indicating a calibration value of 0, an indicator 2102 indicating the maximum negative value (-20 in this case), and an indicator 2103 indicating the maximum positive value (+20 in this case) are displayed near the bottom of the axis 2100. The sign of the calibration value is set so that positive indicates the front focus direction and negative indicates the back focus direction, so that the positional relationship between the mark and the reference is the same as in FIG. 17. Note that indicators indicating values other than these may also be added. In the example of FIG. 21(A), indicators indicating -10 and +10 are added.

[0199] Further, near the axis 2100, a mark 2104 indicating the calibration value of the right-eye optical system 301R and a mark 2105 indicating the calibration value of the left-eye optical system 301L are arranged. Furthermore, marks 1704 and 1705 indicating the degree of focus of the right-eye optical system 301R and the left-eye optical system 301L are arranged on the axis 2100. Here, the display positions of the marks 1704 and 1705 are determined with the position of a calibration value of 0 being the defocus amount of 0. Further, near the left end of the axis 2100, an indicator 2106 is arranged which numerically indicates the difference in the calibration values of the right-eye optical system 301R and the left-eye optical system 301L. Here, the relative value of the calibration value of the right-eye optical system 301R when the calibration value of the left-eye optical system 301L is set to 0 is shown as the difference in calibration values.

[0200] 21A shows that the defocus amounts of the right-eye optical system 301R and the left-eye optical system 301L are both 0, but the focus lens positions have been corrected by the calibration value. Specifically, the focus lens position corresponding to a defocus amount of 0 is corrected by 2 in the front-focus direction (+) for the right-eye optical system 301R, and by 4 in the back-focus direction (-) for the left-eye optical system 301L. Here, the unit of the calibration value is predetermined by the camera 100. For example, it may be the number of pulses used to drive the focus lens.

[0201] With the marks 1704 and 1705 indicating the in-focus state, the user adjusts the focus lens positions of the right-eye optical system 301R and the left-eye optical system 301L, for example, while observing the right image 1800R and the left image 1800L of the live-view image 1800. Then, when the user feels that the degree of focus in the focus detection areas of the right image 1800R and the left image 1800L is at its highest, the user issues an instruction to the camera 100 via the operation member 230. When the system control unit 218 detects this instruction, it stores the calibration values of the right-eye optical system 301R and the left-eye optical system 301L at that time in the non-volatile memory 220.

[0202] The system control unit 218 corrects the lens driving amount based on the defocus amount based on the calibration value stored in the nonvolatile memory 220, and then transmits the corrected amount to the lens system control circuit 303. This makes it possible to obtain an image that the user perceives as most in-focus when the defocus amount is 0. In other words, it is possible to correct the discrepancy between the degree of focus perceived by the user and the degree of focus determined by the camera 100.

[0203] FIG. 21B shows an example of a second display form of the calibration guide 2110. In the second display form, the information presented in the first display form is shown for each imaging optical system. Specifically, axes 2100R and 2100L are provided for each imaging optical system, and marks 1704 (1705) indicating the focus level and marks 2104 (2105) indicating the calibration value are arranged on the corresponding axes. Furthermore, instead of indicator 2106 indicating the difference in calibration values, indicators 2106R and 2106L indicating the calibration value for each imaging optical system are arranged on the corresponding axes 2100R and 2100L. In the second display form, since information is presented for each imaging optical system, the display area is large, but the user can easily grasp the information. Note that information regarding one imaging optical system may be displayed in response to a user instruction.

[0204] FIG. 21C shows an example of a third display form of the calibration guide 2110. This display form is similar to the first display form shown in FIG. 21A, but the marks 1704 and 1705 are both at +4, indicating a front-focus state. The marks 2104 and 2105 indicating the calibration value indicate the calibration value at the position when the marks 1704 and 1705 are at the defocus amount 0 position (FIG. 21A). When the marks 1704 and 1705 are not at the defocus amount 0 position, the marks 2104 and 2105 indicate the calibration value based on the difference in position from the marks 1704 and 1705. Therefore, in FIG. 21C, the calibration value of the right-eye optical system 301R is −2, and the calibration value of the left-eye optical system 301L is 0. Note that the calibration value may be indicated by the display positions of the marks 2104 and 2105 regardless of the defocus amounts of the right-eye optical system 301R and the left-eye optical system 301L.

[0205] A method of focus calibration when the defocus amounts of the right-eye optical system 301R and the left-eye optical system 301L are not 0 will be described with reference to FIG. Fig. 22(A) shows an example of a sample image corresponding to the focus degree in Fig. 21(C). When focus calibration is performed with the defocus amount not equal to 0, the system control unit 218 displays a sample image corresponding to the current focus degree and a live view image (Fig. 22(B)) on the display unit 108 or the EVF 217.

[0206] When calibrating the focus state as in Fig. 21(A), the sample image in Fig. 22(A) is not necessary. The sample image in Fig. 22(A) is, for example, an image corresponding to the current focus degree, which is predicted from the design state of the imaging optical system by using a line image stored in advance in non-volatile memory 220.

[0207] The user compares the image in Fig. 22(A) with the live view image in Fig. 22(B) (left image 2201L and right image 2201R). For example, if only the right image 2201R in the live view image in Fig. 22(B) appears to have a large amount of blur, the user adjusts the focus lens position of the right eye optical system 301R to a position where the right image 2201R appears to be in focus to the same degree as the left image 2201L. This sets the calibration value of the right eye optical system 301R.

[0208] 21(C) corresponds to an example of the display of the calibration guide after the calibration value of the right-eye optical system 301R has been set in this way. By changing the calibration amount of the right-eye optical system 301R from the original value (0) to -2, the blur conditions of the left and right images become consistent, and the desired image can be obtained. Note that, without displaying a sample image, the focus lens position of the imaging optical system that forms the other image may be adjusted so that the left or right image in the live view image has the same degree of focus as the one that is perceived to have a higher degree of focus.

[0209] The calibration values may also be stored in at least one of the camera 100 and the twin lens unit 300. This allows the calibration values to be acquired from the twin lens unit 300 and used in a camera other than the camera that performed the calibration of the twin lens unit 300.

[0210] According to this embodiment, it is possible to set a calibration value that corrects the focus lens position for the imaging optical system of the twin lens unit, which makes it possible to correct the difference between the focus state determined by the camera and the focus state perceived by the user for the twin lens unit.

[0211] ●(Seventh embodiment) Next, a seventh embodiment of the present invention will be described. This embodiment relates to a focus calibration function when a pair of parallax images, such as a left image and a right image captured by a twin lens unit 300, is recorded in a refocusable format. A refocusable image is an image for which the subject distance at which the image is focused can be changed after capture (recording). For example, the image may be an image captured by a light field camera, or an image in which a group of images captured of the same scene with different focus distances are associated and recorded.

[0212] The focus calibration function provided by this embodiment can be used, for example, to set a calibration value for correcting the difference in focus (focus state) between the right and left images when a user views a recorded pair of parallax images. Here, it is assumed that the user views the pair of parallax images while wearing XR goggles, which are display devices having a display unit for the left eye and a display unit for the right eye. Here, XR is a general term for VR (virtual reality), AR (augmented reality), and MR (mixed reality).

[0213] The left diagram of Fig. 23(a) is a perspective view showing an example of the appearance of XR goggles 2300. As shown in the right diagram of Fig. 23(a), XR goggles 2300 are generally worn on the eye area SO of the head. Fig. 23(b) is a perspective view showing an example of the appearance of the XR goggles 2300 as seen from the wearing surface side. Fig. 23(c) is a diagram schematically showing the positional relationship between the eyepiece lenses 2301R and 2301L, the right eye display unit 2308R and the left eye display unit 2308L, and the user's right eye 501R and left eye 501L when the XR goggles 2300 are worn.

[0214] The XR goggles 2300, for example, display a right image obtained by the twin lens unit 300 on the right-eye display unit 2308R and a left image on the left-eye display unit 2308L. Because the right and left images are a pair of parallax images, the user recognizes the right and left images as 3D images by viewing the right image with the right eye 501R through the eyepiece 302R and viewing the left image with the left eye 501L through the eyepiece 302L. Note that the pair of parallax images displayed on the XR goggles 2300 is not limited to those captured by the twin lens unit 300. For example, stereoscopic viewing is also possible when right and left images captured by a stereo camera are displayed on the right-eye display unit 2308R and left-eye display unit 2308L of the XR goggles 2300.

[0215] This embodiment assumes a case where it is possible to independently change the focal distances of the right and left images that make up a pair of parallax images displayed on the XR goggles 2300. For example, video data in which the right and left images are recorded in a refocusable format may be played back on a computing device and viewed on the XR goggles 2300 connected to the computing device.

[0216] 24A and 24B are diagrams showing a schematic diagram of a change in the focused subject by refocusing. Fig. 24A shows a state in which subject 2403 is in focus, but subjects 2402 and 2404 are not in focus. Note that the images of subjects 2402 and 2404 are not blurred and are out of focus because they are located outside the depth of field.

[0217] If the image shown in Figure 24(a) (a still image or a frame of a video) is recorded in a refocusable format, it can be altered to focus on object 2402 or object 2404. Figures 24(b) and 24(c) show the image altered to focus on objects 2402 and 2404, respectively.

[0218] For example, when a user specifies a position to be focused, the system control unit 218 changes the image so that the specified position is in focus. Any known method can be used to change the focused subject through refocusing. When capturing a refocusable image with the camera 100, the number of photodiodes that share the microlens 401 can be increased in both the horizontal and vertical directions.

[0219] Fig. 25 shows an example of a right-eye image 2501R and a left-eye image 2501L corresponding to the image shown in Fig. 24(a). By displaying the right-eye image 2501R and the left-eye image 2501L on the right-eye display unit 2308R and the left-eye display unit 2308L of the XR goggles 2300, a user wearing the XR goggles 2300 can get the feeling that they are present in the scene shown in Fig. 24(a). By detecting the movement of the user's hand and associating it with a position in the image, the user can virtually touch the subject 2403 to designate the subject 2403 as the subject to be focused on.

[0220] FIG. 26 is a block diagram showing an example of the functional configuration of a computer 2600 that can be used as a computing device in this embodiment.

[0221] The display 2701 displays information about data being processed by an application program, various message menus, and the like, and is configured with an LCD (Liquid Crystal Display) or the like. The display 2701 may be a touch display. A display controller 2702 controls the screen display on the display 2701. A keyboard 2703 and a pointing device 2704 are used to input characters and to point to icons and buttons in a GUI (Graphical User Interface). A CPU 2705 controls the entire computer 2600.

[0222] A ROM 2706 (Read Only Memory) stores programs and parameters executed by the CPU 2705. A RAM (Random Access Memory) 2707 is used as a work area when the CPU 2705 executes various programs, a temporary save area during error processing, etc.

[0223] A hard disk drive (HDD) 2708 and a removable media drive (RMD) 2709 function as external storage devices. A removable media drive is a device that reads and writes data from or to a removable recording medium, and may be a flexible disk drive, optical disk drive, magneto-optical disk drive, memory card reader, or even a removable HDD. In addition to or instead of the HDD 2708, an SSD (Solid State Drive) may be provided.

[0224] Programs that realize various functions of the computer 2600 described in this embodiment, an OS, application programs such as a browser, data, libraries, etc. are stored in one or more of the ROM 2706, HDD 2708, and RMD 2709.

[0225] The expansion slot 2710 is a slot for installing an expansion card that complies with, for example, the PCI (Peripheral Component Interconnect) bus standard. Various expansion boards such as a video capture board, a sound board, and a GPIB board can be installed in the expansion slot 2710.

[0226] The external IF 2711 is an interface for communicatively connecting the computer 2600 and an external device, and complies with one or more wired and / or wireless communication standards. The external IF 2711 may have an interface that complies with one or more of, for example, wireless LAN, USB (Universal Serial Bus), HDMI (registered trademark), Bluetooth (registered trademark), 4G (LTE), 5G, etc.

[0227] The bus 2712 comprises an address bus, a data bus, and a control bus, and connects the above-mentioned units.

[0228] Next, the focusing calibration function in this embodiment will be described. Here, the focusing calibration function for the pair of parallax images displayed on the XR goggles 2300 is provided by executing an application program that provides the focusing calibration function in the above-mentioned computer 2600. However, the focusing calibration function can also be provided by having the system control unit 218 of the camera 100 execute a similar program.

[0229] Here, it is assumed that data of a pair of parallax images (right and left images) is recorded in a refocusable format in a storage device (e.g., HDD 2708) of computer 2600. The data of the right and left images may be acquired from camera 100 or other external devices via external IF 2711. Also, as shown in FIG. 27(a), XR goggles 2300 are connected to external IF 2711 of computer 2600, and the right and left images can be displayed.

[0230] The focus calibration function provided by the computer 2600 displays a pair of parallax images 2602 and a calibration guide 2601 on a display 2701 .

[0231] 27(b) shows an example of a calibration guide 2601. The calibration guide 2601 has an axis 2610 that indicates a distance range from the closest end to infinity. A scale is provided on the axis 2610, and values are indicated at some of the scale marks. Here, an indicator 2611 indicating the minimum value (close end) and an indicator 2612 indicating the maximum value (infinity) are displayed near the bottom of the axis 2610. Indicators indicating values for several distances between the minimum and maximum values are also displayed.

[0232] Also, a mark 2613 indicates the position of the axis 2610 corresponding to the focal distance when the currently displayed pair of parallax images was captured. The focal distance when capturing is recorded together with the image data as one piece of information when capturing.

[0233] In the example shown in FIG. 27(b), mark 2613 indicates 1 m. The image at this time is assumed to be, for example, an image focused on subject 2403 shown in FIG. 24(a). By moving mark 2613 to change the distance indicated by mark 2613, the user can indicate the changed focus distance to CPU 2705. The user can move mark 2613 to a desired position by operating keyboard 2703 or pointing device 2704, or by touching display 2701.

[0234] For example, as shown in FIG. 27(c), assume that the user moves the mark 2613 to a position where the axis 2610 points to 2 m. The CPU 2705 changes the focal distance of the image data according to the changed position of the mark 2613. The CPU 2705 can change the focal distance using a method according to the recording method. If a light field image is recorded, the focal distance can be changed by a shift calculation. Furthermore, if a group of images with different focal distances are recorded, it is sufficient to extract an image focused at a specified distance (an image whose depth of field includes the specified distance). Here, it is assumed that the image shown in FIG. 24(b) is an image with a focal distance of 2 m, and the image shown in FIG. 24(c) is an image with a focal distance of 3 m.

[0235] There should be no difference in the degree of focus between the right and left images because they are images focused at the distance indicated by mark 2613. Differences in the degree of focus between the right and left images can occur due to factors such as manufacturing errors, deterioration over time, and the environment of the components such as the multiple lenses and reflecting mirrors included in each of the two imaging optical systems that form the right and left images.

[0236] Therefore, the user may feel that the focus levels of the right and left images observed through the display 2701 or the XR goggles 2300 are different. In particular, if the focus level of one image is lower than that of the other image, the two images are displayed adjacent to each other, making it easy for the user to notice the difference in focus level. Therefore, in this embodiment, a focus calibration function is provided to correct the discrepancy between the focus state expected by the user for a specified distance and the focus level of the displayed image.

[0237] Fig. 27(d) is a diagram showing an example of the display form of the calibration guide 2601 when setting the calibration values. In this embodiment, as shown in Fig. 27(a), the calibration values for the left and right images are set by fine-tuning the focus distance while checking the focus degree of the parallax image pair 2602 displayed on the display 2701.

[0238] 27(d) shows a state in which the CPU 2705 changes the display mode of the calibration guide 2601 in response to a user instruction to transition from the state of FIG. 27(c) to a mode for setting calibration values (calibration mode). Therefore, the currently displayed disparity image pair 2602 is an image presented by the CPU 2705 as an image focused at an object distance of 2 m.

[0239] Here, when transitioning to calibration mode, the indicator attached to the scale of axis 2610 is changed from showing distance to showing a numerical value with the current setting value set to 0. This is to make it easier to grasp the magnitude and direction of the calibration value. However, as shown in FIG. 26(c), the indicator may remain showing distance.

[0240] Furthermore, when the CPU 2705 transitions to the calibration mode, it displays an indicator 2624 that indicates the current calibration value numerically. Furthermore, the CPU 2705 displays a mark 2623 for setting the calibration value of the right image and a mark 2622 for setting the calibration value of the left image. The user can move the marks 2622 and 2623 by touching the keyboard 2703, the pointing device 2704, the display 2701, or the like.

[0241] When a movement operation of the mark 2623 is detected, the CPU 2705 changes the focus distance of the right image of the pair of parallax images 2602 according to the direction and amount of movement from the initial position (0). For example, when movement of the mark 2623 in the + direction is detected, the CPU 2705 changes the focus distance toward infinity, and when movement of the mark 2623 in the - direction is detected, the CPU 2705 changes the focus distance toward the closest end. Note that the amount of change in distance per scale may be a predetermined constant value. When a movement operation of the mark 2622 is detected, the CPU 2705 changes the focus distance of the left image in the same way as the right image. The user operates the marks 2622 and 2623 so that the right and left images appear to be in focus at the specified distance.

[0242] Figure 27(d) shows a state where the calibration value for the left image is 0 and the calibration value for the right image is -4. This shows that there is no need to correct the set distance for the left image, and by adjusting the set distance for the right image toward the closest end by a calibration value of -4, the in-focus area of the right image can be perceived as being the same as the in-focus area of the left image.

[0243] When the calibration is completed, the user instructs the computer 2600 to end the calibration by touching the keyboard 2703, the pointing device 2704, or the display 2701, for example.

[0244] When the CPU 2705 detects this instruction, it stores the calibration value set at that time as a setting value for the application, for example, in the HDD 2708. Furthermore, when updating the parallax image pair 2602, the CPU 2705 reflects the calibration value in the distance setting value. Note that the calibration value can be applied not only when playing back the image data used for the setting, but also to other image data captured by the same device that captured the image data. Furthermore, when a calibration value is set for one frame of video data, the calibration can also be applied to other frames. The set calibration value may be applied automatically, or may be applied when the user instructs it to be applied.

[0245] According to this embodiment, it is possible to provide a function for calibrating the focal distance for a pair of parallax images recorded in a refocusable format.

[0246] (Variation) An indicator showing the hyperfocal distance may be added to the calibration guide 2601. This allows the user to easily obtain a deep-focus image. The hyperfocal distance can be calculated from the focal length and F-number of the lens unit, which are recorded together with the image data as information at the time of shooting, and the allowable circle of confusion diameter, which is, for example, the pixel pitch of the image sensor.

[0247] (Other embodiments) In the above-described embodiment, the display positions of the marks or indicators relating to the right and left images may be reversed. Furthermore, the form of the GUI (Graphical User Interface) constituting the various indicators and guides is not limited to that shown in the drawings. Any form of GUI that can present the same information as the indicators and guides exemplified in the embodiment to the user may be used.

[0248] Furthermore, the various controls described above as being performed by the system control unit 218 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0249] Furthermore, although the present invention has been described in detail based on exemplary embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0250] Furthermore, in the above-described embodiment, the present invention has been described with reference to a digital camera (image capture device), but the present invention is not limited to this example and can be applied to any display control device that can display focus-related display items. In other words, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, printers equipped with displays, digital photo frames, music players, game consoles, e-book readers, and the like.

[0251] The present invention is not limited to application to the imaging device itself, but can also be applied to a control device that communicates with an imaging device (including a network camera) via wired or wireless communication and remotely controls the imaging device. Examples of devices that remotely control an imaging device include smartphones, tablet PCs, and desktop PCs. The imaging device can be remotely controlled by issuing commands from the control device to the imaging device to perform various operations and settings based on operations performed on the control device or processing performed on the control device. Furthermore, live view images captured by the imaging device can be received via wired or wireless communication and displayed on the control device.

[0252] Examples of embodiments disclosed herein are listed below. [Embodiment 1] an image sensor having a plurality of focus detection pixels that receive light beams passing through different pupil partial regions of an imaging optical system; a focus detection unit that acquires a defocus amount of the imaging optical system based on a pair of signals from the focus detection pixels; a focus detection adjustment value used when acquiring the defocus amount; In an imaging device having an imaging device characterized in that the imaging optical system has a plurality of optical axes, the focus detection adjustment values correspond to the plurality of optical axes, optical information for each optical axis of the imaging optical system is converted based on the position of each optical axis on the imaging device, and the focus detection adjustment values for the imaging optical system are calculated from the converted optical information. [Embodiment 2] An imaging device described in embodiment 1, characterized in that the focus detection adjustment value includes at least one of a conversion coefficient for calculating the defocus amount or a correction value used in a correction means for suppressing the difference in intensity of a pair of focus detection signals. [Embodiment 3] The imaging device described in embodiment 1 or 2 is characterized in that it has focus detection adjustment values when an imaging optical system having one optical axis at the center of the imaging element is attached, and focus detection adjustment values when an imaging optical system having multiple optical axes is attached at a position other than the center of the imaging element. [Embodiment 4] An imaging device described in embodiment 3, characterized in that the focus detection adjustment value to be used is switched based on information that determines whether the imaging optical system is an imaging optical system having one optical axis at the center of the image sensor, or an imaging optical system having multiple optical axes at positions other than the center of the image sensor. [Embodiment 5] An imaging device as described in embodiment 4, characterized in that the information is stored in the imaging optical system, and when attached to the imaging device, the imaging device acquires the information via communication. [Embodiment 6] An imaging device as described in embodiment 5, characterized in that the information includes the optical axis position on the imaging element. [Embodiment 7] an image sensor having a plurality of focus detection pixels that receive light beams passing through different pupil partial regions of an imaging optical system; a focus detection unit that acquires a defocus amount of the imaging optical system based on a pair of signals from the focus detection pixels; a calculation means for calculating a correction value for correcting the defocus amount; In an imaging device having an imaging device characterized in that the correction value is made up of coefficients centered on an optical axis of the imaging optical system, the imaging optical system having a plurality of optical axes, and the calculation means calculates the correction value by using the coordinates of the optical axis position on an imaging element of each optical axis and the coefficients. [Embodiment 8] 8. The imaging device according to claim 7, wherein the correction value is a correction value for correcting the focus detection result to a more suitable in-focus position for imaging. [Embodiment 9] An imaging device as described in embodiment 7 or 8, characterized in that the coordinates of the optical axis position are stored in the imaging optical system, and when attached to the imaging device, the imaging device acquires information about the optical axis position via communication. [Embodiment 10] An electronic device that displays a captured image in live view, an acquisition means for acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same direction as the first optical system and having parallax with respect to the first live view image; a display control means for controlling the display of a display item indicating information related to focus by superimposing the first live view image and the second live view image, so that the first display item and the second display item are displayed at the same image height; An electronic device comprising: [Embodiment 11] An electronic device that displays a captured image in live view, a first live view image captured through a first optical system; an acquisition unit that acquires a second live view image that is captured through a second optical system that is oriented in the same direction as the first optical system and has parallax with respect to the first live view image; a display control means for controlling the display of a third display item indicating a difference in information relating to focus between the first live view image and the second live view image so as to be superimposed on the live view image; An electronic device comprising: [Embodiment 12] 12. The electronic device according to claim 10 or 11, wherein the focus information is information about the focus of a focus detection area based on a display position. [Embodiment 13] The electronic device described in embodiment 10, characterized in that the display control means moves the position of a display item indicating information regarding focus, which is displayed superimposed on the first live view image and the second live view image, in tandem. [Embodiment 14] An electronic device described in embodiment 10 or 13, characterized in that the display control means displays the display positions of the first display item and the second display item at the same image height based on the respective positions of the optical axis position of the first optical system and the optical axis position of the second optical system. [Embodiment 15] The electronic device described in embodiment 10, characterized in that the display control means selectively displays a display item indicating information regarding focus in either the first live view image and the second live view image, only the first live view image, or only the second live view image. [Embodiment 16] The electronic device described in embodiment 10 or 11, characterized in that the display control means controls when a lens having a first focus ring for adjusting the focus of the first live view image and a second focus ring for adjusting the focus of the second live view image is attached. [Embodiment 17] An electronic device as described in embodiment 16, characterized in that the display color of the first display item is the same as the color of the first focus ring, and the display color of the second display item is the same as the color of the second focus ring, thereby making the display colors of the first display item and the second display item different. [Embodiment 18] The electronic device described in embodiment 10 or 11, characterized in that the display control means controls when a lens is attached that has a third focus ring that adjusts the focus of the first live view image and the second live view image simultaneously and a fourth focus ring that adjusts the focus of either the first live view image or the second live view image. [Embodiment 19] An electronic device as described in embodiment 18, characterized in that the display color of display items whose focus can be adjusted using only the third focus ring is different from the display color of display items whose focus can be adjusted using the third and fourth focus rings. [Embodiment 20] 12. The electronic device according to claim 10 or 11, wherein the display control means displays the first display item and the second display item when displaying the third display item. [Embodiment 21] The electronic device described in embodiment 10 or 11, characterized in that the display control means displays the third display item showing the difference between the information regarding the first focus displayed in the first display item and the information regarding the second focus displayed in the second display item as differential information based on the information regarding the first focus or the information regarding the second focus. [Embodiment 22] An electronic device as described in embodiment 10 or 11, characterized in that the display control means displays the third display item only when an adjustment mode is performed to adjust the difference between information regarding the first focus displayed on the first display item and information regarding the second focus displayed on the second display item. [Embodiment 23] an acquiring step of acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same orientation as the first optical system and having parallax with respect to the first live view image; a display control step of superimposing a display item indicating information related to focus on the first live view image and the second live view image, and controlling the first display item and the second display item to be displayed at the same image height; 1. A method for controlling an electronic device, comprising: [Embodiment 24] an acquiring step of acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same orientation as the first optical system and having parallax with respect to the first live view image; a display control step of controlling the display of a third display item indicating a difference in information relating to focus between the first live view image and the second live view image so as to be superimposed on the live view image; 1. A method for controlling an electronic device, comprising: [Embodiment 25] A program for causing a computer to function as each means of an electronic device described in any one of embodiments 10 to 22. [Embodiment 26] A computer-readable storage medium storing a program for causing a computer to function as each means of an electronic device described in any one of embodiments 10 to 22. [Embodiment 27] An electronic device that displays a captured image in live view, an acquisition means for acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same direction as the first optical system and having parallax with respect to the first live view image; a first display item showing information about a focus of the first live view image; and a second display item showing information about a focus of the second live view image. a display control means for controlling the display so that the image is superimposed on the live view display; An electronic device comprising: [Embodiment 28] An electronic device described in embodiment 27, characterized in that the display control means displays a plurality of the first display items and the second display items superimposed on the live view display. [Embodiment 29] An electronic device as described in embodiment 27, characterized in that the first display item and the second display item are displayed at positions based on the optical axis positions of the first optical system and the second optical system. [Embodiment 30] An electronic device as described in embodiment 27 or 29, characterized in that when the display position of the first or second display item in the live view display is changed, the first or second display item also moves according to the optical axis positions of the first optical system and the second optical system. [Embodiment 31] an acquisition means for acquiring the first defocus amount detected by phase difference detection using a pair of light beams that have passed through different exit pupils of the first optical system including a first focus ring, and the second defocus amount detected by phase difference detection using a pair of light beams that have passed through different exit pupils of the second optical system including a second focus ring; an adjusting unit that automatically adjusts the drive amount of the first focus ring or the second focus ring based on the difference between the first defocus amount and the second defocus amount; An imaging device comprising: [Embodiment 32] An imaging device that displays a captured image in live view, an acquisition means for acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same direction as the first optical system and having parallax with respect to the first live view image; an adjustment means for adjusting the focus states of the first and second live view images in the same manner; a storage means for storing the calibration value set by the adjustment means, An imaging device, characterized in that the calibration value can be applied at any timing. [Embodiment 33] A display device that displays a pair of parallax images including a first image and a second image, an adjusting means for adjusting the focus states of the first image and the second image in the same manner; a storage means for storing the calibration value set by the adjustment means; and A display device that enables a focus state to be changed by applying the calibration value to an image. [Embodiment 34] A display device according to embodiment 33, characterized in that the focus state is changed by the adjustment means by selecting from a plurality of images with different focus states. [Embodiment 35] 35. The display device according to claim 34, wherein the change of the focus state by the adjustment means is performed by a refocusing process. [Embodiment 36] A display device described in any one of embodiments 33 to 35, characterized in that when the image is a video, the calibration value is set in one frame and applied to other frames.

[0253] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0254] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0255] 100... camera body, 108... display unit, 200... single lens unit, 211... imaging unit, 214... image processing unit, 218... system control unit, 300... twin lens unit

Claims

1. an image sensor capable of generating a signal pair used for focus detection using a phase difference detection method; a calculation means for calculating a focal length of the attached lens unit based on a defocus amount obtained using the signal pair; an adjustment unit for adjusting the focal length of the lens unit based on the focal length, When the lens unit is a multi-lens unit having a plurality of imaging optical systems with different optical axes, the calculation means calculates the focusing distance using an adjustment value obtained based on an optical axis position, which is a position on the image sensor through which the optical axis of the multi-lens unit passes. An imaging device characterized by:

2. 2. The imaging device according to claim 1, wherein the adjustment value is a shading correction value used to make the intensities of the signal pairs uniform.

3. 3. The imaging apparatus according to claim 1, wherein the adjustment value is a conversion coefficient for converting the amount of deviation of the signal pair into a defocus amount.

4. 4. The image pickup apparatus according to claim 3, wherein the conversion coefficient is based on the position of the optical axis of the plurality of imaging optical systems that is closest to a focus detection area.

5. 5. The imaging apparatus according to claim 1, wherein the adjustment value is a correction value for correcting a focus distance based on the defocus amount.

6. 6. The imaging device according to claim 1, wherein, when the lens unit has one optical axis, the calculation means calculates the focal length using the adjustment value obtained assuming that the optical axis passes through the center of the imaging element.

7. 7. The imaging device according to claim 1, further comprising a determination unit that determines whether the lens unit is the multi-lens unit.

8. 8. The imaging device according to claim 1, further comprising an acquisition unit that acquires the optical axis position from the multi-lens unit.

9. 8. The imaging device according to claim 1, further comprising a storage means for storing the optical axis position of the multi-lens unit.

10. 9. The imaging device according to claim 1, wherein the multi-lens unit is a lens unit in which the plurality of imaging optical systems are provided within a single lens barrel.

11. 11. The imaging device according to claim 1, wherein at least some of the pixels of the imaging element have a plurality of photodiodes that share a microlens provided in the pixel, and the signal pair is generated from signals of the pixels having the plurality of photodiodes.

12. 11. The imaging device according to claim 1, wherein the imaging element has a plurality of pixels, and at least some of the plurality of pixels are dedicated pixels for generating the signal pairs.

13. A method executed by an imaging device having an image sensor capable of generating a signal pair used for focus detection using a phase difference detection method, comprising: a calculation step of calculating a focal length of the attached lens unit based on the defocus amount obtained using the signal pair; an adjusting step of adjusting the focal length of the lens unit based on the focal length, When the lens unit is a multi-lens unit having a plurality of imaging optical systems with different optical axes, the calculation step calculates the focal length using an adjustment value obtained based on an optical axis position, which is a position on the image sensor through which the optical axis of the multi-lens unit passes. A method characterized by:

14. A program for causing a computer included in an imaging device having an image sensor capable of generating a signal pair used for focus detection using a phase difference detection method to function as each of the means included in the imaging device described in any one of claims 1 to 8.

Images