Imaging apparatus and method

Abstract

To provide an imaging apparatus capable of focus detection of an imaging surface phase difference detection system even when a lens unit having a plurality of optical axes is mounted thereon.SOLUTION: An imaging apparatus has: an image pick-up device that can generate a signal pair used for focus detection of a phase difference detection system; calculation means that calculates the focal length of a lens unit mounted on the apparatus on the basis of a defocus amount obtained by using the signal pair; and adjustment means that adjusts the focal length of the lens unit on the basis of the focal length. When the lens unit is a multiple-lens unit having a plurality of image forming optical systems with different optical axes, the calculation means calculates the focal length by using an adjustment value obtained on the basis of an optical axis position that is a position on the image pick-up device through which the optical axes of the multiple-lens unit pass.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to an imaging device and method, and particularly to an imaging device and method capable of using a lens unit having a plurality of optical axes.

Background Art

[0002] Conventionally, a stereo camera equipped with a plurality of imaging optical systems and capable of capturing a stereo image with one image sensor has been known (Patent Document 1). On the other hand, in recent years, due to the reduction in price of VR goggles and the like, a more convenient method for capturing a stereo image has been demanded.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, by incorporating two imaging optical systems into one lens barrel and preparing it as an interchangeable lens unit, it is conceivable to capture a stereo image with a general lens interchangeable imaging device.

[0005] However, currently, focus detection using the imaging surface phase difference detection method, which is mainly used in mirrorless cameras, assumes that the lens unit has one optical axis. Therefore, when a lens unit having a plurality of optical axes, such as a lens unit in which two imaging optical systems are incorporated into one lens barrel, is mounted, the accuracy of focus detection may decrease.

[0006] One of the objects of the present invention is to provide an imaging device and method capable of performing accurate focus detection using the imaging surface phase difference detection method even when a lens unit having a plurality of optical axes is mounted.

Means for Solving the Problems

[0007] The above objective is achieved by an imaging device comprising: an image sensor capable of generating a signal pair used for phase-difference detection of focus; a calculation means for calculating the focal distance of a mounted lens unit based on the amount of defocus obtained using the signal pair; and an adjustment means for adjusting the focal distance of the lens unit based on the focal distance, wherein, in the case of a multi-lens unit having multiple imaging optical systems with different optical axes, the calculation means calculates the focal distance using an adjustment value obtained based on the optical axis position, which is the position on the image sensor through which the optical axis of the multi-lens unit passes. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an imaging device and method that can perform accurate focus detection using an imaging plane phase difference detection method even when a lens unit having multiple optical axes is attached. [Brief explanation of the drawing]

[0009] [Figure 1] A perspective view showing an example of the external appearance of a camera 100, which is an example of an imaging device according to the embodiment. [Figure 2] Block diagram showing an example of the functional configuration of a camera system. [Figure 3] Block diagram showing another example of the camera system's functional configuration. [Figure 4] A diagram showing an example of the pixel arrangement of the image sensor of camera 100. [Figure 5] Schematic diagram showing the relationship between the amount of defocus and image shift due to a pair of focus detection signals. [Figure 6] Flowchart relating to focus detection processing in the first embodiment [Figure 7] Schematic cross-sectional view of an optical system having an image sensor and multiple optical axes. [Figure 8] This figure shows an example of the change in the light intensity of the focus detection signal when an optical system with multiple optical axes is installed. [Figure 9] This figure shows an example of displaying a live view image in the second embodiment. [Figure 10]Figure showing an example of the display form of the focus guide [Figure 11] Flowchart related to focus guide display processing [Figure 12] Flowchart related to the live view display operation in the second embodiment [Figure 13] Figure showing an example of the display form of an index presenting the difference in focus degrees between left and right images in the third embodiment [Figure 14] Flowchart related to the display control processing of the index in the third embodiment [Figure 15] Flowchart related to the live view display operation in the third embodiment [Figure 16] Figure showing an example of the display of a live view image in the third embodiment [Figure 17] Figure showing an example of the display form of an index presenting the difference in focus degrees between left and right images in the fourth embodiment [Figure 18] Figure showing an example of the display of a live view image in the fourth embodiment [Figure 19] Flowchart related to the deviation adjustment processing in the fifth embodiment [Figure 20] Figure showing an example of the display form of the index of the fourth embodiment corresponding to the sixth embodiment [Figure 21] Figure showing an example of the display form of the calibration guide in the sixth embodiment [Figure 22] Figure for explaining the calibration method in the sixth embodiment [Figure 23] Figure related to the XR goggles used in the seventh embodiment [Figure 24] Figure showing an example of a recorded image in the seventh embodiment [Figure 25] Figure showing an example of an operation for changing the in-focus subject of a recorded image in the seventh embodiment [Figure 26] Block diagram showing an example of the configuration of a computer capable of implementing the seventh embodiment [Figure 27] Figure for explaining the calibration method in the seventh embodiment

Modes for Carrying Out the Invention

[0010] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0011] In the following embodiments, the present invention will be described in relation to its implementation using an interchangeable-lens digital camera. However, the present invention can also be implemented in any electronic device that may have a camera equipped with an image plane phase-difference detection focus detection function. Such electronic devices include the following: general imaging devices (video cameras, surveillance cameras, etc.), computer equipment (personal computers, tablets, media players, PDAs, etc.), communication equipment (mobile phones, smartphones, IoT devices, etc.), game consoles, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented in other electronic devices.

[0012] ●(First Embodiment) [Overall structure] Figure 1 is a perspective view showing an example of the external appearance of the main body 100 (hereinafter referred to as camera 100) of a lens-interchangeable mirrorless digital camera as an example of an imaging device according to the first embodiment of the present invention. Figure 1(a) is a perspective view of camera 100 from the front diagonal upward direction, and Figure 1(b) is a perspective view of camera 100 from the rear diagonal upward direction.

[0013] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface. The shutter button 101 is an operation for preparing to shoot or giving a shooting command. The power switch 102 is an operation for switching the power of camera 100 on and off. The mode selector switch 103 is an operation for switching between various modes. The main electronic dial 104 is a rotary operation for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation for moving the selection frame (cursor) and advancing images. The video button 106 is an operation for giving a command to start and stop video recording. The external viewfinder display 107 displays various settings such as shutter speed and aperture.

[0014] The camera 100 has a display unit 108, a touch panel 109, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back. The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108.

[0015] The directional keys 110 are an operation unit consisting of keys that can be pressed in the up, down, left, and right directions (4-way keys). Operations can be performed according to the position where the directional keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed when confirming a selection item. The AE lock button 112 is an operation unit that is pressed when fixing the exposure state in shooting standby mode. The zoom button 113 is an operation unit that switches the zoom mode on and off in the live view display (LV display) of the shooting mode. When the zoom mode is on, the live view image (LV image) can be enlarged or reduced by operating the main electronic dial 104. The zoom button 113 is also used in playback mode to enlarge the playback image or increase the magnification ratio.

[0016] The playback button 114 is an operation unit for switching between shooting mode and playback mode. Pressing the playback button 114 in shooting mode switches to playback mode, and the latest image recorded on the recording medium 228 (described later) can be displayed on the display unit 108. The menu button 115 is an operation unit that is pressed to display a menu screen on the display unit 108 that allows for various settings. The user can make various settings of the camera 100 by operating the menu screen displayed on the display unit 108 using the directional keys 110 and the SET button 111. Alternatively, the menu screen may be operated using the touch panel 109 instead of, or in combination with, the buttons.

[0017] The eyepiece section 116 is a window for looking through the eyepiece viewfinder (a type of viewfinder) 117. The user can view the image displayed on the internal EVF (Electronic View Finder) 217, which will be described later, through the eyepiece section 116. The eyepiece detection unit 118 is a sensor that detects whether or not an object is close to the eyepiece section 116.

[0018] The touch bar 119 is a line-shaped touch operation area (line touch sensor) capable of accepting touch operations. The touch bar 119 is positioned so that it can be touched by the right thumb when the grip section 120 is held with the right hand (with the little finger, ring finger, and middle finger) so that the shutter button 101 can be pressed with the right index finger. In other words, the touch bar 119 can be operated when looking through the eyepiece viewfinder 117 through the eyepiece section 116 and holding the camera ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (an operation where the touch position is touched and then released within a predetermined period without moving the touch position), left and right slide operations (an operation where the touch position is moved while touching), etc. The touch bar 119 is a different operation area from the touch panel 109 and does not have a display function. In this embodiment, the touch bar 119 functions as a multifunction bar (M-Fn bar).

[0019] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. When the camera 100 is held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand.

[0020] The thumb rest section 121 (thumb waiting position) is a grip section located on the back of the camera 100, in a place where the thumb of the right hand holding the grip section 120 can be easily rested when no controls are being operated. The thumb rest section 121 is made of rubber material or the like to enhance the holding force (grip). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 228 and the slot for storing the recording medium 228, which will be described later, by closing the slot. The communication terminal 124 is a terminal for the camera 100 to communicate with the lens unit 200, which will be described later and can be attached to and detached.

[0021] <Internal configuration of Camera 100> Figure 2 is a block diagram showing an example of the internal configuration (functional configuration) of a camera system in which a replaceable lens unit 200 is attached to a camera 100. In Figure 2, the components shown in Figure 1 are denoted by the same reference numerals as in Figure 1. Explanations of components already described in Figure 1 will be omitted as appropriate.

[0022] First, let me explain the lens unit 200. The lens unit 200 is an example of a detachable interchangeable lens for the camera 100. The lens unit 200 is a typical single-lens reflex lens (a lens with one optical axis). The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, etc.

[0023] The aperture 201 is configured to have an adjustable aperture diameter. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the focus lens included in the lens 202 to adjust the distance at which the lens unit 200 focuses.

[0024] The lens system control circuit 205 includes, for example, a CPU, ROM, and RAM. The CPU executes a program stored in the ROM, thereby controlling the operation of each part of the lens unit 200. The lens unit 200 and the camera 100 are electrically connected via communication terminals 206 and 124, and the lens system control circuit 205 and the system control unit 218 of the camera 100 can communicate with each other. Based on instructions from the system control unit 218, the lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, and the like.

[0025] Next, I will explain camera 100. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 218.

[0026] The shutter 210 is a focal-plane shutter that operates based on instructions from the system control unit 218 and controls the exposure time of the imaging unit 211. The imaging unit 211 is an image sensor composed of a CCD or CMOS element that converts an optical image into an electrical signal. In this embodiment, the imaging unit 211 is an image sensor that supports focus detection using the image plane phase difference detection method. Specifically, the imaging unit 211 is capable of outputting a focus detection signal pair to realize focus detection using the phase difference detection method.

[0027] The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal (image data). The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing, and color conversion) on the data input through the A / D converter 212 or the memory control unit 213. The image processing unit 214 also performs predetermined calculations using the captured image data to calculate evaluation values ​​used for AF and AE. Based on the obtained calculation results, the system control unit 218 performs exposure control and focus detection control. The image processing unit 214 also calculates the defocus amount based on the focus detection signal pair obtained from the imaging unit 211 as one of the evaluation values. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data and performs AWB (auto white balance) processing on the image data based on the obtained calculation results.

[0028] Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data output by the A / D converter 212 and image data generated by the image processing unit 214. The image data generated by the image processing unit 214 includes display image data for display on the display unit 108 and EVF 217, and recording image data for recording on the recording medium 228. The memory 215 has sufficient storage capacity to store a predetermined number of still image data, a predetermined amount of moving image data, and audio data. In addition, a portion of the memory 215 is used as video memory for the display unit 108.

[0029] The D / A converter 216 converts the image data stored in the memory 215 into an analog signal suitable for display on the display unit 108 or EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or EVF 217 via the D / A converter 216. The display unit 108 or EVF 217 displays according to the analog signal from the D / A converter 216. The display unit 108 or EVF 217 is, for example, a display such as an LCD or an organic EL.

[0030] While the imaging unit 211 is shooting video, the image data stored in the memory 215 via the A / D converter 212 is converted into an analog signal by the D / A converter 216 and sequentially transferred to the display unit 108 and EVF 217 for display. This enables live view display on the display unit 108 and EVF 217.

[0031] The system control unit 218 is a control unit consisting of at least one processor (CPU) and / or at least one circuit. That is, the system control unit 218 may be a processor (CPU), a circuit, or a combination of a processor and a circuit. For example, if the system control unit 218 has a processor (CPU), the system control unit 218 controls the entire camera 100 by reading a program stored in the non-volatile memory 220 into the system memory 219 and executing it with the processor. The system control unit 218 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0032] The camera 100 also includes a system memory 219, a non-volatile memory 220, a system timer 221, a communication unit 222, a posture detection unit 223, and an eyepiece detection unit 118. System memory 219 may be RAM, for example. System memory 219 stores constants and variables for the operation of the system control unit 218, programs read from non-volatile memory 220, and the like. The non-volatile memory 220 may be, for example, an electrically erasable and recordable EEPROM. Constants for the operation of the system control unit 218, programs, etc., are stored in the non-volatile memory 220.

[0033] The system timer 221 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 222 transmits and receives image signals and audio signals to and from external devices connected wirelessly or via wired cable. The communication unit 222 can communicate with external devices compliant with wireless LAN (Local Area Network) and devices on the internet. In addition, the communication unit 222 can communicate with external devices compliant with Bluetooth®. The communication unit 222 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 228, and can receive image data and other various information from external devices.

[0034] The attitude detection unit 223 outputs a signal representing the attitude of the camera 100 relative to the direction of gravity. Based on the signal output by the attitude detection unit 223, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 218 can add orientation information corresponding to the signal output by the attitude detection unit 223 to the image file of the image captured by the imaging unit 211, or rotate the image before recording. The attitude detection unit 223 can use, for example, an acceleration sensor or a gyroscope. The system control unit 218 can also detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.) based on the output signal of the attitude detection unit 223.

[0035] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece section 116 of the eyepiece viewfinder 117, which incorporates the EVF 217. The eyepiece detection unit 118 can use, for example, an infrared proximity sensor. When an object approaches, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The presence or absence of an object approaching the eyepiece section 116 can be determined by the amount of infrared light received.

[0036] The system control unit 218 switches the display (on / off) state of the display unit 108 and EVF 217 depending on whether or not a nearby object is detected by the eyepiece detection unit 118. Specifically, at least in the shooting standby state and when the display destination switching setting is set to automatic switching, if no nearby object is detected, the display unit 108 is turned on and the EVF 217 is turned off. If a nearby object is detected, the EVF 217 is turned on and the display unit 108 is turned off. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as an eyepiece may be used.

[0037] The camera 100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 224, a power control unit 225, a power supply unit 226, a recording medium interface 227, an operation unit 229, and the like. The external viewfinder display unit 107 displays various settings of the camera 100, such as shutter speed and aperture, via the external viewfinder display drive circuit 224. The power control unit 225 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are powered, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 225 also controls the DC-DC converter based on the detection results and instructions from the system control unit 218, supplying the necessary voltage to each part, including the recording medium 228, for the required period. The power supply unit 226 includes primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li batteries, and an AC adapter. The recording medium I / F 227 is an interface with the recording medium 228, such as a memory card or hard disk. The recording medium 228 is a memory card or the like for recording captured images, and consists of semiconductor memory or a magnetic disk. The recording medium 228 may be detachable or built-in.

[0038] The operation unit 229 is an input unit that receives user input (user operation) and is used to input various instructions to the system control unit 218. The operation unit 229 includes a shutter button 101, a power switch 102, a mode selector switch 103, a touch panel 109, and other operating components 230. The other operating components 230 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, a touch bar 119, and the like.

[0039] The shutter button 101 has a first shutter switch 231 and a second shutter switch 232. The first shutter switch 231 turns on during operation of the shutter button 101, so-called half-press, and generates a first shutter switch signal SW1. The system control unit 218 interprets the first shutter switch signal SW1 as a shooting preparation instruction and starts the shooting preparation process. The shooting preparation process includes AF processing, AE processing, AWB processing, flash pre-flash processing, etc.

[0040] The second shutter switch 232 turns on when the shutter button 101 is fully pressed, generating the second shutter switch signal SW2. The system control unit 218 interprets the second shutter switch signal SW2 as a still image capture instruction and starts the still image capture operation based on the exposure conditions determined by the AE processing. It then controls each unit to execute a series of shooting processes, from reading the signal from the imaging unit 211 to generating an image file containing the still image data obtained from the capture and writing it to the recording medium 228.

[0041] The mode switch 103 switches the operating mode of the system control unit 218 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. Modes included in the still image shooting mode include auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide shooting settings for different shooting scenes. The user can directly switch to any of the above shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to any of the displayed modes using the operation unit 229. Similarly, the video shooting mode may also include multiple modes.

[0042] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 can be mounted on top of the display surface of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. GUI stands for Graphical User Interface. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109. Either method is acceptable.

[0043] The system control unit 218 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). The finger or pen that was touching the touch panel 109 is lifted (released), meaning the touch action ends (hereinafter referred to as "Touch-Up"). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0044] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0045] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 218. Based on the notified information, the system control unit 218 determines what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation was performed. An operation in which a finger is touched on the touch panel 109 and then quickly moved a certain distance and released is called a flick. In other words, a flick is an operation in which the finger is quickly traced across the touch panel 109 as if flicking it. If a touch move of a predetermined distance or more at a predetermined speed or faster is detected, and a touch-up is then detected, it is determined that a flick was performed (it can be determined that a flick followed a slide operation). Furthermore, touching multiple points (for example, two points) simultaneously (multitouch) to bring them closer together is called pinch-in, and touching them further apart is called pinch-out. Pinch-out and pinch-in are collectively referred to as pinch operations (or simply pinch).

[0046] <Configuration of the multi-lens unit> Figure 3 is a schematic diagram showing an example configuration of a two-lens unit 300 as an example of a multi-lens unit. In this specification, "multi-lens" refers to a lens unit configured in which multiple imaging optical systems are provided within a single lens mount (or lens barrel), and which has multiple optical axes. Figure 3 shows the two-lens unit 300 mounted on the camera 100. Note that Figure 3 shows only a part of the configuration of the camera 100 shown in Figure 2.

[0047] The twin-lens unit 300 is a type of interchangeable lens that can be attached to the camera 100. The twin-lens unit 300 has two imaging optical systems 301L and 301R within a single lens barrel, and therefore has two optical axes.

[0048] Here, it is assumed that when the twin-lens unit 300 is mounted on the camera 100, the two imaging optical systems 301L and 301R are arranged so that the two optical axes are aligned on a horizontal line. The two imaging optical systems 301L and 301R have a field of view of approximately 180 degrees and can capture the area of ​​the front hemisphere. Specifically, the two imaging optical systems 301L and 301R can capture a field of view of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation / depression angle, pitch angle). The two imaging optical systems 301L and 301R are A pair of parallax images with parallax between the left and right eyes are formed on the imaging surface of the imaging unit 211. In the following description, the imaging optical system 301L will be referred to as the left eye optical system 301L, and the imaging optical system 301R will be referred to as the right eye optical system 301R.

[0049] The right eye optical system 301R and the left eye optical system 301L each have multiple lenses and a reflective mirror, etc. The multiple lenses include at least a focusing lens for adjusting the focusing distance. The two-lens unit 300 also has a lens system control circuit 303. The right eye optical system 301R is an example of a first optical system, and the left eye optical system 301L is an example of a second optical system. In the right eye optical system 301R and the left eye optical system 301L, the lenses 302R and 302L located on the subject side are oriented in the same direction, and their optical axes are approximately parallel.

[0050] Although not shown in Figure 3, the two-lens unit 300 has a similar configuration to the AF drive circuit 204. In this case, it may have one or more AF drive circuits: one that drives the focus lenses of the right eye optical system 301R and the left eye optical system 301L in conjunction, and another that drives at least one of the focus lenses of the right eye optical system 301R and the left eye optical system 301L independently. The drive of the focus lenses is performed by the lens system control circuit 303 based on the control of the system control unit 218 (adjustment means).

[0051] Furthermore, the twin-lens unit 300 includes an encoder that detects the amount and direction of rotation of the focus ring provided on the lens barrel. The lens system control circuit 303 provides a so-called by-wire manual focus function by controlling the AF drive circuit in accordance with the focus lens operation detected by the encoder. In this case, the twin-lens unit 300 may also have a switch that allows the user to switch between the focus lenses driven by the focus ring operation.

[0052] The two-lens unit 300 is a VR180 lens for capturing images in the VR180 format, a VR image format that enables binocular stereoscopic viewing, with the camera 100. The VR180 lens has a right-eye optical system 301R and a left-eye optical system 301L, each having a fisheye lens with a field of view of approximately 180 degrees. The right-eye optical system 301R and the left-eye optical system 301L only need to be able to acquire images that enable binocular VR display as VR180, and the field of view may be around 160 degrees. The VR180 lens can form the right image (first image) by the right-eye optical system 301R and the left image (second image) by the left-eye optical system 301L on the same imaging surface. Here, the imaging unit 211 of the camera 100 has one image sensor, and the two-lens unit 300 forms the right and left images on the imaging surface of that one image sensor. However, the camera 100 may have two image sensors arranged in parallel, and the twin-lens unit 300 may form a right image on the imaging surface of one image sensor and a left image on the imaging surface of the other image sensor.

[0053] The two-lens unit 300 includes a focus ring for adjusting the focus of the right eye optical system 301R and a focus ring for adjusting the focus of the left eye optical system 301L. Alternatively, it may include a focus ring for simultaneously adjusting the focus of both the right eye optical system 301R and the left eye optical system 301L, and a focus ring for adjusting the focus of either the right eye optical system 301R or the left eye optical system 301L. By operating these focus rings, the user can manually adjust the focusing distance of the right eye optical system 301R and the left eye optical system 301L. These focus rings may be provided individually, or in the case of a by-wire system, this may be achieved by switching the function of a single focus ring.

[0054] The twin-lens unit 300 is attached to the camera 100 via a mount, similar to the (single-lens) lens unit 200. The mount consists of a lens mount section 304 and a camera mount section 305. When the twin-lens unit 300 is attached to the camera 100, the communication terminal 124 of the camera 100 and the communication terminal 306 of the twin-lens unit 300 are electrically connected. This enables the system control section 218 of the camera 100 and the lens system control circuit 303 of the twin-lens unit 300 to communicate with each other.

[0055] In this embodiment, the right image and the left image are formed on the imaging surface of the imaging unit 211, spaced apart in the left-right direction. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single image sensor. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. In this way, by attaching the dual-lens unit 300, a pair of disparity images (right image and left image) formed by the right-eye optical system 301R and the left-eye optical system 301L can be acquired in a single shot. Furthermore, by displaying the acquired right and left images as images for the right eye and left eye in VR, the user can observe a three-dimensional VR image, a so-called VR180 image, with a range of approximately 180 degrees.

[0056] Here, a VR image is an image that can be displayed in VR, as described later. VR images include omnidirectional images (spherical images) taken with an omnidirectional camera (spherical camera), and panoramic images that have a wider image range (effective image range) than the display range that can be displayed on the display unit at once. Furthermore, VR images may be either still images or videos. Videos may be pre-recorded videos or live images (images acquired from the camera in near real-time).

[0057] VR images have an image range (effective image range) of up to 360 degrees horizontally and vertically. VR images also include images with a wider field of view than that of a normal camera, or an image range wider than that that can be displayed on a display unit at once, even if the field of view is less than 360 degrees horizontally or vertically. Images captured by camera 100 using the aforementioned twin-lens unit 300 are a type of VR image. VR images can be displayed in VR, for example, by setting the display mode of a display device (a display device capable of displaying VR images) to "VR View". By displaying a VR image with a 360-degree field of view in VR and changing the orientation of the display device horizontally (horizontal rotation direction), the user can view a seamless, omnidirectional image in the horizontal direction.

[0058] Here, VR display (VR view) is a display mode that displays images of a predetermined range of the field of view captured in the VR image, according to the orientation of the display device. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by performing a transformation (a transformation that corrects distortion) that maps the VR image to a virtual sphere. VR display also includes "two-eye VR display (two-eye VR view)," which displays the VR image for the left eye and the VR image for the right eye side-by-side in the left and right regions by performing transformations that map them to virtual spheres.

[0059] Stereoscopic viewing is possible by using "two-eye VR display" with VR images for the left eye and VR images for the right eye that have parallax between them. In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), the image displayed will correspond to the user's field of view. For example, suppose a VR image is displayed with a field of view centered at 0 degrees horizontally (a specific direction, e.g., north) and 90 degrees vertically (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is then reversed (for example, changing the display surface from facing south to facing north), the display range of the same VR image will change to an image with a field of view centered at 180 degrees horizontally (the opposite direction, e.g., south) and 90 degrees vertically. In other words, if a user wearing an HMD turns their face from north to south (i.e., turns their back), the image displayed on the HMD will also change from a north image to a south image.

[0060] It should be noted that the VR image captured using the twin-lens unit 300 of this embodiment is a VR180 format image capturing a range of approximately 180 degrees in front, and there is no image of a range of approximately 180 degrees behind. If such a VR180 format image is displayed in VR and the orientation of the display device is changed to the side where there is no image, a blank area will be displayed, for example.

[0061] By displaying VR images in VR in this way, users can visually experience the sensation of being inside the VR image (in VR space). Note that the method of displaying VR images is not limited to changing the orientation of the display device. For example, the display range may be moved (scrolled) in response to user operations via a touch panel or directional buttons. Furthermore, during VR display (in the "VR View" display mode), in addition to changing the display range due to changes in orientation, the display range may also be changed in response to touch movements on the touch panel, drag operations with a mouse, or pressing directional buttons. Note that a configuration in which a display device such as a smartphone is attached to VR goggles (head-mounted adapter) is a type of HMD (Head-Mounted Display).

[0062] <Configuration of the imaging unit 211 (image sensor)> Figure 4 is a schematic diagram showing an example of the pixel arrangement of the imaging unit 211 (image sensor) in this embodiment. The image sensor constituting the imaging unit 211 in this embodiment is capable of generating signal pairs used for phase-difference detection of focus. Figure 4 shows the pixel arrangement of the image sensor (2D CMOS sensor) in a 4x4 range for imaging pixels (8x4 range for the focus detection pixel arrangement). Hereafter, when simply referred to as "pixel," it means imaging pixel.

[0063] The imaging unit 211 is provided with a primary color Bayer array color filter. The pixel group 400 represents a 2x2 pixel arrangement, which is the repeating unit of the color filter. The pixel group 400 includes pixels 400R having spectral sensitivity for red (R), pixels 400Gr and 400Gb having spectral sensitivity for green (G), and pixels 400B having spectral sensitivity for blue (B). In addition, one microlens 401 is provided for each pixel.

[0064] To enable phase-difference focusing on the imaging plane, the imaging unit 211 is provided with two photodiodes (photoelectric conversion units) 402 and 403, which share a microlens 401, for each of the multiple pixels arranged two-dimensionally on the imaging unit 211. The first photodiode 402 and the second photodiode 403 each function as a sub-pixel or a focus detection pixel. In other words, one pixel functions as two focus detection pixels. Furthermore, by treating the first photodiode 402 and the second photodiode 403 together as a single photodiode, one pixel functions as one imaging pixel. Hereinafter, the signal obtained by the first photodiode 402 will be called the A signal, the signal obtained by the second photodiode 403 will be called the B signal, and the signal obtained by adding the A signal and the B signal obtained from the same pixel will be called the A+B signal. The A signal and the B signal will be called the focus detection signals, and the A+B signal will be called the imaging signal. Note that the A signal (B signal) may also be obtained by subtracting the B signal (A signal) from the A+B signal.

[0065] In this embodiment, each pixel is provided with two photodiodes that share a microlens 401, but the number of photodiodes provided in each pixel may be three or more. Alternatively, a dedicated pixel capable of outputting only A or B signals may be provided for focus detection. There are no restrictions on the configuration of pixels provided in the image sensor, as long as they can output signals capable of realizing phase-difference detection focus detection. Furthermore, in this embodiment, all pixels have multiple photodiodes, but a configuration where only some pixels have multiple photodiodes may also be used.

[0066] <Relationship between defocus amount and image displacement amount> Using Figure 5, we will explain the relationship between the amount of defocus and the amount of image shift obtained from the A and B signals that can be acquired by the image sensor shown in Figure 4. The amount of defocus is calculated using a pair of image signals consisting of an A image signal obtained by concatenating multiple A signals and a B image signal obtained by concatenating multiple B signals. Both the A image signal and the B image signal are also called focus detection signals. Here, we assume that the imaging center (the center of the pixel area used for imaging in the image sensor) and the optical axis center coincide.

[0067] Figure 5 schematically shows the relationship between the amount of defocus d and the amount of image shift between a pair of focus detection signals (image A signal and image B signal). 1300 is the imaging plane of the image sensor. The exit pupil of the imaging optical system is divided into a first pupil region 1303 and a second pupil region 1304 by the first photodiode 402 and the second photodiode 403, which share a single microlens.

[0068] The amount of defocus d, whose magnitude |d| represents the distance from the imaging position of the subject image to the image sensor 1300, is defined as follows: in a front-focus state where the imaging position of the subject image is closer to the subject than the image sensor 1300, the defocus value is negative (d<0), and in a back-focus state, it is positive (d>0). In a focused state where the imaging position of the subject image is on the image sensor 1300 (i.e., the in-focus position), d=0. For example, subject 1301 is in focus because it is imaged at the in-focus state (d=0). Also, subject 1302 is in a front-focus state because its imaging position is closer to the subject than the image sensor 1300 (d<0). Hereafter, the front-focus state (d<0) and the back-focus state (d>0) are collectively referred to as a defocus state (|d|>0).

[0069] In the front-focused state (d<0), the light beam received from the subject 1302 that passes through the first pupil region 1303 (or second pupil region 1304) is focused and then spreads out with a width Γ1 (or Γ2) centered on the centroid position G1 (or G2) of the light beam. In this case, the image of the subject 1302 on the imaging surface 1300 becomes blurred. The blurred image is received by the first photodiode 402 (or second photodiode 403) provided in each pixel arranged in the image sensor, and an A signal (or B signal) is generated.

[0070] Therefore, the pair of focus detection signals (A image signal and B image signal) are stored in memory as image data of a (blurred) subject image with width Γ1 (or Γ2) at the centroid position G1 (or G2) on the imaging surface 1300. The width Γ1 (or Γ2) of the subject image increases roughly proportionally with the increase in the magnitude of the defocus amount d |d|. Similarly, if the image shift amount between the first focus detection signal and the second focus detection signal is "p", then the magnitude of the image shift amount |p| increases with the increase in the magnitude of the defocus amount d |d|.

[0071] For example, as shown in Figure 5, the image displacement p can be defined as the difference in the centroid positions of the light beam, "G1-G2," and its magnitude |p| increases roughly in proportion to the increase in the magnitude of the defocus amount |d|. Note that in the back-focused state (d>0), the direction of image displacement between the pair of focus detection signals (A image signal and B image signal) is opposite to that in the front-focused state, but the magnitude of the image displacement |p| is proportional to the magnitude of the defocus amount |d|.

[0072] Therefore, phase-detection autofocus (AF) can be achieved by detecting the image shift amount p between a pair of focus detection signals (A image signal and B image signal) and converting the image shift amount p into a defocus amount using a conversion factor K. The image shift amount p between a pair of focus detection signals (A image signal and B image signal) can be determined by relatively shifting the A image signal and the B image signal and calculating the correlation amount, and then selecting the shift amount that yields a good correlation (signal agreement). The conversion factor K has a value that depends on the incident angle, F-number, and optical axis position of the imaging optical system. Therefore, a conversion factor K appropriate to the lens unit should be used.

[0073] The conversion coefficient K is stored, for example, in the non-volatile memory of the lens system control circuit 205 of the lens unit, and the system control unit 218 can retrieve it from the mounted lens unit. Of course, the conversion coefficient K may also be obtained by other means, such as storing the conversion coefficient K in the non-volatile memory 220 in association with the identification information of the lens unit, and retrieving the conversion coefficient K from the non-volatile memory 220 based on the identification information of the mounted lens unit.

[0074] A pair of focus detection signals are typically generated based on the signals of pixels within the focus detection area. Therefore, if a focus detection area is set for each imaging optical system, a defocus amount is calculated for each individual focus detection area. If the focus lenses of the imaging optical system can be driven individually, the focusing distance can be adjusted for each imaging optical system. Alternatively, a single defocus amount calculated for each focus detection area may be used to adjust the focusing distance of multiple imaging optical systems. This single defocus amount may be, for example, an average value or a representative value.

[0075] <Defocus Amount Calculation Process> The process for calculating the defocus amount will be explained using the flowchart shown in Figure 6. In this embodiment, the image processing unit 214 generates a first focus detection signal (A image signal) by concatenating the A signals obtained by the first photodiodes 402 of multiple pixels in the image sensor. It also generates a second focus detection signal (B image signal) by concatenating the B signals obtained by the second photodiodes 403 of each pixel used to generate the A image signal.

[0076] In S1401, the image processing unit 214 (calculation means) acquires an A signal and a B signal from each of the multiple pixels included in the region of the image sensor corresponding to the focus detection region. As described above, the A signal (or B signal) may be acquired by subtracting the B signal (or A signal) from the A + B signal.

[0077] In S1402, the image processing unit 214 adds the A signals of same-colored pixels that are in the same position horizontally (row direction) vertically (column direction) to reduce the amount of data for the A and B image signals. This compresses the signal to two rows. Furthermore, the image processing unit 214 generates a luminance signal Y by adding the green (Gr), red (R), blue (B), and green (Gb) signals to the A signals added in the column direction. These multiple luminance signals Y arranged in the row direction are used as the A image signal. The image processing unit 214 applies a similar addition process to the B signal to generate the B image signal. By adding the signals, the Nyquist frequency in the addition direction becomes 1 / n of the non-addition frequency, where n is the number of pixels added.

[0078] In S1403, the image processing unit 214 applies shading correction processing (optical correction processing) to the A image signal and the B image signal to equalize the signal intensity (suppress the difference in signal intensity). The shading correction value has a value that depends on the incident angle, F-number, and optical axis position of the imaging optical system. Similar to the conversion coefficient K, the shading correction value can also be obtained from the lens unit or non-volatile memory 220.

[0079] In S1404, the image processing unit 214 applies a spatial bandpass filter with a specific passband to the A image signal and the B image signal in order to improve the correlation (signal agreement) between the A image signal and the B image signal and to improve the focus detection accuracy. Examples of bandpass filters include difference filters such as {1, 4, 4, 4, 0, -4, -4, -4, -1} that cut the DC component and extract edges, and additive filters such as {1, 2, 1} that suppress high-frequency noise components.

[0080] In S1405, the image processing unit 214 calculates the correlation between the A image signal and the B image signal after the filtering process is applied. The correlation is calculated for each shift amount while changing the relative shift amount of the A image signal and the B image signal in the pupil division direction.

[0081] Let W (>2) be the number of signals that make up the A and B image signals after bandpass filtering, and let A(k) and B(k) be the k-th (1≦k≦W) signals. If the shift amount is s and the range of the shift amount s is Γ, the correlation amount COR is calculated by equation (1). COR(s)=Σ(k∈W)|A(k)-B(ks)|, s∈Γ (1)

[0082] The absolute difference between the k-th A image signal A(k) and the ks-th B image signal B(ks) is accumulated over k in the range of signal number W, and the correlation amount COR(s) with respect to the shift amount s is calculated. The shift amount is, for example, one pixel. If there are multiple A and B image signals in the vertical direction, the correlation amounts calculated for each pair of A and B image signals for the same shift amount may be added together.

[0083] In S1406, the image processing unit 214 calculates a shift amount in units of less than one pixel that minimizes the correlation amount COR(s) calculated for each pixel-level shift. The image processing unit 214 then uses the calculated shift amount as the image displacement amount p between the A image signal and the B image signal. Furthermore, the image processing unit 214 applies the above-mentioned conversion coefficient K to the image displacement amount p (for example, by multiplication) to convert it into a defocus amount d. Through the above processing, the defocus amount d is calculated.

[0084] Thus, the correction values ​​used in signal correction performed during the calculation of the defocus amount d, and the conversion coefficient K that converts the image displacement amount p to the defocus amount d, have values ​​that depend on the characteristics of the lens unit. Typically, the focus detection adjustment value is calculated assuming a lens unit with one optical axis that passes through the center of the image sensor. Here, the statement that the optical axis passes through the center of the image sensor represents a design or ideal state, and deviations due to manufacturing errors may occur. For example, vignetting of incident light due to the lens frame differs between cases with one optical axis and cases with multiple optical axes, even if the aperture of the lens frame is the same. Therefore, even if the focus detection adjustment value is calculated from the lens information for a lens unit with two optical axes in the same way as for a lens unit with one optical axis, it may not result in an appropriate focus detection adjustment value, and as a result, the accuracy of the defocus amount may decrease.

[0085] In this embodiment, even when an imaging optical system having multiple optical axes is mounted on a main body having one image sensor, the reduction in the accuracy of the defocus amount is suppressed by using appropriate focus detection adjustment values. In this embodiment, as an example of focus detection adjustment values ​​for a lens unit having multiple optical axes, • Shading correction coefficient for equalizing the intensity of a pair of focus detection signals • Conversion coefficient K to convert the image displacement amount p to the defocus amount d. • Best focus correction value that corrects the focus distance based on the amount of defocus. This will be explained. However, the concept of this embodiment can be applied similarly to other focus detection adjustment values. Focus detection adjustment values ​​are basically used for each focus detection region, that is, for each imaging optical system.

[0086] <Shading Correction Factor> Figure 7 shows a schematic cross-sectional view of the optical system when a camera 100 having one image sensor is fitted with a twin-lens unit 300 having two optical axes. In this embodiment, the twin-lens unit 300 has two imaging optical systems, and the optical axes of each imaging optical system are on a straight line in the horizontal direction (parallel to the long side of the image sensor) passing through the center of the image sensor (imaging center) and at a position equidistant from the imaging center. Therefore, Figure 7 shows a horizontal cross-section including the center of the image sensor and the two optical axes.

[0087] The diameter of the image circle of each imaging optical system is assumed to be approximately half the length of the long side of the effective pixel area of ​​the image sensor. Figure 8(a) shows an example of the signal intensity of the A image signal and B image signal obtained with the twin-lens unit 300 attached. As shown in Figure 8, since there are optical axes on both the left and right sides of the center of the image sensor, the signal intensity shows a discontinuous change at the center of the image sensor. When an imaging optical system with one optical axis passing through the center of the image sensor is attached, the intensity changes of the A image signal and B image signal are continuous. Therefore, the discontinuous intensity changes of the A image signal and B image signal are characteristic of an imaging optical system with multiple optical axes attached to a single image sensor.

[0088] Figure 8(b) shows an example of shading correction values ​​for matching the intensity of the A image signal and the B image signal, which have the intensity changes shown in Figure 8(a). In order to correct the discontinuous changes in signal intensity of the A image signal and the B image signal, the shading correction values ​​also change discontinuously at the boundary between the two image circles.

[0089] In this embodiment, the shading correction value for the two-lens unit 300 having two optical axes is pre-stored in the camera 100, for example, in its non-volatile memory 220, similar to the shading correction value for the lens unit 200 having one optical axis. The system control unit 218 selects the correction value to be used for shading correction according to the number of optical axes of the mounted lens unit. The shading correction value may be pre-calculated for each lens unit model and stored in the non-volatile memory 220 in association with the lens unit's identification information. Alternatively, a formula for calculating the correction value according to the number of optical axes may be stored in the non-volatile memory 220, and the information necessary for calculating the shading correction value (for example, from the mounted lens unit) may be obtained and applied to the calculation formula to calculate the shading correction value.

[0090] When calculating shading correction values, the number of optical axes and the positions on the image sensor through which the optical axes pass (hereinafter referred to as optical axis positions) are necessary. Optical information with the optical axis center as the origin must be treated with the optical axis positions as the origin. Therefore, for lens units where the optical axes do not pass through the center of the image sensor, such as the twin-lens unit 300 with multiple optical axes, the optical axis positions are important for calculating shading correction values.

[0091] In this embodiment, the optical axis position (Lx,Ly) is obtained, for example, from the twin-lens unit 300, and the optical information of the imaging optical system, which is expressed with the optical axis as the origin, is converted into coordinate system information of the image sensor. For example, as shown in Table 1, suppose that the optical information of one imaging optical system is that the angle of incidence of light at a position 10 mm away from the optical axis is 5 degrees (relative to the optical axis), and the optical axis position (Lx,Ly) = (5 mm, 0 mm) is obtained.

[0092] The optical axis position (5mm, 0mm) indicates that the optical axis passes through the position x=5mm, y=0mm in a Cartesian coordinate system with the center of the image sensor as the origin. Therefore, from the optical information, it can be seen that the angle of incidence of light at a point on the image sensor 10mm away from the optical axis (for example, (15mm, 0mm) or (-5mm, 0mm)) is 5 degrees.

[0093] In this way, by obtaining the optical axis position (Lx,Ly) expressed in the coordinate system of the image sensor, optical information expressed with the optical axis as the origin can be converted into optical information in the coordinate system of the image sensor. Then, by using the optical information in the coordinate system of the image sensor, an appropriate shading correction value can be calculated. [Table 1]

[0094] The conversion coefficients for converting the amount of image displacement into the amount of defocus are also pre-stored in the camera 100's non-volatile memory 220, similar to the shading correction values, for the two-lens unit 300 having two optical axes and the lens unit 200 having one optical axis. For lens units with multiple optical axes, using the conversion coefficient related to the optical axis position closest to the coordinate (or focus detection area) to which the correction is applied is advantageous in terms of conversion accuracy.

[0095] <Best focus correction value> Images captured by focusing the imaging optical system to a focal distance based on the defocus amount obtained by image plane phase-difference focus detection may not match the image that humans perceive as being in optimal focus. This is thought to be partly due to the fact that the spatial frequency band used to calculate the defocus amount does not match the spatial frequency band observed by the human eye. Therefore, it is known to correct the focal distance detected in the imaging device to a focal distance that yields an image perceived as being in optimal focus by humans. The correction value used for this focal distance correction is called the best focus correction value.

[0096] Furthermore, the best focus correction value may have a value that corresponds to the distance from the optical axis position. For example, the best focus correction value may be expressed by the following formula depending on the relative coordinates (x,y) with the optical axis position as the origin. Best focus correction value = a00 + a10*x + a01*y + a11*xy Here, a00, a10, a01, and a11 are coefficients. When the optical axis position coincides with the center of the image sensor, the best focus correction value can be calculated at any relative coordinate (x,y) on the image sensor using the above formula. The best focus correction value is determined for the focus detection region, and the detected focus distance for that focus detection region is corrected by the best focus correction value. By adjusting the focus distance of the imaging optical system based on the corrected focus distance, an image that humans perceive as being in optimal focus can be obtained.

[0097] However, if the optical axis position does not coincide with the center of the image sensor, the above formula cannot express the best focus correction value. In this embodiment, by modifying the above formula based on the optical axis position (Lx,Ly) described above, it becomes possible to calculate the best focus correction value at any relative coordinate (x,y) with the center of the image sensor as the origin, even for imaging optical systems where the optical axis position does not coincide with the center of the image sensor.

[0098] Specifically, by modifying the above equation using the optical axis position (Lx,Ly) as follows, it becomes possible to calculate the best focus correction value for an imaging optical system with optical axis position (Lx,Ly) at any relative coordinate (x,y) with the center of the image sensor as the origin. Best focus correction value = a00 + a10(x-Lx) + a01(y-Ly) + a11(x-Lx)(y-Ly)

[0099] By correcting the best focus correction value, which is expressed as a function using coordinates with the optical axis as the origin, based on the optical axis position, an appropriate correction value can be calculated for lens units with imaging optical systems where the optical axis position differs from the center of the image sensor, such as lens units with multiple optical axes. The optical axis position (Lx,Ly), expressed in the coordinate system of the image sensor, can be obtained from the lens unit via communication. Alternatively, the optical axis position (Lx,Ly), which is pre-stored in the imaging device in association with the identification information of the lens unit, can be obtained by referencing it based on the identification information of the mounted lens unit. For lens units with multiple optical axes, accurate correction is possible by using the best focus correction value corrected using the optical axis position closest to the coordinate (x,y) to be corrected.

[0100] In this embodiment, the shading correction value, conversion coefficient, and best focus correction value were described as focus detection adjustment values ​​assuming that the optical axis position is at the center of the image sensor. However, the technical concept of this embodiment is to correct the focus detection adjustment value, which assumes that the optical axis position is at the center of the image sensor, based on the optical axis position information, and is applicable to any focus detection adjustment value assuming that the optical axis position is at the center of the image sensor.

[0101] Furthermore, for lens units with multiple imaging optical systems with different optical axes, it is generally necessary to acquire or store optical information (such as optical axis position and incident angle) for each imaging optical system. However, for imaging optical systems where optical information other than the optical axis position is common, the memory capacity of the lens unit and the imaging device itself can be reduced by acquiring or storing optical information other than the optical axis position for only one imaging optical system.

[0102] As described above, according to this embodiment, when a lens unit having multiple optical axes is attached to an imaging device having one image sensor, the correction value used in the defocus amount calculation process is calculated using the positional information of each optical axis passing through the image sensor. Therefore, even for correction values ​​that cannot be obtained using a calculation method that assumes a general lens unit having one optical axis passing through the center of the image sensor, an appropriate value can be obtained. As a result, even when a lens unit having multiple optical axes is attached to an imaging device having one image sensor, accurate image plane phase difference detection type focus detection can be realized.

[0103] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. The first embodiment demonstrated that accurate focus detection using an image plane phase difference detection method can be achieved even when a lens unit having multiple optical axes is mounted on an imaging device having one image sensor. This embodiment relates to a configuration (focus guide function) that supports manual focus operation when a lens unit having multiple optical axes is mounted on an imaging device having one image sensor.

[0104] The focus guide function is a feature that displays to the user the position of the focus detection area and the degree of focus within that area. For example, by overlaying a GUI such as an indicator or mark representing the position and degree of focus of the focus detection area onto the live view display, the position and degree of focus of the focus detection area can be displayed to the user.

[0105] When a standard lens unit 200, which is assumed to have one optical axis passing through the center of the image sensor, is attached, only one image is formed on the image sensor. Therefore, the focus guide function only needs to be provided for one image. However, when a lens unit with multiple optical axes is attached, multiple images are formed on the image sensor. For example, when a twin-lens unit 300 is attached to the camera 100, two images are formed on the image sensor (imaging unit 211).

[0106] In this case, if a focus guide function is provided for only one image, it becomes difficult to accurately manually focus on images that do not have a focus guide. Furthermore, it becomes impossible to grasp the degree of focus of each individual image collectively. Therefore, it is necessary to provide a focus guide function suitable for cases where multiple images are formed on a single image sensor.

[0107] An example of the focus guide function provided in this embodiment is shown in Figure 9. Figure 9 shows that the focus guide function is provided to the right image 800R and the left image 800L of the live view image captured using the right eye optical system 301R and the left eye optical system 301L, respectively. Specifically, indices 801R and 801L, which indicate information regarding the position of the focus detection area and the degree of focus, are superimposed on the right image 800R and the left image 800L, respectively. Indices 801R and 801L are displayed at positions having the same image height. Figure 9 shows an example in which indices 801R and 801L are displayed at the same relative coordinates with the optical axis position of each imaging optical system as the origin.

[0108] In this embodiment, the focus guide function is provided by the image processing unit 214 in accordance with the control of the system control unit 218. Specifically, in parallel with the live view display processing, the image processing unit 214 calculates the amount of defocus of the imaging optical system for the focus detection area notified by the system control unit 218 and outputs it to the system control unit 218 along with its reliability. Then, it generates an indicator image based on the display format instructed by the system control unit 218 and writes the indicator image to the address area of ​​the video memory area in the system memory 219 that corresponds to the focus detection area. As a result, the indicator is superimposed on the live view image, and information regarding the position and degree of focus of the focus detection area can be provided to the user. Note that calculating the amount of defocus for the focus detection area means calculating the amount of defocus based on the A image signal and B image signal obtained from the pixels within the focus detection area.

[0109] Next, we will explain specific examples of the indicators provided by the focus guide function using Figures 10(A) to 10(D). The indicator includes a frame-shaped first indicator 500 displayed on the outer periphery of the focus detection area, and a third indicator 502 displayed at a position tangent to a virtual circle 510 that encompasses the focus detection area and has a common center with the focus detection area. The indicator also includes a second indicator 501 displayed at a position tangent to a virtual circle 511 whose radius is larger than that of the virtual circle 510 by the length of the third indicator 502.

[0110] Furthermore, the first indicator 500 indicates the position and size of the focus detection area, and the display mode indicates whether the focus detection area is in focus or out of focus. The second indicator 501 and the third indicator 502 indicate the degree of focus in the focus detection area based on their display format and positional relationship. Specifically, in addition to the in-focus and out-of-focus states, the out-of-focus state is distinguished as either focused on the subject at close range (front focus) or focused on the subject at infinity (back focus), and the amount of deviation from the in-focus state is also indicated.

[0111] Figure 10(A) shows an example of the display form of the indicators when the focus detection area is in focus. In the focused state, the first indicator 500 is displayed as a continuous frame. The second indicator 501 is a downward-pointing wedge shape, and the third indicator 502 is an upward-pointing wedge shape. The second indicator 501 and the third indicator 502 are displayed opposite each other on a vertical line passing through the center of the first indicator 500, with their tips touching. In the focused state, the second indicator 501 and the third indicator 502 have a display form with their interiors filled in. The display forms of the first to third indicators 500 to 502 may differ in attributes other than shape and whether they are filled in, such as color, brightness, and whether they blink, as long as the focused and out-of-focus states can be visually distinguished. For example, the second indicator 501 and the third indicator 502 can be displayed in green when in focus and in white when out of focus.

[0112] Figures 10(B) and 10(C) show examples of how to display the index when the reliability of the defocus amount is high in an out-of-focus state. Figure 10(B) shows an example of the display format for the front-focus indicator. In the out-of-focus state, the first indicator 500 is displayed as a frame with a break. The display position of the second indicator 501 remains the same as in the in-focus state, but the display format is different. Here, an example is shown where it is filled in in the in-focus state, but hollow in the out-of-focus state. On the other hand, the third indicator 502 consists of two indicators 502A and 502B, which are displayed at positions to the left (right) of the display position in the in-focus state by a distance corresponding to the amount of defocus. The downward-facing second indicator 501 indicates that it is in focus closer to the subject, and its display position is the same as in the in-focus state, making it easier to grasp the amount of defocus indicated by indicator 502A (or indicator 502B).

[0113] Figure 10(C) shows an example of the display format for the back-focused indicator. Because the subject is out of focus, the first indicator 500 is displayed as a frame with a gap. The display position of the third indicator 502 is the same as in the in-focus state, but the display format is different. Here, an example is shown where it is filled in in the in-focus state, but hollow in the out-of-focus state. On the other hand, the second indicator 501 becomes two indicators 501A and 501B, which are displayed to the left and right of the display position in the in-focus state by a distance corresponding to the amount of defocus. The third indicator 502 is pointing upwards, indicating that the subject is in focus at a distance from it, and by being in the same display position as in the in-focus state, it makes it easier to grasp the amount of defocus indicated by indicator 501A (or indicator 501B).

[0114] In this example, we have assumed that there are two third indicators 502 when the pin is ahead, but it is not necessary to increase the number. You can simply display either indicator 502A or indicator 502B in Figure 10(B) (i.e., change the display format and position of the third indicator 502). Similarly, when the pin is behind, you can display either indicator 501A or indicator 501B in Figure 10(C) (i.e., change the display format and position of the second indicator 502).

[0115] Figure 10(D) shows an example of how the indicator is displayed when the amount of defocus is large and the reliability of the focus detection result is low (for example, when the image is very blurry). In this case, neither the front / back focus status (direction of defocus) nor the magnitude of the defocus amount is displayed. The indicator is displayed in a way that informs the user that sufficient focus detection is not possible to provide the focus guide function.

[0116] Here, the first indicator 500 is used to display the out-of-focus state, while the second indicator 501 and the third indicator 502 are used to display the in-focus state and the out-of-focus state, which can display the defocus direction and amount of defocus, in a different way. Specifically, the shape of the second indicator 501 and the third indicator 502 is changed from a wedge shape to a rod or line of constant thickness, and they are displayed in a different color (e.g., gray) than the other in-focus states. In addition, the display positions of the second indicator 501 and the third indicator 502 are set to predetermined fixed positions. In the example shown in Figure 10(D), the third indicator 502 is divided into two indicators 502A and 502B, similar to the front-focus state, but it is not necessarily required to increase the number of indicators. The reliability of the amount of defocus can be calculated by the image processing unit 214 using any known method. For example, if the maximum value of the correlation is less than the threshold, the reliability of the amount of defocus may be considered low.

[0117] In this embodiment, the focus guide function is provided based on the amount and direction of defocus obtained using the image plane phase-difference focus detection configuration. However, the basic technical concept of this embodiment does not depend on the method for obtaining the degree of focus in the focus detection area. Therefore, the focus guide function may be provided for each imaging optical system based on other evaluation values ​​that depend on the degree of focus of the image formed by each imaging optical system, such as contrast evaluation values.

[0118] <Focus guide display control processing> Next, the focus guide display control process executed by the system control unit 218 will be explained using the flowchart shown in Figure 11. This process is achieved by the system control unit 218 executing a program stored in the non-volatile memory 220 after it has been loaded into the system memory 219. The focus guide display control process is executed in parallel with the live view display process.

[0119] The focus guide display control process described here does not depend on the number of optical axes (number of imaging optical systems) of the mounted lens unit. The focus guide display control process should be executed for each image formed by the imaging optical system.

[0120] In S601, the system control unit 218 notifies the image processing unit 214 of the position and size of the focus detection region. There are no particular restrictions on how the position and size of the focus detection region are determined when a two-lens unit 300 with multiple optical axes is installed. As with the case when a lens unit 200 with one optical axis is installed, it may be specified by the user, set based on a feature area such as a face, or set to a predetermined position and size. However, it is assumed that the same position and size of focus detection region is set for the image formed by each imaging optical system. The image processing unit 214 calculates the defocus amount for the focus detection region notified by the system control unit 218 as described in the first embodiment.

[0121] In S602, the system control unit 218 obtains the amount of defocus of the imaging optical system in the focus detection region and its reliability from the image processing unit 214. The reliability may be, for example, a correlation quantity corresponding to the amount of defocus.

[0122] In S604, the system control unit 218 determines whether the reliability of the defocus amount obtained in S602 is high. For example, if the obtained reliability is above a predetermined threshold, the system control unit 218 can determine that the reliability of the defocus amount is high.

[0123] The system control unit 218 executes S605 if it determines that the defocus amount is highly reliable, and S610 if it does not. In S610, the system control unit 218 decides to display the indicator in a highly blurred state (fourth display mode) and executes S611. The fourth display mode is the display mode shown in Figure 10(D).

[0124] In S605, the system control unit 218 determines whether the focus detection area is in focus or out of focus. The system control unit 218 can determine that a focus detection area where the absolute value of the defocus amount is less than or equal to a threshold is in focus, and a focus detection area where the absolute value of the defocus amount exceeds the threshold is out of focus. If the system control unit 218 determines that the area is in focus, it executes S607; if it determines that the area is out of focus, it executes S606.

[0125] In S607, the system control unit 218 decides to display the indicator in the focus state display mode (first display mode) and executes S611. The first display mode is the display mode shown in Figure 10(A).

[0126] In S606, the system control unit 218 determines whether the device is in a front-pin or back-pin state. The system control unit 218 can determine whether the device is in a front-pin or back-pin state based on the sign (direction of defocus) of the defocus amount. If the system control unit 218 determines that the device is in a front-pin state, it executes S608; if it determines that the device is in a back-pin state, it executes S609.

[0127] In S608, the system control unit 218 decides to display the indicator in the display mode of the previous pin state (second display mode) and executes S611. The second display mode is the display mode shown in Figure 10(B). In S609, the system control unit 218 decides to display the indicator in a rear-pin state display mode (third display mode) and executes S611. The third display mode is the display mode shown in Figure 10(C).

[0128] In S611, the system control unit 218 determines the display position of the indicator and notifies the image processing unit 214 of the display mode. The image processing unit 214 generates an image of the indicator according to the display mode according to the notification and writes the image of the indicator to the address of the video memory area corresponding to the display position. As a result, the indicator is superimposed on the live view image and displayed on the EVF 217 or display unit 108.

[0129] Next, the live view display operation will be explained using the flowchart shown in Figure 12. This process is achieved by the system control unit 218 executing a program stored in the non-volatile memory 220, which is then loaded into the system memory 219. This process can be executed when live view display is required, such as during shooting standby or video recording.

[0130] At S701, the system control unit 218 starts displaying the live view on the EVF 217 or the display unit 108. Specifically, the system control unit 218 controls the image processing unit 214 to generate display images while the image capture unit 211 continues to record video, and to sequentially write these images to the video memory area of ​​the system memory 219. As a result, the live view image is displayed on the EVF 217 or the display unit 108. Processing from S702 onward is performed in parallel with the live view display.

[0131] In S702, the system control unit 218 confirms the type (number of optical axes) of the installed lens unit. The system control unit 218 communicates with the lens system control circuit 205 or 303 via communication terminal 124 and communication terminal 206 or 306 to confirm the lens type information. The lens type information includes information that can identify the model of the lens unit and the number of optical axes. Note that the lens type information is included in the lens information transmitted from the lens unit to the camera 100 when the camera 100 is powered on or when the lens unit is replaced. Therefore, in S702, the system control unit may refer to already acquired lens information without communicating with the lens unit.

[0132] The system control unit 218 executes S703 if it determines that a single lens unit with one optical axis is installed, and executes S704 if it determines that multiple (in this case, two) lens units with multiple optical axes are installed.

[0133] In step S703, the system control unit 218 performs focus guide display processing for a lens unit with one optical axis (a normal lens unit). The focus guide display processing may be the focus guide display control processing described using the flowchart in Figure 11. As a result, one focus guide is displayed on the live view image formed by one imaging optical system.

[0134] In steps S704 and S705, the system control unit 218 performs focus guide display processing for each of the images (left image and right image) formed by the two imaging optical systems. Specifically, in step S704, the system control unit 218 performs focus guide display processing for the left image, and in step S705, it performs focus guide display processing for the right image. The focus guide display processing for each image may be the focus guide display control processing described using the flowchart in Figure 11. As a result, indicator 801L is displayed on the left image 800L and indicator 801R is displayed on the right image 800R of the live view image 800 shown in Figure 9. Indicators 801L and 801R are displayed at the same position in the image.

[0135] In S706, the system control unit 218 determines whether or not it has detected an operation to terminate the live view display. If it determines that it has detected an termination operation, the system control unit 218 terminates the live view display; otherwise, it repeats the process from S702 to continue the live view display. Although the process from S702 is repeated here, after processing S702 once, S703 and S706, or S704 to S706, may be repeatedly executed according to the result of the determination in S702 until the lens unit is removed.

[0136] As described above, in this embodiment, a focus guide is displayed for each live view image formed by the imaging optical system, according to the number of optical axes of the mounted lens unit. Therefore, when a two-lens unit 300 having two optical axes (imaging optical systems) is mounted, a focus guide is displayed for both the left and right images of the live view image.

[0137] According to this embodiment, for example, when a two-lens unit 300 for capturing images for VR180 is attached to the camera 100, a focus guide is displayed for each of the two live view images on the screen. Therefore, the degree of focus of each live view image can be easily grasped. Furthermore, when adjusting the focus distance of the imaging optical system, if the focus distances of all imaging optical systems are adjusted in conjunction, the image to be prioritized for focus can be selected. Also, if the focus distance can be adjusted for each imaging optical system, the focus distance can be precisely adjusted for each imaging optical system. In either case, manual focus operation when multiple lenses are attached can be effectively supported.

[0138] In this embodiment, the defocus amount obtained as a result of focus detection is used for displaying the focus guide. However, the defocus amount may also be calculated using the correction value described in the first embodiment. This makes it possible to achieve a focus guide display that shows the focus state with greater accuracy.

[0139] ●(Third embodiment) Next, a third embodiment of the present invention will be described. This embodiment relates to a guide function for manually adjusting the difference in the degree of focus between imaging optical systems.

[0140] Figure 13 shows an example of an index provided by the focus guide function when the twin-lens unit 300 is attached in this embodiment and live view is displayed. In this embodiment, an index is provided that shows the difference in the degree of focus in the focus detection area of ​​the left image and the right image.

[0141] Specifically, an index indicating the difference in the degree of focus is displayed based on the amount of defocus of the left eye optical system 301L and the right eye optical system 301R acquired in the focus guide display processing in S704 and S705 of Figure 12. As shown in Figure 13, the index has an axis 900, an indicator 901 that shows the origin (difference = 0), positive and negative indicators 902 and 903 that show the direction of the difference, and a mark 904 that shows the actual difference in the degree of focus.

[0142] The difference in the degree of focus is expressed as a relative value and direction with respect to the amount of defocus relative to one of the imaging optical systems. For example, when the amount of defocus of the left eye optical system 301L is used as the reference, the difference in the degree of focus can be obtained by the following formula. Difference in degree of focus = Defocus amount of right eye optical system 301R - Defocus amount of left eye optical system 301L The sign of the difference in focus is positive for front focus and negative for back focus. Therefore, the difference in focus is positive if the other image (right image) is front focus compared to the reference image (left image), and negative if it is back focus.

[0143] Figure 13(A) shows an example of the first display form of the indicator when the degree of focus of the left and right images (the amount of defocus of the left eye optical system 301L and the right eye optical system 301R) is determined to be the same (no difference). When the difference in the degree of focus is determined to be 0, the mark 904 is positioned on axis 900 to point to the position indicated by indicator 901 (reference position). In addition, when the difference in the degree of focus is determined to be 0, the color of the indicator and visual effects (such as whether or not it blinks) may be different from other cases. For example, the indicator can be green when the difference in the degree of focus is determined to be 0, and white in all other cases.

[0144] Figures 13(B) and 13(C) show examples of the second and third display modes, respectively. These display modes are used to indicate the magnitude and direction of the difference in the degree of focus based on the difference in the amount of defocus, when the difference in the degree of focus is not zero, but the reliability of the amount of defocus is high.

[0145] Figure 13(B) shows an example of a second display mode for the indicator when the focus distance of the other image (right image) is shifted to the near side (positive direction) relative to the reference image (left image), that is, when the right image is in front focus relative to the left image. In this case, mark 904 is positioned to the right (positive direction) of indicator 901, moving by a distance corresponding to the difference in the degree of focus.

[0146] Figure 13(C) shows an example of a third display mode for the indicator when the focus distance of the other image (right image) is shifted toward infinity (negative direction) relative to the reference image (left image), that is, when the right image is back-focused relative to the left image. In this case, mark 904 is positioned to the left (negative direction) of indicator 901, moving by a distance corresponding to the difference in the degree of focus.

[0147] In the second and third display modes, the position of mark 904 can indicate to the user the magnitude and direction of the difference in the relative degree of focus between the left and right images. Specifically, the magnitude of the difference in degree of focus can be indicated by the distance of mark 904 from the position on axis 900 pointed to by indicator 901, which indicates a reference point where the difference is 0. Furthermore, depending on whether mark 904 is located to the right or left of the position on axis 900 pointed to by indicator 901, it can be indicated whether the focus distance of the other image is shifted towards near focus or infinity relative to the reference image.

[0148] Figure 13(D) shows an example of a fourth display mode for the indicator when the reliability of the amount of defocus for at least one of the imaging optical systems is low, such as when the left image, the right image, or both are significantly blurred. In this case, the reliability of the difference in the degree of focus obtained by the above formula is also low, so the mark 904, which indicates the magnitude and direction of the difference in the degree of focus, is not displayed. In addition, for the axis 900 and indicators 901-903 of the indicator that are always displayed, the fourth display mode may have different colors and visual effects (such as whether or not they blink) compared to the first to third display modes. For example, the display color can be set to gray.

[0149] Note that the display formats shown in Figures 13(A) to 13(D) are merely examples, and other forms of indicators may be used to show the difference in the degree of focus, such as displaying the difference using numerical values ​​or using marks or indicators of different shapes.

[0150] Next, the display control process for the difference in the degree of focus between the left and right images, executed by the system control unit 218, will be explained using the flowchart shown in Figure 14. This process is executed when the twin-lens unit 300 is attached to the camera 100. This process is realized by the system control unit 218 executing a program stored in the non-volatile memory 220 by loading it into the system memory 219. The display control process for the difference in the degree of focus between the left and right images is executed in parallel with the live view display process. It can also be executed in parallel with the display control process for the focus guide described in the second embodiment.

[0151] In S1001, the system control unit 218 obtains the defocus amount and reliability of the right eye optical system 301R and the left eye optical system 301L from the image processing unit 214. If the display control processing of the focus guide is being performed, the system control unit 218 may refer to the defocus amount and reliability obtained in S602 in Figure 11. The system control unit 218 then determines whether the reliability of at least one of the defocus amounts is low. If the system control unit 218 determines that the reliability of at least one of the defocus amounts is low, it executes S1008; otherwise, it executes S1002.

[0152] In S1002, the system control unit 218 calculates the difference in the degree of focus based on one of the left or right images. When the left image is used as the reference, the system control unit 218 can calculate the difference in the degree of focus using the calculation formula described above. After calculating the difference in the degree of focus, the system control unit 218 executes S1003.

[0153] In S1003, the system control unit 218 determines whether the difference in the degree of focus calculated in S1002 is zero. If it determines that the difference is zero, it executes S1005; otherwise, it executes S1004.

[0154] In S1004, the system control unit 218 determines whether the sign of the difference in the degree of focus calculated in S1002 is positive or negative. If the system control unit 218 determines that the sign of the difference is positive, it executes S1006; if it determines that the sign of the difference is negative, it executes S1007.

[0155] In S1005, the system control unit 218 decides to display the indicator in the first display format (Figure 13(A)) and executes S1009. In S1006, the system control unit 218 decides to display the indicator in a second display format (Figure 13(B)) and executes S1009. In S1007, the system control unit 218 decides to display the indicator in a third display format (Figure 13(C)) and executes S1009. In S1008, the system control unit 218 decides to display the indicator in a fourth display format (Figure 13(D)) and executes S1009.

[0156] In S1009, the system control unit 218 notifies the image processing unit 214 of the display mode of the indicator determined in S1005 to S1008 and the display position of the mark 904 corresponding to the difference in the degree of focus. The image processing unit 214 generates an image of the indicator according to the display mode according to the notification and writes the image of the indicator to the address of the video memory area corresponding to the predetermined display position of the indicator. As a result, an indicator showing the difference in the degree of focus between the left and right images is superimposed on the live view image and displayed on the EVF 217 or display unit 108.

[0157] Next, the live view display operation in this embodiment will be explained using the flowchart shown in Figure 15. In Figure 15, operations similar to those in the second embodiment are given the same reference numerals as in Figure 12. This process is achieved by the system control unit 218 executing a program stored in the non-volatile memory 220 by loading it into the system memory 219. This process can be executed when live view display is required, such as in the shooting standby state or during video recording.

[0158] The processing in S701 to S705 is the same as in the second embodiment, so the explanation is omitted. When the focus guide display processing for the right image in S705 is completed, in S1106 the system control unit 218 determines whether or not to display the difference in the degree of focus between the left and right images.

[0159] Whether or not to display the difference in the degree of focus between the left and right images may be determined by the system control unit 218 depending on the operating mode of the camera 100, for example, by one of the user settings. For example, if the video mode of the camera 100 is set to an adjustment mode for the user to adjust the difference in the degree of focus between the imaging optical systems of the twin-lens unit 300, the system control unit 218 will determine to display the difference in the degree of focus between the left and right images. In adjustment mode, the user can adjust the focusing distance between the right eye optical system 301R and the left eye optical system 301L by, for example, operating the individual focus rings, so that there is no difference in the focusing distance between the right eye optical system 301R and the left eye optical system 301L.

[0160] The system control unit 218 executes S1107 if it determines that the difference in the degree of focus between the left and right images should be displayed, and executes S706 if it does not determine that the difference should be displayed. In S1107, the system control unit 218 executes the display control process described using the flowchart in Figure 14. The user can adjust the focus distance of the imaging optical system (in this case, the right eye optical system 301R) that is forming an image other than the reference image, while looking at the displayed indicator, so that there is no difference in the degree of focus.

[0161] Figure 16 shows an example of the live view display in S1107. Similar to the second embodiment, one focus guide 1201L and 1201R for the left image 1200L and the right image 1200R within a single frame of the live view image 1200 obtained by the image sensor constituting the imaging unit 211 are displayed. In this embodiment, an index 1202 indicating the difference in the degree of focus between the left and right images is further displayed between the left image 1200L and the right image 1200R.

[0162] The processing of S706 is the same as in the second embodiment, so its explanation is omitted. In this embodiment, when the twin-lens unit 300 is attached to the camera 100, an index is displayed that shows the magnitude and direction of the difference in the degree of focus between the two imaging optical systems or the image. Therefore, the user can easily grasp the shift in the focus distance of the imaging optical system caused by aging or other factors. The user can adjust the system so that the shift in the focus distance between the imaging optical systems becomes zero while looking at the index.

[0163] In the second and third embodiments, assistance for manual focus operation was described by displaying a focus guide. However, the basic technical concept is to provide some kind of display to assist in shooting for the image formed by each imaging optical system when a multi-lens unit is installed, and the display content is not limited to a focus guide that shows the degree of focus. For example, a peaking pattern indicating overexposed or underexposed areas may be superimposed on the live view image.

[0164] In this embodiment as well, the amount of defocus used for displaying the focus guide may be calculated using the correction value described in the first embodiment. This makes it possible to achieve a focus guide display that shows the focus state with greater accuracy.

[0165] ●(Fourth embodiment) Next, a fourth embodiment of the present invention will be described. This embodiment relates to an index that collectively shows the degree of focus and the difference in the degree of focus for individual images.

[0166] Figures 17(A) to 17(D) show examples of the first to fourth display forms of the indicator according to this embodiment. The indicator comprises an axis 1700, an indicator 1701 positioned near the center of the axis 1700 to indicate the focus position, and indicators 1702 indicating back-focused state and indicators 1703 indicating front-focused state, positioned near both ends of the axis 1700. The indicator further comprises a mark 1704 indicating the degree of focus of the right image and a mark 1705 indicating the degree of focus of the left image, positioned near the top and bottom of the axis 1700 to indicate their position on the axis 1700.

[0167] Figure 17(A) shows an example of the first display mode when both the right and left images are in focus (both the right eye optical system 301R and the left eye optical system 301L have a defocus amount of 0). In this case, since there is no difference in the degree of focus between the right and left images, marks 1704 and 1705 are positioned to point to the same position on axis 1700. Also, since the defocus amount is 0, marks 1704 and 1705 are positioned to point to the position on axis 1700 indicated by indicator 1701. In this case, similar to the first display mode used when in focus in the second embodiment, at least a portion of the indicators may be displayed using different colors or visual effects than the other display modes.

[0168] Figure 17(B) shows an example of a second display mode when there is no difference in the degree of focus between the right and left images, but the image is front-focused. In this case, since there is no difference in the degree of focus between the right and left images, marks 1704 and 1705 are positioned to point to the same position on axis 1700, similar to the first display mode. Also, because the image is front-focused, marks 1704 and 1705 are positioned to point to a position on axis 1700 that is moved to the end of indicator 1703 by a distance corresponding to the amount of defocusing, compared to the position on axis 1701 pointed to by indicator 1701. To bring the left and right images into focus while there is no difference in the degree of focus between the left and right images, a display may be provided prompting the user to drive the focus ring to simultaneously adjust the focusing distance of the right eye optical system 301R and the left eye optical system 301L.

[0169] Figure 17(C) shows an example of a third display mode when there is a difference in the degree of focus between the right and left images, and neither is in focus. Here, both the right and left images are assumed to be in a front-focus state. In this case, mark 1704 is positioned to point to a position on axis 1700 that is moved to the end of indicator 1703 by a distance corresponding to the amount of defocus of the right eye optical system 301R, compared to the position on axis 1701 pointed to by indicator 1701. Similarly, mark 1705 is positioned to point to a position on axis 1700 that is moved to the end of indicator 1703 by a distance corresponding to the amount of defocus of the left eye optical system 301L, compared to the position on axis 1701 pointed to by indicator 1701. In the example shown in Figure 17(C), since the amount of defocus is greater for the left eye optical system 301L, mark 1705 is positioned to point to a position closer to the end of indicator 1703 on axis 1700 than mark 1704.

[0170] The user adjusts the focusing distance of the left eye optical system 301L while looking at the indicators so that marks 1704 and 1705 point to the same position on axis 1700. Then, by simultaneously adjusting the focusing distances of the right eye optical system 301R and the left eye optical system 301L, the left and right images can be brought into focus.

[0171] In this embodiment, the position on axis 1700 indicated by marks 1704 and 1705 can be used to show the user the degree of focus of the right and left images, the difference between them, and whether the focus state is front-focused or back-focused.

[0172] In this embodiment, since marks 1704 and 1705 independently indicate the magnitude and direction of the defocus amount of the right eye optical system 301R and the left eye optical system 301L, it is not necessary to calculate the difference in the degree of focus calculated in the previous embodiment. The system control unit 218 can determine the position on axis 1700 that marks 1704 and 1705 should point to (i.e., the display position of marks 1704 and 1705) based on the defocus amount and sign of the right eye optical system 301R and the left eye optical system 301L.

[0173] Figure 17(D) shows an example of a fourth display mode for the indicators when the reliability of the defocus amount of the right eye optical system 301R and the left eye optical system 301L is low, such as when both the right and left images are significantly blurred. In this case, marks 1704 and 1705 are not displayed because the reliability of the defocus amount is low. In addition, for the axis 1700 and indicators 1701-1703 of the indicators that are always displayed, the fourth display mode may have different colors and visual effects (such as whether or not they blink) compared to the first to third display modes. For example, the display color can be set to gray.

[0174] Although not shown here, if the reliability of the defocus amount of one of the right eye optical system 301R and the left eye optical system 301L is low, and the reliability of the defocus amount of the other is high, mark 1704 or 1705 based on the more reliable defocus amount may be displayed.

[0175] Note that the display formats shown in Figures 17(A) to 17(D) are merely examples. The degree of focus and its difference may also be indicated using other forms of indicators, such as showing the magnitude and direction of the defocus amount of the right eye optical system 301R and the left eye optical system 301L as values, or using marks or indicators of different shapes.

[0176] The system control unit 218 can display the indicators of this embodiment in S1107, for example, in a live view display operation in which the processes S704 and S705 are removed from the flowchart of Figure 15 described in the third embodiment. In this case, in S1107, the system control unit 218 obtains the amount of defocus and its reliability for each imaging optical system from the image processing unit 214. If the reliability is low for all of them, the fourth display mode is determined. If the reliability of the defocus amounts is high for all of them, the system control unit 218 determines the first display mode if the defocus amounts are all 0, the second display mode if the defocus amounts are not all 0 and there is no difference, and the third display mode if there is a difference in the defocus amounts. When the first to third display modes are determined, the system control unit 218 also determines the display positions of marks 1704 and 1705. The system control unit 218 then notifies the image processing unit 214 of the determined display mode and, if marks 1704 and 1705 are to be displayed, their display positions.

[0177] Figure 18 shows an example of live view display in this embodiment. Focus guides 1801L and 1801R are displayed on the left image 1800L and the right image 1800R, respectively, within the live view image 1800 of one frame obtained by the image sensor constituting the imaging unit 211. In this embodiment, an index 1802 indicating the difference in the degree of focus between the left and right images, as well as the magnitude and direction of the amount of defocus for each imaging optical system, is further displayed between the left image 1800L and the right image 1800R.

[0178] In this embodiment, in addition to the effects of the third embodiment, the magnitude and direction of the defocus amount for each imaging optical system are presented in a different way than in the second embodiment, so that users can refer to indicators that are easy for them to understand and obtain the necessary information.

[0179] In this embodiment as well, the basic technical concept is to provide some kind of display to assist in shooting with the images formed by each imaging optical system when a multi-lens unit is installed, and the display content is not limited to the degree of focus or the difference between them. For example, a peaking pattern indicating overexposed or underexposed areas may be superimposed on the live view image.

[0180] ●(Fifth embodiment) Next, a fifth embodiment of the present invention will be described. This embodiment relates to a technique for automatically adjusting the difference in the degree of focus between imaging optical systems when a multi-lens unit is installed.

[0181] In the third embodiment, the difference in the degree of focus between the right eye optical system 301R and the left eye optical system 301L (the difference in the amount of defocus obtained for the right image and the left image, respectively) was acquired. By using this difference to drive the focus lens of the right eye optical system 301R or the left eye optical system 301L, the difference in the degree of focus between the imaging optical systems (focus distance shift) can be automatically adjusted.

[0182] Figure 19 is a flowchart relating to the misalignment adjustment process between imaging optical systems, which is executed by the system control unit 218. This process is realized by the system control unit 218 executing a program stored in the non-volatile memory 220 by loading it into the system memory 219. The misalignment adjustment process may be executed in response to user instructions or automatically at predetermined timings. An example of predetermined timings is when the system control unit 218 determines that the installed lens unit is a multi-lens unit (for example, when the lens unit is replaced or when the camera 100 is started up).

[0183] When the camera 100's operating mode is the adjustment mode described in the third embodiment, the user manually adjusts the misalignment between the imaging optical systems. Therefore, in this embodiment, the misalignment adjustment process automatically performed by the system control unit 218 may be performed only when the operating mode is not the adjustment mode. The misalignment adjustment process is performed in parallel with the live view display operation.

[0184] In S1901, the system control unit 218 sets one of the right-eye optical system 301R and the left-eye optical system 301L of the twin-lens unit 300 as the reference imaging optical system. For example, the twin-lens unit 300 can either drive the right-eye optical system 301R and the left-eye optical system 301L in conjunction, or drive only the left-eye optical system 301L, by operating the focus ring. In this case, the system control unit 218 sets the right-eye optical system 301R, which cannot be driven independently, as the reference.

[0185] Furthermore, the system control unit 218 may set the imaging optical system corresponding to the eye pre-set as the user's dominant eye as the reference imaging optical system. Alternatively, if there is a difference in the subject detection accuracy or focus detection accuracy of the right eye optical system 301R and the left eye optical system 301L, the system control unit 218 may set the imaging optical system with the better accuracy as the reference.

[0186] In S1902, the system control unit 218 retrieves the current focus detection region information for the right eye optical system 301R and the left eye optical system 301L, respectively, which is stored in the system memory 219. If the focus detection regions for the right eye optical system 301R and the left eye optical system 301L are set to the same position relative to the optical axis, it is sufficient to read the information for either one of the focus detection regions. The system control unit 218 notifies the image processing unit 214 of the focus detection region information and instructs it to calculate the amount of defocus. As described above, the image processing unit 214 calculates the amount of defocus and its reliability based on the signal of the focus detection region for both the right and left images.

[0187] In S1903, the system control unit 218 obtains the defocus amount DEF_L of the left eye optical system 301L and the defocus amount DEF_R of the right eye optical system 301R from the image processing unit 214.

[0188] In S1904, the system control unit 218 calculates the difference in the defocus amount of other imaging optical systems with respect to the defocus amount of the imaging optical system set as the reference in S1901. For example, if the right eye optical system 301R is set as the reference in S1901, the system control unit 218 calculates the difference in defocus amounts DEF_dif using the following formula. DEF_dif = DEF_L - DEF_R

[0189] Next, in S1905, the system control unit 218 drives the focus lens of the non-reference imaging optical system (in this case, the left eye optical system 301L) in the optical axis direction by PLS_dif, which is the drive amount and drive direction corresponding to DEF_dif detected in S1904. The lens drive amount PLS_dif is, for example, PLS_dif = DEF_dif / SENS_L To desire more.

[0190] Here, SENS_L is a conversion factor that converts the amount of defocus of the non-reference (i.e., the one being adjusted) imaging optical system (in this case, the left eye optical system 301L) into a lens drive amount, and is stored in the lens unit beforehand. The two-lens unit 300 stores the focus sensitivity for the right eye optical system 301R and the left eye optical system 301L, respectively, in the non-volatile memory inside the lens system control circuit 303. Note that if there are multiple imaging optical systems with the same configuration, as in the two-lens unit 300, one focus sensitivity may be used in common for each imaging optical system.

[0191] Through the above process, the difference in the degree of focus between imaging optical systems (deviation in focus distance) can be automatically adjusted to eliminate it. According to this embodiment, the difference in the degree of focus between imaging optical systems (deviation in focus distance) that was manually adjusted in the third embodiment can be automatically adjusted, thereby reducing the user effort required for adjustment and improving usability.

[0192] ●(Sixth Embodiment) Next, a sixth embodiment of the present invention will be described. In this embodiment, a focus calibration function is provided when a multi-lens unit is installed.

[0193] Figure 20 shows the indicators described using Figure 17 in the fourth embodiment. Figure 20(A) is the third display mode, where marks 1704 and 1705 indicate that the right image is in focus, but the left image is front-focused. From this state, the user adjusts the focus distance of the left eye optical system 301L by operating the focus ring so that mark 1704, which indicates the degree of focus of the left image, is at the position pointed to by indicator 1701 (so that the indicators are in the state shown in Figure 20(B)).

[0194] As a result, for example, both the right image 1800R and the left image 1800L of the live view image 1800 displayed in Figure 18 should be in focus. However, due to manufacturing errors, aging deterioration, and environmental factors, a discrepancy may occur between the calculated defocus amount and the degree of image focus due to manufacturing errors, deterioration over time, and other factors of the components such as lenses and reflective mirrors contained in the right eye optical system 301R and the left eye optical system 301L, respectively.

[0195] Therefore, even if marks 1704 and 1705 indicate that the image is in focus, the user may perceive that the right and / or left images observed through the display unit 108 and EVF 217 are not in focus. In particular, if the degree of focus of one image is lower than that of the other, the user is likely to notice the difference in focus because the two images are displayed adjacent to each other. Therefore, in this embodiment, a function (focus calibration function) is provided to correct the difference between the state in which the user perceives the image as in focus and the state in which the camera 100 determines the image to be in focus.

[0196] In this embodiment, the correction value used to correct the difference between the state in which the user perceives an image as being in focus and the state in which the camera 100 determines that the image is in focus is called the calibration value. The calibration value can be set and maintained independently for the right eye optical system 301R and the left eye optical system 301L.

[0197] Figure 21 shows an example of a calibration guide displayed on the display unit 108 and EVF 217 when the focus calibration function is executed. Here, for the sake of explanation and understanding, it is assumed that the focus lens of the left eye optical system 301L and the focus lens of the right eye optical system 301R can be driven independently. However, the focus calibration function can be realized even if the focus lenses of both imaging optical systems are driven in conjunction, or if one of the imaging optical systems has a configuration in which the focus lens can be driven independently. In this case, the calibration of the imaging optical system whose focus lens cannot be driven independently should be performed first, and then the calibration of the remaining imaging optical system should be performed.

[0198] Figure 21(A) shows an example of the first display configuration of the calibration guide 2110. The calibration guide 2110 has an axis 2100 corresponding to a range of calibration values. The axis 2100 is marked with a scale, and values ​​are indicated on some of the scales. Here, an indicator 2101 indicating a calibration value of 0, an indicator 2102 indicating the maximum negative value (here, -20), and an indicator 2103 indicating the maximum positive value (here, +20) are displayed near the bottom of the axis 2100. The sign of the calibration value is such that the positional relationship between the mark and the reference is the same as in Figure 17, with positive in the front pin direction and negative in the back pin direction. Note that indicators indicating other values ​​may also be added. In the example in Figure 21(A), indicators indicating -10 and +10 are added.

[0199] Furthermore, near axis 2100, mark 2104 indicating the calibration value of the right eye optical system 301R and mark 2105 indicating the calibration value of the left eye optical system 301L are positioned. In addition, marks 1704 and 1705 indicating the degree of focus of the right eye optical system 301R and the left eye optical system 301L are positioned on axis 2100. Here, the position of marks 1704 and 1705 is determined by setting the position of calibration value 0 to a defocus amount of 0. Also, near the left end of axis 2100, indicator 2106 is positioned to numerically show the difference in calibration values ​​between the right eye optical system 301R and the left eye optical system 301L. Here, the relative value of the calibration value of the right eye optical system 301R when the calibration value of the left eye optical system 301L is set to 0 is shown as the difference in calibration values.

[0200] Figure 21(A) shows that the defocus amount for both the right eye optical system 301R and the left eye optical system 301L is 0, but the focus lens position has been corrected by the calibration value. Specifically, it shows that the focus lens position corresponding to a defocus amount of 0 has been corrected by 2 in the front focus direction (+) for the right eye optical system 301R and by 4 in the back focus direction (-) for the left eye optical system 301L. Here, the unit of the calibration value is predetermined by the camera 100. For example, it may be the number of pulses when driving the focus lens.

[0201] When marks 1704 and 1705 indicate that the image is in focus, the user adjusts the focus lens positions of the right-eye optical system 301R and the left-eye optical system 301L, respectively, while observing the right image 1800R and the left image 1800L of the live view image 1800. When the user feels that the degree of focus in the focus detection area of ​​the right image 1800R and the left image 1800L is at its highest, they give an instruction to the camera 100 via the operating member 230. When the system control unit 218 detects this instruction, it saves the calibration values ​​of the right-eye optical system 301R and the left-eye optical system 301L at that time to the non-volatile memory 220.

[0202] The system control unit 218 corrects the lens drive amount based on the defocus amount using calibration values ​​stored in the non-volatile memory 220, and then transmits it to the lens system control circuit 303. This allows the user to obtain an image that they perceive as having the highest degree of focus when the defocus amount is 0. In other words, it corrects the discrepancy between the degree of focus perceived by the user and the degree of focus determined by the camera 100.

[0203] Figure 21(B) shows an example of the second display mode of the calibration guide 2110. In the second display mode, the information presented in the first display mode is shown for each imaging optical system. Specifically, axes 2100R and 2100L are provided for each imaging optical system, and marks 1704 (1705) indicating the degree of focus and marks 2104 (2105) indicating the calibration value are placed on the corresponding axes. In addition, instead of the indicator 2106 that shows the difference in calibration values, indicators 2106R and 2106L that show the calibration value for each imaging optical system are placed on the corresponding axes 2100R and 2100L. In the second display mode, the display area is larger because information is presented for each imaging optical system, but it is easier for the user to grasp the information. It is also possible to display information for one imaging optical system according to the user's instructions.

[0204] Figure 21(C) shows an example of the third display mode of the calibration guide 2110. This display mode is similar to the first display mode shown in Figure 21(A), but marks 1704 and 1705 are both at the +4 position, indicating a front-focus state. Marks 2104 and 2105, which indicate the calibration value, show the calibration value at the position when marks 1704 and 1705 are at the defocus amount 0 position (Figure 21(A)). If marks 1704 and 1705 are not at the defocus amount 0 position, marks 2104 and 2105 indicate the calibration value by the difference in position from marks 1704 and 1705. Therefore, in Figure 21(C), the calibration value of the right eye optical system 301R is -2, and the calibration value of the left eye optical system 301L is 0. Note that the calibration value may be indicated by the display position of marks 2104 and 2105 regardless of the defocus amount of the right eye optical system 301R and the left eye optical system 301L.

[0205] The method for focusing calibration when the defocus amount of the right eye optical system 301R and the left eye optical system 301L is not zero will be explained using Figure 22. Figure 22(A) shows an example of a sample image corresponding to the degree of focus in Figure 21(C). When performing focus calibration with a defocus amount that is not zero, the system control unit 218 displays the sample image corresponding to the current degree of focus and the live view image (Figure 22(B)) on the display unit 108 or EVF 217.

[0206] When performing focus calibration as shown in Figure 21(A), the sample image in Figure 22(A) is not necessary. The sample image in Figure 22(A) is an image that corresponds to the current degree of focus, which is predicted by using a line image stored in the non-volatile memory 220 in advance, based on the design state of the imaging optical system.

[0207] The user compares the image in Figure 22(A) with the live view image in Figure 22(B) (left image 2201L and right image 2201R). For example, if the amount of blur in the right image 2201R of the live view image in Figure 22(B) appears to be large, the user adjusts the focus lens position of the right eye optical system 301R to a position where the right image 2201R appears to have the same degree of focus as the left image 2201L. This sets the calibration value for the right eye optical system 301R.

[0208] Figure 21(C) shows an example of the calibration guide display after the calibration value of the right eye optical system 301R has been set in this manner. By setting the calibration amount of the right eye optical system 301R from the original value (0) to -2, the blur state of the left and right images becomes consistent, and the desired image can be obtained. Alternatively, without displaying a sample image, the focus lens position of the imaging optical system that forms the other image can be adjusted so that the degree of focus is equivalent to that of the left and right images in the live view image, whichever is perceived as having a higher degree of focus.

[0209] Furthermore, the calibration values ​​may be stored in at least one of the camera 100 and the twin-lens unit 300. This allows the calibration values ​​to be obtained from the twin-lens unit 300 and used in a camera other than the one that performed the calibration on the twin-lens unit 300.

[0210] According to this embodiment, a calibration value can be set for correcting the focus lens position in the imaging optical system of the twin-lens unit. Therefore, it becomes possible to correct the difference between the focus state determined by the camera and the focus state perceived by the user for the twin-lens unit.

[0211] ● (Seventh embodiment) Next, a seventh embodiment of the present invention will be described. This embodiment relates to a focus calibration function when a pair of disparity images, such as a left image and a right image, captured by a twin-lens unit 300, is recorded in a refocusable format. A refocusable image is an image in which the subject distance at which the image is in focus can be changed after capture (recording). For example, this may be an image captured by a light field camera, or an image recorded in association with a group of images taken of the same scene at different focus distances.

[0212] The focus calibration function provided in this embodiment can be used, for example, to set a calibration value to correct the difference in the degree of focus (focus state) between the right and left images when a user views a recorded pair of parallax images. Here, it is assumed that the user views the pair of parallax images while wearing XR goggles, which are display devices having a display unit for the left eye and a display unit for the right eye. Here, XR is a general term for VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality).

[0213] The left image in Figure 23(a) is a perspective view showing an example of the appearance of the XR goggles 2300. The XR goggles 2300 are typically worn on the eye area SO of the head, as shown in the right image in Figure 23(a). Figure 23(b) is a perspective view showing an example of the appearance of the XR goggles 2300 as seen from the wearing side. Figure 23(c) is a schematic diagram showing the positional relationship between the eyepieces 2301R and 2301L, the right eye display unit 2308R and the left eye display unit 2308L, and the user's right eye 501R and left eye 501L when the XR goggles 2300 are worn.

[0214] The XR goggles 2300, for example, displays the right image obtained by the dual-lens unit 300 on the right-eye display unit 2308R and the left image on the left-eye display unit 2308L. Since the right and left images are a pair of parallax images, the user views the right image with the right eye 501R through the eyepiece 302R and the left image with the left eye 501L through the eyepiece 302L, thereby recognizing the right and left images as 3D images. Note that the pair of parallax images displayed on the XR goggles 2300 is not limited to those captured by the dual-lens unit 300. For example, stereoscopic viewing is possible even if the right and left images captured by a stereo camera are displayed on the right-eye display unit 2308R and the left-eye display unit 2308L of the XR goggles 2300.

[0215] This embodiment assumes a case where the focus distances of the right and left images constituting the parallax image pair displayed on the XR goggles 2300 can be changed independently. For example, video data in which the right and left images are recorded in a refocusable format may be played back on a computing device and observed on the XR goggles 2300 connected to the computing device.

[0216] Figure 24 schematically illustrates the change in the in-focus subject due to refocusing. Figure 24(a) shows a state where subject 2403 is in focus, but subjects 2402 and 2404 are not. It is assumed that the images of subjects 2402 and 2404 are not in focus because they are not subject blurred and are located outside the depth of field.

[0217] If the image shown in Figure 24(a) (a still image or a single frame from a video) is recorded in a refocusable format, it can be modified to focus on either subject 2402 or 2404. Figures 24(b) and 24(c) show the images modified to focus on subjects 2402 and 2404, respectively.

[0218] The system control unit 218 modifies the image so that the specified position is in focus, for example, when the user specifies a position to be in focus. Any known method can be used to change the subject in focus through refocusing. When a refocusable image is captured by the camera 100, the number of photodiodes sharing the microlens 401 can be increased in both the horizontal and vertical directions.

[0219] Figure 25 shows examples of the right-eye image 2501R and left-eye image 2501L corresponding to the image shown in Figure 24(a). By displaying the right-eye image 2501R and left-eye image 2501L on the right-eye display unit 2308R and left-eye display unit 2308L of the XR goggles 2300, the user wearing the XR goggles 2300 can get the feeling of being in the scene shown in Figure 24(a). By detecting the user's hand movements and associating them with their positions in the image, the user can virtually touch the subject 2403 to designate it as the subject to focus on.

[0220] Figure 26 is a block diagram showing an example of the functional configuration of a computer 2600 that can be used as a computing device in this embodiment.

[0221] The display 2701 displays information about data being processed by the application program, various message menus, etc., and is composed of an LCD (Liquid Crystal Display) or the like. The display 2701 may also be a touch display. The display controller 2702 controls the screen display on the display 2701. The keyboard 2703 and pointing device 2704 are used for inputting characters, pointing to icons and buttons in the GUI (Graphical User Interface), etc. The CPU 2705 controls the entire computer 2600.

[0222] ROM2706 (Read Only Memory) stores programs and parameters executed by CPU2705. RAM (Random Access Memory)2707 is used as a work area when CPU2705 executes various programs, as well as a temporary storage area for error handling.

[0223] The hard disk drive (HDD) 2708 and removable media drive (RMD) 2709 function as external storage devices. The removable media drive is a device that reads or writes to removable recording media and may be a flexible disk drive, optical disk drive, magneto-optical disk drive, memory card reader, or even a removable HDD. In addition to or instead of the HDD 2708, an SSD (Solid State Drive) may also be included.

[0224] In this embodiment, the programs that implement the various functions of the computer 2600 described, the OS, application programs such as a browser, data, libraries, etc., are stored in one or more of the ROM 2706, HDD 2708, and RMD 2709.

[0225] Expansion slot 2710 is a slot for installing expansion cards that comply with standards such as the PCI (Peripheral Component Interconnect) bus. Various expansion boards, such as video capture boards, sound boards, and GPIB boards, can be installed in expansion slot 2710.

[0226] The external IF2711 is an interface for enabling communication between the computer 2600 and external devices, and complies with one or more wired and / or wireless communication standards. The external IF2711 may have interfaces compliant with one or more of the following: wireless LAN, USB (Universal Serial Bus), HDMI (registered trademark), Bluetooth (registered trademark), 4G (LTE), 5G, etc.

[0227] Bus 2712 consists of an address bus, a data bus, and a control bus, and connects the aforementioned units.

[0228] Next, the focus calibration function in this embodiment will be described. Here, the focus calibration function for the parallax image pair displayed on the XR goggles 2300 is provided by the computer 2600 described above executing an application program that provides the focus calibration function. However, the system control unit 218 of the camera 100 can also provide the focus calibration function by executing a similar program.

[0229] Here, it is assumed that the data for the disparity image pair (right image and left image) is recorded in a refocusable format on the storage device of the computer 2600 (e.g., HDD 2708). The data for the right and left images may be acquired from the camera 100 or other external devices via the external IF 2711. Furthermore, as shown in Figure 27(a), the XR goggles 2300 are connected to the external IF 2711 of the computer 2600, and the right and left images can be displayed.

[0230] The focus calibration function provided by computer 2600 displays the parallax image pair 2602 and the calibration guide 2601 on display 2701.

[0231] Figure 27(b) shows an example of a calibration guide 2601. The calibration guide 2601 has an axis 2610 that indicates a distance range from the nearest end to infinity. The axis 2610 is marked with scales, and values ​​are shown on some of the scales. Here, an indicator 2611 indicating the minimum value (nearest end) and an indicator 2612 indicating the maximum value (infinity) are displayed near the bottom of the axis 2610. Indicators showing values ​​for several distances between the minimum and maximum values ​​are also displayed.

[0232] Furthermore, mark 2613 indicates the position of axis 2610 corresponding to the focus distance when the currently displayed parallax image pair was captured. The focus distance at the time of capture is recorded along with the image data as one of the pieces of information at the time of capture.

[0233] In the example shown in Figure 27(b), mark 2613 indicates 1m. The image at this time is, for example, an image focused on the subject 2403 shown in Figure 24(a). By moving mark 2613, the distance indicated by mark 2613 is changed, and the user can instruct the CPU 2705 to change the focus distance. The user can move mark 2613 to the desired position by operating the keyboard 2703 or pointing device 2704, or by touching the display 2701.

[0234] For example, suppose the user moves mark 2613 and axis 2610 to a position pointing to 2m, as shown in Figure 27(c). The CPU 2705 changes the focus distance of the image data according to the changed position of mark 2613. The CPU 2705 can change the focus distance in a manner appropriate to the recording method. If a light field image is recorded, the focus distance can be changed by a shift operation. Also, if a group of images with different focus distances are recorded, it is sufficient to extract the image that is in focus at the specified distance (an image whose depth of field includes the specified distance). Here, assume that the image shown in Figure 24(b) is an image with a focus distance of 2m, and the image shown in Figure 24(c) is an image with a focus distance of 3m.

[0235] Since the right and left images are focused at the distance indicated by Mark 2613, there should be no difference in the degree of focus between the right and left images. However, manufacturing errors, aging degradation, and environmental factors may cause a difference in the degree of focus between the right and left images due to the multiple lenses, reflective mirrors, and other components contained in each of the two imaging optical systems that form the right and left images.

[0236] Therefore, users may perceive that the degree of focus of the right and left images observed through the display 2701 and XR goggles 2300 is different. In particular, if the degree of focus of one image is lower than that of the other, the user is likely to notice the difference in focus because the two images are displayed adjacent to each other. Therefore, in this embodiment, a focus calibration function is provided to correct the discrepancy between the expected focus state for a distance specified by the user and the degree of focus of the displayed image.

[0237] Figure 27(d) shows an example of the display format of the calibration guide 2601 when setting calibration values. In this embodiment, as shown in Figure 27(a), the calibration values ​​for the left and right images are set by fine-tuning the focus distance while checking the degree of focus of the parallax image pair 2602 displayed on the display 2701.

[0238] Figure 27(d) shows the state in which the CPU 2705 has changed the display mode of the calibration guide 2601 in response to a user instruction to transition from the state in Figure 27(c) to the mode for setting calibration values ​​(calibration mode). Therefore, the currently displayed disparity image pair 2602 is the image presented by the CPU 2705 as an image focused on a subject distance of 2m.

[0239] Here, when transitioning to calibration mode, the indicator on the scale of axis 2610 is changed from showing distance to displaying a numerical value with the current setting value set to 0. This is to make it easier to understand the magnitude and direction of the calibration value. Alternatively, as shown in Figure 26(c), the indicator showing distance may be maintained.

[0240] Furthermore, when the CPU 2705 transitions to calibration mode, it displays an indicator 2624 that shows the current calibration value numerically. In addition, the CPU 2705 displays a mark 2623 for setting the calibration value of the right image and a mark 2622 for setting the calibration value of the left image. The user can move marks 2622 and 2623 by using the keyboard 2703, pointing device 2704, or touch operation on the display 2701.

[0241] When CPU2705 detects movement of mark 2623, it changes the focus distance of the right image in the disparity image pair 2602 according to the direction and amount of movement from the initial position (0). For example, if movement of mark 2623 in the + direction is detected, CPU2705 changes the focus distance toward infinity, and if movement of mark 2623 in the - direction is detected, it changes the focus distance toward the nearest end. The amount of change in distance per division may be a predetermined constant value. If movement of mark 2622 is detected, CPU2705 changes the focus distance of the left image in the same way as the right image. The user operates marks 2622 and 2623 so that the right and left images appear to be in focus at the specified distance.

[0242] Figure 27(d) shows the state where the calibration value of the left image is 0 and the calibration value of the right image is -4. This indicates that the left image does not require correction of the set distance, and by adjusting the calibration value of the right image to the near end, the in-focus area of ​​the right image appears to be the same as the in-focus area of ​​the left image.

[0243] Once calibration is complete, the user signals the computer 2600 to finish calibration by touching the keyboard 2703, pointing device 2704, or display 2701.

[0244] When CPU2705 detects this instruction, it stores the currently set calibration value as the application setting value, for example, in HDD2708. Furthermore, when updating the disparity image pair 2602, CPU2705 reflects the calibration value in the distance setting. Note that the calibration value can be applied not only when playing back the image data used for the setting, but also to other image data captured by the same device that captured the image data in question. Also, if a calibration value is set for one frame of video data, the calibration can be applied to other frames as well. The set calibration value may be applied automatically, or it may be applied only when instructed by the user.

[0245] According to this embodiment, it becomes possible to provide a focusing distance calibration function for pairs of disparity images recorded in a refocusable format.

[0246] (modified version) An indicator showing the hyperfocal distance may be added to the calibration guide 2601. This allows the user to easily obtain a pan-focus image. The hyperfocal distance can be determined from the focal length and F-number of the lens unit, which are recorded along with the image data as information at the time of shooting, and the acceptable circle of confusion diameter, which is, for example, the pixel pitch of the image sensor.

[0247] (Other embodiments) In the embodiments described above, the display positions of the marks or indicators for the right and left images may be reversed. Furthermore, the form of the GUI (Graphical User Interface) that constitutes the various indicators and guides is not limited to those shown. Any form of GUI capable of presenting the same information to the user as the indicators and guides exemplified in the embodiments can be used.

[0248] Furthermore, the various controls described above, which are performed by the system control unit 218, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0249] Furthermore, although the present invention has been described in detail based on exemplary embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0250] Furthermore, although the above-described embodiments used the application of the present invention to a digital camera (imaging device) as an example, the invention is not limited to this example and can be applied to any display control device capable of displaying display items related to focus. In other words, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, printer devices equipped with displays, digital photo frames, music players, game consoles, e-book readers, and the like.

[0251] Moreover, the present invention is applicable not only to the imaging device main body but also to a control device that communicates with an imaging device (including a network camera) via wired or wireless communication and remotely controls the imaging device. Examples of devices for remotely controlling an imaging device include devices such as smartphones, tablet PCs, and desktop PCs. By notifying the imaging device from the control device side of commands for causing the imaging device to perform various operations and settings based on operations performed on the control device side or processes performed on the control device side, the imaging device can be remotely controlled. Also, a live view image captured by the imaging device can be received via wired or wireless communication and displayed on the control device side.

[0252] Examples of the embodiments disclosed in this specification are listed below. [Embodiment 1] An imaging device having an imaging element with a plurality of focus detection pixels that receive light beams passing through different pupil partial regions of an imaging optical system, focus detection means for obtaining a defocus amount of the imaging optical system based on a pair of signals from the focus detection pixels, a focus detection adjustment value used when obtaining the defocus amount, In the imaging device having, the imaging optical system has a plurality of optical axes, the focus detection adjustment value corresponds to the plurality of optical axes, converts optical information on each optical axis of the imaging optical system based on the position of each optical axis on the imaging device, and calculates a focus detection adjustment value in the imaging optical system from the converted optical information. The imaging device is characterized by this. [Embodiment 2] The imaging device according to Embodiment 1, wherein the focus detection adjustment value includes at least one of a conversion coefficient for calculating a defocus amount or a correction value used for correction means for suppressing the difference in intensity between a pair of focus detection signals. [Embodiment 3] The imaging device according to Embodiment 1 or 2, wherein the imaging device has a focus detection adjustment value when an imaging optical system having one optical axis at the center of the imaging element is mounted and a focus detection adjustment value when an imaging optical system having a plurality of optical axes at a position different from the center of the imaging element is mounted. [Embodiment 4] The imaging device according to Embodiment 3, wherein the focus detection adjustment value to be used is switched based on information for discriminating whether the imaging optical system is an imaging optical system having one optical axis at the center of the imaging element or an imaging optical system having a plurality of optical axes at a position different from the center of the imaging element. [Embodiment 5] The imaging device according to Embodiment 4, wherein the information is stored in the imaging optical system, and when the imaging device is mounted, the imaging device acquires the information via communication. [Embodiment 6] The imaging device according to Embodiment 5, wherein the information includes the optical axis position on the imaging element. [Embodiment 7] An imaging device having an imaging element having a plurality of focus detection pixels that receive light fluxes passing through different pupil partial regions of an imaging optical system, <​​​​​​​​​​​​​​​​​​​​Acquisition means for acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system oriented in the same direction as the first optical system, which has parallax with respect to the first live view image, A display control means that superimposes a display item indicating focus information onto the first live view image and the second live view image, and controls the display of the first display item and the second display item at the same image height. An electronic device characterized by having the following features. [Embodiment 11] An electronic device that displays captured images in live view, A first live view image captured through the first optical system, Acquisition means for acquiring a second live view image which is captured via a second optical system oriented in the same direction as the first optical system and has parallax with respect to the first live view image, A display control means controls the display of a third display item, which indicates the difference in focus information between the first live view image and the second live view image, superimposed on the live view image. An electronic device characterized by having the following features. [Embodiment 12] The electronic device according to embodiment 10 or 11, characterized in that the focus information is focus information for a focus detection region based on the display position. [Embodiment 13] The electronic device according to embodiment 10, characterized in that the display control means moves in conjunction with the position of a display item indicating focus information, which is displayed superimposed on the first live view image and the second live view image. [Embodiment 14] The electronic device according to embodiment 10 or 13, characterized in that the display control means displays the display positions of the first display item and the second display item at the same image height with reference to the optical axis position of the first optical system and the optical axis position of the second optical system, respectively. [Embodiment 15] The electronic device according to Embodiment 10, characterized in that the display control means selectively displays a display item indicating focus information on either the first live view image and the second live view image, only the first live view image, or only the second live view image. [Embodiment 16] The electronic device according to embodiment 10 or 11, characterized in that the display control means controls when a lens having a first focus ring for adjusting the focus of the first live view image and a second focus ring for adjusting the focus of the second live view image is attached. [Embodiment 17] The electronic device according to embodiment 16, characterized in that the display color of the first display item is the same as the color of the first focus ring, and the display color of the second display item is the same as the color of the second focus ring, thereby making the display colors of the first display item and the second display item different. [Embodiment 18] The electronic device according to embodiment 10 or 11, characterized in that the display control means controls when a lens is attached that has a third focus ring for simultaneously adjusting the focus of the first live view image and the second live view image, and a fourth focus ring for adjusting the focus of either the first live view image or the second live view image. [Embodiment 19] The electronic device according to embodiment 18, characterized in that the display color of a display item whose focus can be adjusted using only the third focus ring is different from the display color of a display item whose focus can be adjusted using both the third and fourth focus rings. [Embodiment 20] The electronic device according to embodiment 10 or 11, characterized in that the display control means displays the first display item and the second display item when displaying the third display item. [Embodiment 21] The electronic device according to embodiment 10 or 11, characterized in that the display control means displays a third display item which shows the difference between the first focus information displayed on the first display item and the second focus information displayed on the second display item, with the difference information based on the first focus information or the second focus information. [Embodiment 22] The electronic device according to embodiment 10 or 11, characterized in that the display control means displays the third display item only when performing an adjustment mode that adjusts the difference between the first focus information displayed on the first display item and the second focus information displayed on the second display item. [Embodiment 23] An acquisition step of acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system oriented in the same direction as the first optical system, which has parallax with respect to the first live view image, A display control step that superimposes a display item indicating focus information onto the first live view image and the second live view image, and controls the display so that the first display item and the second display item are displayed at the same image height. A method for controlling electronic equipment, characterized by having the following features. [Embodiment 24] An acquisition step of acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system oriented in the same direction as the first optical system, which has parallax with respect to the first live view image, A display control step that controls the display to superimpose a third display item, which indicates the difference in focus information between the first live view image and the second live view image, onto the live view image. A method for controlling electronic equipment, characterized by having the following features. [Embodiment 25] A program for causing a computer to function as each means of the electronic device according to any one of Embodiments 10 to 22. [Embodiment 26] A computer-readable storage medium storing a program for causing a computer to function as each means of the electronic device according to any one of Embodiments 10 to 22. [Embodiment 27] An electronic device for live-view displaying a captured image, acquisition means for acquiring a first live-view image captured via a first optical system and a second live-view image captured via a second optical system having the same orientation as the first optical system and having a parallax with respect to the first live-view image; a first display item indicating information regarding focus of the first live-view image and a second display item indicating information regarding focus of the second live-view image; display control means for controlling to superimpose and display the first display item and the second display item on the live-view display; An electronic device characterized by having the above. [Embodiment 28] The electronic device according to Embodiment 27, wherein the display control means superimposes and displays a plurality of the first display items and the second display items on the live-view display. [Embodiment 29] The electronic device according to Embodiment 27, wherein the first display item and the second display item are displayed at positions based on the optical axis positions of the first optical system and the second optical system. [Embodiment 30] The electronic device according to Embodiment 27 or Embodiment 29, wherein when the display position of the first or second display item in the live-view display is changed, the first or second display item also moves according to the optical axis positions of the first optical system and the second optical system. [Embodiment 31] An acquisition means for acquiring a first defocus amount detected by phase difference using a pair of light beams passing through different exit pupils of the first optical system including a first focus ring, and a second defocus amount detected by phase difference using a pair of light beams passing through different exit pupils of the second optical system including a second focus ring, An adjustment means that automatically adjusts the amount of drive of the first focus ring or the second focus ring based on the difference between the first defocus amount and the second defocus amount, An imaging device characterized by having the following features. [Embodiment 32] An imaging device that displays captured images in live view, Acquisition means for acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system oriented in the same direction as the first optical system, which has parallax with respect to the first live view image, An adjustment means for similarly adjusting the focus state of the first and second live view images, The system includes a storage means for storing the calibration value set by the adjustment means, An imaging device characterized by allowing the calibration value to be applied at any time. [Embodiment 33] A display device that displays a pair of disparity images including a first image and a second image, An adjustment means for similarly adjusting the focus state of the first image and the second image, A storage means for storing the calibration value set by the adjustment means, It has, A display device characterized by enabling the change in focus state by applying the aforementioned calibration value to an image. [Embodiment 34] The display device according to embodiment 33, characterized in that the change in the focus state by the adjustment means is performed by selecting from a plurality of images with different focus states. [Embodiment 35] The display device according to embodiment 34, characterized in that the change in focus state by the adjustment means is performed by a refocus process. [Embodiment 36] The display device according to any one of embodiments 33 to 35, characterized in that the calibration value is set in one frame and applied to other frames when the image is a video.

[0253] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0254] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0255] 100...Camera body, 108...Display unit, 200...Single-lens unit, 211...Imaging unit, 214...Image processing unit, 218...System control unit, 300...Twin-lens unit

Claims

1. A detection means for detecting a first defocus amount based on the phase difference between a pair of light beams passing through different exit pupils of a first optical system including a first focusing lens, and a second defocus amount based on the phase difference between a pair of light beams passing through different exit pupils of a second optical system including a second focusing lens, An imaging apparatus characterized by having an adjustment means for automatically adjusting the drive amount of the first focus lens or the drive amount of the second focus lens based on the difference between the first defocus amount and the second defocus amount.

2. The system further includes setting means for setting one of the first optical system and the second optical system as a reference optical system, The imaging apparatus according to claim 1, characterized in that the adjustment means calculates the difference between the defocus amount of another optical system and the defocus amount of an optical system set as a reference, and determines the drive amount of the non-reference optical system based on the result of said calculation.

3. The imaging apparatus according to claim 2, characterized in that the setting means sets one of the following as the reference optical system: the optical system that cannot be driven independently, the optical system that is set in advance by the user, the optical system that has high subject detection accuracy, or the optical system that has high focus detection accuracy.

4. A first adjustment mode adjusts the difference in imaging position between the first optical system and the second optical system by moving at least one of the first optical system and the second optical system in the optical axis direction in response to the operation of an operating member by the user, The system includes a second adjustment mode that automatically adjusts the difference in imaging position between the first optical system and the second optical system according to the difference in the amount of defocus of the other optical system with respect to the amount of defocus of the optical system set as a reference, The imaging apparatus according to claim 1, characterized in that the adjustment means performs the adjustment in the case of the second adjustment mode.

5. A user-operated operating component, The imaging apparatus according to claim 1, further comprising: a storage means for storing a correction value for correcting the amount of defocus detected by the focus detection means, which is an adjustment value operable by the operating member.

6. A method performed by an imaging device, A detection step for detecting a first defocus amount based on the phase difference between a pair of light beams passing through different exit pupils of a first optical system including a first focusing lens, and a second defocus amount based on the phase difference between a pair of light beams passing through different exit pupils of a second optical system including a second focusing lens, A method characterized by comprising an adjustment step of automatically adjusting the drive amount of the first focus lens or the drive amount of the second focus lens based on the difference between the first defocus amount and the second defocus amount.

7. A program for causing a computer to perform each step of the method described in claim 6.

8. A computer-readable storage medium storing a program for causing a computer to perform each step of the method according to claim 6.

Images