Image pickup apparatus, control method thereof, computer program product, and storage medium

Abstract

The invention relates to an imaging apparatus, a control method thereof, a computer program product, and a storage medium. The image pickup apparatus includes: an image pickup device configured to photoelectrically convert a first optical image formed by a first image pickup optical system and further configured to photoelectrically convert a second optical image formed by a second image pickup optical system arranged in parallel with the first image pickup optical system; a calculation unit configured to calculate a three-dimensional range, which is a range of a subject distance of the imaging device, in which a three-dimensional image appears to be three-dimensional; and a control unit configured to control the first imaging optical system and the second imaging optical system to limit a range of a subject distance in which auto-focusing of the first imaging optical system and the second imaging optical system is performed to be within the three-dimensional range and perform auto-focusing.

Description

Technical Field

[0001] The present invention relates to a camera device for photographing three-dimensional images. Background Art

[0002] A known technology for displaying stereoscopic virtual reality (VR) images uses two optical systems to obtain wide-view images with parallax, and maps these images onto a virtual sphere for display. A dual-lens VR camera for capturing images with parallax includes two optical systems facing the same direction, and can obtain two images with parallax via a single camera.

[0003] In a dual-lens VR camera, each optical system can capture a wide range of images of at least 180° vertically and horizontally (hemisphere: 90° in all directions from the center of the image). In addition, known methods for displaying VR images include: "single-lens VR display", which performs deformation to map the VR image on a virtual sphere and displays one image; and "dual-lens VR display", which displays VR images for the left eye and the right eye side by side in the left and right areas.

[0004] However, it is known that regarding the distance to the subject during VR videography, there is a three-dimensional range that allows comfortable stereoscopic viewing. When the subject is closer than the three-dimensional range, the parallax of the subject increases, which causes double vision of the image. In addition, when the subject is farther than the three-dimensional range, the parallax decreases, which causes the image to appear to be merely flat and thus lose the stereoscopic effect.

[0005] Various techniques have been proposed to solve this problem.

[0006] Japanese Patent Laid-Open No. 2004-356775 describes a technique for obtaining an image for the left eye and an image for the right eye suitable for three-dimensional viewing, which technique includes adjusting the distance between the optical axes of the imaging optical systems of the left lens and the right lens based on the distance to the subject measured by a rangefinder.

[0007] In Japanese Patent Laid-Open No. 2012-015620, the following technology is described. An appropriate parallax range is calculated based on the size of the display screen, a three-dimensional image is divided into a plurality of regions, and it is determined whether the parallax in each region is within an appropriate parallax range. In the case where the parallax is not within an appropriate parallax range, a warning is displayed by an image display unit that displays the three-dimensional image. In the case where the parallax is not within an appropriate parallax range but the image of the region is blurred, no warning is displayed. In this way, the user can be made aware of an area with inappropriate parallax when a three-dimensional image is captured.

[0008] However, in Japanese Patent Laid-Open No. 2004-356775, with a dual lens in which the distance between the optical axes of the left lens and the right lens is fixed, optimal left-eye and right-eye stereoscopic images cannot be obtained.

[0009] In Japanese Patent Laid-Open No. 2012-015620, during VR moving image shooting, in a case where the parallax is not within an appropriate parallax range, only a warning is displayed, and the moving image is recorded as it is. Summary of the invention

[0010] The present invention has been made in view of the above-mentioned problems, and makes it possible to realize an image pickup apparatus that can pick up an image with an appropriate three-dimensional effect.

[0011] According to a first aspect of the present invention, there is provided an imaging device, comprising: an imaging device configured to perform photoelectric conversion on a first optical image formed by a first imaging optical system to output a first image, and also configured to perform photoelectric conversion on a second optical image formed by a second imaging optical system arranged in parallel with the first imaging optical system to output a second image; a calculation unit configured to calculate, when the first image and the second image are displayed on a display device, a three-dimensional range as a range of a subject distance of the imaging device within which the three-dimensional image appears to be three-dimensional; and a control unit configured to control the first imaging optical system and the second imaging optical system to limit the range of the subject distance for automatic focusing of the first imaging optical system and the second imaging optical system to within the three-dimensional range and perform automatic focusing.

[0012] According to a second aspect of the present invention, a control method for an imaging device is provided, the imaging device including an imaging device, the imaging device being configured to perform photoelectric conversion on a first optical image formed by a first imaging optical system to output a first image, and also being configured to perform photoelectric conversion on a second optical image formed by a second imaging optical system arranged in parallel with the first imaging optical system to output a second image, the control method comprising: when the first image and the second image are displayed on a display device, calculating a three-dimensional range as a range of a subject distance of the imaging device within which the three-dimensional image is made to appear three-dimensional; and controlling the first imaging optical system and the second imaging optical system to limit the range of a subject distance for automatic focusing of the first imaging optical system and the second imaging optical system to within the three-dimensional range and perform automatic focusing.

[0013] According to a third aspect of the present invention, a computer program product is provided, for causing a computer to execute the above control method.

[0014] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a program for causing a computer to execute the above-mentioned control method.

[0015] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A and Figure 1B is a diagram illustrating an example of the external appearance structure of a camera.

[0017] Figure 2 is a diagram illustrating an example of the internal structure of a camera when a single lens unit is mounted.

[0018] Figure 3 is a schematic diagram illustrating an example of the structure of a dual lens unit.

[0019] Figure 4 is a schematic diagram illustrating an example of a pixel array of an image sensor inside an image pickup unit.

[0020] FIG. 5A to FIG. 5C is a diagram illustrating an example of display of a live view image in a camera mounted with a twin-lens unit.

[0021] Figure 6 This is a flowchart illustrating an example of image capture processing.

[0022] Figure 7 is a flowchart illustrating a moving image shooting process.

[0023] Figure 8 is a flowchart illustrating still image shooting processing.

[0024] Fig. 9 is a flowchart illustrating Example 1 of focus detection processing.

[0025] Fig.10 is a flowchart illustrating Example 2 of focus detection processing.

[0026] Fig.11 2 is a flowchart illustrating the focus stop state processing.

[0027] Fig.12 is a flowchart illustrating the focus state processing.

[0028] Fig.13 is a flowchart illustrating Example 1 of the lens driving process.

[0029] Fig.14 is a flowchart illustrating Example 2 of the lens driving process.

[0030] Fig.15is a flowchart illustrating Example 3 of the lens driving process.

[0031] Fig.16A and Fig. 16B It is a graph related to three-dimensional range calculation.

[0032] Fig.17 It is a diagram illustrating a display example of a three-dimensional range displayed in a live view image.

[0033] Fig.18 is a flowchart illustrating Example 1 of a three-dimensional out-of-range warning.

[0034] FIG. 19A to FIG. 19C : is a diagram illustrating an example of display of text and an icon according to Example 1 of a three-dimensional out-of-range warning.

[0035] Fig. 20 is a flowchart illustrating Example 2 of a three-dimensional out-of-range warning.

[0036] Fig.21 is a flowchart illustrating Example 3 of a three-dimensional out-of-range warning.

[0037] Fig.22A and Fig. 22B : is a diagram illustrating an example of AF frame display according to Example 3 of a three-dimensional out-of-range warning.

[0038] Fig.23 is a flowchart illustrating Example 4 of a three-dimensional out-of-range warning.

[0039] Fig.24 : is a diagram illustrating an example of superimposed display according to Example 4 of a three-dimensional out-of-range warning. DETAILED DESCRIPTION

[0040] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but are not limited to an invention requiring all of these features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar structures, and redundant descriptions thereof are omitted.

[0041] Camera appearance structure

[0042] Figure 1A and Figure 1B 1 is a diagram illustrating an example of an external appearance structure of a digital camera 100 (hereinafter referred to as a camera) as an embodiment of an image pickup apparatus of the present invention. Figure 1A is a perspective view of the camera 100 as seen from the front, and Figure 1B It is a perspective view of the camera 100 as seen from the back.

[0043] The camera 100 includes a shutter button 101, a power switch 102, a mode selection switch 103, a main electronic dial 104, a sub-electronic dial 105, a moving image button 106, and a non-viewfinder display unit 107 on its upper surface. The shutter button 101 is an operation unit for performing image capture preparation or image capture instructions. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode selection switch 103 is an operation unit for selecting various modes. The main electronic dial 104 is a rotation operation unit for changing setting values ​​such as shutter speed and aperture. The sub-electronic dial 105 is a rotation operation unit for moving a selection frame (cursor) and image switching. The moving image button 106 is an operation unit for issuing an instruction to start or stop moving image capture (recording). The non-viewfinder display unit 107 displays various setting values ​​such as shutter speed and aperture.

[0044] Furthermore, the camera 100 includes a display unit 108, a touch panel 109, a direction key 110, a set (SET) button 111, an AE lock button 112, an AF frame selection / magnification button 113, a reproduction button 114, a menu button 115, an eyepiece unit 116, an eye approach detection unit 118, and a touch bar 119 on its back. The display unit 108 displays images and various types of information. The touch panel 109 is an operation unit that detects a touch operation on the display surface (touch operation screen) of the display unit 108. The direction key 110 is an operation unit composed of keys (four-direction keys) that can be pressed upward, downward, leftward, and rightward. An operation corresponding to the pressed position of the direction key 110 can be performed. The set button 111 is an operation unit that is pressed mainly to determine a selection item. The AE lock button 112 is an operation unit that is pressed to fix the exposure state in the image capture standby state. The AF frame selection / enlargement button 113 is a common operation unit in the live view display (LV display) in the camera mode, which is used for selection operation to enable the movement operation of the AF frame and for switching the enlargement mode to on and off. By operating the main electronic dial 104 when the enlargement mode is on, the live view image (LV image) is enlarged or reduced. In addition, when enlarging the reproduced image in the reproduction mode, increasing the magnification, etc., the AF frame selection / enlargement button 113 is used. The reproduction button 114 is an operation unit for switching between the camera mode and the reproduction mode. When the reproduction button 114 is pressed in the camera mode, the mode changes to the reproduction mode, and the latest image among the images recorded in the recording medium 229 described below can be displayed on the display unit 108.

[0045] The menu button 115 is an operation unit pressed to display a menu screen on the display unit 108 in which various settings can be set. The user can intuitively set various settings using the menu screen displayed on the display unit 108, the direction key 110, and the setting button 111. The eyepiece unit 116 is a portion to which the eye approaches to view the eyepiece viewfinder 117 (view-type viewfinder). The user can look through the eyepiece unit 116 to see an image displayed on an internal electronic viewfinder (EVF) 217 ​​described below. The eye approach detection unit 118 is a sensor that detects whether the user's eye approaches the eyepiece unit 116.

[0046] The touch bar 119 is a linear touch operation unit (line touch sensor) that can receive a touch operation. The touch bar 119 is arranged at a position where the touch bar 119 can be touch-operated (touchable) with the thumb of the right hand when the grip unit 120 is held with the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. In other words, the touch bar 119 can be operated in a state where the shutter button 101 can be pressed at any time when the user brings their eyes close to the eyepiece unit 116 and looks through the eyepiece finder 117. The touch bar 119 can receive a tap operation (an operation of touching and releasing without moving the touch position within a predetermined period) and a left or right slide operation (an operation of moving the touch position while maintaining the touch state) and the like on the touch bar 119. The touch bar 119 is an operation unit different from the touch panel 109, and does not include a display function. The touch bar 119 according to the present embodiment is a multi-function bar, and is used as an M-Fn bar, for example.

[0047] Furthermore, the camera 100 includes a grip unit 120, a thumb rest unit 121, a terminal cover 122, a cover 123, a communication terminal 124, and the like. The grip unit 120 is a holding unit formed in a shape that is easy to hold with the right hand when the user holds the camera 100. The shutter button 101 and the main electronic dial 104 are arranged at a position that allows operation using the index finger of the right hand when the camera 100 is held by holding the grip unit 120 with the little finger, ring finger, and middle finger of the right hand. Furthermore, the sub-electronic dial 105 and the touch bar 119 are arranged at a position that allows operation by the thumb of the right hand when in a similar state. The thumb rest unit 121 (thumb standby position) is a grip unit provided at a position on the back of the camera 100 where the thumb of the right hand holding the grip unit 120 can easily rest when the operation unit is not being operated. The thumb rest unit 121 is composed of a rubber member for improving the holding force (gripping effect). The terminal cover 122 protects a connection cable and similar connectors that connect the camera 100 to an external device. The cover 123 protects a recording medium 229 and the slot by closing the slot that accommodates the recording medium 229 described below. The communication terminal 124 is a terminal for communicating with the lens unit 200 described below that can be detachably attached to the camera 100.

[0048] Camera internal structure

[0049] Figure 2 is a diagram illustrating an example of the internal structure of the camera 100. Figure 1A and Figure 1B The same components are given the same reference numerals, and description thereof is appropriately omitted. A lens unit 200 is installed in the camera 100 .

[0050] First, the lens unit 200 will be described.

[0051] The lens unit 200 is a kind of interchangeable lens detachably attached to the camera 100. The lens unit 200 is a single lens, and is an example of a typical lens.

[0052] The lens unit 200 includes an aperture 201 , a lens 202 , an aperture driving circuit 203 , an auto focus (AF) driving circuit 204 , a lens system control circuit 205 , a communication terminal 206 , and the like.

[0053] The aperture 201 is configured with an adjustable opening diameter. The lens 202 is composed of a plurality of lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the opening diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 for focusing. The lens system control circuit 205 controls the aperture drive circuit 203 and the AF drive circuit 204, etc. based on an instruction from a system control unit 218 described below. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203, and changes the position of the lens 202 via the AF drive circuit 204 for focusing. The lens system control circuit 205 can communicate with the camera 100. Specifically, the communication terminal 206 of the lens unit 200 communicates with the camera 100 using the communication terminal 124. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100 side.

[0054] Next, the camera 100 will be described.

[0055] The camera 100 includes a shutter 210 , an imaging unit 211 and an A / D converter 212 , a memory control unit 213 , an image processing unit 214 , a memory 215 , a D / A converter 216 , an EVF 217 , the display unit 108 , and a system control unit 218 .

[0056] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on an instruction from the system control unit 218. The imaging unit 211 is equipped with an image sensor composed of a CCD and a CMOS element, etc., for converting an optical image into an electrical signal. The imaging unit 211 may include an image plane phase detection sensor that outputs defocus amount information to the system control unit 218. The A / D converter 212 converts an analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing processing such as reduction, and color conversion processing, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. In addition, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and the system control unit 218 performs exposure control and distance measurement control based on the obtained arithmetic operation result. Through this processing, through-the-lens (TTL) type auto focus (AF) processing, automatic exposure (AE) processing, flash pre-emission (EF) processing, etc. are performed. In addition, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and performs TTL type automatic white balance (AWB) based on the obtained arithmetic operation result.

[0057] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 or the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213, bypassing the image processing unit 214. The memory 215 stores image data obtained by the camera unit 211 and converted into digital data by the A / D converter 212, image data for display on the display unit 108 or the EVF 217, and the like. The memory 215 is equipped with a storage capacity sufficient to store a predetermined number of still images, and a predetermined amount of time of video and audio, and the like. In addition, the memory 215 is also used as a memory (video memory) for image display.

[0058] The D / A converter 216 converts the data for image display stored in the memory 215 into an analog signal, and supplies the analog signal to the display unit 108 or the EVF 217. In this way, the image data for display written in the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display corresponding to the analog signal from the D / A converter 216. The display unit 108 and the EVF 217 are, for example, LCD, organic EL, or similar displays. Live view display is performed by using the D / A converter 216 to convert the digital signal obtained by the A / D converter 212 through A / D conversion and accumulated in the memory 215 into an analog signal, and successively transmitting and displaying the signals on the display unit 108 or the EVF 217.

[0059] The system control unit 218 is a control unit including at least one processor and / or at least one circuit. In other words, the system control unit 218 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 218 controls the entire camera 100. The processing of the flowchart described below is realized by the system control unit 218 executing a program stored in the nonvolatile memory 220. In addition, the system control unit 218 controls the memory 215, the D / A converter 216, the display unit 108, the EVF 217, and the like to perform display control.

[0060] Furthermore, the camera 100 includes a system memory 219 , a nonvolatile memory 220 , a system timer 221 , a communication unit 222 , a gesture detection unit 223 , and an eye proximity detection unit 118 .

[0061] The system memory 219 uses, for example, a RAM. Constants and variables used for the operation of the system control unit 218 and a program read out from the nonvolatile memory 220 are loaded onto the system memory 219. The nonvolatile memory 220 is an electrically erasable and recordable memory, and, for example, an EEPROM can be used. Constants and programs used for the operation of the system control unit 218 and the like are stored in the nonvolatile memory 220. Here, the program refers to a program for executing the flowchart described below.

[0062] The system timer 221 is a time measurement unit that measures the time used for various controls, the time of the built-in timer, etc. The communication unit 222 exchanges image signals and audio signals with an external device connected wirelessly or with a wired cable. The communication unit 222 can be connected to a wireless local area network (LAN) and the Internet. In addition, the communication unit 222 can communicate with an external device via Bluetooth (registered trademark) or Bluetooth low energy. The communication unit 222 can send images (including real-time images) captured by the camera unit 211 and images recorded on the recording medium 229, and can receive image data and various other types of information from an external device.

[0063] The posture detection unit 223 detects the posture of the camera 100 with respect to the gravity direction. It can be determined whether the image captured by the camera unit 211 is an image captured by the camera 100 in the horizontal or vertical direction based on the posture detected by the posture detection unit 223. The system control unit 218 can add orientation information based on the posture detected by the posture detection unit 223 to the image file of the image captured by the camera unit 211, or rotate and record the image. The posture detection unit 223 can use, for example, an acceleration sensor or a gyro sensor. By using the posture detection unit 223, it is also possible to detect the movement of the camera 100 (such as panning, tilting, lifting, and whether the digital camera 100 is stationary).

[0064] The sound generator 224 can output sounds via a command from the system control unit 218. For example, effect sounds at the time of operation, AF focus state notification sounds, and audio from captured moving images and the like can be emitted.

[0065] The eye approach detection unit 118 can detect a certain object approaching the eyepiece unit 116 of the eyepiece finder 117 including the built-in EVF 217. The eye approach detection unit 118 can use, for example, an infrared proximity sensor. When an object is nearby, infrared light from the light projection unit of the eye approach detection unit 118 is reflected by the object and received by the light receiving unit of the infrared proximity sensor. The amount of received infrared light can be used to determine the distance from the eyepiece unit 116 to the object. In this way, the eye approach detection unit 118 detects eye approach by detecting the approach distance of the object relative to the eyepiece unit 116. The eye approach detection unit 118 is an eye approach detection sensor that detects the approach (eye approach) and retreat (eye retreat) of the eyepiece unit 116 of the eyepiece finder 117. In the non-eye approach state (non-approach state), when an object is detected within a predetermined distance near the eyepiece unit 116, eye approach is detected. In the eye approach state (approach state), when the object detected as approaching moves away by more than a predetermined distance, eye separation is detected. For example, the threshold for detecting eye approach and the threshold for detecting eye separation may be made different by providing hysteresis, etc. The state from when eye approach is detected until when eye separation is detected is determined as the eye approach state. The state from when eye separation is detected until when eye approach is detected is determined as the non-eye approach state.

[0066] The system control unit 218 switches between display (display state) and non-display (non-display state) of the display unit 108 and the EVF 217 according to the state detected by the eye approach detection unit 118. Specifically, in a case where at least the imaging standby state is active and the display destination switching is set to automatic, when the eye is not nearby, the display destination is set to the display unit 108 and the display is turned on and the EVF 217 is set to non-display. Furthermore, when the eye is nearby, the display destination is set to the EVF 217 and the display is turned on and the display unit 108 is set to non-display. Note that the eye approach detection unit 118 is not limited to the infrared proximity sensor, and other sensors may be used as long as they can detect a state defined as eye approach.

[0067] Furthermore, the camera 100 includes the non-viewfinder display unit 107 , a non-viewfinder display drive circuit 225 , a power control unit 226 , a power supply unit 227 , a recording medium I / F 228 , an operation unit 230 , and the like.

[0068] The non-viewfinder display unit 107 displays various setting values ​​of the camera 100 including shutter speed and aperture, etc., via the non-viewfinder display drive circuit 225. The power control unit 226 includes a battery detection circuit, a DC-DC converter, and a switch circuit for switching a block to be powered, and detects whether a battery is mounted, the type of battery, and the remaining battery power. In addition, the power control unit 226 controls the DC-DC converter based on the detection result and the instruction from the system control unit 218, and supplies the required voltage to various components including the recording medium 229 at the required time. The power unit 227 includes a primary battery (such as an alkaline battery and a lithium battery), a secondary battery (such as a nickel-cadmium (NiCd) battery, a nickel-metal hydride (NiMH) battery, and a lithium-ion (Li) battery), etc.) and / or an alternating current (AC) adapter. The recording medium I / F 228 is an interface with a recording medium 229 (such as a memory card or a hard disk). The recording medium 229 is a memory card or the like for recording a captured image, and is composed of a semiconductor memory or a magnetic disk, etc. The recording medium 229 may be detachable, or may be built-in.

[0069] The operation unit 230 is an input unit that receives an operation from a user (user operation), and is used to input various types of instructions to the system control unit 218. The operation unit 230 includes the shutter button 101, the power switch 102, the mode selection switch 103, the touch panel 109, and other operation members 231, etc. The other operation members 231 include the main electronic dial 104, the sub-electronic dial 105, the moving image button 106, the direction key 110, the setting button 111, the AE lock button 112, the AF frame selection / magnification button 113, the reproduction button 114, the menu button 115, the touch bar 119, and the like.

[0070] The shutter button 101 includes a first shutter switch 232 and a second shutter switch 233. The first shutter switch 232 is turned on in the middle of the operation of the shutter button 101, or in other words, when it is half-pressed (imaging preparation instruction), and generates a first shutter switch signal SW1. The system control unit 218 starts AF processing, AE processing, AWB processing, EF processing, and similar imaging preparation processing in response to the first shutter switch signal SW1. The second shutter switch 233 is turned on when the operation of the shutter button 101 is completed, or in other words, when it is fully pressed (imaging instruction), and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 218 starts a series of imaging processing operations from reading out a signal from the imaging unit 211 until an image file including a captured image is generated and written to the recording medium 229.

[0071] The mode selection switch 103 switches the operation mode of the system control unit 218 to any one of a still image shooting mode, a video shooting mode, a reproduction mode, or the like. The modes included in the still image shooting mode are an automatic image shooting mode, an automatic scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, various types of scene modes (which include shooting settings specific to each shooting scene), and a custom mode are also included. The user can directly switch to any of the above-mentioned shooting modes via the mode selection switch 103. Alternatively, after the user switches to a screen listing shooting modes via the mode selection switch 103, the user can selectively switch to any of the displayed multiple modes via the operation unit 230. In a similar manner, the video shooting mode can include multiple modes.

[0072] The touch panel 109 is a touch sensor that detects various types of touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 may be formed integrally. For example, the touch panel 109 is set to have light transmittance that does not hinder the display of the display unit 108, and is attached to the upper layer of the display surface of the display unit 108. In addition, by associating the input coordinates of the touch panel 109 with the display coordinates on the display surface of the display unit 108, a graphical user interface (GUI) that allows the user to directly operate the screen displayed on the display unit 108 can be constituted. For the touch panel 109, various method types such as a resistive film type, an electrostatic capacitance type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type can be used. According to the method, a touch is detected in the case of touching the touch panel 109, or a touch is detected in the case of a finger or a stylus pen approaching the touch panel 109, but any method can be adopted.

[0073] The system control unit 218 can detect the following operations and states with respect to the touch panel 109 .

[0074] A finger or a stylus pen that has not touched the touch panel 109 newly touches the touch panel 109 , or in other words, the start of touching (hereinafter referred to as a touchdown).

[0075] A state where a finger or a stylus pen is touching the touch panel 109 (hereinafter referred to as touch hold).

[0076] A state in which a finger or a stylus pen moves while touching the touch panel 109 (hereinafter referred to as touch movement).

[0077] The finger or stylus pen that was touching the touch panel 109 is separated (released) from the touch panel 109 , or in other words, the end of the touch (hereinafter referred to as touch stop).

[0078] A state where nothing is touching the touch panel 109 (hereinafter referred to as untouched).

[0079] When a touch is detected, touch continuation can also be detected at the same time. After the touch, as long as the touch stop is not detected, the touch continuation is usually detected continuously. In addition, in the case where the touch movement is detected, the touch continuation is detected at the same time. Even if the touch continuation is detected, the touch movement is not detected unless the touch position is moving. No touch is related to after the touch stop is detected for all fingers and styluses that are touching.

[0080] These operations and states, as well as the position coordinates of the finger or stylus that is touching the touch panel 109, are communicated to the system control unit 218 via the internal bus. The system control unit 218 determines which operation (touch operation) is performed on the touch panel 109 based on the communicated information. With regard to touch movement, the moving direction of the finger or stylus that moves on the touch panel 109 can be determined for each vertical component and horizontal component on the touch panel 109 based on the change in the position coordinates. In the case where a touch movement of more than a predetermined distance is detected, it is determined that a sliding operation is performed. An operation in which a finger quickly moves a certain distance while touching the touch panel 109 and then releases is called a flick. In other words, a flick is an operation in which a finger quickly passes over the touch panel 109 and then releases. In the case where a touch movement of more than a predetermined distance at a predetermined speed is detected and then the touch stops, it can be determined that a flick is performed (it can be determined that a flick is performed after a sliding operation). In addition, a touch operation of touching multiple points (e.g., two points) (multi-touch) and bringing these touch positions closer together is called a pinch, and a touch operation that further separates these touch positions is called a pinch-in. Pinch-to-pinch and pinch-to-close are collectively referred to as a pinch operation (or simply pinch).

[0081] Lens unit structure

[0082] Figure 3 1 is a schematic diagram illustrating an example of a structure of a dual lens unit 300 that can be mounted on the camera 100. The dual lens unit 300 can be mounted on the camera 100 instead of the single lens unit 200. Figure 3 The lens unit 300 is illustrated in a state where it is mounted on the camera 100. Note that Figure 3 The camera 100 shown in the example is used Figure 2 The same components as those described above are given the same reference numerals, and the description thereof is appropriately omitted.

[0083] The lens unit 300 is a kind of interchangeable lens that is detachably attached to the camera 100. The lens unit 300 is a dual lens that can capture left and right images with parallax. The lens unit 300 includes two optical systems, both of which have a wide viewing angle of about 180 degrees and can capture images within the front hemisphere. Specifically, each of the two optical systems of the lens unit 300 can capture an image of a subject in a field of view (viewing angle) of 180 degrees in the left and right directions (horizontal angle, posture angle, yaw angle) and 180 degrees in the up and down directions (vertical angle, pitch angle, pitch angle).

[0084] The lens unit 300 includes: a right-eye optical system 301R including a plurality of lenses, a reflector, etc.; a left-eye optical system 301L including a plurality of lenses, a reflector, etc.; and a lens system control circuit 303. The right-eye optical system 301R corresponds to an example of a first imaging optical system, and the left-eye optical system 301L corresponds to an example of a second imaging optical system. The right-eye optical system 301R and the left-eye optical system 301L include lenses 302R and 302L located on the subject side, respectively, facing the same direction, wherein the optical axes are substantially parallel to each other. The optical axis distance 307 is the interval between the optical axis of the left-eye optical system 301L and the optical axis of the right-eye optical system 301R.

[0085] The lens unit 300 according to the present embodiment is a lens for virtual reality (VR) 180 for capturing an image in a VR image format for VR 180 that enables two-lens stereoscopic viewing, or in other words, an image for VR 180. The lens for VR 180 includes a lens that can capture a range of about 180 degrees of the left-eye optical system 301L and the right-eye optical system 301R. Note that, for the lens for VR 180, it is sufficient that the left-eye optical system 301L and the right-eye optical system 301R can obtain an image that enables two-lens VR display as VR 180, and the lens for VR 180 may be a lens that can capture a wide viewing angle range of 160 degrees that is smaller than 180 degrees.

[0086] The lens used in the VR 180 can form a right image (first image) formed via the right eye optical system 301R, and a left image (second image) formed via the left eye optical system 301L having a parallax with the right image, on one or both of the image sensors of the mounted camera 100. The lens unit 300 includes an aperture drive circuit and an AF drive circuit as in the lens unit 200. Although not shown, two drive circuits (an AF drive circuit for focusing by driving a lens of the right image formed via the right eye optical system 301R and an AF drive circuit for focusing by driving a lens of the left image formed via the left eye optical system 301L) are provided. The AF drive circuit can focus by simultaneously driving the lens of the left image formed via the left eye optical system 301L and the lens of the right image formed via the right eye optical system 301R.

[0087] Furthermore, the lens unit 300 is mounted on the camera 100 via the lens mount portion 304 and the camera mount portion 305 of the camera 100. By mounting the lens unit 300 on the camera 100, the system control unit 218 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0088] In the present embodiment, a right image formed via the right-eye optical system 301R and a left image formed via the left-eye optical system 301L having parallax with the right image are imaged side by side in the imaging unit 211 of the camera 100. In other words, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on one image sensor. The imaging unit 211 converts the formed subject image (optical signal) into an analog electric signal. In this way, by using the lens unit 300, two images (as a set) having parallax can be obtained simultaneously from two positions (optical systems) corresponding to the right-eye optical system 301R and the left-eye optical system 301L. In addition, by separating the obtained images into an image for the left eye and an image for the right eye and performing VR display, the user can view a three-dimensional VR image (in other words, VR 180) in a range of about 180 degrees.

[0089] Here, the VR image is an image that can be VR displayed as described below. VR images may include omnidirectional images (360-degree images) captured by an omnidirectional camera (360-degree camera), and panoramic images having an image range (effective image range) larger than the display range that can be displayed on the display unit at one time. In addition, VR images are not limited to still images, and may include moving images or real-time images (images obtained from the camera are basically real-time). The VR image has a maximum field of view of 360 degrees in the left and right directions and an image range (effective image range) of 360 degrees in the up and down directions. In addition, the VR image may include an image with a viewing angle, or an image with an image range wider than the display range that can be displayed on the display unit at one time, when the viewing angle is less than 360 degrees in the left and right directions and less than 360 degrees in the up and down directions, having a wider range than the viewing angle that can be captured by an ordinary camera. The above-mentioned image captured by the camera 100 using the lens unit 300 is a VR image. For example, the VR image can be VR displayed by setting the display mode of the display device (display device that can display the VR image) to "VR view". A VR image having a viewing angle of 360 degrees is VR displayed, and the user changes the posture of the display device in the left-right direction (horizontal rotation direction) to seamlessly view an omnidirectional image in the left-right direction.

[0090] Here, VR display (VR view) is a display method (display mode) that displays an image of a field of view corresponding to the posture of the display device in a VR image, and can change the display range. VR display may include a "single-lens VR display (single-lens VR view)", which performs a transformation (distortion correction) to map a VR image on a virtual sphere and then displays one image. VR display may also include a "dual-lens VR display (dual-lens VR view)", which performs a transformation to map a VR image for the left eye and a VR image for the right eye on a virtual sphere, respectively, and then displays these images side by side in the left and right areas. By performing a "dual-lens VR display" using a VR image for the left eye and a VR image for the right eye with parallax, these VR images can be viewed stereoscopically.

[0091] In any case of VR display, when the user wears the display device, for example, as a head-mounted display (HMD), an image of a visual field range corresponding to the user's facial direction is displayed. For example, at a certain point in time, it is assumed that a VR image in a visual field range centered on 0 degrees in the left-right direction (a specific orientation, such as north, etc.) and 90 degrees in the up-down direction (90° from the zenith, i.e., horizontal) is displayed. If the posture of the display device is reversed from this state front to back (for example, the display surface changes from facing south to facing north), the display range of the VR image changes to an image in a visual field range centered on 180 degrees in the left-right direction (opposite orientation, such as south, etc.) and 90 degrees in the up-down direction. In other words, when the user wearing the HMD turns their face from north to south (turns back), the image displayed on the HMD also changes from an image of the north to an image of the south. Note that the VR image captured using the lens unit 300 according to the present embodiment is an image of VR180 that captures a range of approximately 180 degrees in the forward direction, and an image of a range of approximately 180 degrees in the backward direction does not exist. When the image of the VR 180 is VR-displayed in this manner and the posture of the display device is changed to the side where the image does not exist, a blank area is displayed.

[0092] By VR displaying VR images in this way, users can experience the feeling as if they are in the VR image (VR space) from a visual perspective. Note that the method for displaying VR images is not limited to the method for changing the posture of the display device. For example, the display range can be moved (scrolled) according to user operations via a touch panel or direction buttons, etc. In addition, during VR display (in the display mode "VR view"), in addition to changing the display range by changing the posture, for example, the display range can also be changed by touch movement on the touch panel, drag operation using a mouse, etc., or pressing a direction button, etc. Note that the smartphone mounted on the VR goggles (head-mounted adapter) is a type of HMD.

[0093] The structure of the image sensor inside the camera unit

[0094] Figure 4 is a diagram schematically illustrating a pixel array of an image sensor inside the image pickup unit 211 according to the present embodiment.

[0095] Figure 4 This is a diagram illustrating the pixel array of a two-dimensional CMOS sensor used as an image sensor inside the imaging unit 211 according to the present embodiment in a range corresponding to a 4×4 array of imaging pixels (a range corresponding to an 8×4 array as a focus detection pixel array).

[0096] In the present embodiment, the pixel group 400 includes pixels in a 2×2 array and is covered by a Bayer pattern color filter. In addition, in the pixel group 400, a pixel 400R having a red (R) spectral sensitivity is arranged at the upper left position, a pixel 400G having a green (G) spectral sensitivity is arranged at the upper right position and the lower left position, and a pixel 400B having a blue (B) spectral sensitivity is arranged at the lower right position. In addition, the image sensor inside the imaging unit 211 according to the present embodiment maintains a plurality of photodiodes (photoelectric conversion units) for each pixel per microlens 401 to perform focus detection using an image plane phase detection method. In the present embodiment, each pixel includes two photodiodes (photoelectric conversion elements) 402 and 403 in a 2×1 array.

[0097] Since each includes Figure 4 A plurality of pixel groups 400 of the illustrated 2×2 array of pixels (4×2 array of photodiodes) are arranged on the imaging plane, so that the image sensor inside the imaging unit 211 can obtain imaging signals and focus signals.

[0098] At each pixel having such a structure, a light beam that travels through the imaging optical system and is incident is separated into each pupil area at the microlens 401 and is imaged at the photodiodes 402 and 403. Then, a signal (A+B signal) that is the sum of the signals from the two photodiodes 402 and 403 is used as an imaging signal, and two signals (A and B image signals) read out from each photodiode 402 and 403 are used as a focus signal. Note that the imaging signal and the focus signal may be read out separately, but in consideration of the processing load, the following operation may also be performed. In other words, the imaging signal (A+B signal) and the focus signal of the photodiode 402 or 403 (for example, the A signal) may be read out, and the difference may be obtained to obtain another focus signal (for example, the B signal).

[0099] Note that in the present embodiment, each pixel has a structure in which two photodiodes 402 and 403 are provided for one microlens 401. However, the number of photodiodes is not limited to two, and two or more may be used. In addition, a plurality of pixels may be provided in which the opening position of the light receiving portion is different with respect to the microlens 401. In other words, it is sufficient to use a structure that can obtain two signals for phase detection capable of detecting the phase difference between the A image signal and the B image signal as a result.

[0100] In addition, Figure 4 In the embodiment, all pixels include a plurality of photodiodes, but such limitation is not intended. In another structure, a plurality of photodiodes may be discretely arranged inside a common pixel forming an image sensor inside the imaging unit 211. Figure 4The pixels used for focus detection illustrated as an example are pixels used for focus detection.

[0101] Live view image display example

[0102] FIG. 5A to FIG. 5C is a diagram illustrating a display example of a live view image on the display unit 108 in a case where the twin lens unit 300 is mounted on the camera 100 .

[0103] Figure 5A 1 is a diagram illustrating an example of focus detection using only the left eye optical system 301L. A left image 500L formed by the left eye optical system 301L of the lens unit 300 and a right image 500R formed by the right eye optical system 301R are displayed on the display unit 108 as a live view image 500. Since the live view image 500 is displayed inverted left and right, the image 501L of the character "L" displayed in the upper right of the screen and the image 501R of the character "R" displayed in the upper left of the screen are user interface (UI) displays to prevent misrecognition by the user. The left image AF area (focus detection area) 502L is a display frame indicating an intended position for performing auto focus (AF), and is displayed in the left image 500L.

[0104] Figure 5B 301R is a diagram illustrating an example of focus detection performed only with the right-eye optical system 301R, and a right image AF area (focus detection area) 502R is displayed in the right image 500R.

[0105] Figure 5C 301R is a diagram illustrating an example of simultaneously performing focus detection using the left-eye optical system 301L and the right-eye optical system 301R. A right-image AF area 502R is displayed on the right image 500R, and a left-image AF area 502L is displayed on the left image 500L.

[0106] The following will use Figures 6 to 24 The operation of the camera system in the case where the dual lens unit 300 is installed will be described.

[0107] Camera Processing

[0108] Figure 6 1 is a flowchart illustrating an imaging processing operation of the camera 100 according to the present embodiment. The operation of the flowchart is realized by the system control unit 218 loading a control program stored in the nonvolatile memory 220 onto the system memory 219 and executing the control program. This also applies to the operations of other flowcharts described below.

[0109] First, in step S601 , the system control unit 218 executes initialization processing such as camera settings, etc. Then, the processing proceeds to step S602 .

[0110] In step S602, the system control unit 218 determines whether the imaging mode of the camera 100 is the moving image shooting mode or the still image shooting mode. In the case of the moving image shooting mode, the process proceeds to step S603. In the case of the still image shooting mode, the process proceeds to step S604.

[0111] In step S603, the system control unit 218 executes moving image capturing processing, and the processing proceeds to step S605. Figure 7 The moving image capturing process of step S603 will be described in detail.

[0112] In step S604, the system control unit 218 executes still image shooting processing, and the processing proceeds to step S605. Figure 8 The still image capturing process of step S604 will be described in detail.

[0113] In step S605, the system control unit 218 determines whether the image capture process is stopped. If the image capture process is not stopped, the process proceeds to step S606. If the image capture process is stopped, the image capture process ends. Stopping the image capture process may include turning off the power of the camera 100 via the operation unit 230, or performing operations other than image capture (such as camera user setting processing, and playback processing for confirmation of captured images / moving images, etc.).

[0114] In step S606, the system control unit 218 determines whether the imaging mode has changed. If the imaging mode has changed, the process returns to step S601. If the imaging mode has not changed, the process returns to step S602.

[0115] If the imaging mode has not changed, the system control unit 218 continues to execute the current imaging mode. If the imaging mode has changed, the processing of the imaging mode changed to is executed after the initialization processing of step S601 is executed.

[0116] Motion picture capture processing

[0117] Now we will use Figure 7 The flowchart is used to illustrate Figure 6 The moving image capturing process of step S603 is performed.

[0118] In steps S701 to S704 , the system control unit 218 performs control related to moving image recording.

[0119] In step S701, the system control unit 218 determines whether the moving image button 106 is on. If the moving image button 106 is on, the process proceeds to step S702. If the moving image button 106 is not on, the process proceeds to step S705.

[0120] In step S702, the system control unit 218 determines whether moving image recording is currently in progress. If moving image recording is not in progress, in step S703, the system control unit 218 starts moving image recording, and the process proceeds to step S705. If moving image recording is in progress, in step S704, moving image recording is stopped, and the process proceeds to step S705.

[0121] In the present embodiment, moving image recording is started and stopped each time the moving image button 106 is pressed. However, a different method may be used to start and stop recording, such as using different buttons to start and stop recording, and using a switch for switching.

[0122] In step S705, the system control unit 218 performs focus detection processing, and the process proceeds to step S706. The focus detection processing is a process performed by the system control unit 218 to obtain defocus information and reliability information for AF. Fig. 9 To explain the details.

[0123] In step S706, the system control unit 218 determines whether the focusing is currently stopped. If the focusing is stopped, the process proceeds to step S707. If the focusing is not stopped, the process proceeds to step S708.

[0124] In step S707, the system control unit 218 performs processing in the focus stop state, and the moving image shooting processing ends. The processing of step S707 is a processing for determining whether to perform focus control again due to movement or change of the main subject, which is performed in a state where the subject is in focus. Fig.11 The flowchart is used to illustrate the details.

[0125] In step S708, the system control unit 218 performs processing in the focused state, and the moving image shooting processing ends. In step S708, focus control is performed based on the information in step S705. Fig.12 To explain the details.

[0126] Still image capture processing

[0127] Now we will use Figure 8 The flowchart is used to illustrate Figure 6 The still image capturing process of step S604 is performed.

[0128] In step S801, the system control unit 218 determines whether the first shutter switch 232 is pressed. If the first shutter switch 232 is pressed, the process proceeds to step S802. If the first shutter switch 232 is not pressed, the still image capturing process ends.

[0129] In step S802 , the system control unit 218 executes focus detection processing, and the processing proceeds to step S803 .

[0130] and Figure 7 As with step S705 of FIG. 1 , the focus detection processing of step S802 is a process executed by the system control unit 218 to obtain defocus information and reliability information for performing AF. Fig. 9 To explain the details.

[0131] In step S803, the system control unit 218 executes focus state processing, and the process proceeds to step S804. Figure 7 In step S803, the system control unit 218 performs focus control based on the information in step S802, as in step S708. Fig.12 To explain the details.

[0132] In step S804, the system control unit 218 determines whether focusing is completed. If focusing is completed, the process proceeds to step S805. If focusing is not completed, the still image capturing process ends.

[0133] In step S805, the system control unit 218 determines whether the second shutter switch 233 as an imaging start button has been pressed. If the second shutter switch 233 has been pressed, the process proceeds to step S806. If the second shutter switch 233 has not been pressed, the still image capturing process ends.

[0134] In step S806 , the system control unit 218 records the still image on the recording medium 229 , and the still image capturing processing ends.

[0135] In this embodiment, only when it is determined in step S804 that focusing is completed, can the image capturing in step S805 be started.

[0136] Note that the present embodiment is not limited to this configuration, and in other configurations, for example, recording of a still image may be started in the case where the imaging start button is pressed before focusing is completed. Furthermore, in the case where the still image shooting process is started again in step S805 without pressing the imaging start button after focusing is completed in step S804, focus detection is performed in step S802. In this case, in the focus state process in step S803, the lenses 302R and 302L may be driven, or the lenses 302R and 302L may not be driven. Furthermore, after focusing is completed in step S804, the processes from step S802 to step S803 may not be performed.

[0137] Example 1 of focus detection processing

[0138] Will use Fig. 9 The flowchart is used to illustrate the Figure 7 Step S705 and Figure 8 Example 1 of focus detection processing performed in step S802 of FIG.

[0139] like Figure 5A That way, Fig. 9 The focus detection process is illustrated in the case where the left image AF area 502L (focus detection area) is set for the left image 500L. Figure 5B That way, Fig. 9 The focus detection process in the case where the right image AF area 502R (focus detection area) is set for the right image 500R is exemplified.

[0140] First, in step S901 , the system control unit 218 sets the focus detection area to a position within the focus detection range of the imaging screen at which focus state detection is performed based on information based on the current AF area setting mode and the display frame position.

[0141] In step S902 , the system control unit 218 obtains a pair of image signals used for AF from pixels included in the set focus detection area.

[0142] In step S903 , the system control unit 218 calculates the amount of correlation between the obtained image signals, and then in step S904 , calculates the amount of correlation change from the amount of correlation calculated in step S903 .

[0143] In step S905 , the system control unit 218 calculates an off-focus amount based on the correlation change amount.

[0144] In step S906, the system control unit 218 calculates a reliability indicating the degree to which the defocus amount can be trusted. This processing is performed for each focus detection area present in the focus detection range set in step S901.

[0145] In step S907 , the system control unit 218 converts the out-of-focus amount into the defocus amount for each focus detection area, and the focus state detection processing ends.

[0146] Example 2 of focus detection processing

[0147] Will use Fig.10 The flowchart is used to illustrate the Figure 7 Step S705 and Figure 8 Example 2 of focus detection processing performed in step S802 of FIG.

[0148] like Figure 5C That way, Fig.10 The focus detection process in the case where the left image AF area 502L is set for the left image 500L and the right image AF area 502R is set for the right image 500R at the same time is illustrated.

[0149] Steps S901 to S907 and Fig. 9 Note that the processing of steps S901 to S907 is performed for the left image 500L and the right image 500R.

[0150] In step S1001 , the system control unit 218 obtains the defocus amount and the reliability thereof of the focus detection area with respect to the same object for the left image 500L and the right image 500R.

[0151] In step S1002, the system control unit 218 compares the reliabilities of the defocus amounts corresponding to the left image 500L and the right image 500R. If the two reliabilities are equal, the process proceeds to step S1003. If the two reliabilities are not equal, the process proceeds to step S1004.

[0152] In step S1003 , the system control unit 218 calculates the average value of the defocus amounts of the left image 500L and the right image 500R, and determines the average value as the final defocus amount. Then, the focus state detection process ends.

[0153] In step S1004, the system control unit 218 compares the reliability of the defocus amounts of the left image 500L and the right image 500R. If the reliability of the left image 500L is higher, the process proceeds to step S1005. Otherwise, the process proceeds to step S1006.

[0154] In step S1005 , the system control unit 218 determines the defocus amount of the left image 500L as the final defocus amount. Then, the focus state detection process ends.

[0155] In step S1006 , the system control unit 218 determines the defocus amount of the right image 500R as the final defocus amount. Then, the focus state detection process ends.

[0156] Focus stop state processing

[0157] Next, use Fig.11 The flowchart is used to illustrate the Figure 7 The focus stopped state processing is a processing for determining whether to drive the lenses 302R and 302L again when it is determined to be “focused” and the lenses 302R and 302L are stopped.

[0158] In step S1101, the system control unit 218 determines whether the calculated defocus amount is less than a predetermined multiple (e.g., 1 or more) of the depth of focus. If the calculated defocus amount is less than the predetermined multiple of the depth of focus, the process proceeds to step S1102. If the calculated defocus amount is greater than the predetermined multiple of the depth of focus, the process proceeds to step S1104.

[0159] In step S1102, the system control unit 218 determines whether the calculated reliability is greater than a predetermined value. If the calculated reliability is greater than the predetermined value, the process proceeds to step S1103. Otherwise, the process proceeds to step S1104.

[0160] In step S1103 , the system control unit 218 resets the AF restart counter, and the process proceeds to step S1105 .

[0161] In step S1104 , the system control unit 218 counts up the AF restart counter, and the process proceeds to step S1105 .

[0162] In this way, when the defocus amount is equal to or greater than a predetermined amount or the reliability of the defocus amount is less than a predetermined value, the system control unit 218 determines that the main subject being imaged has changed. Then, in step S1104, preparations are made to restart AF (re-drive the lenses 302R and 302L). On the other hand, when it is determined from the magnitude and reliability of the defocus amount that the main subject has not changed, in step S1103, the system control unit 218 does not restart AF (maintains the stopped state of the lenses 302R and 302L).

[0163] The threshold value of the defocus amount set in step S1101 is set by experience or experiment so that AF restart is performed when the main subject changes, and AF restart is not performed when the main subject does not change. As an example, 1 times the depth of focus is set so that the main subject looks blurred.

[0164] Furthermore, the threshold value of the reliability set in step S1102 is set so that, for example, if the reliability is so low that it is difficult to determine the defocus direction, it is determined that the main subject has changed and the AF is to be restarted.

[0165] This determination of steps S1101 and S1102 can be referred to as processing to determine whether the main subject has changed. Therefore, this processing can be replaced with any processing capable of making similar determinations, and the type and value of the threshold used are set according to the processing method.

[0166] In step S1105, the system control unit 218 determines whether the value of the AF restart counter is equal to or greater than a predetermined threshold. If the value is equal to or greater than the threshold, the process proceeds to step S1106. If the value is less than the threshold, the focus stop state process ends.

[0167] In step S1106 , the system control unit 218 makes settings to shift to a focused state, and the processing proceeds to step S1107 .

[0168] In step S1107 , the system control unit 218 resets the AF restart counter, and the focus stop state processing ends.

[0169] In the present embodiment, each time the defocus amount is calculated, the system control unit 218 determines whether the value of the AF restart counter added in step S1104 is equal to or larger than a predetermined threshold.

[0170] In other words, the AF restart judgment (whether the main subject has changed) is not based on a one-time judgment of the size of the defocus amount and the reliability, and the AF restart judgment is expected to be set based on the statistics of the judgment results obtained within a certain period of time. A smaller threshold for the AF restart judgment results in closer tracking of the subject. However, this leads to the possibility of inadvertent focusing in cases where other subjects cut in across the main subject, etc. Therefore, the threshold for the AF restart judgment is set in consideration of this problem. In addition, in step S1107, when the state changes from the focused state to the focused stop state again, the AF restart counter is reset.

[0171] Focus state handling

[0172] The following will use Fig.12 The flowchart is used to illustrate Figure 7 Step S708 and Figure 8 The focus state processing of step S803 is performed. The focus state processing is a processing performed to determine whether the lens is driven and the focus is stopped in a state where the focus is not stopped.

[0173] In step S1201, the system control unit 218 determines whether the magnitude of the defocus amount is equal to or less than a predetermined amount (in this example, the depth of focus), and whether the reliability of the defocus amount is greater than a predetermined value. If these conditions are met, the system control unit 218 causes the processing to enter step S1202. Otherwise, the processing enters step S1203. In the present embodiment, the threshold used in step S1201 is 1 times the depth of focus. However, the threshold can be set to a larger or smaller value as needed. The threshold for reliability set in step S1201 is set so that at least the focusing accuracy can be guaranteed.

[0174] In step S1202, the system control unit 218 sets the state to transition to the focus stop state, and the focus state processing ends. Figure 7 In the moving image shooting process of , if it is determined that the subject is in focus, the state is changed from the state of driving lenses 302R and 302L to the stop state. Figure 7 The focus stop state processing of step S707 is performed to perform a restart judgment to judge whether to drive (restart) the lenses 302R and 302L again.

[0175] In step S1203, the system control unit 218 executes processing to drive the lenses 302R and 302L. Figures 13 to 15 The process for driving lenses 302R and 302L will be described in detail.

[0176] In step S1204, the three-dimensional range UI display is performed, and the focus state processing ends. Fig.17 To explain the 3D range UI display in detail.

[0177] Note that the three-dimensional range has been described in the background technology section of the present invention, but will be described again here. The three-dimensional range is the range of the distance from the camera to the subject (subject distance) when shooting a VR image, which allows comfortable stereoscopic viewing when displaying the captured image. When the subject is closer than the three-dimensional range, the parallax of the subject increases, which causes double vision of the image. In addition, when the subject is farther away from the three-dimensional range, the parallax decreases, which causes the image to appear to be merely flat and thereby lose the stereoscopic effect. In other words, in order to achieve a good stereoscopic effect (where a close subject appears to jump out to the front and a distant subject appears to recede), it is necessary to shoot at a subject distance within the three-dimensional range.

[0178] Example 1 of lens drive processing

[0179] Next, use Fig.13 The flowchart is used to illustrate the Fig.12 Example 1 of the lens driving process performed in step S1203 of . If the current imaging distance is within the three-dimensional range, lens driving is performed within the three-dimensional range. If the current imaging distance is outside the three-dimensional range, lens driving is performed by searching and driving from the three-dimensional range end position.

[0180] In step S1301, the system control unit 218 calculates the three-dimensional range based on the interval between the optical axes of the left-eye optical system 301L and the right-eye optical system 301R (the distance between optical axes 307), the display device information viewed as a three-dimensional image, the viewing angle information at the time of image capture, and the parallax angle (which is the difference between the convergence angle and the angle in the case where the three-dimensional image jumps out or recedes at the virtual position). Then, the process proceeds to step S1302.

[0181] The interval between the optical axes is obtained by the system control unit 218 communicating with the lens system control circuit 303 of the lens unit 300 when the camera 100 is activated. In addition, the system control unit 218 obtains a setting value set from the menu button 115 of the camera 100.

[0182] The display device information viewed as a three-dimensional image is the display device viewing distance and the display device size information, and the system control unit 218 obtains the setting value set from the menu button 115 of the camera 100 .

[0183] The HMD viewing distance corresponds to the distance to the display in the virtual space. The viewing angle information is obtained by the system control unit 218 communicating with the lens system control circuit 303 of the lens unit 300. For the parallax angle, which is the difference between the convergence angle and the angle in the case where the three-dimensional image jumps out or recedes at the virtual position, the system control unit 218 obtains the setting value set from the menu button 115 of the camera 100.

[0184] Normally, the comfortable parallax range (parallax angle) of a three-dimensional image is set within ±1.0 degrees. In addition, it may be possible to set the parallax angle independently for when jumping out at a virtual position (close distance side of the camera distance) and when retreating at a virtual position (infinite distance side of the camera distance). In addition, for when retreating at a virtual position (infinite distance side of the camera distance), it may be set so that no limit is set on the three-dimensional range. The following will use Fig.16A and Fig. 16B To illustrate the relationship between display device information, viewing angle information during video recording, convergence angle and parallax angle, and lens optical axis interval and three-dimensional range.

[0185] In step S1302, the system control unit 218 calculates the current imaging distance, and the process proceeds to step S1303. The current imaging distance information is obtained and calculated by the system control unit 218 communicating imaging distance information based on the positions of the lenses 302R and 302L with the lens system control circuit 303 of the lens unit 300. Alternatively, the system control unit 218 may calculate the current imaging distance information using a triangulation method.

[0186] In step S1303, the system control unit 218 Fig. 9 The target focus position for driving the lens is calculated based on the defocus amount calculated in step S907. Then, the target imaging distance information corresponding to the target focus position is calculated based on the information of the lens unit 300, and the processing enters step S1304. Here, the lens driving method is also determined. For example, in the case where the reliability of the defocus amount is high, the lens (focus lens) is driven based on the defocus amount. In the case where the reliability of the defocus amount is low, the defocus amount is ignored, and, for example, a selection is made so that the three-dimensional range area calculated in step S1301 is searched and driven in the entire area to detect the focus position. In addition, parameters used for driving such as the lens driving speed are set.

[0187] In step S1304, the system control unit 218 determines whether the current imaging distance calculated in step S1302 is within the three-dimensional range calculated in step S1301. If these conditions are met, the process proceeds to step S1305. Otherwise, the process proceeds to step S1308.

[0188] In step S1305, the system control unit 218 determines whether the target imaging distance calculated in step S1303 is within the three-dimensional range calculated in step S1301. If this condition is satisfied, the process proceeds to step S1306. Otherwise, the process proceeds to step S1307.

[0189] In step S1306 , the system control unit 218 drives the lens to the target focus position calculated in step S1303 , and the lens driving process ends.

[0190] In step S1307, the lens is driven to a focus position corresponding to the end of the three-dimensional range calculated in step S1301 in the direction of the target focus position calculated in step S1303, and the lens driving process ends. If the direction of the target imaging distance is the close distance side, the lens is driven to a focus position corresponding to the close distance end in the three-dimensional range. If the direction of the target imaging distance is the infinite distance side, the lens is driven to a focus position corresponding to the infinite distance end in the three-dimensional range. In other words, the lens driving is limited to a focus position corresponding to the three-dimensional range. Here, in steps S1306 and S1307, if the reliability of the defocus amount is low and the defocus amount cannot be trusted, the three-dimensional range calculated in step S1301 is searched and driven in the entire area to detect the focus position.

[0191] In step S1308, if the current imaging distance is outside the three-dimensional range (such as when AF is started) and the current imaging distance corresponds to the close distance side of the three-dimensional range, the system control unit 218 advances the processing to step S1309. Otherwise (in the case of the infinite distance side of the three-dimensional range), the processing advances to step S1310.

[0192] In this case, in order to perform full-area search and drive of the three-dimensional range area, in step S1309, search and drive are performed in the infinite distance direction from the focus position corresponding to the close distance end of the three-dimensional range, and the lens drive process ends. In step S1310, search and drive are performed in the close distance direction from the focus position corresponding to the infinite distance end of the three-dimensional range, and the lens drive process ends.

[0193] Example 2 of lens drive processing

[0194] Next, use Fig.14 The flowchart is used to illustrate the Fig.12 Example 2 of the lens driving process performed in step S1203 of . If the current imaging distance is within the three-dimensional range, lens driving is performed until the focus position corresponding to within the three-dimensional range. If the current imaging distance is outside the three-dimensional range, lens driving is not performed, and it is communicated that the current imaging distance is outside the three-dimensional range.

[0195] Steps S1301 to S1307 and Fig.13 Steps S1301 to S1307 are the same.

[0196] like Fig.14 As illustrated, in step S1304 , if the current imaging distance is outside the three-dimensional range, the system control unit 218 advances the processing to step S1401 .

[0197] In step S1401, the system control unit 218 notifies the user that the current imaging distance is outside the three-dimensional range, and ends the lens driving process without performing lens driving. Figures 18 to 20 to describe the notification details.

[0198] Example 3 of lens drive processing

[0199] Next, use Fig.15 The flowchart is used to illustrate the Fig.12 Example 3 of the lens driving process performed in step S1203 of . If the target imaging distance is within the three-dimensional range, lens driving is performed until the target focus position. If the current imaging distance is outside the three-dimensional range, lens driving is not performed, and it is communicated that the current imaging distance is outside the three-dimensional range.

[0200] Steps S1301 to S1306 and steps S1308 to S1310 are Fig.13 Same as in.

[0201] like Fig.15 As illustrated, in step S1305 , if the target imaging distance is outside the three-dimensional range, the system control unit 218 advances the processing to step S1501 .

[0202] In step S1501, the system control unit 218 notifies the user that the target imaging distance is outside the three-dimensional range, and ends the lens driving process without performing lens driving. Figures 18 to 24 to describe the notification details.

[0203] 3D range calculation example

[0204] Now we will use Fig.16A and Fig. 16B The calculation of the three-dimensional range in step S1301 is described.

[0205] Fig.16A The viewing space viewed from above when a person views the display device 1601 is illustrated. Fig. 16B Example in Figure 3 The illustrated imaging space is viewed from above when the camera 100 mounted with the dual lens unit 300 is capturing an image.

[0206] exist Fig.16A , the optical axis distance 1603 (Dbh) between the eyes of a person is obtained from the distance between the left eye 1602L and the right eye 1602R. The optical axis distance 1603 (Dbh) between the eyes of a person may be an average value of the optical axis distances of typical people, or a value set from a menu may be used.

[0207] The display device viewing distance 1604 (Dd) is obtained from the distance from the display device 1601 to the left eye 1602L and the right eye 1602R. The convergence angle 1607 (θc) indicates the convergence angle when viewing a three-dimensional image at the position of the display device, and is obtained from the optical axis distance 1603 (Dbh) between the human eye and the display device viewing distance 1604 (Dd) using equation (1).

[0208] θc = 2tan -1 (Dbh / 2Dd) (1)

[0209] The limit convergence angle at which the three-dimensional image looks like jumping out on the close distance side is represented by the close distance side limit convergence angle 1608 (α), and the limit convergence angle at which the three-dimensional image looks like receding on the infinite distance side is represented by the infinite distance side limit convergence angle 1609 (β).

[0210] The near-distance side limiting convergence angle 1608 (α) and the infinite distance side limiting convergence angle 1609 (β) are obtained from the parallax angle (θp) (which is a parallax angle allowing comfortable viewing) using the convergence angle 1607 (θc) as a reference. Thus, the convergence angles have the relationship in equations (2-1) and (2-2).

[0211] α = θc + θp (2-1)

[0212] β = θc - θp (2-2)

[0213] Note that, as described above, generally, the parallax angle (θp) is about 1.0 degrees. Here, the same parallax angle (θp) is used for the short distance side and the infinite distance side. However, the parallax angle may be different for the short distance side and the infinite distance side.

[0214] The near distance side limit three-dimensional viewing distance 1605 (Dnh) and the infinite distance side limit three-dimensional viewing distance 1606 (Dih) indicate the distances from the position where the three-dimensional image corresponding to the near distance side limit convergence angle 1608 (α) and the infinite distance side limit convergence angle 1609 (β) can be seen to the left eye 1602L and the right eye 1602R. The near distance side limit three-dimensional viewing distance 1605 (Dnh) and the infinite distance side limit three-dimensional viewing distance 1606 (Dih) are obtained from equations (3-1) and (3-2).

[0215] Dnh = Dbh / (2tan(α / 2)) (3-1)

[0216] Dih = Dbh / (2tan(β / 2)) (3-2)

[0217] Fig. 16B The close distance side limit three-dimensional imaging distance 1611 (Dnr) and the infinite distance side limit three-dimensional imaging distance 1612 (Dir) are the distances from the left eye optical system 301L and the right eye optical system 301R to the imaging subject positions of the respective limits that allow comfortable viewing. The close distance side limit three-dimensional imaging distance 1611 (Dnr) and the infinite distance side limit three-dimensional imaging distance 1612 (Dir) are obtained by converting the close distance side limit three-dimensional viewing distance 1605 (Dnh) and the infinite distance side limit three-dimensional viewing distance 1606 (Dih) in the above-mentioned viewing space into the imaging space. In the conversion, here, the optical axis distance conversion and the viewing angle conversion are used, but other conversions may be used.

[0218] The optical axis distance conversion includes multiplying the optical axis distance conversion coefficient (Cb) calculated based on the optical axis distance 1603 (Dbh) between the eyes of the person and the optical axis distance 307 (Dbc) of the camera 100 by the close distance side limit 3D viewing distance 1605 (Dnh) and the infinite distance side limit 3D viewing distance 1606 (Dih). The optical axis distance conversion coefficient (Cb) is expressed by equation (4).

[0219] Cb = Dbc / Dbh (4)

[0220] The viewing angle conversion includes multiplying the viewing angle conversion coefficient (Ca) by the near-distance side limit three-dimensional viewing distance 1605 (Dnh) and the infinite distance side limit three-dimensional viewing distance 1606 (Dih). The viewing angle conversion coefficient (Ca) is calculated using the viewing angle (γ) calculated based on the display size of the display device 1601 during viewing and the display device viewing distance 1604 (Dd), and the shooting angle (δ) of the lens unit 300 during shooting. The viewing angle (γ) is calculated based on the horizontal display size 1610 (Ds) of the display device 1601 and the display device viewing distance 1604 (Dd) using formula (5).

[0221] γ = tan -1 (Ds / 2Dd) (5)

[0222] Note that, in general, for the display device viewing distance 1604 (Dd), three times the vertical display size of the display device 1601 is appropriate. Based on this relationship, the viewing angle (γ) can be obtained based on the display device viewing distance 1604 (Dd), or the vertical or horizontal display size and aspect ratio of the display device 1601.

[0223] The imaging angle of view (δ) is obtained by the system control unit 218 through communication with the lens section 300. Alternatively, the imaging angle of view (δ) is calculated from the horizontal size Dsh of the image sensor and the obtained focal length F of the lens using equation (6).

[0224] δ = 2tan -1 (Dsh / 2F) (6)

[0225] Note that here, the angles in the horizontal direction of the viewing angle of view and the camera angle of view are used, but the angles in the vertical direction or the diagonal direction may be used.

[0226] The viewing angle conversion coefficient (Ca) is calculated from the viewing angle (γ) and the imaging angle (δ) using equation (7).

[0227] Ca = tan(γ / 2) / tan(δ / 2) (7)

[0228] The short-distance side limit three-dimensional imaging distance 1611 (Dnr) and the infinite-distance side limit three-dimensional imaging distance 1612 (Dir) are calculated using equations (8-1) and (8-2), respectively.

[0229] Dnr = Dnh × Cb × Ca (8-1)

[0230] Dir = Dih × Cb × Ca (8-2)

[0231] The three-dimensional range 1613 in the imaging space is an area between the close-distance side limit three-dimensional imaging distance 1611 (Dnr) and the infinite distance side limit three-dimensional imaging distance 1612 (Dir) obtained as described above. In addition, in the case where the three-dimensional range 1613 is limited only to the close-distance side, the range can be farther than the close-distance side limit three-dimensional imaging distance 1611 (Dnr).

[0232] 3D range UI display example

[0233] Now we will use Fig.17 To illustrate Fig.12 The comparison performed in step S1204 Fig.13 An example of a UI display of a live view image of the three-dimensional range calculated in step S1301. Fig.17 Example FIG. 5A to FIG. 5C The imaging distance information 1701 is displayed on the live view image.

[0234] Current camera distance position 1702 indication Fig.13 The current imaging distance calculated in step S1302. The target imaging distance position 1703 indicates Fig.13 The short-distance side lens driving restriction area 1704 and the infinite distance side lens driving restriction area 1705 indicate the target imaging distance calculated in step S1303. Fig.13 The imaging distance area is outside the area of ​​the three-dimensional range calculated in step S1301.

[0235] Display of the imaging distance information 1701 corresponds to 17a when the three-dimensional range is not limited, corresponds to 17b when the close distance side and the infinite distance side are limited, and corresponds to 17c when only the close distance side is limited.

[0236] By displaying the three-dimensional range on the imaging distance information display in this manner, the user can easily check whether the current imaging distance and the target imaging distance are within the three-dimensional range.

[0237] Example 1 of notification outside the three-dimensional range (notification via text and icon)

[0238] Now we will use Fig.18 and FIG. 19A to FIG. 19C To illustrate Fig.14 Step S1401 and Fig.15 An example of notification for a three-dimensional out-of-range warning displayed via text and an icon on the live view image 500 in step S1501 of FIG.

[0239] Fig.18 It is a flowchart for explaining the operation of the three-dimensional out-of-range notification processing of steps S1401 and S1501.

[0240] First, in step S1801, the system control unit 218 determines text or icons to be displayed on the live view image 500. In step S1802, the system control unit 218 displays the text or icon determined in step S1801 on the live view image 500, and ends the three-dimensional out-of-range notification process.

[0241] Now we will use FIG. 19A to FIG. 19C An example of displaying text and icons is shown below. Fig.19A and Fig.19B The current imaging distance warning text 1901 or the target imaging distance warning text 1902 displayed on the live view image 500 when the current imaging distance or the target imaging distance is outside the three-dimensional range is illustrated. The current imaging distance warning text 1901 and the target imaging distance warning text 1902 are used to convey Fig.18 In addition, for the target camera distance warning text 1902, a text for conveying how much the subject needs to move to the close distance side or the infinite distance side to enter the three-dimensional range can be displayed. In a similar manner, for the current camera distance warning text 1901, a text for conveying how much the current camera distance needs to be moved to enter the three-dimensional range can be displayed.

[0242] Fig.19C The warning icon 1903 displayed on the live view image 500 when the current shooting distance or the target shooting distance is outside the three-dimensional range is illustrated. The warning icon 1903 is an icon for conveying the "outside the three-dimensional range" determined in step S1801. The warning icon 1903 can be displayed in a flashing manner to emphasize the warning. In addition, the warning icon 1903 can be a different icon for warnings for the current shooting distance and the target shooting distance.

[0243] Example 2 of notification of being out of three-dimensional range (notification via sound)

[0244] Now we will use Fig. 20 An example of notification of steps S1401 and S1501 in which a sound is emitted from the sound generator 224 to warn that the current imaging distance and the target imaging distance are outside the three-dimensional range will be described.

[0245] In step S2001, the system control unit 218 determines the content of the sound to be emitted to convey that the current imaging distance and the target imaging distance are outside the three-dimensional range. The content of the sound to be emitted may be a sound for conveying how much the subject needs to move from the target imaging distance and the three-dimensional range to the close distance side or the infinite distance side to enter the three-dimensional range. In a similar manner, the content may convey how much the current imaging distance needs to be moved to enter the three-dimensional range.

[0246] In step S2002, the system control unit 218 emits the sound content determined in step S2001 from the sound emitter 224, and ends the three-dimensional out-of-range notification processing. Note that during moving image recording, warning notification using sound is not performed due to audio recording.

[0247] Example 3 of 3D out-of-range notification (notification via AF frame display)

[0248] Now we will use Fig.21 as well as Fig.22A and Fig. 22B To illustrate Fig.15 An example of notification performed in step S1501 of FIG. 5 for warning that the target imaging distance is outside the three-dimensional range via different display of the AF frame displayed on the live view image 500.

[0249] Fig.21 This is a flowchart for explaining the three-dimensional out-of-range notification processing of step S1501.

[0250] In step S2101 , the system control unit 218 determines an AF frame to be displayed on the live view image 500 to indicate that the target imaging distance in step S1501 is outside the three-dimensional range.

[0251] In step S2102 , the system control unit 218 changes the display of the AF frame on the live view image 500 , and ends the three-dimensional out-of-range notification processing.

[0252] Now we will use Fig.22A and Fig. 22B An example of display of an AF frame to be displayed on the live view image 500 indicating that the target imaging distance of step S1501 is outside the three-dimensional range will be described.

[0253] Fig.22A The AF frame 2201 is illustrated when being displayed within the three-dimensional range when the target imaging distance is within the three-dimensional range. Fig. 22B The AF frame 2202 when out of the three-dimensional range is illustrated when being displayed in a case where the target imaging distance is out of the three-dimensional range. The AF frame 2201 when in the three-dimensional range is drawn with a solid line, and the AF frame 2202 when out of the three-dimensional range is drawn with a dotted line. A three-dimensional range out warning notification is being performed using these different displays. Note that the AF frame 2201 when in the three-dimensional range and the AF frame 2202 when out of the three-dimensional range can include AF frames of different colors, transparencies, and shapes in the warning notification.

[0254] Example 4 of notification of out-of-three-dimensional range (notification via superimposed display on subject)

[0255] Now we will use Fig.23 and Fig.24 An example of notification performed in step S1501 for warning that the target imaging distance is outside the three-dimensional range via a superimposed display on the subject in the live view image 500 will be described.

[0256] Fig.23 This is a flowchart for explaining the three-dimensional out-of-range notification processing of step S1501.

[0257] In step S2301, the system control unit 218 obtains focus detection area information “outside the three-dimensional range” based on the three-dimensional range inside and outside determination of each focus detection area in step S1305.

[0258] In step S2302 , the system control unit 218 superimposes the focus detection area outside the three-dimensional range obtained in step S2301 on the live view image 500 (live view screen), and ends the three-dimensional out-range notification processing.

[0259] Now we will use Fig.24 An example of superimposed display on the subject in the live view image 500 indicating that the target imaging distance performed in step S1501 is outside the three-dimensional range will be described.

[0260] Fig.24 The three-dimensional range out-of-range superimposed display 2401 is a superimposed display of the focus detection area outside the three-dimensional range displayed in step S2302. The three-dimensional range out-of-range superimposed display 2401 is superimposed in a semi-transparent color. Note that patterns such as spots or oblique lines can be used in the superimposed display. In addition, the color, transparency or pattern can be made different in the display depending on how far the target camera distance is from the three-dimensional range or in the direction toward the close distance side or infinite distance side of the three-dimensional range. The three-dimensional range out-of-range superimposed display 2401 can be displayed only in the left lens image 501L or the right lens image 501R, or can be displayed in both. Alternatively, the focus detection area within the three-dimensional range can be superimposed in the display.

[0261] Note that in the above-described embodiment, a phase detection method is used in a focus detection unit for detecting the defocus amount. However, for a focus detection unit for detecting the focus state of a subject image based on the difference (contrast) of high-frequency components included in an imaging signal obtained by an image sensor, a so-called contrast detection method may be used.

[0262] Other embodiments

[0263] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0264] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A camera device, comprising: an imaging device configured to photoelectrically convert a first optical image formed by a first imaging optical system to output a first image, and further configured to photoelectrically convert a second optical image formed by a second imaging optical system arranged in parallel with the first imaging optical system to output a second image; a calculation unit configured to calculate a three-dimensional range as a range of a subject distance of the imaging device in which a three-dimensional image is made to look three-dimensional when the first image and the second image are displayed on a display device; as well as A control unit configured to control the first imaging optical system and the second imaging optical system to restrict a range of subject distances for performing automatic focusing of the first imaging optical system and the second imaging optical system to within the three-dimensional range and perform automatic focusing.

2. The imaging device according to claim 1, further comprising: a first focus detection unit configured to perform focus detection based on a signal obtained by photoelectric conversion of light incident via the first imaging optical system; as well as A second focus detection unit is configured to perform focus detection based on a signal obtained by photoelectric conversion of light incident via the second imaging optical system.

3. The imaging device according to claim 1, wherein: The calculation unit calculates the three-dimensional range based on the following items: the size of the display device displaying the first image and the second image; the viewing distance; the parallax angle which is the difference between the convergence angle and the angle when the three-dimensional image jumps out and when the three-dimensional image recedes at a virtual position; the interval between the optical axis of the first camera optical system and the optical axis of the second camera optical system; and the viewing angle information during camera shooting.

4. The imaging device according to claim 3, further comprising: An input unit is configured to input the size of the display device and the viewing distance.

5. The imaging device according to claim 3, wherein: The size of the display device and the viewing distance are the size of the display and the viewing distance in the virtual space of the head mounted display.

6. The imaging device according to claim 3, further comprising: An input unit is configured to input the parallax angle.

7. The imaging device according to claim 3, wherein: The interval between the optical axis of the first imaging optical system and the optical axis of the second imaging optical system is obtained from the first imaging optical system and the second imaging optical system, or is input by a user.

8. The imaging device according to claim 3, wherein: The angle of view information at the time of imaging is obtained from the first imaging optical system and the second imaging optical system, or is calculated based on information including the size of an image sensor and the focal lengths of the first imaging optical system and the second imaging optical system.

9. The imaging device according to claim 1, wherein: The display device displays the three-dimensional range on a display screen of a live view image obtained via the imaging device.

10. The imaging device according to claim 1, wherein The control unit does not limit the three-dimensional range on the infinite distance side of the object distance.

11. The imaging device according to claim 2, wherein: The first focus detection unit and the second focus detection unit detect a first defocus amount of the first imaging optical system and a second defocus amount of the second imaging optical system based on signals of pixels including a plurality of photoelectric conversion elements, wherein the plurality of photoelectric conversion elements are used to receive light beams passing through different pupil areas of the first imaging optical system and the second imaging optical system.

12. The imaging device according to claim 11, wherein: The control unit obtains a defocus amount used in focusing of the first imaging optical system and the second imaging optical system based on the first defocus amount and / or the second defocus amount.

13. The imaging device according to claim 12, wherein: The control unit sets a defocus amount having the highest reliability among the first defocus amount and the second defocus amount as a defocus amount used in the focusing.

14. The imaging device according to claim 12, wherein: The control unit calculates an average value of the first defocus amount and the second defocus amount, and sets the average value as the defocus amount used in the focusing.

15. The imaging device according to claim 1, wherein: The calculation unit obtains information of a currently photographed subject distance from the first imaging optical system and the second imaging optical system, or calculates information of a currently photographed subject distance via triangulation.

16. The imaging device according to claim 12, wherein: The control unit focuses the first imaging optical system and the second imaging optical system using the defocus amount used in the focusing, if a target imaging distance based on the defocus amount used in the focusing is within the three-dimensional range.

17. The imaging device according to claim 12, wherein: When a target imaging distance based on a defocus amount used in the focusing is outside the three-dimensional range, the control unit moves the focus lenses of the first imaging optical system and the second imaging optical system to positions at ends of the imaging distance of the three-dimensional range.

18. The imaging device according to claim 12, wherein: When a target photographing distance based on the defocus amount used during the focusing is outside the three-dimensional range and a current photographed subject distance is outside the three-dimensional range, the control unit moves the focusing lens to an end of the photographing distance of the three-dimensional range before searching for and driving the focusing lenses of the first photographing optical system and the second photographing optical system at a predetermined speed.

19. The imaging device according to claim 1, wherein: When the distance of the currently photographed object is outside the three-dimensional range, the control unit does not drive the focus lenses of the first imaging optical system and the second imaging optical system.

20. The imaging device according to claim 1, further comprising: A notification unit is configured to notify that the distance of the currently photographed object is outside the three-dimensional range.

21. The imaging device according to claim 12, further comprising: A notification unit configured to notify that a target imaging distance based on the defocus amount used in the focusing is outside the three-dimensional range.

22. The imaging device according to claim 20, wherein: The notification unit displays a warning on the display device via text or an icon.

23. The imaging device according to claim 20, wherein: The notification unit issues a warning via sound.

24. The imaging device according to claim 20, wherein: The notification unit issues a warning via a different AF frame display.

25. The imaging device according to claim 20, wherein: The notification unit displays the subject in a superimposed manner on the live view screen.

26. A control method for an imaging device, the imaging device comprising an imaging device configured to photoelectrically convert a first optical image formed by a first imaging optical system to output a first image, and further configured to photoelectrically convert a second optical image formed by a second imaging optical system arranged in parallel with the first imaging optical system to output a second image, the control method comprising: calculating a three-dimensional range as a range of a subject distance of the imaging device in which a three-dimensional image is made to look three-dimensional when the first image and the second image are displayed on a display device; as well as The first imaging optical system and the second imaging optical system are controlled so that the range of the subject distance for which the first imaging optical system and the second imaging optical system perform automatic focusing is limited to within the three-dimensional range and automatic focusing is performed.

27. A computer program product, configured to cause a computer to execute the control method according to claim 26.

28. A computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 26.

Images