Imaging device, control method for imaging device, and program

Abstract

To provide an imaging device that reduces the deviation of the display position relative to a subject due to parallax and allows a user to operate in the area set by the user without feeling uncomfortable.SOLUTION: An imaging device can display images in a live view, and includes an imaging element that acquires a first live view image captured through a first optical system and a second live view image captured through a second optical system in the same orientation as that of the first optical system and having parallax with respect to the first live view image, and a display control unit that sets a set area in the first live view image and the second live view image. The display control unit sets the set area of the second live view image according to the amount of positional deviation between the first live view image and the second live view image.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an imaging device, a control method for an imaging device, and a program. [Background technology]

[0002] In recent years, a twin-lens VR camera has been known that uses two optical systems facing the same direction to acquire images with a wide viewing angle with parallax, and maps and displays the images on a virtual sphere to display a three-dimensional virtual reality (VR) image. When two parallax images are displayed as live view images, differences in the live view images occur due to the parallax. In particular, the parallax becomes large during close-up shooting, and the differences in the live view images also become large. Therefore, when displaying setting areas at the same coordinates of each live view image, there is a possibility that a defect (displacement of the display position relative to the subject) will occur in which the objects or areas displayed in the two setting areas are different. In addition, since the parallax changes depending on the position in the optical axis direction, positional deviation occurs depending on the focus state of the subject in the setting area.

[0003] In such cases, when the setting area is an area set by the user, such as a focus area to be focused on by autofocus or manual focus, or an enlarged area to be enlarged, it is difficult to know whether the user's intention is correctly reflected. For example, when the setting area is displayed simultaneously on two parallax images, the target (subject) is different in each parallax image, which confuses the user. Also, when the setting area is displayed on one of the two parallax images, a problem occurs in which the target (subject) changes when switching the selection between the two parallax images.

[0004] Patent Document 1 discloses a configuration that, when there is parallax due to the pixel aperture position, corrects the parallax using a pixel aperture position ratio, and is capable of estimating the subject position with high accuracy.

[0005] Patent Document 2 discloses parallax correction based on the amount of displacement of a specific subject detected when synthesizing parallax images of a two-eye VR system, and the timing of updating the correction, in order to view a stereoscopic image without discomfort. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2013-175898 A [Patent Document 2] JP 2011-171813 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the configuration of Patent Document 1, parallax correction is performed due to the difference in pixel aperture position in a normal single lens, but parallax that occurs due to differences in the optical systems of a twin-lens VR camera or the like cannot be corrected.

[0008] Furthermore, Patent Document 2 does not disclose parallax correction or area display when a specific subject is not detected.

[0009] An object of the present invention is to provide an imaging device that reduces the deviation of the display position relative to the subject caused by parallax and allows the user to perform operations in a region set by the user without feeling uncomfortable. [Means for solving the problem]

[0010] An imaging device according to one aspect of the present invention is an imaging device capable of displaying images in live view, and includes an imaging element that acquires a first live view image captured via a first optical system and a second live view image captured via a second optical system oriented in the same direction as the first optical system and having parallax relative to the first live view image, and a display control unit that sets setting areas in the first live view image and the second live view image, and the display control unit sets the setting area of ​​the second live view image in accordance with an amount of positional shift between the first live view image and the second live view image. Effect of the Invention

[0011] According to the present invention, it is possible to provide an imaging device that reduces the deviation of the display position relative to the subject caused by parallax and allows the user to perform operations in a region set by the user without feeling uncomfortable. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a camera system. [Diagram 2] FIG. 2 is a diagram showing a pixel array of an imaging element. [Diagram 3] FIG. 13 is a diagram illustrating another example of the configuration of the camera system. [Figure 4] 6 is a flowchart showing control of the display position of the focus frame in the first embodiment. [Diagram 5] FIG. 2 is a diagram illustrating a schematic diagram of an optical path of the twin-eye VR lens of the first embodiment. [Figure 6] 6 is a flowchart showing a process for acquiring a display position deviation amount according to the first embodiment. [Figure 7] FIG. 11 is a diagram showing a schematic diagram of an optical path of a twin-eye VR lens according to a second embodiment. [Figure 8] 10 is a flowchart showing a process for acquiring a display position deviation amount according to the second embodiment. [Figure 9] 13A and 13B are diagrams for explaining the angles of view of a left eye image and a right eye image and a defocus map according to the third embodiment. [Figure 10]13 is a flowchart showing a process for acquiring a display position deviation amount according to the third embodiment. [Figure 11] 13 is a flowchart showing defocus matching according to a third embodiment. [Figure 12] 13A and 13B are diagrams for explaining the angles of view and search areas of a left-eye image and a right-eye image according to a fourth embodiment. [Figure 13] 13 is a flowchart showing a process for acquiring a display position deviation amount according to the fourth embodiment. [Figure 14] 13 is a flowchart showing image matching according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to refer to the same components, and duplicated descriptions will be omitted. [First embodiment] 1 is a diagram showing an example of the configuration of a camera system. The camera system has a lens unit 101 and a camera body (imaging device) 102. A lens control unit 117 that controls the operation of the entire lens unit 101 and a camera control unit 141 that controls the operation of the entire camera system 1 including the lens unit 101 can communicate with each other via a terminal provided on the lens mount.

[0014] First, the configuration and operation of the lens unit 101 will be described. The lens unit 101 is a single lens, and is an interchangeable lens that can be attached to and detached from the camera body 102. The photographing lens 111 (optical system) includes a fixed lens 112, an aperture 113, and a focus lens 114. The aperture control unit 115 drives the aperture 113 to adjust the aperture diameter of the aperture 113 and adjust the amount of light during photographing. The focus lens 114 is a lens for focus adjustment, and is simply shown as one lens in FIG. 1, but is usually composed of multiple lenses. The focus control unit 116 receives the lens drive amount determined by the camera control unit 141 via the lens control unit 117, and adjusts the focus by moving the focus lens 114. AF control is realized by the movement control of the focus lens 114 by the focus control unit 116. The aperture control unit 115 and the focus control unit 116 are controlled by the lens control unit 117.

[0015] Next, an outline of the configuration and operation of the camera body 102 will be described. The camera body 102 is configured to be capable of displaying images in a live view. The image sensor 121 is configured, for example, with a CCD or CMOS sensor, and includes a plurality of photoelectric conversion elements that photoelectrically convert a subject image (optical image) into an electric charge. The subject image formed by a light beam incident via the photographing lens 111 is formed on the light receiving surface of the image sensor 121, and is converted into an electric charge in each photoelectric conversion element according to the amount of light, and is accumulated. The electric charge accumulated in each photoelectric conversion element is sequentially read out from the image sensor 121 as a voltage signal according to the electric charge, by a drive pulse output by a timing generator 122.

[0016] Here, the configuration of the image sensor 121 will be described with reference to Fig. 2. Fig. 2 is a diagram showing a pixel array 200 of the image sensor 121. The pixel array 200 is configured by arranging unit pixels 201 in a row and column direction. The unit pixels 201 include a microlens 202 and photodiodes (PD) 203a and 203b. The PDs 203a and 203b are arranged under the microlens 202. With this configuration, the PDs 203a and 203b photoelectrically convert light that has passed through different pupil regions of the photographing lens 111, and can capture images of the same subject with a phase difference. An imaging signal can be obtained by adding up and reading out the charges accumulated in the PDs 203a and 203b for each unit pixel 201. A focus detection signal can be obtained by independently reading out the charges accumulated in the PDs 203a and 203b. Note that a signal corresponding to the charges accumulated in one of the PDs 203a and 203b and the added signal can be read out and the difference taken to obtain a signal corresponding to the charges accumulated in the other of the PDs 203a and 203b.

[0017] 1, the CDS / AGC / AD converter 123 performs correlated double sampling to remove reset noise, adjusts the sensor gain, and digitizes the voltage signals (imaging signals and focus detection signals) read out from the image sensor 121. The CDS / AGC / AD converter 123 also outputs the processed imaging signals to an imaging signal processor 124, and outputs the processed focus detection signals to a focus detection signal processor 125.

[0018] The focus detection signal processor 125 sets and arranges the focus detection area for performing focus detection. In this embodiment, focus detection is performed by extracting focus detection signals output from pixels included in a predetermined area from focus detection signals output from the CDS / AGC / AD converter 123. Specifically, first, for each focus detection area, focus detection signals corresponding to PD203a are collected to generate an A image, and focus detection signals corresponding to PD203b are collected to generate an B image. Next, the signals are converted into a pair of image signals having a phase difference, and a correlation calculation is performed to obtain the defocus amount and reliability information (the degree of coincidence of two images, the degree of steepness of two images). In this embodiment, as described above, focus detection is performed by a so-called image plane phase difference method, but the method of focus detection is not limited to this, and other methods may be used as long as the focus state can be detected numerically for each focus detection area.

[0019] The imaging signal processing unit 124 performs predetermined image processing such as gamma conversion processing, white balance processing, and various correction processing on the imaging signal output from the CDS / AGC / AD converter 123. The imaging signal processing unit 124 also stores the processed image data in the SDRAM 136 via the bus 131. The image data stored in the SDRAM 136 is read out by the display control unit 132 via the bus 131 and displayed on the display unit 133. In an operation mode for recording, the image data stored in the SDRAM 136 is also recorded in the recording medium 135 by the recording medium control unit 134.

[0020] The ROM 137 stores a control program executed by the camera control unit 141 and various data necessary for control. The flash ROM 138 stores various setting information related to the operation of the camera body 102, such as user setting information.

[0021] A shooting preparation switch (SW1) 139 is turned on by, for example, half-pressing a shutter release button (not shown) and instructs the start of shooting preparation operations such as AF and AE (hereinafter referred to as "SW1"). After SW1 is turned on, a shooting switch (SW2) 140 is turned on by, for example, fully pressing a shutter release button (not shown) and instructs shooting.

[0022] The camera control unit 141 determines the lens driving amount based on the defocus amount and reliability information output from the focus detection signal processing unit 125. The lens driving amount is transmitted to the focus control unit 116 via the lens control unit 117. The focus control unit 116 realizes AF by moving the focus lens 114. FIG. 3 is a diagram showing another example of the configuration of the camera system. The camera system has a lens unit 300 and a camera body 102. The lens unit 300 has two optical systems, is a twin lens for VR image shooting capable of shooting with parallax between a left image and a right image, and is an interchangeable lens that can be attached to and detached from the camera body 102. Note that the same reference symbols are used for the same configurations of the camera body 102 shown in FIG. 3 as those described in FIG. 1, and duplicated descriptions are omitted.

[0023] The lens unit 300 has a right-eye optical system 301R including a plurality of lenses and a reflecting mirror, a left-eye optical system 301L including a plurality of lenses and a reflecting mirror, and a lens system control circuit 303. The right-eye optical system 301R corresponds to an example of a first optical system, and the left-eye optical system 301L corresponds to an example of a second optical system. The lenses 302R and 302L located on the subject side of the right-eye optical system 301R and the left-eye optical system 301L face in the same direction (the optical axes of the lenses 302R and 302L are approximately parallel).

[0024] The lens unit 300 is a lens for VR180 for capturing images for VR180, which is a VR image format that allows two-eye stereoscopic viewing. The right-eye optical system 301R and the left-eye optical system 301L each include a fisheye lens that can capture a range of approximately 180 degrees. It is sufficient for the lens unit 300 to be capable of capturing a wide viewing angle range of approximately 160 degrees.

[0025] The lens unit 300 can form a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L having parallax from the right image, on one or two image pickup elements of the attached camera. The lens unit 300 includes a focus ring for adjusting the focus of the right image and a focus ring for adjusting the focus of the left image. The lens unit 300 may also include a focus ring for simultaneously adjusting the focus of the right image and the left image, and a focus ring for adjusting the focus of one of the right image and the left image.

[0026] The lens unit 300 is attached to the camera body 102 via the lens mount portion 304 and the camera mount portion 305 of the camera body 102. By attaching the lens unit 300 to the camera body 102, the camera control unit 141 and the lens system control circuit 303 of the lens unit 300 are electrically connected to each other via the communication terminal 307 of the camera body 102 and the communication terminal 306 of the lens unit 300.

[0027] In this embodiment, the right image and the left image are formed side by side on the light receiving surface of the image sensor 121. That is, two subject images formed by the right eye optical system 301R and the left eye optical system 301L are formed on the light receiving surface of one image sensor 121. By using the lens unit 300, two images with parallax can be simultaneously acquired (as a set) from two locations (optical systems) of the right eye optical system 301R and the left eye optical system 301L. In addition, the two acquired parallax images are divided into an image for the left eye (left eye image) and an image for the right eye (right eye image) and displayed in VR, allowing the user to view a stereoscopic VR image.

[0028] Next, with reference to Fig. 4, a display position control of a focus frame, which is an example of a setting area, performed by the display control unit 132 will be described. Fig. 4 is a flowchart showing the display position control of the focus frame. The focus frame may be an area for focusing by autofocus, or an area for focusing by manual focus. Note that in each embodiment, a focus frame will be described, but as described above, it is also applicable to a case where a setting area is displayed at the same coordinates as an enlarged area, etc.

[0029] Control of the display position of the focus frame involves determining the coordinates (left eye image (xL, yL), right eye image (xR, yR)) of the focus frame to be displayed on the left eye image and right eye image. The focus frame clearly indicates the target to be focused on by superimposing the area to be focused on on the live view image. There are two types of focus frames: subject detection frames, which use the subject detection results to display a focus frame at the subject detection position, and fixed area frames, which display a fixed position of any selected position.

[0030] In step S401, it is determined whether or not a fixed area frame is displayed. If it is determined that a fixed area frame is displayed, the process of step S402 is executed, and if it is determined that a fixed area frame is not displayed, that is, if it is determined that a subject detection frame is displayed, the process of step S405 is executed.

[0031] In step S402, when one of the left eye image and the right eye image is set as a reference image, the amount of display position shift W in the parallax direction (x direction) with respect to the other image is acquired. Here, a description will be given of a process for acquiring the display position shift amount W. Note that in this embodiment, the left eye image is taken as a reference image, and the display position shift amount W with respect to the right eye image (comparison image) is acquired.

[0032] Fig. 5 is a diagram showing a schematic optical path of the lens unit 300, which is a twin-lens VR lens for capturing VR images in Fig. 3. The image of the subject 501 passing through the left-eye optical system 301L is 502L, and the image passing through the right-eye optical system 301R is 502R. The deviation amounts from the respective optical axes (xL=0, xR=0) are xL and xR. Here, the parallax (|xL-xR|) between the left-eye image and the right-eye image can be calculated using the shooting distance a, the focal length f, and the distance k between the optical axes of the left-eye optical system 301L and the right-eye optical system 301R, using the following formula (1).

[0033] |xL-xR|=(k×f) / a (1) FIG. 6 is a flowchart showing the process of acquiring the display position deviation amount W.

[0034] In step S601, lens information (shooting distance a, focal length f, and inter-optical axis distance k) required to calculate parallax X is acquired. In step S602, the parallax X is calculated using equation (1).

[0035] In FIG. 5, when a focus frame is displayed at position (-xL) of image 502L, the parallax X (|xL-xR|) is the amount of display position shift W in the right eye image (comparison image) when the left eye image is the reference image.

[0036] In step S603, the parallax X is set as the display position shift amount W.

[0037] Returning to Fig. 4, in step S403, it is determined whether the display position deviation amount W is greater than a predetermined threshold Th. If it is determined that the display position deviation amount W is greater than the predetermined threshold Th, the process of step S404 is executed, and if it is determined that the display position deviation amount W is smaller than the predetermined threshold Th, the process of step S405 is executed. Note that, when the display position deviation amount W is equal to the predetermined threshold Th, it is possible to arbitrarily set which step to execute.

[0038] The predetermined threshold Th may be determined as a ratio to the width of the screen to be displayed, or may be determined according to the size of the focus frame to be displayed. For example, although it depends on the screen size and the focus frame size, if the predetermined threshold Th is set to about 1 / 4 of the focus frame size, it is possible to correct only when a clear display position shift occurs. It may also be changed according to the display position of the focus frame relative to the screen. In particular, since the lens for VR180 is a fisheye lens, it is preferable to set the predetermined threshold Th larger than the center of the screen, since the image is distorted at the edge of the screen. Also, if the predetermined threshold Th is set to zero, the display position shift can always be corrected, enabling a more natural display.

[0039] In step S404, the focus frame display position (coordinates) of the right eye image, which is the comparison image, is corrected using the display position shift amount W calculated in step S402. Specifically, xR is corrected to XR+W.

[0040] In step S405, focus frames are displayed in the left eye image and the right eye image. The coordinates of the left eye image and the right eye image at this time are (xL, yL) and (xR, yR), respectively.

[0041] As described above, by acquiring the display position shift amount W from the parallax calculated from the lens information, it is possible to reduce the shift of the display position of the focus frame relative to the subject. [Second embodiment] In this embodiment, a process for acquiring the display position shift amount W, which differs from that in the first embodiment, will be described.

[0042] 7 is a diagram showing a schematic optical path of the lens unit 300, which is a twin-lens VR lens for VR image shooting in FIG. 3, and shows a state in which the subject 701 is in front focus (front in the optical axis direction). The image of the subject 701 passing through the left-eye optical system 301L is 702L, and the image of the subject 701 passing through the right-eye optical system 301R is 702R. In FIG. 7, the subject is closer than in FIG. 5, so the display position shift amount W is larger than the parallax X obtained from the lens information (shooting distance a, focal length f, and inter-optical axis distance k) described in FIG. 5. The display position shift amount W can be obtained using the lens information and the defocus amount d at the focus frame position using the following formula (2).

[0043] W=(k×f) / (a+d) (2) FIG. 8 is a flowchart showing the process of acquiring the display position deviation amount W.

[0044] In step S801, lens information required to calculate the parallax X is obtained. In step S802, the defocus amount at the focus frame position of the left eye image, which is the reference image, is acquired.

[0045] In step S803, the subject distance (=a+d) is calculated using the shooting distance acquired in step S801 and the defocus amount acquired in step S802.

[0046] In step S804, the display position shift amount W is obtained and set using the lens information obtained in step S801 and the subject distance calculated in step S803.

[0047] As described above, by obtaining the display position shift amount W using the parallax and defocus amount calculated from the lens information, the display shift can be corrected with high accuracy even when the object within the focus frame is not in focus. [Third embodiment] In this embodiment, a process for acquiring the display position shift amount W, which differs from the first and second embodiments, will be described.

[0048] In this embodiment, the defocus map of the left eye image and the right eye image is used, and the display position shift amount W of the focus frame with respect to the subject is calculated by matching the pattern. The defocus map is obtained by calculating and mapping the defocus amount for each area divided by a predetermined number of divisions. In this embodiment, an example will be described in which a user looks at the left eye image and specifies the focus frame position, and the reference image is the left eye image, and a matching position is searched for from the right eye image, which is a comparison image. First, the parallax effect on the defocus matching accuracy will be described. FIG. 9 is a diagram for explaining the angle of view and defocus map of the left eye image and the right eye image. FIG. 9(a) shows the angle of view 901 (solid line) of the left eye image and the angle of view 902 (dashed line) of the right eye image with respect to subjects 903 and 904. The shift between the angles of view 901 and 902 becomes the parallax X indicated by the double arrow in the figure. FIG. 9(b) is a diagram in which a defocus map 906L and a focus frame 907L are superimposed on a left eye image 905L. Fig. 9(c) is a diagram in which a defocus map 906R and a focus frame 907R are superimposed on a right-eye image 905R. Fig. 9(d) is a diagram in which a defocus map 916R and a focus frame 917R shifted by parallax X are superimposed on a right-eye image 905R. Also, the bars on the right side of Fig. 9(b) to Fig. 9(d) show patterns indicating the near to infinity range of the defocus map. Here, IDs (L0,0 to Ln-1,n-1, R0,0 to Rn-1,n-1, R'0,0 to R'n-1,n-1) are assigned to identify the division positions of the defocus map.

[0049] In Fig. 9(b) and Fig. 9(c), defocus maps 906L and 906R are arranged at the same coordinates with each optical axis as the center. In Fig. 9(d), defocus maps are arranged by shifting by parallax X. For example, if the w x w-th coordinates are Lw,w = (xLw, yLw) and Rm,m = (xRw, yRw), then xLw = xRw and yLw = yRw. In addition, the center coordinates of R'm,m are (xRw + X, yRw). In this embodiment, since the left eye image is the reference image, the parallax X takes a positive value and is shifted to the right in the figure.

[0050] As described above, in Fig. 9(b) and Fig. 9(c), the defocus maps 906L and 906R are mapped at the same coordinates with the optical axis as the center, so that the defocus maps are mapped at positions shifted by the parallax X for the subjects 903 and 904. Therefore, the defocus map 906L of the left eye image 905L and the defocus map 906R of the right eye image 905R do not have the same defocus map pattern, and the matching accuracy is deteriorated. As shown in Fig. 9(d), the defocus map 906R of the right eye image 905R, which is the comparison image, is shifted to the right by the parallax X, so that the accuracy of the pattern match of the defocus map 906L of the left eye image 905L, which is the reference image, can be improved.

[0051] FIG. 10 is a flowchart showing a calculation process of the display position deviation amount W.

[0052] In step S1001, lens information required to calculate the parallax X is obtained. In step S1002, the parallax X is calculated using equation (1).

[0053] In step S1003, as described in FIG. 9D, the defocus map coordinates of the comparison image (right eye image) are shifted by the parallax X calculated in step S1002 to calculate the defocus map.

[0054] In step S1004, the ID and defocus amount of the matching reference area are extracted from the reference image (left eye image). In this embodiment, the defocus map area including the focus frame 907L is extracted as the matching reference area 908 (thick dashed line), and pattern matching with the defocus map of the comparison image is performed. For example, in FIG. 9B, the IDs and defocus amounts of Lw,w, Lw+1,w, Lw+2,w, Lw+n,w, Lw+n,w+1, and Lw+n,w+2 are extracted. Here, as an example, the defocus map area including the focus frame is extracted as the matching reference area, but only the area of ​​the defocus map ID with the widest area where the focus frame overlaps may be used as the matching reference area.

[0055] In step S1005, defocus matching is performed by comparing the defocus amount of the matching reference region extracted in step S1004 with the defocus amount of the defocus map of the comparison image (right eye image).

[0056] FIG. 11 is a flowchart showing defocus matching. In defocus matching, the sum Δi of the absolute value of the difference in the defocus amount from the reference image is calculated while incrementing the ID of the comparison image in the matching target area, and the ID when the sum Δi is the smallest is set as the matching ID. In addition, in this embodiment, the deviation of the left eye optical system and the right eye optical system in the vertical direction of the screen is assumed to be approximately zero due to adjustment during manufacturing, calibration, etc., and the imaging position deviation due to the subject on the close side where the display deviation becomes large is corrected. In this embodiment, as shown in FIG. 9(d), the matching target area 909 (thick dotted line) is from the ID (R'w,w, R'w+n,w) which is the same as the ID at the left end of the matching reference area to the right end of the screen (R'n-1,w, R'n-1,w+n). If the above-mentioned assumption does not apply or if it is desired to correct the imaging position deviation on the infinity side as well, the matching target area may be widened.

[0057] In step S1101, a counter i is set to zero.

[0058] In step S1102, a sum Δi is calculated by adding the absolute value of the difference between the defocus amount of the ID of the matching reference area extracted in step S1004 and the defocus amount of the target ID of the comparison image.

[0059] In step S1103, it is determined whether the counter i is at the initial value or the sum Δi calculated in step S1102 is the minimum. If it is determined that the counter i is at the initial value (i=0) or the sum Δi is the minimum, the process of step S1104 is executed. If it is determined that the counter i is not at the initial value and the sum Δi is not the minimum, the process of step S1105 is executed.

[0060] In step S1104, the minimum value of the sum Δi and the matching ID are updated.

[0061] In step S1105, the counter i is incremented.

[0062] In step S1106, it is determined whether the calculation has been completed up to the right edge of the screen (i>nw). If it is determined that the calculation has been completed up to the right edge of the screen, this flow is ended, and if it is determined that the calculation has not been completed up to the right edge of the screen, the process of step S1102 is executed.

[0063] Returning to FIG. 10, in step S1006, the deviation ΔX is calculated from the coordinates of the matching ID determined in step S1104 and the coordinates of the leading ID of the matching reference area.

[0064] In step S1007, the shift amount of the defocus map of the comparison image (right eye image), that is, the sum of the parallax X calculated in step S1002 and the shift ΔX calculated in step S1006, is set as the display position shift amount W.

[0065] As described above, the positions of the defocus maps of the reference image (left eye image) and the comparison image (right eye image) relative to the subject are aligned using the parallax calculated from the lens information, and a position that matches the pattern of the defocus map area extracted from the reference image is searched for. This makes it possible to accurately correct the display misalignment.

[0066] In this embodiment, the processing time can be reduced by limiting the pattern matching target area to only the area where a deviation occurs in principle with the parallax X as the base point. Also, in the case of a general subject, there is a possibility that erroneous matching will occur if the entire comparison image is compared. For example, the subject 904 in FIG. 9 has the same defocus map pattern as the subject 903 with the focus frame set, and there is a possibility that erroneous matching will occur if the matching target area is expanded in the vertical direction of the screen. In this case, it is necessary to determine the position relative to the reference area and the reliability information of the correlation calculation. [Fourth embodiment] In this embodiment, a process for acquiring the display position shift amount W, which is different from the first to third embodiments, is described. In this embodiment, the display position shift amount W of the focus frame with respect to the subject is calculated by image matching between a left eye image (reference image) and a right eye image (comparison image).

[0067] First, optimization of the search area for image matching will be described. FIG. 12 is a diagram for explaining the angle of view of the left eye image and the right eye image and the search area. FIG. 12(a) is the same as FIG. 9(a), and shows the angle of view 901 (solid line) of the left eye image and the angle of view 902 (dashed line) of the right eye image with respect to the objects 903 and 904, and the parallax X which is the deviation of the angles of view 901 and 902. FIG. 12(b) and FIG. 12(c) are diagrams of the left eye image 905L and the right eye image 905R which are the same as FIG. 9(b) and FIG. 9(c), respectively, but are shown with the optical axes aligned. In FIG. 12(b), the image within the focus frame 1201 is set as the matching reference image, and its upper left pixel is set as xw [pix], its horizontal size is set as Wh [pix], and its vertical size is set as Wv [pix]. In Fig. 12(c), the top left pixel sw[pix] of the initial image matching search range 1202 is the same as the top left pixel xw[pix] of the matching reference image. The horizontal size is set to Wh+X[pix] because it is expected that there will be a shift of parallax X. If no matching is found within the above range, efficient image matching can be performed by sequentially adding search ranges up to the parallax direction size H[pix]. Fig. 13 is a flowchart showing the process of acquiring the display position shift amount W.

[0068] In step S1301, lens information required to calculate the parallax X is obtained. In step S1302, the parallax X is calculated using equation (1).

[0069] In step S1303, a matching reference image is extracted from the reference image (left eye image 905L). In this embodiment, as described in FIG. 12B, an image within the focus frame 1201 is extracted.

[0070] In step S1304, image matching is performed by comparing the image data of the matching reference image extracted in step S1303 with the image data of the comparison image (right eye image 905R).

[0071] FIG. 14 is a flowchart showing image matching. In image matching, the sum ΔG (image difference) of the absolute value of the difference in output value for each pixel between the matching comparison image and the matching reference image is calculated while shifting the matching comparison image by one pixel at a time within the matching search range. Then, the matching comparison image in which the sum ΔG is equal to or smaller than a predetermined threshold and is the smallest is set as the matching image. The initial matching search range of this embodiment is a range expanded by the parallax X from the same position as the matching reference image, as described in FIG. 9(c). As in the third embodiment, it is assumed that there is no vertical deviation, and the purpose is mainly to correct deviation on the close side. If the condition is not satisfied within the matching search range, the matching search range is sequentially added and the search is performed up to the right end of the screen. However, if this does not correspond to the premise or if it is desired to correct the imaging position deviation on the infinity side as well, the matching target area may be expanded.

[0072] In step S1401, a counter n is set to zero.

[0073] In step S1402, the first pixel xw of the matching reference image is set to the first pixel xs of the matching search range, and xs is set to the first pixel xs' of the matching comparison image. In step S1403, it is determined whether the matching comparison image does not extend beyond the comparison image (right eye image 905R) (xs'+Wh>H). If it is determined that the matching comparison image does not extend beyond the comparison image (right eye image 905R), the process of step S1404 is executed. On the other hand, if it is determined that the matching comparison image extends beyond the comparison image (right eye image 905R), the process of step S1411 is executed.

[0074] In step S1404, the sum of absolute values ​​of differences in output values ​​for each pixel between the matching reference image and the matching comparison image, ΔG (image difference), is calculated.

[0075] In step S1405, it is determined whether the sum ΔG calculated in step S1404 satisfies a predetermined condition (smaller than a predetermined threshold and minimum (ΔG<minimum value ΔGmin)). If it is determined that the sum ΔG calculated in step S1404 satisfies the predetermined condition, the process of step S1406 is executed, and if it is determined that the sum ΔG calculated in step S1404 does not satisfy the predetermined condition, the process of step S1407 is executed.

[0076] In step S1406, the minimum value ΔGmin is updated, and the top pixel xs′ of the matching comparison image is set as the matching image MINxs.

[0077] In step S1407, it is determined whether the processing in the matching search range has ended (xs'≦xs+nX). If it is determined that the processing in the matching search range has ended, the processing in step S1408 is executed, and if it is determined that the processing in the matching search range has not ended, the processing in step S1409 is executed.

[0078] In step S1408, it is determined whether the minimum value ΔGmin has been acquired. If it is determined that the minimum value ΔGmin has been acquired, this flow is ended, and if it is determined that the minimum value ΔGmin has not been acquired, the process of step S1410 is executed.

[0079] In step S1409, the top pixel xs' of the matching comparison image is shifted by 1 [pix].

[0080] In step S1410, the counter n is incremented.

[0081] In step S1411, it is determined whether the minimum value ΔGmin has been acquired. If it is determined that the minimum value ΔGmin has been acquired, this flow is ended, and if it is determined that the minimum value ΔGmin has not been acquired, the process of step S1412 is executed.

[0082] In step S1412, the first pixel xw of the matching reference image is set as the first pixel of the matching image (matching image MINxs).

[0083] Returning to FIG. 13, in step S1305, a difference (deviation) ΔX between the leading pixel xw of the matching reference image extracted in step S1303 and the leading pixel Minxs of the matching image set in step S1406 or step S1412 is calculated.

[0084] In step S1306, the difference ΔX calculated in step S1305 is set as the amount of shift W of the display position.

[0085] As described above, by limiting the search range for image matching with the parallax calculated from the lens information, it is possible to efficiently search for a matching image. In addition, in the case of a general subject, there is a possibility of erroneous matching when there is a similar subject, but in this embodiment, by limiting the search to a range where there is a high possibility of matching, it is possible to correct the display deviation with high accuracy in a short time.

[0086] In addition, in this embodiment, region-based matching is used as image matching, but other methods such as feature-based matching may be used. Also, while the image difference is calculated using SAD (Sum of Absolute Differences), other methods such as SSD (Sum of Squared Differences) may be used. [Other Examples] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0087] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An imaging device capable of displaying an image in live view, an image sensor that captures a first live view image captured via a first optical system and a second live view image captured via a second optical system that is oriented in the same direction as the first optical system and has a parallax with respect to the first live view image; a display control unit that sets a setting area in the first live view image and the second live view image, The imaging device, wherein the display control unit sets the setting area of ​​the second live view image according to an amount of positional shift between the first live view image and the second live view image. (Configuration 2) 2. The imaging device according to configuration 1, wherein the set area is an area for performing focusing by autofocus. (Configuration 3) 2. The imaging device according to configuration 1, wherein the set area is an area where focusing is performed by manual focusing. (Configuration 4) 2. The imaging device according to configuration 1, wherein the set area is an area to be displayed in an enlarged scale. (Configuration 5) The imaging device described in any one of configurations 1 to 4, characterized in that the amount of positional deviation is acquired using a focal length and a shooting distance of the first optical system, and a distance between the optical axes of the first optical system and the second optical system. (Configuration 6) The imaging device described in any one of configurations 1 to 4, characterized in that the amount of positional deviation is acquired using a focal length and shooting distance of the first optical system, a distance between the optical axes of the first optical system and the second optical system, and a defocus amount based on a focus detection signal output from the imaging element in response to light incident through the first optical system and the second optical system. (Configuration 7) The imaging device described in any one of configurations 1 to 4, characterized in that the amount of positional deviation is acquired using a result of defocus matching between a first defocus map for the first optical system and a second defocus map for the second optical system, which is obtained by mapping defocus amounts of multiple regions based on focus detection signals output from the imaging element in accordance with light incident through the first optical system and the second optical system, respectively. (Configuration 8) The imaging device described in configuration 7, wherein the second defocus map is a defocus map after being shifted based on the focal length and shooting distance of the first optical system, and the optical axis distance between the first optical system and the second optical system. (Configuration 9) The imaging device according to any one of configurations 1 to 8, characterized in that the amount of positional deviation is acquired using a result of image matching between the first live view image and the second live view image. (Configuration 10) The imaging device described in configuration 9, characterized in that in the image matching, a matching search range is set using the focal length and shooting distance of the first optical system, and the distance between the optical axes of the first optical system and the second optical system. (Configuration 11) 11. The imaging device according to claim 1, wherein the display control unit moves a display position of the setting area of ​​the second live view image along a direction in which parallax occurs. (Method 1) A method for controlling an imaging device capable of displaying an image in live view, comprising the steps of: A first step of acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system having the same orientation as the first optical system and having a parallax with respect to the first live view image; a second step of setting a set area in the first live view image and the second live view image, A control method characterized in that in the second step, the display position of the setting area of ​​the second live view image is set according to the amount of positional deviation between the first live view image and the second live view image. (Configuration 12) A program for causing a computer to execute the imaging device control method according to Method 1.

[0088] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0089] 102 Camera body (imaging device) 121 Image sensor 132 Display control section 301R Right eye optical system (first optical system or second optical system) 301L Left eye optical system (second optical system or first optical system)

Claims

1. An imaging device capable of displaying an image in live view, an image sensor that captures a first live view image captured via a first optical system and a second live view image captured via a second optical system that is oriented in the same direction as the first optical system and has a parallax with respect to the first live view image; a display control unit that sets a setting area in the first live view image and the second live view image, The imaging apparatus, wherein the display control unit sets the setting area of ​​the second live view image according to an amount of positional shift between the first live view image and the second live view image.

2. 2. The imaging apparatus according to claim 1, wherein the set area is an area in which focusing is performed by autofocus.

3. 2. The image pickup apparatus according to claim 1, wherein the set area is an area for performing manual focusing.

4. 2. The image pickup apparatus according to claim 1, wherein the set area is an area to be displayed in an enlarged scale.

5. 5. The imaging device according to claim 1, wherein the amount of positional deviation is obtained using a focal length and a shooting distance of the first optical system, and a distance between the optical axes of the first optical system and the second optical system.

6. 5. The imaging device according to claim 1, wherein the amount of positional deviation is acquired using a focal length and a shooting distance of the first optical system, a distance between the optical axes of the first optical system and the second optical system, and a defocus amount based on a focus detection signal output from the imaging element in response to light incident through the first optical system and the second optical system.

7. The imaging device described in any one of claims 1 to 4, characterized in that the amount of positional shift is obtained using the result of defocus matching between a first defocus map for the first optical system and a second defocus map for the second optical system, which is obtained by mapping the defocus amounts of multiple regions based on the focus detection signal output from the imaging element in accordance with the light incident through the first optical system and the second optical system, respectively.

8. 8. The imaging device according to claim 7, wherein the second defocus map is a defocus map after being shifted based on a focal length and a shooting distance of the first optical system, and a distance between optical axes of the first optical system and the second optical system.

9. 5. The imaging apparatus according to claim 1, wherein the amount of positional deviation is acquired using a result of image matching between the first live view image and the second live view image.

10. 10. The imaging device according to claim 9, wherein in the image matching, a matching search range is set using a focal length and a shooting distance of the first optical system, and a distance between optical axes of the first optical system and the second optical system.

11. 5 . The imaging device according to claim 1 , wherein the display control unit moves a display position of the setting area of ​​the second live view image along a direction in which parallax occurs.

12. A method for controlling an imaging device capable of displaying an image in live view, comprising the steps of: A first step of acquiring a first live view image captured through a first optical system and a second live view image captured through a second optical system having the same orientation as the first optical system and having a parallax with respect to the first live view image; a second step of displaying a set area on the first live view image and the second live view image; A control method characterized in that in the second step, the setting area of ​​the second live view image is set according to an amount of positional deviation between the first live view image and the second live view image.

13. A program for causing a computer to execute the method for controlling an image pickup apparatus according to claim 12.

Images