Image processing apparatus, imaging apparatus, image processing method, and program

Abstract

To provide an image processing apparatus that can create a VR image that can be correctly viewed by a user irrespective of the posture of the user.SOLUTION: An image processing apparatus (10) has: acquisition means that acquires a first image and a second image having parallax, or one image including the first image and the second image having parallax; and control means (12) that can set a first mode and a second mode. In the first mode, the control means outputs one image for stereoscopic viewing, with the first image as an image for the right eye and the second image as an image for the left eye, and in the second mode, outputs one image for stereoscopic viewing as an image with the first image as the image for the left eye and the second image as the image for the right eye, wherein the first image and the second image are respectively rotated around an optical axis by 180°.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses an imaging device capable of capturing images from two optical systems at once, in order to capture two images with parallax using two optical systems and view the captured images in a 3D virtual reality (VR) environment. When viewing with VR goggles, in order to obtain not only a 3D effect but also a sense of realism, it is desirable that the angle of view of the captured video or still image is 180 degrees or more. In order to provide an image of at least 180 degrees, taking into consideration manufacturing errors, it is desirable that the shooting lens is a lens that can capture an angle of view of more than 180 degrees. The form of the input image to a device used for VR viewing, such as VR goggles, is generally expressed in equirectangular projection. As described above, a fisheye lens is used to express an image with an angle of view of about 180 degrees. By appropriately correcting a distorted image captured with a fisheye lens to equirectangular projection, an image close to the state seen by the human eye can be displayed on a display corresponding to the left and right eyes.

[0003] Patent Document 2 discloses an image processing means for detecting a rotational position shift of a captured image and correcting the captured image so that it is horizontal in order to correct the horizontality of an image captured at an angle. The amount of rotational shift may be detected using an acceleration sensor inside the camera, and the rotational shift of the captured image may be corrected according to the amount of rotational shift. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5166650 [Patent Document 2] JP 2021-129169 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, a stereo image can be output as a set of left and right images that pass through left and right lenses on an image sensor. Here, the left and right direction of a twin-lens camera equipped with two optical systems is the x-axis, the up and down direction is the y-axis, and the front and back direction is the z-axis. In this case, for example, if an image is captured with a rotational deviation around the z-axis and viewed as is on a head-mounted display (HMD) that is a VR viewing device, the horizon displayed on the HMD will be tilted by the amount of the rotational deviation. This causes a deviation between the image displayed in VR and the sense of the direction of gravity, which makes it easy to get VR sickness.

[0006] In VR shooting, it is basically desirable for the user to shoot with the twin-lens camera fixed horizontally, but sometimes the image is shot with a slight tilt. Therefore, the camera's posture is detected from information such as the acceleration sensor inside the camera, and the amount of rotational deviation is linked to the shot image as a rotational deviation correction parameter and recorded, thereby correcting the rotational deviation of the shot image. This allows for a more accurate horizontal VR image. However, as the amount of rotational deviation increases, the parallax of the two optical systems increases not only left and right, but also up and down. Even if rotational deviation correction is performed on a shot image with a large amount of rotational deviation, the up and down parallax cannot be removed, so the up and down parallax also affects VR viewing. For example, in an image shot with a twin-lens camera rotated 90 degrees, that is, with the camera in a vertical position, the left and right parallax becomes zero and only the up and down parallax remains. For this reason, a VR image with a 90-degree rotational deviation corrected cannot be viewed in stereoscopic view correctly.

[0007] Also, when the twin camera is rotated 180 degrees, that is, when the camera is upside down, the image captured has left-right parallax but no up-down parallax, but the VR image that has been corrected for 180-degree rotational misalignment cannot be viewed correctly in stereo. This is because, when rotated 180 degrees, the left lens of the twin camera corresponds to the right eye of a human, and the right lens corresponds to the left eye of a human. In other words, if the image from the left lens of the twin camera is output as an image for the left eye and the image from the right lens is output as an image for the right eye, the VR image will have reversed left-right parallax and cannot be viewed correctly in stereo.

[0008] In VR shooting, for example, in some situations, such as shooting with a drone, shooting at a low angle using a tripod, or shooting fixed to the ceiling, it can be difficult to fix the twin-lens camera horizontally. Furthermore, there are cases where the twin-lens camera must be fixed upside down, that is, in a state (posture) rotated 180 degrees. However, if rotational deviation correction is performed using the rotational deviation correction parameters recorded inside the camera and a horizontal image is output, as mentioned above, the VR image will have reversed left and right parallax.

[0009] Therefore, an object of the present invention is to provide an image processing device capable of generating VR images that can be viewed correctly regardless of posture. [Means for solving the problem]

[0010] An image processing device as one aspect of the present invention includes an acquisition means for acquiring a first image and a second image with parallax, or an image including the first image and the second image with parallax, and a control means capable of setting a first mode and a second mode, wherein the control means outputs a single image for stereoscopic viewing in the first mode, with the first image as an image for the right eye and the second image as an image for the left eye, and outputs the single image for stereoscopic viewing in the second mode, with the first image as an image for the left eye, the second image as an image for the right eye, and further with the first image and the second image each rotated 180 degrees around the optical axis.

[0011] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0012] According to the present invention, it is possible to provide an image processing device capable of generating VR images that can be viewed correctly regardless of posture. [Brief description of the drawings]

[0013] [Figure 1] 1 is a cross-sectional view of a lens device according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram of an imaging apparatus according to an embodiment of the present invention. [Diagram 3] 4 is an example of an image captured by the imaging device according to the present embodiment. [Figure 4] 4 is an example of an image obtained by converting the image in FIG. 3 using equirectangular projection. [Diagram 5] 1A and 1B are diagrams illustrating a configuration of an imaging device according to an embodiment of the present invention and examples of captured images. [Figure 6] 5(c) is converted into an example of an image using equirectangular projection. [Figure 7] 3A and 3B are front views of an inverted imaging device according to the present embodiment and examples of captured images. [Figure 8] 8 is an example of an image obtained by converting the image in FIG. 7(b) using equirectangular projection. [Figure 9] 2 is a configuration example of an image processing device (application) according to the present embodiment. [Figure 10] 1A and 1B are examples of a front view and a captured image of an imaging device in an inverted imaging mode and tilted about an optical axis in this embodiment; [Figure 11] 10 is an example of an image obtained by converting a captured image according to the present embodiment using equirectangular projection. [Figure 12] 4 is a flowchart of an image processing method according to the present embodiment. [Figure 13] 4 is an example of an operation screen of the image processing apparatus according to the present embodiment. [Figure 14]FIG. 2 is a side view of the imaging device in the embodiment in an upward orientation. [Figure 15] 4 is an example of a live view image of the imaging device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0015] The lens device (interchangeable lens) of this embodiment has two optical systems (first optical system and second optical system) arranged in parallel (symmetrically) to each other, and is configured to form two image circles in parallel on one imaging element. The two sets of optical systems are arranged horizontally at a predetermined distance (baseline length). When viewed from the image side, an image formed by the right optical system (first optical system) is recorded as a video or still image for the right eye, and an image formed by the left optical system (second optical system) is recorded as a video or still image for the left eye. When playing back a video or still image (image), the image is viewed using a known 3D display or so-called VR goggles, so that an image for the right eye is projected on the user's right eye and an image for the left eye is projected on the left eye. At this time, an image with parallax is projected on the right eye and the left eye due to the base line length of the lens device, so that the user can get a three-dimensional feeling. In this way, the lens device of this embodiment is a lens device for stereoscopic photography that can form two images with parallax by the first optical system and the second optical system.

[0016] With reference to FIG. 1, an interchangeable lens (lens device) 200 in this embodiment will be described. FIG. 1 is a cross-sectional view showing a schematic configuration of a first optical system 201R and a second optical system 201L of the interchangeable lens 200. The interchangeable lens 200 has a first optical system 201R and a second optical system 201L. The first optical system 201R and the second optical system 201L are each capable of shooting at an angle of view of 180 degrees or more. Each optical system has, in order from the subject side to the image side, a first optical axis OA1, a second optical axis OA2 substantially perpendicular to the first optical axis, and a third optical axis OA3 parallel to the first optical axis. Along each optical axis, a first lens 211 having a surface 211A convex toward the subject side is arranged on the first optical axis OA1, a second lens 221 on the second optical axis OA2, and third lenses 231 and 231-2 on the third optical axis OA3. Each optical system further includes a first prism 220 that bends the light flux having the first optical axis OA1 to a second optical axis OA2, and a second prism 230 that bends the light flux having the second optical axis OA2 to a third optical axis OA3. The first optical axis OA1R of the first optical system 201R and the first optical axis OA1L of the second optical system 201L are separated from each other by a base length L1, and can capture an image with parallax corresponding to the distance of the base length L1.

[0017] Fig. 2 is a block diagram of an imaging device 100 capable of capturing stereoscopic images. As shown in Fig. 2, the imaging device 100 has a camera body 110 and an interchangeable lens 200. The interchangeable lens 200 is attached to the camera body 110 when in use. In this embodiment, the interchangeable lens 200 is an interchangeable lens system that is detachable from the camera body 110, but the camera body and the lens device may be configured as an integrated unit.

[0018] The interchangeable lens 200 has a first optical system 201R and a second optical system 201L that constitute an imaging optical system, and captures an image formed on an imaging element 111 of the camera body 110. A lens control unit 203 of the interchangeable lens 200 processes values ​​detected by various detection units such as a temperature detection unit 207 and a focus detection unit 208, outputs the processed information to a system control unit 117, and controls the imaging device 100 in cooperation with the camera body 110. The interchangeable lens 200 also has a storage unit 204. In response to a request from the system control unit 117, the storage unit 204 outputs various information detected by the temperature detection unit 207 or the focus detection unit 208, and identification information such as lens individual information 205 or manufacturing error information 206 to the system control unit 117.

[0019] Camera body 110 has an image sensor 111, an A / D conversion unit 112, an image processing unit 113, a display unit (display means) 114, an operation unit (operation means) 115, a recording unit 116, a system control unit 117, an attitude detection unit (attitude detection means) 119, and a camera mount 122. When interchangeable lens 200 is attached to camera body 110 via lens mount 202 and camera mount 122, system control unit 117 and lens control unit 203 are electrically connected.

[0020] The image of the subject is formed on the image sensor 111 in such a manner that a right eye image formed via the first optical system 201R and a left eye image formed via the second optical system 201L are arranged side by side. The image sensor 111 converts the formed image of the subject (optical signal) into an analog electrical signal. The A / D conversion unit 112 converts the analog electrical signal output from the image sensor 111 into a digital electrical signal (image signal). The image processing unit 113 performs various image processing on the digital electrical signal (image signal) output from the A / D conversion unit 112.

[0021] The display unit 114 displays various types of information. The display unit 114 is realized by using, for example, an electronic viewfinder or a liquid crystal panel. The operation unit 115 functions as a user interface that enables the user to give instructions to the imaging device 100. If the display unit 114 has a touch panel, the touch panel also constitutes the operation unit 115.

[0022] The recording unit 116 records various data such as image data that has been subjected to image processing by the image processing unit 113. The recording unit 116 also stores programs. The recording unit 116 is realized by using, for example, a ROM, a RAM, and a HDD.

[0023] The storage unit 118 stores individual identification information such as individual camera information 120 and camera manufacturing error information 121 of the imaging device 100. In the manufacturing process of the camera body 110, information including, for example, model information of the camera body 110, pixel number information of the imaging element 111, physical size information of the imaging element 111, manufacturing error information for the captured image, etc. is written to the storage unit 118. When an image is captured using the imaging device 100, the storage unit 118 outputs the above-mentioned identification information to the system control unit 117. Then, the identification information is generated as an image together with the identification information of the interchangeable lens 200 sent from the lens control unit 203 of the interchangeable lens 200. This image (information image) is output to the image processing unit 113, and is added to the captured image to record it in the captured data. As described above, the image processed by the image processing unit 113 is written to various storage media, such as a flash memory or HDD, by the recording unit 116.

[0024] The system control unit 117 performs overall control of the entire imaging device 100. The system control unit 117 is realized by using a CPU, for example. The above information is recorded on a medium by the recording unit 116 via the system control unit 117. Meanwhile, light captured by the imaging optical system forms an image on the imaging element 111. In general, the formed image is captured by the imaging element 111, and various processes are performed by the A / D conversion unit 112 and the image processing unit 113, after which it is configured as a captured image and written to a recording medium by the recording unit 116.

[0025] Next, we will explain an example of a fisheye image captured using the imaging device 100 and an example of an image converted to equirectangular projection. When an image is captured using an imaging device with a single general lens optical system, an image that is 180 degrees inverted is formed on the imaging element. When this is made into an image, a 180-degree inversion process is performed to align the up-down direction.

[0026] 3 is an example of an image captured by the imaging device 100. An image (e.g., image 300) captured through the interchangeable lens 200 is formed on one imaging element 111 via a first optical system 201R corresponding to the right eye and a second optical system 201L corresponding to the left eye in normal shooting, and is inverted 180 degrees for each optical system. In order to match the top and bottom of the captured image, the entire captured image is inverted 180 degrees as in a general optical system. At this time, an image 300 is output in which the imaging result of the first optical system corresponding to the right eye in normal shooting (fisheye image for right eye 301R) is captured on the left side of the captured image, and similarly, the imaging result of the second optical system corresponding to the left eye (fisheye image for left eye 301L) is captured on the right side of the captured image.

[0027] 4(a) and (b) show an example of converting the image 300 of FIG. 3 into an image of equirectangular projection. In general, as shown in FIG. 4(a), an image 400 is arranged side by side with a left-eye equirectangular image 401L on the left side and a right-eye equirectangular image 401R on the right side, so that VR display can be performed in an HMD (head mounted display). Alternatively, as shown in FIG. 4(b), an image 410 may be arranged vertically with a left-eye equirectangular image 401L on the top and a right-eye equirectangular image 401R on the bottom. Here, the shooting data includes lens individual information 205 and manufacturing error information 206, and the fisheye shooting data (image 300) of FIG. 3 can be converted into the equirectangular image (image 400) of FIG. 4 based on the lens individual information 205 and manufacturing error information 206.

[0028] The HMD can determine whether the image is for the left eye or the right eye based on a fixed arrangement such as side-by-side or top-down, and display each image on the left eye display or right eye display of the HMD. Depending on the image playback player in the HMD, the user may be able to set the left and right display. It is also possible to load the fisheye image (image 300) as is into the HMD. In that case, mesh data is linked to the fisheye image as metadata, and the image is displayed on the HMD according to the mesh data. The mesh data consists of a mesh projected onto a virtual sphere, and is conversion coordinate information that associates the coordinates on the virtual sphere with the center coordinates and peripheral coordinates of the fisheye image. In this method, metadata is added to the image so that the mesh for the left eye is linked to the fisheye image for the left eye 301L, and the mesh for the right eye is linked to the fisheye image for the right eye 301R. Furthermore, the mesh data includes lens individual information 205 and manufacturing error information 206, enabling more accurate and correct VR display. As described above, regardless of whether the image is a fisheye image or an equirectangular image, the left eye image and the right eye image can be set by the image arrangement, mesh data, etc., making it possible to display the image in VR on the HMD.

[0029] Here, VR display (VR view) is a display method (display mode) that can change the display range and displays images of VR images with a viewing range that corresponds to the attitude of the display device. VR display includes "single-eye VR display (single-eye VR view)" that displays one image by performing a transformation (transformation that performs distortion correction) that maps the VR image onto a virtual sphere. VR display also includes "two-eye VR display (two-eye VR view)" that displays a VR image for the left eye and a VR image for the right eye side by side in left and right regions by performing a transformation that maps each image onto a virtual sphere. Stereoscopic viewing is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax from each other.

[0030] Regardless of the VR display, for example, when a user wears a display device such as an HMD (head mounted display), an image with a visual field range according to the direction of the user's face is displayed. For example, assume that an image with a visual field range centered on 0 degrees left and right (a specific direction, for example, north) and 90 degrees up and down (90 degrees from the zenith, i.e. horizontal) is displayed at a certain point in time among the VR images. If the attitude of the display device is reversed from this state (for example, the display surface is changed from facing south to facing north), the display range is changed to an image with a visual field range centered on 180 degrees left and right (the opposite direction, for example, south) and 90 degrees up and down among the same VR image. That is, when the user faces from north to south (i.e. turns backwards) while wearing the HMD, the image displayed on the HMD is also changed from a north image to a south image. Note that the VR image captured using the interchangeable lens 200 of this embodiment is a VR180 image capturing a range of approximately 180 degrees forward, and there is no image with a range of approximately 180 degrees backward. When such a VR180 image is displayed in VR and the attitude of the display device is changed to a side where no image is present, a blank area is displayed.

[0031] By displaying the VR image in this way, the user visually feels as if he or she is inside the VR image (in the VR space). The method of displaying the VR image is not limited to changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operation via a touch panel or directional buttons. Furthermore, during VR display (when in the "VR view" display mode), in addition to changing the display range due to posture changes, the display range may also be changed in response to touch-move on the touch panel, dragging with a mouse, pressing a directional button, etc. A smartphone attached to VR goggles (head-mounted adapter) is a type of HMD.

[0032] Normally, when shooting with an imaging device capable of stereoscopic imaging such as imaging device 100, the two lenses are fixed or held so that they are aligned horizontally. If a fisheye image shot in a perfectly horizontal position is converted into an equirectangular projection image or mesh data is embedded, and the image is displayed as a VR image on an HMD, good VR viewing can be achieved.

[0033] However, if the image captured with the imaging device 100 slightly tilted during shooting is subjected to the same image processing as described above and viewed in VR on an HMD, the image displayed in VR will be tilted by the amount it was tilted during shooting, even if the posture of the viewer and the HMD are correct. As a result, the posture of the viewer and the posture of the VR display will differ, and the viewer will become sick due to the mismatch in sensations, which is known as VR sickness.

[0034] Therefore, the imaging device 100 of this embodiment has an attitude detection unit 119 that detects the attitude of the imaging device 100 during imaging and outputs attitude data, and can detect the tilt during shooting and correct it to the horizontal. The attitude detection unit 119 is, for example, an acceleration sensor or a gyro sensor. The imaging device 100 acquires the attitude data (horizontal correction value acquired from the attitude data) from the attitude detection unit 119. The attitude data is embedded in the shooting data and linked to the shooting data. Then, when converting the shot image into an image in equirectangular projection, correction is performed using the attitude data in addition to the lens individual information 205 and manufacturing error information 206, and a horizontally corrected VR image can be output.

[0035] 5(a) to (c) are diagrams of the configuration of the imaging device 100 and examples of captured images. For example, as shown in FIG. 5(a), the left-right direction of the twin-lens imaging device 100 equipped with two optical systems, a first optical system 201R and a second optical system 201L, is the x-axis, the up-down direction is the y-axis, and the front-back direction is the z-axis. When the imaging device 100 is captured in a state inclined at an angle θ with respect to the horizontal in the rotation direction (roll direction) around the z-axis as shown in FIG. 5(b) as viewed from the front, an image (captured image) 500 as shown in FIG. 5(c) is output. That is, the image (captured image) 500 is output as a captured image in which a fisheye image 501R captured by the first optical system 201R and a fisheye image 501L captured by the second optical system 201L are inclined at an angle θ. The angle θ can be calculated by detecting the gravity direction G by an attitude detection unit 119 such as an acceleration sensor provided in the imaging device 100. When this image is converted into an image in equirectangular projection, as shown in FIG. 6(a), an output image 600 is output in which tilted equirectangular images 601R and 601L are output as VR images.

[0036] Here, the attitude of the imaging device 100 is detected by an attitude detection unit 119 such as an acceleration sensor, and attitude data indicating that the image is tilted by an angle θ in the roll direction is written as metadata to the image 500. By performing correction based on this attitude data, as shown in FIG. 6B, the image 610 is output as a VR image in the form of horizontally corrected equirectangular images 611R and 611L. This horizontal correction is performed simultaneously when the image 500 is converted into an image of equirectangular projection, and the image 610 can be output from the image 500. By viewing the image thus horizontally corrected on an HMD, VR sickness can be suppressed.

[0037] In the case of mesh data, by linking information that the data is tilted by an angle θ in the roll direction as metadata, the data can be displayed with only the roll angle θ corrected in the VR display on the HMD. However, when the angle θ is large, not only the left-right parallax H but also the up-down parallax V occurs as shown in FIG. 5(b). The left-right parallax H also becomes smaller than the base line length L1, which is the parallax when horizontal. Furthermore, when the posture of the image capture device 100 is tilted by 90° in the roll direction, the left-right parallax H becomes zero, and when the posture of the image capture device 100 is tilted by more than 90°, the left-right parallax H becomes parallax in the opposite direction.

[0038] In image 610 in Fig. 6(b) which has been corrected horizontally based on the posture data, the tilt in the roll direction has been corrected, but vertical parallax remains in the image due to the vertical parallax V. If the vertical parallax is small, it may not cause VR sickness, but the larger the vertical parallax, the more likely it is that VR sickness will occur, so it is preferable to hold the camera as horizontally as possible when shooting.

[0039] Next, a case where the imaging device 100 captures an image in an upside-down state (inverted posture) will be described with reference to Figs. 7(a), (b) and Figs. 8(a) to (c). Figs. 7(a) and (b) are examples of a front view of the imaging device in an inverted posture and a captured image. Figs. 8(a) to (c) are examples of images obtained by converting the image in Fig. 7(b) using equirectangular projection.

[0040] When the roll angle θ is 180° and the image is captured in the state shown in FIG. 7(a), an upside-down image (captured image) 700 as shown in FIG. 7(b) is obtained. When the image 700 is directly converted into an equirectangular projection image, an upside-down image (VR image) 800 as shown in FIG. 8(a) is output. Similarly, when the roll angle θ is corrected to 180° based on the posture data, a horizontally corrected image 810 as shown in FIG. 8(b) is output. Here, the corrected image 811L is an image obtained by rotating and correcting the image 801L by 180°, and the image 811R is an image obtained by rotating and correcting the image 801R by 180°. However, when the image is corrected in this way, the left and right parallax is reversed.

[0041] That is, when photographing in the posture shown in FIG. 7(a), in order to view stereoscopically with the correct posture and the correct parallax, the first optical system 201R must be recognized as an image for the left eye, and the second optical system 201L must be recognized as an image for the right eye. However, when horizontal correction is performed using the same algorithm, the image 701R captured by the first optical system 201R is output as an image 801R in the image 800 converted by equirectangular projection. Also, in the image 810 rotated and corrected by 180°, it is output as an image 811R, and is output as an image for the right eye. Similarly, the image captured by the second optical system 201L is output as an image 801L in the image 800 converted by equirectangular projection, and is output as an image 811L in the image 810 rotated and corrected by 180°, and is output as an image for the left eye.

[0042] That is, the left and right images are reversed between image 400 captured in a normal position and converted into an equirectangular projection image, and image 810 captured in an upside-down position, rotated and corrected by 180°, and converted into an equirectangular projection image. To create a VR image with correct parallax, it is necessary to output image 811R, which is rotated and corrected by 180°, for the left eye, and image 811L, which is rotated and corrected for the right eye, as in image 820.

[0043] As a specific algorithm for performing 180° rotation correction while switching between images for the left eye and the right eye and outputting them in this way, there is a method of outputting image 820 in which image 801R is rotated 180° and placed on the left side, and image 801L is rotated 180° and placed on the right side. Alternatively, a method may be used in which image 800 before correction is regarded as one image, and image 820 is output by rotating the entire image 800 by 180°. The image processing described above is performed simultaneously with the process of converting image 700 into an equirectangular projection image, and image 820 can be output from image 700.

[0044] When mesh data is linked to a fisheye image, data indicating that the image 701R is a right-eye image is linked, and data indicating that the image 701L is a left-eye image is linked. Data indicating that the image is rotated 180° in the roll direction is linked, and even when the image is displayed after being corrected 180° in the roll direction, the data indicating that the image 701R is an image for the right eye and the image 701L is an image for the left eye remains the same, and the image is displayed on the HMD with the parallax reversed. In other words, in the case of a VR image having an image for the left eye and an image for the right eye, in order to make an image captured upside down into an image that can be viewed in the correct posture, not only is it necessary to perform the 180° rotation correction, but it is also necessary to swap the image for the left eye and the image for the right eye. In other words, it is necessary to rewrite the data to indicate that the image 701R is an image for the left eye and the image 701L is an image for the right eye.

[0045] Therefore, in this embodiment, the imaging device 100 has a normal imaging mode (first mode) and an inverted imaging mode (second mode). In the normal imaging mode, for example, an image captured by the first optical system 201R is output as an image for the right eye, and an image captured by the second optical system 201L is output as an image for the left eye. In this case, in the inverted imaging mode, an image captured by the first optical system 201R is output as an image for the left eye, and an image captured by the second optical system 201L is output as an image for the right eye.

[0046] The image processing device of this embodiment operates on an application (software) that handles images and videos captured by a digital camera (imaging device) equipped with a twin lens (VR180 lens), and is usually realized on a terminal such as a personal computer (PC) or a smartphone. The application may be installed inside the digital camera.

[0047] Next, a configuration example of an image processing device (application) in this embodiment will be described with reference to Fig. 9(a) to (c). Fig. 9(a) and (b) show a configuration example of an image processing device (application). Fig. 9(c) is a block diagram of the image processing device.

[0048] 9(a), imaging data captured by the imaging device 100 is transmitted to a terminal 910 such as a PC, a smartphone, or a tablet via wireless 901 such as WIFI or Bluetooth (registered trademark) or a cable 902. Alternatively, the imaging data is stored in a memory card 903, and the imaging data stored in the memory card 903 can be read by the terminal 910. Then, a VR image is output by an application (image processing device) installed in the terminal 910.

[0049] In the configuration shown in FIG. 9(b), an application (image processing device) is installed in the imaging device 100, and by operating the operation unit 115, a VR image can be output from the imaging data and displayed on the display unit 114, and the VR image can be recorded to a medium by the recording unit 116.

[0050] As shown in FIG. 9(c), the image processing device 10 of this embodiment has an acquisition means 11, a control means 12, and an operation means 13. The acquisition means 11 acquires a first image and a second image with parallax, or an image including the first image and the second image with parallax. The control means 12 can set a first mode and a second mode. In the first mode, the control means 12 outputs one image for stereoscopic viewing (an image converted by equirectangular projection) with the first image as an image for the right eye and the second image as an image for the left eye. In the second mode, the control means 12 outputs one image for stereoscopic viewing with the first image as an image for the left eye, the second image as an image for the right eye, and the first image and the second image each rotated 180° around the optical axis. The operation means 13 can be operated by the user, but is not essential in this embodiment. The control means 12 can set the first mode and the second mode by the user's selection. However, the present embodiment is not limited to this, and the control means 12 may set the first mode and the second mode based on a predetermined condition that does not depend on the user's selection.

[0051] Next, a case where a slight tilt of the image capture device 100 in the inverted shooting mode is corrected will be described with reference to Figs. 10(a), (b) and Figs. 11(a) to (c). Figs. 10(a) and (b) are respectively a front view of the image capture device 100 in the inverted shooting mode and tilted around the optical axis, and an example of a captured image. Figs. 11(a) to (c) are examples of images obtained by converting the captured image using equirectangular projection.

[0052] Consider an image captured in an inverted shooting mode posture in which the first optical system 201R is for the left eye and the second optical system 201L is for the right eye, tilted by an angle α, as shown in Fig. 10(a). That is, the image is captured in a posture rotated in the +α roll direction at an angle θ=180° from the horizontal state. In this case, as shown in Fig. 10(b), an image 1000 is obtained that is approximately upside down and tilted by an angle α. If the image 1000 is directly converted into an image of equirectangular projection, a VR image 1100 that is approximately upside down and tilted by an angle α is output, as shown in Fig. 11(a).

[0053] Since the posture shown in Fig. 10(a) is substantially upside down, it is desirable to output the image captured by the first optical system 201R for the left eye and the image captured by the second optical system 201L for the right eye. Therefore, when a 180° rotation correction is performed in the roll direction in the upside-down shooting mode, an image 1110 as shown in Fig. 11(b) is output. That is, an image 1110 is output as an image 1111R obtained by 180° rotation correction of the image 1101R for the left eye, and as an image 1111L obtained by 180° rotation correction of the image 1101L for the right eye.

[0054] However, when only 180° rotation correction is performed in the inverted shooting mode, the image is still tilted by angle α. Therefore, in order to make the image available for VR viewing in the correct posture, it is necessary to further rotate and correct the angle α to make the VR image horizontal. Here, the conventional horizontal correction algorithm cannot be applied as it is. This is because the shooting data shot in the posture shown in FIG. 10(a) contains posture data detected by the posture detection unit 119, and the posture data contains posture information that the image is tilted by angle θ in the roll direction. Here, when performing horizontal correction on the image 1110, if the conventional horizontal correction algorithm is applied as it is, the image will be corrected by angle θ in the roll direction. To perform correct horizontal correction and output the image 1120 shown in FIG. 11(c), it is necessary to perform horizontal correction in the roll direction at angle α, which is the angle θ minus 180°.

[0055] In the above, for ease of understanding, the case where only inversion is performed in the inverted shooting mode and the case where horizontal correction is further performed are described in order, but this embodiment is not limited to this. When horizontal correction is further performed in the inverted shooting mode, for example, image 1000 can be processed simultaneously when it is converted into an image of equirectangular projection, and image 1120 can be output from image 1000.

[0056] Next, an image processing method by the image processing device having an inverted shooting mode and a horizontal correction function in this embodiment will be described with reference to Fig. 12. Fig. 12 is an example of a flowchart showing the image processing method in this embodiment.

[0057] First, in step S1200, the image processing device (application) reads an image file. The images read here are fisheye images such as images 300, 500, 700, and 1000. Here, posture data and shooting information contained in the image are also read as metadata.

[0058] Next, in step S1201, the image processing device determines whether the image read in step S1200 is an image captured in an inverted shooting mode (whether the image is captured in a normal shooting mode or an inverted shooting mode). Here, the means for determining the inverted shooting mode may be selectable by the user through an operation means of the image processing device, or may be automatically determined by an application from image information. The means for determining the inverted shooting mode will be described later.

[0059] If it is determined in step S1201 that the mode is the inverted shooting mode, the process proceeds to step S1202. In step S1202, the image processing device determines whether or not to perform horizontal correction. Similarly, the determination of horizontal correction may be made by a user through an operation means of the image processing device, or may be automatically determined by an application based on image information.

[0060] If it is determined in step S1202 that horizontal correction is to be performed, the process proceeds to step S1203. In step S1203, since the mode is one in which horizontal correction is further performed in the inverted shooting mode, it is necessary to set the roll direction correction value of the attitude data to a value of -180° as described above. Next, in step S1204, the image processing device performs horizontal correction using the roll direction correction value of -180°. Also, since the image processing device is in the inverted shooting mode, the image is rotated 180° in the roll direction, and the image of the first optical system 201R is converted into an image for the left eye and the image of the second optical system 201L is converted into an image of equirectangular projection (image for stereoscopic viewing) as an image for the right eye. In this embodiment, the rotation correction angle around the optical axis for the input image of the image for stereoscopic viewing horizontally corrected in the normal shooting mode and the rotation correction angle around the optical axis for the input image of the image for stereoscopic viewing horizontally corrected in the inverted shooting mode differ from each other by 180°.

[0061] By using the algorithms described above in steps S1203 and S1204, the algorithm for processing horizontal correction and the algorithm for processing in inverted shooting mode can be used as is simply by setting the argument of the roll direction correction value to -180°.

[0062] As another solution, when further performing horizontal correction in the inverted shooting mode, horizontal correction may be performed using a roll correction value exceeding 180°, and the image of the first optical system 201R may be output as an image for the left eye, and the image of the second optical system 201L may be output as an image for the right eye, in an equirectangular projection image. However, in this case, the algorithm of the inverted shooting mode itself changes depending on whether or not horizontal correction is performed, so two types of algorithms for the inverted shooting mode are required. In other words, when performing horizontal correction, the correct posture is corrected by the horizontal correction process, so there is no need to rotate 180° in the roll direction again in the inverted shooting mode. On the other hand, when horizontal correction is not performed, it is necessary to rotate 180° in the roll direction in the inverted shooting mode to the correct posture.

[0063] If it is determined in step S1202 that horizontal correction is not to be performed, the process proceeds to step S1205. In step S1205, the image processing device rotates the image by 180° in the roll direction, and converts the image of the first optical system 201R as an image for the left eye and the image of the second optical system 201L as an image for the right eye into an equirectangular projection image without using any correction value.

[0064] In steps S1204 and S1205, since both are inverted shooting mode, the image is rotated 180° in the roll direction, and the image of the first optical system 201R is output as an image for the left eye, and the image of the second optical system 201L is output as an image for the right eye. When performing this processing, the image for the left eye and the image for the right eye may be processed and pasted together to output the final equirectangular projection image. Alternatively, an algorithm similar to that of the normal shooting mode may be used to create equirectangular projection images arranged side by side, which may be used as a single image to rotate the entire image 180° and output the final equirectangular projection image. The image processing device of this embodiment may finally output the image of the first optical system 201R as an image for the left eye and the image of the second optical system 201L as an image for the right eye, and output the image rotated 180° in the roll direction for the image shot in the inverted shooting mode. As long as such a result can be obtained, any processing may be performed inside the algorithm for the image shot in the inverted shooting mode.

[0065] If it is determined in step S1201 that the mode is not the inverted shooting mode, the process proceeds to step S1206. In step S1206, the image processing device determines whether or not to perform horizontal correction, similar to step S1202. If it is determined that horizontal correction is to be performed, the process proceeds to step S1207. In step S1207, the image processing device acquires attitude data contained in the image and obtains a correction value. Next, in step S1208, the image processing device performs horizontal correction based on the obtained correction value, and converts the image of the first optical system 201R as an image for the right eye and the image of the second optical system 201L as an image for the left eye into equirectangular projection images.

[0066] If it is determined in step S1206 that horizontal correction is not to be performed, the process proceeds to step S1209. In step S1209, the image processing device converts the image of the first optical system 201R as an image for the right eye and the image of the second optical system 201L as an image for the left eye into equirectangular projection images.

[0067] As described above, the application can operate according to the algorithm flowchart in this embodiment.

[0068] Next, an operation screen of the image processing device (application) of this embodiment will be described with reference to Figs. 13(a) and (b). Figs. 13(a) and (b) are examples of the operation screen of the image processing device. In the window screen 1300 of the application shown in Fig. 13(a), an image 1301R obtained by converting an image by the first optical system 201R into an image by equirectangular projection is displayed as an image for the left eye. Also, an image 1301L obtained by converting an image by the second optical system 201L into an image by equirectangular projection is displayed as an image for the right eye.

[0069] Also displayed on the window screen 1300 are a horizontal correction checkbox 1302 and an inverted shooting checkbox 1303. The user can switch between whether or not to perform horizontal correction by switching on / off the checkbox 1302. Also, the user can switch between whether or not to switch to the inverted shooting mode by switching on / off the checkbox 1303.

[0070] 13(b), when the check box 1303 is turned on, an image 1301R obtained by converting an image by the first optical system 201R is displayed as an image for the left eye, and an image 1301L obtained by converting an image by the second optical system 201L is displayed as an image for the right eye. For example, depending on whether the check boxes 1302 and 1303 are on or off, determinations are made in steps S1201, S1202, and S1206 in FIG.

[0071] Although the above description has been given of an example in which the user can select the inverted shooting mode and horizontal correction, the application may determine the same from an image that includes posture data at the time of shooting.

[0072] As described above, the left-right direction of the image capture device 100 shown in FIG. 5(a) is the x-axis, the up-down direction is the y-axis, and the front-back direction is the z-axis. The image capture device 100 can acquire the direction of gravity G as orientation data by the orientation detection unit 119, such as an acceleration sensor. For example, when the image capture device 100 is in the normal orientation as shown in FIG. 5(a), the direction of gravity G is x=0, y=-1, z=0. When the image capture device 100 is in a completely upside-down orientation, the direction of gravity G is x=0, y=1, z=0. When determining whether the orientation is upside-down, a threshold value is set somewhere between the normal orientation and the completely upside-down orientation, and it is necessary to determine whether the threshold value is exceeded. Here, the x, y, and z components of the direction of gravity G each take a value in the range of -1 to 1, and the magnitude of the three-dimensional vector made up of each component is 1. In other words, √(x 2 +y 2 +z 2 Each component is output so that ) = 1.

[0073] Here, a threshold value for determining whether the image is upside down will be described. For example, focusing on the y-axis of the image capture device 100, the value of the y-axis component indicating the direction of gravity G is negative when the image capture device is in a normal position, and the value of the y-axis component indicating the direction of gravity G is positive when the image capture device is in an upside down position. The positive and negative values ​​of the y-axis component, that is, the threshold value y=0, can be used as a boundary for determining whether the image capture device is upside down.

[0074] 14(a) and (b) are side views of the imaging device 100 in an upward orientation. In FIG. 14(a), for simplicity, a posture in which the imaging device 100 rotates from a normal orientation to an upward orientation around the x-axis is considered. When y=0 is set as a threshold, as shown in FIG. 14(a), a normal shooting determination region 1400 is a region in which the imaging device 100 shoots directly above from a normal orientation, and an inverted shooting determination region 1401 is a region in which the imaging device 100 shoots directly above from a completely inverted orientation. In other words, in terms of the angle around the x-axis, a posture from a normal orientation to a rotation of an angle of 90° around the x-axis is the normal shooting determination region 1400, and a region rotated 90° to 180° around the x-axis is the inverted shooting determination region 1401.

[0075] When performing horizontal correction on an image captured in a posture within the range of normal shooting judgment area 1400, the image is rotated in the direction of horizontal correction direction 1402 shown in Fig. 14(a) so that the z-axis direction, which is the forward shooting direction of image capturing device 100, becomes horizontal. In other words, the forward shooting direction of image capturing device 100 is corrected to a direction for shooting the left direction in Fig. 14(a).

[0076] When performing horizontal correction on an image captured in a posture within the range of the inverted shooting determination area 1401, the image is rotated in the direction of horizontal correction direction 1403 shown in Fig. 14(a) so that the z-axis direction, which is the forward shooting direction of the imaging device 100, becomes horizontal. In other words, the forward shooting direction of the imaging device 100 is corrected to a direction for shooting the right direction in Fig. 14(a).

[0077] When the image is taken in a posture close to directly above, the direction displayed as a VR image after horizontal correction is reversed in the normal shooting mode and the inverted shooting mode. When the application automatically determines the direction regardless of the user's intention, the determination of the inverted shooting may differ from the user's intention. For example, when taking a picture of a starry sky, the user may take the image with the image capture device 100 facing directly above as shown in FIG. 14(a). Here, the user assumes that the image is taken facing directly above, and that the horizontal correction is performed in the normal shooting mode when the horizontal correction is performed. However, if the image capture device 100 is slightly tilted toward the inverted shooting determination area 1401 side rather than directly above, the application may determine the image as being inverted, and the horizontal correction may be performed in the inverted shooting mode against the user's intention. When the user takes a picture by hand, it is assumed that the image is often determined as being inverted, against the user's intention. Furthermore, when taking pictures multiple times, even if the user thinks that the image was taken facing directly above in the same posture, some images are determined as being inverted shooting and some are determined as being inverted shooting, and the images displayed when the horizontal correction is performed on multiple images are mixed. Therefore, it is not appropriate to set the threshold value y=0 for automatically determining whether an image has been taken upside down.

[0078] 14(b), the orientation from the normal orientation until rotation around the x-axis at an angle of 90°+a predetermined angle β is set as a normal photography determination area 1400, and the orientation from rotation at an angle of 90°+β until rotation at an angle of 180° is set as an inverted photography determination area 1401. This makes it possible to reduce the possibility of inverted photography determination (erroneous determination) that is contrary to the user's intention as described above.

[0079] By setting the value of the y component in the gravity direction by the acceleration sensor to sinβ instead of 0 as the threshold value, it is possible to make it difficult for a slight deviation from the straight up posture to be determined as upside down photography contrary to the user's intention, as described above. When the y component in the gravity direction is in the range of -1 to sinβ, it is determined as normal photography, and when the y component in the gravity direction is in the range of sinβ to 1, it is determined as upside down photography. For example, values ​​such as 5° and 10° are adopted for β, but it is not limited to these. By setting β to a number larger than 0, as shown in FIG. 14(b), a normal photography determination area (first area) 1400 determined as normal photography is larger than an upside down photography determination area (second area) 1401 determined as upside down photography.

[0080] In this embodiment, when the inverted shooting determination is performed automatically, the calculated roll correction value may be set to 90°+γ as the threshold value. As described above, the left and right parallax becomes zero when the tilt angle in the roll direction is 90°, and the left and right parallax becomes reversed when the tilt angle exceeds 90°. By performing horizontal correction on an image taken vertically rotated 90° in the roll direction, it is not possible to obtain an image that can be viewed stereoscopically in practice. However, by setting the roll correction value of 90°+γ as the threshold value, it is possible to obtain a result similar to that obtained when determining from the y component of the acceleration sensor described above. The roll correction value described above is calculated from the values ​​of the x component, y component, and z component in the direction of gravity, which are the orientation data, by the orientation detection unit 119 such as the acceleration sensor described above. Therefore, it is synonymous with setting a certain value of the y component in the direction of gravity as the threshold value.

[0081] An example has been described above in which the attitude of the imaging device 100 is read from attitude data recorded in a captured image and whether normal or upside-down shooting is determined, but the attitude of the imaging device 100 may be determined by processing within the imaging device 100 when no image is being captured, for example, during live view of the imaging device. By providing a function for determining the attitude before shooting and notifying the user, the user can know in advance which mode the image should be processed in after shooting, or which mode has been automatically determined.

[0082] 15(a) and (b) are examples of live view images of the imaging device 100. As shown in FIG. 15(a) and (b), a posture determination mark 1501 is displayed in a live view display 1500, so that the user can know the determination result by the imaging device 100 in advance. For example, as shown in FIG. 15(a), if normal shooting is determined, a white diamond mark is displayed, and as shown in FIG. 15(b), if upside-down shooting is determined, a black diamond mark is displayed, so that the user can be informed. Note that the display content for notifying the user of the determination result is not limited to a diamond mark, and may be other marks or characters.

[0083] In addition, at this time, the user may be able to select normal shooting or upside-down shooting when displaying the live view before shooting. For example, if the user selects normal shooting, even if the image capture device 100 automatically determines that the area is upside-down shooting, the image data may be saved with a flag indicating that the image was taken normally. When image conversion is performed on image data flagged as normal shooting by an application, image processing may be performed by prioritizing the flag rather than the posture data.

[0084] Next, in the case where the posture of the image capturing device 100 changes during video capture, if the posture is changed from a normal posture to an inverted posture during video capture, the image capturing device 100 will determine that the posture is inverted midway when referring to posture data in real time. In this case, the right eye image and the left eye image are switched midway through the video, which affects VR viewing. Therefore, it is preferable to set the posture not to be inverted during video capture. For example, the user can capture a video by setting a determination flag for normal posture or inverted posture. Alternatively, the posture may be determined from posture data in a certain frame in the video, and the determination may be applied to all frames.

[0085] In this embodiment, the imaging device 100 is composed of the camera body 110 and the interchangeable lens 200 that is detachable from the camera body 110, but the lens and the camera body may be integrated into an imaging device. In this embodiment, an example in which an image by the first optical system 201R and an image by the second optical system 201L are captured by the same imaging element 111 has been described, but this is not limiting. For example, an imaging sensor that captures an image by the first optical system 201R and an imaging sensor that captures an image by the second optical system 201L may be provided separately.

[0086] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0087] According to each embodiment, it is possible to provide an image processing device, an image processing method, and a program capable of outputting a VR image that can be viewed correctly regardless of posture.

[0088] The disclosure of each embodiment includes the following configurations and methods. (Configuration 1) An acquisition means for acquiring a first image and a second image with parallax, or an image including the first image and the second image with parallax; A control means capable of setting a first mode and a second mode, The control means In the first mode, the first image is output as a right-eye image and the second image is output as a left-eye image, and one image for stereoscopic viewing is output; In the second mode, the image processing device outputs one image for stereoscopic viewing, with the first image as an image for the left eye, the second image as an image for the right eye, and the first image and the second image each as an image rotated 180° around an optical axis. (Configuration 2) 2. The image processing device according to configuration 1, wherein in the image for stereoscopic viewing, the image for the left eye is arranged on the left side, and the image for the right eye is arranged on the right side. (Configuration 3) 2. The image processing device according to configuration 1, wherein in the image for stereoscopic viewing, the image for the left eye is arranged on the upper side, and the image for the right eye is arranged on the lower side. (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the image for stereoscopic viewing is an image converted using equirectangular projection. (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein the image for stereoscopic viewing is a fisheye image including transformation coordinate information and information for determining whether the image is for the left eye or the right eye. (Configuration 6) 6. The image processing device according to any one of configurations 1 to 5, wherein the control means is capable of setting the first mode and the second mode in response to a user's selection. (Configuration 7) the first image or the second image includes pose data; The image processing device according to any one of configurations 1 to 5, wherein the control means determines whether image processing for the first image and the second image is to be performed in the first mode or the second mode based on the posture data. (Configuration 8) 8. The image processing device according to claim 7, wherein a first range of the posture data determined to be in the first mode is larger than a second range of the posture data determined to be in the second mode. (Configuration 9) the first image or the second image includes pose data; The control means obtaining a horizontal correction value based on the attitude data; 9. The image processing device according to any one of configurations 1 to 8, wherein the image for stereoscopic viewing is outputted after being horizontally corrected based on the horizontal correction value. (Configuration 10) The image processing device described in configuration 9, characterized in that a rotation correction angle about the optical axis for the input image of the image for stereoscopic viewing that has been horizontally corrected in the first mode and a rotation correction angle about the optical axis for the input image of the image for stereoscopic viewing that has been horizontally corrected in the second mode differ from each other by 180°. (Configuration 11) An imaging device capable of acquiring a first image and a second image having parallax, A first optical system; A second optical system arranged in parallel with the first optical system; an acquisition means for acquiring one image including the first image and the second image, or the first image and the second image; A control means capable of setting a first mode and a second mode, The control means In the first mode, the first image is output as a right-eye image and the second image is output as a left-eye image, and one image for stereoscopic viewing is output; In the second mode, the imaging device outputs one image for stereoscopic viewing, with the first image as an image for the left eye, the second image as an image for the right eye, and the first image and the second image each as an image rotated 180 degrees around an optical axis. (Configuration 12) the first optical system is disposed on the right side as viewed from the rear side of the imaging device, and the second optical system is disposed on the left side as viewed from the rear side of the imaging device, the first image is formed by the first optical system; 12. The imaging device according to claim 11, further comprising the imaging device, wherein the second image is formed by the second optical system. (Configuration 13) a posture detection unit that detects a posture of the imaging device when the first image and the second image are captured and outputs posture data; 13. The imaging device according to claim 11, wherein the first image and the second image include the posture data. (Configuration 14) 14. The imaging apparatus according to configuration 13, wherein the control means determines whether the imaging apparatus is in the first mode or the second mode based on the attitude data. (Configuration 15) The device further includes an operation means operable by a user, 15. The image processing device according to any one of configurations 11 to 14, wherein the control means sets the first mode and the second mode in response to a user's selection. (Configuration 16) 16. The imaging device according to any one of configurations 12 to 15, further comprising a display means for displaying whether the mode in which the first image and the second image were captured was the first mode or the second mode. (Configuration 17) 17. The imaging device of any one of configurations 12 to 16, wherein at least one of the first image and the second image includes data indicating whether the image was captured in the first mode or the second mode. (Configuration 18) 18. The imaging device according to claim 17, wherein the control means determines, based on the data, whether image processing for the first image and the second image should be performed in the first mode or the second mode. (Method 1) acquiring a first image and a second image with parallax or an image including the first image and the second image with parallax; A control step capable of setting a first mode and a second mode, In the control step, In the first mode, the first image is output as an image for a right eye, and the second image is output as an image for a left eye, so that one image for stereoscopic viewing is output; In the second mode, the image processing method outputs one image for stereoscopic viewing, with the first image as an image for the left eye, the second image as an image for the right eye, and the first image and the second image each rotated 180° around the optical axis. (Configuration 19) A program for causing a computer to execute the image processing method according to Method 1.

[0089] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0090] 10 Image processing device 11 Acquisition method 12 Control Means

Claims

1. An acquisition means for acquiring a first image and a second image with parallax, or an image including the first image and the second image with parallax; A control means capable of setting a first mode and a second mode, The control means In the first mode, the first image is output as a right-eye image and the second image is output as a left-eye image, and one image for stereoscopic viewing is output; In the second mode, the image processing device outputs one image for stereoscopic viewing, with the first image as an image for the left eye, the second image as an image for the right eye, and the first image and the second image each as an image rotated 180° around an optical axis.

2. 2 . The image processing device according to claim 1 , wherein the image for left eye is arranged on the left side of the image for stereoscopic viewing, and the image for right eye is arranged on the right side.

3. 2 . The image processing device according to claim 1 , wherein the image for left eye is arranged on the upper side and the image for right eye is arranged on the lower side in the image for stereoscopic viewing.

4. 2. The image processing device according to claim 1, wherein the image for stereoscopic viewing is an image converted using equirectangular projection.

5. 2 . The image processing device according to claim 1 , wherein the image for stereoscopic viewing is a fisheye image including transformation coordinate information and information for determining whether the image is for the left eye or the right eye.

6. 6. The image processing apparatus according to claim 1, wherein the control means is capable of setting the first mode and the second mode in response to a user's selection.

7. the first image or the second image includes pose data; 6. The image processing device according to claim 1, wherein the control means determines whether image processing for the first image and the second image is to be performed in the first mode or the second mode based on the posture data.

8. 8. The image processing apparatus according to claim 7, wherein a first range of the posture data determined to be in the first mode is larger than a second range of the posture data determined to be in the second mode.

9. the first image or the second image includes pose data; The control means obtaining a horizontal correction value based on the attitude data; 6. The image processing device according to claim 1, wherein the image for stereoscopic viewing is outputted after being horizontally corrected based on the horizontal correction value.

10. 10. The image processing device according to claim 9, wherein a rotation correction angle about an optical axis for the input image of the image for stereoscopic viewing horizontally corrected in the first mode and a rotation correction angle about an optical axis for the input image of the image for stereoscopic viewing horizontally corrected in the second mode differ from each other by 180 degrees.

11. An imaging device capable of acquiring a first image and a second image having parallax, A first optical system; A second optical system arranged in parallel with the first optical system; an acquisition means for acquiring one image including the first image and the second image, or the first image and the second image; A control means capable of setting a first mode and a second mode, The control means In the first mode, the first image is output as a right-eye image and the second image is output as a left-eye image, and one image for stereoscopic viewing is output; In the second mode, the imaging device outputs one image for stereoscopic viewing, with the first image as an image for the left eye, the second image as an image for the right eye, and the first image and the second image each as an image rotated 180° around the optical axis.

12. the first optical system is disposed on the right side as viewed from the rear side of the imaging device, and the second optical system is disposed on the left side as viewed from the rear side of the imaging device, the first image is formed by the first optical system; The imaging device of claim 11 , wherein the second image is formed by the second optical system.

13. a posture detection unit that detects a posture of the imaging device when the first image and the second image are captured and outputs posture data; The imaging apparatus of claim 11 , wherein the first image and the second image include the pose data.

14. 14. The imaging apparatus according to claim 13, wherein the control means determines whether the imaging mode is the first mode or the second mode based on the attitude data.

15. The device further includes an operation means operable by a user, 12. The image processing apparatus according to claim 11, wherein the control means sets the first mode and the second mode in response to a user's selection.

16. 16. The imaging device according to claim 12, further comprising a display means for displaying whether the mode in which the first image and the second image were captured was the first mode or the second mode.

17. 16. The imaging device according to claim 12, wherein at least one of the first image and the second image includes data indicating whether the image was captured in the first mode or the second mode.

18. 18. The imaging apparatus according to claim 17, wherein the control means determines whether image processing for the first image and the second image is to be performed in the first mode or the second mode based on the data.

19. acquiring a first image and a second image with parallax or an image including the first image and the second image with parallax; A control step capable of setting a first mode and a second mode, In the control step, In the first mode, the first image is output as an image for a right eye, and the second image is output as an image for a left eye, so that one image for stereoscopic viewing is output; The image processing method is characterized in that, in the second mode, the first image is an image for the left eye, the second image is an image for the right eye, and the first image and the second image are each an image rotated 180° around the optical axis to output one image for stereoscopic viewing.

20. A program causing a computer to execute the image processing method according to claim 19.

Images