Image processing apparatus, imaging apparatus, image processing method, storage medium, and computer program product

Abstract

The invention discloses an image processing apparatus, an imaging apparatus, an image processing method, a storage medium, and a computer program product. The image processing apparatus includes: an acquisition unit configured to acquire a first image and a second image having parallax, or one image including the first image and the second image; and a control unit configured to output a stereoscopic visual image by setting the first image as a right-eye image and the second image as a left-eye image in a first mode, and output a stereoscopic visual image by setting the second image as a left-eye image in a second mode. And outputting a stereoscopic visual image by setting the first image as a left eye image and the second image as a right eye image, and by rotating the first image and the second image about an optical axis.

Description

Technical Field

[0001] One aspect of the embodiments relates to an image processing apparatus, an imaging apparatus, an image processing method, and a storage medium. Background Art

[0002] Japanese Patent No. 5166650 discloses a camera device that can capture images from two optical systems at the same time, so as to capture two images with parallax using two optical systems and allow these images to be viewed stereoscopically with a virtual reality (VR) device. When viewed with VR goggles, in order to provide not only a three-dimensional sense but also a sense of immersion, the captured moving or still image can have a viewing angle of 180 degrees or more. In order to provide an image with a viewing angle of at least 180 degrees under manufacturing errors, the camera lens can be a lens configured to capture images with a viewing angle of more than 180 degrees. The form of the image input to a VR viewing device such as VR goggles is a form widely expressed in an equidistant cylindrical projection method. As described above, a fisheye lens is used to express an image with a viewing angle of about 180°. Properly correcting the distorted image captured with a fisheye lens using the equidistant cylindrical projection method can display an image close to the image seen by the human eye on a display unit corresponding to each of the left and right eyes.

[0003] Japanese Patent Laid-Open No. 2021-129169 discloses an image processing unit configured to detect a rotational position offset of a captured image and correct the captured image to horizontalize the tilted captured image. The rotational offset amount can be detected using an acceleration sensor in a camera, and the rotational offset of the captured image can be corrected based on the rotational offset amount.

[0004] The camera device disclosed in Japanese Patent No. 5166650 can output a stereoscopic image, which is a set of a left image and a right image formed by a left lens and a right lens on an image sensor. Assume that the lateral direction of a dual-lens camera including two optical systems is the x-axis, the vertical direction is the y-axis, and the depth direction is the z-axis. In this case, for example, when an image is captured in a state of rotational offset about the z-axis and the image is viewed on a head-mounted display (HMD) as a VR viewing device, the HMD displays a horizontal line tilted by the rotational offset. Therefore, VR motion sickness may occur due to the offset between the image displayed by VR and the sensation in the direction of gravity.

[0005] In VR videography, basically, the user can fix the twin-lens camera horizontally, but the image may be taken at a slight tilt. Therefore, the posture (or orientation) of the camera is detected from information such as an acceleration sensor in the camera, the rotation offset is recorded as a rotation offset correction parameter associated with the captured image, and the rotation offset of the captured image is corrected. As a result, a VR image can be created in a high-precision horizontal state. As the rotation offset increases, not only the horizontal parallax (parallax between left and right) between the two optical systems increases, but also the vertical parallax (parallax between top and bottom) between the two optical systems increases. Even if the captured image with a large rotation offset is corrected for the rotation offset, the vertical parallax cannot be removed, so the vertical parallax also affects VR viewing. For example, in an image taken with a twin-lens camera rotated 90 degrees, that is, with a camera in a vertical state, the horizontal parallax becomes zero, and only the vertical parallax exists. Therefore, a VR image in which the 90-degree rotation offset has been corrected cannot provide accurate stereoscopic viewing.

[0006] An image captured by a twin-lens camera rotated 180 degrees, i.e., an inverted camera, has horizontal parallax and no vertical parallax, but a VR image in which the rotation offset of 180 degrees has been corrected cannot be viewed stereoscopically. This is because, in a state of being rotated 180 degrees, the left lens of the twin-lens camera corresponds to the right eye of a person, and the right lens corresponds to the left eye of a person. In other words, in a case where the image from the left lens of the twin-lens camera is output as a left-eye image and the image from the right lens is output as a right-eye image, the VR image cannot be correctly viewed stereoscopically because the horizontal parallax is reversed.

[0007] In VR videography, it may be difficult to horizontally fix the twin-lens camera according to situations such as videography using a drone, low-angle videography using a tripod, and videography with a camera fixed to a ceiling. In addition, the twin-lens camera may be fixed upside down, that is, in a state (posture) rotated 180 degrees. As described above, in the case of correcting the rotational offset using the rotational offset correction parameter recorded in the camera and outputting a horizontal image, a VR image in which the horizontal parallax is inverted may be output. Summary of the invention

[0008] An image processing device according to one aspect of the present disclosure includes: an acquisition unit configured to acquire a first image and a second image with parallax, or an image including the first image and the second image; and a control unit configured to: in a first mode, output a stereoscopic visible image by setting the first image as a right eye image and the second image as a left eye image, and in a second mode, output a stereoscopic visible image by setting the first image as a left eye image and the second image as a right eye image, and by rotating the first image and the second image around an optical axis. A camera device including the above-mentioned image processing device also constitutes another aspect of the present disclosure. An image processing method corresponding to the above-mentioned image processing device also constitutes another aspect of the present disclosure. A storage medium storing a program for causing a computer to execute the above-mentioned image processing method also constitutes another aspect of the present disclosure.

[0009] Further features of various embodiments of the present disclosure will become apparent from the following description of the embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a cross-sectional view of the lens apparatus according to the present embodiment.

[0011] Figure 2 is a block diagram of an image pickup apparatus according to the present embodiment.

[0012] Figure 3 Example images captured by the image capturing apparatus according to the present embodiment are shown.

[0013] Figure 4A and Figure 4B shows the conversion by using the equidistant cylindrical projection method Figure 3 Example images obtained from the images in .

[0014] Figure 5A , Figure 5B and Figure 5C are images captured by the imaging apparatus according to the present embodiment and example configuration diagrams.

[0015] Fig. 6A and Figure 6B It is converted by using the equidistant cylindrical projection method Figure 5C Example images obtained from the images in .

[0016] Fig. 7A and Figure 7B : is a front view of the imaging apparatus in an inverted posture according to the present embodiment and an example of a captured image.

[0017] Fig. 8A , Figure 8B and Figure 8Cis converted using the equidistant cylindrical projection Figure 7B Example images obtained from the images in .

[0018] Fig.9A , Fig. 9B and Fig. 9C is a configuration example of the image processing apparatus (application) according to the present embodiment.

[0019] Fig. 10A and Fig. 10B 1 is an example of a front view of the imaging apparatus tilted about the optical axis and a captured image in the inverted imaging mode according to the present embodiment.

[0020] Fig.11A , Fig. 11B and Fig. 11C is an example image obtained by converting the captured image according to the present embodiment using the equidistant cylindrical projection method.

[0021] Fig.12 is a flowchart of the imaging method according to this embodiment.

[0022] Fig.13A and Fig. 13B is an example operation screen of the image processing apparatus according to the present embodiment.

[0023] Fig.14A and Fig. 14B is a side view of the image pickup apparatus according to the present embodiment in an upward posture.

[0024] Fig.15A and Fig. 15B is an example live view image of the imaging apparatus according to the present embodiment. DETAILED DESCRIPTION

[0025] Hereinafter, the term "unit" may refer to a software context, a hardware context, or a combination of software and hardware contexts. In the software context, the term "unit" refers to functionality, an application, a software module, a function, a routine, an instruction set, or a program that can be executed by a programmable processor (such as a microprocessor, a central processing unit (CPU), or a specially designed programmable device or controller). The memory contains instructions or programs, wherein the instructions or programs, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In the hardware context, the term "unit" refers to a hardware element, circuit, assembly, physical structure, system, module, or subsystem. According to a specific embodiment, the term "unit" may include mechanical, optical, or electronic components, or any combination thereof. The term "unit" may include active (e.g., transistors) or passive (e.g., capacitors) components. The term "unit" may include a semiconductor device having a substrate and other material layers, wherein the other material layers have various conductivity concentrations. The term "unit" may include a CPU or a programmable processor that can execute a program stored in a memory to perform a specified function. The term "unit" may include a logic element (e.g., AND, OR) implemented by a transistor circuit or any other switching circuit. In a combination of software and hardware contexts, the term "unit" or "circuit" refers to any combination of software and hardware contexts as described above. In addition, the term "element," "assembly," "component," or "device" may also refer to a "circuit" that is integrated with or not integrated with a packaging material.

[0026] Now, with reference to the drawings, a detailed description will be given of embodiments according to the present disclosure.

[0027] The lens device (interchangeable lens) according to the present embodiment includes two optical systems (a first optical system and a second optical system) arranged parallel (symmetrically) to each other, and is configured so that two image circles are imaged in parallel on a single image sensor. The two optical systems are arranged horizontally and spaced apart by a predetermined distance (baseline length). When viewed from the image side, the image formed by the right optical system (the first optical system) is recorded as a motion or still image of the right eye, and the image formed by the left optical system (the second optical system) is recorded as a motion or still image of the left eye. When watching the replayed motion or still image using a known three-dimensional display unit or so-called VR goggles, the user's right eye watches the right eye image, and the left eye watches the left eye image. At this time, due to the baseline length of the lens device, an image with parallax is projected to the right eye and the left eye, so the user can get a three-dimensional sense. Therefore, the lens device according to the present embodiment is a lens device for stereoscopic photography that can form two images with parallax using the first optical system and the second optical system.

[0028] refer to Figure 1, a description will be given of an interchangeable lens (lens device) 200 according to the present embodiment. Figure 1 2 is a cross-sectional view showing a schematic configuration of a first optical system 201R and a second optical system 201L of an interchangeable lens 200. The interchangeable lens 200 includes a first optical system 201R and a second optical system 201L. The first optical system 201R and the second optical system 201L can each perform imaging at a viewing angle of 180 degrees or more. Each optical system has, in order from the subject side to the image side, a first optical axis OA1, a second optical axis OA2 approximately orthogonal to the first optical axis, and a third optical axis OA3 parallel to the first optical axis OA1. Along each optical axis, a first lens 211 having a convex surface 211A on the subject side is arranged on the first optical axis OA1, a second lens 221 is arranged on the second optical axis OA2, and third lenses 231 and 231-2 are arranged on the third optical axis OA3. Each optical system further includes a first prism 220 configured to bend a light beam on the first optical axis OA1 and guide the light beam to the second optical axis OA2, and a second prism 230 configured to bend a light beam on the second optical axis OA2 and guide the light beam to the third optical axis OA3. The first optical axis OA1R of the first optical system 201R and the first optical axis OA1L of the second optical system 201L are separated from each other by a base line length L1, and an image having a parallax corresponding to the base line length L1 can be captured.

[0029] Figure 2 FIG. 1 is a block diagram of an imaging device 100 capable of capturing stereoscopic images. Figure 2 As shown, the imaging apparatus 100 includes a camera body 110 and an interchangeable lens 200. The interchangeable lens 200 is attached to and used with the camera body 110. In the present embodiment, the interchangeable lens 200 is an interchangeable lens system that is attachable to and detachable from the camera body 110, but the camera body and the lens apparatus may be integrated with each other.

[0030] The interchangeable lens 200 includes a first optical system 201R and a second optical system 201L constituting an image pickup optical system, and forms an image on the image sensor 111 of the camera body 110. The lens control unit (LENS CTRL) 203 of the interchangeable lens 200 processes values ​​detected by various detectors such as a temperature (TEMP) detector 207 and a focus detector 208, outputs the processed information to the system control unit 117, communicates the processed information with the camera body 110, and controls the image pickup apparatus 100 in cooperation with the camera body 110. The interchangeable lens 200 also includes a memory (storage unit) 204. The memory 204 stores various information detected by the temperature detector 207 and the focus detector 208, and outputs identification information such as lens individual information 205 or manufacturing (MFG) error information 206 to the system control unit 117 in response to a request from the system control unit 117.

[0031] The camera body 110 includes an image sensor 111, an A / D converter 112, an image processing (PROC) unit 113, a display unit 114, an operation unit 115, a recorder 116, a system control unit 117, a posture detector 119, and a camera mount 122. With the interchangeable lens 200 attached to the camera body 110 via the lens mount 202 and the camera mount 122, the system control unit 117 and the lens control unit 203 are electrically connected.

[0032] As subject images, a right eye image formed by the first optical system 201R and a left eye image formed by the second optical system 201L are formed side by side on the image sensor 111. The image sensor 111 converts the formed subject image (light signal) into an analog electric signal. The A / D converter 112 converts the analog electric signal output from the image sensor 111 into a digital electric signal (image signal). The image processing unit 113 performs various image processing on the digital electric signal (image signal) output from the A / D converter 112.

[0033] The display unit 114 displays various information. The display unit 114 is implemented by, for example, an electronic viewfinder or a liquid crystal panel. The operation unit 115 has a function as a user interface for a user to issue instructions to the imaging apparatus 100. When the display unit 114 has a touch panel, the touch panel also serves as a part of the operation unit 115.

[0034] The recorder 116 records various data such as image data that has received image processing by the image processing unit 113. The recorder 116 also stores programs. The recorder 116 is realized by, for example, a ROM, a RAM, and a HDD.

[0035] The memory 118 stores individual identification information such as camera individual information 120 and camera manufacturing error information 121 of the imaging apparatus 100. In the manufacturing process of the camera body 110, the memory 118 stores information including, for example, model type information of the camera body 110, pixel number information of the image sensor 111, physical size information of the image sensor 111, manufacturing error information related to the captured image, and the like. In the imaging using the imaging apparatus 100, the memory 118 outputs the above identification information to the system control unit 117. Then, together with the identification information related to the interchangeable lens 200 sent from the lens control unit 203 in the interchangeable lens 200, the identification information is generated as an image. This image (information image) is output to the image processing unit 113, added to the captured image, and recorded in the imaging data. As described above, the image processed by the image processing unit 113 can be written to various storage media such as a flash memory or an HDD by the recorder 116.

[0036] The system control unit 117 centrally controls the entire image pickup apparatus 100. For example, the system control unit 117 is realized by using a CPU. These information are recorded on a medium by the recorder 116 via the system control unit 117. On the other hand, light captured by the image pickup optical system forms an image on the image sensor 111. Generally, the formed image is captured by the image sensor 111, undergoes various processes by the A / D converter 112 and the image processing unit 113, and is then written as a captured image.

[0037] A description will now be given of an example fisheye image captured by the image pickup apparatus 100 and an example image converted into an equirectangular projection method. In the case of capturing an image with an image pickup apparatus having a single general lens optical system, an image inverted by 180 degrees is formed on an image sensor. When converting the image into a normal image, a 180-degree inversion process is performed to adjust the vertical direction.

[0038] Figure 3An example image captured by the imaging device 100 is shown. An image captured by the interchangeable lens 200 (e.g., image 300) is imaged on a single image sensor 111 by a first optical system 201R corresponding to the right eye in normal imaging and a second optical system 201L corresponding to the left eye, and the image is inverted 180 degrees for each optical system. In order to adjust the top and bottom of the captured image, the entire captured image is inverted 180 degrees, similar to a general optical system. At this time, an image 300 is output in which the image capture result of the first optical system corresponding to the right eye in normal imaging (right-eye fisheye image 301R) is captured on the left side of the captured image, and the image capture result of the second optical system corresponding to the left eye in normal imaging (left-eye fisheye image 301L) is captured on the right side of the captured image.

[0039] Figure 4A and Figure 4B Shows the Figure 3 An example of converting the image 300 in FIG. 1 into an image of the equidistant cylindrical projection method (hereinafter referred to as an “equidistant cylindrical projection image”). Generally, as Figure 4A As shown, an image 400 is formed in which a left-eye equidistant cylindrical projection image 401L and a right-eye equidistant cylindrical projection image 401R are arranged side by side, and the image 400 can be displayed on a VR device such as an HMD. Alternatively, as Figure 4B As shown, an image 410 may be formed with a left-eye equidistant cylindrical projection image 401L at the top and a right-eye equidistant cylindrical projection image 401R at the bottom. Here, the imaging data includes the lens individual information 205 and the manufacturing error information 206, and based on the lens individual information 205 and the manufacturing error information 206, Figure 3 The fisheye camera data (image 300) can be converted into an equirectangular projection image (image 400).

[0040] The HMD can determine whether the image is a left-eye image or a right-eye image based on a fixed layout such as horizontal arrangement or vertical arrangement, and can display each image at the left-eye or right-eye display position of the HMD. According to the image player in the HMD, the user can set the left display and the right display. The user also enables the HMD to directly read the fisheye image (image 300). In this case, the grid data can be associated with the fisheye image as metadata in the image, and the image can be displayed on the HMD according to the grid data. The grid data includes a grid projected onto a virtual sphere, and is the transformed coordinate information in which the coordinates on the virtual sphere are associated with the center coordinates and the peripheral coordinates of the fisheye image. The method adds metadata to the image so that the left-eye fisheye image 301L is associated with the left-eye grid and the right-eye fisheye image 301R is associated with the right-eye grid. The grid data including the lens individual information 205 and the manufacturing error information 206 can provide a more accurate and correct VR display. As described above, whether it is a fisheye image or an equidistant cylindrical projection image, setting the left-eye image and the right-eye image based on the image arrangement and the grid data enables the image to be VR displayed on the HMD.

[0041] Here, VR display (VR view) is a display method (display mode) that can change the display range of an image in a viewing range corresponding to the posture (or orientation) of the display device for displaying a VR image. VR display includes a "single-lens VR display (single-lens VR view)" for displaying a single image by performing a transformation (for distortion correction) that maps the VR image to a virtual sphere. VR display also includes a "dual-lens VR display (dual-lens VR view)" for performing a transformation that maps a left-eye VR image and a right-eye VR image onto a virtual sphere and for displaying them side by side in left and right areas. "Dual-lens VR display" can provide stereoscopic viewing using a left-eye VR image and a right-eye VR image that have parallax with each other.

[0042] In any type of VR display, for example, in the case where a user wears a display device such as an HMD, an image is displayed in a viewing range corresponding to the orientation of the user's face. For example, it is assumed that in a VR image, at a certain time, an image having a viewing range centered on 0 degrees in the horizontal direction (a specific direction, such as north) and 90 degrees in the vertical direction (90 degrees relative to the zenith, i.e., the horizontal direction) is displayed. In the case where the posture of the display device is reversed from this state (for example, by changing the display surface from south to north), the display range of the same VR image is changed to an image having a viewing range of 180 degrees in the horizontal direction (opposite direction, for example, south) and 90 degrees in the vertical direction. That is, when the user turns his / her face from north to south (i.e., turns back) while wearing the HMD, the image displayed on the HMD also changes from a north image to a south image. The VR image captured with the interchangeable lens 200 according to the present embodiment includes a VR 180 image obtained by capturing the front in a range of approximately 180 degrees, and does not include an image in a range of approximately 180 degrees in the rear. In the case where the VR 180 image is VR displayed and the posture of the display device is changed to the side where the image does not exist, a blank area is displayed.

[0043] By VR displaying the VR image in this way, the user visually feels as if he is inside the VR image (in the VR space). The VR image display method is not limited to the method of changing the posture of the display device. For example, the display range can be moved (scrolled) in response to user operations via a touch panel or direction buttons, etc. In addition, during VR display (in the "VR view" display mode), in addition to changing the display range due to posture changes, the display range can also be changed in response to touch movement on the touch panel, dragging operations with a mouse, etc., pressing direction buttons, etc. A smartphone attached to VR goggles (head-mounted adapter) is a type of HMD.

[0044] Generally, imaging using an imaging device capable of three-dimensional imaging such as the imaging device 100 requires fixing or holding two lenses so that the two lenses are arranged horizontally. In the case where a fisheye image captured in an accurate horizontal state is converted into an equirectangular projection image or embedded mesh data, and the user views those images as VR images displayed on the HMD, he can enjoy excellent VR viewing.

[0045] However, in the case where an image captured while the imaging apparatus 100 is slightly tilted during imaging is subjected to image processing similar to the above-described image processing and the image is VR-displayed on the HMD, even if the postures of the viewer and the HMD are correct, the image displayed by VR is tilted by the tilt amount during imaging. As a result, the posture of the viewer and the posture of the VR display are different, and the viewer is likely to experience a phenomenon called VR sickness that causes motion sickness symptoms due to inconsistency of sensation.

[0046] Therefore, according to the present embodiment, the camera device 100 includes a posture detector 119, which is configured to detect the posture of the camera device 100 during camera shooting and output posture data, and the camera device 100 can detect the tilt during camera shooting and perform horizontal correction on the tilt. The posture detector 119 is, for example, an acceleration sensor or a gyroscope sensor. The camera device 100 obtains posture data (tilt correction value (horizontal correction value or horizontalization value) obtained from the posture detector 119). The posture data is embedded in the camera data and associated with the camera data. When converting the captured image into an equidistant cylindrical projection image, in addition to the lens individual information 205 and the manufacturing error information 206, the camera device 100 also uses the posture data for correction and outputs a VR image after horizontal correction.

[0047] Figure 5A , Figure 5B and Figure 5C 1 is a configuration diagram of the imaging device 100 and an example captured image. Figure 5A As shown, it is assumed that the x-axis is the horizontal direction of the dual-lens imaging device 100 having two optical systems (i.e., the first optical system 201R and the second optical system 201L), the y-axis is the vertical direction, and the z-axis is the depth direction. Figure 5B As shown in the figure, when the image is taken in a state where the image is tilted at an angle θ relative to the horizontal direction in the rotation direction (roll direction) around the z-axis, the output Figure 5C 100. That is, in the image (captured image) 500, the fisheye image 501R captured by the first optical system 201R and the fisheye image 501L captured by the second optical system 201L are output as captured images tilted by an angle θ. For example, the angle θ can be calculated by detecting the gravity direction G using the posture detector 119 such as an acceleration sensor provided in the imaging device 100. In the case where the image is converted into an equidistant cylindrical projection image, as shown in FIG. Fig. 6A As shown, the output image 600 includes tilted equirectangular cylindrical projection images 601R and 601L as VR images.

[0048] Here, the posture of the imaging apparatus 100 is detected by a posture detector 119 such as an acceleration sensor, and posture data indicating that the imaging apparatus 100 is tilted at an angle θ in the roll direction is written as metadata into the image 500. By performing correction based on this posture data, as shown in FIG. Figure 6B As shown, image 610 is output as a VR image including horizontally corrected equirectangular projection images 611R and 611L. When image 500 is converted into an equirectangular projection image, the tilt correction (horizontalization or horizontal correction) is performed at the same time, and image 610 can be output from image 500. Viewing such a horizontally corrected image on an HMD can suppress VR motion sickness.

[0049] In the case of mesh data, information about the tilt angle θ in the roll direction is associated as metadata, which can correct the image by the angle θ in the roll direction and display the image in VR on the HMD. However, in the case where the angle θ is large, such as Figure 5B As shown, not only the horizontal parallax H but also the vertical parallax V appears. The horizontal parallax H is also smaller than the baseline length L1 which is the parallax in the horizontal state. In addition, when the posture of the camera 100 is tilted 90 degrees in the roll direction, the horizontal parallax H becomes zero, and when the posture of the camera 100 is tilted at an angle greater than 90 degrees, the horizontal parallax H becomes the parallax in the opposite direction.

[0050] exist Figure 6B In the image 610 shown to have been horizontally corrected based on the posture data, the tilt in the roll direction has been corrected, but the vertical parallax still exists in the image due to the vertical parallax V. In the case where the vertical parallax is small, VR motion sickness may not occur, but as the vertical parallax increases, the user is more likely to feel VR motion sickness, so the subject can be kept as horizontal as possible during shooting.

[0051] Reference now 7A to 8C , a description will be given of performing imaging with the imaging apparatus 100 maintained in an inverted state (inverted posture). Fig. 7A and Figure 7B is an example of a front view of the imaging device in an inverted posture and the captured image. Fig. 8A , Figure 8B and Figure 8C It is converted by using the equidistant cylindrical projection method Figure 7B An example of an image obtained with the image in .

[0052] exist Fig. 7A The camera image at an angle of 180 degrees in the roll direction in the state shown provides Figure 7B The inverted image (photographed image) 700 is shown. In the case where the image 700 is directly converted into an equidistant cylindrical projection image, the output Fig. 8A The inverted image (VR image) 800 shown in FIG. Similarly, when the angle θ in the roll direction is corrected to 180 degrees based on the posture data, the output Figure 8B Horizontally corrected image 810 is shown. Here, corrected image 811L is an image obtained by correcting the rotation of image 801L by 180 degrees, and corrected image 811R is an image obtained by correcting the rotation of image 801R by 180 degrees. However, these corrections result in horizontal parallax inversion.

[0053] That is, in order to see the scene in three dimensions with the correct posture and parallax Fig. 7A 1. For an image captured in the posture shown in FIG. 1, the first optical system 201R will be used for the left eye image, and the second optical system 201L will be used for the right eye image. However, in the case where a similar algorithm is used to correct the tilt of the image (the image is horizontalized), the image 701R captured by the first optical system 201R is output as image 801R in the image 800 converted by the equidistant cylindrical projection method. In addition, it is output as image 811R in the image 810 that has been rotated and corrected by 180 degrees, and is used as the right eye image. Similarly, the image captured by the second optical system 201L is output as image 801L in the image 800 converted by the equidistant cylindrical projection method. In addition, it is output as image 811L in the image 810 that has been rotated and corrected by 180 degrees, and is used as the left eye image.

[0054] In other words, the image 400 captured in a normal posture and converted into an equirectangular projection image and the image 810 captured in an inverted posture, rotationally corrected by 180 degrees, and converted into an equirectangular projection image are left-right inverted images. In order to obtain a VR image with correct parallax, as in the image 820, the image 811R obtained by rotationally correcting the image 801R by 180 degrees is used for the left eye, and the image 811L obtained by rotationally correcting the image 801L by 180 degrees is used for the right eye.

[0055] In this manner, a specific algorithm for performing rotation correction of 180 degrees, exchanging the left-eye and right-eye images, and outputting the result is a method for rotation correction of image 801R by 180 degrees and placing the result on the left side of image 820 and for rotation correction of image 801L by 180 degrees and placing the result on the right side of image 820. Another method is to regard the pre-correction image 800 as a single image, rotate the entire image 800 by 180 degrees, and output image 820. The above-mentioned image processing can be performed simultaneously with the process of converting the image 700 into an equidistant cylindrical projection image, and image 820 can be output from image 700.

[0056] When associating the grid data to the fisheye image, image 701R is associated with data indicating that it is a right eye image, and image 701L is associated with data indicating that it is a left eye image. Data indicating that the image has been rotated 180 degrees in the roll direction is also associated. Even when the image is corrected 180 degrees in the roll direction and displayed, the data that image 701R is a right eye image and image 701L is a left eye image is maintained, and they are displayed on the HMD with inverted parallax. In other words, in order to convert a VR image that includes a left eye image and a right eye image and has been shot upside down into an image that can be viewed in the correct posture, a 180-degree rotation correction and an exchange of the left eye image and the right eye image are required. In other words, data that image 701R is a left eye image and image 701L is a right eye image is written.

[0057] Therefore, the imaging device 100 according to the present embodiment has a normal imaging mode (first mode) and an inverted imaging mode (second mode). In the normal imaging mode, for example, the image captured by the first optical system 201R is output as a right eye image, and the image captured by the second optical system 201L is output as a left eye image. In the inverted imaging mode, the image captured by the first optical system 201R is output as a left eye image, and the image captured by the second optical system 201L is output as a right eye image.

[0058] The image processing device according to the present embodiment runs on an application (software) that supports still and moving images captured using a digital camera (camera) including a dual lens (VR 180 lens), and is generally implemented by a terminal such as a personal computer (PC) or a smart phone, etc. The above application can be installed in the digital camera.

[0059] Reference Fig.9A , Fig. 9B and Fig. 9C , a description will be given of a configuration example of an image processing device (application) according to the present embodiment. Fig.9A and Fig. 9B is a configuration example of an image processing device (application). Fig. 9C It is a block diagram of an image processing device.

[0060] exist Fig.9A In the configuration shown, the image data captured by the camera device 100 is transmitted to a terminal 910 such as a PC, a smart phone, or a tablet computer via wireless communication 901 such as WIFI or Bluetooth (registered trademark) or a cable (wire) 902. Alternatively, the camera data stored in the memory card 903 can be loaded by the terminal 910. This configuration is to output a VR image using an application (image processing device) installed on the terminal 910.

[0061] exist Fig. 9B In the configuration shown, an application (image processing device) is installed in the camera device 100 and can be operated by the operation unit 115 to output a VR image from the camera data, display the VR image on the display unit 114, and record the VR image on a medium through the recorder 116.

[0062] like Fig. 9C As shown, the image processing device 10 according to the present embodiment includes an acquisition unit 11, a control unit 12 and an operation unit 13. The acquisition unit 11 acquires a first image and a second image with parallax, or a single image including a first image and a second image with parallax. The control unit 12 can set a first mode and a second mode. In the first mode, the control unit 12 outputs a single stereoscopic visible image (equidistant cylindrical projection image) by setting the first image as a right eye image and the second image as a left eye image. In the second mode, the control unit 12 outputs a single stereoscopic visible image by setting the first image as a left eye image and the second image as a right eye image and by setting the first image and the second image as images rotated 180 degrees around the optical axis. The operation unit 13 can be operated by a user, but is not necessary according to the present embodiment. The control unit 12 can set the first mode and the second mode according to the user's selection. However, the present embodiment is not limited to this example, and the control unit 12 can set the first mode and the second mode based on a predetermined condition that does not depend on the user's selection.

[0063] Reference now FIG. 10A to FIG. 11C , a description will be given of correcting a slight tilt of the image pickup apparatus 100 in the inverted image pickup mode. Fig. 10A and Fig. 10B 1 and 2 are examples of a front view and a captured image of the image pickup apparatus 100 tilted about the optical axis in the inverted image pickup mode, respectively. Fig.11A , Fig. 11B and Fig. 11C An example of an image obtained by converting a captured image using the equidistant cylindrical projection method is shown.

[0064] like Fig. 10A As shown, it is assumed that the image is taken by the camera device with the posture of the inverted camera mode in which the first optical system 201R is set to the left eye and the second optical system 201L is set to the right eye and tilted at an angle α. In other words, the image is taken in a posture after being rotated from the horizontal state at an angle θ=180° and in the +α roll direction. In this case, as Fig. 10B As shown in FIG. 1 , an image 1000 approximately inverted and tilted by an angle α is obtained. In the case where the image 1000 is directly converted into an equidistant cylindrical projection image, as shown in FIG. Fig.11A As shown, a VR image 1100 that is approximately inverted and tilted by α is output.

[0065] because Fig. 10A The posture shown is approximately inverted, so the image captured by the first optical system 201R will be output as the left eye image, and the image captured by the second optical system 201L will be output as the right eye image. Therefore, in the case where the rotation is corrected 180 degrees in the roll direction in the inverted camera mode, the output Fig. 11B That is, image 1111R obtained by correcting image 1101R by 180 degrees is set as a left eye image and image 1111L obtained by correcting image 1101L by 180 degrees is set as a right eye image, thereby outputting image 1110.

[0066] However, if only 180° rotation correction is performed in the inverted camera mode, the image is still tilted by angle α. Therefore, in order to create a VR viewable image with a correct posture, the rotation of angle α will be corrected to create a horizontal VR image. Simply applying the traditional tilt correction (horizontalization) algorithm does not work properly. This is because, in Fig. 10A The camera data captured in the posture shown contains posture data detected by the posture detector 119, and the posture data includes posture information indicating that the image is tilted at an angle θ in the roll direction. Here, if a conventional tilt correction algorithm is simply applied to correct the tilt of the image 1110, the angle θ is corrected in the roll direction. In order to correctly level and output Fig. 11C The image 1120 is shown to be horizontalized in the roll direction using an angle α obtained by subtracting 180 degrees from the angle θ.

[0067] For ease of understanding, the inversion conversion in the inverted camera mode and the tilt correction other than the inversion conversion have been described in order so far, but the present embodiment is not limited to this example. When correcting the tilt in the inverted camera mode, for example, when converting the image 1000 into an equidistant cylindrical projection image, simultaneous processing is achieved, and the image 1120 can be output from the image 1000.

[0068] Reference Fig.12 , a description will be given of an image capturing method using an image processing device having an inverted image capturing mode and a tilt correction function according to the present embodiment. Fig.12 is an example of a flowchart illustrating an image capturing method according to the present embodiment.

[0069] First, in step S1200, the image processing device (application) reads an image file. The image read here is a fisheye image such as the images 300, 500, 700, and 1000. At this time, the posture data and imaging information included in the image are also read as metadata.

[0070] Next, in step S1201, the image processing device determines whether the image read in step S1200 is an image captured in the inverted camera mode (whether the image is captured in the normal camera mode or in the inverted camera mode). Here, the method for determining the inverted camera mode can be selected by the user using the operation unit in the image processing device, or can be automatically determined by the application based on the image information. The method for determining the inverted camera mode will be described later.

[0071] In the case where it is determined in step S1201 that the mode is the inverted camera mode, the flow proceeds to step S1202. In step S1202, the image processing device determines whether to perform tilt correction. Similarly, the determination of tilt correction can be selected by the user using the operation unit of the image processing device, or automatically determined by the application based on image information.

[0072] In the case where it is determined in step S1202 that tilt correction is to be performed, the flow proceeds to step S1203. In step S1203, since tilt correction is further performed in the inverted camera mode, as described above, the correction value in the roll direction in the posture data is a value minus 180°. Next, in step S1204, the image processing device performs tilt correction using the correction value in the roll direction that has been adjusted to a value minus 180°. Since the image processing device is in the inverted camera mode, the image is converted into an equidistant cylindrical projection image (stereoscopic visible image) by rotating the image 180 degrees in the roll direction and by setting the image from the first optical system 201R as the left eye image and the image from the second optical system 201L as the right eye image. In this embodiment, the rotation correction angle of the stereoscopic visible image that has been horizontalized in the normal camera mode relative to the input image around the optical axis and the rotation correction angle of the stereoscopic visible image that has been horizontalized in the inverted camera mode relative to the input image around the optical axis differ from each other by 180 degrees.

[0073] By using the algorithm described above in steps S1203 and S1204, the tilt correction algorithm and the algorithm for processing in the inverted imaging mode can be used as they are by simply changing the argument of the correction value in the roll direction to a value minus 180 degrees.

[0074] Another solution for tilt correction in the inverted camera mode is to output an equidistant cylindrical projection image by using a tilt correction value higher than 180° for tilt correction, setting the image from the first optical system 201R as the left eye image, and setting the image from the second optical system 201L as the right eye image. However, this case requires two algorithms for the inverted camera mode because the algorithm for the inverted camera mode itself changes depending on whether tilt correction is performed. In other words, in the case where tilt correction is to be performed, since the tilt correction process provides the correct posture, it is not necessary to rotate 180 degrees in the roll direction for the inverted camera mode. On the other hand, in the case where tilt correction is not performed, it is necessary to rotate 180 degrees in the roll direction for the inverted camera mode to obtain the correct posture.

[0075] In the case where it is determined in step S1202 that tilt correction is not to be performed, the flow proceeds to step S1205. In step S1205, the image processing apparatus rotates the image 180 degrees in the roll direction, sets the image from the first optical system 201R as the left eye image and the image from the second optical system 201L as the right eye image without using a correction value, and converts the image into an equidistant cylindrical projection image.

[0076] Both steps S1204 and S1205 are in the inverted camera mode, so the image is rotated 180 degrees in the roll direction, and the image from the first optical system 201R is set as the left eye image, and the image from the second optical system 201L is set as the right eye image, and these images are output. During this process, the left eye image and the right eye image can be processed and pasted together to output the final equidistant cylindrical projection image. Alternatively, an algorithm similar to the algorithm of the normal camera mode can be used to create a horizontally arranged equidistant cylindrical projection image, the equidistant cylindrical projection image is regarded as a single image, the entire image is rotated 180 degrees, and then the final equidistant cylindrical projection image is output. Based on the image captured in the inverted camera mode, the image processing device according to the present embodiment can finally output the image from the first optical system 201R as the left eye image and output the image from the second optical system 201L as the right eye image, wherein the left eye image and the right eye image are rotated 180 degrees in the roll direction. Any processing can be done inside the algorithm for images captured in inverted camera mode, as long as such a result can be obtained.

[0077] In the case where it is determined in step S1201 that it is not an inverted camera mode, the process proceeds to step S1206. In step S1206, similarly to step S1202, the image processing device determines whether to perform tilt correction. In the case where it is determined that tilt correction is to be performed, the process proceeds to step S1207. In step S1207, the image processing device obtains the posture data included in the image and obtains the correction value. Next, in step S1208, the image processing device uses the obtained correction value to correct the image horizontally, and sets the image from the first optical system 201R as the right eye image, sets the image from the second optical system 201L as the left eye image, and converts the image into an equidistant cylindrical projection image.

[0078] If it is determined in step S1206 that tilt correction is not to be performed, the flow proceeds to step S1209. In step S1209, the image processing apparatus sets the image from the first optical system 201R as a right eye image and the image from the second optical system 201L as a left eye image, and converts these images into equidistant cylindrical projection images.

[0079] As described above, according to the present embodiment, the application runs on the algorithm flow chart.

[0080] Reference now Fig.13A and Fig. 13B , a description will be given of an operation screen of the image processing apparatus (application) according to the present embodiment. Fig.13A and Fig. 13B is a sample operation screen of an image processing device. Fig.13A On the application window screen 1300 shown, an image 1301R obtained by converting the image from the first optical system 201R into an equirectangular projection image is displayed as a left-eye image, and an image 1301L obtained by converting the image from the second optical system 201L into an equirectangular projection image is displayed as a right-eye image.

[0081] A tilt correction check box 1302 and an inverted image pickup check box 1303 are displayed on the window screen 1300. The user can switch whether to perform tilt correction by switching the check box 1302 on and off. The user can switch whether to use the inverted image pickup mode by switching the check box 1303 on and off.

[0082] For example, Fig. 13B As shown in FIG. 13, in the case where the check box 1303 is checked, the image 1301R obtained by converting the image from the first optical system 201R is displayed as the left eye image, and the image 1301L obtained by converting the image from the second optical system 201L is displayed as the right eye image. For example, the image 1301R is displayed as the left eye image according to whether the check boxes 1302 and 1303 are turned on or off. Fig.12 The judgments in steps S1201, S1202 and S1206.

[0083] In the above description, the user can select the inverted imaging mode and the tilt correction, but the application can determine whether to perform each of them based on an image including posture data during imaging.

[0084] As mentioned above, assuming the x-axis is Figure 5A The y-axis is the horizontal direction of the camera 100 shown, the y-axis is the vertical direction, and the z-axis is the depth direction, and the camera 100 can use a posture detector 119 such as an acceleration sensor to obtain the gravity direction G as posture data. Figure 5A In the case of the normal posture shown, the gravity direction G is defined as x=0, y=-1, and z=0. In the case of the completely inverted posture, the gravity direction G is defined as x=0, y=1, and z=0. The judgment of the inverted posture can use a threshold somewhere between the normal posture and the completely inverted posture, and judge whether the value exceeds the threshold. Here, the x component, y component, and z component in the gravity direction G each take a value in the range of -1 or greater than -1 and 1 or less than 1, and the scalar of the three-dimensional vector including these components is 1. In other words, the individual components are output so that

[0085] A description will now be given of a threshold value for determining an inverted posture. For example, it is assumed that, with respect to the y-axis of the imaging apparatus 100, the value of the y-component indicating the gravity direction G is negative when the imaging apparatus 100 is in a normal posture, and the value of the y-component indicating the gravity direction G is positive when the imaging apparatus 100 is in an inverted posture. Then, by determining whether the y-component is positive or negative, that is, using a threshold value of 0 for the y-component, an inverted posture can be determined.

[0086] Fig.14A and Fig. 14B is a side view of the imaging apparatus 100 in an upward posture. Fig.14A In the above, for simplicity, the posture of the camera 100 rotating in the upward direction from the normal posture around the x-axis is considered. In the case where y=0 is set as the threshold value, as shown in FIG. Fig.14A As shown, the normal image pickup determination area 1400 ranges from the normal posture of the image pickup apparatus 100 to the posture for directly above image pickup, and the inverted image pickup determination area 1401 ranges from the posture for directly above image pickup to the completely inverted posture. In other words, in terms of angles around the x-axis, the normal image pickup determination area 1400 ranges from the normal posture to the posture rotated by 90° around the x-axis, and the inverted image pickup determination area 1401 ranges from the posture rotated by 90° around the x-axis to the posture rotated by 180° around the x-axis.

[0087] When the image captured in the normal imaging determination area 1400 is horizontalized, the image is horizontally aligned in the z-axis direction as the forward imaging direction of the imaging device 100. Fig.14A In other words, the forward imaging direction of the imaging device 100 is corrected to Fig.14A in the left direction.

[0088] When the image captured in the posture within the range of the inverted image capturing determination area 1401 is horizontalized, the image is horizontally positioned in the z-axis direction as the forward image capturing direction of the image capturing apparatus 100. Fig.14A In other words, the forward imaging direction of the imaging device 100 is corrected to Fig.14A in the right direction.

[0089] When shooting in a posture facing almost straight up, if the tilt correction is performed between the normal shooting mode and the inverted shooting mode, the direction of the displayed VR image is reversed. In the case where the application makes an automatic judgment regardless of the user's intention, the judgment of inverted shooting may be different from the user's intention. For example, when shooting a starry sky, the user can Fig.14A As shown, the camera device 100 is pointed directly upward. Now assume that the user is taking pictures while facing directly upward, and performs tilt correction in the normal camera mode. However, if the camera device 100 is slightly tilted from directly above toward the inverted camera judgment area 1401, the application can determine that the camera device 100 is in an inverted posture, and may perform tilt correction in the inverted camera mode against the user's intention. In the case where the user performs handheld camera shooting, it is more likely that the inverted camera judgment is performed against the user's intention. In multiple camera shots, even if the user believes that he is facing directly upward in the same posture, some of the captured images may be judged to belong to normal camera shooting, while others may be judged to belong to inverted camera shooting. As a result, in the case of displaying multiple horizontalized images, these displayed images include images horizontalized by different methods. Therefore, it is inappropriate to set the threshold value to y=0 for automatic inverted camera judgment.

[0090] Therefore, if Fig. 14B As shown, the present embodiment provides a normal image pickup determination area 1400 from a normal posture to a posture at an angle of 90 degrees + a predetermined angle β rotated around the x-axis, and an inverted image pickup determination area 1401 from a posture at an angle of (90 + β) degrees rotated around the x-axis to a posture at an angle of 180 degrees rotated around the x-axis. Therefore, as described above, the possibility of an inverted image pickup determination (false determination) contrary to the user's intention can be small.

[0091] By setting the value of the y component in the gravity direction determined by the acceleration sensor to sinβ instead of 0, as described above, in the case of a slight deviation from the directly above posture, it is possible to prevent the judgment of inverted photography that goes against the user's intention. In the case where the y component in the gravity direction is in the range of values ​​from -1 to sinβ, it is judged as normal photography, and in the case where the y component in the gravity direction is in the range of values ​​from sinβ to 1, it is judged as inverted photography. For example, a value such as 5° or 10° is used as β, but the value is not limited to this example. By setting β to a value greater than 0, such as Fig. 14B As shown, a normal imaging determination area (first range) 1400 for determining normal imaging is larger than an inverted imaging determination area (second range) 1401 for determining inverted imaging.

[0092] In the automatic inverted camera judgment, the present embodiment can set the calculated roll correction value 90°+γ as a threshold. As described above, when the tilt angle in the roll direction is 90 degrees, the horizontal parallax becomes zero, and when the tilt angle exceeds 90 degrees, the sign of the horizontal parallax becomes reversed. An actually stereoscopic visible image cannot be obtained simply by horizontalizing an image taken in a vertical posture rotated 90 degrees in the roll direction. However, setting the roll correction value 90°+γ as a threshold can provide an effect similar to the effect of judging based on the y component of the acceleration sensor described above. The above roll correction value is calculated by a posture detector 119 such as the above-mentioned acceleration sensor based on the values ​​of the x component, y component and z component in the gravity direction in the posture data. Therefore, it is equivalent to setting a certain value of the y component in the gravity direction as a threshold.

[0093] An example has been described so far in which the posture of the imaging device 100 is read from the posture data recorded in the captured image and whether it is normal imaging or inverted imaging is determined. The posture of the imaging device 100 may also be determined by processing within the imaging device 100 in a pre-imaging state (for example, in a live view state). By adding a function of determining the posture in advance to imaging and notifying the user, etc., the user can recognize in advance which mode should be used to process the image after imaging, or which mode should be determined automatically.

[0094] Fig.15A and Fig. 15B is an example live view image of the camera apparatus 100. Fig.15A and Fig. 15B As shown in FIG. 1 , the user can recognize the judgment result of the imaging apparatus 100 in advance by displaying the posture judgment mark 1501 in the live view display 1500. For example, Fig.15A As shown in FIG. 1 , when it is determined that the image is taken normally, a white diamond mark is displayed, and Fig. 15BAs shown, in the case where it is determined that the camera is inverted, a black diamond mark is displayed. Thus, the user can recognize the determination result. The display content for notifying the user of the determination result is not limited to the diamond mark, and may be other marks or characters.

[0095] At this time, the user can select normal shooting or inverted shooting during the live view display before shooting. For example, in the case where the user selects normal shooting, even if the camera 100 automatically determines that it is in the inverted shooting determination area, a flag indicating normal shooting can be added to the shooting data and saved. When the shooting data with the normal shooting flag is converted by the application, the flag can be prioritized rather than the posture data for subsequent image processing.

[0096] Next, in the case where the posture of the camera 100 changes during the motion image shooting (for example, from the normal posture to the inverted posture) and the posture data is referred to in real time, the camera 100 determines that it is in an inverted posture during the shooting. Then, during the motion image shooting, the right eye image and the left eye image are exchanged, which affects the VR viewing. Therefore, during the motion image shooting, the posture change may not be set. For example, the user can shoot the motion image by attaching a judgment mark of the normal posture or the inverted posture. Alternatively, the posture can be judged based on the posture data of a certain frame in the motion image, and the judgment can be applied to all frames.

[0097] In the present embodiment, the image pickup apparatus 100 includes a camera body 110 and an interchangeable lens 200 that is attachable to and detachable from the camera body 110, but the image pickup apparatus may be an image pickup apparatus in which the lens and the camera body are integrated. In the present embodiment, the image from the first optical system 201R and the image from the second optical system 201L are captured by the same image sensor 111, but the present embodiment is not limited to this example. For example, an image sensor that captures an image from the first optical system 201R and an image sensor that captures an image from the second optical system 201L may be provided separately.

[0098] Other embodiments

[0099] The computer of a system or device of the present disclosure can also be implemented by reading and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which can also be more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above (one or more) embodiments and / or including one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions in the above (one or more) embodiments and by the following method, wherein the computer of the system or device performs the above method by, for example, reading and executing computer executable instructions from a storage medium to perform one or more functions in the above (one or more) embodiments and / or controlling the one or more circuits to perform one or more functions in the above (one or more) embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read and execute computer executable instructions. For example, these computer executable instructions can be provided to the computer from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage portion of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)). TM etc.), one or more of a flash storage device and a memory card, etc.

[0100] Although the disclosure describes example embodiments, it is to be understood that some embodiments are not limited to the disclosed embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0101] Various embodiments may provide an image processing device that can generate a correctly visible VR image regardless of the posture of the image processing device during imaging.

Claims

1. An image processing device, comprising: an acquisition unit configured to acquire a first image and a second image having parallax, or an image including the first image and the second image; as well as A control unit configured to: In a first mode, a stereoscopic visible image is output by setting the first image as a right-eye image and the second image as a left-eye image, and In the second mode, a stereoscopic visible image is output by setting the first image as a left-eye image and the second image as a right-eye image, and by rotating the first image and the second image around an optical axis.

2. The image processing device according to claim 1, characterized in that The control unit is configured to rotate the first image and the second image 180 degrees around an optical axis in the second mode.

3. The image processing device according to claim 1, characterized in that In the stereoscopically visible image, the left-eye image is arranged on the left side and the right-eye image is arranged on the right side.

4. The image processing device according to claim 1, characterized in that In the stereoscopically visible image, the left-eye image is arranged on a top side and the right-eye image is arranged on a bottom side.

5. The image processing device according to claim 1, characterized in that The stereoscopic visible image is an image converted using an equidistant cylindrical projection method.

6. The image processing device according to claim 1, characterized in that The stereoscopic visible image is a fisheye image including conversion coordinate information and information for determining the left-eye image or the right-eye image.

7. The image processing device according to claim 1, characterized in that The control unit is configured to set the first mode and the second mode according to a user's selection.

8. The image processing device according to claim 1, characterized in that The control unit is configured to obtain posture data of an imaging device that has generated the first image and the second image, Wherein, based on the posture data, the control unit is configured to determine which of the first mode and the second mode is to be used for image processing of the first image and the second image.

9. The image processing device according to claim 8, characterized in that In the gesture data, a first range of gesture data determined to belong to the first mode is wider than a second range of gesture data determined to belong to the second mode.

10. The image processing device according to claim 8, characterized in that The first image or the second image includes the posture data.

11. The image processing device according to claim 1, characterized in that The control unit is configured to obtain posture data of an imaging device that has generated the first image and the second image, The control unit is configured to obtain a tilt correction value based on the posture data, and output a stereoscopically visible image that has been horizontalized based on the tilt correction value.

12. The image processing device according to claim 1, characterized in that A rotation correction angle about the optical axis of the stereoscopically visible image horizontalized in the first mode and a rotation correction angle about the optical axis of the stereoscopically visible image horizontalized in the second mode are different from each other.

13. The image processing device according to claim 12, characterized in that A rotation correction angle of the stereoscopically visible image horizontalized in the first mode and a rotation correction angle of the stereoscopically visible image horizontalized in the second mode relative to the input image about the optical axis differ from each other by 180 degrees.

14. The image processing device according to claim 11, characterized in that The first image or the second image includes the posture data.

15. A camera device configured to acquire a first image and a second image having parallax, the camera device comprising: A first optical system; a second optical system, arranged in parallel with the first optical system; an acquisition unit configured to acquire the first image and the second image having parallax, or an image including the first image and the second image; as well as A control unit configured to: In a first mode, a stereoscopic visible image is output by setting the first image as a right-eye image and the second image as a left-eye image, and In the second mode, a stereoscopic visible image is output by setting the first image as a left-eye image and the second image as a right-eye image, and by rotating the first image and the second image around an optical axis.

16. The imaging device according to claim 15, characterized in that The control unit is configured to rotate the first image and the second image 180 degrees around an optical axis in the second mode.

17. The imaging device according to claim 15, characterized in that When viewed from the back side of the imaging device, the first optical system is arranged on the right side, wherein, when viewed from the back side of the imaging device, the second optical system is arranged on the left side, wherein the first image is formed by the first optical system, and Wherein, the second image is formed by the second optical system. 18 . The imaging apparatus according to claim 15 , further comprising a posture detector configured to detect a posture of the imaging apparatus during capturing of the first image and the second image and output posture data.

19. The imaging device according to claim 18, wherein: The control unit is configured to determine which of the first mode and the second mode is to be used based on the gesture data.

20. The imaging device according to claim 18, wherein: The first image or the second image includes the posture data.

21. The imaging device according to claim 15, further comprising an operation unit operable by a user, It is characterized in that The control unit is configured to set the first mode and the second mode according to a user's selection on the operation unit. 22 . The imaging apparatus according to claim 15 , further comprising a display unit configured to display which of the first mode and the second mode is to be used for image processing of the first image and the second image.

23. The imaging device according to claim 15, characterized in that At least one of the first image and the second image includes data indicating which of the first mode and the second mode is to be used for image processing of the first image and the second image.

24. The imaging device according to claim 18, wherein: Based on the gesture data, the control unit is configured to determine which of the first mode and the second mode is to be used for image processing of the first image and the second image.

25. An image processing method comprising the following steps: Acquire a first image and a second image having parallax, or an image including the first image and the second image; In a first mode, outputting a stereoscopic visible image by setting the first image as a right-eye image and the second image as a left-eye image; and In the second mode, a stereoscopic visible image is output by setting the first image as a left-eye image and the second image as a right-eye image, and by rotating the first image and the second image around an optical axis. 26 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 25 .

27. A computer program product comprising a program for causing a computer to execute the image processing method according to claim 25.

Images