Image processing device, control method for image processing device, and program

Abstract

The objective is to provide an image processing device, a control method for the image processing device, and a program that can reproduce two images with parallax in a stereoscopic viewable manner without any sense of unnaturalness, even if parts of each image are deleted. [Solution] The camera 100 includes an image acquisition means (recording medium 209) that acquires a first image and a second image having parallax between them, which are images of the same subject; an information acquisition means (control unit 203) that acquires tilt information regarding the tilt when the first image and the second image are rotated and tilted around the optical axis of the camera 100 when they are captured; a correction means (control unit 203) that performs rotation correction processing to rotate the first image and the second image in a direction that reduces the tilt based on the tilt information; and an image generation means (image processing unit 208) that generates an array image in which the first image and the second image that have undergone rotation correction processing are arranged side by side.

Description

Technical Field

[0004] , , , ,

[0005] , , , ,

[0001] The present invention relates to an image processing apparatus, a control method of the image processing apparatus, and a program.

Background Art

[0002] There is known a technique (hereinafter referred to as "virtual reality image display technique") for displaying two images (left-eye image and right-eye image) having parallax captured by one camera as a stereoscopic virtual reality (VR) image. For example, Patent Document 1 discloses an apparatus including two optical systems capable of acquiring two images having parallax. The apparatus described in Patent Document 1 reverses the left-right positional relationship of the two images from a state where the two images are arranged side by side. The images obtained by this reversal are displayed in correspondence with the two optical systems.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the virtual reality image display technique, when two images are respectively captured with the camera tilted, a part of each of the two images may be deleted (erased) and displayed. In this case, the virtual reality image may cause a sense of discomfort during reproduction (viewing) due to the deletion of a part of each image.

[0005] The present invention has been made in view of the above problems. An object of the present invention is to provide an image processing apparatus, a control method of the image processing apparatus, and a program capable of reproducing stereoscopic viewing without a sense of discomfort even when a part of each of two images having parallax is deleted.

Means for Solving the Problems

[0006] To achieve the above objective, the image processing apparatus of the present invention is characterized by comprising: an image acquisition means for acquiring a first image and a second image having parallax between them, which are images of the same subject; an information acquisition means for acquiring tilt information relating to the tilt when the imaging device rotates and tilts around the optical axis of the imaging device when the first image and the second image are captured by the imaging device; a correction means for performing rotation correction processing on the first image and the second image, respectively, based on the tilt information, in a direction that reduces the tilt; and an image generation means for generating an array image in which the first image and the second image, which have undergone the rotation correction processing, are arranged side by side. [Effects of the Invention]

[0007] According to the present invention, even if parts of two images having parallax are erased, they can be reproduced in a stereoscopic manner without any sense of discomfort. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing an example of the overall configuration when the image processing apparatus according to the first embodiment is applied to a digital camera. [Figure 2] A block diagram showing an example of a digital camera hardware configuration. [Figure 3] This figure shows an example of the mechanical configuration of a lens. [Figure 4] This flowchart shows the processes performed by a camera with a lens attached. [Figure 5] This figure shows an example of an image taken by a camera with a lens attached. [Figure 6] This flowchart shows the detailed processing performed in step S409 (subroutine) of the flowchart shown in Figure 4. [Figure 7] This is a conceptual diagram illustrating an example of the process performed in step S409 of the flowchart shown in Figure 4. [Figure 8A]This is a conceptual diagram illustrating an example of the process performed in step S410 of the flowchart shown in Figure 4. [Figure 8B] Figure 8B is a conceptual diagram illustrating an example of the effects in this embodiment. [Figure 9] This is a flowchart showing the process performed by the camera according to the second embodiment. [Figure 10] Figure 9 is a conceptual diagram illustrating an example of the process shown in the flowchart. [Figure 11] This is a conceptual diagram illustrating an example of processing performed by the camera according to the third embodiment. [Modes for carrying out the invention]

[0009] The embodiments of the present invention will be described in detail below with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of the present invention is not limited to the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can perform a similar function. In addition, any configuration may be added. Furthermore, any two or more configurations (features) from each embodiment can be combined.

[0010] <First Embodiment> The first embodiment will be described below with reference to Figures 1 to 8B. Figure 1 is a schematic diagram showing an example of the overall configuration when the image processing apparatus according to the first embodiment is applied to a digital camera. The digital camera (hereinafter referred to as "camera") 100 shown in Figure 1 is an imaging device capable of capturing still images and videos. The format for still images may be, for example, JPEG or RAW. The format for videos may be, for example, MP4 or RAW. A lens 300 is detachably attached to the camera 100. The lens 300 is a twin-lens (VR180 lens) with the function of a fisheye lens. With this lens 300, still images and videos can be captured as twin-lens fisheye images with parallax that enable stereoscopic viewing.

[0011] Figure 2 is a block diagram showing an example of the hardware configuration of a digital camera. As shown in Figure 2, the camera 100 includes a shutter 202, a control unit 203, RAM 204, ROM 205, an imaging unit (imaging means) 206, a developing unit 207, and an image processing unit 208. The camera 100 also includes a recording medium 209, a battery 210, a display unit 211, a communication unit 212, a sensor 213, and an operation unit 214. These hardware components of the camera 100 are connected to each other via an internal bus 215 so as to be able to communicate with one another. The shutter 202 is a focal-plane shutter that can adjust the exposure time of the imaging unit 206 based on the control of the control unit 203. The control unit 203 is a computer that controls the entire camera 100, for example, having a CPU. The control of the entire camera 100 may be performed by the division of processing among the hardware components. The RAM 204 has the function of a buffer memory for temporarily storing various data and the function of a work area for the control unit 203. The ROM 205 stores programs and various data that the control unit 203 executes. These programs include, for example, programs that cause the control unit 203 to execute each process (control method for the image processing device) described later. The imaging unit 206 has an image sensor, etc., that converts light into an electrical video signal based on the control of the control unit 203. In this embodiment, the imaging unit 206 consists of an optical system that controls the aperture, zoom, focus, etc., of the lens 300, and an image sensor, etc., for converting light introduced through the optical system into an electrical video signal.

[0012] The developing unit 207 reads the image data after imaging processing stored in the RAM 204. The developing unit 207 then performs developing processing on this image data. The developing processing is not particularly limited and may include, for example, pixel interpolation, filtering, resizing, color conversion, and format conversion to Y, Cb, or Cr formats, which are suitable formats for saving compressed image data. The image data after developing processing is stored in the RAM 204. The image processing unit 208 reads the image data after developing processing stored in the RAM 204. The image processing unit 208 then performs image processing on this image data. The image processing is not particularly limited and may include, for example, transformation processing such as rotation or movement of images contained in the image data. The image data after image processing is also stored in the RAM 204. Various data is stored in the recording medium 209. The recording medium 209 may be fixed to and built into the camera 100 beforehand, or it may be detachable from the camera 100.

[0013] The battery 210 is a power source that supplies power for the camera 100 to operate. The battery 210 is not particularly limited, and for example, a rechargeable secondary battery that can be charged by an external battery charger can be used. The display unit 211 displays images being captured by the imaging unit 206 and images stored after capture. The display unit 211 also displays a menu screen on which the operating conditions of the camera 100 can be set. The display unit 211 is not particularly limited, and for example, a liquid crystal display can be used. The communication unit 212 has, for example, an antenna and a communication controller for wireless communication. This allows the camera 100 to communicate with an external device. The sensor 213 is a detection unit that detects the movement and orientation of the camera 100. The sensor 213 is not particularly limited, and for example, an accelerometer or a gyroscope can be used. The operation unit 214 accepts user input. The operation unit 214 includes, for example, a power button to switch the battery 210 ON and OFF, and an image capture start button to start image capture. In this embodiment, the camera 100 has both an imaging function and an image processing function, but is not limited to this. For example, the camera 100 may perform the imaging function, and an image processing device configured separately from the camera 100 may perform the image processing function.

[0014] Figure 3 shows an example of the mechanical configuration of a lens. The lens 300 shown in Figure 3 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens 300 is a bilens that, when mounted on the camera 100, is capable of capturing images with parallax between the two lenses when imaging the same subject. The lens 300 has a left-eye optical system 301L and a right-eye optical system 301R. Both the left-eye optical system 301L and the right-eye optical system 301R have a wide field of view of 180 degrees, enabling imaging of the area of ​​the front hemisphere. Specifically, both the left-eye optical system 301L and the right-eye optical system 301R are capable of capturing subjects within a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation / depression angle, pitch angle). Note that the field of view of the left-eye optical system 301L and the right-eye optical system 301R is not limited to 180 degrees, and may be, for example, around 160 degrees. The left eye optical system 301L includes a lens 302L positioned at the very front, as well as a reflective mirror and the like. The right eye optical system 301R is also positioned at the very front and includes a lens 302R, as well as a reflective mirror and the like. The optical axes of lens 302L and lens 302R are parallel to each other. When the same subject is imaged, lens 300 can form a right image (first image) formed via the right eye optical system 301R and a left image (second image) formed via the left eye optical system 301L, which has parallax with the right image, on the imaging unit 206 of the camera 100. The right and left images captured by the imaging unit 206 are stored on the recording medium 209. This allows the right and left images to be acquired (image acquisition step). In this embodiment, the recording medium 209 functions as an image acquisition means for acquiring the right and left images from the imaging unit 206. Note that since the right and left images are images that have passed through lens 300, they are fisheye images. In this embodiment, the right image and the left image are simultaneously and side-by-side formed on the imaging unit 206. That is, the two optical images formed by the right eye optical system 301R and the left eye optical system 301L are formed on the imaging unit 206. The imaging unit 206 converts the formed right image and left image (optical signals) into analog electrical signals, respectively. The right image and left image are then transmitted to, for example, a head-mounted display (not shown). The head-mounted display displays the right image and the left image separately.As a result, a user wearing a head-mounted display on their head can view a stereoscopic VR image (VR180) within a range of approximately 180 degrees.

[0015] Here, the "VR image" refers to an image that can be VR-displayed, which will be described later. VR images include, for example, omnidirectional images (all-spherical images) captured by an omnidirectional camera (all-spherical camera), panoramic images having a video range (effective video range) wider than the display range that can be displayed on the display unit 211, etc. Also, VR images include still images, moving images, and live images (images acquired in real time from the camera 100). The VR image has a video range (effective video range) corresponding to a visual field of up to 360 degrees in the left-right direction and 360 degrees in the up-down direction. Further, VR images include images having an angle of view wider than that of a general camera even if the angle in the left-right direction is less than 360 degrees and the angle in the up-down direction is less than 360 degrees, and images having a video range wider than the display range that can be displayed on the display unit 211. The image captured by the camera 100 using the lens 300 is a type of VR image. The VR image is VR-displayed, for example, by setting the display mode of a display device capable of displaying VR images such as a head-mounted display to "VR view". By VR-displaying a VR image with a 360-degree angle of view and having the user change the posture of the display device in the left-right direction, that is, the horizontal rotation direction, it becomes possible to view an omnidirectional video without seams in the left-right direction. In this embodiment, the VR image captured using the lens 300 is a VR180 image captured within a 180-degree range in the front direction and does not include an image captured within a 180-degree range in the rear direction. When such a VR180 image is VR-displayed and the posture of the display device is changed to the side where there is no image, a blank area is displayed on the display device. Also, the lens 300 has a lens system control circuit 303 that is communicably connected to the control unit 203 of the camera 100 in a state of being attached to the camera 100. The lens system control circuit 303 controls, for example, the driving of the lenses of the left-eye optical system 301L and the right-eye optical system 301R.

[0016] FIG. 4 is a flowchart showing the processes executed by a camera with a lens attached. A program based on the flowchart shown in FIG. 4 starts when the power button of the camera 100 is switched from the OFF state to the ON state. As shown in FIG. 4, in step S401, the control unit 203 of the camera 100 acquires the design value of the lens 300 from the lens 300. The design value of the lens 300 is stored in advance in a storage medium (not shown) built into the lens 300. The design value of the lens 300 is not particularly limited, and examples include the focal length and the like. Also, the design value of the lens 300 is used in the positive distance cylindrical conversion process described later.

[0017] In step S402, the control unit 203 acquires the individual value of the lens 300 from the lens 300. The individual value of the lens 300 is stored in advance in the storage medium built into the lens 300, similar to the design value of the lens 300. The individual value of the lens 300 is not particularly limited, and examples include the manufacturing error of the lens 300 and the like. Also, the individual value of the lens 300 is used in the positive distance cylindrical conversion process together with the design value of the lens 300. Thereby, a more preferable processing result can be obtained than when only the design value of the lens 300 is used in the positive distance cylindrical conversion process.

[0018] In step S403, the control unit 203 acquires the images (right image and left image) captured by the imaging unit 206 from the imaging unit 206. After acquiring this image, the control unit 203 executes development processing on the image in the development unit 207.

[0019] In step S404, the control unit 203 displays the image on which the development processing was executed in step S403 on the display unit 211. Thereby, live view display on the display unit 211 is performed.

[0020] In step S405, the control unit 203 determines whether a recording start command has been issued by a user operation on the operation unit 214 to start recording images. If the control unit 203 determines that a recording start command has been issued as a result of the determination in step S405, the process proceeds to step S406. On the other hand, if the control unit 203 determines that a recording start command has not been issued as a result of the determination in step S405, the process returns to step S403 and the subsequent steps are executed in order. Therefore, the camera 100 continues to display the live view on the display unit 211 until it is determined in step S405 that a recording start command has been issued. The recording start command may be a recording start command to start recording still images or a recording start command to start recording video.

[0021] In step S406, similar to step S403, the control unit 203 acquires an image from the imaging unit 206, and then the developing unit 207 performs a developing process on the image.

[0022] In step S407, the control unit 203 acquires shooting information and metadata (information acquisition step). The shooting information is not particularly limited and includes, for example, the shutter speed at the time of shooting. The metadata is not particularly limited and includes, for example, the attitude information of the camera 100 at the time of shooting, i.e., tilt information related to the tilt when the camera 100 rotates and tilts in the roll direction (around the optical axis of the camera 100). The attitude information is acquired from the sensor 213. Thus, in this embodiment, the control unit 203 also functions as an information acquisition means for acquiring various types of information. Note that in the camera 100, a part that functions as an information acquisition means may be provided separately from the control unit 203. Furthermore, when RAW image shooting is performed with the camera 100, the control unit 203 also acquires metadata necessary for developing the RAW image.

[0023] In step S408, the control unit 203 determines whether the camera 100 is tilted in the roll direction based on the attitude information acquired in step S407. If the control unit 203 determines that the camera 100 is tilted as a result of the determination in step S408, correction for the tilt is required, and the process proceeds to step S409. On the other hand, if the control unit 203 determines that the camera 100 is not tilted as a result of the determination in step S408, the correction for the tilt can be omitted, and the process proceeds to step S410.

[0024] In step S409, the control unit 203 controls the image processing unit 208 to perform correction processing for the tilt of the camera 100 (correction step). Details of the processing in step S409 will be described later. Thus, in this embodiment, the control unit 203 also functions as a correction means for performing correction processing. Note that in the camera 100, the part that functions as a correction means may be provided separately from the control unit 203. Also, if imaging is performed with the camera 100 tilted in the roll direction, the captured image will be played back on the head-mounted display in that tilted state if the correction processing is omitted. Therefore, a user wearing a head-mounted display may feel discomfort when viewing the image, or may experience so-called VR sickness.

[0025] In step S410, the control unit 203 controls the image processing unit 208 to perform a process (equirectangular transformation) that converts the image developed in step S406 into a recorded image. Details of the process in step S410 will be described later.

[0026] In step S411, the control unit 203 encodes the image processed in step S410. The control unit 203 also stores this encoded image as a file on the recording medium 209.

[0027] In step S412, the control unit 203 associates the shooting information and metadata acquired in step S407 with the file stored in step S411 and stores it in the recording medium 209.

[0028] In step S413, the control unit 203 associates the design value of the lens 300 acquired in step S401 with the individual value of the lens 300 acquired in step S402 with the file stored in step S412, and stores it in the recording medium 209.

[0029] In step S414, the control unit 203 determines whether a recording end instruction has been given by a user operation on the operation unit 214 to terminate image recording. In the case of still image shooting, since each image is taken individually, the recording start instruction and recording end instruction are given simultaneously. In the case of video recording, the recording instruction is terminated when the operation unit 214 is operated again. If the control unit 203 determines, as a result of the determination in step S414, that a recording end instruction has been given, the process ends. On the other hand, if the control unit 203 determines, as a result of the determination in step S414, that a recording end instruction has not been given, the process returns to step S406 and the subsequent steps are executed in order.

[0030] Figure 5 shows an example of an image captured by a camera equipped with a lens. The image 500 shown in Figure 5 is an image that has been developed by the developing unit 207 and is stored in the RAM 204. Image 500 includes a left-side circular fisheye image (left image) 501 formed by the left-eye optical system 301L of the lens 300, and a right-side circular fisheye image (right image) 502 formed by the right-eye optical system 301R. Position 505 (X coordinate) is the vertical center position of the left-side circular fisheye image 501 and the right-side circular fisheye image 502. Position 503 (Y1 coordinate) is the horizontal center position of the left-side circular fisheye image 501. Position 504 (Y2 coordinate) is the horizontal center position of the right-side circular fisheye image 502. Therefore, the intersection of position 505 and position 503 is the center position of the left-side circular fisheye image 501. Furthermore, the intersection of position 505 and position 504 is the center position of the right-hand circular fisheye image 502.

[0031] Figure 6 is a flowchart showing the detailed processing performed in step S409 (subroutine) of the flowchart shown in Figure 4. As shown in Figure 6, in step S601, the control unit 203 obtains tilt information regarding the tilt of the camera 100 in the roll direction from the attitude information acquired in step S407.

[0032] In step S602, the control unit 203 controls the image processing unit 208 to perform rotation correction processing. Specifically, based on the tilt information acquired in step S601, the image processing unit 208 performs rotation correction processing to rotate the left circular fisheye image 501 and the right circular fisheye image 502 in a direction that reduces (cancels out) the tilt of the camera 100 in the roll direction.

[0033] Figure 7 is a conceptual diagram illustrating an example of the process performed in step S409 of the flowchart shown in Figure 4. Figure 7(a) shows the circular fisheye image before processing. Figure 7(b) shows the circular fisheye image after processing. Figure 7(c) shows the camera orientation when capturing the circular fisheye image shown in Figure 7(a). The image 700 shown in Figure 7(a) is the image held in RAM 204 after development processing in development unit 207. This image 700 is also the image when it is determined in step S408 that the camera 100 is tilted. Image 700 includes the left circular fisheye image (left image) 701 formed by the left eye optical system 301L of lens 300 and the right circular fisheye image (right image) 702 formed by the right eye optical system 301R. Position 705 (X coordinate) is the vertical center position of the left circular fisheye image 701 and the right circular fisheye image 702. Position 703 (Y1 coordinate) is the horizontal center of the left circular fisheye image 701. Position 704 (Y2 coordinate) is the horizontal center of the right circular fisheye image 702. The intersection point 706 of position 705 and position 703 is the center point of the left circular fisheye image 701. The intersection point 707 of position 705 and position 704 is the center point of the right circular fisheye image 702. When capturing such an image 700, the camera 100 will be in the orientation shown in Figure 7(c). Specifically, the camera 100 will be tilted at an angle θ712 in the counterclockwise direction (roll direction) in Figure 7(c) with respect to the horizontal direction 711 in the imaging space, with respect to the optical axis of the camera 100.

[0034] By performing rotation correction processing on image 700, image 710 shown in Figure 7(b) is obtained. Image 710 includes a left circular fisheye image 708 and a right circular fisheye image 709. The left circular fisheye image 708 is the image obtained by rotating the left circular fisheye image 701 clockwise around the intersection (center point) 706 by an angle θ713, which is the same magnitude as the angle θ712, as shown in Figure 7(b). The right circular fisheye image 709 is the image obtained by rotating the right circular fisheye image 702 clockwise around the intersection (center point) 707 by an angle θ713, which is the same magnitude as the angle θ712, as shown in Figure 7(b). In both the left circular fisheye image 708 and the right circular fisheye image 709, the horizontal direction coincides with the horizontal direction 711 in the imaging space of camera 100.

[0035] Figure 8A is a conceptual diagram illustrating an example of the process performed in step S410 of the flowchart shown in Figure 4. Figure 8A(a) shows the circular fisheye image before processing. Figure 8A(b) shows the circular fisheye image after processing. In addition, there is a "Side-by-Side format" for recording the processed circular fisheye image, in which the left circular fisheye image and the right circular fisheye image are placed side by side. Another format is the "Top-Bottom format," in which the left circular fisheye image and the right circular fisheye image are placed side by side vertically. Furthermore, when using the Top-Bottom format, before placing the left and right circular fisheye images vertically, a portion of each circular fisheye image may be erased (deleted) before placing them vertically. In this case, the range of stereoscopic viewing is narrower in the circular fisheye image displayed on the head-mounted display compared to when no portion of the circular fisheye image is reduced, the amount of data can be reduced, and the sense of resolution is maintained. This Top-Bottom format, which captures a portion of a circular fisheye image, is applied, for example, when the processing power of camera 100 is insufficient, or when it is desirable to reduce the amount of data as much as possible for streaming distribution, etc.

[0036] Image 800, shown in Figure 8A(a), is an image obtained by performing rotation correction processing, similar to Image 710 (see Figure 7(b)). Image 800 includes a left-side circular fisheye image 801 formed by the left-eye optical system 301L of lens 300, and a right-side circular fisheye image 802 formed by the right-eye optical system 301R. Position 805 (X coordinate) is the vertical center position of the left-side circular fisheye image 801 and the right-side circular fisheye image 802. Position 803 (Y1 coordinate) is the horizontal center position of the left-side circular fisheye image 801. Position 804 (Y2 coordinate) is the horizontal center position of the right-side circular fisheye image 802. Position 812 (Y3 coordinate) is the horizontal center position of image 800. The reference line 806 is the line used when an erasure process is performed on the left circular fisheye image 801 and the right circular fisheye image 802 to erase the upper portion. For the left circular fisheye image 801 and the right circular fisheye image 802, the portion above the reference line 806 (the upper gray portion in Figure 8A(a)) is erased. The reference line 807 is the line used when an erasure process is performed on the left circular fisheye image 801 and the right circular fisheye image 802 to erase the lower portion. For the left circular fisheye image 801 and the right circular fisheye image 802, the portion below the reference line 807 (the lower gray portion in Figure 8A(a)) is erased. In this embodiment, the image processing unit 208 functions as the erasure means that executes the erasure process, but it is not limited to this. For example, a part that functions as an erasure means may be provided separately from the image processing unit 208. Furthermore, in this embodiment, the upper and lower parts of each circular fisheye image are erased, but this is not limited to this, and either the upper or lower part may be erased. Also, the vertical positions of the reference lines 806 and 807 in the image 800 can be changed, for example, by user operation.

[0037] As shown in Figure 8A(b), the control unit 203 controls the image processing unit 208 to move image 809, the portion of the left circular fisheye image 801 between the reference lines 806 and 807, to area 811 on the RAM 204. The control unit 203 also controls the image processing unit 208 to move image 810, the portion of the right circular fisheye image 802 between the reference lines 806 and 807, to area 811 on the RAM 204. Then, the control unit 203 controls the image processing unit 208 to generate an array image in which images 809 and 810 are arranged vertically within area 811 (image generation step). In this array image, the centers of images 809 and 810 are aligned at position 812. In this embodiment, the image processing unit 208 functions as an image generation means for generating the array image, but is not limited to this. For example, a part that functions as an image generation means may be provided separately from the image processing unit 208.

[0038] Figure 8B is a conceptual diagram illustrating an example of the effects in this embodiment. Figure 8B(a) shows a circular fisheye image before processing. Image 8000 shown in Figure 8B(a) is an image held in RAM 204 after development processing in development unit 207. This image 8000 is also the image when it is determined in step S408 that the camera 100 is tilted. Image 8000 includes a left circular fisheye image 8001 formed by the left eye optical system 301L of lens 300 and a right circular fisheye image 8002 formed by the right eye optical system 301R.

[0039] Figure 8B(b) shows a circular fisheye image (comparative example) after equirectangular transformation processing. Equirectangular transformation processing here includes rotation correction processing. Image 8000' shown in Figure 8B(b) includes the left circular fisheye image 8003 and the right circular fisheye image 8004 arranged in a side-by-side format. Image 8000' also includes regions 8005 and 8006 that were erased by an erasure process before the left circular fisheye image 8003 and the right circular fisheye image 8004 were arranged in a side-by-side format. Regions 8005 and 8006 become prominently visible after equirectangular transformation processing. Therefore, when viewing image 8000' with a head-mounted display, image 8000' may appear as an unnatural stereoscopic image.

[0040] Figure 8B(c) shows a circular fisheye image (example) after rotation correction processing. By performing rotation correction processing on image 8000, image 8007 shown in Figure 8B(c) is obtained. Image 8007 includes the left circular fisheye image 8008 and the right circular fisheye image 8009. The left circular fisheye image 8008 is an image obtained by rotating the left circular fisheye image 8001 clockwise by a predetermined angle in Figure 8B. The right circular fisheye image 8009 is an image obtained by rotating the right circular fisheye image 8002 clockwise by the same predetermined angle in Figure 8B.

[0041] Figure 8B(d) shows a circular fisheye image (example) after equirectangular transformation processing. Image 8007' shown in Figure 8B(d) includes a left circular fisheye image 8010 and a right circular fisheye image 8011 arranged in a side-by-side format. Image 8007' also includes regions 8012 and 8013 that were erased by an erasure process after rotation correction processing and before the left circular fisheye image 8010 and the right circular fisheye image 8011 were arranged in a side-by-side format. When such an image 8007' is viewed with a head-mounted display, the image 8007' becomes a stereoscopic image without any sense of incongruity. With the camera 100 configured as described above, even if parts of the left circular fisheye image and the right circular fisheye image, which have parallax, are erased, the head-mounted display can observe (reproduce) the left circular fisheye image and the right circular fisheye image stereoscopically without any sense of incongruity.

[0042] <Second Embodiment> The second embodiment will be described below with reference to Figures 9 and 10, focusing on the differences from the previously described embodiment, and omitting explanations of similar matters. In this embodiment, the case where the camera 100 tilts in the roll direction during imaging with the camera 100, resulting in a significant height difference between the left eye optical system 301L and the right eye optical system 301R of the lens 300, will be described. Figure 9 is a flowchart of the processing performed by the camera according to the second embodiment. The flowchart shown in Figure 9 is a flowchart of the detailed processing performed in step S409 of the flowchart shown in Figure 4. As shown in Figure 9, in step S901, similar to step S601, the control unit 203 of the camera 100 acquires tilt information regarding the tilt of the camera 100 in the roll direction from the attitude information acquired in step S407.

[0043] In step S902, similar to step S602, the control unit 203 controls the image processing unit 208 to perform rotation correction processing.

[0044] In step S903, the control unit 203 calculates the height between the left eye optical system 301L and the right eye optical system 301R during shooting, based on the tilt information acquired in step S901 and the lens design value acquired in step S401.

[0045] In step S904, the control unit 203 controls the image processing unit 208 to move one of the two circular fisheye images parallel to the Y coordinate by the height calculated in step S903. This aligns the centers of the two circular fisheye images.

[0046] Figure 10 is a conceptual diagram illustrating an example of the processing in the flowchart shown in Figure 9. Figure 10(a) shows the circular fisheye image before processing. Figure 10(b) shows the circular fisheye image after step S902 is executed. Figure 10(c) shows the circular fisheye image after step S904 is executed. Figure 10(d) shows the camera's orientation when capturing the circular fisheye image shown in Figure 10(a). Image 700 shown in Figure 10(a) is an image held in RAM 204 after development processing in development unit 207. Image 1000 is an image taken when it is determined in step S408 that the camera 100 is tilted. Image 1000 includes a left circular fisheye image 1001 formed by the left eye optical system 301L of lens 300 and a right circular fisheye image 1002 formed by the right eye optical system 301R. Position 1005 (X coordinate) is the vertical center position of the left circular fisheye image 1001 and the right circular fisheye image 1002. Position 1003 (Y1 coordinate) is the horizontal center position of the left circular fisheye image 1001. Position 1004 (Y2 coordinate) is the horizontal center position of the right circular fisheye image 1002. The intersection point 1006 of position 1005 and position 1003 is the center point of the left circular fisheye image 1001. Also, the intersection point 1007 of position 1005 and position 1004 is the center point of the right circular fisheye image 1002. When capturing such an image 1000, the camera 100 will be in the orientation shown in Figure 10(d). Specifically, the camera 100 is tilted at an angle θ1024 in the counterclockwise direction (roll direction) in Figure 10(d) with respect to the horizontal directions 1021 and 1022 in the imaging space, with respect to the optical axis of the camera 100. The height H1023 is the height between the left eye optical system 301L and the right eye optical system 301R.

[0047] By performing the processing (rotation correction processing) in step S902 on image 1000, image 1000' shown in Figure 10(b) is obtained. Image 1000' includes the left circular fisheye image 1008 and the right circular fisheye image 1009. The left circular fisheye image 1008 is the image obtained by rotating the left circular fisheye image 1001 clockwise around the intersection (center point) 1010 by an angle of the same magnitude as angle θ1024 in Figure 10(b). The right circular fisheye image 1009 is the image obtained by rotating the right circular fisheye image 1002 clockwise around the intersection (center point) 1011 by an angle of the same magnitude as angle θ1012 in Figure 10(b). Point 1012 is the point in the right circular fisheye image 1002 that corresponds to intersection point 1010. Furthermore, both the left circular fisheye image 1008 and the right circular fisheye image 1009 have a horizontal direction that coincides with the horizontal direction 1021 (horizontal direction 1022) in the imaging space of the camera 100. However, because the center position of the subject being photographed differs between the left circular fisheye image 1008 and the right circular fisheye image 1009, these images may not be convertible into stereoscopic images that can be viewed without discomfort on a head-mounted display.

[0048] By performing the processing in step S904 on image 1000', image 1000'' shown in Figure 10(c) is obtained. Image 1000'' includes the left circular fisheye image 1013 and the moved right circular fisheye image 1014. Center 1015 is the center of the left circular fisheye image 1013. Center 1017 is the center of the moved right circular fisheye image 1014. Position 1019 (X1 coordinate) is the same as position 1005 (X1 coordinate). Position 1020 (X2 coordinate) is the position that was the center of the moved right circular fisheye image 1014. The amount of movement AM1018 is vertical information relating to the vertical difference between the center of the right circular fisheye image 1014 (subject) and the center of the left circular fisheye image 1013 (subject), and is calculated based on the height H1023. Then, based on the displacement amount AM1018, a reduction correction process is performed to reduce the height H1023, so that the original center 1016 of the right circular fisheye image 1014 moves by the displacement amount AM1018. As a result, the vertical positions of the center 1015 of the left circular fisheye image 1013 and the center 1017 of the right circular fisheye image 1014 are aligned. This ensures that even if a significant height difference occurs between the left eye optical system 301L and the right eye optical system 301R, the image on the head-mounted display becomes a stereoscopic image that can be observed without discomfort. In this embodiment, the right circular fisheye image after movement was acquired using the center of the left circular fisheye image as the reference, but this is not limited to this. For example, the right circular fisheye image may be used as the reference, or each circular fisheye image may be moved using the midpoint between the center of the left circular fisheye image and the center of the right circular fisheye image as the reference.

[0049] <Third Embodiment> The third embodiment will now be described with reference to Figure 11, focusing on the differences from the previously described embodiment, and omitting similar matters. In this embodiment, the case where the left circular fisheye image and the right circular fisheye image are arranged side by side will be described. Figure 11 is a conceptual diagram illustrating an example of processing performed by the camera according to the third embodiment. Specifically, Figure 11 is a conceptual diagram illustrating an example of processing performed in step S410 of the flowchart shown in Figure 4. Figure 11(a) shows the circular fisheye image before processing. Figure 11(b) shows the circular fisheye image after processing. Image 1100 shown in Figure 11(a) is an image obtained by performing rotation correction processing. Image 1100 includes the left circular fisheye image 1101 formed by the left eye optical system 301L of the lens 300 and the right circular fisheye image 1102 formed by the right eye optical system 301R. Position 1109 (X coordinate) is the vertical center position of the left circular fisheye image 1101 and the right circular fisheye image 1102. Position 1110 (Y coordinate) is the horizontal center position of image 1100. The reference line 1103 is the line used when an erasure process is performed on the left circular fisheye image 1101 to erase the left portion. For the left circular fisheye image 1101, everything to the left of reference line 1103 is erased. The reference line 1104 is the line used when an erasure process is performed on the left circular fisheye image 1101 to erase the right portion. For the left circular fisheye image 1101, everything to the right of reference line 1104 is erased. The reference line 1105 is the line used when an erasure process is performed on the right circular fisheye image 1102 to erase the left portion. For the right circular fisheye image 1102, everything to the left of reference line 1105 is erased. The reference line 1106 is the line used when an erasure process is performed on the right-hand circular fisheye image 1102 to erase the right-hand portion. For the right-hand circular fisheye image 1102, everything to the right of the reference line 1106 is erased. In this embodiment, the left and right sides of each circular fisheye image are erased, but this is not limited to this, and either the left or right side may be erased. Furthermore, the positions of the reference lines 1103 to 1106 in the left-right direction within the image 1100 can be changed, for example, by user operation.

[0050] As shown in Figure 11(b), the control unit 203 controls the image processing unit 208 to move image 1107, the portion of the left circular fisheye image 1101 between reference lines 1103 and 1104, to region 1111 on the RAM 204. The control unit 203 also controls the image processing unit 208 to move image 1108, the portion of the right circular fisheye image 1102 between reference lines 1105 and 1106, to region 1111 on the RAM 204. Then, the control unit 203 controls the image processing unit 208 to generate an array image in which images 1107 and 1108 are placed side by side within region 1111 (image generation process). In this array image, the centers of images 1107 and 1108 are aligned at position 1112. When such images 1107 and 1108 are viewed with a head-mounted display, the images become a stereoscopic image without any sense of incongruity.

[0051] While preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. For example, in addition to reducing the left and right portions of two circular fisheye images to form a Side-by-Side format, a Top-Bottom format may also be used. In addition to reducing the top and bottom portions of two circular fisheye images to form a Top-Bottom format, a Side-by-Side format may also be used. The present invention provides a program that implements one or more functions of the above embodiments to a system or device via a network or storage medium. It can also be implemented by a process in which one or more general-purpose processors (ASICs) in the computer of the system or device read and execute the program. Furthermore, the present invention can also be implemented by a dedicated processor (e.g., an ASIC or FPGA) that implements one or more functions. Moreover, the present invention can also be implemented by a combination of a general-purpose processor and a dedicated processor. Here, "processor" refers to a processor in a broad sense and includes both general-purpose processors and dedicated processors. Furthermore, the process of implementing the present invention may be executed by only one processor, or by the cooperation of multiple processors located in physically separate locations.

[0052] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Configuration 1) Image acquisition means that acquires a first image and a second image having parallax between them, which are images of the same subject, Information acquisition means for acquiring tilt information relating to the tilt when the imaging device rotates and tilts around the optical axis of the imaging device when the first image and the second image are captured by the imaging device, Correction means that performs rotation correction processing to rotate the first image and the second image in a direction that reduces the tilt based on the tilt information, An image processing apparatus characterized by comprising: an image generation means for generating an array image obtained by arranging the first image and the second image, which have undergone the rotation correction process, side by side. (Configuration 2) The information acquisition means acquires vertical information relating to the vertical difference between the center of the subject included in the first image and the center of the subject included in the second image, which is caused by the tilt. The image processing apparatus according to configuration 1, characterized in that the correction means performs a reduction correction process to reduce the difference based on the vertical information. (Configuration 3) The image processing apparatus according to Configuration 2, characterized in that, after the rotation correction process, the correction means performs the reduction correction process by aligning the vertical position of the center of the subject included in one of the first and second images with the vertical position of the center of the subject included in the other image. (Configuration 4) An image processing apparatus according to any one of Configurations 1 to 3, characterized in that, prior to the generation of the array image, it comprises erasing means for performing an erasing process on the first image and the second image on which the rotation correction process has been performed, to erase at least one of the upper portion and the lower portion. (Configuration 5) An image processing apparatus according to any one of Configurations 1 to 3, characterized in that it comprises erasing means for erasing at least one of the left portion and the right portion of the first image and the second image, which have undergone the rotation correction processing prior to the generation of the array image. (Configuration 6) The image processing apparatus according to any one of Configurations 1 to 5, characterized in that the image generation means generates an image in which the first image and the second image are arranged vertically as the array image. (Configuration 7) The image processing apparatus according to any one of Configurations 1 to 5, characterized in that the image generation means generates an image in which the first image and the second image are arranged side by side as the array image. (Configuration 8) The image processing apparatus according to any one of Configurations 1 to 7, characterized in that the image acquisition means acquires fisheye images as the first image and the second image, respectively. (Configuration 9) The image processing device is an imaging device that includes imaging means for capturing the first image and the second image, The image processing apparatus according to any one of configurations 1 to 8, characterized in that the image acquisition means acquires the first image and the second image from the imaging means. (Method 1) A method for controlling an image processing device capable of image processing, An image acquisition process that acquires a first image and a second image having parallax between them, both of which are images of the same subject, An information acquisition step to acquire tilt information relating to the tilt when the imaging device rotates and tilts around the optical axis of the imaging device when the first image and the second image are captured by the imaging device, A correction step in which a rotation correction process is performed to rotate the first image and the second image in a direction that reduces the tilt, based on the tilt information, A control method for an image processing apparatus, characterized by comprising: an image generation step of generating an array image obtained by arranging the first image and the second image, which have undergone the rotation correction process, side by side. (Program 1) A program characterized by causing a computer to execute the control method described in Configuration 10. [Explanation of Symbols]

[0053] 100 Digital Cameras 203 Control Unit 206 Imaging Department 208 Image Processing Unit

Claims

1. Image acquisition means for acquiring a first image and a second image having parallax between them, both of which are images of the same subject, Information acquisition means for acquiring tilt information relating to the tilt when the imaging device rotates and tilts around the optical axis of the imaging device when the first image and the second image are captured by the imaging device, Correction means that performs rotation correction processing to rotate the first image and the second image in a direction that reduces the tilt based on the tilt information, An image processing apparatus characterized by comprising: an image generation means for generating an array image obtained by arranging the first image and the second image, which have undergone the rotation correction process, side by side.

2. The information acquisition means acquires vertical information relating to the vertical difference between the center of the subject included in the first image and the center of the subject included in the second image, which is caused by the tilt. The image processing apparatus according to claim 1, characterized in that the correction means performs a reduction correction process to reduce the difference based on the vertical information.

3. The image processing apparatus according to claim 2, characterized in that the correction means performs the reduction correction process by aligning the vertical position of the center of the subject included in one of the first and second images with the vertical position of the center of the subject included in the other image after the rotation correction process.

4. The image processing apparatus according to claim 1, further comprising erasing means for performing an erasing process on the first image and the second image, which have undergone the rotation correction process prior to the generation of the array image, to erase at least one of the upper portion and the lower portion.

5. The image processing apparatus according to claim 1, further comprising erasing means for performing an erasing process on the first image and the second image, which have undergone the rotation correction process prior to the generation of the array image, to erase at least one of the left portion and the right portion.

6. The image processing apparatus according to claim 1, characterized in that the image generation means generates an image in which the first image and the second image are arranged vertically as the array image.

7. The image processing apparatus according to claim 1, characterized in that the image generation means generates an image in which the first image and the second image are arranged side by side as the array image.

8. The image processing apparatus according to claim 1, characterized in that the image acquisition means acquires fisheye images as the first image and the second image, respectively.

9. The image processing apparatus is an imaging apparatus that includes imaging means for capturing the first image and the second image, The image processing apparatus according to claim 1, characterized in that the image acquisition means acquires the first image and the second image from the imaging means.

10. A method for controlling an image processing device capable of image processing, An image acquisition process that acquires a first image and a second image having parallax between them, both of which are images of the same subject, An information acquisition step to acquire tilt information relating to the tilt when the imaging device rotates and tilts around the optical axis of the imaging device when the first image and the second image are captured by the imaging device, A correction step in which a rotation correction process is performed to rotate the first image and the second image in a direction that reduces the tilt, based on the tilt information, A control method for an image processing apparatus, characterized by comprising: an image generation step of generating an array image obtained by arranging the first image and the second image, which have undergone the rotation correction process, side by side.

11. A program characterized by causing a computer to execute the control method described in claim 10.

Images