Image processing device, image processing method, and program

Abstract

To provide a technology that appropriately processes a circumferential fisheye image which is accompanied by a missing area having no pixel value.SOLUTION: An image processing device comprising: acquisition means that acquires a first circumferential fisheye image that is accompanied by a first missing area having no pixel value; and creation means that creates a first azimuthal equidistant cylindrical diagram method image by performing first azimuthal equidistant cylindrical transformation processing on the basis of the first circumferential fisheye image, the creation means creating the first azimuthal equidistant cylindrical diagram method image for a first corresponding area corresponding to the first missing area to have a pixel value in the first azimuthal equidistant cylindrical diagram method image, is provided.SELECTED DRAWING: Figure 8A

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] There is a known technology that uses two cameras to capture images with parallax and reproduces the captured parallax images as virtual reality (VR) images with a three-dimensional effect. There is also a known device that has two optical systems in one lens mount and can capture images with parallax at the same time (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-141052 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology in Patent Document 1 addresses the problem of the left and right images passing through the left and right lenses on the image sensor being captured in different directions, but does not take into consideration how to process circular fisheye images that contain missing areas where pixel values do not exist.

[0005] The present invention has been made in view of the above circumstances, and has as its object to provide a technique for appropriately processing circular fisheye images that include missing areas where pixel values do not exist. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention provides an image processing method comprising: acquiring means for acquiring a first circular fisheye image that is not recorded in a complete form, the first circular fisheye image having a first missing area where no pixel values exist within a first circular area corresponding to the complete form of the first circular fisheye image; and generating means for generating a first equirectangular projection image corresponding to the first circular area by performing a first equirectangular conversion process based on the first circular fisheye image, the generating means comprising: Before the first equirectangular conversion process, a pixel value indicating a predetermined color is set in the first missing area of the first circular fisheye image, The image processing device generates the first equirectangular projection image so that a first corresponding area of the first equirectangular projection image, which corresponds to the first missing area in the first circular area, has a pixel value; the acquisition means acquires a second circular fisheye image having parallax with respect to the first circular fisheye image; the first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, and the first subject image is formed via a first fisheye lens, and the second subject image is formed via a second fisheye lens. [Effects of the Invention]

[0007] According to the present invention, it is possible to appropriately process a circular fisheye image that includes missing areas where no pixel values exist.

[0008] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiment of the present invention. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the overall configuration of the system. [Figure 2] FIG. 1 is a diagram showing the appearance of a camera 100. [Figure 3] FIG. 2 is a diagram showing an example of the internal configuration of the camera 100. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a VR180 lens 300 that can be attached to the camera 100. [Figure 5] Block diagram showing the configuration of PC500. [Figure 6A] 4 is a flowchart of a photographing process by the camera 100. [Figure 6B] 4 is a flowchart of a photographing process by the camera 100. [Figure 7] 10 is an overall flowchart of equirectangular conversion processing by PC500. [Figure 8A] 10 is a flowchart showing details of the process in S712 (processing for creating an equirectangular projection image) according to the first embodiment. [Figure 8B] 10 is a flowchart showing details of the process in S712 (processing for creating an equirectangular projection image) according to the second embodiment. [Figure 8C] 11 is a flowchart showing details of the process in S712 (processing for creating an equirectangular projection image) according to the third embodiment. [Figure 8D] 13 is a flowchart showing details of the process of S712 (processing for creating an equirectangular projection image) according to the fourth embodiment. [Figure 9] 1 is a diagram showing the structure of an image file recorded by the camera 100 (an image file processed by the PC 500). [Figure 10] 10(a) and 10(b) are diagrams showing information acquired from a VR180 lens 300, and 10(c) is a diagram showing information acquired from a camera 100. FIG. [Figure 11] (a) (b) A diagram showing an example of an image captured and recorded by a camera 100 equipped with a VR180 lens 300, and (c) a diagram showing an example of a state in which pixel values have been set in missing areas of a circular fisheye image. [Figure 12A] FIG. 10 is a conceptual diagram of the process in S712 (processing for creating an equirectangular projection image) according to the first embodiment. [Figure 12B] FIG. 11 is a conceptual diagram of the process in S712 (processing for creating an equirectangular projection image) according to the second embodiment. [Figure 12C] FIG. 11 is a conceptual diagram of the process in S712 (processing for creating an equirectangular projection image) according to the third embodiment. [Figure 12D] FIG. 13 is a conceptual diagram of the process of S712 (processing for creating an equirectangular projection image) according to the fourth embodiment. [Figure 13]10 is a flowchart of image processing according to the fifth embodiment. [Figure 14] 10 is a flowchart showing details of image processing in S1310. [Figure 15] FIG. 13 is a diagram showing a display setting screen according to the fifth embodiment. [Figure 16A] FIG. 13 is a conceptual diagram of image processing according to the fifth embodiment. [Figure 16B] FIG. 13 is a conceptual diagram of image processing according to the fifth embodiment. [Figure 16C] FIG. 13 is a conceptual diagram of image processing according to the fifth embodiment. [Figure 16D] FIG. 13 is a conceptual diagram of image processing according to the fifth embodiment. [Figure 17] 13 is a flowchart of image processing according to the sixth embodiment. [Figure 18] 10 is a flowchart showing details of the processing of S1715. [Figure 19] FIG. 20 is a diagram showing an application screen on a PC 500 according to the sixth embodiment. [Figure 20] FIG. 13 is a conceptual diagram of image processing according to the sixth embodiment. [Figure 21] FIG. 13 is a conceptual diagram of image processing according to the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [First embodiment] ● Overall system configuration FIG. 1 is a diagram showing the overall configuration of a system according to a first embodiment. In FIG. 1, 100 is a digital camera (hereinafter referred to as "camera") capable of capturing images (still images or videos). 250 is a VR180 lens that can be attached to the camera 100. The VR180 lens 300 is a twin lens, and by capturing images using the camera 100 to which the VR180 lens 300 is attached, it is possible to obtain images with parallax. The VR180 lens 300 is a lens for capturing images for so-called VR180, which is a VR image format that allows twin-eye stereoscopic viewing and has a 180-degree field of view.

[0012] Reference numeral 500 denotes a personal computer (PC), which is an example of an image processing device that processes images captured by the camera 100. The method by which the PC 500 acquires images from the camera 100 is not particularly limited. For example, as shown in FIG. 1(a), the PC 500 may acquire images from the camera 100 via wireless communication. Alternatively, as shown in FIG. 1(b), the PC 500 may acquire images recorded as files via an external storage device (e.g., a memory card) of the camera 100.

[0013] ●Configuration of Camera 100 Figure 2 shows the external appearance of the camera 100. Figure 2(a) is a perspective view of the camera 100 seen from the front, and Figure 2(b) is a perspective view of the camera 100 seen from the back.

[0014] The camera 100 has, on its top surface, a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an extra-viewfinder display 107. The shutter button 101 is an operation unit for preparing for shooting or issuing instructions for shooting. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode switch 103 is an operation unit for switching between various modes. The main electronic dial 104 is a rotary operation unit for changing setting values such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation unit for moving the selection frame (cursor), scrolling through images, etc. The video button 106 is an operation unit for issuing instructions to start and stop video shooting (recording). The extra-viewfinder display 107 displays various setting values such as shutter speed and aperture.

[0015] The camera 100 also has a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, and a menu button 115 on the back side. The camera 100 also has an eyepiece unit 116, an eyepiece finder 117 (a peer-type finder), an eyepiece detection unit 118, and a touch bar 119. The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are an operation unit consisting of keys (four-way keys) that can be pressed up, down, left, and right. Operations can be performed according to the position of the directional keys 110 that are pressed. The SET button 111 is an operation unit that is mainly pressed to confirm a selection item. The AE lock button 112 is an operation unit that is pressed to fix the exposure state in a shooting standby state. The enlargement button 113 is an operation unit for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is also used to enlarge the playback image or increase the magnification in the playback mode. The playback button 114 is an operation unit for switching between the shooting mode and the playback mode. In the shooting mode, pressing the playback button 114 switches to the playback mode, and the latest image recorded on the recording medium 227 (described later) can be displayed on the display unit 108.

[0016] The menu button 115 is an operation unit that is pressed when a menu screen that allows various settings to be displayed on the display unit 108. The user can intuitively make various settings using the menu screen displayed on the display unit 108, the direction keys 110, and the SET button 111. The eyepiece unit 116 is a part for placing an eye on the eyepiece finder 117. The user can view an image displayed on an internal EVF 217 (Electronic View Finder) described below through the eyepiece unit 116. The eyepiece detection unit 118 is a sensor that detects whether the user has placed an eye on the eyepiece unit 116.

[0017] The touch bar 119 is a line-shaped touch operation unit (line touch sensor) capable of receiving touch operations. The touch bar 119 is positioned so that it can be touched by the thumb of the right hand when the grip unit 120 is held in the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. That is, the touch bar 119 can be operated while the user places their eye on the eyepiece unit 116, looks through the eyepiece viewfinder 117, and is in a position (shooting posture) in which the user is ready to press the shutter button 101 at any time. The touch bar 119 can receive tap operations (operations in which the user touches and then releases the touch bar without moving within a predetermined period of time), slide operations to the left or right (operations in which the user touches and then moves the touched position while keeping the touch), and the like. The touch bar 119 is an operation unit different from the touch panel 109 and does not have a display function. The touch bar 119 of this embodiment is a multi-function bar and functions, for example, as an M-Fn bar.

[0018] The camera 100 also has a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, and a communication terminal 124. The grip section 120 is a holding section shaped to be easily gripped with the user's right hand when holding the camera 100. The shutter button 101 and main electronic dial 104 are positioned so that they can be operated with the index finger of the right hand when the user holds the camera 100 by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand. Similarly, the sub electronic dial 105 and touch bar 119 are positioned so that they can be operated with the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section provided on the back side of the camera 100, in a position where it is easy to place the thumb of the right hand holding the grip section 120 when none of the operation sections are being operated. The thumb rest section 121 is made of a rubber member or the like to enhance holding strength (grip feeling). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 227 and the slot by closing the slot for storing the recording medium 227, which will be described later. The communication terminal 124 is a terminal for communicating with the detachable lens unit 200, which will be described later, of the camera 100.

[0019] Fig. 3 is a diagram showing an example of the internal configuration of the camera 100. Note that the same components as those in Fig. 2 are given the same reference numerals and descriptions thereof will be omitted where appropriate. A lens unit 200 can be attached to the camera 100.

[0020] First, a description will be given of the lens unit 200. The lens unit 200 is a type of interchangeable lens that can be attached to and detached from the camera 100. The lens unit 200 is a single lens, and is an example of a normal lens.

[0021] The lens unit 200 has an aperture 201, a lens 202, an aperture drive circuit 203, an AF drive circuit 204 (autofocus drive circuit), a lens system control circuit 205, and a communication terminal 206. The aperture 201 is configured so that its aperture diameter is adjustable. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit 50 (described later). The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203 and adjusts the focus by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, communication is performed via a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100 side.

[0022] Next, the camera 100 will be described. The camera 100 has a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50. The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an imaging element (image sensor) formed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 211 may also have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts an analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined calculations using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. This processing allows for TTL (through-the-lens) AF processing, AE (auto exposure) processing, EF (pre-flash) processing, etc. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data, and performs TTL AWB (auto white balance) processing based on the obtained calculation results.

[0023] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data to be displayed on the display unit 108 and EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 215 also serves as a memory for displaying images (video memory).

[0024] The D / A converter 216 converts image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 or the EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display in accordance with the analog signal from the D / A converter 216. The display unit 108 or the EVF 217 is, for example, an LCD or organic EL display. A digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216 and then sequentially transferred to and displayed on the display unit 108 or the EVF 217, thereby performing live view display.

[0025] The system control unit 50 is a control unit including at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 executes programs recorded in the nonvolatile memory 219 to realize each process in the flowcharts described below. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0026] The camera 100 also includes a system memory 218, a nonvolatile memory 219, a system timer 220, a communication unit 221, an attitude detection unit 222, and an eyepiece detection unit 118. The system memory 218 may be, for example, a RAM. Constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are loaded into the system memory 218. The nonvolatile memory 219 is an electrically erasable and recordable memory, for example, an EEPROM. The nonvolatile memory 219 stores constants and programs for the operation of the system control unit 50. The programs referred to here are programs for executing flowcharts, which will be described later. The system timer 220 is a timing unit that measures the time used for various controls and the time of a built-in clock. The communication unit 221 transmits and receives video and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 221 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 221 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive image data and other various information from external devices. The orientation detection unit 222 detects the orientation of the camera 100 relative to the direction of gravity. Based on the orientation detected by the orientation detection unit 222, it is possible to determine whether an image captured by the imaging unit 211 was captured with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate and record the image. The orientation detection unit 222 can use, for example, an acceleration sensor or a gyro sensor. The orientation detection unit 222 can also be used to detect movement of the camera 100 (panning, tilting, lifting, whether the camera is stationary, etc.).

[0027] The eyepiece detection unit 118 can detect the approach of an object to the eyepiece 116 of the eyepiece finder 117, which incorporates the EVF 217. The eyepiece detection unit 118 can be, for example, an infrared proximity sensor. When an object approaches, infrared light is emitted from a light-emitting unit of the eyepiece detection unit 118, reflected by the object, and received by a light-receiving unit of the infrared proximity sensor. The amount of received infrared light can be used to determine the distance from the eyepiece 116 to the object. In this way, the eyepiece detection unit 118 performs eyepiece detection, which detects the proximity of the object to the eyepiece 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (approach) and departure (away from) of an eye (object) to the eyepiece 116 of the eyepiece finder 117. When an object is detected approaching within a predetermined distance from the eyepiece 116 from a non-eyepiece state (not approaching state), it is detected that the eye has been placed in proximity. On the other hand, when an object whose proximity has been detected moves away from the eye-closed state (approaching state) by a distance greater than a predetermined distance, it is detected that the eye has moved away. The threshold for detecting eye-closedness and the threshold for detecting eye-away may be different, for example, by providing hysteresis. Furthermore, after detecting eye-closedness, the eye-closed state is maintained until eye-away is detected. After detecting eye-away, the non-eye-closed state is maintained until eye-closedness is detected. The system control unit 50 switches the display unit 108 and the EVF 217 between display (display state) and non-display (non-display state) depending on the state detected by the eye-closedness detection unit 118. Specifically, when at least in a shooting standby state and the display destination switching setting is automatic switching, the display unit 108 is set as the display destination and the display is turned on, and the EVF 217 is hidden, when the eye is not placed near the object. Furthermore, when the eye is placed near the object, the EVF 217 is set as the display destination and the display is turned on, and the display unit 108 is hidden. Note that the eye proximity detection unit 118 is not limited to an infrared proximity sensor, and any other sensor may be used as long as it can detect a state that can be considered as eye proximity. The camera 100 also has an outside-finder display unit 107, an outside-finder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium I / F 226, and an operation unit 228. The outside-finder display unit 107 displays various setting values of the camera 100, such as the shutter speed and aperture, via the outside-finder display drive circuit 223.The power supply control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery charge. The power supply control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each unit, including the recording medium 227. The power supply unit 225 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The recording medium I / F 226 is an interface with the recording medium 227, such as a memory card or a hard disk. The recording medium 227 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 227 may be removable or built-in.

[0028] The operation unit 228 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode selector switch 103, the touch panel 109, and other operation units 229. The other operation units 229 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction keys 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, and the touch bar 119.

[0029] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 is turned on when the shutter button 101 is pressed halfway (a shooting preparation command) and generates a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processes such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 231 is turned on when the shutter button 101 is pressed fully (a shooting command) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of shooting processes in response to the second shutter switch signal SW2, from reading out a signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227.

[0030] The mode selector switch 103 switches the operation mode of the system control unit 50 to one of still image capture mode, video capture mode, playback mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide capture settings for different capture scenes. The user can directly switch to one of the above-mentioned capture modes using the mode selector switch 103. Alternatively, the user can first switch to a list screen of capture modes using the mode selector switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video capture mode may also include multiple modes.

[0031] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating input coordinates on the touch panel 109 with display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user is directly operating the screen displayed on the display unit 108. The touch panel 109 can be any of a variety of systems, including a resistive film system, a capacitive system, a surface acoustic wave system, an infrared system, an electromagnetic induction system, an image recognition system, and an optical sensor system. Depending on the system, there are systems that detect a touch by contact with the touch panel 109, and systems that detect a touch by the approach of a finger or a pen to the touch panel 109, but either system may be used.

[0032] The system control unit 50 can detect the following operations or states on the touch panel 109.

[0033] A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down).

[0034] A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On).

[0035] The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move).

[0036] The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the end of the touch (hereinafter referred to as "touch-up").

[0037] A state in which nothing is touching the touch panel 109 (hereinafter referred to as Touch-Off).

[0038] When a touch down is detected, a touch on is also detected at the same time. After a touch down, a touch on is usually continued to be detected unless a touch up is detected. If a touch move is detected, a touch on is also detected at the same time. Even if a touch on is detected, a touch move is not detected unless the touch position moves. Once it is detected that all fingers or pens that were touching have touched up, a touch off occurs.

[0039] These operation states and the position coordinates of the finger or pen touching the touch panel 109 are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding touch-move, the movement direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced across the touch panel 109 as if flicking. If a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is then detected, it is determined that a flick has been performed (it can be determined that a flick occurred following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called a pinch in, and a touch operation in which the touch positions are moved farther apart is called a pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply pinch).

[0040] ●VR180 Lens 300 Configuration Fig. 4 is a diagram showing an example of the configuration of a VR180 lens 300 that can be attached to the camera 100. Fig. 4 shows the VR180 lens 300 attached to the camera 100. Note that in the camera 100 shown in Fig. 4, the same components as those described in Fig. 3 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0041] The VR180 lens 300 is a type of interchangeable lens that can be attached to and detached from the camera 100. The VR180 lens 300 is a twin lens that can capture images with parallax between left and right images. The VR180 lens 300 has two optical systems, each with a wide viewing angle of approximately 180 degrees, and can capture images of the range of the forward hemisphere. Specifically, the two optical systems of the VR180 lens 300 can each capture an object with a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle).

[0042] The VR180 lens 300 includes a right-eye optical system 301R having a plurality of lenses and a reflecting mirror, a left-eye optical system 301L having a plurality of lenses and a reflecting mirror, and a lens system control circuit 303. The right-eye optical system 301R corresponds to an example of a first optical system, and the left-eye optical system 301L corresponds to an example of a second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the lenses 302R and 302L located on the subject side face in the same direction, and their optical axes are approximately parallel.

[0043] The VR180 lens 300 of this embodiment is a lens for capturing images for so-called VR180, which is a VR image format that enables binocular stereoscopic viewing and has a 180-degree field of view. The VR180 lens 300 has fisheye lenses that enable the right-eye optical system 301R and the left-eye optical system 301L to each capture an approximately 180-degree range. Note that the VR180 lens 300 may be a lens that can capture a wide viewing angle range of approximately 160 degrees, which is narrower than the 180-degree range, as long as the right-eye optical system 301R and the left-eye optical system 301L can each acquire an image that allows binocular VR display as VR180. The VR180 lens 300 can form a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L, which has parallax from the right image, on one or two image pickup elements of the attached camera.

[0044] Furthermore, the VR180 lens 300 is attached to the camera 100 via the lens mount unit 304 and the camera mount unit 305 of the camera 100. By attaching the VR180 lens 300 to the camera 100, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the VR180 lens 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the VR180 lens 300.

[0045] In this embodiment, a right image formed via the right-eye optical system 301R and a left image formed via the left-eye optical system 301L, which has parallax from the right image, are formed side by side on the imaging unit 211 of the camera 100. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single imaging element. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. In this way, by using the VR180 lens 300, two images with parallax can be simultaneously acquired (as a set) from two locations (optical systems), the right-eye optical system 301R and the left-eye optical system 301L. Furthermore, by dividing the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image with a range of approximately 180 degrees, known as VR180.

[0046] Here, a VR image is an image that can be displayed in VR, as described below. VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera) and panoramic images with a wider image range (effective image range) than the display range that can be displayed at one time on a display unit. VR images are not limited to still images, but also include videos and live images (images acquired from a camera in almost real time). VR images have an image range (effective image range) of up to 360 degrees horizontally and vertically. VR images also include images with a wider angle of view than the angle of view that can be captured by a normal camera, or an image range that can be displayed at one time on a display unit, even if the field of view is less than 360 degrees horizontally or vertically. An image captured by the camera 100 using the VR180 lens 300 described above is a type of VR image. VR images can be displayed in VR by, for example, setting the display mode of a display device (a display device that can display VR images) to "VR view." By displaying a VR image with a 360-degree angle of view and changing the orientation of the display device left and right (horizontal rotation direction), the user can view seamless, omnidirectional images left and right.

[0047] Here, VR display (VR view) refers to a display method (display mode) that displays a VR image with a field of view that corresponds to the orientation of the display device, allowing for a change in display range. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by mapping a VR image onto a virtual sphere (deformation that corrects distortion). VR display also includes "two-eye VR display (two-eye VR view)," which displays a VR image for the left eye and a VR image for the right eye side by side by mapping them onto a virtual sphere. Stereoscopic viewing is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax. Regardless of the VR display, for example, when a user wears a display device such as an HMD (head-mounted display), the image displayed corresponds to the orientation of the user's face. For example, suppose a VR image is displayed with a field of view centered at 0 degrees left and right (a specific direction, e.g., north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is flipped from this state (for example, the display surface is changed from facing south to facing north), the display range of the same VR image will be changed to an image with a field of view centered at 180 degrees left and right (the opposite direction, for example south) and 90 degrees up and down. In other words, when the user is wearing the HMD and turns their face from north to south (i.e., turns around), the image displayed on the HMD will also change from a north image to a south image.

[0048] The VR image captured using the VR180 lens 300 of this embodiment is a VR180 image capturing a range of approximately 180 degrees forward, and does not contain any image in a range of approximately 180 degrees backward. If such a VR180 image is displayed in VR and the position of the display device is changed to the side where no image exists, a blank area will be displayed.

[0049] By displaying VR images in this way, the user visually feels as if they are inside the VR image (in the VR space). Note that the method of displaying VR images is not limited to changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operation via a touch panel or directional buttons. Furthermore, during VR display (display mode "VR view"), in addition to changing the display range due to posture changes, the display range may also be changed in response to touch-move on the touch panel, dragging with a mouse, pressing directional buttons, etc. Note that a smartphone attached to VR goggles (head-mounted adapter) is a type of HMD.

[0050] ●PC500 configuration Fig. 5 is a block diagram showing the configuration of PC 500. In Fig. 5, 501 is a control unit that controls the entire PC 500, such as a Central Processing Unit (CPU). 502 is a Read Only Memory (ROM) that stores programs and parameters that do not require modification. ROM 502 stores an information processing program as program code that can be read by control unit 501, and this information processing program code is executed by control unit 501. 503 is a Random Access Memory (RAM) that temporarily stores programs and data supplied from an external device, etc.

[0051] An external storage device 504 includes a hard disk or flash memory that is fixedly installed in the PC 500. Alternatively, the external storage device 504 may be an external storage device including an optical disk such as a floppy disk (FD) or compact disk (CD) that is detachable from the PC 500, a magnetic or optical card, an IC card, a memory card, etc. Image files acquired by the PC 500 from the camera 100 are stored in the external storage device 504.

[0052] Reference numeral 505 denotes an operation unit such as buttons or a touch panel that receives user operations and inputs data. Reference numeral 506 denotes a display unit that displays data held by the PC 500 and data supplied thereto. Reference numeral 507 denotes a communication unit that communicates with external devices such as the camera 100. Reference numeral 508 denotes a system bus that connects the components of the PC 500 so that they can communicate with each other.

[0053] ●Features of images taken with the VR180 lens 300 In the case of a single lens optical system, an image rotated 180 degrees is formed on the imaging element. When generating a normal image from the 180-degree rotated image, the camera 100 performs a 180-degree rotation process to align the up-down direction of the image with the up-down direction of the subject. When capturing an image using the camera 100 equipped with the VR180 lens 300, an image from each optical system is formed on a single imaging element via the right-eye optical system 301R and the left-eye optical system 301L. At this time, the image from each optical system is rotated 180 degrees for each optical system. As in the case of a single lens optical system, the camera 100 rotates the entire image 180 degrees to align the up-down direction of the image with the up-down direction of the subject. While rotation during image formation occurs on an optical system-by-optical system basis, rotation during image generation occurs for the entire image. Therefore, the image corresponding to the left-eye optical system moves to the right of the overall image, and the image corresponding to the right-eye optical system moves to the left of the overall image.

[0054] ●Photography processing by camera 100 Next, with reference to Figures 6A and 6B, a description will be given of the photographing process by the camera 100. When the user turns on the power of the camera 100, the process of the flowcharts in Figures 6A and 6B starts.

[0055] In S601, the system control unit 50 determines whether the firmware of the camera 100 is compatible with VR180 lenses. If it is determined that the firmware is compatible with VR180 lenses, the processing proceeds to S602. If it is determined that the firmware is not compatible with VR180 lenses, the processing proceeds to S622. Because the optical system of a VR180 lens is different from that of a general lens, the camera 100 needs to be able to read and record VR180 lens metadata for post-processing. Therefore, the system control unit 50 determines whether the firmware is compatible with VR180 lenses.

[0056] In S602, the system control unit 50 determines whether a VR180 lens (for example, the VR180 lens 300 shown in FIG. 4) is attached to the camera 100. If it is determined that a VR180 lens is attached, the process proceeds to S603. If it is determined that a VR180 lens is not attached, the process proceeds to S622.

[0057] In S603, the system control unit 50 acquires the design values of the VR180 lens. The lens design values are used in the left-right swap processing and the equirectangular conversion processing, which will be described later.

[0058] In S604, the system control unit 50 acquires individual values of the VR180 lens. In addition to lens design values, the VR180 lens also stores information on individual values such as manufacturing errors. In the equirectangular conversion process, using the manufacturing error values can produce better results than equirectangular conversion using only the design values.

[0059] In S605, the system control unit 50 acquires an image from the imaging unit 211.

[0060] In S606, the system control unit 50 displays the image acquired in S605 on the EVF 217 (live view display).

[0061] In S607, the system control unit 50 determines whether a recording start instruction has been issued by operating the shutter button 101. If it is determined that a recording start instruction has been issued, the process proceeds to step S608. If it is determined that a recording start instruction has not been issued, the process proceeds to step S605. Therefore, live view display on the EVF is repeated until a recording start instruction is issued by the user. The recording start instruction may be an instruction to shoot a still image or a video.

[0062] In S608, the system control unit 50 acquires an image from the imaging unit 211.

[0063] In S609, the system control unit 50 acquires shooting information such as the shutter speed at the time of shooting, and metadata such as orientation information of the camera 100 at the time of shooting. The orientation information is acquired from the orientation detection unit 222. Furthermore, in the case of RAW image shooting, the system control unit 50 also acquires metadata necessary for development.

[0064] In S610, the system control unit 50 records the image acquired in S608 in a file.

[0065] In S611, the system control unit 50 records the shooting information and metadata acquired in S609 in association with a file. For example, the system control unit 50 records the shooting information and metadata in the same file as the image.

[0066] In S612, the system control unit 50 records the information about the VR180 lens acquired in S603 and S604 in association with a file. For example, the system control unit 50 records the information about the VR180 lens in the same file as the image.

[0067] In S613, the system control unit 50 determines whether the user has issued an instruction to end recording. In the case of still image shooting, since a single image is being shot, it is assumed that the instruction to start recording and the instruction to end recording are issued simultaneously. In the case of video shooting, the user presses the shutter button 101 again to end the recording instruction. If it is determined that an instruction to end recording has been issued, the processing of this flowchart ends. If it is determined that an instruction to end recording has not been issued, the processing proceeds to S608. The system control unit 50 repeats the processing of S608 to S613, allowing video images to be recorded one after another into a file.

[0068] Next, a case where the processing step shifts from S601 or S602 to S622 will be described. Note that when the processing step shifts from S601 to S622, the firmware of the camera 100 does not support a VR180 lens. Therefore, the camera 100 cannot perform processing specific to a VR180 lens. Therefore, even if a VR180 lens is attached to the camera 100, the camera 100 performs the same processing as when a normal lens is attached.

[0069] In S622, the system control unit 50 determines whether a lens is attached to the camera 100. If it is determined that a lens is attached, the process proceeds to step S623. If it is determined that a lens is not attached, the process of this flowchart ends.

[0070] In S623, the system control unit 50 acquires general information about the lens (for example, the lens name).

[0071] In S624, the system controller 50 determines whether the general information about the lens was acquired in S623. If the general information about the lens was acquired, the process proceeds to S625. If the general information about the lens was not acquired, the process proceeds to S626. For example, if the camera 100 and the lens are from different manufacturers and the camera 100 cannot communicate with the lens, the camera 100 cannot acquire information from the lens.

[0072] In S625, the system control unit 50 stores the general information about the lens acquired in S623 in the system memory 218.

[0073] The processing in S626 to S632 is the same as the processing in S605 to S611, and therefore the description thereof will be omitted.

[0074] In S633, the system controller 50 determines whether general lens information is stored in the system memory 218 (see S625). If general lens information is stored, the process proceeds to S634. If general lens information is not stored, the process proceeds to S635.

[0075] In S634, the system control unit 50 records the general information about the lens stored in the system memory 218 in association with a file. For example, the system control unit 50 records the general information about the lens in the same file as the image.

[0076] The process of S635 is the same as the process of S613, and if an instruction to end recording is given, the process of this flowchart ends, and if an instruction to end recording is not given, the process proceeds to step S629. The system control unit 50 repeats the processes of S629 to S635, allowing moving images to be recorded one after another in a file.

[0077] 6A and 6B may be still images or moving images. Furthermore, the images may be RAW still images or RAW moving images other than general JPG still images or MP4 moving images.

[0078] ●Image taken by camera 100 equipped with VR180 lens 300 Next, with reference to FIGS. 11(a) and 11(b), an example of an image captured and recorded by the camera 100 equipped with the VR180 lens 300 will be described.

[0079] The VR180 lens 300 is designed with specific values in mind regarding the size of the image sensor and the angle of view for recording. For example, in the case of a VR180 lens 300 designed for a DCI 8K angle of view, the expected angle of view is 8192 x 4320.

[0080] On the other hand, depending on the shooting settings, the camera 100 can also record at an angle of view other than the angle of view assumed by the VR180 lens 300. For example, in the case of a UHD 8K setting, the angle of view is 7680 x 4320. In this case, even though the entire image from the VR180 lens 300 is input to the imaging element, part of it (for example, both ends) will not be recorded.

[0081] 11(a) is an example of an image recorded at an angle of view that uses the entire image sensor using a camera 100 with an image sensor of the size assumed by the VR180 lens 300. This is the case when the size of the image sensor assumed by the VR180 lens 300 is DCI 8K (8192 x 4320) and the recording size is also DCI 8K. In this case, two circular fisheye images lined up side by side are recorded in their entirety.

[0082] FIG. 11(b) shows an example of an image recorded using a camera 100 with an image sensor of a size intended for a VR180 lens, with an angle of view that uses only a portion of the image sensor. For example, this is the case when the image sensor size intended for the VR180 lens is DCI 8K (8192x4320) and the recording size is UHD 8K (7680x4320). In this case, two circular fisheye images aligned side by side are completely formed on the image sensor, but they are recorded with portions missing. For example, as shown in FIG. 11(b), of the two circular fisheye images aligned side by side, a portion of the left side of the left circular image and a portion of the right side of the right circular image are recorded with portions missing. In the following description, the missing areas of the circular fisheye images are also referred to as "missing areas."

[0083] ● Equirectangular conversion processing using PC500 7 and 8A, the equirectangular conversion process by the PC 500 will be described. The PC 500 performs the equirectangular conversion process on an image included in an image file acquired from the camera 100. By the equirectangular conversion process, two equirectangular projection images on the left and right are generated from two circular fisheye images on the left and right as described with reference to FIGS. 11(a) and 11(b).

[0084] In S701, the control unit 501 reads an image from an image file. This is a still image or a moving image from an image file shown in FIG. 9, which will be described later.

[0085] In S702, the control unit 501 reads the shooting information and metadata other than the VR180 lens from the image file. This is the shooting information and metadata other than the VR180 lens of the image file shown in Fig. 9, which will be described later.

[0086] In S703, the control unit 501 determines whether the image file has VR180 lens metadata. This is the VR180 lens metadata of the image file shown in FIG. 9, which will be described later. If the image file has VR180 lens metadata, the processing step proceeds to S704. If the image file does not have VR180 lens metadata, the processing step proceeds to S706.

[0087] In S704, the control unit 501 reads the VR180 lens metadata from the image file.

[0088] In S705, the control unit 501 acquires the center coordinates and radii of the two left and right circular images in the recorded image from the design values of the VR180 lens among the VR180 lens metadata read in S704. Furthermore, if the VR180 lens metadata has an individual value (manufacturing error) of the VR lens, the control unit 501 also reflects the individual value to acquire the center coordinates and radii of the two left and right circular images.

[0089] When the processing step proceeds from S703 to S706, the control unit 501 determines whether the image file has a lens name. For example, if the firmware of the camera 100 is not compatible with the VR180 lens, the camera 100 may not record VR180 lens metadata, but may record the lens name in a metadata area other than the VR180 lens. If the image file has a lens name, the processing step proceeds to S707. If the image file does not have a lens name, the processing step proceeds to S709.

[0090] In S707, the control unit 501 acquires the lens name from the image file.

[0091] In S708, the control unit 501 determines whether the lens name acquired in S707 indicates a known VR180 lens. If the lens name acquired in S707 indicates a known VR180 lens, the processing proceeds to S709. If the lens name acquired in S707 does not indicate a known VR180 lens, the processing proceeds to S710.

[0092] In S709, the control unit 501 determines the center coordinates and radii of the two circular images (left and right) in the recorded image based on the lens name acquired in S707. To achieve this processing, the program of the PC 500 is configured to store the lens name and the design values of the center coordinates and radii of the two circular images (left and right) in the image captured with that lens. Alternatively, this information may be provided to the program from an external source.

[0093] When the process proceeds from S706 or S708 to S710, the control unit 501 determines whether the image read in S701 is a pair of left and right circular fisheye images. For example, the control unit 501 creates a monochrome binary image in which the pixel value of the read image is black if the pixel brightness value is equal to or less than a threshold value (a value close to black), and white if the pixel brightness value exceeds the threshold value. The control unit 501 can then determine whether the image read in S701 is a pair of left and right circular fisheye images based on whether the white areas in the monochrome binary image are two circles. If the read image is a pair of left and right circular fisheye images, the process proceeds to S711. If the read image is not a pair of left and right circular fisheye images, the process of this flowchart ends.

[0094] In S711, the control unit 501 detects the center coordinates and radii of the two circular fisheye images of the image read in S701. It is known that when two lines are drawn intersecting two points on the circumference of a circle and perpendicular lines are drawn from the midpoints of each line, the intersection of the two perpendicular lines is the center of the circle. Therefore, for example, using the monochrome binary image created in S710, the control unit 501 examines pixel values in the horizontal direction at any vertical position in a circular area, and determines the X coordinate X1 of pixels that change from black to white and the X coordinate X2 of pixels converted from white to black. The control unit 501 also examines pixel values in the vertical direction at any horizontal position, and determines the Y coordinate Y1 of pixels that change from black to white and the Y coordinate Y2 of pixels converted from white to black. In this case, the center coordinates of the circle are (X1 + X2) / 2 and (Y1 + Y2) / 2, respectively. Furthermore, if pixel values are examined horizontally from the center coordinate of the circle and the X coordinate of the pixel that changes from white to black is X3, the radius of the circle is X3-((X1+X2) / 2).

[0095] In S712, the control unit 501 creates an equirectangular projection image from the two circular fisheye images using the center coordinates and radii of the two circular fisheye images obtained by the processing of S705, S709, or S711. Here, as described with reference to Fig. 11(b), even when images are taken with a camera 100 equipped with a VR180 lens, the two circular fisheye images may not be recorded in their entirety (may have missing parts).

[0096] FIG. 8A is a flowchart showing details of the process of S712 (processing for creating an equirectangular projection image) according to the first embodiment.

[0097] In S801, the control unit 501 calculates the image size required to record the entire image captured by the VR180 lens. This can be calculated from the center coordinates and radii of the two circular fisheye images obtained by the processing of S705, S709, or S711. For example, if two circular fisheye images are recorded side by side, let X1 be the distance from the center of the image to the X coordinate of the center position of the left circular fisheye image, X2 be the distance from the center of the image to the X coordinate of the center position of the right circular fisheye image, and R be the radius of the left and right circular fisheye images. Here, (R<=X1) and (R<=X2). In this case, the image size required to completely record the two left and right circular fisheye images is width ((X1+X2)×2) and height (R×2).

[0098] In S802, the control unit 501 determines whether the image size acquired from the image file is smaller than the image size calculated in S801. If the image size acquired from the image file is smaller than the image size calculated in S801, the processing proceeds to S803. This means that the two circular fisheye images have been recorded with some of them missing. If the image size acquired from the image file is equal to or larger than the image size calculated in S801, the processing proceeds to S806. This means that the two circular fisheye images have been recorded in their entirety.

[0099] In S803, the control unit 501 allocates a buffer corresponding to the image size calculated in S801 to the RAM 503, and sets all pixel values to black.

[0100] In S804, the control unit 501 copies the image data acquired in S701 to the buffer allocated in S803. At this time, the control unit 501 copies the image data so that the distance from the horizontal center of the buffer allocated in S803 to the X coordinate of the center position of the left circular fisheye image is X1 calculated in S801. As a result, the image in the buffer becomes as shown in FIG. 11(c).

[0101] In S805, the control unit 501 selects the image in the buffer as the source image for equirectangular transformation.

[0102] The upper diagram in FIG. 12A shows the source image stored in the buffer. The lower diagram in FIG. 12A shows the destination image (conversion result image) of the equirectangular conversion. As indicated by arrow 1201, the coordinates of the right edge of the left-eye equirectangular projection image correspond to the coordinates of the right edge of the left-eye circular fisheye image, but the pixel at that coordinate has not been recorded, so it points to the black pixel set in S803. Similarly, as indicated by arrow 1202, the coordinates of the left edge of the right-eye equirectangular projection image correspond to the coordinates of the left edge of the right-eye circular fisheye image, but the pixel at that coordinate has not been recorded, so it points to the black pixel set in S803.

[0103] When the processing step proceeds from S802 to S806, the control unit 501 selects the image acquired in S701 as the source image for equirectangular transformation.

[0104] In S807, the control unit 501 determines whether processing of all pixels of the converted image (equirectangular projection image) is complete. If processing of all pixels of the converted image is complete, the processing of this flowchart ends. If unprocessed pixels exist, the processing proceeds to S808.

[0105] In S808, the control unit 501 determines (identifies) the coordinates of the source image (circular fisheye image) that correspond to the coordinates of the pixel to be processed in the destination image (equirectangular projection image). This can be done using a common method for equirectangular transformation. As shown in FIG. 12A, even if a portion of the recorded circular fisheye image is missing, by using the image in the buffer prepared in S803 to S805 as the source image, invalid areas are not referenced during equirectangular transformation. As a result, areas of the equirectangular projection image that correspond to areas that do not exist in the circular fisheye image can be displayed as black pixels.

[0106] In S809, the control unit 501 acquires a pixel value from the coordinates of the source image obtained in S808, and sets the acquired pixel value to the pixel to be processed in the destination image.

[0107] In S810, the control unit 501 selects the next pixel of the destination image as the pixel to be processed. For example, when processing from the upper left pixel to the lower right pixel of the destination image, the next pixel is the pixel immediately to the right in the same row. However, if the X coordinate of the pixel immediately to the right reaches the image width, the leftmost pixel one row below becomes the next pixel.

[0108] By performing the above processing, even when equirectangular transformation processing is performed on a circular fisheye image with missing areas, the coordinates of the circular fisheye image corresponding to the coordinates in the equirectangular projection image will no longer point to the missing areas of the circular fisheye image.

[0109] ●Image file format Next, the structure of an image file recorded by the camera 100 (an image file processed by the PC 500) will be described with reference to FIG. 9. In FIG. 9, 901 is a file header. The file header records information such as the type of image. 902 is shooting information at the time the image recorded in the file was captured. The shooting information records information such as the shutter speed and aperture. 903 is VR180 lens metadata (metadata related to the VR180 lens). As the VR180 lens metadata, information such as the lens name, the radius of the circular fisheye lens, and information on manufacturing errors are recorded. 904 is metadata for lenses other than the VR180 lens. In the case of videos, information that changes with each frame is recorded in the metadata for lenses other than the VR180 lens, and in the case of RAW, data required for development is recorded. A camera equipped with firmware that is not compatible with VR180 lenses records the lens name in the metadata for lenses other than the VR180 lens. 905 is still image or video data. In the case of videos, audio data is also recorded in addition to images.

[0110] 9, the image file records shooting information 902, VR180 lens metadata 903, and metadata other than the VR180 lens 904. However, a configuration may be adopted in which the shooting information 902, VR180 lens metadata 903, and metadata other than the VR180 lens 904 are recorded in a file different from the image file, and the recorded information is associated with the image file.

[0111] Information acquired from the VR180 lens 300 and the camera 100 10(a), an example of information that the camera 100 acquires from the VR180 lens 300 will be described. The following information is acquired from the VR180 lens 300. 1. Lens design values 2. Lens individual value (manufacturing error value) 3. Lens flag information 4. Lens focal length information 5. Lens thermometer information

[0112] The lens design values are design value data common to both eyes of VR180 for aberration correction. Details of the lens design values will be described later with reference to FIG. 10(b).

[0113] The lens individual value (manufacturing error value) is data that indicates the manufacturing error of the VR180 lens 300. During the manufacturing process of the VR180 lens 300, errors occur on both the left and right sides. If equirectangular conversion processing is performed by the PC500 when errors exist, the quality of the 3D display will deteriorate. To mitigate this problem, measurement results of errors detected during the lens manufacturing process are recorded inside the lens. Details of the lens individual value will be described later with reference to FIG. 10(b).

[0114] When performing image processing, the PC 500 needs to determine whether or not to perform conversion processing on image data captured with the VR180 lens 300. The lens flag is recorded to enable this determination. The lens flag is a flag that indicates that the image data is image data captured with a VR180 lens.

[0115] The lens focal length is the distance from the "principal point" at the center of the lens to the image formed on the imaging element. When image data captured with the VR180 lens 300 is converted by the PC 500, a highly accurate (decimal point) value for the lens focal length is required to maintain the quality of the 3D display.

[0116] The lens thermometer information indicates the temperature of the VR180 lens 300. Because lens temperature information is recorded during shooting, the PC 500 can grasp the ambient temperature.

[0117] 10(b), the lens design values and lens individual values (manufacturing error values) will be described. This information is used for left-right swap processing, equirectangular conversion processing, etc. The lens design values include, for example, the following information: 1. Image circle position 2. Image circle diameter 3. Angle of View 4. Distortion correction coefficient

[0118] The image circle position is the design value of the center position of the image circle formed on the imaging element. For example, the image circle position is defined as horizontal and vertical coordinates with the mount as the origin.

[0119] The image circle diameter is the diameter of the image circle formed on the image sensor.

[0120] The field angle is the range of the field angle of an image formed within an image circle.

[0121] The distortion correction coefficient is the ratio of the design image height to the ideal image height of the lens. A value may be set for each image height, and the values may be interpolated between them or approximated by a polynomial.

[0122] Furthermore, when an app on the PC 500 is connected to the camera 100 and displayed, a magic window display is performed on the screen of the PC 500. In this case, the "image circle position," "image circle diameter," and "angle of view" are used to maintain the posture and display quality. The PC 500 edits this information to match the app and uses it to properly display the image on the screen. For example, the PC 500 multiplies the "image circle position" and "image circle diameter" by coefficients to match the magic window display.

[0123] The lens individual value (manufacturing error value) includes, for example, the following information: 5. Image circle misalignment 6. Optical axis tilt 7. Image magnification deviation

[0124] Image circle positional deviation is the deviation of the center position of the image circle formed on the image sensor from the design value. For example, the image circle positional deviation is defined by horizontal and vertical coordinates with the design position as the origin.

[0125] The optical axis tilt is the deviation of the optical axis on the subject side. For example, the horizontal and vertical tilt deviations are expressed as angles.

[0126] The image magnification deviation is the deviation of the image size from the design value. For example, the image magnification deviation is expressed as a ratio to the design value.

[0127] These individual values vary from lens to lens because they are generated by manufacturing errors such as lens decentering and tilt. Therefore, the lens individual values are measured and recorded for each of the left and right optical systems.

[0128] 10(c), the metadata acquired from the camera 100 will be described. This metadata is used to maintain 3D quality based on the display area and orientation information when performing conversion processing in an application on the PC 500. The metadata includes, for example, the following information: 1. Camera recording area information 2.In-camera accelerometer information 3. Right eye aperture correction information

[0129] Camera recording area information refers to the effective image area. The effective image area that can be displayed varies depending on the camera's imaging element and recording mode. This information is used to display images more accurately when conversion processing is performed by the PC500 app.

[0130] The in-camera accelerometer information is the roll and pitch of the attitude information measured using the in-camera accelerometer (level). This information is used for electronic image stabilization and horizontal correction in the PC500 app.

[0131] The right eye aperture correction information is the exposure setting value for the left eye. This information is used to maintain 3D quality and ensure a natural playback display when conversion processing is performed by the PC500 application.

[0132] As described above, according to the first embodiment, the PC 500 generates an equirectangular projection image by performing equirectangular conversion processing on a circular fisheye image that includes missing areas where no pixel values exist. Furthermore, before the equirectangular conversion processing, the PC 500 sets predetermined pixel values in the missing areas of the circular fisheye image. This makes it possible to generate an equirectangular projection image with no missing pixel values, even if the circular fisheye image includes missing areas.

[0133] In the above example, the predetermined pixel value is a pixel value indicating black, but the predetermined pixel value is not limited to a particular color and may be, for example, a pixel value indicating white.

[0134] [Second embodiment] In the second embodiment, another example of the process of creating an equirectangular projection image (S712 in FIG. 7) will be described. In this embodiment, the basic configurations of the camera 100 and the PC 500 are the same as those in the first embodiment. Below, differences from the first embodiment will be mainly described.

[0135] 8B is a flowchart showing details of the process of S712 (processing for creating an equirectangular projection image) according to the second embodiment. In Fig. 8B, unlike Fig. 8A (first embodiment), the control unit 501 uses the acquired circular fisheye image as the source image for equirectangular conversion, regardless of whether the acquired circular fisheye image has any defects.

[0136] The processing in S811 and S812 is similar to the processing in S807 and S808 in FIG. 8A, and therefore a description thereof will be omitted.

[0137] In S813, the control unit 501 determines whether the source coordinates (coordinates of the source image corresponding to the coordinates of the pixel to be processed in the destination image) calculated in S812 are within the source image (circular fisheye image). If the source coordinates are within the source image, the process proceeds to S814. If the source coordinates are not within the source image, the process proceeds to S815.

[0138] The process of S814 (processing when the source coordinates are within the source image) is the same as the process of S809 in FIG. 8A, and therefore a description thereof will be omitted.

[0139] On the other hand, if the source coordinates are not within the source image, in S815, the control unit 501 sets black to the coordinates of the pixel to be processed in the destination image (equirectangular projection image).

[0140] The upper diagram in FIG. 12B shows a partially clipped circular fisheye image used as the source image for equirectangular conversion. The lower diagram in FIG. 12B shows the destination image (conversion result image) of equirectangular conversion. As indicated by arrow 1211, the coordinates of the right edge of the left-eye equirectangular projection image correspond to the coordinates of the right edge of the left-eye circular fisheye image, but the pixel at those coordinates is not recorded. Therefore, in S815, the control unit 501 sets black to the pixel to be processed in the destination image. Similarly, as indicated by arrow 1212, the coordinates of the left edge of the right-eye equirectangular projection image correspond to the coordinates of the left edge of the right-eye circular fisheye image, but the pixel at those coordinates is not recorded. Therefore, in S815, the control unit 501 sets black to the pixel to be processed in the destination image.

[0141] The process of S816 is the same as the process of S810 in FIG. 8A, and therefore a description thereof will be omitted.

[0142] As described above, according to the second embodiment, when the coordinates of the source image (circular fisheye image) corresponding to the coordinates of the processing target pixel in the conversion destination image (equirectangular projection image) of the equirectangular conversion process point to a missing area, the PC 500 sets a pixel value (predetermined pixel value) indicating black to the processing target pixel. This makes it possible to perform the equirectangular conversion process normally even if the circular fisheye image has a missing area.

[0143] [Third embodiment] In the third embodiment, yet another example of the process for creating an equirectangular projection image (S712 in FIG. 7) will be described. In this embodiment, the basic configurations of the camera 100 and the PC 500 are the same as those in the first embodiment. Below, differences from the first embodiment will be mainly described.

[0144] When an equirectangular projection image created by equirectangular conversion processing from two circular fisheye images captured with a VR180 lens is viewed on a head-mounted display (HMD), the user can view the two equirectangular projection images as a single stereoscopic image. When an equirectangular projection image is created according to the first or second embodiment, as shown in FIGS. 12A and 12B , pixels in the equirectangular projection image may be black for only one of the left and right eyes in some areas. When a user views such two equirectangular projection images as a single stereoscopic image on an HMD, a sense of discomfort may occur. In the third embodiment, a configuration for suppressing such discomfort will be described.

[0145] 8C is a flowchart showing details of the process of S712 (processing for creating an equirectangular projection image) according to the third embodiment. In Fig. 8C, unlike Fig. 8A (first embodiment), the control unit 501 uses the acquired circular fisheye image as the source image for equirectangular conversion, regardless of whether the acquired circular fisheye image has any defects.

[0146] In S821, the control unit 501 determines whether processing of all pixels of the left eye destination image (equirectangular projection image corresponding to the left eye) has been completed. In this embodiment, in one processing loop of S821 to S827, in addition to the pixel to be processed in the left eye destination image, pixels located at the same coordinates as the pixel to be processed in the right eye destination image (equirectangular projection image corresponding to the right eye) are also processed. Therefore, when processing of all pixels of the left eye destination image has been completed, processing of all pixels of the right eye destination image has also been completed. Therefore, when processing of all pixels of the left eye destination image has been completed, the processing of this flowchart ends. If there are unprocessed pixels in the left eye destination image, the processing proceeds to step S822.

[0147] In S822, the control unit 501 obtains coordinates (left eye source coordinates) in the left eye source image (left eye circular fisheye image) corresponding to the coordinates (left eye destination coordinates) of the pixel to be processed in the left eye destination image. Here, the left eye destination coordinates are in a coordinate system with the upper left corner of the left eye destination image as the origin. The process of obtaining the left eye source coordinates corresponding to the left eye destination coordinates can be performed using a general method for equirectangular conversion.

[0148] In S823, the control unit 501 obtains coordinates (right-eye source coordinates) in the right-eye source image (right-eye circular fisheye image) corresponding to the coordinates (right-eye destination coordinates) of the pixel to be processed in the right-eye destination image. Here, the right-eye destination coordinates are in a coordinate system with the upper right corner of the right-eye destination image as the origin. The process of obtaining the right-eye source coordinates corresponding to the right-eye destination coordinates can be performed using a general method for equirectangular transformation.

[0149] In S824, the control unit 501 determines whether the left eye source coordinates are in the left eye source image and the right eye source coordinates are in the right eye source image. If the left eye source coordinates are in the left eye source image and the right eye source coordinates are in the right eye source image, the processing proceeds to S825. If at least one of the left eye source coordinates and the right eye source coordinates is not in the corresponding source image, the processing proceeds to S826.

[0150] In S825, the control unit 501 acquires pixel values from the left eye source coordinates of the left eye source image and sets them to the processing target pixel of the left eye destination image. Similarly, the control unit 501 acquires pixel values from the right eye source coordinates of the right eye source image and sets them to the processing target pixel of the right eye destination image.

[0151] On the other hand, when the processing step proceeds from S824 to S826, the control unit 501 sets black to both the pixel to be processed of the left eye destination image and the pixel to be processed of the right eye destination image.

[0152] The top diagram in Figure 12C shows a partially clipped circular fisheye image used as the source image for equirectangular transformation, and the bottom diagram in Figure 12C shows the destination image (conversion result image) of equirectangular transformation.

[0153] As indicated by the arrow 1221, the coordinates of the left edge of the left-eye equirectangular projection image correspond to the coordinates of the left edge of the left-eye circular fisheye image, and the pixel at the coordinates of the left edge of the left-eye circular fisheye image is recorded. On the other hand, the starting point of the arrow 1222 in the right-eye equirectangular projection image is at a position corresponding to the starting point of the arrow 1221 in the left-eye equirectangular projection image, but no pixel is recorded at the position indicated by the arrow 1222 in the right-eye circular fisheye image (the coordinates of the left edge of the right-eye circular fisheye image). In this case, the color black is set for both the starting point of the arrow 1221 in the left-eye equirectangular projection image and the starting point of the arrow 1222 in the right-eye equirectangular projection image.

[0154] As indicated by the arrow 1223, the coordinates of the right edge of the right-eye equirectangular projection image correspond to the coordinates of the right edge of the right-eye circular fisheye image, and the pixel at the coordinates of the left edge of the right-eye circular fisheye image is recorded. On the other hand, the starting point of the arrow 1224 in the left-eye equirectangular projection image is at a position corresponding to the starting point of the arrow 1223 in the right-eye equirectangular projection image, but no pixel is recorded at the position indicated by the arrow 1224 in the left-eye circular fisheye image (the coordinates of the right edge of the left-eye circular fisheye image). In this case, the color black is set for both the starting point of the arrow 1223 in the right-eye equirectangular projection image and the starting point of the arrow 1224 in the left-eye equirectangular projection image.

[0155] In S827, the control unit 501 selects the next pixel as the pixel to be processed for each of the left eye destination image and the right eye destination image. For example, when processing is performed from the upper left pixel to the lower right pixel of the left eye destination image, the next pixel is the pixel immediately to the right in the same row. However, if the X coordinate of the pixel immediately to the right reaches the width of the left eye destination image, the leftmost pixel of the left eye destination image in the row below is selected as the next pixel. The same applies to the right eye destination image.

[0156] As described above, according to the third embodiment, when the same coordinates of a left-eye equirectangular projection image and a right-eye equirectangular projection image correspond to a missing area in one image, the PC 500 sets a pixel value (predetermined pixel value) indicating black in the other image as well. This makes it possible to prevent pixels from becoming black only in one image for the same coordinates.

[0157] [Fourth embodiment] In the fourth embodiment, yet another example of the process of creating an equirectangular projection image (S712 in FIG. 7) will be described. In the fourth embodiment, similar to the third embodiment, the PC 500 performs a process for suppressing pixels from becoming black only in one of the images for the same coordinates in the left-eye equirectangular projection image and the right-eye equirectangular projection image, but the specific processing content for this is different from that in the third embodiment. In this embodiment, the basic configurations of the camera 100 and the PC 500 are the same as those in the third embodiment. Below, differences from the third embodiment will be mainly described.

[0158] 8D is a flowchart showing details of the process of S712 (processing for creating an equirectangular projection image) according to the fourth embodiment. In Fig. 8D, similar to Fig. 8C (third embodiment), the control unit 501 uses the acquired circular fisheye image as the source image for equirectangular conversion, regardless of whether the acquired circular fisheye image has any defects.

[0159] The processing in steps S831 to S835 is the same as the processing in steps S821 to S825 in FIG. 8C, and therefore a description thereof will be omitted.

[0160] In S836, the control unit 501 determines whether the left eye source coordinates are outside the left eye source image and the right eye source coordinates are outside the right eye source image. If the left eye source coordinates are outside the left eye source image and the right eye source coordinates are outside the right eye source image, the process proceeds to S837. If not, the process proceeds to S838.

[0161] In S837, the control unit 501 sets black to both the pixel to be processed in the left eye destination image and the pixel to be processed in the right eye destination image.

[0162] In S838, the control unit 501 determines whether the left eye source coordinates are within the left eye source image. If the left eye source coordinates are within the left eye source image, the process proceeds to S839. If the left eye source coordinates are not within the left eye source image (i.e., if the right eye source coordinates are within the right eye source image), the process proceeds to S840.

[0163] In S839, the control unit 501 acquires a pixel value from the left eye source coordinates of the left eye source image, and sets the acquired pixel value to the processing target pixel of both the left eye destination image and the right eye destination image.

[0164] In S840, the control unit 501 acquires a pixel value from the right eye source coordinates of the right eye source image, and sets the acquired pixel value to the processing target pixel of both the left eye destination image and the right eye destination image.

[0165] The top diagram in Figure 12D shows a partially clipped circular fisheye image used as the source image for equirectangular transformation, and the bottom diagram in Figure 12D shows the destination image (conversion result image) for equirectangular transformation.

[0166] As indicated by the arrow 1231, the coordinates of the left edge of the left-eye equirectangular projection image correspond to the coordinates of the left edge of the left-eye circular fisheye image, and the pixel at the coordinates of the left edge of the left-eye circular fisheye image is recorded. On the other hand, the starting point of the arrow 1232 in the right-eye equirectangular projection image is at a position corresponding to the starting point of the arrow 1231 in the left-eye equirectangular projection image, but no pixel is recorded at the position indicated by the arrow 1232 in the right-eye circular fisheye image (the coordinates of the left edge of the right-eye circular fisheye image). In this case, the processing of S839 sets the pixel value of the position indicated by the arrow 1231 in the left-eye circular fisheye image (the coordinates of the left edge of the left-eye circular fisheye image) for both the starting point of the arrow 1231 in the left-eye equirectangular projection image and the starting point of the arrow 1232 in the right-eye equirectangular projection image. In other words, as indicated by the dashed arrow 1233, pixel values of the corresponding area of the left-eye equirectangular projection image are copied to the area of the right-eye equirectangular projection image that corresponds to the missing area of the right-eye circular fisheye image.

[0167] Similarly, as indicated by the arrow 1234, the coordinates of the right edge of the right-eye equirectangular projection image correspond to the coordinates of the right edge of the right-eye circumferential fisheye image, and the pixel at the coordinates of the right edge of the right-eye circumferential fisheye image is recorded. On the other hand, the starting point of the arrow 1235 in the left-eye equirectangular projection image is at a position corresponding to the starting point of the arrow 1234 in the right-eye equirectangular projection image, but no pixel is recorded at the position indicated by the arrow 1235 in the left-eye circumferential fisheye image (the coordinates of the right edge of the left-eye circumferential fisheye image). In this case, the processing of S840 sets the pixel value of the position indicated by the arrow 1234 in the right-eye circumferential fisheye image (the coordinates of the right edge of the right-eye circumferential fisheye image) for both the starting point of the arrow 1234 in the right-eye equirectangular projection image and the starting point of the arrow 1235 in the left-eye equirectangular projection image. In other words, as indicated by the dashed arrow 1236, pixel values of the corresponding area of the right-eye equirectangular projection image are copied to the area of the left-eye equirectangular projection image that corresponds to the missing area of the left-eye circular fisheye image.

[0168] The process of S841 is the same as the process of S827 in FIG. 8C, and therefore a description thereof will be omitted.

[0169] As described above, according to the fourth embodiment, for a missing area in the first circular fisheye image, if the corresponding area in the second circular fisheye image is not a missing area, the PC 500 sets the pixel values of the corresponding area in the second circular fisheye image to the corresponding area in the first equirectangular projection image that corresponds to the missing area in the first circular fisheye image. This makes it possible to prevent pixels from becoming black only in one image for the same coordinates and to reduce black areas.

[0170] [Fifth embodiment] When an equirectangular projection image is created according to the first or second embodiment, there is a possibility that pixels of the equirectangular projection image will be black for only one of the left eye or the right eye in some areas, as shown in Figures 12A and 12B. When a user performs VR viewing based on such two equirectangular projection images, there is a possibility that a problem will occur in which black is displayed only in one of the left eye image or the right eye image, making it difficult to see. In the fifth embodiment, a configuration for suppressing this problem during viewing will be described.

[0171] Referring to FIG. 15, the selection of processing for blacked-out areas outside the angle of view will be described. Radio button 1501 corresponds to processing for blacking out pixels on the opposite side of pixels outside the angle of view. Radio button 1502 corresponds to processing for filling in blacked-out pixels outside the angle of view with pixels on the opposite side. Radio button 1503 corresponds to processing for displaying the opposite image on the left and right when pixels outside the angle of view are to be displayed. Only one of radio buttons 1501, 1502, and 1503 can be selected. 1504 is an OK button. The user can close the setting screen of FIG. 15 by pressing OK button 1504 after selecting the processing.

[0172] An example of viewing the left edge with VR180 and an example of processing the blacked-out areas outside the angle of view will be described with reference to FIGS. 16A to 16D.

[0173] FIG. 16A shows an example where no processing is performed during display. 1601 is an image from the left lens of the VR180 lens. 1602 is the display area of the image for the left eye when viewing with VR180. 1603 indicates the areas that are blacked out and are outside the angle of view of the left lens of the VR180 lens. 1604 is an image from the right lens of the VR180 lens. 1605 is the display area of the image for the right eye when viewing with VR180. 1606 indicates the areas that are blacked out and are outside the angle of view of the right lens of the VR180 lens. 1607 is an arrow indicating that the blacked out areas outside the angle of view appear only in the image for the right eye. 1609 is the display of the image for the left eye when viewing with VR180. 1610 is the display of the image for the right eye when viewing with VR180. As indicated by the arrow 1607, there is a blacked-out area in the right-eye image outside the angle of view where no image is displayed.

[0174] FIG. 16B is a display example when pixels on the opposite side corresponding to pixels outside the angle of view when radio button 1501 in FIG. 15 is selected are colored black. 1609 is a display example of the image for the left eye before processing. 1610 is a display example of the image for the right eye before processing. 1611 is a display example of the image for the left eye after processing. 1612 is a display example of the image for the right eye after processing. 1613 is a blacked-out area added by processing. Adding the blacked-out area eliminates the difficulty in viewing caused by the image being on only one side, as the same areas are blacked out in the image for the right eye and the image for the left eye.

[0175] FIG. 16C is a display example when pixels outside the angle of view when radio button 1502 in FIG. 15 is selected are supplemented with pixels on the opposite side. Explanation of the same reference numerals as in FIG. 16B will be omitted. 1614 is a display example of the left eye image after processing. 1615 is a display example of the right eye image after processing. 1616 indicates the source area from which pixels are copied to the right eye image. 1617 indicates the state in which an image is displayed by copying pixels from the left eye image. 1618 is an arrow indicating the process of copying pixels from the left eye image to the right eye image. By copying pixels, an image is added to the area that was previously blacked out in the right eye image, eliminating the difficult-to-see state where only one side of the right eye image is blacked out in the left eye image.

[0176] FIG. 16D is a display example when the image that is not outside the angle of view when radio button 1503 in FIG. 15 is selected is displayed on both. Explanation of the same reference numerals as in FIG. 16B will be omitted. 1619 is a display example of the image for the left eye after processing. 1620 is a display example of the image for the right eye after processing. 1621 is an arrow indicating that the image for the left eye is being displayed on the image for the right eye. By replacing the entire display screen where blackout display occurs outside the angle of view with the image for the left eye, the difficult-to-view condition where only one side of the image for the right eye and the image for the left eye is blacked out is eliminated.

[0177] Fig. 13 is a flowchart of image processing according to the fifth embodiment. The processing of this flowchart selects a method for processing blackened areas outside the angle of view, as explained with reference to Fig. 15, and determines whether or not there are blackened areas outside the angle of view. The processing of this flowchart starts when an instruction to play a VR180 image is input to PC 500.

[0178] In S1301, the control unit 501 displays a display setting screen for the area outside the angle of view. That is, the setting screen in FIG. 15 is displayed. In this embodiment, this screen is displayed first as a playback method setting. When the user operates the operation unit 505 to select radio button 1501, 1502, or 1503 on the screen and presses the OK button 1504, the processing proceeds to step S1302.

[0179] In S1302, the control unit 501 acquires the setting selected in S1301.

[0180] In S1303, the control unit 501 acquires the input image. In this embodiment, since there is an area outside the angle of view, the control unit 501 acquires an image after equirectangular conversion in which part of the image is missing. The acquired images are image 1601 of the left lens and image 1604 of the right lens in FIG. 16A.

[0181] In S1304, the control unit 501 cuts out the center portions of the left and right eye images (initial positions to be displayed first) from the acquired equirectangular converted images, and reads them as images to be displayed.

[0182] In S1305, the control unit 501 displays (outputs) the left and right eye images read in S1304.

[0183] In S1306, the control unit 501 determines whether an instruction to end the display has been issued. Any existing technology can be used for the UI and operation for ending the display.

[0184] In S1307, the control unit 501 determines whether an instruction to change the display position has been received. The instruction to change the display position is given by the user operating the operation unit 505 to move the display position using the cursor keys or joystick. Note that a gyro sensor may be incorporated in the display monitor itself, and the control unit 501 may acquire movement based on the acceleration of the display monitor and treat that movement as an instruction to change the display position.

[0185] In S1308, the control unit 501 acquires the display images for the left and right eyes corresponding to the change instruction made in S1307.

[0186] In S1309, the control unit 501 determines whether or not the image acquired in S1308 contains a blacked-out area outside the angle of view. One method for this determination is to determine whether or not there is a black area where the pixel values are all zero, RGB values. Another method for this determination may be to detect an area where the RGB values are below a certain threshold. Since black areas appear at the edges of an image, the determination may be limited to the edges of the image. If meta information about black areas is added to the image, the determination may be made based on whether or not the display area includes the area specified by the meta information. If it is determined that there is no area outside the angle of view, the processing proceeds to S1305. If it is determined that there is an area outside the angle of view, the processing proceeds to S1310.

[0187] In S1310, the control unit 501 processes the left and right eye images in accordance with the settings acquired in S1302.

[0188] FIG. 14 is a flowchart showing the details of the image processing in S1310.

[0189] In S1401, the control unit 501 identifies on which side, the left or right eye, the area outside the angle of view is located. Here, the control unit 501 identifies on which side, the left or right eye, the area outside the angle of view that is painted black is located, using the same method as in S1309. In the following description, it is assumed that the area outside the angle of view that is painted black is located in the image for the right eye, as shown in FIG. 16A.

[0190] In S1402, the control unit 501 determines whether or not the radio button 1503 (setting to display the same image on the left and right) was selected in S1301. If it is determined that the radio button 1503 was selected, the process proceeds to S1403. If not, the process proceeds to S1404.

[0191] In S1403, control unit 501 replaces the image including the area outside the angle of view with an image that does not include the area outside the angle of view. That is, as indicated by arrow 1621 in FIG. 16D , control unit 501 displays pre-processing left-eye image 1609 as post-processing right-eye image 1620. As a result, an image identical to left-eye image 1619 that does not include the area outside the angle of view is displayed as right-eye image 1620. Note that instead of replacing an image including an area outside the angle of view with an image that does not include the area outside the angle of view, an image that does not include the area outside the angle of view may be replaced with an image that includes the area outside the angle of view, thereby eliminating the state in which a blacked-out area is present on only one side.

[0192] In S1404, the control unit 501 selects one pixel to be processed from the area outside the angle of view in an image including the area outside the angle of view. In S1404 to S1408, the entire area outside the angle of view that is painted black is ultimately processed, so the control unit 501 may select pixels to be processed in order from the top left of the area outside the angle of view.

[0193] In S1405, the control unit 501 determines whether or not radio button 1501 (setting to make pixels in the image of the opposite eye that correspond to pixels outside the angle of view black) was selected in S1301. If it is determined that radio button 1501 was selected, the processing proceeds to S1406. If not (radio button 1502 was selected), the processing proceeds to S1407.

[0194] In S1406, the control unit 501 changes the pixel at the same position in the image for the opposite eye that corresponds to the pixel to be processed to black. The process of changing the pixel to black may involve setting an RGB value to zero, or copying the pixel value from the screen that is outside the angle of view. The processing result is as shown in FIG. 16B. As a result of processing the left-eye image 1609 that does not include the area outside the angle of view, a black area 1613 is added to the left-eye image 1611. By adding the black area to the left-eye image 1611, the same image (black) is displayed on both the left and right sides in the area that corresponds to the area outside the angle of view in the right-eye image 1612.

[0195] In S1407, the control unit 501 copies the pixel value of the same position in the image for the opposite eye that corresponds to the pixel to be processed to the pixel to be processed. The processing result is as shown in FIG. 16C. As indicated by arrow 1618, pixel values are copied from area 1616 to area 1617, eliminating the black area outside the angle of view. As a result of this processing, the same image (the image in area 1616) is displayed on both the left and right sides in the area corresponding to the area outside the angle of view of the image for the right eye 1612.

[0196] In S1408, the control unit 501 determines whether or not processing of all pixels in the area outside the angle of view has been completed. If it is determined that processing of all pixels in the area outside the angle of view has been completed, the processing of this flowchart ends. The image processed in this manner is used as the image to be displayed in S1305 of FIG. 13. If it is determined that processing of all pixels in the area outside the angle of view has not been completed, the processing step proceeds to S1404. Then, by repeating the processing loop of S1404 to S1408, processing is performed on all pixels in the area outside the angle of view.

[0197] As described above, according to the fifth embodiment, when a first specific area exists in the first display image (one of the left-eye image 1609 or the right-eye image 1610) that overlaps with a corresponding area in the equirectangular projection image that corresponds to a missing area in the circular fisheye image, the PC 500 corrects the first display image or the second display image so that the first specific area and a second specific area in the second display image (the other of the left-eye image 1609 or the right-eye image 1610) that is located at a position corresponding to the first specific area contain the same image. This correction is performed before the first display image or the second display image is output (before it is displayed). This makes it possible to improve the display quality of stereoscopic virtual reality.

[0198] In the present embodiment, the PC 500 receives an image file, but the PC 500 may receive the image as a video signal. The image may be a moving image as well as a still image.

[0199] [Sixth embodiment] In the sixth embodiment, a configuration will be described in which one circular fisheye image is cut out (extracted) from an image including two circular fisheye images having parallax.

[0200] FIG. 19 shows an application screen on the PC 500. Reference numeral 1900 denotes the entire application screen. Two circular fisheye images, side by side and recorded in a single image file, are displayed on the screen. Reference numeral 1901 denotes a circular fisheye image captured with one of the left and right optical systems of the VR180 lens. Reference numeral 1902 denotes a circular fisheye image captured with the optical system opposite to the circular fisheye image 1901, of the left and right optical systems of the VR180 lens. Reference numeral 1903 denotes a button for saving the circular fisheye image 1901 on the left side being displayed. Reference numeral 1904 denotes a button for saving the circular fisheye image 1902 on the right side being displayed. Reference numeral 1905 denotes a checkbox for selecting whether to perform additional processing (described below) when there is a loss of angle of view in the circular fisheye image.

[0201] The two circular fisheye images arranged side by side before saving and the circular fisheye image after saving that are used in this embodiment will be described with reference to Fig. 20. 2001 shows the two circular fisheye images arranged side by side after the file has been read. 2002 is an arrow indicating that the circular fisheye image on the left will be saved. 2003 is the circular fisheye image after saving.

[0202] 17 and 18, a process for selecting any one of two circular fisheye images arranged side by side, either left or right, and then cutting out and saving that circular fisheye image will be described. When the user inserts the recording medium 227, on which images are recorded by the camera 100, into the external storage device 504 and operates the operation unit 505 to select an image file, the process of the flowchart in Fig. 17 begins.

[0203] The processing of S701 to S711 in FIG. 17 is the same as the processing of S701 to S711 in FIG. 7, and therefore a description thereof will be omitted.

[0204] In S1712, the control unit 501 displays the two circular fisheye images. The application screen displaying the images is as shown in FIG.

[0205] In S1713, the control unit 501 determines whether an instruction to save the left circular fisheye image has been issued. The user issues the instruction to save by operating the operation unit 505 and pressing button 1903 in FIG. 19. If it is determined that an instruction to save the left circular fisheye image has been issued, the process proceeds to S1715. If it is determined that an instruction to save the left circular fisheye image has not been issued, the process proceeds to S1714.

[0206] In S1714, the control unit 501 determines whether an instruction to save the right circular fisheye image has been issued. The user issues a save instruction by operating the operation unit 505 and pressing button 1904 in FIG. 19. If it is determined that an instruction to save the right circular fisheye image has been issued, the process proceeds to S1715. If it is determined that an instruction to save the right circular fisheye image has not been issued, the process proceeds to S1713. The control unit 501 continues to display the two circular fisheye images until a save instruction is issued in S1713 or S1714.

[0207] In S1715, the control unit 501 stores the selected circular fisheye image. Details of the processing in S1715 will be described later with reference to FIG.

[0208] When the left circular fisheye image is saved, the saved result is circular fisheye image 2003 in Fig. 20. While the example of saving the left circular fisheye image has been described here, the right circular fisheye image can also be saved using the same procedure.

[0209] FIG. 18 is a flowchart showing the details of the process of S1715.

[0210] In S1801, the control unit 501 determines whether missing areas exist in the left and right circular fisheye images. One method for determining this is to use the center coordinates and radii of the two circular fisheye images calculated in S1705, S1709, or S1711 to determine whether the point extending the radius from the center of the left circular fisheye image to the left is within or outside the image range. While the determination is based on the left circular fisheye image here, a determination based on the right circular fisheye image may also be used. Alternatively, a determination may be made using both the left and right circular fisheye images. If it is determined that missing areas exist in the left and right circular fisheye images, the process proceeds to S1802. The process for this case will be described later. If it is determined that no missing areas exist in the left and right circular fisheye images, the process proceeds to S1811.

[0211] In S1811, the control unit 501 calculates the necessary storage range for the circular fisheye image to be saved based on the center coordinates and radius of the circular fisheye image determined in S1705, S1709, or S1711. The storage range is determined so that the center of the optical axis of the circular fisheye image is at the center of the saved image and the entire circular fisheye image is covered. The circular fisheye image may have no margins (black areas) at the top, bottom, left, and right, or may be saved with a certain size margin.

[0212] In S1812, the control unit 501 saves the images in the storage range determined in S1811. As a result of the saving, a circular fisheye image 2003 shown in FIG.

[0213] Instead of determining the center coordinates of the two circular fisheye images by image processing, a configuration may be adopted in which the center coordinates are specified by a user operation.

[0214] By performing the above process, if there are no missing areas in the left and right circular fisheye images, one circular fisheye image can be extracted. If the circular fisheye image is a video rather than a still image, the control unit 501 can select a specific frame from the video and extract one circular fisheye image from the selected frame.

[0215] When the processing step moves from S1801 to S1802, the control unit 501 identifies a missing area in the circular fisheye image to be saved. Here, an example will be described in which the circular fisheye image on the left side is the image to be saved. One method for identifying the missing area is to extend the calculated circular fisheye image to the left by a radius and determine whether the missing area is within a valid area within the image or outside the image range. The missing area can be identified from the difference between the valid distance within the image range in the left direction and the radius. For example, missing area 2102 in FIG. 21(a) is identified as the missing area.

[0216] In S1803, the control unit 501 identifies the area of the circular fisheye image on the opposite side that corresponds to the area identified in S1802. For example, this can be done by overlapping the two circular fisheye images at the center of the optical axis. Alternatively, the area may be identified by calculating the distance from the center of the optical axis of the circular fisheye image. The valid area of the image within the frame 2103 in FIG. 21(a) is identified as the corresponding portion.

[0217] In S1804, the control unit 501 determines whether or not overlapping is performed when synthesizing the fisheye missing area. This determination can be made based on whether or not the user operates the operation unit 505 to check the check box 1905 in FIG. 19. If it is determined that overlapping is performed, the processing proceeds to S1806. The processing in this case will be described later. If it is determined that overlapping is not performed, the processing proceeds to S1805.

[0218] In S1805, the control unit 501 copies the corresponding area on the opposite side (the area enclosed by the frame 2103 in FIG. 21A) to the missing area (the missing area 2102 in FIG. 21A). The copy result is shown by the frame 2105 in FIG. 21A.

[0219] In S1809, the control unit 501 calculates the necessary storage range based on the optical axis center coordinate and radius of the circular fisheye image after synthesis processing. The method of calculating the storage range is the same as in S1811.

[0220] In S1810, the control unit 501 saves the image in the saving range calculated in S1809. As a result of saving, a circular fisheye image 2106 shown in FIG.

[0221] FIG. 21(a) will now be described in more detail. 2101 is an image in which the left side of the left circular fisheye image to be saved is missing. 2102 is the missing area of the circular fisheye image identified when saving the left side. 2103 is a frame indicating the portion of the right image corresponding to the missing area 2102. 2104 is an arrow indicating the destination of synthesis. In the example of FIG. 21(a), synthesis is performed by copying a portion of the right image into the missing area of the left image. 2105 is a frame indicating the result of copying a portion of the right image into the missing area of the left image. 2106 is the left circular fisheye image obtained by saving the synthesis result. Here, an example of saving the left circular fisheye image has been described, but the right circular fisheye image can also be saved using a similar procedure.

[0222] By performing the above processing, if there are missing areas in the left and right circular fisheye images, it is possible to extract a single circular fisheye image by filling in the missing area in one circular fisheye image with the corresponding area in the other circular fisheye image.

[0223] When the processing step proceeds from S1804 to S1806, the control unit 501 identifies the overlap area where the left and right images are to be superimposed. In this embodiment, half of the circular fisheye image is identified as the overlap area. In the example of FIG. 21(b), the area of the right circular fisheye image corresponding to the frame 2108 is the overlap area.

[0224] In S1807, the control unit 501 determines the composition rate according to the composition position when the left and right images are superimposed. One method for determining the composition rate is to change the composition ratio of the left and right images from the missing area of the circular fisheye image toward the center of the optical axis of the circular fisheye image. As an example of a changing composition rate, for the missing area of the circular fisheye image on the left image, the composition rate of the right image is set to 100%. Then, the composition rate is determined by decreasing the usage rate of the right image toward the center of the circular fisheye image, so that when the center of the circular fisheye image is reached, the usage rate of the right image is 0% and the left image is 100%.

[0225] In S1808, the control unit 501 combines the right circular fisheye image with the left circular fisheye image according to the combination ratio obtained in S1807. The combined image is shown in a frame 2112 in FIG.

[0226] The processes in S1809 and S1810 following S1808 are the same as the processes in S1809 and S1810 following S1805. As a result of the storage, a circular fisheye image 2113 shown in Fig. 21(b) is obtained.

[0227] FIG. 21(b) will now be described in more detail. Reference numeral 2106 denotes an image in which the left side of the left circular fisheye image to be saved is missing. Reference numeral 2107 denotes a missing area of the circular fisheye image identified when saving the left side. Reference numeral 2108 denotes an overlapping area to reduce the sense of incongruity in the composite result, taking into account the parallax between the left and right images. Reference numeral 2109 denotes a frame indicating the right image area to be overlapped with the left image. Reference numeral 2110 denotes an arrow indicating the composite destination of the overlapping area of the left image. In this embodiment, composite processing is performed with the corresponding area of the right image. Reference numeral 2111 denotes an arrow indicating the composite destination of the overlapping area of the right image. In this embodiment, the right image is copied into the missing area of the left circular fisheye image. In the overlapping area, composite processing is performed while adjusting the composite ratio between the left and right images according to the composite position. Reference numeral 2112 denotes a circular fisheye image resulting from the composite processing. Reference numeral 2113 denotes a left circular fisheye image obtained by saving the composite result. Here, an example in which the left circular fisheye image is saved has been described, but the right circular fisheye image can also be saved using the same procedure.

[0228] In the synthesis process here, the synthesis ratio is adjusted to reduce the sense of incongruity in the synthesis result, but this may be combined with other image processing such as applying blur.

[0229] By performing the above processing, if there are missing areas in the left and right circular fisheye images, one circular fisheye image can be extracted by overlapping the overlap area containing the missing area of one circular fisheye image with the corresponding area of the other circular fisheye image.

[0230] As described above, according to the sixth embodiment, the PC 500 cuts out (extracts) one circular fisheye image from an image containing two circular fisheye images with parallax. If the circular fisheye image to be extracted contains a missing area, the PC 500 sets the pixel values of the area of the other circular fisheye image at a position corresponding to the missing area in the missing area of the circular fisheye image to be extracted. This makes it possible to extract a circular fisheye image containing the pixel values set in the missing area.

[0231] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0232] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0233] 500...PC, 501...controller, 502...ROM, 503...RAM, 504...external storage device, 505...operation unit, 506...display unit, 507...communication unit, 508...system bus

Claims

1. an acquisition means for acquiring a first circular fisheye image that is not recorded in a complete form, the first circular fisheye image having a first missing region where no pixel values exist within a first circular region corresponding to the complete form of the first circular fisheye image; a generating means for generating a first equirectangular projection image corresponding to the first circular region by performing a first equirectangular conversion process on the first circular fisheye image; Equipped with the generating means generates the first equirectangular projection image by setting a pixel value indicating a predetermined color in the first missing area of the first circular fisheye image before the first equirectangular conversion process, so that a first corresponding area of the first equirectangular projection image, which corresponds to the first missing area in the first circular area, has a pixel value; the acquiring means acquires a second circular fisheye image having a parallax with respect to the first circular fisheye image; The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens.

1. An image processing device comprising:

2. the second circular fisheye image includes a second missing region where no pixel values exist; the generating means performs a second equirectangular conversion process based on the second circular fisheye image to generate the second equirectangular projection image so that a second corresponding area in the second equirectangular projection image that corresponds to the second missing area has a pixel value that indicates the predetermined color; The generating means sets a pixel value indicating the predetermined color in an area of the first equirectangular projection image that is located at a position corresponding to the second corresponding area of the second equirectangular projection image.

2. The image processing device according to claim 1, wherein:

3. The pixel value indicating the predetermined color is a value indicating black.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. An acquisition means for acquiring a first circular fisheye image that is not recorded in a complete form, the first circular fisheye image having a first missing area where no pixel values exist within a first circular area corresponding to the complete form of the first circular fisheye image; a generating means for generating a first equirectangular projection image corresponding to the first circular region by performing a first equirectangular conversion process on the first circular fisheye image; Equipped with the generating means identifies coordinates of the first circular fisheye image corresponding to a pixel to be processed of the first equirectangular projection image in the first equirectangular conversion process, and if a pixel value does not exist at the identified coordinates, sets a pixel value indicating a predetermined color to the pixel to be processed of the first equirectangular projection image, thereby generating the first equirectangular projection image so that a first corresponding area of the first equirectangular projection image corresponding to the first missing area in the first circular area has a pixel value; the acquiring means acquires a second circular fisheye image having a parallax with respect to the first circular fisheye image; The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens.

1. An image processing device comprising:

5. An acquisition means for acquiring a first circular fisheye image that is not recorded in a complete form, the first circular fisheye image having a first missing area where no pixel values exist within a first circular area corresponding to the complete form of the first circular fisheye image; a generating means for generating a first equirectangular projection image corresponding to the first circular region by performing a first equirectangular conversion process on the first circular fisheye image; Equipped with the generating means generates the first equirectangular projection image so that a first corresponding area of the first equirectangular projection image, which corresponds to the first missing area in the first circular area, has a pixel value; the acquiring means acquires a second circular fisheye image having a parallax with respect to the first circular fisheye image; the first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens; the second circular fisheye image includes a second missing region where no pixel values exist; the generating means sets pixel values of an area of the second circular fisheye image that is located at a position corresponding to the first missing area of the first circular fisheye image in the first corresponding area of the first equirectangular projection image; When an area of the second circular fisheye image that is located at a position corresponding to a specific area of the first missing area of the first circular fisheye image is included in the second missing area, the generating means sets a pixel value that indicates a predetermined color for an area of the first corresponding area of the first equirectangular projection image that corresponds to the specific area.

1. An image processing device comprising:

6. An acquisition means for acquiring a first equirectangular projection image generated by a first equirectangular conversion process based on a first circular fisheye image that is not recorded in a complete form, and a second equirectangular projection image generated by a second equirectangular conversion process based on a second circular fisheye image that has a parallax with respect to the first circular fisheye image, the first circular fisheye image has a missing region where no pixel values exist within a circular region corresponding to the complete shape of the first circular fisheye image; the first equirectangular image corresponds to the circular region; a predetermined pixel value is set in a corresponding area of the first equirectangular projection image that corresponds to the missing area in the circular area; Acquisition means; a designation means for designating a display position for a stereoscopic virtual reality based on the first equirectangular projection image and the second equirectangular projection image; an output means for outputting a first display image corresponding to the display position in the first equirectangular projection image and a second display image corresponding to the display position in the second equirectangular projection image; a correction means for correcting the first display image or the second display image before outputting the first display image or the second display image, when a first specific area overlapping the corresponding area exists in the first display image, so that the first specific area and a second specific area in the second display image that is located at a position corresponding to the first specific area include the same image; Equipped with The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens.

1. An image processing device comprising:

7. The correction means sets the predetermined pixel value in the second specific region of the second display image.

7. The image processing device according to claim 6,

8. The correction means replaces the image of the first specific area of the first display image with the image of the second specific area of the second display image.

7. The image processing device according to claim 6,

9. The correction means replaces the first display image with the second display image.

7. The image processing device according to claim 6,

10. The predetermined pixel value is a value indicating black.

10. The image processing device according to claim 6, wherein the image processing device is a computer.

11. an acquisition means for acquiring an image including a first circular fisheye image that is not recorded in a complete form and a second circular fisheye image that has a parallax with respect to the first circular fisheye image, the first circular fisheye image having a missing area where no pixel values exist within a circular area corresponding to the complete form of the first circular fisheye image; a setting means for setting, in the missing area of the first circular fisheye image, pixel values of an area of the second circular fisheye image that is located at a position corresponding to the missing area; an extracting means for extracting the first circular fisheye image including the pixel values set in the missing region from the image; a synthesis means for synthesizing pixel values of a second specific area of the second circular fisheye image, the second specific area being located at a position corresponding to the first specific area, with pixel values of the first specific area within a predetermined range from the missing area of the first circular fisheye image; Equipped with the first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens; The combining means controls the combining ratio of the pixel value of the second specific region to the pixel value of the first specific region so that the closer the pixel is to the missing region, the greater the combining ratio of the pixel value of the second specific region to the pixel value of the first specific region.

1. An image processing device comprising:

12. An image processing method executed by an image processing device, an acquiring step of acquiring a first circular fisheye image that is not recorded in a complete form, the first circular fisheye image having a first missing region where no pixel values exist within a first circular region corresponding to the complete form of the first circular fisheye image; a generating step of generating a first equirectangular projection image corresponding to the first circular region by performing a first equirectangular conversion process on the first circular fisheye image; Equipped with the generating step generates the first equirectangular projection image by setting a pixel value indicating a predetermined color in the first missing area of the first circular fisheye image before the first equirectangular conversion process, so that a first corresponding area of the first equirectangular projection image corresponding to the first missing area in the first circular area has a pixel value; the acquiring step acquires a second circular fisheye image having a parallax with respect to the first circular fisheye image; The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens. An image processing method comprising:

13. An image processing method executed by an image processing device, comprising: an acquiring step of acquiring a first circular fisheye image that is not recorded in a complete form, the first circular fisheye image having a first missing region where no pixel values exist within a first circular region corresponding to the complete form of the first circular fisheye image; a generating step of generating a first equirectangular projection image corresponding to the first circular region by performing a first equirectangular conversion process on the first circular fisheye image; Equipped with the generating step identifies coordinates of the first circular fisheye image corresponding to a pixel to be processed of the first equirectangular projection image in the first equirectangular conversion process, and if a pixel value does not exist at the identified coordinates, sets a pixel value indicating a predetermined color to the pixel to be processed of the first equirectangular projection image, thereby generating the first equirectangular projection image so that a first corresponding area of the first equirectangular projection image corresponding to the first missing area in the first circular area has a pixel value; the acquiring step acquires a second circular fisheye image having a parallax with respect to the first circular fisheye image; The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens. An image processing method comprising:

14. An image processing method executed by an image processing device, comprising: an acquiring step of acquiring a first circular fisheye image that is not recorded in a complete form, the first circular fisheye image having a first missing region where no pixel values exist within a first circular region corresponding to the complete form of the first circular fisheye image; a generating step of generating a first equirectangular projection image corresponding to the first circular region by performing a first equirectangular conversion process on the first circular fisheye image; Equipped with the generating step generates the first equirectangular projection image so that a first corresponding area of the first equirectangular projection image, which corresponds to the first missing area in the first circular area, has a pixel value; the acquiring step acquires a second circular fisheye image having a parallax with respect to the first circular fisheye image; the first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens; the second circular fisheye image includes a second missing region where no pixel values exist; the generating step includes setting pixel values of an area of the second circular fisheye image that is located at a position corresponding to the first missing area of the first circular fisheye image in the first corresponding area of the first equirectangular projection image; When a region of the second circular fisheye image that is located at a position corresponding to a specific region of the first missing region of the first circular fisheye image is included in the second missing region, the generating step sets a pixel value indicating a predetermined color for a region of the first corresponding region of the first equirectangular projection image that corresponds to the specific region. An image processing method comprising:

15. An image processing method executed by an image processing device, an acquisition step of acquiring a first equirectangular projection image generated by a first equirectangular conversion process based on a first circular fisheye image that is not recorded in a complete form, and a second equirectangular projection image generated by a second equirectangular conversion process based on a second circular fisheye image that has a parallax with respect to the first circular fisheye image, the first circular fisheye image has a missing region where no pixel values exist within a circular region corresponding to the complete shape of the first circular fisheye image; the first equirectangular image corresponds to the circular region; a predetermined pixel value is set in a corresponding area of the first equirectangular projection image that corresponds to the missing area in the circular area; an acquisition step; a designation step of designating a display position for a stereoscopic virtual reality based on the first equirectangular projection image and the second equirectangular projection image; an output step of outputting a first display image corresponding to the display position in the first equirectangular projection image and a second display image corresponding to the display position in the second equirectangular projection image; a correction step of correcting the first display image or the second display image before outputting it, when a first specific area overlapping the corresponding area is present in the first display image, so that the first specific area and a second specific area in the second display image that is located at a position corresponding to the first specific area contain the same image; Equipped with The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens. An image processing method comprising:

16. An image processing method executed by an image processing device, an acquiring step of acquiring an image including a first circular fisheye image that is not recorded in a complete form and a second circular fisheye image that has a parallax with respect to the first circular fisheye image, the first circular fisheye image having a missing area where no pixel values exist within a circular area corresponding to the complete form of the first circular fisheye image; a setting step of setting, in the missing area of the first circular fisheye image, pixel values of an area of the second circular fisheye image that is located at a position corresponding to the missing area; an extraction step of extracting the first circular fisheye image including the pixel values set in the missing region from the image; a combining step of combining pixel values of a second specific area of the second circular fisheye image, the second specific area being located at a position corresponding to the first specific area, with pixel values of the first specific area within a predetermined range from the missing area in the first circular fisheye image; Equipped with the first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, the first subject image being formed via a first fisheye lens, and the second subject image being formed via a second fisheye lens; The combining step controls the combining ratio of the pixel value of the second specific region to the pixel value of the first specific region so that the closer the pixel is to the missing region, the greater the combining ratio of the pixel value of the second specific region to the pixel value of the first specific region. An image processing method comprising:

17. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 3.

18. A program for causing a computer to function as each means of the image processing device described in claim 4.

19. A program for causing a computer to function as each means of the image processing device described in claim 5.

20. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 6 to 10.

21. A program for causing a computer to function as each of the means of the image processing apparatus according to claim 11.

Images