Image processing device, image processing method, and program

Abstract

To provide a technique for reducing the processing load of processing for developing a RAW image including an area of a circumferential fisheye image.SOLUTION: An image processing device includes acquisition means for acquiring a first RAW image including an area of a first circumferential fisheye image and developing means for developing the first RAW image. The developing means does not develop pixels outside the first circumferential fisheye image in the first RAW image.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] There is a known technology that uses two cameras to capture circular fisheye images with parallax and then reproduces the captured parallax images as virtual reality (VR) images with a three-dimensional effect. There is also a known device that has two optical systems in one lens mount and can capture images with parallax at the same time (Patent Document 1). There is also a known camera that can capture images in RAW format (RAW images). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-141052 Summary of the Invention [Problem to be solved by the invention]

[0004] To reproduce a RAW image containing a circular fisheye image as a VR image, both development processing and equirectangular conversion are required. This places a relatively high processing load on image processing devices such as personal computers (PCs), and no technology has been known to reduce this processing load.

[0005] The present invention has been made in view of the above circumstances, and has an object to provide a technique for reducing the processing load of developing a RAW image that includes a circular fisheye image area. [Means for solving the problem]

[0006] In order to solve the above problem, the present invention provides an image processing device comprising: a first acquisition means for acquiring a first RAW image including a region of a first circular fisheye image and formed via a first optical system; and a second RAW image including a region of a second circular fisheye image having parallax relative to the first circular fisheye image and formed via a second optical system different from the first optical system; and a second acquisition means for acquiring a first image obtained by developing the first RAW image, wherein the first image does not include a portion of the first circular fisheye image where pixels of a specific portion located at a position corresponding to a missing portion of the second circular fisheye image are developed, and the missing portion is a portion of an image formed via the second optical system that is not included in the region of the second circular fisheye image. [Effects of the Invention]

[0007] According to the present invention, it is possible to reduce the processing load of the process of developing a RAW image that includes a region of a circular fisheye image.

[0008] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiment of the present invention. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the overall configuration of a system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing the appearance of a camera 100. [Figure 3] FIG. 2 is a diagram showing an example of the internal configuration of the camera 100. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a VR180 lens 300 that can be attached to the camera 100. [Figure 5] Block diagram showing the configuration of PC500. [Figure 6A] 4 is a flowchart of a photographing process by the camera 100 according to the first embodiment. [Figure 6B] 4 is a flowchart of a photographing process by the camera 100 according to the first embodiment. [Figure 7]10 is an overall flowchart of development processing and equirectangular conversion processing by the PC 500 according to the first embodiment. [Figure 8] 10 is a flowchart showing details of the processes of S712 and S1412 (processing for creating an equirectangular projection image). [Figure 9] 1 is a diagram showing the structure of an image file recorded by the camera 100 (an image file processed by the PC 500). [Figure 10] 10(a) and 10(b) are diagrams showing information acquired from a VR180 lens 300, and 10(c) is a diagram showing information acquired from a camera 100. FIG. [Figure 11] 1A and 1B are diagrams showing examples of images captured and recorded by a camera 100 equipped with a VR180 lens 300. [Figure 12] 10 is a conceptual diagram of the processes of S712 and S1412 (processing for creating an equirectangular projection image). [Figure 13A] 10 is a flowchart of a video signal output process by the camera 100 according to the second embodiment. [Figure 13B] 10 is a flowchart of a video signal output process by the camera 100 according to the second embodiment. [Figure 14] 10 is an overall flowchart of development processing and equirectangular conversion processing by a PC 500 according to a second embodiment. [Figure 15] FIG. 10 is a diagram showing the overall configuration of a system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [First embodiment] ● Overall system configuration FIG. 1 is a diagram showing the overall configuration of a system according to a first embodiment. In FIG. 1, 100 is a digital camera (hereinafter referred to as "camera") capable of capturing RAW format images (RAW format still images or RAW format videos). 250 is a VR180 lens that can be attached to the camera 100. The VR180 lens 300 is a twin lens, and by capturing images with the camera 100 to which the VR180 lens 300 is attached, it is possible to obtain images with parallax. The VR180 lens 300 is a lens for capturing images for so-called VR180, which is a VR image format that allows twin-eye stereoscopic viewing and has a 180-degree field of view.

[0012] Reference numeral 500 denotes a personal computer (PC), which is an example of an image processing device that processes images captured by the camera 100. The method by which the PC 500 acquires images from the camera 100 is not particularly limited. For example, as shown in FIG. 1(a), the PC 500 may acquire images from the camera 100 via wireless communication. Alternatively, as shown in FIG. 1(b), the PC 500 may acquire images recorded as files via an external storage device (e.g., a memory card) of the camera 100.

[0013] ●Configuration of Camera 100 Figure 2 shows the external appearance of the camera 100. Figure 2(a) is a perspective view of the camera 100 seen from the front, and Figure 2(b) is a perspective view of the camera 100 seen from the back.

[0014] The camera 100 has, on its top surface, a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an extra-viewfinder display 107. The shutter button 101 is an operation unit for preparing for shooting or issuing instructions for shooting. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode switch 103 is an operation unit for switching between various modes. The main electronic dial 104 is a rotary operation unit for changing setting values ​​such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation unit for moving the selection frame (cursor), scrolling through images, etc. The video button 106 is an operation unit for issuing instructions to start and stop video shooting (recording). The extra-viewfinder display 107 displays various setting values ​​such as shutter speed and aperture.

[0015] The camera 100 also has a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, and a menu button 115 on the back side. The camera 100 also has an eyepiece unit 116, an eyepiece finder 117 (a peer-type finder), an eyepiece detection unit 118, and a touch bar 119. The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are an operation unit consisting of keys (four-way keys) that can be pressed up, down, left, and right. Operations can be performed according to the position of the directional keys 110 that are pressed. The SET button 111 is an operation unit that is mainly pressed to confirm a selection item. The AE lock button 112 is an operation unit that is pressed to fix the exposure state in a shooting standby state. The enlargement button 113 is an operation unit for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is also used to enlarge the playback image or increase the magnification in the playback mode. The playback button 114 is an operation unit for switching between the shooting mode and the playback mode. In the shooting mode, pressing the playback button 114 switches to the playback mode, and the latest image recorded on the recording medium 227 (described later) can be displayed on the display unit 108.

[0016] The menu button 115 is an operation unit that is pressed when a menu screen that allows various settings to be displayed on the display unit 108. The user can intuitively make various settings using the menu screen displayed on the display unit 108, the direction keys 110, and the SET button 111. The eyepiece unit 116 is a part for placing an eye on the eyepiece finder 117. The user can view an image displayed on an internal EVF 217 (Electronic View Finder) described below through the eyepiece unit 116. The eyepiece detection unit 118 is a sensor that detects whether the user has placed an eye on the eyepiece unit 116.

[0017] The touch bar 119 is a line-shaped touch operation unit (line touch sensor) capable of receiving touch operations. The touch bar 119 is positioned so that it can be touched by the thumb of the right hand when the grip unit 120 is held in the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. That is, the touch bar 119 can be operated while the user places their eye on the eyepiece unit 116, looks through the eyepiece viewfinder 117, and is in a position (shooting posture) in which the user is ready to press the shutter button 101 at any time. The touch bar 119 can receive tap operations (operations in which the user touches and then releases the touch bar without moving within a predetermined period of time), slide operations to the left or right (operations in which the user touches and then moves the touched position while keeping the touch), and the like. The touch bar 119 is an operation unit different from the touch panel 109 and does not have a display function. The touch bar 119 of this embodiment is a multi-function bar and functions, for example, as an M-Fn bar.

[0018] The camera 100 also has a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, and a communication terminal 124. The grip section 120 is a holding section shaped to be easily gripped with the user's right hand when holding the camera 100. The shutter button 101 and main electronic dial 104 are positioned so that they can be operated with the index finger of the right hand when the user holds the camera 100 by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand. Similarly, the sub electronic dial 105 and touch bar 119 are positioned so that they can be operated with the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section provided on the back side of the camera 100, in a position where it is easy to place the thumb of the right hand holding the grip section 120 when none of the operation sections are being operated. The thumb rest section 121 is made of a rubber member or the like to enhance holding strength (grip feeling). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 227 and the slot by closing the slot for storing the recording medium 227, which will be described later. The communication terminal 124 is a terminal for communicating with the detachable lens unit 200, which will be described later, of the camera 100.

[0019] Fig. 3 is a diagram showing an example of the internal configuration of the camera 100. Note that the same components as those in Fig. 2 are given the same reference numerals and descriptions thereof will be omitted where appropriate. A lens unit 200 can be attached to the camera 100.

[0020] First, a description will be given of the lens unit 200. The lens unit 200 is a type of interchangeable lens that can be attached to and detached from the camera 100. The lens unit 200 is a single lens, and is an example of a normal lens.

[0021] The lens unit 200 has an aperture 201, a lens 202, an aperture drive circuit 203, an AF drive circuit 204 (autofocus drive circuit), a lens system control circuit 205, and a communication terminal 206. The aperture 201 is configured so that its aperture diameter is adjustable. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit 50 (described later). The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203 and adjusts the focus by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, communication is performed via a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100 side.

[0022] Next, the camera 100 will be described. The camera 100 has a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50. The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an imaging element (image sensor) formed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 211 may also have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts an analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined calculations using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. This processing allows for TTL (through-the-lens) AF processing, AE (auto exposure) processing, EF (pre-flash) processing, etc. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data, and performs TTL AWB (auto white balance) processing based on the obtained calculation results.

[0023] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data to be displayed on the display unit 108 and EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 215 also serves as a memory for displaying images (video memory).

[0024] The D / A converter 216 converts image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 or the EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display in accordance with the analog signal from the D / A converter 216. The display unit 108 or the EVF 217 is, for example, an LCD or organic EL display. A digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216 and then sequentially transferred to and displayed on the display unit 108 or the EVF 217, thereby performing live view display.

[0025] The system control unit 50 is a control unit including at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 executes programs recorded in the nonvolatile memory 219 to realize each process in the flowcharts described below. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0026] The camera 100 also includes a system memory 218, a nonvolatile memory 219, a system timer 220, a communication unit 221, an attitude detection unit 222, and an eyepiece detection unit 118. The system memory 218 may be, for example, a RAM. Constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are loaded into the system memory 218. The nonvolatile memory 219 is an electrically erasable and recordable memory, for example, an EEPROM. The nonvolatile memory 219 stores constants and programs for the operation of the system control unit 50. The programs referred to here are programs for executing flowcharts, which will be described later. The system timer 220 is a timing unit that measures the time used for various controls and the time of a built-in clock. The communication unit 221 transmits and receives video and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 221 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 221 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive image data and other various information from external devices. The orientation detection unit 222 detects the orientation of the camera 100 relative to the direction of gravity. Based on the orientation detected by the orientation detection unit 222, it is possible to determine whether an image captured by the imaging unit 211 was captured with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate and record the image. The orientation detection unit 222 can use, for example, an acceleration sensor or a gyro sensor. The orientation detection unit 222 can also be used to detect movement of the camera 100 (panning, tilting, lifting, whether the camera is stationary, etc.).

[0027] The eyepiece detection unit 118 can detect the approach of an object to the eyepiece 116 of the eyepiece finder 117, which incorporates the EVF 217. The eyepiece detection unit 118 can be, for example, an infrared proximity sensor. When an object approaches, infrared light is emitted from a light-emitting unit of the eyepiece detection unit 118, reflected by the object, and received by a light-receiving unit of the infrared proximity sensor. The amount of received infrared light can be used to determine the distance from the eyepiece 116 to the object. In this way, the eyepiece detection unit 118 performs eyepiece detection, which detects the proximity of the object to the eyepiece 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (approach) and departure (away from) of an eye (object) to the eyepiece 116 of the eyepiece finder 117. When an object is detected approaching within a predetermined distance from the eyepiece 116 from a non-eyepiece state (not approaching state), it is detected that the eye has been placed in proximity. On the other hand, when an object whose proximity has been detected moves away from the eye-closed state (approaching state) by a distance greater than a predetermined distance, it is detected that the eye has moved away. The threshold for detecting eye-closedness and the threshold for detecting eye-away may be different, for example, by providing hysteresis. Furthermore, after detecting eye-closedness, the eye-closed state is maintained until eye-away is detected. After detecting eye-away, the non-eye-closed state is maintained until eye-closedness is detected. The system control unit 50 switches the display unit 108 and the EVF 217 between display (display state) and non-display (non-display state) depending on the state detected by the eye-closedness detection unit 118. Specifically, when at least in a shooting standby state and the display destination switching setting is automatic switching, the display unit 108 is set as the display destination and the display is turned on, and the EVF 217 is hidden, when the eye is not placed near the object. Furthermore, when the eye is placed near the object, the EVF 217 is set as the display destination and the display is turned on, and the display unit 108 is hidden. Note that eye proximity detection unit 118 is not limited to an infrared proximity sensor, and other sensors may be used as long as they can detect a state that can be considered as eye proximity. Camera 100 also has an outside-finder display unit 107, an outside-finder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium I / F 226, an operation unit 228, and a video signal output I / F 240. Outside-finder display unit 107 displays various settings of camera 100, such as shutter speed and aperture, via outside-finder display drive circuit 223.The power supply control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery charge. The power supply control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each unit, including the recording medium 227. The power supply unit 225 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The recording medium I / F 226 is an interface with the recording medium 227, such as a memory card or a hard disk. The recording medium 227 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 227 may be removable or built-in. The video signal output I / F 240 is an interface (HDMI terminal) for HDMI (High-Definition Multimedia Interface) output. The video signal from the camera 100 is output to a video signal receiving device 241 such as an external display or an external recorder via an HDMI cable.

[0028] The operation unit 228 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode selector switch 103, the touch panel 109, and other operation units 229. The other operation units 229 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction keys 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, and the touch bar 119.

[0029] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 is turned on when the shutter button 101 is pressed halfway (a shooting preparation command) and generates a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processes such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 231 is turned on when the shutter button 101 is pressed fully (a shooting command) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of shooting processes in response to the second shutter switch signal SW2, from reading out a signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227.

[0030] The mode selector switch 103 switches the operation mode of the system control unit 50 to one of still image capture mode, video capture mode, playback mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide capture settings for different capture scenes. The user can directly switch to one of the above-mentioned capture modes using the mode selector switch 103. Alternatively, the user can first switch to a list screen of capture modes using the mode selector switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video capture mode may also include multiple modes.

[0031] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating input coordinates on the touch panel 109 with display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user is directly operating the screen displayed on the display unit 108. The touch panel 109 can be any of a variety of systems, including a resistive film system, a capacitive system, a surface acoustic wave system, an infrared system, an electromagnetic induction system, an image recognition system, and an optical sensor system. Depending on the system, there are systems that detect a touch by contact with the touch panel 109, and systems that detect a touch by the approach of a finger or a pen to the touch panel 109, but either system may be used.

[0032] The system control unit 50 can detect the following operations or states on the touch panel 109.

[0033] A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down).

[0034] A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On).

[0035] The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move).

[0036] The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the end of the touch (hereinafter referred to as "touch-up").

[0037] A state in which nothing is touching the touch panel 109 (hereinafter referred to as Touch-Off).

[0038] When a touch down is detected, a touch on is also detected at the same time. After a touch down, a touch on is usually continued to be detected unless a touch up is detected. If a touch move is detected, a touch on is also detected at the same time. Even if a touch on is detected, a touch move is not detected unless the touch position moves. Once it is detected that all fingers or pens that were touching have touched up, a touch off occurs.

[0039] These operation states and the position coordinates of the finger or pen touching the touch panel 109 are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding touch-move, the movement direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced across the touch panel 109 as if flicking. If a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is then detected, it is determined that a flick has been performed (it can be determined that a flick occurred following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called a pinch in, and a touch operation in which the touch positions are moved farther apart is called a pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply pinch).

[0040] ●VR180 Lens 300 Configuration Fig. 4 is a diagram showing an example of the configuration of a VR180 lens 300 that can be attached to the camera 100. Fig. 4 shows the VR180 lens 300 attached to the camera 100. Note that in the camera 100 shown in Fig. 4, the same components as those described in Fig. 3 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0041] The VR180 lens 300 is a type of interchangeable lens that can be attached to and detached from the camera 100. The VR180 lens 300 is a twin lens that can capture images with parallax between left and right images. The VR180 lens 300 has two optical systems, each with a wide viewing angle of approximately 180 degrees, and can capture images of the range of the forward hemisphere. Specifically, the two optical systems of the VR180 lens 300 can each capture an object with a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle).

[0042] The VR180 lens 300 includes a right-eye optical system 301R having a plurality of lenses and a reflecting mirror, a left-eye optical system 301L having a plurality of lenses and a reflecting mirror, and a lens system control circuit 303. The right-eye optical system 301R corresponds to an example of a first optical system, and the left-eye optical system 301L corresponds to an example of a second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the lenses 302R and 302L located on the subject side face in the same direction, and their optical axes are approximately parallel.

[0043] The VR180 lens 300 of this embodiment is a lens for capturing images for so-called VR180, which is a VR image format that enables binocular stereoscopic viewing and has a 180-degree field of view. The VR180 lens 300 has fisheye lenses that enable the right-eye optical system 301R and the left-eye optical system 301L to each capture an approximately 180-degree range. Note that the VR180 lens 300 may be a lens that can capture a wide viewing angle range of approximately 160 degrees, which is narrower than the 180-degree range, as long as the right-eye optical system 301R and the left-eye optical system 301L can each acquire an image that allows binocular VR display as VR180. The VR180 lens 300 can form a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L, which has parallax from the right image, on one or two image pickup elements of the attached camera.

[0044] Furthermore, the VR180 lens 300 is attached to the camera 100 via the lens mount unit 304 and the camera mount unit 305 of the camera 100. By attaching the VR180 lens 300 to the camera 100, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the VR180 lens 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the VR180 lens 300.

[0045] In this embodiment, a right image formed via the right-eye optical system 301R and a left image formed via the left-eye optical system 301L, which has parallax from the right image, are formed side by side on the imaging unit 211 of the camera 100. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single imaging element. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. In this way, by using the VR180 lens 300, two images with parallax can be simultaneously acquired (as a set) from two locations (optical systems), the right-eye optical system 301R and the left-eye optical system 301L. Furthermore, by dividing the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image with a range of approximately 180 degrees, known as VR180.

[0046] Here, a VR image is an image that can be displayed in VR, as described below. VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera) and panoramic images with a wider image range (effective image range) than the display range that can be displayed at one time on a display unit. VR images are not limited to still images, but also include videos and live images (images acquired from a camera in almost real time). VR images have an image range (effective image range) of up to 360 degrees horizontally and vertically. VR images also include images with a wider angle of view than the angle of view that can be captured by a normal camera, or an image range that can be displayed at one time on a display unit, even if the field of view is less than 360 degrees horizontally or vertically. An image captured by the camera 100 using the VR180 lens 300 described above is a type of VR image. VR images can be displayed in VR by, for example, setting the display mode of a display device (a display device that can display VR images) to "VR view." By displaying a VR image with a 360-degree angle of view and changing the orientation of the display device left and right (horizontal rotation direction), the user can view seamless, omnidirectional images left and right.

[0047] Here, VR display (VR view) refers to a display method (display mode) that displays a VR image with a field of view that corresponds to the orientation of the display device, allowing for a change in display range. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by mapping a VR image onto a virtual sphere (deformation that corrects distortion). VR display also includes "two-eye VR display (two-eye VR view)," which displays a VR image for the left eye and a VR image for the right eye side by side by mapping them onto a virtual sphere. Stereoscopic viewing is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax. Regardless of the VR display, for example, when a user wears a display device such as an HMD (head-mounted display), the image displayed corresponds to the orientation of the user's face. For example, suppose a VR image is displayed with a field of view centered at 0 degrees left and right (a specific direction, e.g., north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is flipped from this state (for example, the display surface is changed from facing south to facing north), the display range of the same VR image will be changed to an image with a field of view centered at 180 degrees left and right (the opposite direction, for example south) and 90 degrees up and down. In other words, when the user is wearing the HMD and turns their face from north to south (i.e., turns around), the image displayed on the HMD will also change from a north image to a south image.

[0048] The VR image captured using the VR180 lens 300 of this embodiment is a VR180 image capturing a range of approximately 180 degrees forward, and does not contain any image in a range of approximately 180 degrees backward. If such a VR180 image is displayed in VR and the position of the display device is changed to the side where no image exists, a blank area will be displayed.

[0049] By displaying VR images in this way, the user visually feels as if they are inside the VR image (in the VR space). Note that the method of displaying VR images is not limited to changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operation via a touch panel or directional buttons. Furthermore, during VR display (display mode "VR view"), in addition to changing the display range due to posture changes, the display range may also be changed in response to touch-move on the touch panel, dragging with a mouse, pressing directional buttons, etc. Note that a smartphone attached to VR goggles (head-mounted adapter) is a type of HMD.

[0050] ●PC500 configuration Fig. 5 is a block diagram showing the configuration of PC 500. In Fig. 5, 501 is a control unit that controls the entire PC 500, such as a Central Processing Unit (CPU). 502 is a Read Only Memory (ROM) that stores programs and parameters that do not require modification. ROM 502 stores an information processing program as program code that can be read by control unit 501, and this information processing program code is executed by control unit 501. 503 is a Random Access Memory (RAM) that temporarily stores programs and data supplied from an external device, etc.

[0051] An external storage device 504 includes a hard disk or flash memory that is fixedly installed in the PC 500. Alternatively, the external storage device 504 may be an external storage device including an optical disk such as a floppy disk (FD) or compact disk (CD) that is detachable from the PC 500, a magnetic or optical card, an IC card, a memory card, etc. Image files acquired by the PC 500 from the camera 100 are stored in the external storage device 504.

[0052] Reference numeral 505 denotes an operation unit such as buttons or a touch panel that receives user operations and inputs data. Reference numeral 506 denotes a display unit that displays data held by the PC 500 or supplied data. Reference numeral 507 denotes a communication unit that communicates with external devices such as the camera 100. Reference numeral 508 denotes an external I / F that transmits and receives video signals and files to and from external devices. Reference numeral 509 denotes a system bus that connects the components of the PC 500 so that they can communicate with each other.

[0053] ●Features of images taken with the VR180 lens 300 In the case of a single lens optical system, an image rotated 180 degrees is formed on the imaging element. When generating a normal image from the 180-degree rotated image, the camera 100 performs a 180-degree rotation process to align the up-down direction of the image with the up-down direction of the subject. When capturing an image using the camera 100 equipped with the VR180 lens 300, the images of each optical system are formed on a single imaging element via the right-eye optical system 301R and the left-eye optical system 301L. At this time, the images of each optical system are rotated 180 degrees for each optical system. As in the case of a single lens optical system, the camera 100 rotates the entire image 180 degrees to align the up-down direction of the image with the up-down direction of the subject. While rotation during image formation occurs on an optical system-by-optical system basis, rotation during image generation occurs for the entire image. Therefore, the image corresponding to the left-eye optical system moves to the right of the overall image, and the image corresponding to the right-eye optical system moves to the left of the overall image. Therefore, to display the left and right images in the correct positional relationship, it is necessary to perform a process of swapping the left and right images.

[0054] ●Photography processing by camera 100 Next, with reference to Figures 6A and 6B, a description will be given of the photographing process by the camera 100. When the user turns on the power of the camera 100, the process of the flowcharts in Figures 6A and 6B starts.

[0055] In S601, the system control unit 50 determines whether the firmware of the camera 100 is compatible with VR180 lenses. If it is determined that the firmware is compatible with VR180 lenses, the processing proceeds to S602. If it is determined that the firmware is not compatible with VR180 lenses, the processing proceeds to S622. Because the optical system of a VR180 lens is different from that of a general lens, the camera 100 needs to be able to read and record VR180 lens metadata for post-processing. Therefore, the system control unit 50 determines whether the firmware is compatible with VR180 lenses.

[0056] In S602, the system control unit 50 determines whether a VR180 lens (for example, the VR180 lens 300 shown in FIG. 4) is attached to the camera 100. If it is determined that a VR180 lens is attached, the process proceeds to S603. If it is determined that a VR180 lens is not attached, the process proceeds to S622.

[0057] In S603, the system control unit 50 acquires the design values ​​of the VR180 lens. The lens design values ​​are used in the left-right swap processing and the equirectangular conversion processing, which will be described later.

[0058] In S604, the system control unit 50 acquires individual values ​​of the VR180 lens. In addition to lens design values, the VR180 lens also stores information on individual values ​​such as manufacturing errors. In the equirectangular conversion process, using the manufacturing error values ​​can produce better results than equirectangular conversion using only the design values.

[0059] In S605, the system control unit 50 acquires an image from the imaging unit 211.

[0060] In S606, the system control unit 50 displays the image acquired in S605 on the EVF 217 (live view display).

[0061] In S607, the system control unit 50 determines whether a recording start instruction has been issued by operating the shutter button 101. If it is determined that a recording start instruction has been issued, the process proceeds to step S608. If it is determined that a recording start instruction has not been issued, the process proceeds to step S605. Therefore, live view display on the EVF is repeated until a recording start instruction is issued by the user. The recording start instruction may be an instruction to shoot a still image or a video.

[0062] In S608, the system control unit 50 acquires an image from the imaging unit 211.

[0063] In S609, the system control unit 50 acquires shooting information such as the shutter speed at the time of shooting, and metadata such as orientation information of the camera 100 at the time of shooting. The orientation information is acquired from the orientation detection unit 222. The system control unit 50 also acquires metadata necessary for developing an image in RAW format (RAW image).

[0064] In S610, the system control unit 50 records the image acquired in S608 in a file as an image in RAW format (RAW image).

[0065] In S611, the system control unit 50 records the shooting information and metadata acquired in S609 in association with a file. For example, the system control unit 50 records the shooting information and metadata in the same file as the image.

[0066] In S612, the system control unit 50 records the information about the VR180 lens acquired in S603 and S604 in association with a file. For example, the system control unit 50 records the information about the VR180 lens in the same file as the image.

[0067] In S613, the system control unit 50 determines whether the user has issued an instruction to end recording. In the case of still image shooting, since a single image is being shot, it is assumed that the instruction to start recording and the instruction to end recording are issued simultaneously. In the case of video shooting, the user presses the shutter button 101 again to end the recording instruction. If it is determined that an instruction to end recording has been issued, the processing of this flowchart ends. If it is determined that an instruction to end recording has not been issued, the processing proceeds to S608. The system control unit 50 repeats the processing of S608 to S613, allowing video images to be recorded one after another into a file.

[0068] Next, a case where the processing step shifts from S601 or S602 to S622 will be described. Note that when the processing step shifts from S601 to S622, the firmware of the camera 100 does not support a VR180 lens. Therefore, the camera 100 cannot perform processing specific to a VR180 lens. Therefore, even if a VR180 lens is attached to the camera 100, the camera 100 performs the same processing as when a normal lens is attached.

[0069] In S622, the system control unit 50 determines whether a lens is attached to the camera 100. If it is determined that a lens is attached, the process proceeds to step S623. If it is determined that a lens is not attached, the process of this flowchart ends.

[0070] In S623, the system control unit 50 acquires general information about the lens (for example, the lens name).

[0071] In S624, the system controller 50 determines whether the general information about the lens was acquired in S623. If the general information about the lens was acquired, the process proceeds to S625. If the general information about the lens was not acquired, the process proceeds to S626. For example, if the camera 100 and the lens are from different manufacturers and the camera 100 cannot communicate with the lens, the camera 100 cannot acquire information from the lens.

[0072] In S625, the system control unit 50 stores the general information about the lens acquired in S623 in the system memory 218.

[0073] The processing in S626 to S632 is the same as the processing in S605 to S611, and therefore the description thereof will be omitted.

[0074] In S633, the system controller 50 determines whether general lens information is stored in the system memory 218 (see S625). If general lens information is stored, the process proceeds to S634. If general lens information is not stored, the process proceeds to S635.

[0075] In S634, the system control unit 50 records the general information about the lens stored in the system memory 218 in association with a file. For example, the system control unit 50 records the general information about the lens in the same file as the image.

[0076] The process of S635 is the same as the process of S613, and if an instruction to end recording is given, the process of this flowchart ends, and if an instruction to end recording is not given, the process proceeds to step S629. The system control unit 50 repeats the processes of S629 to S635, allowing moving images to be recorded one after another in a file.

[0077] The RAW images captured in the flowcharts of FIGS. 6A and 6B may be still images (RAW still images) or moving images (RAW moving images).

[0078] ●Image taken by camera 100 equipped with VR180 lens 300 Next, with reference to FIGS. 11(a) and 11(b), an example of an image captured and recorded by the camera 100 equipped with the VR180 lens 300 will be described.

[0079] The VR180 lens 300 is designed with specific values ​​in mind regarding the size of the image sensor and the angle of view for recording. For example, in the case of a VR180 lens 300 designed for a DCI 8K angle of view, the expected angle of view is 8192 x 4320.

[0080] On the other hand, depending on the shooting settings, the camera 100 can also record at an angle of view other than the angle of view assumed by the VR180 lens 300. For example, in the case of a UHD 8K setting, the angle of view is 7680 x 4320. In this case, even though the entire image from the VR180 lens 300 is input to the imaging element, part of it (for example, both ends) will not be recorded.

[0081] 11(a) is an example of an image recorded at an angle of view that uses the entire image sensor using a camera 100 with an image sensor of the size assumed by the VR180 lens 300. This is the case when the size of the image sensor assumed by the VR180 lens 300 is DCI 8K (8192 x 4320) and the recording size is also DCI 8K. In this case, two circular fisheye images lined up side by side are recorded in their entirety.

[0082] FIG. 11(b) shows an example of an image recorded using a camera 100 with an image sensor of a size intended for a VR180 lens, with an angle of view that uses only a portion of the image sensor. For example, this is the case when the image sensor size intended for the VR180 lens is DCI 8K (8192x4320) and the recording size is UHD 8K (7680x4320). In this case, two circular fisheye images aligned side by side are completely formed on the image sensor, but they are recorded with portions missing. For example, as shown in FIG. 11(b), of the two circular fisheye images aligned side by side, a portion of the left side of the left circular image and a portion of the right side of the right circular image are recorded with portions missing. In the following description, the missing areas (portions) of the circular fisheye images are also referred to as "missing areas" or "missing portions."

[0083] ●Development processing and equirectangular conversion processing using PC500 The development process and equirectangular conversion process by the PC 500 will be described with reference to Figures 7 and 8. The PC 500 performs development process and equirectangular conversion process on a RAW image included in an image file acquired from the camera 100. Through the development process and equirectangular conversion process, two equirectangular projection images on the left and right are generated from the two circular fisheye images on the left and right as described with reference to Figures 11(a) and 11(b). Note that in the example of Figure 7, the development process is included in the process of S712 (the process of creating an equirectangular projection image).

[0084] In S701, the control unit 501 reads a RAW image from an image file. This is a still image or a moving image from an image file shown in FIG. 9, which will be described later.

[0085] In S702, the control unit 501 reads the shooting information and metadata other than the VR180 lens from the image file. This is the shooting information and metadata other than the VR180 lens of the image file shown in Fig. 9, which will be described later. The metadata other than the VR180 lens includes metadata necessary for developing a RAW image (RAW development).

[0086] In S703, the control unit 501 determines whether the image file has VR180 lens metadata. This is the VR180 lens metadata of the image file shown in FIG. 9, which will be described later. If the image file has VR180 lens metadata, the processing step proceeds to S704. If the image file does not have VR180 lens metadata, the processing step proceeds to S706.

[0087] In S704, the control unit 501 reads the VR180 lens metadata from the image file.

[0088] In S705, the control unit 501 acquires the center coordinates and radii of the two left and right circular images in the recorded image from the design values ​​of the VR180 lens among the VR180 lens metadata read in S704. Furthermore, if the VR180 lens metadata has an individual value (manufacturing error) of the VR lens, the control unit 501 also reflects the individual value to acquire the center coordinates and radii of the two left and right circular images.

[0089] When the processing step proceeds from S703 to S706, the control unit 501 determines whether the image file has a lens name. For example, if the firmware of the camera 100 is not compatible with the VR180 lens, the camera 100 may not record VR180 lens metadata, but may record the lens name in a metadata area other than the VR180 lens. If the image file has a lens name, the processing step proceeds to S707. If the image file does not have a lens name, the processing step proceeds to S709.

[0090] In S707, the control unit 501 acquires the lens name from the image file.

[0091] In S708, the control unit 501 determines whether the lens name acquired in S707 indicates a known VR180 lens. If the lens name acquired in S707 indicates a known VR180 lens, the processing proceeds to S709. If the lens name acquired in S707 does not indicate a known VR180 lens, the processing proceeds to S710.

[0092] In S709, the control unit 501 determines the center coordinates and radii of the two circular images (left and right) in the recorded image based on the lens name acquired in S707. To achieve this processing, the program of the PC 500 is configured to store the lens name and the design values ​​of the center coordinates and radii of the two circular images (left and right) in the image captured with that lens. Alternatively, this information may be provided to the program from an external source.

[0093] When the process proceeds from S706 or S708 to S710, the control unit 501 determines whether the image read in S701 is a pair of left and right circular fisheye images. For example, the control unit 501 creates a monochrome binary image in which the pixel value of the read image is black if the pixel brightness value is equal to or less than a threshold value (a value close to black), and white if the pixel brightness value exceeds the threshold value. The control unit 501 can then determine whether the image read in S701 is a pair of left and right circular fisheye images based on whether the white areas in the monochrome binary image are two circles. If the read image is a pair of left and right circular fisheye images, the process proceeds to S711. If the read image is not a pair of left and right circular fisheye images, the process of this flowchart ends.

[0094] In S711, the control unit 501 detects the center coordinates and radii of the two circular fisheye images of the image read in S701. It is known that when two lines are drawn intersecting two points on the circumference of a circle and perpendicular lines are drawn from the midpoints of each line, the intersection of the two perpendicular lines is the center of the circle. Therefore, for example, using the monochrome binary image created in S710, the control unit 501 examines pixel values ​​in the horizontal direction at any vertical position in a circular area, and determines the X coordinate X1 of pixels that change from black to white and the X coordinate X2 of pixels converted from white to black. The control unit 501 also examines pixel values ​​in the vertical direction at any horizontal position, and determines the Y coordinate Y1 of pixels that change from black to white and the Y coordinate Y2 of pixels converted from white to black. In this case, the center coordinates of the circle are (X1 + X2) / 2 and (Y1 + Y2) / 2, respectively. Furthermore, if pixel values ​​are examined horizontally from the center coordinate of the circle and the X coordinate of the pixel that changes from white to black is X3, the radius of the circle is X3-((X1+X2) / 2).

[0095] In S712, the control unit 501 creates an equirectangular projection image from the two circular fisheye images using the center coordinates and radii of the two circular fisheye images obtained by the processing of S705, S709, or S711. Here, as described with reference to Fig. 11(b), even when images are taken with a camera 100 equipped with a VR180 lens, the two circular fisheye images may not be recorded in their entirety (may have missing parts).

[0096] 8 is a flowchart showing the details of the process of S712 (processing for creating equirectangular projection images). In this flowchart, the control unit 501 generates two equirectangular projection images by performing equirectangular conversion processing on two circular fisheye images with missing areas. Here, the circular fisheye images are included in the RAW image (Bayer data). Therefore, in this flowchart, the control unit 501 also performs development processing of the RAW images (processing for generating RGB data).

[0097] In S821, the control unit 501 determines whether processing of all pixels of the left eye destination image (equirectangular projection image corresponding to the left eye) has been completed. In this embodiment, in one processing loop of S821 to S827, in addition to the pixel to be processed in the left eye destination image, pixels located at the same coordinates as the pixel to be processed in the right eye destination image (equirectangular projection image corresponding to the right eye) are also processed. Therefore, when processing of all pixels of the left eye destination image has been completed, processing of all pixels of the right eye destination image has also been completed. Therefore, when processing of all pixels of the left eye destination image has been completed, the processing of this flowchart ends. If there are unprocessed pixels in the left eye destination image, the processing proceeds to step S822.

[0098] In S822, the control unit 501 determines (identifies) the coordinates (left eye source coordinates) of the left eye source image (left eye circular fisheye image) corresponding to the coordinates (left eye destination coordinates) of the pixel to be processed in the left eye destination image. Here, the left eye destination coordinates are in a coordinate system with the upper left corner of the left eye destination image as the origin. The process of determining the left eye source coordinates corresponding to the left eye destination coordinates can be performed using a general method for equirectangular transformation.

[0099] In S823, the control unit 501 determines (identifies) the coordinates (right-eye source coordinates) of the right-eye source image (right-eye circular fisheye image) corresponding to the coordinates (right-eye destination coordinates) of the pixel to be processed in the right-eye destination image. Here, the right-eye destination coordinates are in a coordinate system with the upper right corner of the right-eye destination image as the origin. The process of determining the right-eye source coordinates corresponding to the right-eye destination coordinates can be performed using a general method for equirectangular transformation.

[0100] In S824, the control unit 501 determines whether the left eye source coordinates are in the left eye source image and the right eye source coordinates are in the right eye source image. If the left eye source coordinates are in the left eye source image and the right eye source coordinates are in the right eye source image, the processing proceeds to S825. If at least one of the left eye source coordinates and the right eye source coordinates is not in the corresponding source image, the processing proceeds to S826.

[0101] In S825, the control unit 501 develops the pixel at the left eye source coordinates in the left eye source image and sets the pixel value obtained by the development to the pixel to be processed in the left eye destination image.Similarly, the control unit 501 develops the pixel at the right eye source coordinates in the right eye source image and sets the pixel value obtained by the development to the pixel to be processed in the right eye destination image.

[0102] An example of the development process in S825 will be described. The control unit 501 acquires pixel values ​​(Bayer state) of an N×N pixel (N is a natural number) region centered on the left eye source coordinates of the left eye source image. Then, the control unit 501 acquires pixel values ​​(RGB values) of the left eye source coordinates by performing development processing (interpolation processing) on ​​the pixels of the left eye source coordinates based on the pixel values ​​of the N×N pixel region and the metadata required for RAW development acquired in S702 of FIG. 7. Similarly, the control unit 501 acquires pixel values ​​(Bayer state) of an N×N pixel (N is a natural number) region centered on the right eye source coordinates of the right eye source image. Then, the control unit 501 acquires pixel values ​​(RGB values) of the right eye source coordinates by performing development processing (interpolation processing) on ​​the pixels of the right eye source coordinates based on the pixel values ​​of the N×N pixel region and the metadata required for RAW development acquired in S702 of FIG.

[0103] In addition, in development processing using pixel values ​​of an NxN pixel area, if the NxN pixel area includes an area outside the area of ​​the circular fisheye image, more accurate pixel values ​​can be calculated by not using pixels in the area outside the area of ​​the circular fisheye image in the development processing.

[0104] On the other hand, when the processing step proceeds from S824 to S826, the control unit 501 sets black to both the pixel to be processed of the left eye destination image and the pixel to be processed of the right eye destination image.

[0105] The top image in Figure 12 shows a partially clipped circular fisheye image used as the source image for equirectangular conversion, and the bottom image in Figure 12 shows the destination image (conversion result image) for equirectangular conversion.

[0106] As indicated by the arrow 1221, the coordinates of the left edge of the left-eye equirectangular projection image correspond to the coordinates of the left edge of the left-eye circular fisheye image, and the pixel at the coordinates of the left edge of the left-eye circular fisheye image is recorded. On the other hand, the starting point of the arrow 1222 in the right-eye equirectangular projection image is at a position corresponding to the starting point of the arrow 1221 in the left-eye equirectangular projection image, but no pixel is recorded at the position indicated by the arrow 1222 in the right-eye circular fisheye image (the coordinates of the left edge of the right-eye circular fisheye image). In this case, the color black is set for both the starting point of the arrow 1221 in the left-eye equirectangular projection image and the starting point of the arrow 1222 in the right-eye equirectangular projection image.

[0107] As indicated by the arrow 1223, the coordinates of the right edge of the right-eye equirectangular projection image correspond to the coordinates of the right edge of the right-eye circular fisheye image, and the pixel at the coordinates of the left edge of the right-eye circular fisheye image is recorded. On the other hand, the starting point of the arrow 1224 in the left-eye equirectangular projection image is at a position corresponding to the starting point of the arrow 1223 in the right-eye equirectangular projection image, but no pixel is recorded at the position indicated by the arrow 1224 in the left-eye circular fisheye image (the coordinates of the right edge of the left-eye circular fisheye image). In this case, the color black is set for both the starting point of the arrow 1223 in the right-eye equirectangular projection image and the starting point of the arrow 1224 in the left-eye equirectangular projection image.

[0108] In this way, for the left-eye equirectangular projection image, black is set not only in the area (starting point of arrow 1224) corresponding to the missing area (end point of arrow 1224) in the left-eye circumferential fisheye image, but also in the area (starting point of arrow 1221) corresponding to the area (end point of arrow 1221) at a position corresponding to the missing area (end point of arrow 1222) in the right-eye circumferential fisheye image. Similarly, for the right-eye equirectangular projection image, black is set not only in the area (starting point of arrow 1222) corresponding to the missing area (end point of arrow 1222) in the right-eye circumferential fisheye image, but also in the area (starting point of arrow 1223) corresponding to the area (end point of arrow 1223) at a position corresponding to the missing area (end point of arrow 1224) in the left-eye circumferential fisheye image. This makes it possible to prevent black pixel areas from appearing in only one of the left and right equirectangular projection images. This improves the image quality when a user views the left and right equirectangular projection images as a single stereoscopic image using a head-mounted display (HMD).

[0109] 12, one RAW image includes a region of a RAW image (first RAW image) that includes a region of a left-eye circular fisheye image (first circular fisheye image) and a region of a RAW image (second RAW image) that includes a region of a right-eye circular fisheye image (second circular fisheye image). However, the control unit 501 may acquire the RAW image (first RAW image) that includes a region of a left-eye circular fisheye image (first circular fisheye image) and the RAW image region (second RAW image) that includes a region of a right-eye circular fisheye image (second circular fisheye image) as separate RAW images.

[0110] In S827, the control unit 501 selects the next pixel as the pixel to be processed for each of the left eye destination image and the right eye destination image. For example, when processing is performed from the upper left pixel to the lower right pixel of the left eye destination image, the next pixel is the pixel immediately to the right in the same row. However, if the X coordinate of the pixel immediately to the right reaches the width of the left eye destination image, the leftmost pixel of the left eye destination image in the row below is selected as the next pixel. The same applies to the right eye destination image.

[0111] Through the above processing, two equirectangular projection images are generated from two circular fisheye images with missing regions (missing portions).

[0112] According to the process for creating an equirectangular projection image shown in Fig. 8, the control unit 501 develops a RAW image including two circular fisheye images, but does not develop pixels outside the area of ​​the circular fisheye images. In other words, the control unit 501 does not develop pixels in areas that are not referenced as source coordinates for the equirectangular conversion process (black areas in the upper diagram of Fig. 12). This makes it possible to reduce the processing load of development.

[0113] Furthermore, the control unit 501 does not develop pixels in a specific portion (end point of arrow 1221) of the left-eye circular fisheye image that is located at a position corresponding to the missing portion of the right-eye circular fisheye image (end point of arrow 1222 in FIG. 12). This is because black is set in step S826 in the area (start point of arrow 1221) of the left-eye equirectangular projection image that corresponds to the specific portion of the left-eye circular fisheye image (end point of arrow 1221), so developed pixel values ​​are unnecessary for the specific portion. Similarly, the control unit 501 does not develop pixels in a specific portion (end point of arrow 1223) of the right-eye circular fisheye image that is located at a position corresponding to the missing portion of the left-eye circular fisheye image (end point of arrow 1224 in FIG. 12). This makes it possible to further reduce the processing load of development.

[0114] According to the process for creating an equirectangular projection image shown in FIG. 8, the control unit 501 performs development processing and equirectangular conversion processing for each pixel to be processed in the processing loop from S821 to S827. In other words, the control unit 501 executes development processing and equirectangular conversion processing in parallel. However, the control unit 501 may also perform development processing on the entire RAW image before equirectangular conversion processing. In this case, the control unit 501 may set a predetermined pixel value (e.g., a pixel value indicating black) to pixels in areas not targeted for development processing. The areas not targeted for development processing here refer to the black area in the upper diagram of FIG. 12, a specific portion of the right-eye circular fisheye image located at a position corresponding to a missing portion of the left-eye circular fisheye image, and a specific portion of the left-eye circular fisheye image located at a position corresponding to a missing portion of the right-eye circular fisheye image. Whether each pixel of the RAW image is within a circular fisheye area can be determined based on whether the pixel values ​​of all four pixels in the corresponding Bayer array are equal to or below a certain level.

[0115] ●Image file format Next, the structure of an image file recorded by the camera 100 (an image file processed by the PC 500) will be described with reference to FIG. 9. In FIG. 9, 901 is a file header. The file header records information such as the type of image. 902 is shooting information at the time the image recorded in the file was captured. The shooting information records information such as the shutter speed and aperture. 903 is VR180 lens metadata (metadata related to the VR180 lens). As the VR180 lens metadata, information such as the lens name, the radius of the circular fisheye lens, and information on manufacturing errors are recorded. 904 is metadata for lenses other than the VR180 lens. In the case of videos, information that changes with each frame is recorded in the metadata for lenses other than the VR180 lens, and in the case of RAW, data required for development is recorded. A camera equipped with firmware that is not compatible with VR180 lenses records the lens name in the metadata for lenses other than the VR180 lens. 905 is still image or video data. In the case of videos, audio data is also recorded in addition to images.

[0116] 9, the image file records shooting information 902, VR180 lens metadata 903, and metadata other than the VR180 lens 904. However, a configuration may be adopted in which the shooting information 902, VR180 lens metadata 903, and metadata other than the VR180 lens 904 are recorded in a file different from the image file, and the recorded information is associated with the image file.

[0117] Information acquired from the VR180 lens 300 and the camera 100 10(a), an example of information that the camera 100 acquires from the VR180 lens 300 will be described. The following information is acquired from the VR180 lens 300. 1. Lens design values 2. Lens individual value (manufacturing error value) 3. Lens flag information 4. Lens focal length information 5. Lens thermometer information

[0118] The lens design values ​​are design value data common to both eyes of VR180 for aberration correction. Details of the lens design values ​​will be described later with reference to FIG. 10(b).

[0119] The lens individual value (manufacturing error value) is data that indicates the manufacturing error of the VR180 lens 300. During the manufacturing process of the VR180 lens 300, errors occur on both the left and right sides. If equirectangular conversion processing is performed by the PC500 when errors exist, the quality of the 3D display will deteriorate. To mitigate this problem, measurement results of errors detected during the lens manufacturing process are recorded inside the lens. Details of the lens individual value will be described later with reference to FIG. 10(b).

[0120] When performing image processing, the PC 500 needs to determine whether or not to perform conversion processing on image data captured with the VR180 lens 300. The lens flag is recorded to enable this determination. The lens flag is a flag that indicates that the image data is image data captured with a VR180 lens.

[0121] The lens focal length is the distance from the "principal point" at the center of the lens to the image formed on the imaging element. When image data captured with the VR180 lens 300 is converted by the PC 500, a highly accurate (decimal point) value for the lens focal length is required to maintain the quality of the 3D display.

[0122] The lens thermometer information indicates the temperature of the VR180 lens 300. Because lens temperature information is recorded during shooting, the PC 500 can grasp the ambient temperature.

[0123] 10(b), the lens design values ​​and lens individual values ​​(manufacturing error values) will be described. This information is used for left-right swap processing, equirectangular conversion processing, etc. The lens design values ​​include, for example, the following information: 1. Image circle position 2. Image circle diameter 3. Angle of View 4. Distortion correction coefficient

[0124] The image circle position is the design value of the center position of the image circle formed on the imaging element. For example, the image circle position is defined as horizontal and vertical coordinates with the mount as the origin.

[0125] The image circle diameter is the diameter of the image circle formed on the image sensor.

[0126] The field angle is the range of the field angle of an image formed within an image circle.

[0127] The distortion correction coefficient is the ratio of the design image height to the ideal image height of the lens. A value may be set for each image height, and the values ​​may be interpolated between them or approximated by a polynomial.

[0128] Furthermore, when an app on the PC 500 is connected to the camera 100 and displayed, a magic window display is performed on the screen of the PC 500. In this case, the "image circle position," "image circle diameter," and "angle of view" are used to maintain the posture and display quality. The PC 500 edits this information to match the app and uses it to properly display the image on the screen. For example, the PC 500 multiplies the "image circle position" and "image circle diameter" by coefficients to match the magic window display.

[0129] The lens individual value (manufacturing error value) includes, for example, the following information: 5. Image circle misalignment 6. Optical axis tilt 7. Image magnification deviation

[0130] Image circle positional deviation is the deviation of the center position of the image circle formed on the image sensor from the design value. For example, the image circle positional deviation is defined by horizontal and vertical coordinates with the design position as the origin.

[0131] The optical axis tilt is the deviation of the optical axis on the subject side. For example, the horizontal and vertical tilt deviations are expressed as angles.

[0132] The image magnification deviation is the deviation of the image size from the design value. For example, the image magnification deviation is expressed as a ratio to the design value.

[0133] These individual values ​​vary from lens to lens because they are generated by manufacturing errors such as lens decentering and tilt. Therefore, the lens individual values ​​are measured and recorded for each of the left and right optical systems.

[0134] 10(c), the metadata acquired from the camera 100 will be described. This metadata is used to maintain 3D quality based on the display area and orientation information when performing conversion processing in an application on the PC 500. The metadata includes, for example, the following information: 1. Camera recording area information 2.In-camera accelerometer information 3.Aperture value right eye correction information

[0135] Camera recording area information refers to the effective image area. The effective image area that can be displayed varies depending on the camera's imaging element and recording mode. This information is used to display images more accurately when conversion processing is performed by the PC500 app.

[0136] The in-camera accelerometer information is the roll and pitch of the attitude information measured using the in-camera accelerometer (level). This information is used for electronic image stabilization and horizontal correction in the PC500 app.

[0137] The aperture value for the right eye is the exposure setting for the left eye. This information is used to maintain 3D quality and ensure a natural playback display when conversion processing is performed by the PC500 application.

[0138] As described above, according to the first embodiment, the PC 500 acquires a RAW image (for example, the upper diagram in FIG. 12) that includes an area of ​​the circular fisheye image, and develops the acquired RAW image. In developing the RAW image, the PC 500 does not develop pixels in the RAW image that are outside the area of ​​the circular fisheye image (i.e., pixels in an area that is not referenced as the source coordinates for the equirectangular conversion process (for example, pixels in the black area in the upper diagram in FIG. 12)). This makes it possible to reduce the processing load of development.

[0139] The PC 500 may also omit development of pixels in a specific portion of the first circular fisheye image (for example, the region of the left-eye circular fisheye image corresponding to the end point of the arrow 1221 in FIG. 12) that is located at a position corresponding to the missing portion of the second circular fisheye image (for example, the missing portion of the right-eye circular fisheye image corresponding to the end point of the arrow 1222 in FIG. 12). This makes it possible to further reduce the processing load of development.

[0140] [Second embodiment] In the first embodiment, a configuration was described in which the PC 500 acquires a RAW image from an image file containing the RAW image. In the second embodiment, a configuration will be described in which the PC 500 acquires a RAW image by receiving a video signal containing the RAW image from the camera 100 (external device) in accordance with a predetermined video signal transmission standard. In this embodiment, the basic configurations of the camera 100 and the PC 500 are the same as in the first embodiment. Below, differences from the first embodiment will mainly be described.

[0141] In the following description, the HDMI standard is used as the video signal transmission standard, but the video signal transmission standard in this embodiment is not limited to the HDMI standard and may be, for example, the DisplayPort standard.

[0142] ● Overall system configuration Fig. 15 is a diagram showing the overall configuration of a system according to the second embodiment. In Fig. 15, 190 is an HDMI cable that transmits a video signal output from a camera 100. The video signal from the camera 100 is input to a PC 500 via an external I / F 508. The PC 500 can display the video signal on its display in real time. The PC 500 also converts the video signal into a format that can be streamed, and can transmit the video directly from the PC 500 to an HMD 191 (head-mounted display) or distribute the video to multiple users 193 via a cloud 192.

[0143] ●Video signal output processing by camera 100 13A and 13B, a video signal output process by camera 100 will be described. When the user turns on the power of camera 100, the process of the flowcharts in Figures 13A and 13B starts.

[0144] The processing in S1301 to S1306 is the same as the processing in S601 to S606 in FIG. 6A, and therefore a description thereof will be omitted.

[0145] In S1307, the system control unit 50 determines whether or not an HDMI cable is connected to the HDMI terminal (video signal output I / F 240). If it is determined that an HDMI cable is connected, the process proceeds to S1308, and if it is determined that an HDMI cable is not connected, the process proceeds to S1305.

[0146] The processing in steps S1308 to S1309 is the same as the processing in steps S1308 to S1309 in FIG. 6A, and therefore a description thereof will be omitted.

[0147] In S1310, the system control unit 50 stores the data acquired in S1303, S1304, S1308, and S1309 in a predetermined format in a buffer for HDMI signal output.

[0148] In S1311, the system control unit 50 outputs the data stored in S1310 as an HDMI signal.

[0149] In S1312, the system control unit 50 determines whether the HDMI cable connected to the HDMI terminal (video signal output I / F 240) has been unplugged. If it is determined that the HDMI cable has been unplugged, the processing of this flowchart ends. If the HDMI cable has not been unplugged, the processing proceeds to S1308. The system control unit 50 repeats the processing of S1308 to S1312, allowing the camera 100 to continue outputting an HDMI signal.

[0150] Next, a case where the processing step shifts from S1301 or S1302 to S1322 will be described. Note that when the processing step shifts from S1301 to S1322, the firmware of the camera 100 does not support a VR180 lens. Therefore, the camera 100 cannot perform processing specific to a VR180 lens. Therefore, even if a VR180 lens is attached to the camera 100, the camera 100 performs the same processing as when a normal lens is attached.

[0151] The processing of S1322 to S1327 is the same as the processing of S622 to S627 in FIG. 6B, and therefore a description thereof will be omitted.

[0152] In S1328, the system control unit 50 determines whether or not an HDMI cable is connected to the HDMI terminal (video signal output I / F 240). If it is determined that an HDMI cable is connected, the process proceeds to S1329, and if it is determined that an HDMI cable is not connected, the process proceeds to S1326.

[0153] The processing in S1329 to S1330 is the same as the processing in S629 to S630 in FIG. 6B, and therefore a description thereof will be omitted.

[0154] In S1331, the system controller 50 determines whether general lens information is stored in the system memory 218 (see S1325). If general lens information is stored, the process proceeds to S1332. If general lens information is not stored, the process proceeds to S1333.

[0155] In S1332, the system control unit 50 acquires general information about the lens stored in the system memory 218.

[0156] In S1333, the system control unit 50 stores the data acquired in S1329, S1330, and S1332 in a predetermined format in a buffer for HDMI signal output.

[0157] In S1334, the system controller 50 outputs the data stored in S1333 as an HDMI signal.

[0158] In S1335, the system control unit 50 determines whether the HDMI cable connected to the HDMI terminal (video signal output I / F 240) has been unplugged. If it is determined that the HDMI cable has been unplugged, the processing of this flowchart ends. If the HDMI cable has not been unplugged, the processing proceeds to S1329. The system control unit 50 repeats the processing of S1329 to S1335, allowing the camera 100 to continue outputting an HDMI signal.

[0159] The RAW images captured in the flowcharts of FIGS. 13A and 13B may be still images (RAW still images) or moving images (RAW moving images).

[0160] ●Development processing and equirectangular conversion processing using PC500 The development process and equirectangular conversion process by the PC 500 will be described with reference to Fig. 14. The PC 500 performs development process and equirectangular conversion process on a RAW image included in a video signal (HDMI signal) received from the camera 100. Through the development process and equirectangular conversion process, two equirectangular projection images on the left and right are generated from the two circular fisheye images on the left and right as described with reference to Figs. 11(a) and 11(b). Note that in the example of Fig. 14, the development process is included in the process of S1412 (the process of creating an equirectangular projection image).

[0161] In S1401, the control unit 501 determines whether or not an HDMI signal has been received from the external device (camera 100). If an HDMI signal has been received, the process proceeds to processing step S1402. If an HDMI signal has not been received, the control unit 501 repeats the process of S1401.

[0162] In S1402, the control unit 501 reads the RAW image, shooting information, and metadata other than that of the VR180 lens from the first frame of the received HDMI signal.

[0163] In S1403, the control unit 501 determines whether VR180 lens metadata is superimposed on the HDMI signal. If VR180 lens metadata is superimposed on the HDMI signal, the processing proceeds to S1404. If VR180 lens metadata is not superimposed on the HDMI signal, the processing proceeds to S1406.

[0164] In S1404, the control unit 501 reads the VR180 lens metadata from the HDMI signal.

[0165] The process of S1405 is the same as the process of S705 in FIG. 7, and therefore a description thereof will be omitted.

[0166] When the processing step proceeds from S1403 to S1406, the control unit 501 determines whether a lens name is superimposed on the HDMI signal. For example, if the firmware of the camera 100 does not support VR180 lenses, the camera 100 may not superimpose VR180 lens metadata, but may superimpose the lens name as metadata for a lens other than the VR180 lens. If the lens name is superimposed on the HDMI signal, the processing step proceeds to S1407. If the lens name is not superimposed on the HDMI signal, the processing step proceeds to S1409.

[0167] In S1407, the control unit 501 acquires the lens name from the HDMI signal.

[0168] In S1408, the control unit 501 determines whether the lens name acquired in S1407 indicates a known VR180 lens. If the lens name acquired in S1407 indicates a known VR180 lens, the processing proceeds to S1409. If the lens name acquired in S1407 does not indicate a known VR180 lens, the processing proceeds to S1410.

[0169] The process of S1409 is the same as the process of S709 in FIG. 7, and therefore a description thereof will be omitted.

[0170] When the processing step proceeds from S1406 or S1408 to S1410, the control unit 501 determines whether the image read in S1402 is a pair of left and right circular fisheye images. For example, the control unit 501 creates a monochrome binary image in which the pixel value of the read image is black if the luminance value of the pixel is equal to or less than a threshold (a value close to black), and white if the luminance value of the pixel is greater than the threshold. The control unit 501 can then determine whether the image read in S1402 is a pair of left and right circular fisheye images based on whether the white areas in the monochrome binary image are two circles. If the read image is a pair of left and right circular fisheye images, the processing step proceeds to S1411. If the read image is not a pair of left and right circular fisheye images, the processing of this flowchart ends.

[0171] The process of S1411 is the same as the process of S711 in FIG. 7, and therefore a description thereof will be omitted.

[0172] In S1412, the control unit 501 creates an equirectangular projection image from the two circular fisheye images using the center coordinates and radii of the two circular fisheye images obtained by the processing of S1405, S1409, or S1411. Details of the processing of S1412 are the same as the processing of S712 in Fig. 7 (i.e., the processing in the flowchart of Fig. 8), and therefore will not be described again.

[0173] For subsequent frames of the HDMI signal, the control unit 501 repeatedly executes the process of the flowchart in FIG. 8 until it detects the end of the HDMI signal.

[0174] Through the above processing, two equirectangular projection images are generated from two circular fisheye images with missing regions (missing portions).

[0175] As described above, according to the second embodiment, the PC 500 receives a video signal including a RAW image that includes a circular fisheye image area from the camera 100 in accordance with the HDMI standard, and acquires the RAW image from the received video signal. Then, similar to the first embodiment, the PC 500 develops the RAW image by omitting development of some pixels. Therefore, even when the PC 500 acquires a RAW image from a video signal (instead of an image file), it is possible to reduce the processing load of development, similar to the first embodiment.

[0176] In the above-described embodiments, a configuration has been described in which the development process of a RAW image, the left-right swapping of a circular fisheye image, and the equirectangular conversion process are performed all at once. However, a configuration may be adopted in which the development process and the equirectangular conversion process are performed after the left-right swapping of a circular fisheye image is performed in the state of a Bayer image (RAW image). Alternatively, a configuration may be adopted in which the development process and the equirectangular conversion are performed on necessary pixels, and then the left-right swapping is performed in the state of an RGB image.

[0177] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0178] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0179] 500...PC, 501...controller, 502...ROM, 503...RAM, 504...external storage device, 505...operation unit, 506...display unit, 507...communication unit, 508...external I / F, 509...system bus

Claims

1. a first acquisition means for acquiring a first RAW image including a region of a first circular fisheye image and formed via a first optical system, and a second RAW image including a region of a second circular fisheye image having parallax with respect to the first circular fisheye image and formed via a second optical system different from the first optical system; a second acquisition means for acquiring a first image obtained by developing the first RAW image; Equipped with the first image does not include a portion of the first circular fisheye image in which pixels of a specific portion located at a position corresponding to a missing portion of the second circular fisheye image are developed; The missing portion is a portion of the image formed through the second optical system that is not included in the area of ​​the second circular fisheye image.

1. An image processing device comprising:

2. The first acquisition means acquires the first RAW image and the second RAW image by acquiring one RAW image including an area of ​​the first RAW image and an area of ​​the second RAW image.

2. The image processing device according to claim 1, wherein:

3. The first image is an image in which predetermined pixel values ​​are set for pixels in the first RAW image that are not to be developed.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. a generating unit for generating an equirectangular projection image by performing equirectangular conversion processing based on the first circular fisheye image; In the equirectangular conversion process, the generating means identifies a pixel of the first circular fisheye image that corresponds to a pixel to be processed of the equirectangular projection image, and sets a pixel value obtained by developing the identified pixel to the pixel to be processed of the equirectangular projection image.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

5. The first acquisition means acquires the first RAW image from an image file containing the first RAW image.

5. The image processing device according to claim 1, wherein the image processing device is a computer.

6. further comprising a receiving means for receiving a video signal including the first RAW image from an external device in accordance with a predetermined video signal transmission standard; The first acquisition means acquires the first RAW image from the video signal.

5. The image processing device according to claim 1, wherein the image processing device is a computer.

7. The circular fisheye image is a circular area included in the RAW image.

7. The image processing device according to claim 1, wherein the image processing device is a computer.

8. The first image is an image in which pixels outside the area of ​​the first circular fisheye image in the first RAW image are not developed.

8. The image processing device according to claim 1, wherein the image processing device is a computer.

9. An image processing method executed by an image processing device, a first acquisition step of acquiring a first RAW image including a region of a first circular fisheye image and formed via a first optical system, and a second RAW image including a region of a second circular fisheye image having parallax with respect to the first circular fisheye image and formed via a second optical system different from the first optical system; a second acquisition step of acquiring a first image obtained by developing the first RAW image; Equipped with the first image does not include a portion of the first circular fisheye image in which pixels of a specific portion located at a position corresponding to a missing portion of the second circular fisheye image are developed; The missing portion is a portion of the image formed through the second optical system that is not included in the area of ​​the second circular fisheye image. An image processing method comprising:

10. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 8.

Images