Image processing device, image processing method and storage medium

Abstract

The present invention relates to an image processing device, an image processing method and a storage medium. The image processing device includes: an acquisition unit, which acquires a first circular fisheye image accompanied by a first missing area with no pixel value; a generation unit, which generates a first equidistant cylindrical projection image by performing a first equidistant cylindrical transformation process based on the first circular fisheye image. The generation unit generates the first equidistant cylindrical projection image so that in the first equidistant cylindrical projection image, a first corresponding area corresponding to the first missing area has a pixel value.

Description

Technical Field

[0001] The invention relates to an image processing device, an image processing method and a storage medium. Background Art

[0002] A technique is known that uses two cameras to capture images with parallax and reproduces the captured parallax images as three-dimensional virtual reality (VR) images. Also known is a device that has two optical systems on a single lens mount and can capture images with parallax at once (Japanese Patent Laid-Open No. 2013-141052).

[0003] The technology of Japanese Patent Laid-Open No. 2013-141052 solves the problem that left and right images through left and right lenses are captured in different directions on the image sensor. However, the technology does not consider processing of a circular fisheye image with a missing area where no pixel value exists. Summary of the invention

[0004] In view of the above circumstances having been achieved, the present invention provides a technique for appropriately processing a circular fisheye image accompanied by missing regions where no pixel values ​​exist.

[0005] According to a first aspect of the present invention, an image processing device is provided, comprising: an obtaining unit configured to obtain a first circular fisheye image accompanied by a first missing area with no pixel values ​​present; and a generating unit configured to generate a first equidistant cylindrical projection image by performing a first equidistant cylindrical transformation processing based on the first circular fisheye image, wherein the generating unit generates the first equidistant cylindrical projection image so that in the first equidistant cylindrical projection image, a first corresponding area corresponding to the first missing area has a pixel value.

[0006] According to a second aspect of the present invention, there is provided an image processing device, comprising: an obtaining unit, which is configured to obtain a first equidistant cylindrical projection image and a second equidistant cylindrical projection image, wherein the first equidistant cylindrical projection image is generated by a first equidistant cylindrical transformation process based on a first circular fisheye image accompanied by a missing area with no pixel values, and the second equidistant cylindrical projection image is generated by a second equidistant cylindrical transformation process based on a second circular fisheye image having a parallax relative to the first circular fisheye image, and a predetermined pixel value is set in a corresponding area of ​​the first equidistant cylindrical projection image corresponding to the missing area; and a specifying unit, which is configured to specify a predetermined pixel value based on the first equidistant cylindrical projection image. and a display position of a three-dimensional virtual reality of the second equidistant cylindrical projection image; an output unit, which is configured to output a first display image corresponding to the display position in the first equidistant cylindrical projection image and a second display image corresponding to the display position in the second equidistant cylindrical projection image; and a correction unit, which is configured to correct the first display image or the second display image before outputting the first display image or the second display image when there is a first specific area overlapping with the corresponding area in the first display image, so that the first specific area and the second specific area located at the position of the second display image corresponding to the first specific area include the same image.

[0007] According to a third aspect of the present invention, there is provided an image processing device, comprising: an obtaining unit, configured to obtain an image including a first circular fisheye image and a second circular fisheye image, wherein the first circular fisheye image is accompanied by a missing area where no pixel values ​​exist, and the second circular fisheye image has parallax relative to the first circular fisheye image; a setting unit, configured to set pixel values ​​of an area of ​​the second circular fisheye image located at a position corresponding to the missing area in the missing area of ​​the first circular fisheye image; and an extraction unit, configured to extract the first circular fisheye image including the pixel values ​​set in the missing area from the image.

[0008] According to a fourth aspect of the present invention, there is provided an image processing method, which is performed by an image processing device, and the image processing method comprises: obtaining a first circular fisheye image accompanied by a first missing area in which no pixel values ​​exist; and generating a first equidistant cylindrical projection image by performing a first equidistant cylindrical transformation processing based on the first circular fisheye image, wherein the first equidistant cylindrical projection image is generated so that in the first equidistant cylindrical projection image, a first corresponding area corresponding to the first missing area has a pixel value.

[0009] According to a fifth aspect of the present invention, there is provided an image processing method, which is performed by an image processing device, the image processing method comprising: obtaining a first equidistant cylindrical projection image and a second equidistant cylindrical projection image, the first equidistant cylindrical projection image being generated by a first equidistant cylindrical transformation process based on a first circular fisheye image accompanied by a missing area with no pixel values, the second equidistant cylindrical projection image being generated by a second equidistant cylindrical transformation process based on a second circular fisheye image having a parallax relative to the first circular fisheye image, and a predetermined pixel value being set in a corresponding area of ​​the first equidistant cylindrical projection image corresponding to the missing area; specifying a display position of a three-dimensional virtual reality based on the first equidistant cylindrical projection image and the second equidistant cylindrical projection image; outputting a first display image corresponding to the display position in the first equidistant cylindrical projection image and a second display image corresponding to the display position in the second equidistant cylindrical projection image; and when there is a first specific area in the first display image that overlaps with the corresponding area, correcting the first display image or the second display image before outputting the first display image or the second display image so that the first specific area and the second specific area located at the position of the second display image corresponding to the first specific area include the same image.

[0010] According to a sixth aspect of the present invention, there is provided an image processing method, which is performed by an image processing device, and the image processing method comprises: obtaining an image including a first circular fisheye image and a second circular fisheye image, wherein the first circular fisheye image is accompanied by a missing area where no pixel values ​​exist, and the second circular fisheye image has parallax relative to the first circular fisheye image; setting pixel values ​​of an area of ​​the second circular fisheye image located at a position corresponding to the missing area in the missing area of ​​the first circular fisheye image; and extracting the first circular fisheye image including the pixel values ​​set in the missing area from the image.

[0011] According to the seventh aspect of the present invention, a non-temporary computer-readable storage medium is provided, which stores a program for causing a computer to execute an image processing method, the image processing method comprising: obtaining a first circular fisheye image accompanied by a first missing area in which no pixel values ​​exist; and generating a first equidistant cylindrical projection image by performing a first equidistant cylindrical transformation process based on the first circular fisheye image, wherein the first equidistant cylindrical projection image is generated so that in the first equidistant cylindrical projection image, a first corresponding area corresponding to the first missing area has a pixel value.

[0012] According to an eighth aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method, the image processing method comprising: obtaining a first equidistant cylindrical projection image and a second equidistant cylindrical projection image, the first equidistant cylindrical projection image being generated by a first equidistant cylindrical transformation process based on a first circular fisheye image accompanied by a missing area in which no pixel values ​​exist, the second equidistant cylindrical projection image being generated by a second equidistant cylindrical transformation process based on a second circular fisheye image having a parallax relative to the first circular fisheye image, predetermined pixel values ​​being set in corresponding areas of the first equidistant cylindrical projection image corresponding to the missing area specifying a display position of a three-dimensional virtual reality based on the first equidistant cylindrical projection image and the second equidistant cylindrical projection image; outputting a first display image corresponding to the display position in the first equidistant cylindrical projection image and a second display image corresponding to the display position in the second equidistant cylindrical projection image; and when there is a first specific area overlapping with the corresponding area in the first display image, correcting the first display image or the second display image before outputting the first display image or the second display image so that the first specific area and the second specific area located at the position of the second display image corresponding to the first specific area include the same image.

[0013] According to a ninth aspect of the present invention, there is provided a non-temporary computer-readable storage medium storing a program for causing a computer to execute an image processing method, the image processing method comprising: obtaining an image including a first circular fisheye image and a second circular fisheye image, wherein the first circular fisheye image is accompanied by a missing area where no pixel values ​​exist, and the second circular fisheye image has parallax relative to the first circular fisheye image; setting pixel values ​​of an area of ​​the second circular fisheye image located at a position corresponding to the missing area in the missing area of ​​the first circular fisheye image; and extracting from the image the first circular fisheye image including the pixel values ​​set in the missing area.

[0014] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A and Figure 1B is a diagram showing the overall configuration of the system.

[0016] Figure 2A and Figure 2B is a diagram showing the appearance of the camera 100 .

[0017] Figure 3is a diagram illustrating an example of the internal configuration of the camera 100 .

[0018] Figure 4 is a diagram showing an example of the configuration of a VR180 lens 300 that can be mounted to the camera 100 .

[0019] Figure 5 is a block diagram showing the configuration of the PC 500 .

[0020] Fig. 6A is a flowchart showing the photographing process of the camera 100 .

[0021] Figure 6B is a flowchart showing the photographing process of the camera 100 .

[0022] Figure 7 2 is a flowchart showing the entire equidistant cylindrical transformation process performed by the PC 500.

[0023] Fig. 8A : is a flowchart showing in detail the process of step S712 (equidistant cylindrical projection image generation process) according to the first embodiment.

[0024] Figure 8B : is a flowchart showing in detail the process of step S712 (equidistant cylindrical projection image generation process) according to the second embodiment.

[0025] Figure 8C : is a flowchart illustrating in detail the process of step S712 (equidistant cylindrical projection image generation process) according to the third embodiment.

[0026] Fig.8D : is a flowchart illustrating in detail the process of step S712 (equidistant cylindrical projection image generation process) according to the fourth embodiment.

[0027] Fig. 9 is a diagram showing the structure of an image file recorded by the camera 100 (an image file processed by the PC 500 ).

[0028] Fig. 10A and Fig. 10B is a diagram showing information obtained from the VR 180 lens 300 .

[0029] Fig. 10C is a diagram showing information obtained from the camera 100 .

[0030] Fig.11A and Fig. 11B is a diagram showing an example of an image captured and recorded by the camera 100 to which the VR180 lens 300 is mounted.

[0031] Fig. 11C: is a diagram showing an example of a state in which pixel values ​​are set in a missing area of ​​a circular fisheye image.

[0032] Fig. 12A : is a conceptual diagram showing the process of step S712 (equidistant cylindrical projection image generation process) according to the first embodiment.

[0033] Fig. 12B : is a conceptual diagram showing the process of step S712 (equidistant cylindrical projection image generation process) according to the second embodiment.

[0034] Fig. 12C : is a conceptual diagram showing the process of step S712 (equidistant cylindrical projection image generation process) according to the third embodiment.

[0035] Fig.12D : is a conceptual diagram showing the process of step S712 (equidistant cylindrical projection image generation process) according to the fourth embodiment.

[0036] Fig.13 is a flowchart illustrating image processing according to the fifth embodiment.

[0037] Fig.14 : is a flowchart showing the image processing of step S1310 in detail.

[0038] Fig.15 is a diagram showing a display setting screen according to the fifth embodiment.

[0039] Fig.16A is a conceptual diagram illustrating image processing according to the fifth embodiment.

[0040] Fig. 16B is a conceptual diagram illustrating image processing according to the fifth embodiment.

[0041] Fig. 16C is a conceptual diagram illustrating image processing according to the fifth embodiment.

[0042] Fig.16D is a conceptual diagram illustrating image processing according to the fifth embodiment.

[0043] Fig.17A and Fig. 17B is a flowchart illustrating image processing according to the sixth embodiment.

[0044] Fig.18 is a flowchart showing the processing of step S1715 in detail.

[0045] Fig.19 is a diagram showing a screen of an application program running on the PC 500 according to the sixth embodiment.

[0046] Fig. 20is a conceptual diagram illustrating image processing according to the sixth embodiment.

[0047] Fig.21A and Fig.21B is a conceptual diagram illustrating image processing according to the sixth embodiment. DETAILED DESCRIPTION

[0048] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but the invention is not limited to requiring all such features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar configurations, and redundant descriptions thereof are omitted.

[0049] First embodiment

[0050] Overall system configuration

[0051] Figure 1A and Figure 1B 1 is a diagram showing the overall configuration of a system according to a first embodiment. Figure 1A and Figure 1B 100 denotes a digital camera (hereinafter simply referred to as a "camera") capable of capturing images (still images or moving images). 300 denotes a VR180 lens that can be mounted to the camera 100. The VR180 lens 300 is a binocular lens, and an image with parallax can be obtained by capturing an image by the camera 100 mounted with the VR180 lens 300. The VR180 lens 300 is a lens for capturing an image of so-called "VR180" which is a format of a VR image that enables binocular stereoscopic viewing and has a 180-degree field of view.

[0052] 500 denotes a personal computer (PC) as an example of an image processing device that processes an image captured by the camera 100. The method by which the PC 500 obtains an image from the camera 100 is not limited. Figure 1A As shown, the PC 500 may obtain images from the camera 100 via wireless communication. Alternatively, as Figure 1B As shown, the PC 500 can obtain the image recorded as a file via an external storage device (eg, a memory card) of the camera 100 .

[0053] Configuration of the camera 100

[0054] Figure 2A and Figure 2B is a diagram showing the appearance of the camera 100 . Figure 2A is a perspective view of the camera 100 as seen from the front, and Figure 2B It is a perspective view of the camera 100 as seen from the rear.

[0055] The camera 100 has a shutter button 101, a power switch 102, a mode change switch 103, a main electronic dial 104, a sub-electronic dial 105, a moving image button 106, and a viewfinder external display unit 107 on its top surface. The shutter button 101 is an operation unit for making shooting preparations or making shooting instructions. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode change switch 103 is an operation unit for switching among various types of modes. The main electronic dial 104 is a rotation operation unit for changing setting values ​​such as shutter speed and aperture. The sub-electronic dial 105 is a rotation operation unit for moving a selection frame (cursor) and moving through images, etc. The moving image button 106 is an operation unit for instructing the start and stop of moving image shooting (recording). The viewfinder external display unit 107 displays various setting values ​​such as shutter speed and aperture.

[0056] The camera 100 also has a display unit 108, a touch panel 109, a direction key 110, a set (SET) button 111, an AE lock button 112, a magnification button 113, a playback button 114, and a menu button 115 on its rear surface. The camera 100 also includes an eyepiece portion 116, an eyepiece viewfinder 117 (a look-through type viewfinder), an eye proximity sensing unit 118, and a touch bar 119. The display unit 108 displays images and various types of information, etc. The touch panel 109 is an operation unit that detects a touch operation performed on the display surface (touch operation surface) of the display unit 108. The direction key 110 is an operation unit composed of keys (four-way keys) that can be pressed in the up, down, left, and right directions. Operations can be performed according to the position of the direction key 110 that has been pressed. The set button 111 is an operation unit that is pressed mainly when confirming a selected item. The AE lock button 112 is an operation unit that is pressed when locking the exposure state in a shooting standby state. The zoom button 113 is an operation unit for switching the zoom mode on and off during the live view display (LV display) in the shooting mode. The main electronic dial 104 is operated while the zoom mode is on, which causes the live view image (LV image) to be enlarged or reduced. In addition, the zoom button 113 is used to enlarge the playback image and increase the magnification ratio in the playback mode. The playback button 114 is an operation unit for switching between the shooting mode and the playback mode. Pressing the playback button 114 during the shooting mode causes the transition to the playback mode, and the latest image among the images recorded in the recording medium 227 (described later) can be displayed in the display unit 108.

[0057] The menu button 115 is an operation unit that is pressed when a menu screen in which various types of settings can be made is displayed in the display unit 108. The user can intuitively make various types of settings by using the menu screen displayed in the display unit 108, the direction key 110, the setting button 111, and the like. The eyepiece portion 116 is a portion to which the eye approaches to view the eyepiece viewfinder 117. The user can view an image displayed in an internal electronic viewfinder (EVF) 217 ​​(described later) through the eyepiece portion 116. The eye proximity sensing unit 118 is a sensor that senses whether the user's eye is near the eyepiece portion 116.

[0058] The touch bar 119 is a touch-based operation unit (line touch sensor) in a strip shape capable of accepting a touch operation. The touch bar 119 is arranged at a position where, in a state where the shutter button 101 can be pressed by the index finger of the user's right hand, the user can perform a touch operation (can be touched with the thumb of the right hand) while holding the grip 120 with the right hand (with the pinky finger, ring finger, and middle finger of the right hand). In other words, while viewing the eyepiece finder 117 with the eye close to the eyepiece portion 116, the touch bar 119 can be operated in a state (shooting posture) where the shutter button 101 can be pressed at any time. The touch bar 119 can accept a tap operation (an operation of touching and releasing without moving within a predetermined amount of time) on the touch bar 119 and a left and right slide operation (an operation of touching and then moving the touch position while maintaining contact), etc. The touch bar 119 is an operation unit different from the touch panel 109, and can have a display function. The touch bar 119 according to the present embodiment is a multi-function bar, and is used as, for example, an M-Fn bar.

[0059] The camera 100 further includes a grip 120, a thumb rest 121, a terminal cover 122, a cover 123, and a communication terminal 124. The grip 120 is a holding portion formed in a shape that is convenient for the user to hold the camera 100 with his right hand. The shutter button 101 and the main electronic dial 104 are arranged at positions that can be operated by the index finger of the right hand while holding the camera 100 with the little finger, ring finger, and middle finger of the right hand. The sub-electronic dial 105 and the touch bar 119 are arranged at positions that can be operated by the thumb of the right hand in the same state. The thumb rest 121 (thumb standby position) is a grip portion that is provided at a position on the rear of the camera 100 where the thumb of the right hand holding the grip 120 is easily placed while not operating any operating unit. The thumb rest 121 is composed of a rubber member or the like to increase the holding force (gripping feeling). The terminal cover 122 protects a connector such as a connection cable that connects the camera 100 to an external device. The cover 123 protects a recording medium 227 (described later) and a slot for storing the recording medium 227 by covering the slot. The communication terminal 124 is a terminal for communicating with a lens unit 200 (described later) that can be attached to and removed from the camera 100.

[0060] Figure 3 1 is a diagram showing an example of the internal configuration of the camera 100. Figure 2A and Figure 2B The same elements will be given the same reference numerals and will not be described again as appropriate. The lens unit 200 may be mounted to the camera 100 .

[0061] First, the lens unit 200 will be described. The lens unit 200 is an interchangeable lens that can be attached to and removed from the camera 100. The lens unit 200 is a monocular lens, and is an example of a normal lens.

[0062] The lens unit 200 includes an aperture stop 201, a lens 202, an aperture drive circuit 203, an AF drive circuit 204 (auto focus drive circuit), a lens system control circuit 205, and a communication terminal 206. The aperture stop 201 is configured so that the diameter of the opening can be adjusted. The lens 202 is composed of a plurality of lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the diameter of the opening in the aperture stop 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the aperture drive circuit 203 and the AF drive circuit 204, etc. based on an instruction from a system control unit 50 (described later). The lens system control circuit 205 controls the aperture stop 201 through the aperture drive circuit 203, and shifts the position of the lens 202 through the AF drive circuit 204 to focus the lens 202. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication is performed through the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100 side.

[0063] Next, the configuration of the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50. The shutter 210 is a focal plane shutter by which the exposure time of the imaging unit 211 can be freely controlled based on an instruction from the system control unit 50. The imaging unit 211 is an image sensor composed of a CCD or CMOS element or the like that converts an optical image into an electrical signal. The imaging unit 211 may have an imaging plane phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts an analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, size adjustment processing such as reduction, and color conversion processing, etc.) on data from the A / D converter 212 or data from the memory control unit 213. In addition, the image processing unit 214 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control, etc. based on the results obtained from these calculations. Through this processing, through-the-lens (TTL) AF processing, automatic exposure (AE) processing, flash pre-emission (EF) processing, etc. are implemented. In addition, the image processing unit 214 performs predetermined calculation processing using the captured image data to perform TTL automatic white balance (AWB) processing based on the obtained calculation results.

[0064] The image data from the A / D converter 212 is written to the memory 215 through the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 through the memory control unit 213 without passing through the image processing unit 214. The memory 215 stores the image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, and the image data for display in the display unit 108 and the EVF 217, etc. The memory 215 is provided with a storage capacity sufficient to store a predetermined number of still images and a predetermined amount of time of moving images and audio, etc. The memory 215 is also used as an image display memory (video memory).

[0065] The D / A converter 216 converts the data for image display stored in the memory 215 into an analog signal, and supplies the analog signal to the display unit 108 and the EVF 217. Therefore, the image data for display written in the memory 215 is displayed by the display unit 108 or the EVF 217 or the like via the D / A converter 216. The display unit 108 and the EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and the EVF 217 are, for example, LCD, organic EL or similar displays. The digital signal A / D-converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216, and is sequentially transmitted to the display unit 108 or the EVF 217 or the like and displayed by the display unit 108 or the EVF 217 or the like, thereby realizing live view display.

[0066] The system control unit 50 is a control unit composed of at least one processor and / or at least one circuit. In other words, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the camera 100 as a whole. The system control unit 50 implements each process of the flowchart described later by executing a program recorded in the nonvolatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, and the like.

[0067] The camera 100 further includes a system memory 218, a nonvolatile memory 219, a system timer 220, a communication unit 221, a posture sensing unit 222, and an eye proximity sensing unit 118. The system memory 218 is, for example, a RAM. Operation constants and variables of the system control unit 50 and programs and the like read out from the nonvolatile memory 219 are loaded into the system memory 218. The nonvolatile memory 219 is a memory that can be recorded and erased electrically, and is composed of, for example, an EEPROM. Operation constants and programs and the like of the system control unit 50 are recorded in the nonvolatile memory 219. Here, the "program" is a program for executing a flowchart described later. The system timer 220 is a time measurement unit that measures time used in various types of control and measures time of an internal clock and the like. The communication unit 221 transmits a video signal, an audio signal, and the like to an external device connected wirelessly or by a hard-wired cable, and receives a video signal, an audio signal, and the like from the external device. The communication unit 221 can also be connected to a wireless local area network (LAN), the Internet, and the like. The communication unit 221 is also capable of communicating with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy, etc. The communication unit 221 can transmit images (including real-time images) captured by the camera unit 211 and images recorded in the recording medium 227, etc., and can also receive image data and various other types of information from an external device. The posture sensing unit 222 senses the posture of the camera 100 relative to the gravity direction. Based on the posture sensed by the posture sensing unit 222, it can be determined whether the image captured by the camera unit 211 is an image captured while the camera 100 is held horizontally or an image captured while the camera 100 is held vertically. The system control unit 50 can add orientation information based on the posture sensed by the posture sensing unit 222 to an image file of the image captured by the camera unit 211, and record the image in a rotated state, etc. For example, an accelerometer or a gyro sensor, etc. can be used as the posture sensing unit 222. The posture sensing unit 222 can also be used to sense the movement of the camera 100 (panning, pitching, lifting, and whether the camera is stationary, etc.).

[0068] The eye proximity sensing unit 118 can sense the approach of an object to the eyepiece portion 116 of the eyepiece viewfinder 117 including the EVF 217. For example, an infrared proximity sensor can be used for the eye proximity sensing unit 118. When an object is nearby, infrared light emitted from the light emitting unit of the eye proximity sensing unit 118 is reflected by the object and received by the light receiving unit of the infrared proximity sensor. The distance from the eyepiece portion 116 to the object can be determined by the amount of infrared light received. In this way, the eye proximity sensing unit 118 performs eye proximity sensing, in which the distance of the object to the eyepiece portion 116 is sensed. The eye proximity sensing unit 118 is an eye proximity sensor that detects the approach (eye approach) and separation (eye separation) of the eyepiece portion 116 of the eyepiece viewfinder 117. When an object has been sensed within a predetermined distance from the eyepiece portion 116 in the eye non-approach state (non-approach state), it is determined that the eye approach has been sensed. On the other hand, when the object that has been detected as approaching moves away by a distance greater than or equal to a predetermined distance in the eye approaching state (approaching state), it is determined that eye separation has been sensed. The threshold for sensing eye approaching and the threshold for sensing eye separation may be different by, for example, applying hysteresis. In addition, after eye approaching has been sensed, the eye approaching state is considered to be valid until eye separation is sensed. After eye separation has been sensed, the eye non-approaching state is considered to be valid until eye approaching is sensed. The system control unit 50 switches the display unit 108 and the EVF 217 between display (display state) / non-display (non-display state) according to the state sensed by the eye approaching sensing unit 118. Specifically, when the camera is at least in the shooting standby state and the display destination switching setting is set to automatic switching, the display destination is set to the display unit 108 for display, and when the eye is not nearby, the EVF 217 is set to non-display. When the eye is nearby, the display destination is set to the turned-on EVF217, and the display unit 108 is set to non-display. Note that the eye approach sensing unit 118 is not limited to the infrared proximity sensor, and other sensors may be used as long as they can sense a state that can be regarded as eye approach. The camera 100 also includes a viewfinder external display unit 107, a viewfinder external display drive circuit 223, a power control unit 224, a power unit 225, a recording medium I / F 226, and an operation unit 228. The viewfinder external display unit 107 displays various setting values ​​(such as shutter speed, aperture, etc.) of the camera 100 through the viewfinder external display drive circuit 223. The power control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching a block through which power passes, and the like, and detects whether a battery is connected, the type of battery, the remaining battery power, and the like.The power control unit 224 also controls the DC-DC converter based on the detection result and the instruction from the system control unit 50, and supplies necessary voltage to various units including the recording medium 227 at necessary periods. The power supply unit 225 is a primary battery such as an alkaline battery, a lithium battery, etc., a secondary battery such as a NiCd battery, a NiMH battery, a Li battery, etc., and an AC adapter, etc. The recording medium I / F 226 is an interface for the recording medium 227 such as a memory card, a hard disk, etc. The recording medium 227 is a memory card, etc. for recording captured images, and is composed of a semiconductor memory, a magnetic disk, etc. The recording medium 227 may be removable or built-in.

[0069] The operation unit 228 is an input unit that accepts an operation from a user (user operation), and is used to input various types of instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode change switch 103, the touch panel 109, and the other operation units 229. The other operation units 229 include the main electronic dial 104, the sub-electronic dial 105, the moving image button 106, the direction key 110, the set button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, and the touch bar 119.

[0070] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. When the shutter button 101 is operated halfway, or in other words, when the shutter button 101 is half-pressed (shooting preparation instruction), the first shutter switch 230 turns on, and generates a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processing (such as AF processing, AE processing, AWB processing, and EF processing) in response to the first shutter switch signal SW1. When the shutter button 101 is fully operated, or in other words, when the shutter button 101 is fully pressed (shooting instruction), the second shutter switch 231 turns on, and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processing from reading out a signal from the imaging unit 211 to generating an image file containing a captured image and writing the file to the recording medium 227.

[0071] The mode change switch 103 switches the operation mode of the system control unit 50 between a still image shooting mode, a moving image shooting mode, a playback mode, and the like. The still image shooting mode includes an automatic shooting mode, an automatic scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). There are also various types of scene modes, custom modes, and the like as shooting settings for different shooting scenes. The user can directly switch to any of the above-mentioned shooting modes using the mode change switch 103. Alternatively, the user can selectively switch to any of the displayed multiple modes by using the operation unit 228 after first switching to the shooting mode list screen using the mode change switch 103. Similarly, the moving image shooting mode may include multiple modes.

[0072] The touch panel 109 is a touch sensor that detects various types of touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 has a light transmittance that does not interfere with the display of the display unit 108, and is attached to the upper layer of the display surface of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a graphical user interface (GUI) can be configured so that it looks like the user can directly manipulate the screen displayed in the display unit 108. The touch panel 109 can use any of a variety of systems including a resistive film, an electrostatic capacitor, a surface acoustic wave, an infrared ray, an electromagnetic induction, an image recognition, an optical sensor, etc. According to the type, a touch is sensed when contacting with the touch panel 109, or a touch is sensed when a finger or a pen has been close to the touch panel 109, and any of these types can be used.

[0073] The system control unit 50 can detect the following operations or states on the touch panel 109 .

[0074] A finger or a pen that has not touched the touch panel 109 newly touches the touch panel 109, that is, touch starts (hereinafter referred to as “touchdown”).

[0075] A state in which a finger or a pen is touching the touch panel 109 (hereinafter referred to as “touch-on”).

[0076] A finger or a pen moves while touching the touch panel 109 (hereinafter referred to as “touch movement”).

[0077] The finger or pen that has touched the touch panel 109 is separated from (released) the touch panel 109 , that is, the touch ends (hereinafter referred to as “touch stop”).

[0078] When nothing is touching the touch panel 109 (hereinafter referred to as “not touched”).

[0079] When a touch is detected, touch continuation is detected at the same time. After a touch, touch continuation is usually detected as long as touch stop is not detected. When touch move is detected, touch continuation is also detected at the same time. Even if touch continuation is detected, touch move is not detected as long as the touch position does not move. Untouch occurs after touch stop is detected for all the fingers or pens that touched.

[0080] These operations / states and the position coordinates on the touch panel 109 touched by the finger or pen, etc. are communicated to the system control unit 50 via the internal bus. The system control unit 50 determines what type of operation (touch operation) has been performed on the touch panel 109 based on the communicated information. Regarding touch movement, for each of the vertical component and the horizontal component on the touch panel 109, the moving direction of the finger or pen on the touch panel 109 can be determined based on the change in the position coordinates. If a touch movement greater than or equal to a predetermined distance has been detected, it is determined that a sliding operation has been performed. If, while touching the touch panel 109, the finger quickly moves a given distance and then moves away, the operation is called a "flick". In other words, a "flick" is an operation in which the finger quickly flicks on the touch panel 109. If a touch movement greater than or equal to a predetermined distance and greater than or equal to a predetermined speed is detected, and then the touch is detected to be stopped, it is determined that a flick has been performed (it can be determined that a flick has occurred in continuation of a sliding operation). In addition, when multiple parts (for example, two points) are touched simultaneously (multi-touch) and the touch positions are brought together, the touch operation is called "pinch-in", and when the touch positions are moved apart, the touch operation is called "pinch-out". Pinch-out and pinch-in are collectively referred to as a pinch operation (or simply "pinch").

[0081] VR180 lens 300 configuration

[0082] Figure 4 is a diagram showing a configuration example of a VR180 lens 300 that can be mounted to the camera 100 . Figure 4 A VR180 lens 300 is shown mounted to the camera 100. Figure 4 In the camera 100 shown, Figure 3 The same component configurations as those shown are given the same reference numerals and will not be described again as appropriate.

[0083] The VR180 lens 300 is an interchangeable lens that can be attached to and removed from the camera 100. The VR180 lens 300 is a binocular lens that enables shooting with parallax between the left image and the right image. The VR180 lens 300 has two optical systems, each of which has a wide viewing angle of approximately 180 degrees and can shoot a front hemisphere range. Specifically, the two optical systems of the VR180 lens 300 can shoot a subject with a field of view (viewing angle) of 180 degrees in the left and right directions (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up and down directions (vertical angle, pitch angle, tilt angle), respectively.

[0084] The VR180 lens 300 has a right-eye optical system 301R having a plurality of lenses, a reflector, etc., a left-eye optical system 301L having a plurality of lenses, a reflector, etc., and a lens system control circuit 303. The right-eye optical system 301R corresponds to an example of a first optical system, and the left-eye optical system 301L corresponds to an example of a second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the respective lenses 302R and 302L located on the subject side face the same direction, and the optical axes of the respective lenses 302R and 302L are substantially parallel.

[0085] The VR180 lens 300 according to the present embodiment is a lens for capturing an image of so-called "VR180" which is a format of a VR image that enables binocular stereoscopic viewing and has a 180-degree field of view. The VR180 lens 300 includes a fisheye lens that enables each of the right-eye optical system 301R and the left-eye optical system 301L to capture a range of approximately 180 degrees. Note that the VR180 lens 300 may be any lens that enables the right-eye optical system 301R and the left-eye optical system 301L to each obtain an image that can be displayed as VR180 in binocular VR, and may be a lens that can capture a wide viewing angle range of approximately 160 degrees narrower than the 180-degree range. The VR180 lens 300 may form a right image (first image) formed by the right-eye optical system 301R and a left image (second image) formed by the left-eye optical system 301L on one or two image sensors of the mounted camera, wherein the left image has parallax with respect to the right image.

[0086] The VR180 lens 300 is mounted to the camera 100 using the lens mount 304 and the camera mount 305 of the camera 100. By mounting the VR180 lens 300 to the camera 100, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the VR180 lens 300 are electrically connected through the communication terminal 124 of the camera 100 and the communication terminal 306 of the VR180 lens 300.

[0087] In the present embodiment, a right image formed by the right-eye optical system 301R and a left image having parallax with respect to the right image formed by the left-eye optical system 301L are formed as side-by-side images on the imaging unit 211 of the camera 100. In other words, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single image sensor. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. In this way, by using the VR180 lens 300, two images having parallax can be obtained simultaneously (in groups) from two parts (optical systems) (i.e., the right-eye optical system 301R and the left-eye optical system 301L). In addition, by displaying the obtained images as left-eye images and right-eye images respectively in VR, the user can view a three-dimensional VR image over a range of approximately 180 degrees, which is so-called "VR180".

[0088] Here, a "VR image" is an image that can be displayed in VR (described later). VR images include omnidirectional images (full spherical images) captured by an omnidirectional camera (full spherical camera) and panoramic images having an image range (effective image range) wider than the display range that the display unit can display at one time, etc. VR images are also not limited to still images, but also include moving images and real-time images (images obtained from the camera in almost real time). The image range (effective image range) of a VR image is equivalent to a field of view of 360 degrees in the left-right direction and 360 degrees in the up-down direction. Even if the angle is less than 360 degrees in the left-right direction or less than 360 degrees in the up-down direction, the VR image includes an image having a wider angle of view than that which can be captured by an ordinary camera or an image range wider than the image range that the display unit can display at one time. The image captured by the camera 100 using the above-mentioned VR180 lens 300 is a VR image. For example, by setting the display mode of a display device (a display device capable of displaying VR images) to "VR view", a VR image can be displayed in VR. By displaying a VR image with a 360-degree viewing angle in VR, a user can view a seamless omnidirectional image in the left and right directions by changing the posture of the display device in the left and right directions (horizontal rotation direction).

[0089] Here, VR display (VR view) is a display method (display mode) that can change the display range of VR images, in which the image is displayed in a field of view range based on the posture of the display device. VR display includes "monocular VR display (monocular VR view)", in which a single image is displayed by applying a deformation (deformation with distortion correction applied) that maps the VR image onto a virtual sphere. VR display also includes "binocular VR display (binocular VR view)", in which the left eye VR image and the right eye VR image are displayed side by side in the left and right areas by performing a transformation that maps the left eye VR image and the right eye VR image onto a virtual sphere, respectively. Stereoscopic images can be viewed by performing "binocular VR display" using a left eye VR image and a right eye VR image that have parallax to each other. In any VR display, for example, when a user wears a display device such as a head-mounted display (HMD), the image is displayed in a field of view range corresponding to the direction facing the user's face. For example, it is assumed that at a given point in time, the VR image displays a field of view range centered on 0 degrees in the left and right directions (a specific orientation, such as north) and 90 degrees in the up and down directions (90 degrees from the zenith, i.e. horizontal). If the posture of the display device is flipped from front to back from this state (for example, the display surface is changed from facing south to facing north), the display range of the same VR image is changed to an image with a field of view range centered at 180 degrees in the left-right direction (opposite direction, such as south) and 90 degrees in the up-down direction. In other words, when the user turns his face from north to south (i.e., turns around) while wearing the HMD, the image displayed in the HMD also changes from an image of the north to an image of the south.

[0090] Note that the VR image captured using the VR180 lens 300 of the present embodiment is a VR180 image of approximately 180 degrees in the front, and there is no image of approximately 180 degrees in the rear. If such a VR180 image is displayed in VR and the posture of the display device is changed to the side where there is no image, a blank area is displayed.

[0091] By VR displaying a VR image in this manner, the user has a visual sense of actually being in the VR image (in the VR space). Note that the VR image display method is not limited to a method of changing the posture of the display device. For example, the display range can be moved (scrolled) in response to a user operation performed using a touch panel or direction buttons, etc. During VR display (in the "VR view" display mode), in addition to changing the display range by changing the posture, the display range can be changed in response to a touch movement on the touch panel, a drag operation using a mouse, etc., or pressing a direction button, etc. Note that a smartphone attached to VR goggles (head-mounted adapter) is a type of HMD.

[0092] Configuration of PC 500

[0093] Figure 5is a block diagram showing the configuration of PC 500. Figure 5 501 denotes a control unit that controls the entire PC 500, and is, for example, a central processing unit (CPU). 502 denotes a read-only memory (ROM) that stores programs and parameters that do not need to be changed. An information processing program is stored in the ROM 502 as a program code that can be read by the control unit 501, and the control unit 501 executes the information processing program code. 503 denotes a random access memory (RAM) that temporarily stores programs and data supplied from an external device or the like.

[0094] 504 is an external storage device (including a hard disk or a flash memory, etc.) fixedly installed in the PC 500. Alternatively, the external storage device 504 may be an external storage device including a floppy disk (FD), an optical disk such as a compact disk (CD), a magnetic card or an optical card, an IC card, or a memory card, etc. that can be removed from the PC 500. The image file obtained by the PC 500 from the camera 100 is stored in the external storage device 504.

[0095] 505 denotes an operation unit (such as a button or a touch panel) that receives a user operation and inputs data. 506 denotes a display unit for displaying data held by the PC 500 or supplied data. 507 denotes a communication unit for communicating with an external device such as the camera 100. 508 denotes a system bus that communicatively connects components of the PC 500.

[0096] Characteristics of images captured using VR180 lens 300

[0097] When there is a single-lens optical system, an image rotated 180 degrees is formed on the image sensor. When a normal image is generated from an image rotated 180 degrees, the camera 100 aligns the up-down direction of the image with the up-down direction of the subject by performing a 180-degree rotation process. When the camera 100 equipped with the VR180 lens 300 captures an image, the images of the respective optical systems are formed on a single image sensor by the right-eye optical system 301R and the left-eye optical system 301L. At this time, for each optical system, the image of each optical system is rotated 180 degrees. As in the case of a single-lens optical system, the camera 100 aligns the up-down direction of the image with the up-down direction of the subject by rotating the entire image 180 degrees. Although the rotation during image formation occurs in units of optical systems, the rotation during image generation is performed for the entire image, and therefore, the image corresponding to the left-eye optical system moves to the right side of the entire image, and the image corresponding to the right-eye optical system moves to the left side of the entire image.

[0098] Shooting process of camera 100

[0099] Next, we will refer to Fig. 6A and Figure 6B The following describes the photographing process performed by the camera 100. When the user turns on the camera 100, Fig. 6A and 6B The processing in the flowchart in starts.

[0100] In step S601, the system control unit 50 determines whether the firmware of the camera 100 is compatible with the VR180 lens. If the firmware is determined to be compatible with the VR180 lens, the sequence moves to step S602. If the firmware is determined to be incompatible with the VR180 lens, the sequence moves to step S622. Because the optical system of the VR180 lens is different from that of a typical lens, the camera 100 must be able to read and record VR180 lens metadata for post-processing. Therefore, the system control unit 50 determines whether the firmware is compatible with the VR180 lens.

[0101] In step S602, the system control unit 50 determines whether the VR180 lens (eg, Figure 4 The sequence determines whether the VR180 lens 300 shown in FIG. 1 is mounted to the camera 100. If it is determined that the VR180 lens is mounted, the sequence moves to step S603. If it is determined that the VR180 lens is not mounted, the sequence moves to step S622.

[0102] In step S603, the system control unit 50 obtains the design value of the VR180 lens. The lens design value is used for left-right swap processing and equidistant cylindrical transformation processing (described later).

[0103] In step S604, the system control unit 50 obtains the individual values ​​of the VR180 lens. In addition to the lens design values, the VR180 lens also holds additional information related to the individual values ​​(such as manufacturing errors, etc.). In the equidistant cylindrical transformation process, using the manufacturing error values ​​obtains better results than equidistant cylindrical transformation using only the design values.

[0104] In step S605 , the system control unit 50 obtains an image from the imaging unit 211 .

[0105] In step S606 , the system control unit 50 displays the image obtained in step S605 in the EVF 217 (live view display).

[0106] In step S607, the system control unit 50 determines whether a recording start instruction has been made by operating the shutter button 101. If it is determined that a recording start instruction has been made, the sequence moves to step S608. If it is determined that a recording start instruction has not been made, the sequence moves to step S605. Therefore, the live view display in the EVF is repeated until the user makes a recording start instruction. The recording start instruction may be an instruction to shoot a still image or an instruction to shoot a moving image.

[0107] In step S608 , the system control unit 50 obtains an image from the imaging unit 211 .

[0108] In step S609, the system control unit 50 obtains shooting information such as the shutter speed at the time of shooting and metadata such as the posture information of the camera 100 at the time of shooting. The posture information is obtained from the posture sensing unit 222. When a RAW image is captured, the system control unit 50 also obtains metadata necessary for development and the like.

[0109] In step S610 , the system control unit 50 records the image obtained in step S608 in a file.

[0110] In step S611, the system control unit 50 records the shooting information and metadata obtained in step S609 in association with a file. For example, the system control unit 50 records the shooting information and metadata in the same file as the image.

[0111] In step S612, the system control unit 50 records the information of the VR180 lens obtained in steps S603 and S604 in association with the file. For example, the system control unit 50 records the information of the VR180 lens in the same file as the image.

[0112] In step S613, the system control unit 50 determines whether the user has given a recording end instruction. When shooting a still image, a single image is shot, so it is assumed that a recording start instruction and a recording end instruction are given at the same time. When shooting a moving image, the user presses the shutter button 101 again to give a recording end instruction. If it is determined that a recording end instruction has been given, the processing of this flowchart ends. If it is determined that a recording end instruction has not been given, the sequence moves to step S608. The system control unit 50 repeats the processing from steps S608 to S613 to record moving images one by one into a file.

[0113] Next, a case where the sequence moves from step S601 or step S602 to step S622 will be described. Note that if the sequence has moved from step S601 to step S622, the firmware of the camera 100 is not compatible with the VR180 lens. Therefore, the camera 100 cannot perform processing specific to the VR180 lens. Therefore, even if the VR180 lens is attached to the camera 100, the camera 100 performs the same processing as when a normal lens is attached.

[0114] In step S622, the system control unit 50 determines whether a lens is mounted to the camera 100. If it is determined that a lens is mounted, the sequence moves to step S623. If it is determined that a lens is not mounted, the processing of this flowchart ends.

[0115] In step S623 , the system control unit 50 obtains general information about the lens (eg, lens name).

[0116] In step S624, the system control unit 50 determines whether the general information about the lens has been obtained in step S623. If the general information about the lens has been successfully obtained, the sequence moves to step S625. If the general information about the lens cannot be obtained, the sequence moves to step S626. For example, if the camera 100 and the lens have different manufacturers and the camera 100 cannot communicate with the lens, the camera 100 cannot obtain information from the lens.

[0117] In step S625 , the system control unit 50 holds the general information about the lens acquired in step S623 in the system memory 218 .

[0118] The processing of steps S626 to S632 is similar to the processing of steps S605 to S611 and will therefore not be described again.

[0119] In step S633, the system control unit 50 determines whether the general information related to the lens is retained in the system memory 218 (see step S625). If the general information related to the lens is retained, the sequence moves to step S634. If the general information related to the lens is not saved, the sequence moves to step S635.

[0120] In step S634, the system control unit 50 records the general information about the lens held in the system memory 218 in association with the file. For example, the system control unit 50 records the general information about the lens in the same file as the image.

[0121] The processing of step S635 is similar to that of step S613; that is, if a recording end instruction is made, the processing of this flowchart ends, and if a recording end instruction is not made, the sequence moves to step S629. The system control unit 50 repeats the processing from steps S629 to S635 to record moving images one by one into files.

[0122] Note that Fig. 6A and Figure 6B The image captured in the flowchart can be a still image or a moving image. In addition to the general JPG still image and MP4 moving image, the image to be captured can also be an original still image or an original moving image.

[0123] Image captured by the camera 100 equipped with the VR180 lens 300

[0124] Next, we will refer to Fig.11A and Fig. 11B An example of an image captured and recorded by the camera 100 mounted with the VR180 lens 300 is described.

[0125] The VR180 lens 300 is designed to assume specific values ​​regarding the image sensor size and the angle of view to be recorded. For example, in the case where the VR180 lens 300 assumes a DCI 8K angle of view, the assumed angle of view is 8192×4320.

[0126] On the other hand, depending on the shooting setting, the camera 100 may also record at a viewing angle other than the viewing angle assumed by the VR180 lens 300. For example, in the case of a UHD 8K setting, the viewing angle is 7680×4320. In this case, even if the entire image from the VR180 lens 300 is input to the image sensor, a portion of the image (for example, both ends) is not recorded.

[0127] Fig.11A is an example of an image recorded by the camera 100 using an image sensor of a size assumed by the VR180 lens 300 at a viewing angle using the entire image sensor. This is, for example, a case where the image sensor size assumed by the VR180 lens 300 is DCI 8K (8192×4320) and the recording size is also DCI 8K. In this case, two circular fisheye images aligned left and right are completely recorded.

[0128] Fig. 11Bis an example of an image recorded by the camera 100 using an image sensor having a size assumed by a VR180 lens at a viewing angle using a portion of the image sensor. This is a case where, for example, the image sensor size assumed by the VR180 lens is DCI 8K (8192×4320), but the recording size is UHD 8K (7680×4320). In this case, two circular fisheye images aligned left and right are completely formed on the image sensor, but the two circular fisheye images are recorded with a portion of the image missing. For example, Fig. 11B As shown, in a state where the left portion of the left circular image and the right portion of the right circular image are missing, two circular fisheye images aligned left and right are recorded. In the following description, the missing area in the circular fisheye image will also be referred to as a "missing area".

[0129] Equidistant cylindrical transformation processing using PC 500

[0130] Will refer to Figure 7 and Fig. 8A The equirectangular transformation process performed by the PC 500 is described. The PC 500 performs the equirectangular transformation process on the image contained in the image file obtained from the camera 100. As a result of the equirectangular transformation process, the image is converted from the image file such as the reference image. Fig.11A and Fig. 11B The described left and right circular fisheye image pairs are equivalent to the left and right circular fisheye image pairs to generate left and right equirectangular projection image pairs.

[0131] In step S701, the control unit 501 reads an image from an image file. Fig. 9 A still image or a moving image of an image file shown in and described later.

[0132] In step S702, the control unit 501 reads the shooting information and non-VR180 lens metadata from the image file. Fig. 9 Shooting information and non-VR180 shot metadata for image files shown and described later.

[0133] In step S703, the control unit 501 determines whether the image file has VR180 lens metadata. Fig. 9 The VR180 lens metadata of the image file shown in FIG. 7 and described later. If the image file has the VR180 lens metadata, the sequence moves to step S704. If the image file does not have the VR180 lens metadata, the sequence moves to step S706.

[0134] In step S704 , the control unit 501 reads VR180 lens metadata from the image file.

[0135] In step S705, the control unit 501 obtains the center coordinates and radius of the left and right circular image pair in the recorded image from the design value of the VR180 lens in the VR180 lens metadata read in step S704. If the VR180 lens metadata has individual VR lens values ​​(manufacturing error), the control unit 501 obtains the center coordinates and radius of the left and right circular image pair so as to also reflect the individual values.

[0136] If the sequence has moved from step S703 to step S706, the control unit 501 determines whether the image file has a lens name. For example, if the camera 100 firmware does not support VR180 lenses, the camera 100 will not record VR180 lens metadata, but may record the lens name in an area other than VR180 lens metadata. If the image file has a lens name, the sequence moves to step S707. If the image file does not have a lens name, the sequence moves to step S710.

[0137] In step S707 , the control unit 501 obtains the shot name from the image file.

[0138] In step S708, the control unit 501 determines whether the lens name obtained in step S707 indicates a known VR180 lens. If the lens name obtained in step S707 indicates a known VR180 lens, the sequence moves to step S709. If the lens name obtained in step S707 does not indicate a known VR180 lens, the sequence moves to step S710.

[0139] In step S709, the control unit 501 determines the center coordinates and radius of the left and right circular image pair in the recorded image based on the lens name obtained in step S707. To implement this processing, the program of the PC 500 is configured to hold the lens name and the design values ​​of the center coordinates and radius of the left and right circular image pair in the image captured using the lens. Alternatively, this information may be provided to the program from the outside.

[0140] If the sequence moves from step S706 or step S708 to step S710, the control unit 501 determines whether the image read in step S701 is a left and right circular fisheye image pair. For example, the control unit 501 generates a black and white binary image for the brightness value of the pixel in the read image, wherein if the pixel value of the pixel is less than or equal to a threshold value (a value close to black), the pixel value of the pixel is black, and if the value is greater than the threshold value, the pixel value of the pixel is white. The control unit 501 can then determine whether the image read in step S701 is a left and right circular fisheye image pair based on whether the white area of ​​the black and white binary image is two circles. If the read image is a left and right circular fisheye image pair, the sequence moves to step S711. If the read image is not a left and right circular fisheye image pair, the processing in this flowchart ends.

[0141] In step S711, the control unit 501 detects the center coordinates and radius of two circular fisheye images in the image read in step S701. As is well known, in a circle, when two arbitrary straight lines are drawn to intersect with two points on the circumference and vertical lines are drawn from the midpoints of the respective straight lines, the intersection of the two vertical lines is the center of the circle. Therefore, for example, the control unit 501 uses the black and white binary image generated in step S710 to check the pixel values ​​in the horizontal direction at any vertical position relative to the area forming the circle, and finds the X coordinate X1 of the pixel that has changed from black to white and the X coordinate X2 of the pixel that has changed from white to black. The control unit 501 also checks the pixel values ​​in the vertical direction at any horizontal position, and finds the Y coordinate Y1 of the pixel that has changed from black to white and the Y coordinate Y2 of the pixel that has changed from white to black. In this case, the center coordinates of the circle are (X1+X2) / 2 for the X coordinate and (Y1+Y2) / 2 for the Y coordinate. If the pixel values ​​are checked in the horizontal direction from the center coordinate of the circle, and the X coordinate of the pixel that changes from white to black is represented by X3, the radius of the circle is X3-((X1+X2) / 2).

[0142] In step S712, the control unit 501 uses the center coordinates and radius of the two circular fisheye images obtained by the processing in step S705, step S709, or step S711, and generates an equidistant cylindrical projection image from the two circular fisheye images. Fig. 11B As described, even when the images are captured by the camera 100 mounted with a VR180 lens, the two circular fisheye images may not be recorded in a complete form (ie, portions may be missing).

[0143] Fig. 8A : is a flowchart showing in detail the process of step S712 (equidistant cylindrical projection image generation process) according to the first embodiment.

[0144] In step S801, the control unit 501 finds the image size required to record the entire image captured by the VR180 lens. This can be determined based on the center coordinates and radius of the two circular fisheye images obtained from the processing of step S705, step S709, or step S711. For example, assuming that two circular fisheye images side by side are recorded, the distance from the left and right centers of the image to the X coordinate of the center position of the left circular fisheye image is X1, the distance from the left and right centers of the image to the X coordinate of the center position of the right circular fisheye image is X2, and the radius of the left and right circular fisheye images is R, where (R≤X1) and (R≤X2). At this time, the image size required to fully record the left and right circular fisheye image pair is a width of ((X1+X2)×2) and a height of (R×2).

[0145] In step S802, the control unit 501 determines whether the image size obtained from the image file is smaller than the image size obtained in step S801. If the image size obtained from the image file is smaller than the image size obtained in step S801, the sequence moves to step S803. This means that two circular fisheye images have been recorded with a portion thereof missing. If the image size obtained from the image file is greater than or equal to the image size obtained in step S801, the sequence moves to step S806. This means that the two circular fisheye images are recorded completely.

[0146] In step S803, the control unit 501 allocates a buffer equivalent to the size of the image obtained in step S801 in the RAM 503, and sets all pixel values ​​to black.

[0147] In step S804, the control unit 501 copies the image data obtained in step S701 to the buffer allocated in step S803. At this time, the control unit 501 performs the copying so that the distance from the left and right centers of the buffer allocated in step S803 to the X coordinate of the center position of the left circular fisheye image is X1 obtained in step S801. Fig. 11C The resulting buffer image is shown in .

[0148] In step S805 , the control unit 501 selects the buffer image as a transformation source image for equidistant cylindrical transformation.

[0149] Fig. 12A The image at the top represents the transformed source image stored in the buffer. Fig. 12AThe image at the middle bottom represents the transformation destination image (transformation result image) of the equidistant cylindrical transformation. As indicated by arrow 1201, the coordinates at the right end of the left eye equidistant cylindrical projection image correspond to the coordinates at the right end of the left eye circular fisheye image, but the pixels at these coordinates are not recorded, and thus point to the black pixels set in step S803. Similarly, as indicated by arrow 1202, the coordinates at the left end of the right eye equidistant cylindrical projection image correspond to the coordinates at the left end of the right eye circular fisheye image, but the pixels at these coordinates are not recorded, and thus point to the black pixels set in step S803.

[0150] If the sequence has moved from step S802 to step S806 , the control unit 501 selects the image obtained in step S701 as the transformation source image for equidistant cylindrical transformation.

[0151] In step S807, the control unit 501 determines whether all pixels in the transformation destination image (equidistant cylindrical projection image) have been processed. If all pixels in the transformation destination image have been processed, the processing in this flowchart ends. If there are unprocessed pixels, the sequence moves to step S808.

[0152] In step S808, the control unit 501 obtains (identifies) the coordinates of the transformation source image (circular fisheye image) corresponding to the coordinates of the pixel to be processed in the transformation destination image (equidistant cylindrical projection image). This can be done using a general method of equidistant cylindrical transformation. Fig. 12A As shown, even if a portion of the recorded circular fisheye image is missing, using the image in the buffer prepared in steps S803 to S805 as the transformation source image ensures that an invalid area is not referenced during the equidistant cylindrical transformation. As a result, black pixels can be set for areas of the equidistant cylindrical projection image corresponding to areas that do not exist in the circular fisheye image.

[0153] In step S809 , the control unit 501 obtains pixel values ​​from the coordinates of the transformation source image obtained in step S808 , and sets the obtained pixel values ​​to the pixels to be processed in the transformation destination image.

[0154] In step S810, the control unit 501 selects the next pixel in the transformation destination image as the pixel to be processed. For example, when processing from the upper left pixel to the lower right pixel of the transformation destination image, the next pixel is the right adjacent pixel in the same row. However, if the X coordinate of the right adjacent pixel reaches the image width, the left end pixel under one row is used as the next pixel.

[0155] The above process ensures that even when the circular fisheye image with the missing area is subjected to the equidistant cylindrical transformation process, the coordinates of the circular fisheye image corresponding to the coordinates in the equidistant cylindrical projection image do not point to the missing area in the circular fisheye image.

[0156] Image file formats

[0157] Next, we will refer to Fig. 9 The structure of an image file recorded by the camera 100 (an image file processed by the PC 500) is described. Fig. 9 In the figure, 901 represents the header of the file. Information such as the image type is recorded in the header of the file. 902 represents the shooting information when the image recorded in the file was shot. Information such as the shutter speed and aperture is recorded in the shooting information. 903 represents VR180 lens metadata (metadata related to the VR180 lens). Information such as the lens name and the circular fisheye radius and manufacturing error information are recorded as VR180 lens metadata. 904 represents non-VR180 lens metadata. Information on frame-by-frame changes for moving images and data required for RAW development are recorded in non-VR180 lens metadata. For cameras whose firmware does not support VR180 lenses, the lens name is recorded in the non-VR180 lens metadata. 905 represents still image or moving image data. In the case of moving images, audio data is recorded in addition to the image.

[0158] exist Fig. 9 In the example of FIG. 1 , shooting information 902, VR180 lens metadata 903, and non-VR180 lens metadata 904 are recorded in an image file. However, a configuration may be adopted in which shooting information 902, VR180 lens metadata 903, and non-VR180 lens metadata 904 are recorded in a file different from the image file, and then the recorded information is associated with the image file.

[0159] Information obtained from the VR180 lens 300 and the camera 100

[0160] Will refer to Fig. 10A An example of information that the camera 100 obtains from the VR180 lens 300 is described. The following information is obtained from the VR180 lens 300.

[0161] 1. Lens design value

[0162] 2. Individual lens value (manufacturing error value)

[0163] 3. Lens logo information

[0164] 4. Lens focal length information

[0165] 5. Lens thermometer information

[0166] The lens design values ​​are the design value data common to both eyes of VR180 for aberration correction. Fig. 10B Describe the lens design values ​​in detail.

[0167] The individual lens value (manufacturing error value) is data indicating the manufacturing error of the VR180 lens 300. During the process of manufacturing the VR180 lens 300, errors occur on both the left and right sides. If the PC 500 performs equidistant cylindrical transformation processing in the presence of the error, the quality of 3D display will be degraded. In order to suppress this problem, the measurement results of the errors detected during the lens manufacturing process are recorded in the lens. This will be referred to later. Fig. 10B Describes individual lens values ​​in detail.

[0168] When performing image processing, it is necessary for the PC 500 to determine whether to perform conversion processing on the image data captured by the VR180 lens 300. The lens flag is recorded to enable this determination. The lens flag is a flag indicating that the image data is captured by the VR180 lens.

[0169] The lens focal length is the distance from the "principal point" as the center of the lens to the image formed on the image sensor. When the image data captured by the VR180 lens 300 undergoes a conversion process using the PC 500, a high-precision (decimal point unit) value is required as the lens focal length in order to maintain the quality of 3D display.

[0170] The lens thermometer information indicates the temperature of the VR180 lens 300. The lens temperature information is recorded at the time of shooting, and thus the PC 500 can determine the ambient temperature.

[0171] Will refer to Fig. 10B Describes lens design values ​​and individual lens values ​​(manufacturing error values). This information is used for left-right swapping processing and equidistant cylindrical transformation processing, etc. The lens design values ​​include, for example, the following information.

[0172] 1. Image circle position

[0173] 2. Image circle diameter

[0174] 3. Perspective

[0175] 4. Distortion correction coefficient

[0176] The image circle position is the design value of the center position of the image circle formed on the image sensor. For example, the image circle position is defined as the coordinates in the horizontal and vertical directions with the mount as the origin.

[0177] The image circle diameter is the diameter of the image circle formed on the image sensor.

[0178] The viewing angle is the range of viewing angles of the image formed in the image circle.

[0179] The distortion correction factor is the ratio of the design image height of the lens to the ideal image height. Values ​​can be set for various image heights and interpolated between them, or approximated by a polynomial.

[0180] In addition, when the application on the PC 500 is connected to the camera 100 and displayed, a magic window display is performed on the screen of the PC 500. In this case, the "image circle position", "image circle diameter" and "angle of view" are used to maintain the posture and display quality. The PC 500 edits the information into information suitable for the application and uses the information to correctly display the image on the screen. For example, the PC 500 multiplies the "image circle position" and "image circle diameter" by coefficients according to the magic window display.

[0181] The individual lens value (manufacturing error value) includes, for example, the following information.

[0182] 5. Image circle position deviation

[0183] 6. Optical axis tilt

[0184] 7. Image magnification deviation

[0185] Image circle position deviation is a deviation from a design value of the center position of an image circle formed on an image sensor. For example, the image circle position deviation is defined by horizontal and vertical coordinates with the design position as the origin.

[0186] The optical axis tilt is a deviation in the orientation of the optical axis on the object side. For example, the tilt deviation in the horizontal direction and the tilt deviation in the vertical direction are described in terms of angles.

[0187] Image magnification deviation is the deviation of image size from the design value. For example, image magnification deviation is described as a ratio to the design value.

[0188] These individual values ​​are caused by manufacturing errors such as lens decentering and tilt, and therefore vary from lens to lens. Therefore, for each of the left and right optical systems, individual lens values ​​are measured and recorded.

[0189] Will refer to Fig. 10C Describes metadata obtained from the camera 100. This metadata is used to maintain 3D quality based on the display area and posture information when the application in the PC 500 performs conversion processing. The metadata includes, for example, the following information.

[0190] 1. Camera recording area information

[0191] 2. Accelerometer information in the camera

[0192] 3. Aperture value right eye correction information

[0193] The camera recording area information is the effective image area. The effective image area that can be displayed depends on the camera's image sensor and recording mode. This information is used by the application in the PC 500 to perform more accurate image display during the conversion process.

[0194] The in-camera accelerometer information is roll and pitch of posture information measured using an accelerometer (horizontal meter) in the camera. This information is used in an application in the PC 500 for electronic image stabilization and horizontal correction.

[0195] The aperture value right eye correction information is an exposure setting value used as a reference for the left eye. This information is used to maintain 3D quality and perform playback display without discomfort when the application in the PC 500 performs conversion processing.

[0196] As described so far, according to the first embodiment, the PC 500 generates an equidistant cylindrical projection image by performing equidistant cylindrical transformation processing based on a circular fisheye image having a missing area where no pixel value exists. Before the equidistant cylindrical transformation processing, the PC 500 also sets a predetermined pixel value in the missing area of ​​the circular fisheye image. This makes it possible to generate an equidistant cylindrical projection image without missing pixel values ​​even when the circular fisheye image has a missing area.

[0197] Although the foregoing example describes the predetermined pixel value as a pixel value representing black, it should be noted that the predetermined pixel value is not limited to a specific color and may be, for example, a pixel value representing white.

[0198] Second embodiment

[0199] The second embodiment will describe a process for generating an equidistant cylindrical projection image ( Figure 7 In the present embodiment, the basic configuration of the camera 100 and the PC 500 is similar to that of the first embodiment. The areas different from the first embodiment will be mainly described below.

[0200] Figure 8B Detailed description of the process of step S712 (equidistant cylindrical projection image generation process) according to the second embodiment. Figure 8B In, with Fig. 8A Unlike (First Embodiment), regardless of whether the circular fisheye image obtained from the image file has a missing portion, the control unit 501 uses the obtained circular fisheye image as a transformation source image for equidistant cylindrical transformation.

[0201] The processing of steps S811 and S812 is the same as Fig. 8A The processing of steps S807 and S808 in is similar and therefore will not be described again.

[0202] In step S813, the control unit 501 determines whether the transformation source coordinates (the coordinates of the transformation source image corresponding to the coordinates of the pixel to be processed in the transformation destination image) obtained in step S812 are within the transformation source image (circular fisheye image). If the transformation source coordinates are within the transformation source image, the sequence moves to step S814. If the transformation source coordinates are not within the transformation source image, the sequence moves to step S815.

[0203] The processing of step S814 (processing when the transformation source coordinates are within the transformation source image) is the same as Fig. 8A The processing of step S809 in is similar and therefore will not be described again.

[0204] On the other hand, if the transformation source coordinates are not within the transformation source image, then in step S815, the control unit 501 sets the coordinates of the pixel to be processed in the transformation destination image (equidistant cylindrical projection image) to black.

[0205] Fig. 12B The image at the top in the middle represents a circular fisheye image recorded in a partially missing manner, which is used as a transformation source image for equidistant cylindrical transformation. Fig. 12B The image at the middle bottom represents the transformation destination image (transformation result image) of the equidistant cylindrical transformation. As shown by arrow 1211, the coordinates at the right end of the left eye equidistant cylindrical projection image correspond to the coordinates at the right end of the left eye circular fisheye image, but the pixels at these coordinates are not recorded. Therefore, in step S815, the control unit 501 sets the pixels to be processed in the transformation destination image to black. Similarly, as shown by arrow 1212, the coordinates at the left end of the right eye equidistant cylindrical projection image correspond to the coordinates at the left end of the right eye circular fisheye image, but the pixels at these coordinates are not recorded. Therefore, in step S815, the control unit 501 sets the pixels to be processed in the transformation destination image to black.

[0206] The processing of step S816 is Fig. 8A The processing of step S810 in is similar and therefore will not be described again.

[0207] As described so far, according to the second embodiment, if the coordinates of the transformation source image (circular fisheye image) corresponding to the coordinates of the pixel to be processed in the transformation destination image (equidistant cylindrical projection image) of the equidistant cylindrical transformation process point to the missing area, the PC 500 sets the pixel value (predetermined pixel value) representing black to the pixel to be processed. This makes it possible to normally perform the equidistant cylindrical transformation process even when the circular fisheye image is accompanied by the missing area.

[0208] Third embodiment

[0209] The third embodiment will describe a process for generating an equidistant cylindrical projection image ( Figure 7 In the present embodiment, the basic configuration of the camera 100 and the PC 500 is similar to that of the first embodiment. The areas different from the first embodiment will be mainly described below.

[0210] When viewing an equirectangular projection image generated by processing a pair of left and right circular fisheye images captured from a VR180 lens by equirectangular transformation on a head mounted display (HMD), the user can view the two equirectangular projection images as a single three-dimensional image. Fig. 12A and Fig. 12B As shown, when an equidistant cylindrical projection image is generated according to the first embodiment or the second embodiment, for some areas, the pixels of the equidistant cylindrical projection image may become black only for one of the left eye or the right eye. When the user views these two equidistant cylindrical projection images on the HMD as a single three-dimensional image, the user may feel uncomfortable. The third embodiment describes a configuration for suppressing such discomfort.

[0211] Figure 8C Detailed description of the process of step S712 (equidistant cylindrical projection image generation process) according to the third embodiment. Figure 8C In, with Fig. 8A Unlike (First Embodiment), regardless of whether the circular fisheye image obtained from the image file has a missing portion, the control unit 501 uses the obtained circular fisheye image as a transformation source image for equidistant cylindrical transformation.

[0212] In step S821, the control unit 501 determines whether all pixels in the left eye transformation destination image (equidistant cylindrical projection image corresponding to the left eye) have been processed. In the present embodiment, in one processing cycle from step S821 to step S827, in addition to the pixels to be processed in the left eye transformation destination image, pixels located at the same coordinates as the coordinates of the pixels to be processed in the left eye transformation destination image are also processed in the right eye transformation destination image (equidistant cylindrical projection image corresponding to the right eye). Therefore, if all pixels in the left eye transformation destination image have been processed, all pixels in the right eye transformation destination image have also been processed. Therefore, if all pixels in the left eye transformation destination image have been processed, the processing in this flowchart ends. If there are unprocessed pixels in the left eye transformation destination image, the sequence moves to step S822.

[0213] In step S822, the control unit 501 obtains the coordinates of the left-eye transformation source image (left-eye circular fisheye image) corresponding to the coordinates of the pixel to be processed in the left-eye transformation destination image (left-eye transformation destination coordinates) (i.e., left-eye transformation source coordinates). Here, the left-eye transformation destination coordinates are a coordinate system with the upper left of the left-eye transformation destination image as its origin. The process for obtaining the left-eye transformation source coordinates corresponding to the left-eye transformation destination coordinates can be performed using a general method of equidistant cylindrical transformation.

[0214] In step S823, the control unit 501 obtains the coordinates of the right eye transformation source image (right eye circular fisheye image) corresponding to the coordinates of the pixel to be processed in the right eye transformation destination image (right eye transformation destination coordinates) (i.e., right eye transformation source coordinates). Here, the right eye transformation destination coordinates are a coordinate system with the upper right of the right eye transformation destination image as its origin. The process for obtaining the right eye transformation source coordinates corresponding to the right eye transformation destination coordinates can be performed using a general method of equidistant cylindrical transformation.

[0215] In step S824, the control unit 501 determines whether the left eye transform source coordinates are within the left eye transform source image and whether the right eye transform source coordinates are within the right eye transform source image. If the left eye transform source coordinates are within the left eye transform source image and the right eye transform source coordinates are within the right eye transform source image, the sequence moves to step S825. If at least one of the left eye transform source coordinates and the right eye transform source coordinates is not within the corresponding transform source image, the sequence moves to step S826.

[0216] In step S825, the control unit 501 obtains a pixel value from the left eye transformation source coordinates of the left eye transformation source image, and sets the pixel value to the pixel to be processed in the left eye transformation destination image. Similarly, the control unit 501 obtains a pixel value from the right eye transformation source coordinates of the right eye transformation source image, and sets the pixel value to the pixel to be processed in the right eye transformation destination image.

[0217] On the other hand, if the sequence has moved from step S824 to step S826 , the control unit 501 sets both the pixels to be processed in the left eye conversion destination image and the pixels to be processed in the right eye conversion destination image to black.

[0218] Fig. 12C The image at the top in the middle represents a circular fisheye image recorded in a partially missing manner, which is used as a transformation source image for equidistant cylindrical transformation. Fig. 12C The image at the middle bottom represents the transformation destination image (transformation result image) of the equidistant cylindrical transformation.

[0219] As indicated by arrow 1221, the coordinates of the left end of the left eye equidistant cylindrical projection image correspond to the coordinates of the left end of the left eye circular fisheye image, and pixels are recorded at the coordinates of the left end of the left eye circular fisheye image. On the other hand, although the starting point of arrow 1222 in the right eye equidistant cylindrical projection image is located at a position corresponding to the starting point of arrow 1221 in the left eye equidistant cylindrical projection image, no pixels are recorded at the position indicated by arrow 1222 in the right eye circular fisheye image (the coordinates of the left end of the right eye circular fisheye image). In this case, black is set for both the starting point of arrow 1221 in the left eye equidistant cylindrical projection image and the starting point of arrow 1222 in the right eye equidistant cylindrical projection image.

[0220] As indicated by arrow 1223, the coordinates of the right end of the right eye equidistant cylindrical projection image correspond to the coordinates of the right end of the right eye circular fisheye image, and pixels are recorded at the coordinates of the left end of the right eye circular fisheye image. On the other hand, although the starting point of arrow 1224 in the left eye equidistant cylindrical projection image is located at a position corresponding to the starting point of arrow 1223 in the right eye equidistant cylindrical projection image, no pixels are recorded at the position indicated by arrow 1224 in the left eye circular fisheye image (the coordinates of the right end of the left eye circular fisheye image). In this case, black is set for both the starting point of arrow 1223 in the right eye equidistant cylindrical projection image and the starting point of arrow 1224 in the left eye equidistant cylindrical projection image.

[0221] In step S827, the control unit 501 selects the next pixel as a pixel to be processed for each of the left eye transformation destination image and the right eye transformation destination image. For example, when processing from the upper left pixel to the lower right pixel of the left eye transformation destination image, the next pixel is the right adjacent pixel in the same row. However, if the X coordinate of the right adjacent pixel reaches the width of the left eye transformation destination image, the left end pixel under one row in the left eye transformation destination image is taken as the next pixel. The same applies to the right eye transformation destination image.

[0222] As described so far, according to the third embodiment, for the same coordinates in the left-eye equidistant cylindrical projection image and the right-eye equidistant cylindrical projection image, if one of the images corresponds to a missing area, the PC 500 also sets a pixel value (predetermined pixel value) representing black for the other image. This makes it possible to suppress a situation in which pixels become black only at the same coordinates in one of the images.

[0223] Fourth embodiment

[0224] The fourth embodiment will describe a process for generating an equidistant cylindrical projection image ( Figure 7712 in step S714). In the fourth embodiment, similarly to the third embodiment, the PC 500 performs processing to suppress the situation where pixels become black at the same coordinates in only one of the left-eye equidistant cylindrical projection image and the right-eye equidistant cylindrical projection image, but the specific details of the processing are different from the third embodiment. In this embodiment, the basic configuration of the camera 100 and the PC 500 is similar to that of the third embodiment. The areas different from the third embodiment will be mainly described below.

[0225] Fig.8D Detailed description of the process of step S712 (equidistant cylindrical projection image generation process) according to the fourth embodiment. Fig.8D In, with Figure 8C (Third Embodiment) Likewise, regardless of whether the circular fisheye image obtained from the image file has a missing portion, the control unit 501 uses the obtained circular fisheye image as a transformation source image for equidistant cylindrical transformation.

[0226] The processing of steps S831 to S835 is the same as Figure 8C The processing of steps S821 to S825 in is similar and therefore will not be described again.

[0227] In step S836, the control unit 501 determines whether the left eye transformation source coordinates are outside the left eye transformation source image and whether the right eye transformation source coordinates are outside the right eye transformation source image. If the left eye transformation source coordinates are outside the left eye transformation source image and the right eye transformation source coordinates are outside the right eye transformation source image, the sequence moves to step S837. In other cases, the sequence moves to step S838.

[0228] In step S837 , the control unit 501 sets both the pixels to be processed in the left-eye conversion destination image and the pixels to be processed in the right-eye conversion destination image to black.

[0229] In step S838, the control unit 501 determines whether the left eye transformation source coordinates are within the left eye transformation source image. If the left eye transformation source coordinates are within the left eye transformation source image, the sequence moves to step S839. If the left eye transformation source coordinates are not within the left eye transformation source image (that is, if the right eye transformation source coordinates are within the right eye transformation source image), the sequence moves to step S840.

[0230] In step S839 , the control unit 501 obtains pixel values ​​from the left eye transform source coordinates of the left eye transform source image, and sets the obtained pixel values ​​to pixels to be processed in both the left eye transform destination image and the right eye transform destination image.

[0231] In step S840 , the control unit 501 obtains pixel values ​​from the right eye transform source coordinates of the right eye transform source image, and sets the obtained pixel values ​​to pixels to be processed in both the left eye transform destination image and the right eye transform destination image.

[0232] Fig.12D The image at the top in the middle represents a circular fisheye image recorded in a partially missing manner, which is used as a transformation source image for equidistant cylindrical transformation. Fig.12D The image at the middle bottom represents the transformation destination image (transformation result image) of the equidistant cylindrical transformation.

[0233] As indicated by arrow 1231, the coordinates of the left end of the left eye equidistant cylindrical projection image correspond to the coordinates of the left end of the left eye circular fisheye image, and pixels are recorded at the coordinates of the left end of the left eye circular fisheye image. On the other hand, although the starting point of arrow 1232 in the right eye equidistant cylindrical projection image is located at a position corresponding to the starting point of arrow 1231 in the left eye equidistant cylindrical projection image, no pixel is recorded at the position indicated by arrow 1232 in the right eye circular fisheye image (the coordinates of the left end of the right eye circular fisheye image). In this case, as a result of the processing in step S839, for both the starting point of arrow 1231 in the left eye equidistant cylindrical projection image and the starting point of arrow 1232 in the right eye equidistant cylindrical projection image, a pixel value is set in the position in the left eye circular fisheye image indicated by arrow 1231 (the coordinates of the left end of the left eye circular fisheye image). In other words, as indicated by the dotted arrow 1233, the pixel values ​​of the corresponding area of ​​the left eye equidistant cylindrical projection image are copied to the area in the right eye equidistant cylindrical projection image corresponding to the missing area of ​​the right eye circular fisheye image.

[0234] Likewise, as indicated by arrow 1234, the coordinates of the right end of the right eye equidistant cylindrical projection image correspond to the coordinates of the right end of the right eye circular fisheye image, and pixels are recorded at the coordinates of the right end of the right eye circular fisheye image. On the other hand, although the starting point of arrow 1235 in the left eye equidistant cylindrical projection image is located at a position corresponding to the starting point of arrow 1234 in the right eye equidistant cylindrical projection image, no pixel is recorded at the position indicated by arrow 1235 in the left eye circular fisheye image (the coordinates of the right end of the left eye circular fisheye image). In this case, as a result of the processing in step S840, a pixel value is set in the position in the right eye circular fisheye image indicated by arrow 1234 (the coordinates of the right end of the right eye circular fisheye image) for both the starting point of arrow 1234 in the right eye equidistant cylindrical projection image and the starting point of arrow 1235 in the left eye equidistant cylindrical projection image. In other words, as indicated by dashed arrow 1236, the pixel values ​​of the corresponding area of ​​the right-eye equidistant cylindrical projection image are copied to the area in the left-eye equidistant cylindrical projection image corresponding to the missing area of ​​the left-eye circular fisheye image.

[0235] The processing of step S841 is Figure 8C The processing of step S827 in is similar and will therefore not be described again.

[0236] As described above, according to the fourth embodiment, if, for a missing area in the first circular fisheye image, the corresponding area in the second circular fisheye image is not a missing area, the PC 500 sets the pixel value of the corresponding area of ​​the second circular fisheye image in the corresponding area in the first equidistant cylindrical projection image corresponding to the missing area of ​​the first circular fisheye image. This makes it possible to suppress a situation in which pixels become black only at the same coordinates in one of the images, while reducing the black area.

[0237] Fifth embodiment

[0238] like Fig. 12A and Fig. 12B As shown, when an equirectangular projection image is generated according to the first embodiment or the second embodiment, for some areas, the pixels of the equirectangular projection image may become black only for one of the left eye or the right eye. When a user views VR based on two such equirectangular projection images, a problem may occur in which only one of the left eye image or the right eye image is displayed as blackened, making it difficult to see. The fifth embodiment describes a configuration for suppressing this problem during viewing.

[0239] Will refer to Fig.15 Describes the selection of processing for the blackened portion outside the viewing angle. Radio button 1501 corresponds to a process for setting the opposite side pixels corresponding to the pixels outside the viewing angle to black. Radio button 1502 corresponds to a process for using the opposite side pixels to compensate for the portion where the pixels outside the viewing angle are blackened. Radio button 1503 corresponds to a process for displaying the opposite side images on the left and right when the state is a state of displaying pixels outside the viewing angle. Only one of radio buttons 1501, 1502, and 1503 can be selected. 1504 is an OK button. After the user selects the processing, the user can close the process by pressing OK button 1504. Fig.15 The settings screen in .

[0240] Will refer to Figures 16A to 16D An example of viewing the left end in VR180 and an example of processing a portion that is out of the viewing angle and painted black will be described.

[0241] Fig.16AAn example of a case where no processing is performed when displayed is shown. 1601 is an image from the left lens of the VR180 lens. 1602 is a display area of ​​the left eye image when viewed with VR180. 1603 indicates a portion that is out of view and painted black in the left lens of the VR180 lens. 1604 is an image from the right lens of the VR180 lens. 1605 is a display area of ​​the right eye image when viewed with VR180. 1606 indicates a portion that is out of view and painted black in the right lens of the VR180 lens. 1607 is an arrow indicating that the black-painted area out of view appears only in the right eye image. 1609 is the display of the left eye image when viewed with VR180. 1610 is the display of the right eye image when viewed with VR180. As indicated by arrow 1607, only for the right eye image, there is a black-painted area out of view where the image is not displayed.

[0242] Fig. 16B When choosing Fig.15 1609 is an example of a display of a left eye image before processing. 1610 is an example of a display of a right eye image before processing. 1611 is an example of a display of a left eye image after processing. 1612 is an example of a display of a right eye image after processing. 1613 represents a blackened portion added by processing. Adding the blackened portion ensures that the blackening appears in the same area of ​​the right eye image and the left eye image, which eliminates viewing difficulties caused by the image appearing on only one side.

[0243] Fig. 16C When choosing Fig.15 An example of display when using the opposite side pixels to compensate for the portion of the pixel outside the viewing angle when selecting the radio button 1502 in FIG. Fig. 16B 1614 is an example of display of a processed left-eye image. 1615 is an example of display of a processed right-eye image. 1616 indicates that pixels are brought to a copy source area of ​​the right-eye image. 1617 indicates a state of an image displayed as a result of copying pixels from the left-eye image. 1618 is an arrow showing a process of copying pixels from the left-eye image to the right-eye image. By copying pixels and adding the image to the blackened portion of the right-eye image, the state in which only a portion of one of the right-eye image and the left-eye image is blackened, which makes viewing difficult, is eliminated.

[0244] Fig.16D Shows when you select Fig.15 An example of display when the radio button 1503 in the image is displayed for both images when the image is not outside the viewing angle. Fig. 16B1619 is an example of display of the processed left-eye image. 1620 is an example of display of the processed right-eye image. 1621 is an arrow indicating that the left-eye image is displayed in the right-eye image. By replacing the entire display screen in which the area outside the viewing angle is blackened with the left-eye image, the state in which only a part of one of the right-eye image and the left-eye image is blackened, which makes viewing difficult, is eliminated.

[0245] Fig.13 1 is a flowchart showing image processing according to the fifth embodiment. Fig.15 The selection of the processing method for the black-painted area outside the viewing angle and the determination of whether there is a black-painted area outside the viewing angle When a playback instruction of the VR180 image is input to the PC 500, the processing of this flowchart starts.

[0246] In step S1301, the control unit 501 displays a display setting screen of a portion outside the viewing angle. Fig.15 The setting screen shown. In this embodiment, the first display is for setting the playback method. When the user operates the operation unit 505, selects the radio button 1501, 1502 or 1503 on the screen and presses the OK button 1504, the sequence moves to step S1302.

[0247] In step S1302, the control unit 501 obtains the setting selected in step S1301.

[0248] In step S1303, the control unit 501 obtains the input image. In this embodiment, there is an area outside the viewing angle, so the control unit 501 obtains an image after equidistant cylindrical transformation in which a part of the image is missing. The image obtained here is Fig.16A The left lens image 1601 and the right lens image 1604 are shown in FIG.

[0249] In step S1304, the control unit 501 cuts out the central portions (initial positions to be displayed first) of the left and right eye images from the obtained equidistant cylindrically transformed images, and reads these portions as images for display.

[0250] In step S1305 , the control unit 501 displays (outputs) the left and right eye images read in step S1304 .

[0251] In step S1306, the control unit 501 determines whether an instruction to end the display has been made. Any existing technology can be used for the UI and operation of ending the display.

[0252] In step S1307, the control unit 501 determines whether an instruction to change the display position has been given. The instruction to change the display position is given by the user operating the operation unit 505 and moving the display position using the cursor keys or a joystick, etc. Note that the display monitor itself may include a gyro sensor, and the control unit 501 may obtain the movement based on the acceleration of the display monitor and regard the movement as an instruction to change the display position.

[0253] In step S1308 , the control unit 501 obtains left and right eye display images corresponding to the change instruction made in step S1307 .

[0254] In step S1309, the control unit 501 determines whether the image obtained in step S1308 has an area that is outside the viewing angle and is painted black. A method of determining whether there is a black area where the pixel values ​​are all zero RGB values ​​can be used as a determination method. A method of detecting an area where the RGB value is less than or equal to a given threshold can be used as another determination method. Because the black area appears at the edge of the image, the determination may be limited to the edge of the image. If metadata belonging to the black area is added to the image, the determination can be made based on whether the display area includes the area represented by the metadata. If it is determined that there is no area outside the viewing angle, the sequence moves to step S1305. If it is determined that there is an area outside the viewing angle, the sequence moves to step S1310.

[0255] In step S1310 , the control unit 501 processes the left and right eye images according to the settings obtained in step S1302 .

[0256] Fig.14 is a flowchart showing in detail the image processing performed in step S1310.

[0257] In step S1401, the control unit 501 identifies which of the left and right eye sides has an area outside the viewing angle. Here, the control unit 501 identifies which of the left and right eye sides has a blackened area outside the viewing angle by a method similar to the method used in step S1309. Fig.16A As shown, the following description assumes that there is an area outside the viewing angle and painted black in the right eye image.

[0258] In step S1402, the control unit 501 determines whether the radio button 1503 (setting for displaying the same image on both left and right) is selected in step S1301. If it is determined that the radio button 1503 is selected, the sequence moves to step S1403. In other cases, the sequence moves to step S1404.

[0259] In step S1403, the control unit 501 replaces the image including the area outside the viewing angle with the image not including the area outside the viewing angle. Fig.16D As shown by arrow 1621 in , the control unit 501 displays the left eye image 1609 before processing as the right eye image 1620 after processing. As a result, the same image as the left eye image 1619 (which does not have the area outside the viewing angle) is displayed as the right eye image 1620. Instead of replacing the image including the area outside the viewing angle with the image not including the area outside the viewing angle, the image not including the area outside the viewing angle may be replaced with the image including the area outside the viewing angle so as to eliminate the state where the blacked-out area is located only on one side.

[0260] In step S1404, the control unit 501 selects a pixel to be processed from the area outside the viewing angle in the image including the area outside the viewing angle. In steps S1404 to S1408, the entire area outside the viewing angle and painted black will eventually be processed, so the control unit 501 can select pixels to be processed in order starting from the upper left corner of the area outside the viewing angle.

[0261] In step S1405, the control unit 501 determines whether the radio button 1501 (setting for blacking out pixels corresponding to pixels outside the viewing angle in the image relative to the eye) is selected in step S1301. If it is determined that the radio button 1501 is selected, the sequence moves to step S1406. If not (if it is determined that the radio button 1502 is selected), the sequence moves to step S1407.

[0262] In step S1406, the control unit 501 sets the pixel at the same position in the image relative to the eye corresponding to the pixel to be processed to black. The process of setting the pixel to black can be completed by setting a zero value in the RGB value, or by copying the pixel value from the picture outside the viewing angle of the pixel. The result of this process is Fig. 16B As a result of processing the left eye image 1609 that does not include the area outside the viewing angle, a black area 1613 is added to the left eye image 1611. By adding the black area to the left eye image 1611, for the area corresponding to the area outside the viewing angle in the right eye image 1612, the left and right display the same image (black).

[0263] In step S1407, the control unit 501 copies the pixel value at the same position corresponding to the pixel to be processed in the image relative to the eye to the pixel to be processed. The result of this processing is Fig. 16CAs shown in FIG. 16A , the pixel value is copied from region 1616 to region 1617, which eliminates the black area outside the viewing angle as shown by arrow 1618. By performing this process, for the area corresponding to the area outside the viewing angle in the right eye image 1612, the same image (the image of region 1616) is displayed on the left and right.

[0264] In step S1408, the control unit 501 determines whether all pixels in the area outside the angle of view have been processed. If all pixels in the area outside the angle of view have been processed, the processing in this flowchart ends. The image processed in this manner is used as the image to be processed in Fig.13 If it is determined that all pixels in the area outside the field of view have not been processed, the sequence moves to step S1404. Then, by repeating the processing loop from step S1404 to step S1408, all pixels in the area outside the field of view are processed.

[0265] As described above, according to the fifth embodiment, when there is a first specific area in the first display image (one of the left eye image 1609 and the right eye image 1610) that overlaps with a corresponding area in the equidistant cylindrical projection image corresponding to a missing area in the circular fisheye image, the PC 500 corrects the first display image or the second display image (the other of the left eye image 1609 and the right eye image 1610) so that the first specific area and the second specific area located at a position corresponding to the first specific area of ​​the second display image include the same image. This correction is performed before the first display image or the second display image is output (before the first display image or the second display image is displayed). This makes it possible to improve the display quality of three-dimensional virtual reality.

[0266] Although the present embodiment describes an example of a case where the PC 500 receives an image file, the configuration may allow the PC 500 to receive an image as a video signal. The image may be a moving image as opposed to a still image.

[0267] Sixth embodiment

[0268] The sixth embodiment will describe a configuration of cutting out (extracting) one circular fisheye image from an image including two circular fisheye images having parallax.

[0269] Fig.19An application screen on the PC 500 is shown. 1900 denotes the entire application screen. Two circular fisheye images (one on the left and one on the right) recorded in a single image file are displayed on the screen. 1901 denotes a circular fisheye image captured by one of the left and right optical systems of the VR180 lens. 1902 denotes a circular fisheye image opposite to the circular fisheye image 1901 captured by one of the left and right optical systems of the VR180 lens. 1903 denotes a button for saving the displayed left circular fisheye image 1901. 1904 denotes a button for saving the displayed right circular fisheye image 1902. 1905 denotes a check box for selecting whether to perform additional processing (described later) in the case of a lack of viewing angle in the circular fisheye image.

[0270] Will refer to Fig. 20 Describes two circular fisheye images arranged left and right before saving and the circular fisheye image after saving according to this embodiment. 2001 indicates two circular fisheye images arranged left and right after reading the file. 2002 is an arrow indicating that the left circular fisheye image is saved. 2003 indicates the circular fisheye image after saving.

[0271] Will refer to Fig.17A , Fig. 17B and Fig.18 Describes the process of selecting a given circular fisheye image from two circular fisheye images arranged left and right and cropping and saving the circular fisheye image. When the user sets the recording medium 227 in the external storage device 504 and selects an image file by operating the operation unit 505, Fig.17A and Fig. 17B The processing of the flowchart in is started, where an image has already been recorded by the camera 100 in the recording medium 227 .

[0272] Fig.17A and Fig. 17B The processing of steps S701 to S711 in Figure 7 The processing of steps S701 to S711 in is similar and will therefore not be described again.

[0273] In step S1712, the control unit 501 displays two circular fisheye images. The application screen for displaying the images is as follows: Fig.19 shown.

[0274] In step S1713, the control unit 501 determines whether an instruction to save the left circular fisheye image has been given. Fig.19 If it is determined that an instruction to save the left circular fisheye image has been made, the sequence moves to step S1715. If it is determined that an instruction to save the left circular fisheye image has not been made, the sequence moves to step S1714.

[0275] In step S1714, the control unit 501 determines whether an instruction to save the right circular fisheye image has been given. Fig.19 The control unit 501 makes a save instruction by pressing the button 1904 in step S1714. If it is determined that an instruction to save the right circular fisheye image has been made, the sequence moves to step S1715. If it is determined that an instruction to save the right circular fisheye image has not been made, the sequence moves to step S1713. The control unit 501 continues to display the two circular fisheye images until a save instruction is received in step S1713 or step S1714.

[0276] In step S1715, the control unit 501 saves the selected circular fisheye image. Fig.18 The processing of step S1715 is described in detail.

[0277] If the left circular fisheye image is saved, the saved result is Fig. 20 The circular fisheye image 2003 is shown. Even though an example of saving the left circular fisheye image is described here, the right circular fisheye image can also be saved through a similar sequence.

[0278] Fig.18 is a flowchart showing the processing performed in step S1715 in detail.

[0279] In step S1801, the control unit 501 determines whether there is a missing area in the left and right circular fisheye images. The following method can be used as a determination method, in which, based on the center coordinates and radius of the two circular fisheye images calculated in step S1705, step S1709, or step S1711, it is determined whether the amount extending from the center of the left circular fisheye image to the left equal to the radius is within the image range or outside the image range. Although it is described here that the determination is based on the configuration of the left circular fisheye image, it is also possible to adopt a configuration in which the determination is based on the right circular fisheye image. In addition, it is also possible to adopt a configuration in which the determination is made using both the left and right circular fisheye images. If it is determined that there is a missing area in the left and right circular fisheye images, the sequence moves to step S1802. The processing performed in this case will be described later. If it is determined that there is no missing area in the left and right circular fisheye images, the sequence moves to step S1811.

[0280] In step S1811, the control unit 501 calculates the necessary saving range of the circular fisheye image to be saved based on the center coordinates and radius of the circular fisheye image determined in step S1705, step S1709, or step S1711. The saving range is determined as a range in which the center of the optical axis of the circular fisheye image is at the center of the image and the entire circular fisheye image is covered after the image is saved. The upper boundary, bottom boundary, left boundary, and right boundary (black area) of the circular fisheye image may be completely missing, or may be saved to provide a boundary of a given size.

[0281] In step S1812, the control unit 501 saves the image in the storage range determined in step S1811. Fig. 20 A circular fisheye image 2003 is shown as the saved result.

[0282] Note that a configuration in which the center coordinates are specified by a user operation may be adopted instead of using image processing to determine the center coordinates of the two circular fisheye images.

[0283] Through the above processing, when there is no missing area in the left and right circular fisheye images, a single circular fisheye image can be cropped. If the circular fisheye image is a moving image rather than a still image, the control unit 501 can select a specific frame from the moving image and crop a circular fisheye image from the selected frame.

[0284] When the sequence moves from step S1801 to step S1802, the control unit 501 identifies the missing area in the circular fisheye image to be saved. Here, the case where the left circular fisheye image is to be saved will be described as an example. One method of this identification is to determine whether the area is a valid area in the image or outside the image range while extending to the left by an amount equivalent to the radius from the center of the calculated circular fisheye image. The difference between the effective distance within the image range in the left direction and the radius can be used to identify the missing area. For example, Fig.21A The missing region 2102 in is identified as a missing region.

[0285] In step S1803, the control unit 501 identifies an area of ​​the relatively circular fisheye image corresponding to the area identified in step S1802. This identification can be performed by, for example, superimposing two circular fisheye images at the center of the optical axis. Instead of superimposing, this identification can be performed by calculating the distance from the center of the optical axis of the circular fisheye image. Fig.21A The valid part of the image within the frame 2103 in FIG. 2 is identified as the corresponding part.

[0286] In step S1804, the control unit 501 determines whether to provide overlap when synthesizing the missing area of ​​the fisheye image. Fig.19The judgment is made by checking the check box 1905 in . If it is judged that an overlap is to be provided, the sequence moves to step S1806. The processing performed in this case will be described later. If it is judged that an overlap is not to be provided, the sequence moves to step S1805.

[0287] In step S1805, the control unit 501 converts the corresponding area ( Fig.21A The area in box 2103 in the Fig.21A The missing region 2102 in the replication result is Fig.21A Box 2105 in is represented.

[0288] In step S1809, the control unit 501 calculates a necessary storage range based on the optical axis center coordinates and radius of the circular fisheye image after the synthesis process. The method for calculating the storage range is similar to that of step S1811.

[0289] In step S1810, the control unit 501 saves the image in the storage range calculated in step S1809. Fig.21A A circular fisheye image 2106 is shown as the saved result.

[0290] This is described in more detail Fig.21A 2101 indicates the missing image on the left side of the left circular fisheye image to be saved. 2102 indicates the missing area of ​​the circular fisheye image identified when the left side is to be saved. 2103 is a frame indicating the portion of the right image corresponding to the missing area 2102. 2104 is an arrow indicating the synthesis destination. Fig.21A In the example of , synthesis is performed by copying a portion of the right image to the missing area of ​​the left image. 2105 is a box indicating the result of copying a portion of the right image to the missing area of ​​the left image. 2106 indicates a left circular fisheye image obtained by saving the synthesis result. Although an example of saving the left circular fisheye image is described here, the right circular fisheye image can also be saved by a similar sequence.

[0291] Through the above processing, when there are missing areas in the left and right circular fisheye images, one circular fisheye image can be cropped by filling the missing area in one circular fisheye image with the corresponding area in the other circular fisheye image.

[0292] When the sequence moves from step S1804 to step S1806, the control unit 501 identifies the overlapping area to be overlapped in the left and right images. This embodiment assumes that half of the circular fisheye image is identified as the overlapping area. Fig.21B In the example of , the area of ​​the right circular fisheye image corresponding to frame 2108 is the overlapping area.

[0293] In step S1807, the control unit 501 determines the synthesis ratio according to the synthesis position when the left and right images are superimposed. A method in which the synthesis ratio of the left image and the right image is changed from the missing area of ​​the circular fisheye image toward the optical axis center of the circular fisheye image can be used as a determination method. As an example of changing the synthesis ratio, for the missing area of ​​the circular fisheye image in the left image, the synthesis ratio of the right image is set to 100%. The synthesis ratio is then determined so that the usage ratio of the right image decreases toward the center of the circular fisheye image, so that when the center of the circular fisheye image is reached, the usage ratio of the right image is 0% and the usage ratio of the left image is 100%.

[0294] In step S1808, the control unit 501 combines the right circular fisheye image with the left circular fisheye image according to the combination ratio obtained in step S1807. Fig.21B Box 2112 in is represented.

[0295] The processing in steps S1809 and S1810 after step S1808 is similar to the processing in steps S1809 and S1810 after step S1805. Fig.21B The circular fisheye image 2113 is shown as the saved result.

[0296] This is described in more detail Fig.21B . 2106 indicates the missing image on the left side of the left circular fisheye image to be saved. 2107 indicates the missing area of ​​the circular fisheye image identified when the left side is to be saved. 2108 indicates an area where an overlap is provided to reduce the discomfort of the synthesis result by taking into account the parallax between the left and right images. 2109 is a frame showing the right image area overlapping with the left image. 2110 is an arrow indicating the synthesis destination of the overlapping area of ​​the left image. In the present embodiment, synthesis processing is performed with the corresponding area of ​​the right image. 2111 is an arrow indicating the synthesis destination of the overlapping area of ​​the right image. In the present embodiment, the right image is copied to the missing area of ​​the left circular fisheye image. In the overlapping area, synthesis processing is performed while adjusting the synthesis ratio between the left image and the right image according to the synthesis position. 2112 indicates the circular fisheye image generated from the synthesis processing. 2113 indicates the left circular fisheye image obtained by saving the synthesis result. Although an example of saving the left circular fisheye image is described here, the right circular fisheye image can also be saved by a similar sequence.

[0297] Although the compositing process described herein involves reducing the discomfort of the compositing result by adjusting the compositing ratio, the process may be combined with other image processing such as applying blur.

[0298] Through the above processing, when there are missing areas in the left and right circular fisheye images, one circular fisheye image can be cropped by overlapping the overlapping area including the missing area in one circular fisheye image with the corresponding area in the other circular fisheye image.

[0299] As described above, according to the sixth embodiment, the PC 500 cuts out (extracts) a circular fisheye image from an image including two circular fisheye images having parallax. If the circular fisheye image to be extracted is accompanied by a missing area, the PC 500 sets the pixel value of the area of ​​the other circular fisheye image located at a position corresponding to the missing area in the missing area of ​​the circular fisheye image to be extracted. This makes it possible to extract a circular fisheye image including the pixel values ​​set in the missing area.

[0300] Other embodiments

[0301] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0302] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: an obtaining unit configured to obtain a first circular fisheye image that is not recorded in a complete form, wherein the first circular fisheye image is accompanied by a first missing area where no pixel value exists within a first circular area corresponding to the complete form of the first circular fisheye image; and a generating unit configured to generate a first equidistant cylindrical projection image corresponding to the first circular area by performing a first equidistant cylindrical transformation process based on the first circular fisheye image, The generating unit generates the first equidistant cylindrical projection image so that a first corresponding area of ​​the first equidistant cylindrical projection image corresponding to the first missing area within the first circular area has a pixel value.

2. The image processing device according to claim 1, in, The generation unit sets a predetermined pixel value in a first missing area of ​​the first circular fisheye image before the first equidistant cylindrical transform process.

3. The image processing device according to claim 1, in, The generation unit identifies the coordinates of the first circular fisheye image corresponding to the pixels to be processed of the first equidistant cylindrical projection image in the first equidistant cylindrical transformation process, and sets a predetermined pixel value for the pixels to be processed of the first equidistant cylindrical projection image if no pixel value exists at the identified coordinates.

4. The image processing device according to claim 2 or 3, in, The obtaining unit obtains a second circular fisheye image having a parallax with respect to the first circular fisheye image and accompanied by a second missing region having no pixel value. The generating unit generates a second equidistant cylindrical projection image by performing a second equidistant cylindrical transformation process based on the second circular fisheye image, so that in the second equidistant cylindrical projection image, a second corresponding area corresponding to the second missing area has the predetermined pixel value, and The generation unit sets the predetermined pixel value in a region of the first equidistant cylindrical projection image located at a position corresponding to a second corresponding region of the second equidistant cylindrical projection image.

5. The image processing device according to claim 1, in, The obtaining unit obtains a second circular fisheye image having a parallax relative to the first circular fisheye image, and The generation unit sets pixel values ​​of a region of the second circular fisheye image located at a position corresponding to the first missing region of the first circular fisheye image in a first corresponding region of the first equidistant cylindrical projection image.

6. The image processing device according to claim 5, in, The second circular fisheye image is accompanied by a second missing region where no pixel values ​​exist, and When an area of ​​the second circular fisheye image located at a position corresponding to a specific area in the first missing area of ​​the first circular fisheye image is included in the second missing area, the generation unit sets a predetermined pixel value for an area corresponding to the specific area in the first corresponding area of ​​the first equidistant cylindrical projection image.

7. The image processing device according to any one of claims 2, 3 and 6, in, The predetermined pixel value is a value representing black.

8. An image processing device, comprising: an obtaining unit configured to obtain a first equidistant cylindrical projection image and a second equidistant cylindrical projection image, wherein the first equidistant cylindrical projection image is generated by a first equidistant cylindrical transformation process based on a first circular fisheye image that is not recorded in a complete form, and the second equidistant cylindrical projection image is generated by a second equidistant cylindrical transformation process based on a second circular fisheye image having a parallax with respect to the first circular fisheye image, wherein: The first circular fisheye image is accompanied by a missing region in which no pixel value exists within a circular region corresponding to the complete form of the first circular fisheye image; The first equidistant cylindrical projection image corresponds to the circular area; and Predetermined pixel values ​​are set in corresponding areas of the first equidistant cylindrical projection image corresponding to the missing area within the circular area; a designation unit configured to designate a display position of a three-dimensional virtual reality based on the first equidistant cylindrical projection image and the second equidistant cylindrical projection image; an output unit configured to output a first display image corresponding to the display position in the first equidistant cylindrical projection image and a second display image corresponding to the display position in the second equidistant cylindrical projection image; and a correction unit configured to correct the first display image or the second display image before outputting the first display image or the second display image when there is a first specific area overlapping the corresponding area in the first display image so that the first specific area and a second specific area located at a position of the second display image corresponding to the first specific area include the same image, The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, wherein the first subject image is formed via a first fisheye lens and the second subject image is formed via a second fisheye lens.

9. The image processing device according to claim 8, in, The correction unit sets the predetermined pixel value in a second specific area of ​​the second display image.

10. The image processing device according to claim 8, in, The correction unit replaces an image in a first specific area of ​​the first display image with an image in a second specific area of ​​the second display image.

11. The image processing device according to claim 8, in, The correction unit replaces the first display image with the second display image.

12. The image processing device according to any one of claims 8 to 11, in, The predetermined pixel value is a value representing black.

13. An image processing device, comprising: an obtaining unit configured to obtain an image including a first circular fisheye image that is not recorded in a complete form and a second circular fisheye image having a parallax with respect to the first circular fisheye image, wherein the first circular fisheye image is accompanied by a missing region in which no pixel value exists within a circular region corresponding to the complete form of the first circular fisheye image; a setting unit configured to set, in the missing region of the first circular fisheye image, a pixel value of a region of the second circular fisheye image located at a position corresponding to the missing region; and an extracting unit configured to extract the first circular fisheye image including the pixel values ​​set in the missing area from the image, The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, wherein the first subject image is formed via a first fisheye lens and the second subject image is formed via a second fisheye lens.

14. The image processing device according to claim 13, further comprising: A synthesis unit is configured to synthesize pixel values ​​of a first specific area located in the first circular fisheye image within a predetermined range from the missing area and a second specific area of ​​the second circular fisheye image located at a position corresponding to the first specific area.

15. The image processing device according to claim 14, in, The synthesis unit controls a synthesis ratio of the pixel value of the second specific area with respect to the pixel value of the first specific area to increase as the pixel approaches the missing area.

16. An image processing method, which is performed by an image processing device, the image processing method comprising: obtaining a first circular fisheye image that is not recorded in a complete form, wherein the first circular fisheye image is accompanied by a first missing region in which no pixel value exists within a first circular region corresponding to the complete form of the first circular fisheye image; and generating a first equidistant cylindrical projection image corresponding to the first circular area by performing a first equidistant cylindrical transformation process based on the first circular fisheye image, The first equidistant cylindrical projection image is generated so that a first corresponding area of ​​the first equidistant cylindrical projection image corresponding to the first missing area within the first circular area has a pixel value.

17. An image processing method, which is performed by an image processing device, the image processing method comprising: A first equidistant cylindrical projection image and a second equidistant cylindrical projection image are obtained, wherein the first equidistant cylindrical projection image is generated by a first equidistant cylindrical transformation process based on a first circular fisheye image that is not recorded in a complete form, and the second equidistant cylindrical projection image is generated by a second equidistant cylindrical transformation process based on a second circular fisheye image having a parallax relative to the first circular fisheye image, wherein: The first circular fisheye image is accompanied by a missing region in which no pixel value exists within a circular region corresponding to the complete form of the first circular fisheye image; The first equidistant cylindrical projection image corresponds to the circular area; and Predetermined pixel values ​​are set in corresponding areas of the first equidistant cylindrical projection image corresponding to the missing area within the circular area; specifying a display position of a three-dimensional virtual reality based on the first equidistant cylindrical projection image and the second equidistant cylindrical projection image; outputting a first display image corresponding to the display position in the first equidistant cylindrical projection image and a second display image corresponding to the display position in the second equidistant cylindrical projection image; and when there is a first specific area overlapping the corresponding area in the first display image, correcting the first display image or the second display image before outputting the first display image or the second display image so that the first specific area and a second specific area located at a position of the second display image corresponding to the first specific area include the same image, The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, wherein the first subject image is formed via a first fisheye lens and the second subject image is formed via a second fisheye lens.

18. An image processing method, which is performed by an image processing device, the image processing method comprising: obtaining an image including a first circular fisheye image that is not recorded in a complete form and a second circular fisheye image having a parallax with respect to the first circular fisheye image, wherein the first circular fisheye image is accompanied by a missing region in which no pixel value exists within a circular region corresponding to the complete form of the first circular fisheye image; Setting, in the missing region of the first circular fisheye image, a pixel value of a region of the second circular fisheye image located at a position corresponding to the missing region; and extracting the first circular fisheye image including the pixel values ​​set in the missing area from the image, The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, wherein the first subject image is formed via a first fisheye lens and the second subject image is formed via a second fisheye lens.

19. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method, the image processing method comprising: obtaining a first circular fisheye image that is not recorded in a complete form, wherein the first circular fisheye image is accompanied by a first missing region in which no pixel value exists within a first circular region corresponding to the complete form of the first circular fisheye image; and generating a first equidistant cylindrical projection image corresponding to the first circular area by performing a first equidistant cylindrical transformation process based on the first circular fisheye image, The first equidistant cylindrical projection image is generated so that a first corresponding area of ​​the first equidistant cylindrical projection image corresponding to the first missing area within the first circular area has a pixel value.

20. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method, the image processing method comprising: A first equidistant cylindrical projection image and a second equidistant cylindrical projection image are obtained, wherein the first equidistant cylindrical projection image is generated by a first equidistant cylindrical transformation process based on a first circular fisheye image that is not recorded in a complete form, and the second equidistant cylindrical projection image is generated by a second equidistant cylindrical transformation process based on a second circular fisheye image having a parallax relative to the first circular fisheye image, wherein: The first circular fisheye image is accompanied by a missing region in which no pixel value exists within a circular region corresponding to the complete form of the first circular fisheye image; The first equidistant cylindrical projection image corresponds to the circular area; and Predetermined pixel values ​​are set in corresponding areas of the first equidistant cylindrical projection image corresponding to the missing area within the circular area; specifying a display position of a three-dimensional virtual reality based on the first equidistant cylindrical projection image and the second equidistant cylindrical projection image; outputting a first display image corresponding to the display position in the first equidistant cylindrical projection image and a second display image corresponding to the display position in the second equidistant cylindrical projection image; and when there is a first specific area overlapping the corresponding area in the first display image, correcting the first display image or the second display image before outputting the first display image or the second display image so that the first specific area and a second specific area located at a position of the second display image corresponding to the first specific area include the same image, The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, wherein the first subject image is formed via a first fisheye lens and the second subject image is formed via a second fisheye lens.

21. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method, the image processing method comprising: obtaining an image including a first circular fisheye image that is not recorded in a complete form and a second circular fisheye image having a parallax with respect to the first circular fisheye image, wherein the first circular fisheye image is accompanied by a missing region in which no pixel value exists within a circular region corresponding to the complete form of the first circular fisheye image; Setting, in the missing region of the first circular fisheye image, a pixel value of a region of the second circular fisheye image located at a position corresponding to the missing region; and extracting the first circular fisheye image including the pixel values ​​set in the missing area from the image, The first circular fisheye image and the second circular fisheye image are captured by simultaneously forming a first subject image and a second subject image on the same image sensor, wherein the first subject image is formed via a first fisheye lens and the second subject image is formed via a second fisheye lens.

Images