Image processing apparatus and image processing method

Abstract

To provide an image processing device that, even when an imaging apparatus fails to recognize a binocular lens unit, can recognize the type of an image recorded by the imaging apparatus to appropriately process the image.SOLUTION: An image processing device determines whether an image represented by image data includes two circular areas. When determining that the image includes two circular areas, the image processing device executes predetermined processing on the image data.SELECTED DRAWING: Figure 8

Description

Technical Field

[0003]

[0001] The present invention relates to an image processing apparatus and an image processing method.

Background Art

[0002] It is known to display a parallax image pair on a head-mounted display (HMD) or the like to provide stereoscopic vision. Further, Patent Document 1 describes an imaging apparatus capable of capturing a parallax image pair with a single unit using a lens having one lens mount and two imaging optical systems (hereinafter referred to as a two-eye lens).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When forming a pair of images on a single imaging element using a two-eye lens having two imaging optical systems (right-eye optical system and left-eye optical system) whose optical axes are separated in the horizontal direction, the positional relationship between the imaging optical system and the images is reversed between the left and right. That is, an imaging image is obtained in which the image (right image) formed by the right-eye optical system is on the left side and the image (left image) formed by the left-eye optical system is on the right side.

[0005] In order to correctly use the right image and the left image, it is necessary to correctly grasp the relationship between the right image and the left image in the imaging image. For this purpose, the imaging apparatus must be able to recognize that a two-eye lens unit that forms an image in which the positional relationship between the right image and the left image is reversed is attached.

[0006] For example, an imaging device might be able to recognize, based on information about the lens unit (lens information) obtained through communication with the lens unit, that the mounted lens unit is a binocular lens unit that forms an image with the relative positions of the right and left images reversed. However, it is not always possible to obtain lens information from all lens units that can be mounted on the imaging device. Furthermore, the lens information may not contain information that would allow the device to determine whether or not the mounted lens unit is a binocular lens unit that forms an image with the relative positions of the right and left images reversed.

[0007] This invention has been made in view of the problems of the prior art. One of the objects of this invention is to provide an image processing device and an image processing method that can recognize the type of image recorded by an imaging device and process it appropriately, even when the imaging device is unable to recognize a twin-lens unit. [Means for solving the problem]

[0008] The purpose described above is to obtain image data. and metadata of image data A means of obtaining, When metadata does not contain information indicating that the image data represents a specific image, The image represented by the image data It is a specific image. A determination means for determining whether or not the image is It is a specific image. If it is determined that, processing means to perform predetermined processing on the image, A particular image has two circular regions containing the subject image, and the region outside of these two circular regions does not contain the subject image. This is achieved by an image processing device characterized by the following. [Effects of the Invention]

[0009] Even if the imaging device fails to recognize the twin-lens unit, an image processing device and image processing method can be provided that can recognize the type of image recorded by the imaging device and process it appropriately. [Brief explanation of the drawing]

[0010] [Figure 1] A schematic diagram showing an example configuration of an image processing system according to the embodiment. [Figure 2] Perspective view showing an example of the camera's appearance according to the embodiment. [Figure 3] Block diagram showing an example of the functional configuration of the camera according to the embodiment. [Figure 4] Cross-sectional view showing an example configuration of a two-lens unit. [Figure 5] Schematic diagram showing the positional relationship of the images formed by the single-lens unit and the double-lens unit. [Figure 6] Block diagram showing an example of the functional configuration of a PC according to this embodiment. [Figure 7] Flowchart of camera operation in the embodiment [Figure 8] Flowchart of PC operation in the embodiment [Figure 9] A schematic diagram showing an example configuration of an image processing system according to another embodiment. [Figure 10] Flowchart relating to camera operation in another embodiment [Figure 11] Flowchart relating to the operation of the PC in another embodiment [Figure 12] Schematic diagram of the operation of S703 in Figure 8. [Modes for carrying out the invention]

[0011] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0012] In the following embodiments, the present invention will be described with respect to the case of being implemented on a personal computer. However, the present invention can be implemented on any electronic device capable of image processing. Such electronic devices include digital cameras, tablet computers, media players, PDAs, mobile phones, smartphones, game machines, robots, drones, and drive recorders. These are examples, and the present invention can be implemented on other electronic devices as well.

[0013] ●(First Embodiment) FIG. 1 is a schematic diagram of an image processing system including a personal computer (PC) as an example of an image processing apparatus according to an embodiment of the present invention 500 and a digital camera 100 (hereinafter referred to as camera 100) that generates image data to be processed by PC 500.

[0014] Camera 100 can acquire data of a captured image (hereinafter referred to as a VR image) with a stereoscopic viewing angle of 180 degrees and a stereo angle of 180 degrees using a binocular lens unit 300. In the present embodiment, it is assumed that camera 100 is a lens-exchangeable type, but it may not be a lens-exchangeable type. In the present embodiment, the binocular lens unit 300 houses two imaging optical systems that form a circumferential fisheye image with a viewing angle of 180 degrees. Using the circumferential fisheye images with parallax, a stereoscopic virtual reality (VR) image can be reproduced. Details of the configuration of the binocular lens unit 300 will be described later.

[0015] PC 500 is a general personal computer and functions as the image processing apparatus of the present embodiment by executing application software. PC 500 can directly acquire the captured image data obtained by camera 100 performing imaging processing from camera 100 through wired or wireless communication 150 (FIG. 1(a)). Further, PC 500 may acquire the captured image data by reading the captured image data recorded on a removable medium 160 which is, for example, a semiconductor memory card, by camera 100 (FIG. 1(b)).

[0016] Next, we will describe camera 100. Figure 2 is a perspective view showing an example of the camera 100's appearance. Figure 2(a) is a perspective view of camera 100 from the front diagonal upward direction, and Figure 2(b) is a perspective view of camera 100 from the rear diagonal upward direction.

[0017] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface. The shutter button 101 is an operation for preparing to shoot or giving a shooting command. The power switch 102 is an operation for switching the power of camera 100 on and off. The mode selector switch 103 is an operation for switching between various modes. The main electronic dial 104 is a rotary operation for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation for moving the selection frame (cursor) and advancing images. The video button 106 is an operation for giving a command to start and stop video recording. The external viewfinder display 107 displays various settings such as shutter speed and aperture.

[0018] The camera 100 has a display unit 108, a touch panel 109, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back. The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108.

[0019] The directional keys 110 are an operation unit consisting of keys that can be pressed in the up, down, left, and right directions (4-way keys). Operations can be performed according to the position where the directional keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed when confirming a selection item. The AE lock button 112 is an operation unit that is pressed when fixing the exposure state in shooting standby mode. The zoom button 113 is an operation unit that switches the zoom mode on and off in the live view display (LV display) of the shooting mode. When the zoom mode is on, the live view image (LV image) can be enlarged or reduced by operating the main electronic dial 104. The zoom button 113 is also used in playback mode to enlarge the playback image or increase the magnification ratio.

[0020] The playback button 114 is an operation unit for switching between shooting mode and playback mode. Pressing the playback button 114 in shooting mode switches to playback mode, and the latest image recorded on the recording medium 227 (described later) can be displayed on the display unit 108. The menu button 115 is an operation unit that is pressed to display a menu screen on the display unit 108 that allows for various settings. The user can make various settings of the camera 100 by operating the menu screen displayed on the display unit 108 using the directional keys 110 and the SET button 111. Alternatively, the menu screen may be operated using the touch panel 109 instead of, or in combination with, the buttons.

[0021] The eyepiece section 116 is a window for looking through the eyepiece viewfinder (a type of viewfinder) 117. The user can view the image displayed on the internal EVF (Electronic View Finder) 217, which will be described later, through the eyepiece section 116. The eyepiece detection unit 118 is a sensor that detects whether or not an object is close to the eyepiece section 116.

[0022] The touch bar 119 is a line-shaped touch operation area (line touch sensor) capable of accepting touch operations. The touch bar 119 is positioned so that it can be touched by the right thumb when the grip section 120 is held with the right hand (with the little finger, ring finger, and middle finger) so that the shutter button 101 can be pressed with the right index finger. In other words, the touch bar 119 can be operated when looking through the eyepiece viewfinder 117 through the eyepiece section 116 and holding the camera ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (an operation where the touch position is touched and then released within a predetermined period without moving the touch position), left and right slide operations (an operation where the touch position is moved while touching), etc. The touch bar 119 is a different operation area from the touch panel 109 and does not have a display function. In this embodiment, the touch bar 119 functions as a multifunction bar (M-Fn bar).

[0023] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. When the camera 100 is held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand.

[0024] The thumb rest section 121 (thumb waiting position) is a grip section located on the back of the camera 100, in a place where the thumb of the right hand holding the grip section 120 can be easily rested when no controls are being operated. The thumb rest section 121 is made of rubber material or the like to enhance the holding force (grip). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 227 and the slot for storing the recording medium 227, which will be described later, by closing the slot. The communication terminal 124 is a terminal for the camera 100 to communicate with the lens unit 200, which will be described later and can be attached to and detached.

[0025] <Internal configuration of Camera 100> Figure 3 is a block diagram showing an example of the internal configuration (functional configuration) of a camera system in which a replaceable lens unit 200 is attached to the camera 100. In Figure 3, the components shown in Figure 2 are denoted by the same reference numerals as in Figure 2. Explanations of components already described in Figure 2 will be omitted as appropriate.

[0026] First, let me explain the lens unit 200. The lens unit 200 is an example of a detachable interchangeable lens for the camera 100. The lens unit 200 is a typical single-lens reflex lens (a lens with one optical axis). The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, etc.

[0027] The aperture 201 is configured to have an adjustable opening diameter. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the opening diameter of the aperture 201. The AF drive circuit 204 drives the focus lens included in the lens 202 to adjust the distance at which the lens unit 200 focuses.

[0028] The lens system control circuit 205 includes, for example, a CPU, ROM, and RAM. The CPU executes a program stored in the ROM, thereby controlling the operation of each part of the lens unit 200. The lens unit 200 and the camera 100 are electrically connected via communication terminals 206 and 124, and the lens system control circuit 205 and the system control unit 50 of the camera 100 can communicate with each other. Based on instructions from the system control unit 50, the lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, and the like.

[0029] Next, I will explain camera 100. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0030] The shutter 210 is a focal-plane shutter that operates based on instructions from the system control unit 50 and controls the exposure time of the imaging unit 211. The imaging unit 211 is an image sensor composed of a CCD or CMOS element that converts an optical image into an electrical signal. In this embodiment, the imaging unit 211 is an image sensor that supports image plane phase-difference autofocus (image plane phase-difference AF). Specifically, the imaging unit 211 is capable of outputting a focus detection signal pair to realize phase-difference autofocus.

[0031] The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal (image data). The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing, and color conversion) on the data input through the A / D converter 212 or the memory control unit 213. The image processing unit 214 also performs predetermined calculations using the captured image data to calculate evaluation values ​​used for AF and AE. Based on the obtained calculation results, the system control unit 50 performs exposure control and focus detection control. The image processing unit 214 also calculates the defocus amount based on the focus detection signal pair obtained from the imaging unit 211 as one of the evaluation values. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data and performs AWB (auto white balance) processing on the image data based on the obtained calculation results.

[0032] Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data output by the A / D converter 212 and image data generated by the image processing unit 214. The image data generated by the image processing unit 214 includes display image data for display on the display unit 108 and EVF 217, and recording image data for recording on the recording medium 227. The memory 215 has sufficient storage capacity to store a predetermined number of still image data, a predetermined amount of moving image data, and audio data. In addition, a portion of the memory 215 is used as video memory for the display unit 108.

[0033] The D / A converter 216 converts the image data stored in the memory 215 into an analog signal suitable for display on the display unit 108 or EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or EVF 217 via the D / A converter 216. The display unit 108 or EVF 217 displays according to the analog signal from the D / A converter 216. The display unit 108 or EVF 217 is, for example, a display such as an LCD or an organic EL.

[0034] While the imaging unit 211 is shooting video, the image data stored in the memory 215 via the A / D converter 212 is converted into an analog signal by the D / A converter 216 and sequentially transferred to the display unit 108 and EVF 217 for display. This enables live view display on the display unit 108 and EVF 217.

[0035] The system control unit 50 is a control unit consisting of at least one processor (CPU) and / or at least one circuit. That is, the system control unit 50 may be a processor (CPU), a circuit, or a combination of a processor and a circuit. For example, if the system control unit 50 has a processor (CPU), the system control unit 50 controls the entire camera 100 by reading a program stored in the non-volatile memory 219 into the system memory 218 and executing it with the processor. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0036] The camera 100 also includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118. System memory 218 may be RAM, for example. System memory 218 stores constants and variables for the operation of the system control unit 50, programs read from non-volatile memory 219, and the like. The non-volatile memory 219 may be, for example, an electrically erasable and recordable EEPROM. Constants for the operation of the system control unit 50, programs, and the like are stored in the non-volatile memory 219.

[0037] The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives image signals and audio signals to and from external devices connected by wireless or wired cables. The communication unit 221 can communicate with external devices compliant with wireless LAN (Local Area Network) and devices on the internet. In addition, the communication unit 221 can communicate with external devices compliant with Bluetooth®. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive image data and other various information from external devices.

[0038] The attitude detection unit 222 outputs a signal representing the attitude of the camera 100 relative to the direction of gravity. Based on the signal output by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the signal output by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image before recording. The attitude detection unit 222 can use, for example, an acceleration sensor or a gyroscope. Based on the output signal of the attitude detection unit 222, the system control unit 50 can also detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.).

[0039] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece section 116 of the eyepiece viewfinder 117, which incorporates the EVF 217. The eyepiece detection unit 118 can use, for example, an infrared proximity sensor. When an object approaches, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The presence or absence of an object approaching the eyepiece section 116 can be determined by the amount of infrared light received.

[0040] The system control unit 50 switches the display (on / off) state of the display unit 108 and the EVF 217 depending on whether or not a nearby object is detected by the eyepiece detection unit 118. Specifically, at least in the shooting standby state and when the display destination switching setting is set to automatic switching, if no nearby object is detected, the display unit 108 is turned on and the EVF 217 is turned off. If a nearby object is detected, the EVF 217 is turned on and the display unit 108 is turned off. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as an eyepiece may be used.

[0041] The camera 100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium interface 226, an operation unit 228, a video signal output interface 240, and the like.

[0042] The external viewfinder display unit 107 displays various settings of the camera 100, such as shutter speed and aperture, via the external viewfinder display drive circuit 223. The power control unit 224 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are powered, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, supplying the necessary voltage to each part, including the recording medium 227, for the required period. The power supply unit 225 includes primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li batteries, and an AC adapter. The recording medium I / F 226 is an interface with the recording medium 227, such as a memory card or hard disk.

[0043] The recording medium 227 is a memory card or the like for recording captured images, and is composed of semiconductor memory, magnetic disk, or the like. The recording medium 227 may be removable or built-in. The video signal output I / F 240 is an interface for outputting image signals from the camera 100 to an external device. The video signal output I / F 240 has one or more interfaces that conform to a standard. There are no particular restrictions on the standard, but for example, it may be an interface that conforms to the HDMI® standard. The camera 100 outputs, for example, video data being captured to an external device (video signal receiver 241) connected to the video signal output I / F 240. In Figure 1(a), PC500 corresponds to the video signal receiver 241.

[0044] The operation unit 228 is an input unit that receives user input (user operation) and is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode selector switch 103, a touch panel 109, and other operating components 229. Other operating components 229 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, a touch bar 119, and the like.

[0045] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 turns on during operation of the shutter button 101, so-called half-press, and generates a first shutter switch signal SW1. The system control unit 50 interprets the first shutter switch signal SW1 as a shooting preparation instruction and starts the shooting preparation process. The shooting preparation process includes AF processing, AE processing, AWB processing, flash pre-flash processing, etc.

[0046] The second shutter switch 231 turns on when the shutter button 101 is fully pressed, generating a second shutter switch signal SW2. The system control unit 50 interprets the second shutter switch signal SW2 as a still image capture instruction and starts the still image capture operation based on the exposure conditions determined by the AE processing. It then controls each unit to execute a series of shooting processes, from reading the signal from the imaging unit 211 to generating an image file containing the still image data obtained from the capture and writing it to the recording medium 227.

[0047] The mode switch 103 switches the operating mode of the system control unit 50 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. Modes included in the still image shooting mode include auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide shooting settings for different shooting scenes. The user can directly switch to any of the above shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to any of the displayed modes using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0048] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 can be mounted on top of the display surface of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. GUI stands for Graphical User Interface. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109. Either method is acceptable.

[0049] The system control unit 50 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). The finger or pen that was touching the touch panel 109 is lifted (released), meaning the touch action ends (hereinafter referred to as "Touch-Up"). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0050] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0051] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 50. Based on the notified information, the system control unit 50 determines what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation was performed. An operation in which a finger is touched on the touch panel 109 and then quickly moved a certain distance and then released is called a flick. In other words, a flick is an operation in which the finger is quickly traced across the touch panel 109 as if flicking it. If a touch move of a predetermined distance or more at a predetermined speed or faster is detected, and a touch-up is then detected, it is determined that a flick was performed (it can be determined that a flick followed a slide operation). Furthermore, touching multiple points (for example, two points) simultaneously (multitouch) to bring them closer together is called pinch-in, and touching them further apart is called pinch-out. Pinch-out and pinch-in are collectively referred to as pinch operations (or simply pinch).

[0052] <Configuration of the multi-lens unit> Figure 4 is a schematic diagram showing an example configuration of a twin-lens unit 300 as an example of a multi-lens unit. The twin-lens unit 300 is a VR180 lens unit that forms VR images compliant with the VR180 standard. In this specification, "multi-lens" refers to a lens unit configured in which multiple imaging optical systems are provided within a single lens mount (or lens barrel), and which has multiple optical axes. Figure 4 shows the twin-lens unit 300 mounted on the camera 100. Note that Figure 4 shows only a part of the configuration of the camera 100 shown in Figure 3.

[0053] The twin-lens unit 300 is a type of interchangeable lens that can be attached to the camera 100. The twin-lens unit 300 has two imaging optical systems 301L and 301R within a single lens barrel, and therefore has two optical axes.

[0054] Here, it is assumed that when the twin-lens unit 300 is mounted on the camera 100, the two imaging optical systems 301L and 301R are arranged so that their two optical axes are aligned on a horizontal line. The two imaging optical systems 301L and 301R have a field of view of approximately 180 degrees and can capture the area of ​​the front hemisphere. Specifically, the two imaging optical systems 301L and 301R can capture a field of view of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation / depression angle, pitch angle), respectively. The two imaging optical systems 301L and 301R form a pair of parallax images with parallax between the left and right eyes onto the imaging surface of the imaging unit 211. In the following description, the imaging optical system 301L will be referred to as the left-eye optical system 301L, and the imaging optical system 301R will be referred to as the right-eye optical system 301R.

[0055] The right eye optical system 301R and the left eye optical system 301L each have multiple lenses and a reflective mirror, etc. The multiple lenses include at least a focusing lens for adjusting the focusing distance. The two-lens unit 300 also has a lens system control circuit 303. The right eye optical system 301R is an example of a first optical system, and the left eye optical system 301L is an example of a second optical system. In the right eye optical system 301R and the left eye optical system 301L, the lenses 302R and 302L located on the subject side are oriented in the same direction, and their optical axes are approximately parallel.

[0056] Although not shown in Figure 4, the dual-lens unit 300 has a configuration similar to that of the AF drive circuit 204. In this case, it may have one or more AF drive circuits: one that drives the focus lenses of the right-eye optical system 301R and the left-eye optical system 301L in conjunction, and another that drives at least one of the focus lenses of the right-eye optical system 301R and the left-eye optical system 301L independently. The drive of the focus lenses is performed by the lens system control circuit 303 based on the control of the system control unit 50.

[0057] Furthermore, the twin-lens unit 300 includes an encoder that detects the amount and direction of rotation of the focus ring provided on the lens barrel. The lens system control circuit 303 provides a so-called by-wire manual focus function by controlling the AF drive circuit in accordance with the focus lens operation detected by the encoder. In this case, the twin-lens unit 300 may also have a switch that allows the user to switch between the focus lenses driven by the focus ring operation.

[0058] The two-lens unit 300 is a VR180 lens for capturing images in the VR180 format, a VR image format that enables binocular stereoscopic viewing, with the camera 100. The VR180 lens has a right-eye optical system 301R and a left-eye optical system 301L, each having a fisheye lens with a field of view of approximately 180 degrees. The right-eye optical system 301R and the left-eye optical system 301L only need to be able to acquire images that enable binocular VR display as VR180, and the field of view may be around 160 degrees. The VR180 lens can form the right image (first image) by the right-eye optical system 301R and the left image (second image) by the left-eye optical system 301L on the same imaging surface. Here, the imaging unit 211 of the camera 100 has one image sensor, and the two-lens unit 300 forms the right and left images on the imaging surface of that one image sensor. However, the camera 100 may have two image sensors arranged in parallel, and the twin-lens unit 300 may form a right image on the imaging surface of one image sensor and a left image on the imaging surface of the other image sensor.

[0059] The two-lens unit 300 includes a focus ring for adjusting the focus of the right eye optical system 301R and a focus ring for adjusting the focus of the left eye optical system 301L. Alternatively, it may include a focus ring for simultaneously adjusting the focus of both the right eye optical system 301R and the left eye optical system 301L, and a focus ring for adjusting the focus of either the right eye optical system 301R or the left eye optical system 301L. By operating these focus rings, the user can manually adjust the focusing distance of the right eye optical system 301R and the left eye optical system 301L. These focus rings may be provided individually, or in the case of a by-wire system, this may be achieved by switching the function of a single focus ring.

[0060] The twin-lens unit 300 is attached to the camera 100 via a mount, similar to the (single-lens) lens unit 200. The mount consists of a lens mount section 304 and a camera mount section 305. When the twin-lens unit 300 is attached to the camera 100, the communication terminal 124 of the camera 100 and the communication terminal 306 of the twin-lens unit 300 are electrically connected. This enables the system control section 50 of the camera 100 and the lens system control circuit 303 of the twin-lens unit 300 to communicate with each other. However, in this embodiment, the camera 100 cannot obtain information from the twin-lens unit 300 that would allow it to determine that the twin-lens unit 300 is a VR180 lens or that the positional relationship between the right and left images of the circular fisheye image is reversed. Information that would allow it to determine that the twin-lens unit 300 is a VR180 lens could be, for example, the coordinates of the circular fisheye center or the radius of the circular fisheye.

[0061] In this embodiment, the right image and the left image are formed on the imaging surface of the imaging unit 211, spaced apart in the left-right direction. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single image sensor. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. In this way, by attaching the dual-lens unit 300, a pair of disparity images (right image and left image) formed by the right-eye optical system 301R and the left-eye optical system 301L can be acquired in a single shot. Furthermore, by displaying the acquired right and left images as images for the right eye and left eye in VR, the user can observe a three-dimensional VR image, a so-called VR180 image, with a range of approximately 180 degrees.

[0062] Figure 5 is a schematic diagram of the subject image formed on the imaging unit 211 by the (single-lens) lens unit 200 and the twin-lens lens unit 300. The left side shows the (single-lens) lens unit 200, and the right side shows the twin-lens lens unit 300. Here, we assume that a scene with a subject (a person's face) 401 in the center is being photographed. Also, for convenience, the difference in the field of view of the imaging optical systems of each lens unit is ignored.

[0063] The subject image formed by the (single-lens) lens unit 200 on the imaging unit 211 is an inverted upright image (upright image with top, bottom, left, and right reversed). Therefore, when the captured image 402a is rotated 180 degrees around the center of the image, an image 403a is obtained in which the shooting scene and the top, bottom, left, and right orientation of the subject 401 are correct.

[0064] On the other hand, in the case of the two-lens unit 300, the left-eye optical system 301L and the right-eye optical system 301R each form an inverted upright image of the subject 401 on the imaging unit 211, resulting in an image 402b. Rotating the image 402b 180 degrees around the center of the image yields image 403b. As shown in image 403b, the format in which the circular fisheye image formed by the left-eye optical system 301L and the right-eye optical system 301R is recorded is sometimes called the mesh format, and the format in which the two images are arranged side by side is sometimes called the side-by-side format. In image 403b, the top, bottom, left, and right of the subject 401 are the same as in the shooting scene, but the subject image formed by the right-eye optical system 301R (right image) is on the left side, and the subject image formed by the left-eye optical system 301L (left image) is on the right side, reversing the positional relationship between the right and left images. Therefore, if the subject image on the right side of image 403b is used as the right image and the subject image on the left side as the left image, the correct display cannot be achieved.

[0065] If camera 100 can recognize that the captured image contains both a right and a left image, and that the relative positions of the right and left images are reversed, then by recording information about the right and left images in association with the data of the captured image, the right and left images can be correctly utilized from the captured image. However, if camera 100 cannot recognize that the captured image contains both a right and a left image, and that the relative positions of the right and left images are reversed, then it is not possible to record information about the right and left images in association with the data of the captured image. This embodiment makes it possible to correctly utilize the right and left images from the captured image even in such cases. Details will be described later.

[0066] Here, a VR image is an image that can be displayed in VR, as described later. VR images include omnidirectional images (spherical images) taken with an omnidirectional camera (spherical camera), and panoramic images that have a wider image range (effective image range) than the display range that can be displayed on the display unit at once. Furthermore, VR images may be either still images or videos. Videos may be pre-recorded videos or live images (images acquired from the camera in near real-time).

[0067] VR images have an image range (effective image range) of up to 360 degrees horizontally and vertically. VR images also include images with a wider field of view than that of a normal camera, or an image range wider than that that can be displayed on a display unit at once, even if the field of view is less than 360 degrees horizontally or vertically. Images captured by camera 100 using the aforementioned twin-lens unit 300 are a type of VR image. VR images can be displayed in VR, for example, by setting the display mode of a display device (a display device capable of displaying VR images) to "VR View". By displaying a VR image with a 360-degree field of view in VR and changing the orientation of the display device horizontally (horizontal rotation direction), the user can view a seamless, omnidirectional image in the horizontal direction.

[0068] Here, VR display (VR view) is a display mode that displays images of a predetermined range of the field of view captured in the VR image, according to the orientation of the display device. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by performing a transformation (a transformation that corrects distortion) that maps the VR image to a virtual sphere. VR display also includes "two-eye VR display (two-eye VR view)," which displays the VR image for the left eye and the VR image for the right eye side-by-side in the left and right regions by performing transformations that map them to virtual spheres.

[0069] Stereoscopic viewing is possible by using "two-eye VR display" with VR images for the left eye and VR images for the right eye that have parallax between them. In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), the image displayed will correspond to the user's field of view. For example, suppose a VR image is displayed with a field of view centered at 0 degrees horizontally (a specific direction, e.g., north) and 90 degrees vertically (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is then reversed (for example, changing the display surface from facing south to facing north), the display range of the same VR image will change to an image with a field of view centered at 180 degrees horizontally (the opposite direction, e.g., south) and 90 degrees vertically. In other words, if a user wearing an HMD turns their face from north to south (i.e., turns their back), the image displayed on the HMD will also change from a north image to a south image.

[0070] It should be noted that the VR image captured using the twin-lens unit 300 of this embodiment is a VR180 format image capturing a range of approximately 180 degrees in front, and there is no image of a range of approximately 180 degrees behind. If such a VR180 format image is displayed in VR and the orientation of the display device is changed to the side where there is no image, a blank area will be displayed, for example.

[0071] By displaying VR images in VR in this way, users can visually experience the sensation of being inside the VR image (in VR space). Note that the method of displaying VR images is not limited to changing the orientation of the display device. For example, the display range may be moved (scrolled) in response to user operations via a touch panel or directional buttons. Furthermore, during VR display (in the "VR View" display mode), in addition to changing the display range due to changes in orientation, the display range may also be changed in response to touch movements on the touch panel, drag operations with a mouse, or pressing directional buttons. Note that a configuration in which a display device such as a smartphone is attached to VR goggles (head-mounted adapter) is a type of HMD (Head-Mounted Display).

[0072] Next, we will explain the PC500. Figure 6 is a block diagram showing an example of the functional configuration of the PC500. The PC500 may be an electronic device that is commercially available as a personal computer.

[0073] The control unit 501 is a processor capable of executing programs, such as a CPU (Central Processing Unit). The ROM (Read Only Memory) 502 is, for example, an electrically rewritable non-volatile memory. The ROM 502 stores the program executed by the control unit 501, as well as various settings and parameters. The RAM (Random Access Memory) 503 is used to load the program executed by the control unit 501 and to temporarily store various data. Part of the RAM 503 may be used as buffer memory or video memory.

[0074] The external storage device 504 is a large-scale storage device built into the PC 500. Typically, the external storage device 504 is a hard disk drive (HDD) or a solid-state drive (SSD). In addition to the HDD or SSD, the external storage device 504 may also include storage devices that use removable media, such as a memory card reader. The external storage device 504 stores the OS (Operating System), application programs, user data, etc. A portion of the external storage device 504 may be used as a memory swap area. Image data files acquired by receiving from the camera 100 or reading from a memory card are stored in the HDD or SSD of the external storage device 504.

[0075] The operation unit 505 is a general term for user-operable input devices such as keyboards, mice, and touch panels. The display unit 506 is, for example, a liquid crystal display (LCD) and may have a touch panel. The display unit 506 is used by the OS and applications running on the PC 500 to display various information and data.

[0076] The communication unit 507 has, for example, a communication circuit for wireless communication with external devices. The communication unit 507 conforms to one or more wireless communication standards. Typical wireless communication standards include, but are not limited to, Bluetooth® and wireless LAN (IEEE 802.11x).

[0077] The External I / F508 has a communication circuit for wired communication with external devices, for example. The External I / F508 complies with one or more wired communication standards. Typical wired communication standards include, but are not limited to, USB (Universal Serial Bus), HDMI, Thunderbolt®, and Ethernet®. The system bus 509 connects each of the aforementioned blocks in a way that allows them to communicate with one another.

[0078] Next, the shooting operation of camera 100 will be explained using the flowchart in Figure 7. The shooting operation is realized by the system control unit 50 executing a program and performing the necessary controls. The shooting operation shown in Figure 7 starts when the user turns on the power of camera 100, for example by operating the power switch 102. After the system control unit 50 performs the startup process associated with power-on, it puts camera 100 into shooting standby mode. In shooting standby mode, camera 100 continues to shoot video and displays a live view on the EVF 217 or display unit 108.

[0079] In step S601, the system control unit 50 reads an image signal corresponding to one frame of a moving image from the imaging unit 211 and outputs it to the A / D converter 212. In step S602, the system control unit 50 instructs the image processing unit 214 to generate image data for live view display. The image processing unit 214 stores the image data for live view display in the video memory area of ​​memory 215. The D / A converter 216 performs D / A conversion on the image data stored in the video memory area of ​​memory 215 and displays it on at least one of the EVF 217 and the display unit 108. The image processing unit 214 also generates evaluation values ​​based on the image data and outputs them to the system control unit 50. The system control unit 50 can determine exposure conditions or adjust the focus distance of the lens unit based on the evaluation values.

[0080] In S603, the system control unit 50 determines whether or not a recording start command has been issued by the user. For example, the system control unit 50 can determine that a recording start command has been issued if it detects the second shutter switch signal SW2. If the system control unit 50 determines that a recording start command has been issued, it executes S604; otherwise, it restarts from S601 to display the live view of the next frame. In this way, in the shooting standby state, the system control unit 50 continues to perform the live view display operation until it determines that a recording start command has been issued. Note that the recording start command may also be a video recording start command issued by operating the video button 106. Furthermore, if a command other than a recording start command is detected, the system control unit 50 performs an action corresponding to the command, but the details of that are omitted from the explanation.

[0081] In S604, the system control unit 50 reads an image signal from the imaging unit 211 and outputs it to the A / D converter 212. The system control unit 50 reads an image signal with a higher resolution than that used during live view display, for example. The system control unit 50 causes the image processing unit 214 to generate image data for recording. The image processing unit 214 stores the image data for recording in the memory 215.

[0082] In step S605, the system control unit 50 acquires information to be recorded as metadata for the captured image data. For example, the system control unit 50 can acquire shooting conditions (information regarding the settings and status of the camera 100 and lens unit at the time of shooting) and parameters used in the development process. These are examples, and other information may also be acquired. Shooting conditions may include, for example, exposure conditions such as shutter speed, aperture value, and ISO sensitivity, frame rate, resolution, data compression format, color space, gamma value, camera 100 orientation, and whether or not the flash is on. In the case of video data, information indicating the start position of the frame in the video file is also included in the metadata. Furthermore, if lens information can be acquired from the lens unit, lens information can also be included in the metadata. Note that information not exemplified here may also be recorded as metadata.

[0083] In S606, the system control unit 50 records a data file containing the image data for recording generated by the image processing unit 214 in S604 onto, for example, the recording medium 227. In S607, the system control unit 50 records the information acquired in S605 as metadata for the image data, associating it with the data file recorded in S606. In this embodiment, metadata is recorded in the data file recorded in S606.

[0084] In S608, the system control unit 50 determines whether or not the user has issued a recording termination command. The system control unit 50 can determine that a recording termination command has been issued if, for example, the second shutter switch signal SW2 is not detected. In the case of video recording, the system control unit 50 can determine that a recording termination command has been issued if, for example, the operation of the video button 106 is detected. If the system control unit 50 determines that a recording termination command has been issued, it terminates the shooting operation. The system control unit 50 may also execute S601 to return to the shooting standby state. If it is not determined that a recording termination command has been issued, the system control unit 50 returns to S604 and executes the shooting of the next frame of the video or a still image.

[0085] When the camera 100 is equipped with a dual-lens unit 300, two circular fisheye images are arranged left and right (mesh, side-by-side format), and a VR180 image (still image or video) is recorded in which the left-right positional relationship of the circular fisheye images is reversed. There are no particular restrictions on the data format of still images and videos; they may be in RAW format or in a format after development processing (such as JPEG or MPEG format). The captured image may be transmitted to the PC 500 via the video signal output I / F 240 instead of being recorded to the recording medium 227, or in addition to being recorded to the recording medium 227. The PC 500 stores the image data file received via the communication unit 507 or external I / F 508 in the external storage device 504.

[0086] Next, the image data display operation of the PC500 will be explained using the flowchart in Figure 8. The following operations are achieved in the PC500 by the control unit 501 executing the image processing application stored in the external storage device 504.

[0087] In this configuration, the user operates the GUI displayed on the display unit 506 by the image processing application via the operation unit 505 to select an image data file to be processed, and the control unit 501 detects that the user has instructed the user to execute the processing and initiates the operation. The image data file to be processed is selected from, for example, image data files stored in an external storage device 504 (HDD, SSD, or memory card).

[0088] In S701, the control unit 501 obtains metadata about the image data within the selected image data file.

[0089] In S702, the control unit 501 uses the metadata acquired in S701 as needed to read the image data to be processed from the image data file into the RAM 503.

[0090] In S703, the control unit 501 (image processing means) determines the type of image data read in S702. Specifically, the control unit 501 determines whether the image data is data of an image containing two circular fisheye images. This determination corresponds to determining whether the image data is mesh-type VR image data. In the case of an image containing circular fisheye images, the surrounding areas other than the circular fisheye images are black pixels. Therefore, the control unit 501 generates binarized image data by applying a threshold value close to the brightness value of a black pixel to the brightness value of each pixel of the image data, for example. Then, if the control unit 501 (determination means) determines that the image represented by the binarized image data contains two white circular areas, it determines that the type of image data read in S702 is data of an image containing two circular fisheye images.

[0091] Whether a white area is circular or not can be determined by any known method. For example, if several arbitrary lines are drawn intersecting two points on the outer perimeter of the white area, and perpendicular lines passing through the midpoints of these lines intersect at a single point, then it can be determined to be circular. However, considering that errors may occur in practice, it may also be acceptable to determine that a region is circular if the intersection points of multiple perpendicular lines are within a certain distance of each other.

[0092] Furthermore, if two circular regions are detected, the control unit 501 may further determine whether the sizes of the circular regions are the same and whether the distance between the circular regions is less than a threshold. The control unit 501 then determines that the image data is VR image data containing two circular fisheye images if at least one of the following conditions is met: the sizes of the two circular regions are the same and the distance between the circular regions is less than a threshold.

[0093] The fact that the two circular regions are of equal size and that the distance between them is less than a threshold are conditions that increase the likelihood that the image data is VR image data containing two circular fisheye images. The size of a circular region may be, for example, the number of pixels within the region, or the maximum length of a straight line connecting two points on the outer perimeter of the region. If the difference in the number of pixels or the maximum length is less than a threshold, the two circular regions can be determined to be of equal size. The distance between the circular regions may be the shortest distance between the outer perimeters of the two circular regions.

[0094] Alternatively, the control unit 501 may determine that the image data is VR image data containing two circular fisheye images using a different method. For example, the control unit 501 divides the image data into two regions: a left region and a right region. Then, for each region, the control unit 501 detects a subregion formed by pixels equal to or greater than a threshold. The control unit 501 determines whether the subregion is a circular region using the method described above. If a circular region is detected in both the right and left regions, the control unit 501 can determine that the image data is VR image data containing two circular fisheye images. In this case as well, the control unit 501 may determine that the image data is VR image data containing two circular fisheye images if at least one of the following conditions is met: the sizes of the two circular regions are equal, and the distance between the circular regions is less than a threshold.

[0095] The control unit 501 executes S704 if it determines that the image data to be processed is an image containing two circular fisheye images, and executes S707 otherwise.

[0096] Furthermore, if the effective pixel area of ​​the imaging unit 211 is small, an image 1220 may be recorded in which a portion of the subject image (circular fisheye image) 1221 formed by the right eye optical system 301R and the left eye optical system 301L is missing, as shown in Figure 12(b). Even if the edges of the circular fisheye image are missing, the VR image can be displayed and reproduced by using the central portion. However, the field of view of the VR image will be narrower than when there is no missing portion in the circular fisheye image.

[0097] If the white region 1221 reaches the edge of image 1220, the portion of the outer perimeter of the white region 1221 that does not form the edge 1222 of image 1220 can be used to determine whether the white region 1221 is part of a circular region using the method described above. Specifically, multiple lines can be drawn that intersect with two points in the portion of the outer perimeter of the white region 1221 that does not form the edge 1222 of image 1220. If the perpendicular lines passing through the midpoints of these lines intersect at a single point, then the white region 1221 can be determined to be part of a circular region. If the white region 1221 is determined to be part of a circular region, then the image 1220 can be determined to contain a portion of a circular fisheye image.

[0098] For example, if the control unit 501 does not determine in S703 that the image contains two circular fisheye images, it can further determine whether or not it contains two partially missing circular fisheye images. If the control unit 501 determines that the image contains two partially missing circular fisheye images, it may consider the image to contain two circular fisheye images and execute the processing from S704 onward. In this case, S703 can be considered as determining whether or not the image data contains at least two partially circular fisheye images, and S704 to S706 can be considered as processing targeting at least partially circular fisheye images.

[0099] In S704, the control unit 501 determines the center coordinates and radii of the two circular fisheye images contained in the image. Since the perpendicular bisector of the chord of a circle passes through the center of the circle, two arbitrary lines can be drawn that intersect with two points on the circumference of the white region determined to be circular, and the intersection of the perpendicular bisectors of each line can be used as the center coordinates of the white region.

[0100] A concrete example will be explained using Figure 12(a). For example, for each of the two circular regions 1201 detected from the image data 1200 in S703, the pixel values ​​are examined horizontally at an arbitrary vertical position (Y coordinate) to find the X coordinate X1 of the pixel that changed from black to white and the X coordinate X2 of the pixel that was converted from white to black. Also, the pixel values ​​are examined vertically at an arbitrary horizontal position (X coordinate) to find the Y coordinate Y1 of the pixel that changed from black to white and the Y coordinate Y2 of the pixel that was converted from white to black. At this time, the center coordinates 1202(X0,Y0) of the circular region 1201 are (X1+X2) / 2 for the X coordinate (X0) and (Y1+Y2) / 2 for the Y coordinate (Y0). Furthermore, if the pixel values ​​are examined horizontally from the center coordinate and the X coordinate of the pixel that changed from white to black is X3, then the radius of the circle can be found as X3-((X1+X2) / 2). The center coordinates are calculated as image coordinates, for example, with a point within the image (e.g., the center of the image) as the origin. Similarly, for a circular region 1221 with a missing portion, as shown in Figure 12(b), the center coordinates and radius can be determined using the portion of the circumference excluding the part 1222 that forms the edge of the image.

[0101] Note that the method described here is just one example, and the center coordinates and radius of the circular fisheye image may be calculated using other image processing methods. Furthermore, to account for errors, the final center coordinates may be the average of the center coordinates obtained from multiple different combinations of two straight lines for a single circular region. Similarly, the radius may be calculated by averaging the results obtained from the center coordinates in multiple directions.

[0102] In S705, the control unit 501 (generation means) extracts images of two circular regions corresponding to the right and left images from the image data using the center coordinates and radii of the two circular regions obtained in S704. The control unit 501 may extract the images so that the surrounding black region is included so that the circular regions are not cut off. The control unit 501 then generates image data with the right and left images swapped left and right. The control unit 501 arranges the extracted images of the circular regions so that the center coordinates of the left and right circular regions obtained in S704 are maintained even after the swap. For example, the circular region extracted from the left side of the image (right image) is arranged so that its center coordinates match the center coordinates calculated for the circular region on the right side of the image (left image). The same applies to the circular region extracted from the left side of the image.

[0103] Alternatively, if the right and left eye coordinates in the image after the rearrangement are known, the position of the right image may be determined based on the right eye coordinates, and the position of the left image may be determined based on the left eye coordinates.

[0104] In S706, the control unit 501 displays the VR180 image data generated in S705, which has the arrangement of circular fisheye images rearranged, on the display unit 506, or outputs it to a stereoscopic display device such as VR goggles connected via the communication unit 507 or external I / F 508. When displaying VR180 image data, the control unit 501 may, if necessary, apply an equirectangular transformation to the circular fisheye image to convert it from mesh format to equirectangular format.

[0105] In S707, the control unit 501 displays the image data acquired in S702 directly on the display unit 506.

[0106] In Figure 8, it is shown that the display operation will end when S706 and S707 are completed, but it is also possible to return to S701 and wait for instructions to process other image data. Furthermore, if the image data to be processed is VR180 video data, the processes S704 to S706 are repeated for each frame. For video data, the processing results of S703 and S704 for the first frame can be reused for the second frame and beyond, so S704 can be omitted for the second frame and beyond.

[0107] Furthermore, at any point between S704 and S706, the user may be prompted to specify whether or not to swap the positions of the right and left images. If the user is not instructed not to swap the positions, the control unit 501 displays the image data acquired in S702 as is. If it is determined in S703 that the image does not contain two circular fisheye images, it may be determined whether or not it contains two circular fisheye images with parts missing. If it is determined that the image contains two circular fisheye images with parts missing, the processing from S704 onwards may be applied to the circular fisheye images with parts missing.

[0108] Furthermore, even if the metadata does not contain information indicating whether the image data to be processed is VR180 image data, the model name of the lens unit may be included in the metadata as general shooting information. In this case, by reading the model name of the lens unit in S701 and referring to a pre-registered list of VR180 lens model names, it is possible to determine the type of image data, that is, whether or not it is image data taken using a VR180 lens. In this case, the determination in S703 can be performed without image processing. The control unit 501 should execute S704 if the image data was taken using a VR180 lens, or S707 if the image data was taken using another lens.

[0109] Furthermore, the control unit 501 may save at least one of the image data obtained by swapping the positions of the right and left images, and the image data obtained by swapping the positions of the right and left images and further applying an equirectangular transformation to the right and left images, as a separate image file in the external storage device 504. These image data can be directly handled by general VR180 compatible devices such as VR goggles.

[0110] When these image data are saved separately, the metadata recorded in the image data file can include information indicating that the image data is VR180 image data, that the right and left images have been swapped, and that equirectangular transformation has been performed. This simplifies processing in VR systems that use image data.

[0111] In this embodiment, it is assumed that the camera 100 cannot obtain information from the twin-lens unit 300 indicating that the twin-lens unit 300 is a VR180 lens. If the camera 100 can determine that the twin-lens unit 300 is a VR180 lens based on the information obtainable from the twin-lens unit 300, then information indicating that it is VR180 image data can be recorded as metadata in the image data file. This allows the PC 500 to perform the determination in S703 without image processing. The control unit 501 should execute S704 if the image data was captured using a VR180 lens, or S707 if the image data was captured using another lens.

[0112] Furthermore, if the center coordinates and radius of the circular fisheye image (right image and left image) formed by the twin-lens unit 300 can be obtained from the twin-lens unit 300, this information can also be recorded as metadata in the image data file. The control unit 501 can skip processing S703 and S704 if the metadata obtained in S701 includes information on the center coordinates and radius of the circular fisheye image (right image and left image). Therefore, the control unit 501 may determine whether or not the metadata obtained in S701 includes information on the center coordinates and radius of the circular fisheye image (right image and left image) before executing S703. The control unit 501 then executes S703 if it determines that the metadata does not include information on the center coordinates and radius of the circular fisheye image (right image and left image), and executes S705 if it determines that it does.

[0113] As described above, according to this embodiment, when it is determined that the image data is data of an image containing two circular fisheye images, the type of image data is determined to be VR180 image data. Therefore, even if it is not possible to determine from the metadata of the image data whether or not it is VR180 image data, it is possible to correctly recognize the type of image and process the VR180 image data appropriately. Consequently, even if it is not possible to recognize that the lens unit to which the camera is attached is a VR180 lens, the type of image data recorded by the camera can be correctly recognized and handled appropriately.

[0114] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. The second embodiment relates to a configuration in which an image signal read from an image sensor is transmitted in real time as a video signal from a camera 100 equipped with a twin-lens unit 300 to a PC 500. In the following description, components common to the first embodiment will be described using the same reference numerals as in the first embodiment, and their explanation will be omitted.

[0115] Figure 9 is a schematic diagram of the image processing system according to the second embodiment. In this embodiment, the image processing system has a camera 100 and a PC 500 connected by a cable 190, and reads from the imaging unit 211 (image sensor) of the camera 100. Out The system has a configuration in which the captured image signal is transmitted to the PC500 in real time. Cable 190 connects, for example, the video signal output I / F240 of camera 100 and the external I / F508 of PC500. Here, as an example, the image signal is transmitted from camera 100 to PC500 using a transmission method compliant with the HDMI standard. Therefore, the video signal output I / F240 and the external I / F508 are interfaces compliant with the HDMI standard, and cable 190 is an HDMI cable. However, DVI (Digi t Video signals may be transmitted according to other standards such as the Visual Interface, SDI, and USB Video Class.

[0116] The PC500 can display video signals received via the external I / F 508 on the display unit 506 in real time. Furthermore, the PC500 can convert the received video signals into a streamable format and transmit them to the HMD 191 connected to the PC500, or distribute them to multiple users 193 via the cloud service 192.

[0117] Next, the video output operation of the camera 100 in this embodiment will be explained using the flowchart in Figure 10. In Figure 10, steps that perform the same processing as the shooting process described in the first embodiment are given the same reference numerals as in Figure 7 and their explanation is omitted. The video output operation is realized by the system control unit 50 executing a program and performing the necessary control. The shooting operation shown in Figure 10 starts when the user turns on the power of the camera 100, for example by operating the power switch 102. After the system control unit 50 performs the startup process associated with turning on the power, it puts the camera 100 into a shooting standby state.

[0118] In the shooting standby state, camera 100 continues to record video and displays a live view on the EVF217 or display unit 108 (S601~S602). In S903, the system control unit 50 determines whether or not cable 190 is connected to the video signal output I / F240 based on the voltage of the signal line. If it is determined that cable 190 is connected, the system control unit 50 executes S604; otherwise, it executes S601.

[0119] In S604, the system control unit 50 reads out one frame of image signals and instructs the image processing unit 214 to generate image data for display. In S605, the system control unit 50 acquires information to be sent to the PC 500 as metadata. In this embodiment as well, the system control unit 50 is unable to acquire information from the twin-lens unit 300 that would allow it to determine that the twin-lens unit 300 is a VR180 lens.

[0120] In S906, the system control unit 50 stores image data and metadata in a predetermined format in a buffer for HDMI signal output, which is provided as a portion of the memory 215, in parallel with the operation for live view display on the camera 100.

[0121] In S907, the system control unit 50 outputs the image data and metadata stored in the buffer as an HDMI signal via the video signal output I / F240. Details such as the encoding process for converting the data into an HDMI signal (TMDS sequence) are omitted from this explanation.

[0122] In S908, the system control unit 50 determines whether cable 190 has been disconnected from the video signal output interface 240 based on the voltage of the signal line. If it is determined that cable 190 has been disconnected, the video output operation is terminated. If it is not determined that cable 190 has been disconnected, the system control unit 50 repeats the process from S604. As a result, the live view image is continuously displayed on the display unit 506 of the PC 500.

[0123] In this embodiment, the camera 100 is equipped with a twin-lens unit 300, which is a VR180 lens. Therefore, the PC 500 receives a video signal of a VR180 image containing two circular fisheye images, in which the positions of the right and left images are reversed (corresponding to the captured image 402b in Figure 5). Out The signal format of the image data output to the PC500 may be RGB, YCbCr, or RAW. In the case of RAW format, S906 stores the RAW data in the buffer instead of the image data for display. The image data for display generated by the image processing unit 214 has a resolution that matches the resolution of the display device of the camera 100. The image data to be transmitted to the PC500 may have a resolution that takes into account the display resolution of the PC500.

[0124] Next, the video display operation of the PC500 will be explained using the flowchart in Figure 11. The following operation is achieved in the PC500 by the control unit 501 executing an image processing application stored in the external storage device 504. In Figure 11, steps that perform the same processing as the image data display operation described in the first embodiment are given the same reference numerals as in Figure 8 and their explanation is omitted. Here, it is assumed that the external I / F 508 and the video signal output I / F 240 of the camera 100 are connected by cable 190, and that the PC500 is in a state where it can receive a video signal from the camera 100.

[0125] In S1001, the control unit 501 determines whether or not it has received a video signal via the external I / F 508. If the control unit 501 determines that it has received a video signal, it executes S1002; otherwise, it executes S1001.

[0126] In S1002, the control unit 501 determines whether or not the reception of the video signal via the external I / F 508 has stopped. If the control unit 501 determines that the reception of the video signal has stopped, it terminates the video display operation; otherwise, it executes S1003.

[0127] In S1003, the control unit 501 decodes the received video signal and obtains metadata. In S1004, the control unit 501 decodes the received video signal to acquire image data. Thereafter, the control unit 501 executes the processing from S703 onwards as described in the first embodiment.

[0128] Furthermore, at any point between S704 and S706, the user may be prompted to specify whether or not to swap the positions of the right and left images. If the user is not instructed not to swap the positions, the control unit 501 displays the image data acquired in S1004 as is.

[0129] Furthermore, in S706, the right and left images of the image data, with their positions swapped, may be converted from mesh format to equirectangular format by applying an equirectangular transformation to the right and left images before display. The user may be allowed to specify whether to display the mesh format or the equirectangular format.

[0130] Furthermore, even if the metadata does not contain information indicating whether the image data to be processed is VR180 image data, the model name of the lens unit may be included in the metadata as general shooting information. In this case, by reading the model name of the lens unit in S1003 and referring to a pre-registered list of VR180 lens model names, it is possible to determine the type of image data to be processed (whether or not it is image data taken using a VR180 lens). In this case, the determination in S703 can be performed without image processing. The control unit 501 should execute S704 if the image data was taken using a VR180 lens, or S707 if the image data was taken using another lens.

[0131] Furthermore, since the video signal received by the PC500 in this embodiment corresponds to video data, the processing results of S703 and S704 for the first frame can be reused for the second frame and subsequent frames. Therefore, S704 can be omitted for the second frame and subsequent frames.

[0132] Alternatively, the image data obtained by swapping the positions of the right and left images may be converted from mesh format to equirectangular format by applying an equirectangular transformation to the right and left images, and the resulting image data may be converted to a streamable format and supplied externally. For example, the control unit 501 can transmit the streamable image data to an HMD connected to the PC 500, or to a distribution server via the communication unit 507 or external I / F 508. This allows the live view image of the camera 100 to be supplied to an external device or distributed to a user in a remote location.

[0133] Alternatively, the video signal received by the PC500 may be converted into a format that can be streamed directly, and the processing corresponding to S703 to S706 may be performed by an external device (such as a distribution server or HMD).

[0134] In this embodiment, it is assumed that the camera 100 cannot obtain information from the twin-lens unit 300 indicating that the twin-lens unit 300 is a VR180 lens. However, if the camera 100 can determine that the twin-lens unit 300 is a VR180 lens based on information obtainable from the twin-lens unit 300, then information indicating that the image data is VR180 can be included in the video signal as metadata. This allows the PC 500 to perform the determination in S703 without image processing. The control unit 501 should execute S704 if the image data was captured using a VR180 lens, or S707 if the image data was captured using another lens.

[0135] Furthermore, if the center coordinates and radius of the circular fisheye image (right image and left image) formed by the twin-lens unit 300 can be obtained from the twin-lens unit 300, this information can also be recorded as metadata in the image data file. The control unit 501 can skip processing S703 and S704 if the metadata obtained in S1003 includes information on the center coordinates and radius of the circular fisheye image (right image and left image). Therefore, the control unit 501 may determine whether or not the metadata obtained in S1003 includes information on the center coordinates and radius of the circular fisheye image (right image and left image) before executing S703. The control unit 501 then executes S703 if it determines that the metadata does not include information on the center coordinates and radius of the circular fisheye image (right image and left image), and executes S705 if it determines that it does.

[0136] As described above, according to this embodiment, when it is determined that the image data transmitted in the video signal transmitted in real time from the camera is image data containing two circular regions, the type of image data is determined to be VR180 image data. Therefore, even if it is not possible to determine from the metadata of the image data whether or not it is VR180 image data, it becomes possible to process VR180 image data appropriately. Consequently, even if it is not possible to recognize that the lens unit to which the camera is attached is a VR180 lens, the type of image received from the camera can be correctly recognized and handled appropriately.

[0137] (Other embodiments) In the first embodiment, the operations described using the flowchart in Figure 8 are not necessarily performed by the PC 500, but may be performed by the camera 100. In other words, the image processing device in the first embodiment may be the camera 100.

[0138] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0139] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0140] 100...Camera, 102...PC, 501...Control Unit, 502...ROM, 503...RAM, 504...External Storage Device, 505...Operation Unit, 506...Display Unit, 507...Communication Unit, 508...External I / F, 509...System Bus

Claims

1. An acquisition means for acquiring image data and metadata of said image data, A determination means for determining whether the image represented by the image data is the specific image when the metadata does not contain information indicating that the image represented by the image data is a specific image, The device includes a processing means that, if the determination means determines that the image is the specific image, performs a predetermined process on the image. The aforementioned specific image has two circular regions containing the subject image, and the region outside of these two circular regions does not contain the subject image. An image processing apparatus characterized by the following:

2. The image processing apparatus according to claim 1, wherein the determination means detects circular regions in the right region and the left region of the image in the determination.

3. The image processing apparatus according to claim 1 or 2, characterized in that the determination means determines that the image is the specific image when at least one of the following conditions is met: the difference in size between two circular regions detected from the image is less than a threshold, and the distance between the two circular regions is less than a threshold.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that the predetermined processing is a process that generates image data in which the positions of the two circular regions included in the image are swapped.

5. The image processing apparatus according to claim 4, characterized in that the swapping of the positions of the two circular regions is a swap of the left and right sides of the two circular regions.

6. The image processing apparatus according to claim 4 or 5, characterized in that the processing means stores the data of the image on which the processing has been performed in association with metadata indicating that it is of a predetermined type.

7. The two circular regions are a circular fisheye image, The image processing apparatus according to any one of claims 4 to 6, characterized in that the processing means further generates image data obtained by applying an equirectangular transformation to an image in which the positions of the two circular regions have been swapped.

8. The image processing apparatus according to any one of claims 1 to 6, characterized in that the two circular regions are a circular fisheye image.

9. The image processing apparatus according to any one of claims 1 to 8, characterized in that the acquisition means acquires the image data from the imaging device in real time.

10. The image processing apparatus according to any one of claims 1 to 9, characterized in that the particular image is an image obtained by forming two subject images on a single image sensor via two optical systems.

11. The image processing apparatus according to any one of claims 1 to 10, characterized in that the processing means performs the predetermined processing based on the user's instructions.

12. The image processing apparatus according to any one of claims 1 to 11, characterized in that the image processing apparatus is an imaging apparatus that generates the image data by imaging processing.

13. The image processing apparatus according to claim 12, characterized in that the image processing apparatus is a lens-exchangeable imaging apparatus.

14. The image processing apparatus according to any one of claims 1 to 13, characterized in that the determination means determines that the image is the specific image when it is determined that the image contains at least two partial circular regions.

15. An image processing method performed by an image processing device, A step of acquiring image data and metadata of said image data, If the metadata does not contain information indicating that the image represented by the image data is a specific image, the step of determining whether or not the image represented by the image data is the specific image, The process includes the step of performing a predetermined process on the image if it is determined that the image is the specific image. The aforementioned specific image has two circular regions containing the subject image, and the region outside of these two circular regions does not contain the subject image. An image processing method characterized by the following:

16. A program for causing a computer to function as each of the means of the image processing apparatus described in any one of claims 1 to 14.

Images