Image processing apparatus, image processing method, and program

Abstract

To provide a technique for reducing the processing load of processing for developing a RAW image including an area of a circumferential fisheye image.SOLUTION: An image processing device includes acquisition means for acquiring a first RAW image including an area of a first circumferential fisheye image and developing means for developing the first RAW image. The developing means does not develop pixels outside the first circumferential fisheye image in the first RAW image.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] There is known a technique of capturing a circumferential fisheye image having a parallax using two cameras and reproducing the captured parallax image as a three-dimensional virtual reality (VR) image. Further, there is known an apparatus having two optical systems on one lens mount and capable of capturing an image having a parallax at once (Patent Document 1). Also, there is known a camera capable of capturing an image in RAW format (RAW image).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to reproduce a RAW image including a region of a circumferential fisheye image as a VR image, it is necessary to perform both development processing and orthographic cylindrical conversion. Therefore, a relatively high processing load is imposed on an image processing apparatus such as a personal computer (PC), but no technique for reducing such a processing load has been known.

[0005] The present invention has been made in view of such a situation, and an object thereof is to provide a technique for reducing the processing load of processing for developing a RAW image including a region of a circumferential fisheye image.

Means for Solving the Problems

[0006] In order to solve the above problems, the present invention provides a first RAW image including a region of a first circumferential fisheye image and a second RAW image including a region of a second circumferential fisheye image having a parallax with respect to the first circumferential fisheye image and having a missing portion where pixel values do not existAn acquisition means for acquiring, and a developing means for developing the first RAW image, wherein the developing means does not develop pixels outside the region of the first circumferential fisheye image in the first RAW image and pixels of a specific portion of the first circumferential fisheye image located at a position corresponding to the missing portion of the second circumferential fisheye image An image processing apparatus is provided, characterized in that development is not performed for pixels outside the region of the first circumferential fisheye image in the first RAW image.

Advantages of the Invention

[0007] According to the present invention, it is possible to reduce the processing load of the process of developing a RAW image including the region of the circumferential fisheye image.

[0008] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the description in the following mode for carrying out the invention.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 14

Figure 15

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] [First Embodiment] ●Overall Configuration of the System FIG. 1 is a diagram showing the overall configuration of the system according to the first embodiment. In FIG. 1, reference numeral 100 denotes a digital camera (hereinafter referred to as "camera") capable of shooting an image in RAW format (a still image in RAW format or a moving image in RAW format). Reference numeral 250 denotes a VR180 lens that can be attached to the camera 100. The VR180 lens 300 is a binocular lens, and by shooting with the camera 100 to which the VR180 lens 300 is attached, an image having parallax can be obtained. The VR180 lens 300 is a lens for shooting an image for so-called VR180, which is in a format of a VR image enabling binocular stereoscopic vision and having a 180-degree field of view.

[0012] Reference numeral 500 denotes a personal computer (PC), which is an example of an image processing apparatus that processes an image shot by the camera 100. The method by which the PC 500 acquires an image from the camera 100 is not particularly limited. For example, as shown in FIG. 1(a), the PC 500 may acquire an image from the camera 100 by wireless communication. Alternatively, as shown in FIG. 1(b), the PC 500 may acquire an image recorded as a file via an external storage device (e.g., a memory card) of the camera 100.

[0013] ●Configuration of the camera 100 FIG. 2 is a diagram showing the appearance of the camera 100. FIG. 2(a) is a perspective view of the camera 100 seen from the front, and FIG. 2(b) is a perspective view of the camera 100 seen from the back.

[0014] The camera 100 has, on its upper surface, a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub - electronic dial 105, a video button 106, and an external - viewfinder display section 107. The shutter button 101 is an operation section for performing shooting preparation or a shooting instruction. The power switch 102 is an operation section for switching the power of the camera 100 on and off. The mode switch 103 is an operation section for switching various modes. The main electronic dial 104 is a rotary operation section for changing set values such as shutter speed and aperture. The sub - electronic dial 105 is a rotary operation section for moving a selection frame (cursor) or performing image scrolling, etc. The video button 106 is an operation section for giving an instruction to start or stop video shooting (recording). The external - viewfinder display section 107 displays various set values such as shutter speed and aperture.

[0015] In addition, on the back of the camera 100, there are a display unit 108, a touch panel 109, direction keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, and a menu button 115. Furthermore, the camera 100 has an eyepiece part 116, an eyepiece finder 117 (a peeping-type finder), an eyepiece detection unit 118, and a touch bar 119. The display unit 108 displays images and various types of information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108. The direction keys 110 are an operation unit composed of keys that can be pressed in the up, down, left, and right directions (4-direction keys). Operations can be performed according to the position where the direction keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed when determining a selected item. The AE lock button 112 is an operation unit that is pressed when fixing the exposure state in the shooting standby state. The zoom button 113 is an operation unit for switching between on and off of the zoom mode in the live view display (LV display) in the shooting mode. When the zoom mode is on, by operating the main electronic dial 104, the live view image (LV image) can be enlarged or reduced. Also, the zoom button 113 is used when enlarging the playback image or increasing the magnification rate in the playback mode. The playback button 114 is an operation unit for switching between the shooting mode and the playback mode. By pressing the playback button 114 in the shooting mode, the user can shift to the playback mode and display the latest image among the images recorded on the recording medium 227 described later on the display unit 108.

[0016] The menu button 115 is an operation unit that is pressed when displaying a menu screen on which various settings can be made on the display unit 108. The user can intuitively perform various settings using the menu screen displayed on the display unit 108, the direction keys 110, and the SET button 111. The eyepiece part 116 is a part for looking through the eyepiece finder 117. The user can view the video displayed on the internal EVF 217 (Electronic View Finder) described later through the eyepiece part 116. The eyepiece detection unit 118 is a sensor that detects whether the user is looking through the eyepiece part 116.

[0017] The touch bar 119 is a linear touch operation unit (line touch sensor) capable of receiving touch operations. The touch bar 119 is arranged at a position where it can be touched (touched) with the right thumb in a state where the grip portion 120 is held with the right hand (a state where it is held with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. That is, the touch bar 119 is operable in a state (shooting posture) where it is held up to the viewfinder 117 through the eyepiece 116 and the shutter button 101 can be pressed at any time. The touch bar 119 can receive a tap operation (an operation of touching and releasing without moving within a predetermined period) on the touch bar 119, a slide operation to the left and right (an operation of moving the touch position while keeping the touch after touching), and the like. The touch bar 119 is an operation unit different from the touch panel 109 and does not have a display function. The touch bar 119 of the present embodiment is a multifunction bar and functions as, for example, an M-Fn bar.

[0018] The camera 100 also has a grip portion 120, a thumb rest portion 121, a terminal cover 122, a lid 123, and a communication terminal 124. The grip portion 120 is a holding portion formed in a shape that is easy to hold with the right hand when the user holds the camera 100. With the grip portion 120 held by the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are arranged at positions operable by the index finger of the right hand. Also, in the same state, the sub - electronic dial 105 and the touch bar 119 are arranged at positions operable by the thumb of the right hand. The thumb rest portion 121 (thumb standby position) is a grip portion provided at a location on the back side of the camera 100 where it is easy to place the thumb of the right hand that is holding the grip portion 120 without operating any operation portion. The thumb rest portion 121 is composed of a rubber member or the like for enhancing the holding force (grip feeling). The terminal cover 122 protects the connector such as a connection cable for connecting the camera 100 to an external device. The lid 123 protects the recording medium 227 and the slot by closing the slot for storing the recording medium 227 described later. The communication terminal 124 is a terminal for the camera 100 to communicate with the lens unit 200 side which is detachable as described later.

[0019] Figure 3 is a diagram showing an example of the internal configuration of the camera 100. Note that the same components as those in Figure 2 are denoted by the same reference numerals and their descriptions are omitted as appropriate. The lens unit 200 can be attached to the camera 100.

[0020] First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that can be attached to and detached from the camera 100. The lens unit 200 is a single - lens reflex lens and is an example of a normal lens.

[0021] The lens unit 200 includes a diaphragm 201, a lens 202, a diaphragm drive circuit 203, an AF drive circuit 204 (auto focus drive circuit), a lens system control circuit 205, and a communication terminal 206. The diaphragm 201 is configured such that its aperture diameter is adjustable. The lens 202 is composed of a plurality of lenses. The diaphragm drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the diaphragm 201. The AF drive circuit 204 drives the lens 202 to focus. The lens system control circuit 205 controls the diaphragm drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit 50 described later. The lens system control circuit 205 controls the diaphragm 201 via the diaphragm drive circuit 203 and focuses by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, communication is carried out via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100 side.

[0022] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50. The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on an instruction from the system control unit 50. The imaging unit 211 is an image sensor (image sensor) composed of a CCD, a CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 211 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing processing such as reduction, color conversion processing, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. Also, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and based on the obtained arithmetic result, the system control unit 50 performs exposure control and distance measurement control. Through this processing, TTL (through-the-lens) method AF processing, AE (automatic exposure) processing, EF (flash pre-emission) processing, etc. are performed. Furthermore, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and based on the obtained arithmetic result, performs TTL method AWB (auto white balance) processing.

[0023] The image data from the A / D converter 212 is written into the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written into the memory 215 via the memory control unit 213 without passing through the image processing unit 214. The memory 215 stores the image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, and the image data for display on the display unit 108 and the EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images, a moving image for a predetermined time, and audio. Also, the memory 215 doubles as a memory for image display (video memory).

[0024] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 and the EVF 217. Therefore, the image data for display written in the memory 215 is displayed on the display unit 108 and the EVF 217 via the D / A converter 216. The display unit 108 and the EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and the EVF 217 are, for example, displays such as an LCD or an organic EL. The digital signal A / D-converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216 and sequentially transferred to the display unit 108 and the EVF 217 for display, thereby performing live view display.

[0025] The system control unit 50 is a control unit composed of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, may be a circuit, or may be a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 realizes each process of the flowchart described later by executing the program recorded in the non-volatile memory 219. In addition, the system control unit 50 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, etc.

[0026] In addition, the camera 100 includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, an attitude detection unit 222, and an eye detection unit 118. The system memory 218 uses, for example, RAM. Constants, variables, programs read from the non-volatile memory 219, etc., for the operation of the system control unit 50 are expanded in the system memory 218. The non-volatile memory 219 is an electrically erasable and recordable memory, and for example, EEPROM is used. Constants, programs, etc., for the operation of the system control unit 50 are recorded in the non-volatile memory 219. The program here is a program for executing the flowchart described later. The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from external devices connected by a wireless or wired cable. The communication unit 221 can also be connected to a wireless LAN (Local Area Network) or the Internet. In addition, the communication unit 221 can communicate with external devices using Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit the images captured by the imaging unit 211 (including live images) and the images recorded on the recording medium 227, and can receive image data and other various information from external devices. The attitude detection unit 222 detects the attitude of the camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 is an image captured with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate and record the image. The attitude detection unit 222 can use, for example, an acceleration sensor, a gyro sensor, etc. It is also possible to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.) using the attitude detection unit 222.

[0027] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece portion 116 of the viewfinder 117 incorporating the EVF 217. The eyepiece detection unit 118 can use, for example, an infrared proximity sensor. When an object approaches, the infrared rays projected from the light projecting unit of the eyepiece detection unit 118 are reflected by the object and received by the light receiving unit of the infrared proximity sensor. The distance from the eyepiece portion 116 to the object can be determined based on the amount of the received infrared rays. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of the object to the eyepiece portion 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eyepiece) and separation (eyeleaving) of the eye (object) to the eyepiece portion 116 of the viewfinder 117. When an object approaching within a predetermined distance to the eyepiece portion 116 is detected from the non-eyepiece state (non-approaching state), it is detected that the eyepiece is made. On the other hand, when the object whose approach has been detected moves away by a predetermined distance or more from the eyepiece state (approaching state), it is detected that the eye has left. The threshold for detecting the eyepiece and the threshold for detecting the eye leaving may be different, for example, by providing hysteresis. Also, after detecting the eyepiece, it is assumed to be in the eyepiece state until the eye leaving is detected. After detecting the eye leaving, it is assumed to be in the non-eyepiece state until the eyepiece is detected. The system control unit 50 switches the display (display state) / non-display (non-display state) of the display unit 108 and the EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, at least in the shooting standby state and when the switching setting of the display destination is automatic switching, when not in the eyepiece state, the display destination is set to the display unit 108 and the display is turned on, and the EVF 217 is turned off. Also, when in the eyepiece state, the display destination is set to the EVF 217 and the display is turned on, and the display unit 108 is turned off. Note that the eyepiece detection unit 118 is not limited to the case of being an infrared proximity sensor, and other sensors may be used as long as they can detect a state that can be regarded as the eyepiece. Also, the camera 100 includes an external viewfinder display unit 107, an external viewfinder display driving circuit 223, a power control unit 224, a power supply unit 225, a recording medium I / F 226, an operation unit 228, and a video signal output I / F 240. The external viewfinder display unit 107 displays various setting values of the camera 100, such as the shutter speed and aperture, via the external viewfinder display driving circuit 223.The power control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching the energized block, etc., and detects the presence or absence of battery installation, the type of battery, the remaining battery level, etc. Further, the power control unit 224 controls the DC-DC converter based on the detection result and the instruction of the system control unit 50, and supplies the necessary voltage to each part including the recording medium 227 for the necessary period. The power supply unit 225 is a primary battery such as an alkaline battery and a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, and a Li battery, an AC adapter, etc. The recording medium I / F 226 is an interface with a recording medium 227 such as a memory card or a hard disk. The recording medium 227 is a memory card or the like for recording the captured image, and is composed of a semiconductor memory, a magnetic disk, etc. The recording medium 227 may be detachable or built-in. The video signal output I / F 240 is an interface (HDMI terminal) for performing HDMI (High-Definition Multimedia Interface) output. The video signal from the camera 100 is output to a video signal receiving device 241 such as an external display or an external recorder via an HDMI cable.

[0028] The operation unit 228 is an input unit that receives operations (user operations) from the user and is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode change switch 103, a touch panel 109, and other operation units 229. The other operation units 229 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a direction key 110, a SET button 111, an AE lock button 112, a zoom button 113, a play button 114, a menu button 115, and a touch bar 119.

[0029] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 turns on during the operation of the shutter button 101, i.e., during a so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processes such as AF processing, AE processing, AWB processing, and EF processing based on the first shutter switch signal SW1. The second shutter switch 231 turns on when the operation of the shutter button 101 is completed, i.e., during a so-called full-press (shooting instruction), and generates a second shutter switch signal SW2. The system control unit 50 starts a series of shooting processes from reading the signal from the imaging unit 211 to generating an image file including the captured image and writing it to the recording medium 227 based on the second shutter switch signal SW2.

[0030] The mode switch 103 switches the operation mode of the system control unit 50 to any one of a still image shooting mode, a video shooting mode, a playback mode, etc. The modes included in the still image shooting mode are an auto shooting mode, an auto scene discrimination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), a program AE mode (P mode). Also, there are various scene modes and custom modes for shooting settings according to the shooting scene. The user can directly switch to any of the above shooting modes using the mode switch 103. Alternatively, the user can first switch to a list screen of shooting modes using the mode switch 103 and then selectively switch to any of the displayed multiple modes using the operation unit 228. Similarly, the video shooting mode may also include a plurality of modes.

[0031] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be integrally configured. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that the light transmittance does not interfere with the display of the display unit 108. Then, by associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured as if the user can directly operate the screen displayed on the display unit 108. For the touch panel 109, any one of various methods such as the resistive film method, the capacitance method, the surface acoustic wave method, the infrared method, the electromagnetic induction method, the image recognition method, and the optical sensor method can be used. Depending on the method, there are methods that detect a touch when there is contact with the touch panel 109, and methods that detect a touch when a finger or pen approaches the touch panel 109, but any method may be used.

[0032] The system control unit 50 can detect the following operations or states on the touch panel 109.

[0033] · A finger or pen that was not touching the touch panel 109 newly touches the touch panel 109, that is, the start of a touch (hereinafter referred to as Touch-Down).

[0034] · A state where the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On).

[0035] · The finger or pen that is touching the touch panel 109 is moving while touching it (hereinafter referred to as Touch-Move).

[0036] · A finger or pen that was touching the touch panel 109 has left (been released) from the touch panel 109, that is, the end of a touch (hereinafter referred to as Touch-Up).

[0037] · The state where nothing is touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0038] When a touch down is detected, a touch on is also detected simultaneously. After a touch down, unless a touch up is detected, usually a touch on is continuously detected. Even when a touch move is detected, a touch on is detected simultaneously. Even if a touch on is detected, if the touch position does not move, a touch move is not detected. After it is detected that all fingers or pens that were touching have touched up, it becomes Touch-Off.

[0039] These operations / states and the position coordinates where a finger or pen is touching on the touch panel 109 are notified to the system control unit 50 through the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding a touch move, the moving direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical component and horizontal component on the touch panel 109 based on the change in the position coordinates. When it is detected that a touch move has been made by a predetermined distance or more, it is determined that a slide operation has been performed. An operation of quickly moving a finger a certain distance while touching the touch panel 109 and then leaving it is called a flick. In other words, a flick is an operation of quickly tracing on the touch panel 109 as if pushing it with a finger. When it is detected that a touch move has been made by a predetermined distance or more at a predetermined speed or more and a touch up is detected immediately afterwards, it is determined that a flick has been performed (it can be determined that there is a flick following a slide operation). Furthermore, a touch operation of touching multiple locations (for example, two points) together (multi-touching) and bringing the touch positions closer to each other is called pinch in, and a touch operation of moving the touch positions away from each other is called pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply a pinch).

[0040] ● Configuration of the VR180 lens 300 FIG. 4 is a diagram showing an example of the configuration of the VR180 lens 300 that can be attached to the camera 100. FIG. 4 shows a state in which the VR180 lens 300 is attached to the camera 100. In the camera 100 shown in FIG. 4, the same components as those described in FIG. 3 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0041] The VR180 lens 300 is a type of interchangeable lens that can be attached to and detached from the camera 100. The VR180 lens 300 is a binocular lens capable of taking pictures with parallax in the left and right images. The VR180 lens 300 has two optical systems, each having a wide viewing angle of approximately 180 degrees and capable of photographing the range of the front hemisphere. Specifically, the two optical systems of the VR180 lens 300 can photograph subjects with a viewing angle (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, pitch angle, elevation angle).

[0042] The VR180 lens 300 includes a right-eye optical system 301R having a plurality of lenses and a reflection mirror, etc., a left-eye optical system 301L having a plurality of lenses and a reflection mirror, etc., and a lens system control circuit 303. The right-eye optical system 301R corresponds to an example of the first optical system, and the left-eye optical system 301L corresponds to an example of the second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the respective lenses 302R and 302L located on the subject side face the same direction, and the respective optical axes are substantially parallel.

[0043] The VR180 lens 300 of this embodiment is a lens for taking an image in the so-called VR180 format, which enables binocular stereoscopic vision and has a 180-degree field of view. The VR180 lens 300 has a fisheye lens in which the right optical system 301R and the left optical system 301L can each capture a range of approximately 180 degrees. Note that the VR180 lens 300 only needs to be able to acquire an image in which the right optical system 301R and the left optical system 301L can each perform binocular VR display as VR180, and may be a lens that can capture a wide-angle field of view of about 160 degrees, which is narrower than the 180-degree range. The VR180 lens 300 can form a right image (first image) formed via the right optical system 301R and a left image (second image) formed via the left optical system 301L having a parallax with the right image on one or two imaging elements of the mounted camera.

[0044] In addition, the VR180 lens 300 is mounted on the camera 100 via a lens mount portion 304 and a camera mount portion 305 of the camera 100. When the VR180 lens 300 is mounted on the camera 100, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the VR180 lens 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the VR180 lens 300.

[0045] In this embodiment, the right image formed via the right optical system 301R and the left image formed via the left optical system 301L having a parallax with the right image are simultaneously (as a set) imaged on the imaging unit 211 of the camera 100 side by side. That is, two optical images formed by the right optical system 301R and the left optical system 301L are formed on one imaging element. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. In this way, by using the VR180 lens 300, two images with parallax can be simultaneously acquired from two locations (optical systems), namely, the right optical system 301R and the left optical system 301L. Further, by separately VR-displaying the acquired images as an image for the left eye and an image for the right eye, the user can view a stereoscopic VR image in a range of approximately 180 degrees, that is, a so-called VR180.

[0046] Here, a VR image is an image that can be VR - displayed as described later. VR images include omnidirectional images (full - sphere images) captured by an omnidirectional camera (full - sphere camera), panoramic images having a video range (effective video range) wider than the display range that can be displayed at once on the display unit, and the like. Also, VR images include not only still images but also moving images and live images (images acquired almost in real - time from a camera). A VR image has a video range (effective video range) of up to 360 degrees in the left - right direction and 360 degrees in the up - down direction. Further, VR images include images having a wider angle of view than the angle of view that can be captured by a normal camera or a video range wider than the display range that can be displayed at once on the display unit, even if the left - right direction is less than 360 degrees and the up - down direction is less than 360 degrees. The image captured by the camera 100 using the above - mentioned VR180 lens 300 is a kind of VR image. A VR image can be VR - displayed, for example, by setting the display mode of a display device (a display device capable of displaying a VR image) to "VR view". By VR - displaying a VR image having a 360 - degree angle of view and having the user change the posture of the display device in the left - right direction (horizontal rotation direction), an omnidirectional video without seams in the left - right direction can be viewed.

[0047] Here, VR display (VR view) is a display method (display mode) with a variable display range that displays an image of a visual field range corresponding to the posture of a display device among VR images. In VR display, there is "monocular VR display (monocular VR view)" that performs a transformation (a transformation with distortion correction applied) of mapping a VR image onto a virtual sphere to display one image. Also, in VR display, there is "binocular VR display (binocular VR view)" that performs a transformation of mapping a VR image for the left eye and a VR image for the right eye onto a virtual sphere respectively and arranges and displays them in the left and right regions. It is possible to perform stereoscopic viewing by using the VR image for the left eye and the VR image for the right eye with a parallax between them for "binocular VR display". In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), an image of a visual field range corresponding to the direction of the user's face is displayed. For example, assume that among VR images, an image of a visual field range centered at 0 degrees in the left-right direction (a specific azimuth, for example, north) and 90 degrees in the up-down direction (90 degrees from the zenith, that is, horizontal) is being displayed at a certain point in time. When the posture of the display device is reversed front and back from this state (for example, the display surface is changed from facing south to facing north), the display range is changed to an image of a visual field range centered at 180 degrees in the left-right direction (the opposite azimuth, for example, south) and 90 degrees in the up-down direction among the same VR images. That is, when the user wears an HMD and turns the face from north to south (that is, turns to the back), the image displayed on the HMD is also changed from the image of the north to the image of the south.

[0048] Note that the VR image captured using the VR180 lens 300 of this embodiment is a VR180 image that captures a range of approximately 180 degrees in the front, and there is no image of a range of approximately 180 degrees in the back. When such a VR180 image is VR-displayed and the posture of the display device is changed to the side where there is no image, a blank area is displayed.

[0049] By displaying the VR image in this way for VR display, the user will visually feel as if they are inside the VR image (inside the VR space). Note that the method of displaying the VR image is not limited to the method of changing the posture of the display device. For example, the display range may be moved (scrolled) according to user operations via a touch panel, direction buttons, etc. Also, during VR display (when in the display mode "VR View"), in addition to changing the display range due to a change in posture, the display range may be changed according to a touch move on the touch panel, a drag operation with a mouse, etc., or the pressing of a direction button. Note that a smartphone mounted on a VR goggle (head-mounted adapter) is a type of HMD.

[0050] ● Configuration of PC500 Figure 5 is a block diagram showing the configuration of PC500. In Figure 5, 501 is a control unit that controls the entire PC500, for example, a Central Processing Unit (CPU). 502 is a Read Only Memory (ROM) that stores programs and parameters that do not require modification. In ROM502, an information processing program is stored as program code readable by the control unit 501, and the control unit 501 is configured to execute this information processing program code. 503 is a Random Access Memory (RAM) that temporarily stores programs and data supplied from external devices, etc.

[0051] 504 is an external storage device including a hard disk or flash memory fixedly installed in PC500. Alternatively, the external storage device 504 may be an external storage device including a removable floppy disk (FD), an optical disk such as a Compact Disk (CD), a magnetic or optical card, an IC card, a memory card, etc. from PC500. The image file acquired by PC500 from the camera 100 is stored in the external storage device 504.

[0052] 505 is an operation unit such as a button or a touch panel that receives a user's operation and inputs data. 506 is a display unit for displaying data held by the PC 500 or supplied data. 507 is a communication unit for communicating with an external device such as the camera 100. 508 is an external I / F for transmitting and receiving video signals and files to and from an external device. 509 is a system bus that communicably connects components of the PC 500.

[0053] ● Features of an image captured by the VR180 lens 300 When there is one lens optical system, an image rotated 180 degrees on the imaging element is formed. When generating a normal image from the 180-degree rotated image, the camera 100 aligns the vertical direction of the image with the vertical direction of the subject by performing a 180-degree rotation process. When shooting is performed by the camera 100 equipped with the VR180 lens 300, images of each optical system are formed on one imaging element via the right-eye optical system 301R and the left-eye optical system 301L. At that time, the image of each optical system rotates 180 degrees for each optical system. Similar to the case where there is one lens optical system, the camera 100 aligns the vertical direction of the image with the vertical direction of the subject by rotating the entire image 180 degrees. Since the rotation at the time of imaging occurs in units of the optical system while the rotation at the time of image generation is performed on the entire image, the image corresponding to the left-eye optical system moves to the right side of the entire image, and the image corresponding to the right-eye optical system moves to the left side of the entire image. Therefore, in order to display the left and right images in the correct positional relationship, it is necessary to perform a swapping process of the left and right images.

[0054] ● Shooting process by the camera 100 Next, with reference to FIGS. 6A and 6B, the shooting process by the camera 100 will be described. When the user turns on the power of the camera 100, the processes of the flowcharts in FIGS. 6A and 6B start.

[0055] In S601, the system control unit 50 determines whether the firmware of the camera 100 supports the VR180 lens. If it is determined that the firmware supports the VR180 lens, the processing steps proceed to S602. If it is determined that the firmware does not support the VR180 lens, the processing steps proceed to S622. Since the optical system of the VR180 lens is different from that of a general lens, the camera 100 needs to be able to read and record the metadata of the VR180 lens for post-processing. Therefore, the system control unit 50 determines whether the firmware supports the VR180 lens.

[0056] In S602, the system control unit 50 determines whether a VR180 lens (for example, the VR180 lens 300 shown in FIG. 4) is attached to the camera 100. If it is determined that the VR180 lens is attached, the processing steps proceed to S603. If it is determined that the VR180 lens is not attached, the processing steps proceed to S622.

[0057] In S603, the system control unit 50 acquires the design values of the VR180 lens. The lens design values are used in the left-right swapping process and the orthographic cylindrical conversion process described later.

[0058] In S604, the system control unit 50 acquires the individual values of the VR180 lens. In addition to the lens design values, the VR180 lens additionally holds information on individual values such as manufacturing errors. In the orthographic cylindrical conversion process, by using the manufacturing error values, better results can be obtained compared to performing orthographic cylindrical conversion using only the design values.

[0059] In S605, the system control unit 50 acquires an image from the imaging unit 211.

[0060] In S606, the system control unit 50 displays the image acquired in S605 on the EVF 217 (live view display).

[0061] In S607, the system control unit 50 determines whether a recording start instruction has been given by an operation of the shutter button 101. If it is determined that the recording start instruction has been given, the processing step proceeds to S608. If it is determined that the recording start instruction has not been given, the processing step proceeds to S605. Therefore, the live view display on the EVF is repeated until a recording start instruction is given by the user. The recording start instruction may be an instruction for still image shooting or an instruction for video shooting.

[0062] In S608, the system control unit 50 acquires an image from the imaging unit 211.

[0063] In S609, the system control unit 50 acquires shooting information such as the shutter speed at the time of shooting, and metadata such as the attitude information of the camera 100 at the time of shooting. The attitude information is acquired from the attitude detection unit 222. Also, the system control unit 50 acquires metadata necessary for developing a RAW format image (RAW image) and the like.

[0064] In S610, the system control unit 50 records the image acquired in S608 as a RAW format image (RAW image) in a file.

[0065] In S611, the system control unit 50 records the shooting information and metadata acquired in S609 in association with the file. For example, the system control unit 50 records the shooting information and metadata in the same file as the image.

[0066] In S612, the system control unit 50 records the information of the VR180 lens acquired in S603 and S604 in association with the file. For example, the system control unit 50 records the information of the VR180 lens in the same file as the image.

[0067] In S613, the system control unit 50 determines whether a recording end instruction has been given by the user. In the case of still image shooting, since only one image is shot, it is assumed that the recording start instruction and the recording end instruction are given simultaneously. In the case of video shooting, the user presses the shutter button 101 again to end the recording instruction. If it is determined that the recording end instruction has been given, the processing of this flowchart ends. If it is determined that the recording end instruction has not been given, the processing step proceeds to S608. By repeatedly performing the processing of S608 to S613 by the system control unit 50, moving images can be successively recorded in files.

[0068] Next, the case where the processing step proceeds from S601 or S602 to S622 will be described. When the processing step proceeds from S601 to S622, the firmware of the camera 100 does not support the VR180 lens. Therefore, the camera 100 cannot perform processing specific to the VR180 lens. Accordingly, even if the VR180 lens is attached to the camera 100, the camera 100 performs the same processing as when a normal lens is attached.

[0069] In S622, the system control unit 50 determines whether a lens is attached to the camera 100. If it is determined that a lens is attached, the processing step proceeds to S623. If it is determined that no lens is attached, the processing of this flowchart ends.

[0070] In S623, the system control unit 50 acquires general information of the lens (for example, the lens name).

[0071] In S624, the system control unit 50 determines whether the general information of the lens could be acquired in S623. If the general information of the lens could be acquired, the processing step proceeds to S625. If the general information of the lens could not be acquired, the processing step proceeds to S626. For example, when the manufacturers of the camera 100 and the lens are different and the camera 100 cannot communicate with the lens, the camera 100 cannot acquire information from the lens.

[0072] In S625, the system control unit 50 stores the general lens information acquired in S623 in the system memory 218.

[0073] The processes of S626 to S632 are the same as the processes of S605 to S611, so the description is omitted.

[0074] In S633, the system control unit 50 determines whether the general lens information is stored in the system memory 218 (see S625). If the general lens information is stored, the processing step proceeds to S634. If the general lens information is not stored, the processing step proceeds to S635.

[0075] In S634, the system control unit 50 records the general lens information stored in the system memory 218 in association with a file. For example, the system control unit 50 records the general lens information in the same file as the image.

[0076] The process of S635 is the same as the process of S613. When an end-of-recording instruction is given, the processing of this flowchart ends. When an end-of-recording instruction has not been given, the processing step proceeds to S629. By repeating the processes of S629 to S635 by the system control unit 50, moving images can be successively recorded in files.

[0077] Note that the RAW images captured in the flowcharts of FIGS. 6A and 6B may be still images (RAW still images) or moving images (RAW moving images).

[0078] ● Photographed image of the camera 100 with the VR180 lens 300 attached Next, with reference to FIGS. 11(a) and 11(b), an example of an image photographed and recorded by the camera 100 with the VR180 lens 300 attached will be described.

[0079] The VR180 lens 300 is designed assuming specific values regarding the size of the imaging device and the recorded angle of view. For example, in the case of the VR180 lens 300 assuming a DCI 8K angle of view, the assumed angle of view is 8192x4320.

[0080] On the other hand, the camera 100 can record at an angle of view other than the angle of view assumed by the VR180 lens 300 according to the shooting settings. For example, in the case of the UHD 8K setting, the angle of view is 7680x4320. In this case, although the entire image of the VR180 lens 300 is input on the imaging device, a part thereof (for example, both ends) will not be recorded.

[0081] FIG. 11(a) is an example image when recording at an angle of view using the entire imaging device with the camera 100 having an imaging device of the size assumed by the VR180 lens 300. This is, for example, the case where the size of the imaging device assumed by the VR180 lens 300 is DCI 8K (8192x4320) and the recording size is also DCI 8K. In this case, two circumferential fisheye images arranged side by side are completely recorded.

[0082] FIG. 11(b) is an example image when recording at an angle of view using a part of the imaging device with the camera 100 having an imaging device of the size assumed by the VR180 lens. This is, for example, the case where the size of the imaging device assumed by the VR180 lens is DCI 8K (8192x4320) and the recording size is UHD 8K (7680x4320). In this case, two circumferential fisheye images arranged side by side are completely formed on the imaging device, but a part thereof will be recorded in a missing state. For example, as shown in FIG. 11(b), a part on the left side of the left circumferential image and a part on the right side of the right circumferential image among the two circumferential fisheye images arranged side by side are recorded in a missing state. In the following description, the missing region (part) of the circumferential fisheye image is also referred to as the "missing region" or "missing part".

[0083] ● Development processing and orthographic cylindrical conversion processing by the PC500 Referring to FIGS. 7 and 8, the development process and the orthographic cylindrical conversion process by the PC500 will be described. The PC500 performs a development process and an orthographic cylindrical conversion process on the RAW image included in the image file acquired from the camera 100. Through the development process and the orthographic cylindrical conversion process, two left and right orthographic cylindrical projection images are generated from the two left and right circumferential fisheye images as described with reference to FIGS. 11(a) and 11(b). Note that in the example of FIG. 7, the development process is included in the process of S712 (the process of creating an orthographic cylindrical projection image).

[0084] In S701, the control unit 501 reads the RAW image from the image file. This is a still image or a moving image of the image file shown in FIG. 9 described later.

[0085] In S702, the control unit 501 reads the shooting information and metadata other than the VR180 lens from the image file. This is the shooting information and metadata other than the VR180 lens of the image file shown in FIG. 9 described later. The metadata other than the VR180 lens includes the metadata necessary for the development of the RAW image (RAW development).

[0086] In S703, the control unit 501 determines whether the image file has VR180 lens metadata. This is the VR180 lens metadata of the image file shown in FIG. 9 described later. If the image file has VR180 lens metadata, the process step proceeds to S704. If the image file does not have VR180 lens metadata, the process step proceeds to S706.

[0087] In S704, the control unit 501 reads the VR180 lens metadata from the image file.

[0088] In S705, the control unit 501 obtains the center coordinates and radii of the two circumferential images on the left and right in the recorded image from the design values of the VR180 lens among the VR180 lens metadata read in S704. Also, when the VR180 lens metadata has the individual values (manufacturing errors) of the VR lens, the control unit 501 obtains the center coordinates and radii of the two circumferential images on the left and right, reflecting the individual values as well.

[0089] When the processing step migrates from S703 to S706, the control unit 501 determines whether the image file has a lens name. For example, when the firmware of the camera 100 does not support the VR180 lens, the camera 100 does not record the VR180 lens metadata, but it may record the lens name in the metadata area other than the VR180 lens. When the image file has a lens name, the processing step migrates to S707. When the image file does not have a lens name, the processing step migrates to S709.

[0090] In S707, the control unit 501 obtains the lens name from the image file.

[0091] In S708, the control unit 501 determines whether the lens name obtained in S707 indicates a known VR180 lens. When the lens name obtained in S707 indicates a known VR180 lens, the processing step migrates to S709. When the lens name obtained in S707 does not indicate a known VR180 lens, the processing step migrates to S710.

[0092] In S709, the control unit 501 determines the center coordinates and radii of the two circumferential images on the left and right in the recorded image based on the lens name obtained in S707. To implement this processing, the program of the PC500 is configured to hold the lens name and the design values of the center coordinates and radii of the two circumferential images on the left and right of the image taken with that lens. Alternatively, these pieces of information may be given to the program from the outside.

[0093] When the processing step migrates from S706 or S708 to S710, the control unit 501 determines whether the image read in S701 is two circumferential fisheye images on the left and right. For example, regarding the luminance value of the pixels of the read image, when the value is less than or equal to the threshold value (a value close to black), the pixel value of that pixel is set to black, and when the value exceeds the threshold value, the pixel value of that pixel is set to white, to create a binary monochrome image. Then, the control unit 501 can determine whether the image read in S701 is two circumferential fisheye images on the left and right based on whether the white regions of the binary monochrome image are two circles. If the read image is two circumferential fisheye images on the left and right, the processing step migrates to S711. If the read image is not two circumferential fisheye images on the left and right, the processing of this flowchart ends.

[0094] In S711, the control unit 501 detects the center coordinates and radius of the two circumferential fisheye images of the image read in S701. In a circle, it is known that when two arbitrary straight lines intersecting two points on the circumference are drawn and perpendicular lines are drawn from the midpoints of each, the intersection point of the two perpendicular lines is the center of the circle. Therefore, for example, the control unit 501 uses the binary monochrome image created in S710 to examine the pixel values horizontally at an arbitrary vertical position for the region forming a circle, and obtains the X coordinate X1 of the pixel that changes from black to white and the X coordinate X2 of the pixel that changes from white to black. Also, the control unit 501 examines the pixel values vertically at an arbitrary horizontal position and obtains the Y coordinate Y1 of the pixel that changes from black to white and the Y coordinate Y2 of the pixel that changes from white to black. At this time, the center coordinates of the circle are such that the X coordinate is (X1 + X2) / 2 and the Y coordinate is (Y1 + Y2) / 2. Also, if the pixel value is examined horizontally from the center coordinates of the circle and the X coordinate of the pixel that changes from white to black is set as X3, the radius of the circle is X3 - ((X1 + X2) / 2).

[0095] In S712, the control unit 501 creates an orthographic cylindrical projection image from two circumferential fisheye images using the center coordinates and radii of the two circumferential fisheye images obtained by the processes of S705, S709, or S711. Here, as described with reference to FIG. 11(b), even when shooting is performed by the camera 100 equipped with the VR180 lens, the two circumferential fisheye images may not be recorded in a complete form (there may be some missing parts).

[0096] FIG. 8 is a flowchart showing details of the process of S712 (the process of creating an orthographic cylindrical projection image). In this flowchart, the control unit 501 generates two orthographic cylindrical projection images by performing an orthographic cylindrical conversion process on two circumferential fisheye images with missing areas. Here, the circumferential fisheye images are included in the RAW images (Bayer data). Therefore, in this flowchart, the control unit 501 also performs a developing process of the RAW images (a process of generating RGB data).

[0097] In S821, the control unit 501 determines whether the processing of all pixels of the left-eye conversion destination image (the orthographic cylindrical projection image corresponding to the left eye) has been completed. In this embodiment, in one processing loop of S821 to S827, in addition to the pixels to be processed in the left-eye conversion destination image, the pixels located at the same coordinates as the coordinates of the pixels to be processed in the left-eye conversion destination image in the right-eye conversion destination image (the orthographic cylindrical projection image corresponding to the right eye) are also processed. Therefore, when the processing of all pixels of the left-eye conversion destination image has been completed, the processing of all pixels of the right-eye conversion destination image has also been completed. Accordingly, when the processing of all pixels of the left-eye conversion destination image has been completed, the processing of this flowchart ends. If there are unprocessed pixels in the left-eye conversion destination image, the processing step proceeds to S822.

[0098] In S822, the control unit 501 obtains (identifies) the coordinates (left-eye conversion source coordinates) of the left-eye conversion source image (the left-eye circumferential fisheye image) corresponding to the coordinates of the pixel to be processed in the left-eye conversion destination image (left-eye conversion destination coordinates). Here, the left-eye conversion destination coordinates are in a coordinate system with the upper left of the left-eye conversion destination image as the origin. The process of obtaining the left-eye conversion source coordinates corresponding to the left-eye conversion destination coordinates can be performed using a general method of orthographic cylindrical conversion.

[0099] In S823, the control unit 501 obtains (identifies) the coordinates (left-eye transformation source coordinates) of the left-eye transformation source image (left-eye circumferential fisheye image) corresponding to the coordinates (right-eye transformation destination coordinates) of the pixel to be processed in the right-eye transformation destination image. Here, the right-eye transformation destination coordinates are in a coordinate system with the upper right of the right-eye transformation destination image as the origin. The process of obtaining the left-eye transformation source coordinates corresponding to the right-eye transformation destination coordinates can be performed using a general method of orthographic cylindrical transformation.

[0100] In S824, the control unit 501 determines whether the left-eye transformation source coordinates are within the left-eye transformation source image and whether the right-eye transformation source coordinates are within the right-eye transformation source image. If the left-eye transformation source coordinates are within the left-eye transformation source image and the right-eye transformation source coordinates are within the right-eye transformation source image, the processing step proceeds to S825. If at least one of the left-eye transformation source coordinates and the right-eye transformation source coordinates is not within the corresponding transformation source image, the processing step proceeds to S826.

[0101] In S825, the control unit 501 develops the pixel at the left-eye transformation source coordinates of the left-eye transformation source image and sets the pixel value obtained by the development to the pixel to be processed in the left-eye transformation destination image. Similarly, the control unit 501 develops the pixel at the right-eye transformation source coordinates of the right-eye transformation source image and sets the pixel value obtained by the development to the pixel to be processed in the right-eye transformation destination image.

[0102] An example of the development process in S825 will be described. The control unit 501 acquires the pixel values (in the Bayer state) of an N×N pixel (where N is a natural number) area centered on the left-eye original coordinates of the left-eye original image. Then, based on the pixel values of the N×N pixel area and the metadata necessary for RAW development acquired in S702 of FIG. 7, the control unit 501 performs a development process (interpolation process) on the pixel at the left-eye original coordinates to obtain the pixel value (RGB value) at the left-eye original coordinates. Similarly, the control unit 501 acquires the pixel values (in the Bayer state) of an N×N pixel (where N is a natural number) area centered on the right-eye original coordinates of the right-eye original image. Then, based on the pixel values of the N×N pixel area and the metadata necessary for RAW development acquired in S702 of FIG. 7, the control unit 501 performs a development process (interpolation process) on the pixel at the right-eye original coordinates to obtain the pixel value (RGB value) at the right-eye original coordinates.

[0103] In addition, in the development process using the pixel values of the N×N pixel area, when the area outside the circular fisheye image area is included in the N×N pixel area, by not using the pixels in the area outside the circular fisheye image area for the development process, more accurate pixel value calculation is possible.

[0104] On the other hand, when the processing steps shift from S824 to S826, the control unit 501 sets black for both the pixel to be processed in the left-eye destination image and the pixel to be processed in the right-eye destination image.

[0105] The upper diagram in FIG. 12 shows a circular fisheye image with a part missing, which is used as the original image for orthographic cylindrical transformation. The lower diagram in FIG. 12 shows the destination image (transformation result image) of the orthographic cylindrical transformation.

[0106] As indicated by arrow 1221, the coordinates of the left end of the left-eye orthographic cylindrical projection image correspond to the coordinates of the left end of the left-eye circumferential fisheye image, and the pixels at the coordinates of the left end of the left-eye circumferential fisheye image are recorded. On the other hand, the starting point of arrow 1222 in the right-eye orthographic cylindrical projection image is at a position corresponding to the starting point of arrow 1221 in the left-eye orthographic cylindrical projection image, but there are no recorded pixels at the position pointed to by arrow 1222 in the right-eye circumferential fisheye image (the coordinates of the left end of the right-eye circumferential fisheye image). In this case, black is set for both the starting point of arrow 1221 in the left-eye orthographic cylindrical projection image and the starting point of arrow 1222 in the right-eye orthographic cylindrical projection image.

[0107] As indicated by arrow 1223, the coordinates of the right end of the right-eye orthographic cylindrical projection image correspond to the coordinates of the right end of the right-eye circumferential fisheye image, and the pixels at the coordinates of the left end of the right-eye circumferential fisheye image are recorded. On the other hand, the starting point of arrow 1224 in the left-eye orthographic cylindrical projection image is at a position corresponding to the starting point of arrow 1223 in the right-eye orthographic cylindrical projection image, but there are no recorded pixels at the position pointed to by arrow 1224 in the left-eye circumferential fisheye image (the coordinates of the right end of the left-eye circumferential fisheye image). In this case, black is set for both the starting point of arrow 1223 in the right-eye orthographic cylindrical projection image and the starting point of arrow 1224 in the left-eye orthographic cylindrical projection image.

[0108] In this way, for the left-eye orthographic cylindrical projection image, in addition to the region corresponding to the missing region of the left-eye circumferential fisheye image (the end point of arrow 1224) (the start point of arrow 1224), black is also set in the region corresponding to the region (the end point of arrow 1221) located at the position corresponding to the missing region of the right-eye circumferential fisheye image (the end point of arrow 1222) (the start point of arrow 1221). Similarly, for the right-eye orthographic cylindrical projection image, in addition to the region corresponding to the missing region of the right-eye circumferential fisheye image (the end point of arrow 1222) (the start point of arrow 1222), black is also set in the region corresponding to the region (the end point of arrow 1223) located at the position corresponding to the missing region of the left-eye circumferential fisheye image (the end point of arrow 1224) (the start point of arrow 1223). Thereby, it is possible to suppress the occurrence of a region of black pixels in only one of the left and right orthographic cylindrical projection images. Therefore, the image quality when the user views the left and right orthographic cylindrical projection images as one stereoscopic image using a head-mounted display (HMD) can be improved.

[0109] In the example of FIG. 12, one RAW image includes the region of the RAW image (the first RAW image) including the region of the left-eye circumferential fisheye image (the first circumferential fisheye image) and the region of the RAW image (the second RAW image) including the region of the right-eye circumferential fisheye image (the second circumferential fisheye image). However, the control unit 501 may acquire the RAW image (the first RAW image) including the region of the left-eye circumferential fisheye image (the first circumferential fisheye image) and the region of the RAW image (the second RAW image) including the region of the right-eye circumferential fisheye image (the second circumferential fisheye image) as separate RAW images.

[0110] In S827, the control unit 501 selects the following pixel as the pixel to be processed for each of the left-eye converted image and the right-eye converted image. For example, when processing from the upper-left pixel to the lower-right pixel of the left-eye converted image, the following pixel is the pixel adjacent to the right in the same row. However, when the X coordinate of the pixel adjacent to the right reaches the width of the left-eye converted image, the leftmost pixel of the left-eye converted image in the row one row below is set as the following pixel. This is the same for the right-eye converted image.

[0111] Through the above processing, two orthographic cylindrical projection images are generated from two circumferential fisheye images with missing areas (missing parts).

[0112] According to the creation process of the orthographic cylindrical projection image shown in FIG. 8, the control unit 501 develops the RAW image including the two circumferential fisheye images, but does not develop the pixels outside the area of the circumferential fisheye image. In other words, the control unit 501 does not develop the pixels in the area that is not referenced as the conversion source coordinates for the orthographic cylindrical conversion process (the black area in the upper figure of FIG. 12). This makes it possible to reduce the processing load of development.

[0113] In addition, the control unit 501 does not develop the pixels of the specific part (the end point of arrow 1221) of the left-eye circumferential fisheye image at the position corresponding to the missing part (the end point of arrow 1222) of the right-eye circumferential fisheye image. The reason is that since black is set at S826 in the area of the left-eye orthographic cylindrical projection image corresponding to the specific part (the end point of arrow 1221) of the left-eye circumferential fisheye image (the start point of arrow 1221), the developed pixel values for the specific part are unnecessary. Similarly, the control unit 501 does not develop the pixels of the specific part (the end point of arrow 1223) of the right-eye circumferential fisheye image at the position corresponding to the missing part (the end point of arrow 1224) of the left-eye circumferential fisheye image. This makes it possible to further reduce the processing load of development.

[0114] Incidentally, according to the creation process of the orthographic cylindrical projection image shown in FIG. 8, in the processing loop of S821 to S827, the control unit 501 performs development processing and orthographic cylindrical conversion processing for each pixel to be processed. In other words, the control unit 501 executes the development processing and the orthographic cylindrical conversion processing in parallel. However, the control unit 501 may perform the development processing on the entire RAW image before the orthographic cylindrical conversion processing. In this case, the control unit 501 may set a predetermined pixel value (for example, a pixel value indicating black) for the pixels in the area outside the development processing target. Here, the area outside the development processing target refers to the black area in the upper figure of FIG. 12, a specific part of the right-eye circumferential fisheye image at the position corresponding to the missing part of the left-eye circumferential fisheye image, and a specific part of the left-eye circumferential fisheye image at the position corresponding to the missing part of the right-eye circumferential fisheye image. Whether each pixel of the RAW image is within the circumferential fisheye area can be determined based on whether the pixel values of all four pixels of the corresponding Bayer array are below a certain level.

[0115] ● Image file format Next, with reference to FIG. 9, the structure of the image file (the image file processed by the PC 500) recorded by the camera 100 will be described. In FIG. 9, 901 is the header of the file. Information such as the type of the image is recorded in the header of the file. 902 is the shooting information at the time of shooting the image recorded in the file. Information such as the shutter speed and aperture is recorded in the shooting information. 903 is the VR180 lens metadata (metadata related to the VR180 lens). As the VR180 lens metadata, information such as the lens name, the radius of the circumferential fisheye, and information on manufacturing errors is recorded. 904 is the metadata other than the VR180 lens. In the case of a video, information that changes for each frame is recorded in the metadata other than the VR180 lens, and in the case of RAW, data necessary for development is recorded. A camera equipped with firmware that does not support the VR180 lens records the lens name in the metadata other than the VR180 lens. 905 is the data of a still image or a video. In the case of a video, audio data is also recorded in addition to the image.

[0116] In the example of FIG. 9, shooting information 902, VR180 lens metadata 903, and metadata 904 other than the VR180 lens are recorded in the image file. However, a configuration may be adopted in which the shooting information 902, VR180 lens metadata 903, and metadata 904 other than the VR180 lens are recorded in a file different from the image file, and the recorded information is associated with the image file.

[0117] ● Information Obtained from VR180 Lens 300 and Camera 100 With reference to FIG. 10(a), an example of information obtained by camera 100 from VR180 lens 300 will be described. The following information is obtained from VR180 lens 300. 1. Lens design value 2. Lens individual value (manufacturing error value) 3. Lens flag information 4. Lens focal length information 5. Lens thermometer information

[0118] The lens design value is design value data common to both eyes of the VR180 for aberration correction. Details of the lens design value will be described later with reference to FIG. 10(b).

[0119] The lens individual value (manufacturing error value) is data indicating manufacturing errors of VR180 lens 300. In the manufacturing process of VR180 lens 300, errors occur on the left and right sides respectively. When performing orthogonal cylindrical conversion processing by PC500 with errors present, the quality of 3D display deteriorates. To suppress this problem, measurement results of errors detected during the lens manufacturing process are recorded in the lens. Details of the lens individual value will be described later with reference to FIG. 10(b).

[0120] When performing image processing, PC500 needs to determine whether to perform conversion processing on the image data captured by VR180 lens 300. The lens flag is recorded to enable this determination. The lens flag is a flag indicating that the image data is image data captured by the VR180 lens.

[0121] The lens focal length is the distance from the "principal point", which is the center of the lens, to the imaging on the image sensor. When performing conversion processing by the PC500 on the image data captured with the VR180 lens 300, in order to maintain the quality of 3D display, a highly accurate (down to the decimal point) value is required as the lens focal length.

[0122] The lens temperature information indicates the temperature of the VR180 lens 300. Since the temperature information of the lens is recorded during shooting, the PC500 can grasp the ambient temperature.

[0123] Referring to FIG. 10(b), the lens design values and lens individual values (manufacturing error values) will be described. These pieces of information are used for processes such as left-right swapping processing and orthographic cylindrical conversion processing. The lens design values include, for example, the following information. 1. Image circle position 2. Image circle diameter 3. Angle of view 4. Distortion correction coefficient

[0124] The image circle position is the design value of the center position of the image circle formed on the image sensor. For example, the image circle position is defined as coordinates in the horizontal and vertical directions with the mount as the origin.

[0125] The image circle diameter is the diameter of the image circle formed on the image sensor.

[0126] The angle of view is the angular range of the image formed within the image circle.

[0127] The distortion correction coefficient is the ratio of the designed image height to the ideal image height of the lens. Values can be set for each image height, and interpolation can be performed between them, or they can be approximated by a polynomial.

[0128] Also, when connecting and displaying the application on the PC500 with the camera 100, a magic window display is performed on the screen of the PC500. In that case, in order to maintain the posture and display quality, the "image circle position", "image circle diameter", and "field angle" are used. The PC500 edits and uses this information as information adapted to the application in order to appropriately display the image on the screen. For example, the PC500 multiplies a coefficient to the "image circle position" and "image circle diameter" according to the magic window display and uses them.

[0129] The lens individual value (manufacturing error value) includes, for example, the following information. 5. Image circle position deviation 6. Optical axis tilt 7. Image magnification deviation

[0130] The image circle position deviation is the deviation from the designed value of the center position of the image circle formed on the imaging device. For example, the image circle position deviation is defined by coordinates in the horizontal and vertical directions with the designed position as the origin.

[0131] The optical axis tilt is the deviation of the direction of the optical axis on the subject side. For example, the tilt deviation in the horizontal direction and the tilt deviation in the vertical direction are described in angles.

[0132] The image magnification deviation is the deviation from the designed value of the size of the image. For example, the image magnification deviation is described as a ratio to the designed value.

[0133] Since these individual values are generated due to manufacturing errors such as eccentricity and tilt of the lens, they have various values for each individual. Therefore, the lens individual values are measured and recorded for each of the left and right optical systems.

[0134] Referring to FIG. 10(c), the metadata acquired from the camera 100 will be described. This metadata is used to maintain the 3D quality based on the display area and posture information when performing conversion processing in the application of the PC500. The metadata includes, for example, the following information. 1. Camera recording area information 2. Camera internal accelerometer information 3. Aperture value right-eye correction information

[0135] The camera recording area information refers to the effective image area. Depending on the camera's imaging device and recording mode, the displayable effective image area is different. This information is used to perform more accurate image display when performing conversion processing in the PC500 application.

[0136] The camera internal accelerometer information refers to the roll and pitch of the attitude information measured using the accelerometer (level) inside the camera. This information is used for electronic image stabilization and horizontal correction in the PC500 application.

[0137] The aperture value right-eye correction information refers to the exposure setting value when based on the left eye. This information is used to maintain 3D quality and perform playback display without a sense of discomfort when performing conversion processing in the PC500 application.

[0138] As described above, according to the first embodiment, the PC500 acquires a RAW image (for example, the upper figure in FIG. 12) including the area of the circumferential fisheye image, and develops the acquired RAW image. In developing the RAW image, the PC500 does not develop the pixels outside the area of the circumferential fisheye image in the RAW image (that is, the pixels in the area not referred to as the conversion source coordinates of the orthographic cylindrical conversion process (for example, the pixels in the black area in the upper figure of FIG. 12)). Thereby, it becomes possible to reduce the processing load of development.

[0139] Also, the PC500 may also omit the development of the pixels in the specific part of the first circumferential fisheye image (for example, the area of the left-eye circumferential fisheye image corresponding to the end point of arrow 1221 in FIG. 12) that is at the position corresponding to the missing part of the second circumferential fisheye image (for example, the missing part of the right-eye circumferential fisheye image corresponding to the end point of arrow 1222 in FIG. 12). Thereby, it becomes possible to further reduce the processing load of development.

[0140] [Second Embodiment] In the first embodiment, the configuration in which the PC 500 acquires a RAW image from an image file including the RAW image has been described. In the second embodiment, a configuration in which the PC 500 acquires a RAW image by receiving a video signal including the RAW image from the camera 100 (external device) according to a predetermined video signal transmission standard will be described. In this embodiment, the basic configurations of the camera 100 and the PC 500 are the same as those in the first embodiment. Hereinafter, mainly the differences from the first embodiment will be described.

[0141] Hereinafter, the HDMI standard will be used as the video signal transmission standard for description, but the video signal transmission standard of this embodiment is not limited to the HDMI standard, and for example, the DisplayPort standard may be used.

[0142] ● Overall system configuration FIG. 15 is a diagram showing the overall configuration of the system according to the second embodiment. In FIG. 15, reference numeral 190 denotes an HDMI cable that transmits the video signal output from the camera 100. The video signal from the camera 100 is input to the PC 500 via the external I / F 508. In the PC 500, the video signal can be displayed in real time on the display of the PC 500. Further, the PC 500 can convert the video signal into a streamable format and directly transmit the video to the HMD 191 (head-mounted display) from the PC 500, or distribute the video to a plurality of users 193 via the cloud 192.

[0143] ● Video signal output processing by the camera 100 Next, with reference to FIGS. 13A and 13B, the video signal output processing by the camera 100 will be described. When the user turns on the power of the camera 100, the processes of the flowcharts in FIGS. 13A and 13B start.

[0144] The processes of S1301 to S1306 are the same as the processes of S601 to S606 in FIG. 6A, and thus the description thereof will be omitted.

[0145] In S1307, the system control unit 50 determines whether an HDMI cable is connected to the HDMI terminal (video signal output I / F 240). If it is determined that the HDMI cable is connected, the processing step proceeds to S1308. If it is determined that the cable is not connected, the processing step proceeds to S1305.

[0146] The processing in S1308 - S1309 is the same as the processing in S1308 - S1309 in FIG. 6A, so the description is omitted.

[0147] In S1310, the system control unit 50 stores the data acquired in S1303, S1304, S1308, and S1309 in a buffer for HDMI signal output in a predetermined format.

[0148] In S1311, the system control unit 50 outputs the data stored in S1310 as an HDMI signal.

[0149] In S1312, the system control unit 50 determines whether the HDMI cable connected to the HDMI terminal (video signal output I / F 240) has been unplugged. If it is determined that the HDMI cable has been unplugged, the processing of this flowchart ends. If the HDMI cable has not been unplugged, the processing step proceeds to S1308. By repeatedly performing the processing in S1308 - S1312 by the system control unit 50, the camera 100 can continue to output an HDMI signal.

[0150] Next, the case where the processing step proceeds from S1301 or S1302 to S1322 will be described. When the processing step proceeds from S1301 to S1322, the firmware of the camera 100 does not support the VR180 lens. Therefore, the camera 100 cannot perform processing specific to the VR180 lens. Thus, even if the VR180 lens is attached to the camera 100, the camera 100 performs the same processing as when a normal lens is attached.

[0151] The processing in S1322 - S1327 is the same as the processing in S622 - S627 in FIG. 6B, so the description is omitted.

[0152] In S1328, the system control unit 50 determines whether an HDMI cable is connected to the HDMI terminal (video signal output I / F 240). If it is determined that the HDMI cable is connected, the processing step proceeds to S1329. If it is determined that the cable is not connected, the processing step proceeds to S1326.

[0153] The processes of S1329 to S1330 are the same as the processes of S629 to S630 in FIG. 6B, so the description thereof is omitted.

[0154] In S1331, the system control unit 50 determines whether general information of the lens is stored in the system memory 218 (see S1325). If the general information of the lens is stored, the processing step proceeds to S1332. If the general information of the lens is not stored, the processing step proceeds to S1333.

[0155] In S1332, the system control unit 50 acquires the general information of the lens stored in the system memory 218.

[0156] In S1333, the system control unit 50 stores the data acquired in S1329, S1330, and S1332 in a predetermined format in the buffer for HDMI signal output.

[0157] In S1334, the system control unit 50 outputs the data stored in S1333 as an HDMI signal.

[0158] In S1335, the system control unit 50 determines whether the HDMI cable connected to the HDMI terminal (video signal output I / F 240) has been unplugged. If it is determined that the HDMI cable has been unplugged, the processing of this flowchart ends. If the HDMI cable has not been unplugged, the processing step proceeds to S1329. By repeatedly performing the processes of S1329 to S1335 by the system control unit 50, the camera 100 can continue to output an HDMI signal.

[0159] Note that the RAW images captured in the flowcharts of FIGS. 13A and 13B may be still images (RAW still images) or moving images (RAW moving images).

[0160] ● Development processing and orthographic cylindrical conversion processing by PC500 With reference to FIG. 14, the development processing and orthographic cylindrical conversion processing by PC500 will be described. PC500 performs development processing and orthographic cylindrical conversion processing on the RAW image included in the video signal (HDMI signal) received from the camera 100. Through the development processing and orthographic cylindrical conversion processing, two left and right orthographic cylindrical projection images are generated from the two left and right circumferential fisheye images as described with reference to FIGS. 11(a) and 11(b). Note that, in the example of FIG. 14, the development processing is included in the processing of S1412 (creation processing of orthographic cylindrical projection image).

[0161] In S1401, the control unit 501 determines whether an HDMI signal has been received from an external device (camera 100). If an HDMI signal has been received, the process proceeds to processing step S1402. If an HDMI signal has not been received, the control unit 501 repeats the process of S1401.

[0162] In S1402, the control unit 501 reads the RAW image, shooting information, and metadata other than the VR180 lens from the leading frame of the received HDMI signal.

[0163] In S1403, the control unit 501 determines whether VR180 lens metadata is superimposed on the HDMI signal. If VR180 lens metadata is superimposed on the HDMI signal, the process proceeds to processing step S1404. If VR180 lens metadata is not superimposed on the HDMI signal, the process proceeds to processing step S1406.

[0164] In S1404, the control unit 501 reads the VR180 lens metadata from the HDMI signal.

[0165] The process of S1405 is the same as the process of S705 in FIG. 7, so the description is omitted.

[0166] When the processing step shifts from S1403 to S1406, the control unit 501 determines whether a lens name is superimposed on the HDMI signal. For example, when the firmware of the camera 100 does not support the VR180 lens, the camera 100 does not superimpose the VR180 lens metadata, but it is possible to superimpose the lens name as metadata other than the VR180 lens. When a lens name is superimposed on the HDMI signal, the processing step shifts to S1407. When a lens name is not superimposed on the HDMI signal, the processing step shifts to S1409.

[0167] In S1407, the control unit 501 acquires the lens name from the HDMI signal.

[0168] In S1408, the control unit 501 determines whether the lens name acquired in S1407 indicates a known VR180 lens. When the lens name acquired in S1407 indicates a known VR180 lens, the processing step shifts to S1409. When the lens name acquired in S1407 does not indicate a known VR180 lens, the processing step shifts to S1410.

[0169] The process of S1409 is the same as the process of S709 in FIG. 7, so the description is omitted.

[0170] When the processing step shifts from S1406 or S1408 to S1410, the control unit 501 determines whether the image read in S1402 is two circumferential fisheye images on the left and right. For example, regarding the luminance value of the pixels of the read image, when the value is below the threshold (a value close to black), the pixel value of that pixel is set to black, and when the value exceeds the threshold, the pixel value of that pixel is set to white, to create a binary monochrome image. Then, the control unit 501 can determine whether the image read in S1402 is two circumferential fisheye images on the left and right based on whether the white regions of the binary monochrome image are two circles. If the read image is two circumferential fisheye images on the left and right, the processing step shifts to S1411. If the read image is not two circumferential fisheye images on the left and right, the processing of this flowchart ends.

[0171] The processing of S1411 is the same as the processing of S711 in FIG. 7, so the description is omitted.

[0172] In S1412, the control unit 501 creates an orthographic cylindrical projection image from the two circumferential fisheye images using the center coordinates and radii of the two circumferential fisheye images obtained by the processing of S1405, S1409, or S1411. The details of the processing of S1412 are the same as the processing of S712 in FIG. 7 (that is, the processing of the flowchart in FIG. 8), so the description is omitted.

[0173] For subsequent frames of the HDMI signal, the control unit 501 repeatedly executes the processing of the flowchart in FIG. 8 until the stop of the HDMI signal is detected.

[0174] Through the above processing, two orthographic cylindrical projection images are generated from two circumferential fisheye images with missing regions (missing parts).

[0175] As described above, according to the second embodiment, the PC 500 receives, from the camera 100, a video signal including a RAW image including the area of the circumferential fisheye image in accordance with the HDMI standard, and acquires the RAW image from the received video signal. Then, the PC 500 develops the RAW image in the same manner as in the first embodiment, omitting development for some pixels. Therefore, even when the PC 500 acquires the RAW image from the video signal (instead of the image file), it is possible to reduce the processing load of development in the same manner as in the first embodiment.

[0176] In addition, in each of the above-described embodiments, a configuration has been described in which the development process of the RAW image, the left-right swapping of the circumferential fisheye image, and the orthographic cylindrical conversion process are performed at once. However, a configuration may be adopted in which the left-right swapping of the circumferential fisheye image is performed in the state of the Bayer image (RAW image), and then the development process and the orthographic cylindrical conversion process are performed. Alternatively, a configuration may be adopted in which the left-right swapping is performed in the state of the RGB image after performing the development process and the orthographic cylindrical conversion for the necessary pixels.

[0177] [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in the computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0178] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are attached to disclose the scope of the invention.

Description of Reference Numerals

[0179] 500...PC, 501...Control Unit, 502...ROM, 503...RAM, 504...External Storage Device, 505...Operation Unit, 506...Display Unit, 507...Communication Unit, 508...External I / F, 509...System Bus

Claims

1. An acquisition means for acquiring a first RAW image including a region of a first circumferential fisheye image and a second RAW image including a region of a second circumferential fisheye image having a parallax with respect to the first circumferential fisheye image and having a missing portion where pixel values do not exist; A developing means for developing the first RAW image; Comprising: The developing means does not develop pixels outside the region of the first circumferential fisheye image in the first RAW image and pixels of a specific portion of the first circumferential fisheye image located at positions corresponding to the missing portions of the second circumferential fisheye image. An image processing apparatus characterized by the above.

2. The acquisition means acquires the first RAW image and the second RAW image by acquiring one RAW image including the region of the first RAW image and the region of the second RAW image. The image processing apparatus according to claim 1, characterized by the above.

3. The developing means sets a predetermined pixel value for pixels in the first RAW image that are not developed. The image processing apparatus according to claim 1 or 2, characterized by the above.

4. Further comprising a generating means for generating an orthographic cylindrical projection image by performing orthographic cylindrical transformation processing based on the first circumferential fisheye image, In the orthographic cylindrical transformation processing, the generating means identifies pixels of the first circumferential fisheye image corresponding to processing target pixels of the orthographic cylindrical projection image, and sets pixel values obtained by developing the identified pixels for the processing target pixels of the orthographic cylindrical projection image. The image processing apparatus according to any one of claims 1 to 3, characterized by the above.

5. The acquisition means acquires the first RAW image from an image file including the first RAW image. The image processing apparatus according to any one of claims 1 to 4, characterized by the above.

6. Further comprising a receiving means for receiving a video signal including the first RAW image from an external device according to a predetermined video signal transmission standard, The acquisition means acquires the first RAW image from the video signal. The image processing apparatus according to any one of claims 1 to 4, characterized by the above.

7. The circumferential fisheye image is a circular region included in the RAW image. The image processing apparatus according to any one of claims 1 to 6, characterized by the above.

8. An image processing method executed by an image processing apparatus, An acquisition step of acquiring a first RAW image including an area of a first circumferential fisheye image, and a second RAW image including an area of a second circumferential fisheye image having a parallax with respect to the first circumferential fisheye image and having a missing portion where pixel values do not exist A developing step of developing the first RAW image Comprising In the developing step, pixels outside the area of the first circumferential fisheye image in the first RAW image and pixels of a specific portion of the first circumferential fisheye image at positions corresponding to the missing portions of the second circumferential fisheye image are not developed An image processing method characterized by this

9. A program for causing a computer to function as each means of the image processing apparatus according to any one of Claims 1 to 7

Images