Information processing system, information processing method, and program

Abstract

To implement both of confirmation of presence and the like and confirmation of a detail of an imaging target when a user views an image, with a simple configuration.SOLUTION: An information processing system includes: first acquisition means for acquiring a first image including a first region of an image obtained by performing geometrical conversion to a first imaged image acquired by imaging via a first optical system and a second region of an image obtained by performing the geometrical conversion to a second imaged image acquired by imaging via a second optical system; second acquisition means for acquiring a second image having image quality higher than that of the first image on the basis of an image obtained by cutting a first range of part of any of the first imaged image and the second imaged image; and control means for controlling display means so as to display a third image obtained by combining the first image and the second image by arranging the second image into a range corresponding to the first range of the first image.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] In recent years, devices that can simultaneously capture two images with parallax between them by having two optical systems in one lens mount have been put into practical use. After undergoing image processing, these two images are reproduced on a device (such as VR goggles) so that the user can perceive them as a three-dimensional image.

[0003] In recent years, with the digitalization of cameras, it has become possible for a display device to display in real time images captured by a digital camera in real space. In addition, by using VR goggles as a display device, it is also possible to display images being captured by the VR goggles.

[0004] At the shooting site of VR video, users want to check the realism of the shot video and the details of the subject (focus, texture, etc.) using only the lightest equipment possible. However, in order to preview VR video content in stereoscopic vision, geometric transformations such as left-right swapping and equirectangular transformation are required, and the resolution of VR video is very high. For this reason, in order for users to check high-quality video, it takes a very long time to process and requires expensive and large equipment (such as a high-spec computer with high processing power).

[0005] Patent Document 1 discloses a system that enables a high-resolution image of interest to be displayed even when communication is performed with a small transfer bandwidth by synthesizing and displaying multiple live view images with different resolutions on the viewer side. Patent Document 2 discloses an imaging device that has an optical system capable of acquiring images with different angles of view and synthesizes these images in order to make it easier for users to check the composition. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2006-340091 A [Patent Document 2] JP 2014-132771 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, while these technologies enable a user to check the details of the subject being photographed and the image around the point of interest using images of different resolutions, they do not allow the user to check the realism or three-dimensionality of the image.

[0008] Therefore, an object of the present invention is to realize, with a simple configuration, a system that allows a user to check both the sense of realism and the details of a subject when viewing an image. [Means for solving the problem]

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: a first acquisition means for acquiring a first image including a first region of an image obtained by performing a geometric transformation on a first captured image acquired by imaging via a first optical system, and a second region of an image obtained by performing the geometric transformation on a second captured image acquired by imaging via a second optical system; A first range is cut out from a part of either the first captured image or the second captured image. a second acquisition means for acquiring a second image having a higher image quality than the first image based on the acquired image; a control means for controlling a display means to display a third image obtained by combining the first image and the second image by arranging the second image in a range of the first image corresponding to the first range; The information processing system is characterized by having the following features.

[0010] One aspect of the present invention is a method for producing a composition comprising the steps of: a first acquisition step of acquiring a first image including a first region of an image obtained by performing a geometric transformation on a first captured image acquired by imaging via a first optical system, and a second region of an image obtained by performing the geometric transformation on a second captured image acquired by imaging via a second optical system; a second acquisition step of acquiring a second image having a higher image quality than the first image based on an image obtained by cutting out a first range from either the first captured image or the second captured image; a control step of controlling a display means to display a third image obtained by combining the first image and the second image by arranging the second image in a range of the first image corresponding to the first range; The information processing method is characterized by having the following features. Effect of the Invention

[0011] According to the present invention, with a simple configuration, when a user views an image, it is possible to simultaneously confirm the sense of realism and the like and the details of the subject being photographed. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] 1 is an external view of a camera according to a first embodiment. [Diagram 3] FIG. 1 is a diagram showing the internal configuration of a camera according to a first embodiment. [Figure 4] FIG. 1 is a diagram showing a lens unit according to a first embodiment. [Diagram 5] FIG. 1 is a diagram illustrating a display control device according to a first embodiment. [Figure 6] FIG. 2 is a diagram illustrating a controller according to the first embodiment. [Figure 7] 4 is a flowchart of processing in the camera according to the first embodiment. [Figure 8] 4 is a flowchart of processing of the VR goggles according to the first embodiment. [Figure 9]FIG. 2 is a diagram illustrating a live view image according to the first embodiment. [Figure 10] 10A to 10C are diagrams illustrating left-right swapping processing according to the first embodiment. [Figure 11] FIG. 4 is a diagram showing an image after geometric transformation according to the first embodiment. [Figure 12] FIG. 2 is a diagram illustrating a display image according to the first embodiment. [Figure 13] 11 is a flowchart of an image request transmission process according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0014] <Embodiment 1> FIG. 1 is a diagram showing an information processing system 1 according to a first embodiment. The information processing system 1 has a digital camera 100 and VR goggles 530. The digital camera 100 (hereinafter referred to as "camera") is an imaging device (image recording device). The lens unit 300 is attached to the camera 100. The VR goggles 530 is a display device including a display control device 500.

[0015] The camera 100 and the lens unit 300 are in a state in which they can communicate with each other. The camera 100 transmits lens information to the camera 100. The camera 100 and the VR goggles 530 are connected wirelessly or by wire, and transmit and receive various data (lens information, live view images, etc.). When the camera 100 transmits a live view image to the VR goggles 530, the display 505 of the VR goggles 530 displays a display image based on the live view image. At this time, in response to a user's instruction, the CPU 501 of the VR goggles 530 performs image processing on the live view image, and displays the processing result on the display 505 of the VR goggles 530.

[0016] In the first embodiment, an example is shown in which the VR goggles 530 and the camera 100 are connected wirelessly. However, the VR goggles 530 and the camera 100 may be connected by wire. In addition, an example is shown in which the VR goggles 530 are connected to the camera 100, but the present invention is not limited to this. Instead of the VR goggles 530, any external device may be used as long as it is a display control device (display device) that can realize a function of displaying a VR180 image.

[0017] (Explanation of the camera's external configuration) 2A and 2B are diagrams showing an example of the external configuration of camera 100. Fig. 2A is a perspective view of camera 100 as seen from the front, and Fig. 2B is a perspective view of camera 100 as seen from the back.

[0018] The camera 100 has a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub electronic dial 105, a movie button 106, and a viewfinder display 107 on the top surface. The shutter button 101 is an operation unit for preparing for shooting or issuing a shooting instruction. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode switch 103 is an operation unit for switching between various modes. The main electronic dial 104 is a rotary operation unit for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation unit for moving a selection frame (cursor), forwarding images, etc. The movie button 106 is an operation unit for issuing an instruction to start or stop movie shooting (recording). The viewfinder display 107 displays various settings such as shutter speed and aperture.

[0019] The camera 100 has a display unit 108, a touch panel 109, a direction key 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, a menu button 115, an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar 119 on the rear surface.

[0020] The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are an operation unit that includes keys (four-way keys) that can be pressed up, down, left, and right. Operations can be performed according to the positions of the directional keys 110 that are pressed. The SET button 111 is an operation unit that is pressed mainly to confirm a selection item. The AE lock button 112 is an operation unit that is pressed to fix an exposure state in a shooting standby state. The enlargement button 113 is an operation unit for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced when the main electronic dial 104 is operated. The enlargement button 113 is also used to enlarge a playback image or increase the magnification in the playback mode. The playback button 114 is an operation unit for switching between the shooting mode and the playback mode. In the shooting mode, pressing the playback button 114 switches to the playback mode, and the latest image among the images recorded on the recording medium 227 described later can be displayed on the display unit .

[0021] The menu button 115 causes the display unit 108 to display a menu screen on which various settings can be made. The display unit 108 is an operation unit that is pressed when the user wants to change the camera settings. The user can intuitively make various settings by looking at a menu screen displayed on the display unit 108 and using the direction keys 110 and the SET button 111. The eyepiece unit 116 is a part for placing the eye on an eyepiece finder (a peer-type finder) 117. The user can view an image displayed on an internal EVF 217 (Electronic View Finder) described later through the eyepiece unit 116. The eyepiece detection unit 118 is a sensor that detects whether the user places the eye on the eyepiece unit 116.

[0022] The touch bar 119 is a line-shaped touch operation unit (line touch sensor) capable of receiving a touch operation. The touch bar 119 is disposed at a position where it can be touched (touched) by the thumb of the right hand when the grip unit 120 is held by the right hand (holding it with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed by the index finger of the right hand. That is, the touch bar 119 can be operated in a state (shooting posture) where the eyepiece unit 116 is placed close to the eyepiece 116, the eyepiece viewfinder 117 is looked into, and the user is ready to press the shutter button 101 at any time. The touch bar 119 can receive a tap operation (operation of touching and releasing without moving within a predetermined period of time) on the touch bar 119, a slide operation to the left and right (operation of touching and then moving the touch position while keeping the touch), and the like. The touch bar 119 is an operation unit different from the touch panel 109, and does not have a display function. The touch bar 119 of this embodiment is a multi-function bar, and functions as, for example, an M-Fn bar.

[0023] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a cover 123, a communication terminal 124, and the like. The grip section 120 is a holding section formed in a shape that is easy to hold with the right hand when the user holds the camera 100. The shutter button 101 and the main electronic dial 104 are arranged at positions that can be operated with the index finger of the right hand when the camera 100 is held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand. In a similar state, the sub electronic dial 105 and the touch bar 119 are arranged at positions that can be operated with the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section provided on the rear side of the camera 100 at a position where it is easy to place the thumb of the right hand that holds the grip section 120 when none of the operation sections are being operated. The thumb rest section 121 is made of a rubber member or the like for increasing the holding force (grip feeling). The terminal cover 122 protects connectors such as a connection cable that connects the camera 100 to an external device. The cover 123 protects the recording medium 227 and the slot by closing the slot for storing the recording medium 227 described later. The communication terminal 124 is a terminal for communicating with the detachable lens unit 200 described later of the camera 100.

[0024] (Explanation of the internal structure of the camera) Fig. 3 is a diagram showing an example of the internal configuration of the camera 100. Note that the same components as those in Fig. 2 are given the same reference numerals and descriptions thereof will be omitted where appropriate. A lens unit 200 is attached to the camera 100.

[0025] First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that is detachable from the camera 100. The lens unit 200 is a single lens, and is an example of a normal lens. The lens unit 200 has an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0026] The aperture 201 is configured so that the aperture diameter can be adjusted. The lens 202 is composed of a plurality of lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the aperture drive circuit 203 based on instructions from the system control unit 50, which will be described later. , and an AF drive circuit 204. The lens system control circuit 205 controls the aperture 201 via an aperture drive circuit 203, and adjusts the focus by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, the camera 100 and the lens unit 200 communicate with each other via a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100.

[0027] Next, a description will be given of the camera 100. The camera 100 has a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0028] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on an instruction from the system control unit 50. The imaging unit 211 is an imaging element (image sensor) composed of a CCD or CMOS element that converts an optical image into an electric signal. The imaging unit 211 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts an analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion processing, etc.) on data from the A / D converter 212 or data from the memory control unit 213. In addition, the image processing unit 214 performs predetermined arithmetic processing using the captured image data. Based on the obtained arithmetic result, the system control unit 50 performs exposure control or distance measurement control. Through this processing, TTL (through-the-lens) type AF processing, AE (automatic exposure) processing, EF (flash pre-emission) processing, etc. are performed. Furthermore, the image processing unit 214 performs a predetermined calculation process using the captured image data, and performs TTL type AWB (auto white balance) processing based on the obtained calculation results.

[0029] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores "image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212" and "image data to be displayed on the display unit 108 or EVF 217." The memory 215 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 215 also serves as a memory for displaying images (video memory).

[0030] When a live view image is transferred to the VR goggles 530 (display control device 500), the memory 215 also stores the live view image with adjusted resolution, and a high-resolution image (cropped image) cropped from a specified position of the live view image.

[0031] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 or the EVF 217. Therefore, the display image data written in the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display according to the analog signal from the D / A converter 216. The display unit 108 or the EVF 217 is, for example, an LCD or an organic EL display. The digital data A / D converted by the A / D converter 212 is stored in the memory 215 as a digital signal. The digital signal is converted into an analog signal in the D / A converter 216. The analog signal is then sequentially transferred to and displayed on the display unit 108 or the EVF 217, thereby performing a live view display.

[0032] The system control unit 50 includes at least one processor and / or at least one The system control unit 50 is a control unit made up of a circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 executes a program recorded in the non-volatile memory 219 to realize each process of the flowchart described below. The system control unit 50 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, etc.

[0033] The camera 100 also includes a system memory 218 , a non-volatile memory 219 , a system timer 220 , a communication unit 221 , an attitude detection unit 222 , and an eye proximity detection unit 118 .

[0034] The system memory 218 may be, for example, a RAM. Constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are deployed in the system memory 218. The nonvolatile memory 219 is an electrically erasable and recordable memory. For example, an EEPROM may be used for the nonvolatile memory 219. Constants and programs for the operation of the system control unit 50 are recorded in the nonvolatile memory 219. The programs here are programs for executing the flowcharts described below. The system timer 220 is a timing unit that measures times used for various controls and the time of a built-in clock.

[0035] The communication unit 221 transmits and receives video signals and audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 221 can also connect to a wireless LAN (Local Area Network) or the Internet. The communication unit 221 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded in the recording medium 227. The communication unit 221 can receive image data and various other information from an external device.

[0036] The attitude detection unit 222 detects the attitude of the camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 is an image captured by holding the camera 100 horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211. In addition, the system control unit 50 can rotate and record the image according to the attitude detected by the attitude detection unit 222. For example, an acceleration sensor or a gyro sensor can be used for the attitude detection unit 222. It is also possible to detect the movement of the camera 100 (panning, tilting, lifting, whether or not it is stationary, etc.) using the attitude detection unit 222.

[0037] The eyepiece detection unit 118 can detect the approach of an object to the eyepiece unit 116 (the eyepiece unit 116 of the eyepiece finder 117 incorporating the EVF 217). For example, an infrared proximity sensor can be used for the eyepiece detection unit 118. When an object approaches, infrared light projected from a light projecting unit of the eyepiece detection unit 118 is reflected by the object and received by a light receiving unit of the infrared proximity sensor. The eyepiece detection unit 118 can determine the distance from the eyepiece unit 116 to the object based on the amount of infrared light received. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity of the object to the eyepiece unit 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (approach) and departure (away) of an eye (object) to the eyepiece unit 116 of the eyepiece finder 117. The eye contact detection unit 118 detects that an object has been placed near the eye when an object approaching within a predetermined distance of the eye piece unit 116 from a non-eye contact state (non-approaching state) is detected. On the other hand, the eye contact detection unit 118 detects that an object that was detected as approaching from the eye contact state (approaching state) has moved away by more than a predetermined distance. The threshold for detecting the placement of the eye and the threshold for detecting the placement of the eye may be different, for example, by providing a hysteresis.

[0038] In addition, after detecting the eye coming close to the subject, the camera is in the eye-closed state until it detects the eye moving away from the subject. After detecting the eye moving away from the subject, the camera is in the non-eye-closed state until it detects the eye coming close to the subject. The system control unit 50 switches between display (display state) / non-display (non-display state) of the display unit 108 and the EVF 217 according to the state detected by the eye-closed detection unit 118. Specifically, when the system control unit 50 is at least in the shooting standby state and the display destination switching setting is automatic switching, the system control unit 50 turns on the display of the display unit 108 as the display destination and turns off the EVF 217 while the eye is not coming close to the subject. In addition, the system control unit 50 turns on the display of the EVF 217 as the display destination and turns off the display unit 108 while the eye is coming close to the subject. Note that the eye-closed detection unit 118 is not limited to an infrared proximity sensor. The eye-closed detection unit 118 may be any other sensor as long as it can detect a state that can be considered as the eye coming close to the subject.

[0039] The camera 100 also has an outside-finder display unit 107, an outside-finder display drive circuit 223, a power supply control unit 224, a power supply unit 225, a recording medium I / F 226, an operation unit 228, and the like.

[0040] The outside viewfinder display unit 107 displays various settings of the camera 100 such as the shutter speed and aperture via an outside viewfinder display drive circuit 223 .

[0041] The power supply control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, etc. The power supply control unit 224 detects whether a battery is installed, the type of battery, and the remaining battery level. The power supply control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50. In this way, the power supply control unit 224 supplies the necessary voltage to each unit including the recording medium 227 for the necessary period.

[0042] The power supply unit 225 is a primary battery (such as an alkaline battery or a lithium battery), a secondary battery (such as a NiCd battery, a NiMH battery, or a Li battery), an AC adapter, or the like.

[0043] The recording medium I / F 226 is an interface with a recording medium 227 (such as a memory card or a hard disk). The recording medium 227 is a memory card or the like for recording captured images. The recording medium 227 is configured from a semiconductor memory, a magnetic disk, or the like. The recording medium 227 may be detachable from the camera 100. The recording medium 227 may be built into the camera 100.

[0044] The operation unit 228 is an input unit that accepts operations from the user (user operations). The operation unit 228 is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode switching switch 103, the touch panel 109, other operation units 229, etc. The other operation units 229 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction key 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, the touch bar 119, etc.

[0045] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. When the first shutter switch 230 is turned on during operation of the shutter button 101, i.e., when the shutter button 101 is pressed halfway (instruction to prepare for shooting), it generates a first shutter switch signal SW1. In response to the generation of the first shutter switch signal SW1, the system control unit 50 starts shooting preparation processing (AF processing, AE processing, AWB processing, EF processing, etc.). When the second shutter switch 231 is turned on after the operation of the shutter button 101 is completed, i.e., when the shutter button 101 is pressed all the way (instruction to prepare for shooting), it generates a second shutter switch signal SW2. In response to the generation of the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processes (from reading out a signal from the imaging unit 211 to generating an image file including a shot image and writing it to the recording medium 227). Start writing (until writing begins).

[0046] The mode changeover switch 103 changes the operation mode of the system control unit 50 to one of a still image shooting mode, a video shooting mode, a playback mode, etc. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). Modes included in the still image shooting mode include various scene modes and custom modes that are shooting settings according to shooting scenes. A user can directly switch to one of the above-mentioned shooting modes by using the mode changeover switch 103. Alternatively, a user can selectively switch to one of the displayed modes by using the operation unit 228 after once switching to a list screen of shooting modes by using the mode changeover switch 103. Similarly, the video shooting mode may also include a plurality of modes.

[0047] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as one unit. For example, the touch panel 109 has a high light transmittance so as not to interfere with the display of the display unit 108, and is attached to the upper layer of the display surface of the display unit 108. Then, by associating input coordinates on the touch panel 109 with display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured as if the user could directly operate the screen displayed on the display unit 108. The touch panel 109 can be any of various types such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. Depending on the type, there is a type that detects a touch by contact with the touch panel 109, and a type that detects a touch by approaching a finger or a pen to the touch panel 109, but any type may be used.

[0048] The system control unit 50 can detect the following operations or states on the touch panel 109. A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down). A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On). The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the touch ends (hereinafter, referred to as "touch-up"). A state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0049] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on will also be detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. After it is detected that all fingers or pens that were touching have touched up, touch-off will occur.

[0050] These operation states and the position coordinates of the touch panel 109 touched by a finger or pen are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. The touch-move can be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates, with respect to the direction of the finger or pen moving on the touch panel 109. When a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, moved quickly for a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 109 as if flicking. When a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is detected, it is determined that a flick has been performed (it can be determined that a flick has been performed following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touched) and the touch positions are brought closer to each other is called a pinch-in, and a touch operation in which the touch positions are moved away from each other is called a pinch-out. Pinch-out and pinch-in are collectively called pinch operations (or simply pinch).

[0051] (Lens unit configuration) Fig. 4 is a schematic diagram showing an example of the configuration of lens unit 300. Fig. 4 shows a state in which lens unit 300 is attached to camera 100. Note that, among the camera 100 shown in Fig. 4, the same configurations as those explained in Fig. 3 are given the same reference numerals and explanations thereof will be omitted as appropriate.

[0052] Lens unit 300 is a type of interchangeable lens that can be attached to and detached from camera 100. Lens unit 300 is a twin lens that can capture images with parallax between left and right images. Lens unit 300 has two optical systems, each with a wide viewing angle of approximately 180 degrees, and can capture the range of the front hemisphere. Specifically, the two optical systems of lens unit 300 can capture an object with a viewing field (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle).

[0053] The lens unit 300 has a "right-eye optical system 301R having a plurality of lenses and a reflecting mirror, etc.", a "left-eye optical system 301L having a plurality of lenses and a reflecting mirror, etc.", and a lens system control circuit 303. The right-eye optical system 301R corresponds to an example of a first optical system, and the left-eye optical system 301L corresponds to an example of a second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the lenses 302R and 302L located on the subject side face in the same direction, and the optical axes are approximately parallel. The lens unit 300 of this embodiment is a VR180 lens for capturing images in a VR image format (so-called VR180) that allows binocular stereoscopic vision. In the VR180 lens, the right-eye optical system 301R and the left-eye optical system 301L each include a fisheye lens that can capture a range of approximately 180 degrees. In addition, the VR180 lens only needs to obtain an image that allows the right eye optical system 301R and the left eye optical system 301L to display two-eye VR images as VR180. The VR180 lens may be a lens that can capture a wide viewing angle range narrower than the range of 180 degrees (for example, about 160 degrees). The VR180 lens can form a right image (first image) formed via the right eye optical system 301R and a left image (second image) formed via the left eye optical system 301L having parallax from the right image, on one or two imaging elements of the attached camera.

[0054] Furthermore, lens unit 300 is attached to camera 100 via lens mount portion 304 and camera mount portion 305 of camera 100. By attaching lens unit 300 to camera 100, system control unit 50 of camera 100 and lens system control circuit 303 of lens unit 300 are electrically connected to each other via communication terminal 124 and communication terminal 306.

[0055] In this embodiment, a right image formed via the right eye optical system 301R and a left image formed via the left eye optical system 301L having a parallax with respect to the right image are formed side by side on the imaging unit 211 of the camera 100. Two optical images of the object are formed on one image sensor. The image capturing section 211 converts the formed object image (optical signal) into an analog electric signal. In this way, by using the lens unit 300, two images with parallax can be simultaneously acquired (as a set) from two locations (optical systems) of the right eye optical system 301R and the left eye optical system 301L. In addition, by dividing the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image (so-called VR180) with a range of approximately 180 degrees.

[0056] Here, the VR image is an image that can be displayed in VR, which will be described later. The VR image includes an omnidirectional image (spherical image) captured by an omnidirectional camera (spherical camera), a panoramic image having a wider image range (effective image range) than the display range that can be displayed at one time on a display unit, and the like. In addition, the VR image is not limited to a still image, but also includes a video and a live image (image acquired from a camera in almost real time). The VR image has an image range (effective image range) of a field of view of 360 degrees in the left and right directions and 360 degrees in the up and down directions at maximum. In addition, the VR image includes an image having a wider angle of view than the angle of view that can be captured by a normal camera, or a wider image range than the display range that can be displayed at one time on a display unit, even if it is less than 360 degrees in the left and right directions or less than 360 degrees in the up and down directions. The image captured by the camera 100 using the lens unit 300 described above is a type of VR image. The VR image can be displayed in VR by, for example, setting the display mode of a display device (a display device that can display a VR image) to "VR view". By displaying a VR image with a 360-degree angle of view in VR and changing the position of the display device left and right (horizontal rotation direction), the user can view seamless omnidirectional images in the left and right directions.

[0057] Here, VR display (VR view) is a display method (display mode) that can change the display range and displays an image of a VR image with a field of view corresponding to the posture of the display device. VR display includes "single-eye VR display (single-eye VR view)" that displays one image by performing a transformation (transformation in which distortion correction is performed) that maps a VR image onto a virtual sphere. VR display also includes "two-eye VR display (two-eye VR view)" that displays a VR image for the left eye and a VR image for the right eye side by side in the left and right regions by performing a transformation that maps the VR image onto a virtual sphere, respectively. Stereoscopic vision is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax from each other. In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), an image with a field of view corresponding to the direction of the user's face is displayed. For example, assume that a VR image with a field of view centered on 0 degrees left and right (a specific direction, for example, north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) is displayed at a certain point in time. When the attitude of the display device is inverted from this state (for example, the display surface is changed from facing south to facing north), the display range of the same VR image is changed to an image with a field of view centered on 180 degrees in the left-right direction (the opposite direction, for example, south) and 90 degrees in the up-down direction. That is, when the user faces from north to south (i.e. turns backwards) while wearing the HMD, the image displayed on the HMD is also changed from a north image to a south image. Note that the VR image captured using the lens unit 300 of this embodiment is a VR180 image captured in a range of approximately 180 degrees forward, and no image exists in a range of approximately 180 degrees backward. When such a VR180 image is VR displayed and the attitude of the display device is changed to the side where no image exists, a blank area is displayed.

[0058] By displaying a VR image in this way, the user visually feels as if he or she is inside the VR image (in the VR space). Note that the method of displaying a VR image is not limited to changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operation via a touch panel or directional buttons, etc. Also, during VR display (when in "VR view" display mode), in addition to changing the display range due to posture changes, the display range may also be changed in response to touch-move on a touch panel, dragging with a mouse, pressing a directional button, etc. Note that when a smartphone is attached to VR goggles (head-mounted adapter), The smartphone is treated as a type of HMD.

[0059] (VR goggles configuration) FIG. 5A shows an example of an external view of a display control device 500, which is a type of electronic device. The display 505 is a display unit that displays images and various information. The display 505 is integrally configured with a touch panel 506a as described later. The touch panel 506a can detect a touch operation on the display surface of the display 505. The display control device 500 can VR display a VR image (VR content) on the display 505. The operation unit 506 includes a touch panel 506a and operation units 506b, 506c, 506d, and 506e. The operation unit 506b is a power button that accepts an operation to switch the power of the display control device 500 on and off. The operation units 506c and 506d are volume buttons that increase and decrease the volume of the sound output from the sound output unit 512. The operation unit 506e is a home button for displaying a home screen on the display 505. The sound output terminal 512a is an earphone jack, and is a terminal that outputs sound to earphones, an external speaker, or the like. The speaker 512b is a built-in speaker for producing sound.

[0060] 5B shows an example of an external view of another surface (rear surface) of the display control device 500. The imaging unit 515 is a camera capable of capturing images.

[0061] 5C shows an example of the configuration of the display control device 500. The display control device 500 can be configured using a display device such as a smartphone. A CPU 501, a memory 502, a non-volatile memory 503, an image processing unit 504, a display 505, an operation unit 506, a storage medium I / F 507, an external I / F 509, and a communication I / F 510 are connected to an internal bus 550. In addition, an audio output unit 512 and a posture detection unit 513 are also connected to the internal bus 550. Each unit connected to the internal bus 550 can exchange data with each other via the internal bus 550.

[0062] The CPU 501 is a control unit that controls the entire display control device 500. The CPU 501 is composed of at least one processor or circuit. The memory 502 is composed of, for example, a RAM (a volatile memory using a semiconductor element, etc.). The CPU 501 controls each unit of the display control device 500 using the memory 502 as a work memory according to, for example, a program stored in the non-volatile memory 503. The non-volatile memory 503 stores image data, audio data, other data, various programs for the operation of the CPU 501, etc. The non-volatile memory 503 is composed of, for example, a flash memory, a ROM, etc.

[0063] The image processing unit 504 performs various image processing on images stored in the non-volatile memory 503 or the storage medium 508, video signals acquired via the external I / F 509, images acquired via the communication I / F 510, etc., based on the control of the CPU 501. The image processing performed by the image processing unit 504 includes A / D conversion processing, D / A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, color conversion processing, etc. The image processing unit 504 also performs various image processing (panoramic expansion, mapping processing, conversion, etc.) on VR images, which are wide-range images having wide-range (not limited to omnidirectional) data. The image processing unit 504 may be composed of a dedicated circuit block for performing specific image processing. Depending on the type of image processing, the CPU 501 can also perform image processing according to a program without relying on the image processing unit 504.

[0064] The display 505 displays images and "GUI screens constituting a GUI (Graphical User Interface)" under the control of the CPU 501. The CPU 501 generates a display control signal in accordance with a program. The CPU 501 generates a video signal to be displayed on the display 505 and controls the display 50 The display control device 500 controls each unit of the display control device 500 so as to output the image to the display 505. The display 505 displays an image based on the output video signal. Note that the configuration of the display control device 500 itself is limited to an interface for outputting a video signal to be displayed on the display 505, and the display 505 may be configured as an external monitor (such as a television).

[0065] The operation unit 506 includes a character information input device (such as a keyboard), a pointing device (a mouse or a touch panel), a button, a dial, a joystick, a touch sensor, a touch pad, etc. The operation unit 506 is an input device for receiving user operations. The touch panel is configured in a planar manner overlaid on the display 505, and outputs coordinate information according to the touched position.

[0066] A storage medium 508 (memory card, CD, DVD) can be attached to the storage medium I / F 507. The storage medium I / F 507 reads data from the attached storage medium 508 and writes data to the storage medium 508 under the control of the CPU 501. The external I / F 509 is an interface for connecting to an external device via a wired cable or wirelessly and inputting and outputting video signals and audio signals. The communication I / F 510 is an interface for communicating with an external device, the Internet 511, etc. and transmitting and receiving various data such as files and commands. The communication I / F 510 can also communicate with the controller 600, the controller 610, etc. via a wireless connection.

[0067] The audio output unit 512 outputs audio of video and music data, operation sounds, ringtones, various notification sounds, etc. The audio output unit 512 includes an audio output terminal 512a for connecting earphones, etc., and a speaker 512b. The audio output unit 512 may output audio via wireless communication, etc.

[0068] The attitude detection unit 513 detects the attitude of the display control device 500 with respect to the direction of gravity and the inclination of the attitude with respect to each axis of yaw, roll, and pitch. Based on the attitude detected by the attitude detection unit 513, it is possible to determine whether the display control device 500 is held horizontally, held vertically, facing up, facing down, or at an angle. As the attitude detection unit 513, at least one of an acceleration sensor, a gyro (angular velocity) sensor, a geomagnetic sensor, a direction sensor, an altitude sensor, and the like can be used, and a combination of two or more sensors can also be used.

[0069] The estimation unit 514 estimates the self-position and the surrounding environment in the space of the display control device 500 or the VR goggles 530 described later.

[0070] The "self-position" indicates the position of the display control device 500 or the VR goggles 530 described later in space. For example, it is expressed by three parameters that indicate the position in a coordinate system in which a predetermined position in a predetermined range of space is defined as the origin and three mutually orthogonal axes are defined as the X-axis, Y-axis, and Z-axis. It may also be expressed by adding three parameters that indicate the attitude (direction).

[0071] The imaging unit 515 is a camera capable of acquiring an image. The acquired image can be used for various detection processes by a gesture detection unit 506f, an estimation unit 514, etc., which will be described later. The imaging unit 515 can also output an image of the outside world to the display 505.

[0072] The operation unit 506 includes a touch panel 506a. The CPU 501 can detect the following operations or states on the touch panel 506a. A finger or pen that has not been touching the touch panel 506a is now touching the touch panel 506a Touching the touch screen, i.e., the start of touching (hereinafter referred to as Touch-Down). The touch panel 506a is in a state where a finger or a pen is touching the touch panel 506a (hereinafter referred to as Touch-On). A finger or pen is moved while touching the touch panel 506a (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 506a is removed from the touch panel 506a, that is, the touch ends (hereinafter referred to as "touch-up"). A state in which nothing is touching the touch panel 506a (hereinafter referred to as Touch-Off)

[0073] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on is also detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. When it is detected that all fingers or pens that were touching have touched up, touch-off is detected.

[0074] These operations and states, and the position coordinates of the touch panel 506a touched by a finger or pen are notified to the CPU 501 through the internal bus. The CPU 501 determines what kind of operation (touch operation) has been performed on the touch panel 506a based on the notified information. For touch-move, the moving direction of the finger or pen moving on the touch panel 506a can also be determined for each vertical component and horizontal component on the touch panel 106a based on the change in the position coordinates. If a touch-move of a distance greater than a predetermined distance is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 506a, quickly moved for a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 506a as if flicking it. If a touch-move of a distance greater than a predetermined distance at a predetermined speed or faster is detected and a touch-up is detected immediately, it can be determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched simultaneously and the touch positions are brought closer to each other is called pinch in, and a touch operation in which the touch positions are moved away from each other is called pinch out. Pinch out and pinch in are collectively called pinch operation (or simply pinch). The touch panel 506a may be of any of various touch panel types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. There are types that detect a touch by contact with the touch panel, and types that detect a touch by the approach of a finger or a pen to the touch panel, and either type may be used.

[0075] Moreover, the operation unit 506 includes a gesture detection unit 506f. The CPU 501 acquires a gesture image of a user's hand or the like via the imaging unit 515, and can detect a predetermined gesture in the gesture detection unit 506f.

[0076] FIG. 5D shows an external view of VR goggles (head-mounted adapter) 530 to which the display control device 500 can be attached. The display control device 500 can also be used as a head-mounted display by attaching it to the VR goggles 530. The insertion port 531 is an insertion port for inserting the display control device 500. The entire display control device 500 can be inserted into the VR goggles 530 with the display surface of the display 505 facing the headband 532 side (i.e., the user side) for fixing the VR goggles 530 to the user's head. In this manner, the VR goggles 530 with the display control device 500 attached thereto are worn. This allows the user to view the display 505 of the display control device 500 without holding the display control device 500 in their hands while wearing the VR goggles 530 on their head. In this case, when the user moves his / her head or whole body, the attitude of the display control device 500 also changes.

[0077] The orientation detection unit 513 detects a change in the orientation of the display control device 500. Based on this change in orientation, the CPU 501 performs VR display processing. In this case, the detection of the orientation of the display control device 500 by the orientation detection unit 513 is equivalent to detecting the orientation of the user's head (the direction in which the user's line of sight is facing).

[0078] The display control device 500 may detect the user's line of sight or facial expression, and the detection result may be used as an operation for the display control device 500.

[0079] 6A and 6B show external views of controllers (controller 600 and controller 610) capable of communicating with display control device 500. In grip-type controller 600 as shown in Fig. 6A, a user holds holding section 601 in his / her hand and operates members of operation surface 602, whereby an operation event is notified to display control device 500.

[0080] 6B includes a ring-type controller 610 for being worn on a user's finger 613, and a ring operation unit 612. Ring operation unit 612 may be a push-down button or the like, or may be a member capable of detecting contact with a finger, such as a rotary dial or an optical track pad.

[0081] In the following, a case where the VR goggles 530 are worn and used by the user will be described. In the above, the VR goggles 530 to which the display control device 500 can be detached are used for the description, but the display control device 500 does not necessarily have to be detachable. Furthermore, in the following, the VR goggles 530 will be described as "a display device including the display control device 500 (each component of the display control device 500)".

[0082] (About Live View images) The live view image will be described with reference to FIGS. 9A to 9C.

[0083] As shown in Fig. 9B, the live view image 902 is an image captured (photographed) of the subject 901 (see Fig. 9A) by the camera 100 equipped with a VR180 lens. The live view image 902 includes a "circular fisheye image 903 captured by the camera 100 of the subject 901 via the right-eye optical system 301R" and a "circular fisheye image 904 captured by the camera 100 of the subject 901 via the left-eye optical system 301L." The live view image 902 is displayed in real time on the display unit 108 of the camera 100.

[0084] 9B, the circular fisheye image 903 and the circular fisheye image 904 are images (areas) having parallax with each other. Therefore, the live view image 902 is an image (stereoscopic image) that allows the user to perceive the subject 901 three-dimensionally.

[0085] The camera 100 generates a cropped image 907 from a crop range 906 (a part of the live view image 902) of the live view image 902, the range having the central position 905 of the user's viewpoint as the reference (center) (see FIG. 9C). For example, the camera 100 selects one of the circular fisheye image 903 and the circular fisheye image 904, whichever the user's dominant eye is looking at, and cuts out the cropped image 907 from the crop range 906 centered on the position of the dominant eye in the selected image (area). The camera 100 then transmits the live view image 902 and the cropped image 907 to the VR goggles 530 (the VR goggles 530 including the display control device 500). If the communication speed between the camera 100 and the VR goggles 530 is slow, the cropped image 907 may be displayed with a large size (a large total number of pixels). The real-time nature and update frequency of the circular fisheye image 903 and the circular fisheye image 904 may be transmitted separately.

[0086] The cropped image 907 is an image of a range of the live view image 902 with the center of the viewpoint (a position on the display 505 corresponding to the position where either the user's right eye or left eye is looking) as a reference (center). The cropped image 907 is an image having image quality of the pixel-to-pixel magnification of the image sensor of the camera 100 or image quality of the recommended resolution of the VR goggles 530. Note that the size of the crop range 906 (hereinafter referred to as the "crop size") from which the cropped image 907 is extracted may be specified by the VR goggles 530.

[0087] (Camera 100 processing) The processing of the camera 100 according to the first embodiment will be described with reference to the flowchart of FIG.

[0088] In step S701, the system control unit 50 determines whether the firmware version of the camera 100 is a firmware version compatible with a VR180 lens (hereinafter referred to as "VR firmware version"). If it is determined that the firmware version of the camera 100 is the VR firmware version, the process proceeds to step S702. If it is determined that the firmware version of the camera 100 is not the VR firmware version, the process of this flowchart ends.

[0089] In step S702, the system control unit 50 determines whether or not a VR180 lens is attached to the camera 100. If it is determined that a VR180 lens is attached, the process proceeds to step S703. If it is determined that a VR180 lens is not attached, the process of this flowchart ends.

[0090] In step S703, the system controller 50 acquires lens design values ​​from the lens unit 300. The lens design values ​​are design values ​​for performing aberration correction. The lens design values ​​include an image circle position, an image circle diameter, an angle of view, a distortion correction coefficient, and the like.

[0091] In step S704, the system control unit 50 acquires lens individual values ​​(manufacturing errors) from the lens unit 300. The lens individual values ​​are measurement results of errors (errors from set values) detected during the manufacturing process of a twin lens such as a VR180 lens. During the manufacturing process of the twin lens, errors such as lens decentering and tilt occur in each of the two optical systems (the left-eye optical system 301L and the right-eye optical system 301R). The lens individual values ​​include image circle position shift, optical axis tilt, image magnification shift, and the like. This information is prepared by previously measuring each of the two optical systems (the left-eye optical system 301L and the right-eye optical system 301R) of the twin lens.

[0092] In step S705, the system control unit 50 connects the VR goggles 530 including the display control device 500 to the camera 100 (starts communication).

[0093] In step S706, the system control unit 50 determines whether or not a "start request" has been received from the VR goggles 530. The "start request" is an instruction to request the start of processing for displaying an image based on the live view image 902 on the VR goggles 530 (hereinafter referred to as "image display processing"). If it is determined that the "start request" has been received, the process proceeds to step S707. If it is determined that the "start request" has not been received, the process of step S706 is repeated.

[0094] In step S707, the system control unit 50 requests transmission of a live view image 902. It is determined whether an "image request" for the image capture is received from the VR goggles 530. If it is determined that an "image request" has been received, the process proceeds to step S708. If it is determined that an "image request" has not been received, the process of step S707 is repeated.

[0095] In step S708, the system control unit 50 acquires a live view image 902 from the imaging unit 211. Furthermore, when the system control unit 50 acquires information on the center position of the viewpoint and the crop size from the VR goggles 530 together with the "image request" in step S707, the system control unit 50 generates a cropped image 907 based on the information on the center position of the viewpoint and the information on the crop size. Note that the information on the center position of the viewpoint is information indicating the position in the live view image 902 where the user is looking, and is information indicating the center position of the crop range 906.

[0096] The lens information (lens design values ​​and lens individual values) acquired in steps S703 and S704 is then transmitted to the VR goggles 530 and used for image processing of the live view image 902. However, the lens information cannot be used for image processing as it is. For this reason, in step S708, the system control unit 50 converts the lens information acquired in steps S703 and S704 into information in the coordinate system of the live view image 902.

[0097] The lens information includes the center positions of the left and right fisheye lenses, the boundary positions between the left and right fisheye lenses, and the diameter size of the fisheye lenses. The lens information may also include drawing information of the magic window areas for both the left and right eyes (the position of the upper left corner, and information on the width and height) and error information during lens manufacturing. The lens information may also include correction data (such as correction values ​​obtained by calibration) for improving the accuracy of the image conversion process.

[0098] In the first embodiment, the lens information conversion process is performed by the camera 100 (system control unit 50), but may be performed by the VR goggles 530 (CPU 501). In this case, the camera 100 transmits "parameters necessary for the lens information conversion process" to the VR goggles 530 together with the lens information.

[0099] In step S709, the system control unit 50 performs processing to reduce the image quality (or amount of information) of the live view image 902. For example, the system control unit 50 causes the image processing unit 504 to perform resizing processing to reduce the total number of pixels (resolution) of the live view image 902, processing to reduce the number of colors, or processing to increase the compression rate. As a result, the cropped image 907 has higher image quality (for example, relative resolution, i.e., the number of pixels per unit size) than the live view image 902.

[0100] Wireless connections such as wireless LAN do not have high communication speeds. For this reason, when a user checks a subject 901 in real time by looking at the VR goggles 530, the total number of pixels of the live view image 902 transmitted to the VR goggles 530 cannot be made very large. In addition, in consideration of the stability of the update frequency of the drawing, the total number of pixels (resolution) of the live view image 902 after the image quality has been reduced is preferably about 960×640 or less, but can be changed as appropriate depending on the communication speed and stability of the communication.

[0101] In step S710, the system control unit 50 transmits to the VR goggles 530 the lens information converted in step S708 and the live view image 902 (the live view image 902 with reduced image quality).

[0102] In step S710, if the system control unit 50 has acquired the viewpoint center position information from the VR goggles 530 in step S707, the system control unit 50 also transmits the cropped image 907 to the VR goggles 530 at the same time. Alternatively, a communication protocol for transmitting only the cropped image 907 without transmitting the image data 902 may be established in advance with the VR goggles 530.

[0103] In step S711, the system control unit 50 determines whether or not to end the image display process. If it is determined that the image display process is not to be ended, the process proceeds to step S707. If it is determined that the image display process is to be ended, the process of this flowchart ends.

[0104] (VR Goggles 530 Processing) The process of the VR goggles 530 (display control device 500) according to the first embodiment will be described with reference to the flowchart of FIG.

[0105] In step S801, the CPU 501 determines whether or not the VR goggles 530 are connected to the camera 100 (imaging device). If it is determined that the VR goggles 530 are connected to the camera 100, the process proceeds to step S802. If it is determined that the VR goggles 530 are not connected to the camera 100, the process of step S801 is repeated.

[0106] In step S802, CPU 501 determines whether or not camera 100 connected in step S801 is a camera compatible with a VR180 lens as shown in lens unit 300 (hereinafter referred to as a "VR180 camera"). If it is determined that camera 100 is a VR180 camera, the process proceeds to step S803. If it is determined that camera 100 is not a VR180 camera, the process of this flowchart ends.

[0107] In step S803, the CPU 501 determines whether the firmware version of the camera 100 connected in step S801 is the VR firmware version. If it is determined that the firmware version of the camera 100 is the VR firmware version, the process proceeds to step S804. If it is determined that the firmware version of the camera 100 is not the VR firmware version, the process of this flowchart ends.

[0108] In step S804, CPU 501 determines whether or not a VR180 lens as shown in lens unit 300 is attached to camera 100 connected in step S801. If it is determined that a VR180 lens is attached to camera 100, the process proceeds to step S805. If it is determined that a VR180 lens is not attached to camera 100, the process of this flowchart ends.

[0109] In step S805, the CPU 501 transmits to the camera 100 a "start request" for requesting the start of image display processing.

[0110] In step S806, the CPU 501 transmits to the camera 100 an "image request" for requesting transmission of the live view image 902. Note that detailed processing of step S806 will be described later with reference to FIG.

[0111] In step S807, CPU 501 determines whether or not a live view image 902 with reduced image quality (total number of pixels; resolution) and lens information have been received from camera 100. If it is determined that the live view image 902 and the lens information have been received, the process proceeds to step S808. If it is determined that at least one of the live view image 902 and the lens information has not been received, the process of step S807 is repeated. At this time, the lens information has been subjected to conversion processing by the process of camera 100 (process of step S708) to convert it into information in the coordinate system of the live view image 902. Note that, if information on the center position of the viewpoint has been transmitted to camera 100 in step S806, CPU 501 also receives cropped image 907 from camera 100.

[0112] In step S808, the CPU 501 performs a process of swapping the left and right regions in the live view image 902 with reference to the center positions of the left eye optical system 301L and the right eye optical system 301R (hereinafter referred to as "left-right swapping process"). As a result, the CPU 501 acquires a swapped image 1101 as shown in FIG. 11A. It can be seen from FIG. 11A that the coordinate conversion of the crop range 906 (the region from which the crop image 907 is extracted) and the center position 905 of the crop range 906 described with reference to FIG. 9B is also performed. Note that the left-right swapping process uses the lens information acquired in step S807 to swap the left and right regions (circular fisheye image 903 and circular fisheye image 904) so ​​that the position corresponding to the optical axis center does not change, as shown in FIG. 10.

[0113] In step S809, the CPU 501 performs equirectangular transformation on each of the circular fisheye image 903 and the circular fisheye image 904 in the post-replacement image 1101, based on the center positions of the left-eye optical system 301L and the right-eye optical system 301R, etc. As a result, the CPU 501 obtains a planar development image 1102 as shown in Fig. 11B. "Equirectangular transformation" is a process of transforming an image (planar development of an image) so that latitudes and longitudes form right angles, as in the equirectangular projection of a map, when the fisheye is considered to be a sphere.

[0114] Note that the planar developed image 1102 is an image with a small total number of pixels (low absolute resolution), for example, about 960×640, because it has been subjected to image quality degradation processing. For this reason, the CPU 501 performs enlargement processing on the planar developed image 1102 in order to combine it with the cropped image 907, which has not been subjected to image quality degradation processing. At this time, the target size of the enlargement may be determined according to the recommended specifications of the VR goggles 530. The resolution of the cropped image 907 may also be determined by the camera 100 according to the recommended specifications of the VR goggles 530.

[0115] In step S810, CPU 501 generates an image to be displayed on display 505 (hereinafter referred to as a "display image"). If CPU 501 has received cropped image 907, CPU 501 synthesizes cropped image 907 with planar developed image 1102 to generate a display image. On the other hand, if CPU 501 has not received cropped image 907, CPU 501 treats planar developed image 1102 (or live view image 902) as the display image.

[0116] Note that the CPU 501 performs equirectangular conversion on the cropped image 907 before the composition. For example, as shown in FIG. 11B, when the left eye of the user is the dominant eye, the CPU 501 calculates an area 1103 of the left image (an area of ​​the left image corresponding to the cropped image 907) in which the cropped image 907 is to be placed based on the position where the left eye is looking. Furthermore, the CPU 501 calculates an area 1104 (an area of ​​the right image corresponding to the cropped image 907) corresponding to the area 1103 based on parallax information of the left and right images of the planar development image 1102. Then, the CPU 501 composes the cropped image 907 in each of the area 1103 and the area 1104. Note that any method may be used as long as the cropped image 907 can be placed in the position where the user is looking in the planar development image 1102 (= the range of the planar development image 1102 corresponding to the "range where the cropped image 907 in the live view image 902 is acquired"). Incidentally, by arranging the same cropped image 907 in the area 1103 and the area 1104 in this manner, the number of processes in the VR goggles 530 and an increase in the amount of communication between the VR goggles 530 and the camera 100 can be suppressed.

[0117] 12 shows an area 1205 of at least a part of a display image in which a cropped image 907 and a planar developed image 1102 are combined. The user can check the details and focus of the subject 901 by looking at a high-image-quality area 1206 near the center of the viewpoint position in the area 1205. Furthermore, the user can check the details and focus of the subject 901 by looking at an area 1207 (low-image-quality area) around the area 1206. Therefore, a sense of realism such as perspective can be felt. Therefore, a real-time display can be realized in which the center of the user's viewpoint has high image quality and gives a sense of realism to the user wearing the VR goggles 530.

[0118] On the other hand, instead of synthesizing two images, it is also possible to display the cropped image 907 by using an operation panel displayed in the three-dimensional space (180-degree stereoscopic space) of the VR goggles 530. In this case, first, the operation panel is arranged (displayed) in the area 1206, and the cropped image 907 is superimposed on the operation panel. This makes the operation panel appear as if it is floating in the 3D space, but it achieves the same effect as synthesizing two images.

[0119] In step S811, CPU 501 displays the display image generated in step S810 on display 505 (performs VR display). That is, when cropped image 907 is received, CPU 501 displays "an image obtained by combining cropped image 907 with planar development image 1102" on display 505. On the other hand, when cropped image 907 is not received, CPU 501 displays planar development image 1102 on display 505 without displaying "an image obtained by combining cropped image 907 with planar development image 1102" on display 505. Note that when cropped image 907 is not received, this refers to a case where "detail display mode," described later, is not set (is released).

[0120] In step S812, the CPU 501 determines whether or not to end the display of the display image. If it is determined that the display of the display image is to be ended, the process of this flowchart ends. If it is determined that the display of the V display image is not to be ended, the process proceeds to step S806.

[0121] (About left / right swapping) Fig. 10 is a diagram illustrating the left-right swapping process in step S808 in Fig. 8. Fig. 10 is a diagram illustrating the result when the left and right areas of a live view image 902 are swapped using lens information.

[0122] An image 1001 is a circular fisheye image (circular fisheye image data) after imaging (photography) similar to the live view image 902. A center line 1002 is the center line of the image 1001 and the image 1007. An imaging area 1003 is the imaging result (captured image) via the right-eye optical system 301R. An imaging area 1005 is the imaging result (captured image) via the left-eye optical system 301L.

[0123] An image 1007 is an image obtained after the imaging area 1003 and the imaging area 1005 in the image 1001 are swapped. An imaging area 1008 shows the imaging area 1005 after left-right swapping processing. An imaging area 1009 shows the imaging area 1003 after left-right swapping processing.

[0124] An optical axis center line 1004 is a line indicating a position corresponding to the optical axis center of the lens on the left side of the image 1001 and the image 1007. An optical center line 1006 is a line indicating a position corresponding to the optical axis center of the lens on the right side of the image 1001 and the image 1007. In the left-right swapping process, as shown in Fig. 10, the process is performed so that the positions corresponding to the optical axis centers of the image 1001 before the left-right swapping process and the image 1007 after the left-right swapping process match.

[0125] (Details of the process in step S806) Details of the process in step S806 in Fig. 8 will be described with reference to the flowchart in Fig. 13. In the following, CPU 501 determines whether the user's viewing position has changed by determining whether the user's posture (posture of the user's head) has changed, and performs processing according to the determination result.

[0126] In step S1301, the CPU 501 detects the VR goggles posture detected by the posture detection unit 513. It is determined whether or not the user's posture (posture of the user's head) has changed based on the posture information of 530. If it is determined that the user's posture has changed, the process proceeds to step S1302. If it is determined that the user's posture has not changed, the process proceeds to step S1311.

[0127] In step S1302, the CPU 501 determines whether the display mode is the "detail display mode." The "detail display mode" is a display mode in which a display image including the cropped image 907 is displayed on the display 505. In other words, if the display mode is not the "detail display mode," the planar development image 1102 (or the live view image 902) without the cropped image 907 composited therewith is displayed on the display 505.

[0128] When the cropped image 907 is displayed by the VR goggles 530, the amount of image data transferred from the camera 100 to the VR goggles 530 increases, and the processing content in the VR goggles 530 also increases. This causes delays in display and reduces the frame rate. In addition, there is a concern that the power consumption of the VR goggles 530 may increase, so in some cases it may be better not to set the "detail display mode" all the time.

[0129] Whether to set the "detail display mode" may be determined by a user's operation on the VR goggles 530. On the other hand, when the user wears the VR goggles 530 at the shooting site, the user cannot fully check the surroundings. In addition, when adjusting the angle of view or the focus, the user is required to operate the camera 100 while wearing the VR goggles 530. Therefore, it may be troublesome for the user to set the "detail display mode" by operating the VR goggles 530. For this reason, the CPU 501 may automatically set the "detail display mode" according to the user's movement (for example, a specific operation on the camera 100).

[0130] For example, when it is detected that the manual focus ring of the camera 100 has been operated and the VR goggles 530 are notified of the detection of the operation, the CPU 501 sets the mode to "detail display mode" for a certain period of time. After the certain period of time has elapsed, the CPU 501 cancels the setting of "detail display mode." Note that, below, an example will be described in which the mode is set to "detail display mode" when it is detected that the user's posture (a state corresponding to the user's action of continuing to look at one point on the subject) does not change (step S1311).

[0131] In step S1303, CPU 501 determines whether the amount of change in the user's posture detected in step S1301 is greater than a predetermined amount. If it is determined that the amount of change in the user's posture is greater than the predetermined amount (= if it is determined that the direction in which the user is looking has changed by more than a predetermined angle), the process proceeds to step S1315. If it is determined that the amount of change in the user's posture is equal to or less than the predetermined amount, the process proceeds to step S1304.

[0132] In step S1304, CPU 501 calculates the center position of the user's current viewpoint (the position where the user is currently looking) on ​​display 505. Here, the center positions of the viewpoints are calculated for each of the user's left and right eyes, so two positions are calculated.

[0133] In step S1305, the CPU 501 calculates "two positions corresponding to the center positions of the two viewpoints acquired in step S1304" in the display image (the display image displayed on the display 505 of the VR goggles 530).

[0134] In step S1306, the CPU 501 calculates "two positions in the post-replacement image 1101 that correspond to the two positions calculated in step S1305."

[0135] In step S1307, the CPU 501 performs In other words, in steps S1304 to S1307, CPU 501 calculates the position at which the user looks in live view image 902 (the position at which the right eye looks and the position at which the left eye looks).

[0136] In step S1308, CPU 501 selects, from the two positions calculated in step S1307, the position toward which the eye (eg, dominant eye) set in advance by the user is looking.

[0137] Note that the process of step S1308 does not necessarily have to be performed. For example, CPU 501 may calculate only the center position of the viewpoint of the dominant eye on display 505 in step S1304, and calculate the position of live view image 902 corresponding to that center position of the viewpoint in steps S1305 to S1307.

[0138] In step S1309, CPU 501 transmits information on the center position of the viewpoint indicating the position selected in step S1308 and information on the crop size to camera 100. In this way, CPU 501 requests camera 100 to transmit cropped image 907. Note that when processing proceeds from step S1314 to step S1309, CPU 501 transmits information on the crop size determined in step S1314 and information on the center position of the viewpoint currently being used to camera 100.

[0139] In step S1310, CPU 501 requests camera 100 to transmit live view image 902 (= makes an "image request"). Note that, although not shown, if CPU 501 determines that the attitude detection in step S1301 continues to remain unchanged, CPU 501 may not request camera 100 to transmit live view image 902 for a specific period of time. Alternatively, in such a case, CPU 501 may reduce the frequency of requests to camera 100 to transmit live view image 902 (for example, requesting transmission of live view image 902 only once every several times that step S1310 is proceeded to).

[0140] In step S1311, the CPU 501 sets the "detail display mode."

[0141] In step S1312, CPU 501 measures (monitors) the time during which the user's posture remains unchanged (hereinafter, referred to as "posture duration").

[0142] In step S1313, CPU 501 determines whether the posture duration is longer than the threshold. If it is determined that the posture duration is longer than the threshold (= the user's viewing position has not changed for a specific period of time), the process proceeds to step S1314. If it is determined that the posture duration is equal to or shorter than the threshold, the process proceeds to step S1309.

[0143] In step S1314, CPU 501 increases the crop size stepwise from the current size. As a result, when the user focuses and gazes at one point on the subject, area 1206 displayed in high image quality gradually expands, and the range in which the user can check the details of the subject to be photographed (subject) automatically expands.

[0144] In step S1315, CPU 501 cancels the "detail display mode."

[0145] Furthermore, if the CPU 501 determines in step S1303 that the amount of change in the user's posture is within a specific range that is smaller than a predetermined amount and close to zero, then the CPU 501 may use the previous information on the center position of the viewpoint in step S1309. In other words, the processes in steps S1304 to S1308 may not be performed, and the information on the center position of the viewpoint may not be updated. In this way, if the amount of change in the user's posture is within a specific range, the cropped image 907 Since the position at which the cropped image 907 is displayed is fixed, it is possible to prevent the position of the cropped image 907 from changing minutely.

[0146] According to the first embodiment, in the displayed image, the cropped image allows the user to check the details of the central range of the viewing position. In addition, since the surroundings of the central range are also displayed with low image quality, the user can check the three-dimensional effect of the displayed image. Furthermore, since the image quality of the images other than the cropped image is low, the number of processes (calculation amount) of the geometric processes (left-right swapping process and equirectangular conversion process) executed by the VR goggles can also be reduced. This makes it possible to simplify the configuration of the VR goggles (information processing system).

[0147] <Embodiment 2> In the first embodiment, the geometric transformation processes (such as the left-right swap process described with reference to FIG. 10 and the equirectangular transformation described with reference to the planar development image 1102 shown in FIG. 11B) are processes based on lens information. For this reason, it is also possible to perform these processes at high speed by the hardware in the camera 100. This can further improve the performance of stereoscopic vision.

[0148] In the second embodiment, the camera 100 (system control unit 50) performs "processing similar to steps S802, S803, S808, and S809" described as the processing of the VR goggles 530 in the first embodiment in steps S708 and S710. Then, the camera 100 transmits the processed image (planar development image 1102) to the VR goggles 530.

[0149] With this configuration, the processing performed by the VR goggles 530 can be simplified to "enlarging" the planar development image 1102 in step S809 and "fitting" the cropped image 907 into the planar development image 1102. Note that the camera 100 may further execute the processing in step S809 and transmit to the VR goggles 530 a display image in which the planar development image 1102 and the cropped image 907 are combined.

[0150] In this way, by reducing the processing of the VR goggles 530, it is possible to lower the recommended specifications of the VR goggles 530. Therefore, it is possible to provide a monitor function with a smaller, lighter, and less expensive system configuration (simpler configuration) at the shooting site.

[0151] In each of the above embodiments, when the communication between the camera 100 and the VR goggles 530 is smooth, the transmission and reception of the large-sized cropped image 907 has little effect on communication delay. Therefore, the crop size may be determined based on the communication state between the camera 100 and the VR goggles 530. Specifically, the CPU 501 may increase the crop size as the communication speed between the camera 100 and the VR goggles 530 increases. Alternatively, when the camera 100 and the VR goggles 530 communicate with each other by wire, the CPU 501 may increase the crop size compared to when the camera 100 and the VR goggles 530 communicate with each other by wireless communication. According to these configurations, the user can check a wide range of the real space in detail in real time by looking at the large-sized cropped image 907.

[0152] In the above description, the camera 100 cuts out the cropped image 907 from either the circumferential fisheye image 903 or the circumferential fisheye image 904, but the camera 100 may cut out the cropped image 907 from each of the circumferential fisheye image 903 and the circumferential fisheye image 904. For example, the camera 100 cuts out an image in a crop range 906R centered on "a position on the display 505 corresponding to the position where the user's right eye is looking" from the circumferential fisheye image 903 as the cropped image 907R. The camera 100 cuts out an image in a crop range 906R centered on "a position on the display 505 corresponding to the position where the user's right eye is looking" from the circumferential fisheye image 904 as the cropped image 907R. An image of a crop range 906L having a center at a position corresponding to a position where the user's left eye is looking is cut out as a crop image 907L. Then, the VR goggles 530 (display control device 500) synthesizes the crop image 907L at a position where the user's left eye is looking in the planar development image 1102, and synthesizes the crop image 907R at a position where the user's right eye is looking. At this time, since a person looks at the same subject (object; target) with both the left and right eyes, the crop images 907R and 907L (crop ranges 906R and 906L) contain (are shown in) the same subject. According to such a method, the number of processes and the amount of communication are greater than when the same crop image 907 is arranged in two regions on the left and right of the planar development image 1102, but it is possible to display a more appropriate display image.

[0153] In the above description, the camera 100 cuts out a cropped image from a range centered on the position where the user is looking in the live view image 902. However, the camera 100 may cut out a cropped image from a range centered on the center position of the circular fisheye image 903 or the center position of the circular fisheye image 904 in the live view image 902, for example, without detecting the position where the user is looking.

[0154] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0155] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0156] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0157] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0158] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) A first captured image obtained by imaging through a first optical system is subjected to a geometric transformation. a first acquisition means for acquiring a first image including a first region of an image and a second region of an image obtained by performing the geometric transformation on a second captured image acquired by imaging via a second optical system; a second acquisition means for acquiring a second image having a higher image quality than the first image based on an image obtained by cutting out a first range from either the first captured image or the second captured image; a control means for controlling a display means to display a third image obtained by combining the first image and the second image by arranging the second image in a range of the first image corresponding to the first range; An information processing system comprising: (Configuration 2) the control means generates the third image by arranging the second image in a range of the first region that corresponds to the first range, and by arranging the second image in a range of the second region that corresponds to the first range. 2. The information processing system according to configuration 1. (Configuration 3) the second acquisition means 1) acquires the second image based on an image obtained by cutting out the first range of a portion of the first captured image, and 2) acquires a fourth image having a higher image quality than the first image based on an image obtained by cutting out a second range of a portion of the second captured image; the control means generates the third image by arranging the second image in a range of the first area corresponding to the first range, and arranging the fourth image in a range of the second area corresponding to the second range. a subject included in the second range of the second captured image is the same as a subject included in the first range of the first captured image; 2. The information processing system according to configuration 1. (Configuration 4) The first range is a range centered on a first position, the first position corresponds to a second position at which the user is viewing the display means; 4. The information processing system according to any one of configurations 1 to 3. (Configuration 5) The geometric transformation includes a process of controlling positions so as to exchange a position of the first image with a position of the second image. 5. The information processing system according to any one of configurations 1 to 4. (Configuration 6) each of the first optical system and the second optical system includes a fisheye lens; The geometric transformation includes a process of performing planar expansion. 6. The information processing system according to any one of configurations 1 to 5. (Configuration 7) The control means controlling the display means to display the third image for a certain period of time when a specific operation is performed by a user; and controlling the display means so as to display the first image without displaying the third image after the certain period of time has elapsed. 7. The information processing system according to any one of configurations 1 to 6. (Configuration 8) the control means controls the display means to display the first image without displaying the third image when a change in the user's posture is greater than a predetermined amount. 7. The information processing system according to any one of configurations 1 to 6. (Configuration 9) The second acquisition means acquires a position where the user is looking when the position has not changed for a specific period of time. if so, increase the size of the first range from the current size. 9. The information processing system according to any one of configurations 1 to 8. (Configuration 10) The second acquisition means includes: selecting one of the first captured image and the second captured image based on information indicating a dominant eye of a user; acquiring the second image based on an image obtained by cutting out the first range from a part of the selected image; 10. The information processing system according to any one of configurations 1 to 9. (Configuration 11) an imaging device that captures images via the first optical system and the second optical system; a display device having the display means and capable of communicating with the imaging device; having The imaging device transmits at least one of the second image and the third image to the display device. 11. The information processing system according to any one of configurations 1 to 10. (Configuration 12) The second acquisition means controls a size of the first range based on a communication state between the imaging device and the display device. 12. The information processing system according to configuration 11. (method) a first acquisition step of acquiring a first image including a first region of an image obtained by performing a geometric transformation on a first captured image acquired by imaging via a first optical system, and a second region of an image obtained by performing the geometric transformation on a second captured image acquired by imaging via a second optical system; a second acquisition step of acquiring a second image having a higher image quality than the first image based on an image obtained by cutting out a first range from either the first captured image or the second captured image; a control step of controlling a display means to display a third image obtained by combining the first image and the second image by arranging the second image in a range of the first image corresponding to the first range; 13. An information processing method comprising: (program) 13. A program for causing a computer to function as each of the means of the information processing system according to any one of configurations 1 to 12. [Explanation of symbols]

[0159] 1: Information processing system, 100: Camera, 530: VR goggles, 500: Display control device, 302R: Right eye optical system, 302L: Left eye optical system, 211: imaging unit, 50: system control unit, 501: CPU, 505: Display

Claims

1. a first acquisition means for acquiring a first image including a first region of an image obtained by performing a geometric transformation on a first captured image acquired by imaging via a first optical system, and a second region of an image obtained by performing the geometric transformation on a second captured image acquired by imaging via a second optical system; a second acquisition means for acquiring a second image having a higher image quality than the first image based on an image obtained by cutting out a first range from either the first captured image or the second captured image; a control means for controlling a display means to display a third image obtained by combining the first image and the second image by arranging the second image in a range of the first image corresponding to the first range; An information processing system comprising:

2. the control means generates the third image by arranging the second image in a range of the first region corresponding to the first range, and by arranging the second image in a range of the second region corresponding to the first range.

2. The information processing system according to claim 1 .

3. the second acquisition means 1) acquires the second image based on an image obtained by cutting out the first range of a portion of the first captured image, and 2) acquires a fourth image having a higher image quality than the first image based on an image obtained by cutting out a second range of a portion of the second captured image; the control means generates the third image by arranging the second image in a range of the first area corresponding to the first range, and arranging the fourth image in a range of the second area corresponding to the second range. a subject included in the second range of the second captured image is the same as a subject included in the first range of the first captured image; 2. The information processing system according to claim 1 .

4. The first range is a range centered on a first position, the first position corresponds to a second position at which the user is viewing the display means; 4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

5. The geometric transformation includes a process of controlling positions so as to exchange a position of the first image with a position of the second image.

4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

6. each of the first optical system and the second optical system includes a fisheye lens; The geometric transformation includes a process of performing planar expansion.

4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

7. The control means controlling the display means to display the third image for a certain period of time when a specific operation is performed by a user; controlling the display means to display the first image without displaying the third image after the certain time has elapsed; 4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

8. the control means controls the display means to display the first image without displaying the third image when a change in the user's posture is greater than a predetermined amount.

4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

9. the second acquisition means increases the size of the first range from the current size when a position at which the user is looking has not changed continuously for a specific period of time; 4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

10. The second acquisition means includes: selecting one of the first captured image and the second captured image based on information indicating a dominant eye of a user; acquiring the second image based on an image obtained by cutting out the first range from a part of the selected image; 4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

11. an imaging device that captures images via the first optical system and the second optical system; a display device having the display means and capable of communicating with the imaging device; having the imaging device transmits at least one of the second image and the third image to the display device; 4. The information processing system according to claim 1, wherein the information processing system is a data processing system.

12. the second acquisition means controls a size of the first range based on a communication state between the imaging device and the display device.

12. The information processing system according to claim 11.

13. a first acquisition step of acquiring a first image including a first region of an image obtained by performing a geometric transformation on a first captured image acquired by imaging via a first optical system, and a second region of an image obtained by performing the geometric transformation on a second captured image acquired by imaging via a second optical system; a second acquisition step of acquiring a second image having a higher image quality than the first image based on an image obtained by cutting out a first range from either the first captured image or the second captured image; a control step of controlling a display means to display a third image obtained by combining the first image and the second image by arranging the second image in a range of the first image corresponding to the first range; 13. An information processing method comprising:

14. A program for causing a computer to function as each of the means of the information processing system according to any one of claims 1 to 3.

Images