Information processing device, control method for information processing device, program

Abstract

This technology provides a virtual display while also facilitating physical user actions such as button presses. [Solution] An information processing device that is communicatively connected to an imaging device includes: a first acquisition means for acquiring a first image of a space including the imaging device captured according to the user's viewpoint; a second acquisition means for acquiring a second image of the space captured by the imaging device; and a generation means for generating a composite image by combining the first image and the second image. If the user has the imaging device, the generation means generates the composite image in which the second image is arranged so that it does not overlap the imaging device in the first image.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] As a technology for experiencing a virtual space, VR (Virtual Reality) technology is known. Also, as a technology for fusing the real space and the virtual space in real time and seamlessly, so-called MR (Mixed Reality) technology is known. For a device capable of experiencing such a technology, for example, a head-mounted device typified by an HMD (Head Mounted Display) is used.

[0003] Patent Document 1 describes a method of determining which of a virtual object and a real object to display based on whether or not the distance between the position of the viewpoint and the position of the virtual object exceeds a set threshold when the virtual object and the real object overlap.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in Patent Document 1, when a user wants to operate a button on an external imaging device that actually exists, a virtual object such as a virtual window may cover the button. In this case, it becomes difficult for the user to operate the button.

[0006] Therefore, an object of the present invention is to provide a technology that makes it easy to realize physical operations such as pressing a button by a user while performing virtual display. [Means for solving the problem]

[0007] One aspect of the present invention is, An information processing device that is communicatively connected to an imaging device. A first acquisition means for acquiring a first image of the space including the imaging device, captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A generation means for generating a composite image obtained by combining the first image and the second image, Having The generation means, if the user has the imaging device, generates the composite image in which the second image is arranged so as not to overlap the imaging device in the first image. This is an information processing device characterized by the following features. [Effects of the Invention]

[0008] According to the present invention, it becomes easier to perform physical operations such as pressing buttons by the user while simultaneously providing a virtual display. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the system configuration according to Embodiment 1. [Figure 2] This is an external view of the camera according to Embodiment 1. [Figure 3] This is an internal configuration diagram of the camera according to Embodiment 1. [Figure 4] This is an internal configuration diagram of the HMD, etc., according to Embodiment 1. [Figure 5] This is a diagram illustrating the MR space according to Embodiment 1. [Figure 6] This figure illustrates an example of the display on an HMD according to Embodiment 1. [Figure 7] This is a flowchart of the live view processing of the camera according to Embodiment 1. [Figure 8]It is a flowchart of the live view processing of the PC according to Embodiment 1. [Figure 9] It is a flowchart of the UI display processing of the PC according to Embodiment 1. [Figure 10] It is a diagram for explaining the composite image according to Embodiment 1. [Figure 11] It is a diagram for explaining the composite image according to Embodiment 1. [Figure 12] It is a flowchart of the UI display processing of the PC according to Embodiment 2.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] <Embodiment 1> (Configuration of the Whole System) Referring to FIG. 1, an example of the configuration of the whole system according to Embodiment 1 will be described. The information processing system 1 includes a camera 100, an HMD 300, a PC (personal computer) 310, and a controller 320.

[0012] The camera 100 is connected to the PC 310 in a communicable state, either wired or wirelessly. The camera 100 transmits and receives various data (such as live view image data and captured image data). Note that, for example, instead of the camera 100, an imaging device (such as a smartphone or a tablet terminal) that can realize the functions described below may be used. Note that the camera 100 may communicate with the HMD 300, not limited to the PC 310.

[0013] In addition, this embodiment shows an example of obtaining, as a final product, an image obtained by synthesizing an image of the real space captured by the camera 100 and an object in the virtual space. However, the technology according to this embodiment can also be implemented when a user wearing the HMD 300 captures only an object in the real space with the camera 100. Therefore, a virtual object as a subject may not be synthesized in both the image displayed on the display of the HMD 300 and the final product.

[0014] The HMD 300 is a display device (a head-mounted electronic device) that can be worn on the user's head. The HMD 300 displays a composite image in which "a captured image of the range in front of the user captured by the HMD 300" and "content such as CG in a form corresponding to the position and orientation of the HMD 300" are synthesized.

[0015] The PC 310 is an information processing device that controls the HMD 300. The PC 310 is connected to the HMD 300 by wire such as a USB cable or wirelessly such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). For example, the PC 310 generates a composite image by synthesizing a captured image and CG, and transmits the composite image to the HMD 300. In this case, when the PC 310 receives a live view image or a captured image from the camera 100, the PC 310 generates a composite image in which the received image and CG in a form corresponding to the position and orientation of the camera 100 are synthesized. The PC 310 transmits the composite image to the HM D 300.

[0016] Note that a smartphone or a tablet terminal may be used instead of the PC 310. Also, each component of the PC 310 may be possessed by the HMD 300. In Embodiment 1, an example in which the PC 310 and the camera 100 are connected wirelessly is shown, but the PC 310 and the camera 100 may be connected by wire.

[0017] The controller 320 performs various controls on the HMD 300. When the PC 310 is in a specific control mode, the HMD 300 is controlled according to the user's operation when the user operates the controller 320. As shown in Figure 1, the controller 320 is an operating component that is "ring-shaped and can be worn and supported on the user's finger" or "handheld and can be held in the hand". The controller 320 also has physical buttons for making decisions and selections on the display.

[0018] Controller 320 communicates with PC 310 wirelessly via Bluetooth. Controller 320 may also communicate with HMD 300, not just PC 310. The user can change the indicated position on the display by moving Controller 320. The indicated position may be represented as a point, or as a virtual ray connecting the point and the controller with a straight line (line segment) or dotted line. The user can perform menu selection and confirmation operations by pressing physical buttons.

[0019] The controller 320 is ring-shaped or handheld. However, the controller 320 may be any shape as long as it can be supported by a finger, hand, or arm. The buttons on the controller 320 are described as physical buttons, but they may be operable in the form of a trackpad, touch panel, wheel, or trackball. In addition to button presses, the controller 320 may also accept slide, flick, and touch operations. The controller 320 may be wearable on at least one of the fingers, hand, or arm. The controller 320 may also be attached to an object held in the hand, and positional and orientation information of the attached position may be obtained from sensors. Examples of such objects include objects that mimic tools.

[0020] (External structure of a digital camera) Figures 2A and 2B show an example of the external configuration of the camera 100, which is an imaging device. Figure 2A is a perspective view of the camera 100 from the front. Figure 2B is a perspective view of the camera 100 from the rear.

[0021] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface.

[0022] The shutter button 101 is an operation unit for preparing to take a picture or giving a shooting command. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode selector switch 103 is an operation unit for switching between various modes.

[0023] The main electronic dial 104 is a rotary control unit for changing settings such as shutter speed and aperture. The sub-electronic dial 105 is a rotary control unit for moving the selection frame (cursor) and advancing images.

[0024] The video button 106 is an operating unit for instructing the start and stop of video recording. Yes, it does. The external display unit 107 of the viewfinder displays various settings such as shutter speed and aperture.

[0025] The camera 100 also includes a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back.

[0026] The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108.

[0027] The directional keys 110 are an operation unit consisting of keys that can be pressed in the up, down, left, and right directions (4-way keys). The camera 100 can be controlled according to the position where the directional keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed when confirming a selection item.

[0028] The AE lock button 112 is an operation button that is pressed to fix the exposure state when the camera is in shooting standby mode.

[0029] The zoom button 113 is an operation unit used to switch the zoom mode on and off in the live view display (LV display) of the shooting mode. When the zoom mode is on, the live view image (LV image) will be enlarged or reduced when the main electronic dial 104 is operated. The zoom button 113 is also used in playback mode to enlarge the playback image or increase the magnification ratio.

[0030] The playback button 114 is an operation unit for switching between shooting mode and playback mode. When the playback button 114 is pressed in shooting mode, the device switches to playback mode, and the most recent image recorded on the recording medium 227 (described later) is displayed on the display unit 108.

[0031] The menu button 115 is an operation button that is pressed to display a menu screen on the display unit 108 that allows for various settings. The user can intuitively make various settings using the menu screen displayed on the display unit 108, the directional keys 110, and the SET button 111.

[0032] The eyepiece section 116 is the part that allows the user to bring their eye close to (eye-at) the eyepiece viewfinder (a type of viewfinder) 117. The user can view the image displayed on the EVF 217 (Electronic View Finder) through the eyepiece section 116.

[0033] The eyepiece detection unit 118 is a sensor (eyepiece sensor) that detects whether or not a user is looking into the eyepiece unit 116.

[0034] The touch bar 119 is a line-shaped touch operation area (line touch sensor) capable of accepting touch operations. The touch bar 119 is positioned so that it can be touched with the right thumb when the grip section 120 is held with the right hand (with the little finger, ring finger, and middle finger of the right hand), allowing the shutter button 101 to be pressed with the right index finger. In other words, the touch bar 119 can be operated when the user is looking through the eyepiece viewfinder 117 with their eyepiece in the eyepiece section 116 and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (touching and releasing without moving within a predetermined period of time), left and right sliding operations (touching and then moving the touch position while keeping the touch on the surface), etc. The touch bar 119 is a different operation area from the touch panel 109 and does not have a display function. The touch bar 119 in this embodiment is a multifunction bar and functions, for example, as an M-Fn bar.

[0035] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, and the like.

[0036] The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. When the camera 100 is held with the grip section 120 held by the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand.

[0037] The thumb rest section 121 (thumb waiting position) is a grip section located on the back of the camera 100, in a position where it is easy to rest the thumb of the right hand holding the grip section 120 when no controls are being operated. The thumb rest section 121 is made of rubber material or the like to enhance the holding power (grip feel).

[0038] The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 227 and the slot for storing the recording medium 227 by closing the slot.

[0039] The communication terminal 124 is a terminal for communicating with the lens unit 200.

[0040] (Internal structure of a digital camera) Figure 3 shows an example of the internal configuration of camera 100. In Figure 3, components identical to those in Figure 2 are denoted by the same reference numerals, and their descriptions are omitted as appropriate. A lens unit 200 is attached to camera 100.

[0041] First, let's describe the lens unit 200. The lens unit 200 is a type of interchangeable lens that can be attached to and detached from the camera 100. The lens unit 200 is a single-lens reflex lens and is an example of a normal lens. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0042] The aperture diameter of aperture 201 is adjustable. Lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of aperture 201. The AF drive circuit 204 drives lens 202 to focus.

[0043] The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc., based on instructions from the system control unit 50. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203. The lens system control circuit 205 focuses by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication takes place via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100.

[0044] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0045] The shutter 210 controls the exposure time of the imaging unit 211 based on instructions from the system control unit 50. It is a focal-plane shutter that can be freely controlled.

[0046] The imaging unit 211 is an image sensor composed of a CCD or CMOS element, which converts an optical image into an electrical signal. The imaging unit 211 may also have an image plane phase difference sensor that outputs defocus amount information to the system control unit 50.

[0047] The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal.

[0048] The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing, and color conversion) on data from the A / D converter 212 or data from the memory control unit 213. The image processing unit 214 also performs predetermined calculations using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. This process enables TTL (through-the-lens) AF processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data and performs TTL AWB (auto white balance) processing based on the obtained calculation results. Image data from the A / D converter 212 is written to memory 215 via the image processing unit 214 and memory control unit 213. Alternatively, image data from the A / D converter 212 is written to memory 215 via memory control unit 213 without going through the image processing unit 214.

[0049] Memory 215 stores "image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212" and "image data for display on the display unit 108 and EVF 217". Memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. Memory 215 also serves as memory for image display (video memory).

[0050] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies the analog signal to the display unit 108 and EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 and EVF 217 via the D / A converter 216. The display unit 108 and EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and EVF 217 are displays such as LCDs or OLEDs. The digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216. The analog signal is sequentially transferred to the display unit 108 and EVF 217, enabling live view display, which shows an image representing the space in real time.

[0051] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements each process of the flowchart described later by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0052] The camera 100 also includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118.

[0053] For example, RAM is used for the system memory 218. The system memory 218 contains "constants and variables for the operation of the system control unit 50" and "data read from the non-volatile memory 219". Programs such as "Programs" will be launched.

[0054] The non-volatile memory 219 is an electrically erasable and recordable memory. For example, an EEPROM is used for the non-volatile memory 219. Constants for the operation of the system control unit 50, programs, etc., are stored in the non-volatile memory 219. The program here refers to a program for executing the flowchart processing described later.

[0055] The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals or audio signals to and from external devices connected by wireless or wired cables.

[0056] The communication unit 221 can connect to both a wireless LAN (Local Area Network) and the internet. Furthermore, the communication unit 221 can communicate with external devices via Bluetooth® and Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227. The communication unit 221 can also receive image data and other various types of information from external devices.

[0057] The attitude detection unit 222 detects the attitude of the camera 100 relative to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image before recording. For example, an acceleration sensor or a gyroscope sensor can be used for the attitude detection unit 222. The attitude detection unit 222 can also be used to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.).

[0058] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece section 116 of the "eyepiece viewfinder 117 with built-in EVF 217". For example, an infrared proximity sensor can be used for the eyepiece detection unit 118. When an object approaches the eyepiece section 116, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece section 116 to the object can be determined by the amount of infrared light received (=sensor value). In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece section 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eye-to-eye contact) and departure (eye-away) of an eye (object) to the eyepiece section 116 of the eyepiece viewfinder 117. When an object approaches the eyepiece 116 within a predetermined distance from the non-eyepiece state (not close state), it is detected that the eye has been placed. Conversely, when the eyepiece is in the close state, it is detected that the eye has been removed when the object that was detected approaching moves beyond a predetermined distance. The threshold for detecting eye placement and the threshold for detecting eye removal may be different, for example, by providing hysteresis. Furthermore, after eye placement is detected, the eyepiece is considered to be in the eyepiece state until eye removal is detected. After eye removal is detected, the eyepiece is considered to be in the non-eyepiece state until eye placement is detected again.

[0059] The system control unit 50 switches the display (displayed) / hidden (hidden) state on the display unit 108 and EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, at least in the shooting standby state and when the display destination switching setting is set to automatic switching, the system control unit 50 turns on the display on the display unit 108 and hides the EVF 217 when the user is not using an eyepiece. Also, when the user is using an eyepiece, the system control unit 50 turns on the display on the EVF 217 and hides the display unit 108. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor, but can detect any state that can be considered as an eyepiece. If that's the case, other sensors may be used.

[0060] The camera 100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium interface 226, an operation unit 228, and the like.

[0061] The external viewfinder display unit 107 displays various settings of the camera 100 (such as shutter speed and aperture) via the external viewfinder display drive circuit 223.

[0062] The power control unit 224 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and the like. The power control unit 224 detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, supplying the necessary voltage to each part (including the recording medium 227) for the required period.

[0063] The power supply unit 225 is a primary battery (such as an alkaline battery or a lithium battery), a secondary battery (such as a NiCd battery, a NiMH battery or a Li battery), or an AC adapter.

[0064] The recording medium I / F 226 is an interface to the recording medium 227. The recording medium 227 is a memory card or the like for recording captured images. The recording medium 227 is composed of semiconductor memory or a magnetic disk or the like.

[0065] The recording medium 227 may be detachable from the camera 100, or it may be built into the camera 100.

[0066] The operation unit 228 is an input unit that receives user input (user operation). The operation unit 228 is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode selector switch 103, a touch panel 109, and other operation units 229.

[0067] Other control units 229 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a play button 114, a menu button 115, a touch bar 119, and the like.

[0068] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231.

[0069] The first shutter switch 230 turns on during the operation of the shutter button 101, specifically when it is half-pressed (indicating preparation for shooting), and generates the first shutter switch signal SW1. Upon generation of the first shutter switch signal SW1, the system control unit 50 starts the shooting preparation process (AF processing, AE processing, AWB processing, EF processing, etc.).

[0070] The second shutter switch 231 turns on when the shutter button 101 is fully pressed (a shooting instruction), generating the second shutter switch signal SW2. Upon generation of the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processes (from reading the signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227).

[0071] The mode selector switch 103 switches the operating mode of the system control unit 50 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. The modes include auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). The still image shooting mode also includes various scene modes and custom modes, which provide shooting settings for different shooting scenes. The user can directly switch to any of the above shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to one of the displayed modes using the control unit 228. Similarly, the video shooting mode may also include multiple modes.

[0072] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is mounted on top of the display surface of the display unit 108 so that the light transmittance of the touch panel 109 does not interfere with the display of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109. Either method is acceptable.

[0073] The system control unit 50 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). The finger or pen that was touching the touch panel 109 is lifted (released), meaning the touch action ends (hereinafter referred to as "Touch-Up"). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0074] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0075] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 50 via the internal bus. Based on the notified information, the system control unit 50 determines what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If it is detected that a touch move has occurred beyond a predetermined distance, a slide operation is performed. It is determined that a flick has occurred. A flick is an operation in which you touch the touch panel 109 with your finger, move it quickly a certain distance while keeping it on the panel, and then release it. In other words, a flick is an operation in which you quickly trace across the touch panel 109 with your finger as if flicking it. If it is detected that you have touched the panel for a distance greater than a predetermined distance at a speed greater than a predetermined speed, and then a touch-up is detected, it is determined that a flick has occurred (it can be determined that a flick followed a slide operation). Furthermore, a touch operation in which you touch multiple locations (for example, two points) together (multitouch) and bring the touch positions closer together is called a pinch-in, and a touch operation in which you move the touch positions further apart is called a pinch-out. Pinch-out and pinch-in are collectively referred to as a pinch operation (or simply a pinch).

[0076] (HMD configuration) Referring to Figure 4, an example of the configuration of the HMD300 will be described. The HMD300 includes an HMD control unit 301, an imaging unit 302, an image display unit 303, an attitude sensor unit 304, a non-volatile memory 305, a working memory 306, and a gaze imaging unit 307.

[0077] The HMD control unit 301 is a CPU that controls each component of the HMD 300. When the HMD control unit 301 acquires a composite image (an image created by combining an image captured by the imaging unit 302 of the space in front of the user with computer graphics) from the PC 310, it displays the composite image on the image display unit 303. Alternatively, instead of the HMD control unit 301 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0078] The imaging unit 302 includes two cameras (imaging devices). The two cameras capture images used for synthesis with images in a virtual space and for generating positional orientation information. The two cameras have an imaging unit for the left eye and an imaging unit for the right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the HMD300 wearer. The left eye imaging unit outputs an image of each frame in the moving image (captured image). The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the HMD300 wearer. The right eye imaging unit outputs an image of each frame in the moving image (captured image). In other words, the imaging unit 302 acquires captured images as stereo images with parallax that approximately coincides with the positions of the left and right eyes of the HMD300 wearer. In addition, distance information from the two cameras to the subject can be acquired as distance information by measuring distance with the stereo cameras. Furthermore, in HMDs for MR systems, it is preferable that the central optical axis of the imaging range of the imaging unit be positioned so as to substantially coincide with the direction of the HMD wearer's line of sight.

[0079] Each of the left and right eye imaging units includes an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incident light as an image. The images captured by the two cameras of the subject (the area directly in front of the user) are output to the PC 310 and the HMD control unit 301. The imaging unit 302 may output video instead of an image.

[0080] The image display unit 303 displays a composite image. The image display unit 303 has a liquid crystal panel or an organic EL panel, etc. When the user is wearing the HMD 300, the image display unit 303 is positioned in front of each of the user's eyes. It is also possible to use a device with a semi-transparent half-mirror for the image display unit 303. In this case, for example, the image display unit 303 may display an image so that the CG is superimposed directly onto the real space visible through the half-mirror, using a technology generally known as AR (Augmented Reality). Alternatively, the image display unit 303 may display an image of a completely virtual space without using captured images, using a technology generally known as VR (Virtual Reality).

[0081] The attitude sensor unit 304 acquires attitude (and position) information of the HMD 300. The attitude sensor unit 304 may also acquire user (the user wearing the HMD 300) attitude information that corresponds to the attitude (and position) of the HMD 300. For example, the attitude sensor unit 304 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 304 is used to acquire user attitude information, and the HMD control unit 301 outputs the user's attitude information to the PC 310. The attitude information may also be acquired from one or more of the following: a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, or an angular velocity sensor.

[0082] The HMD control unit 301 estimates the position or orientation of each joint point of the user's hand and fingers based on images obtained from the two cameras of the imaging unit 302. The joint points include characteristic points of parts such as the finger joints, fingertips, back of the hand (palm), and arm. Each joint point indicates its coordinate position. The hand's orientation can be estimated based on information from multiple joint points. For estimating the position or orientation of the hand and each joint point, known machine learning methods such as object recognition and pose estimation using convolutional neural networks can be used. Furthermore, the depth-direction position information of each joint point of the hand can be obtained, for example, by calculating the distance from the imaging unit 302 to each joint point through triangulation using stereo matching with images obtained from the two cameras of the imaging unit 302. The estimated coordinate information of each joint point of the hand is output from the HMD control unit 301 to the PC 310. The HMD control unit 301 may also detect the position of camera 100 based on images obtained from the two cameras of the imaging unit 302. The HMD control unit 301 may acquire the attitude information obtained by the attitude detection unit 222 inside the camera 100 as the attitude information of the camera 100.

[0083] The non-volatile memory 305 is an electrically erasable and recordable non-volatile memory. The non-volatile memory 305 stores programs, such as those executed by the control unit 311, as described later.

[0084] The working memory 306 is used as a buffer memory for temporarily holding image data captured by the imaging unit 302, as well as as an image display memory for the image display unit 303 and as a working area for the HMD control unit 301.

[0085] The gaze-tracking unit 307 is a camera that acquires images to detect the user's gaze. The gaze-tracking unit 307 is mounted inside the HMD 300 to capture images of the user's eyes when the user wears the HMD 300. The image captured by the camera of the subject (user's eyes) is output to the control unit 311 of the PC 310 via the HMD control unit 301. The control unit 311 detects the gaze of the user wearing the HMD 300 from the image captured by the gaze-tracking unit 307 and identifies the area the user is fixated on in the image display unit 303.

[0086] (Internal configuration of the PC) Referring to Figure 4, the internal configuration of PC310 will be described. PC310 includes a control unit 311, a non-volatile memory 312, a working memory 313, a communication unit 314, and a recording medium 315.

[0087] The control unit 311 is a CPU that controls various parts of the PC310 according to the input signals and the program described later. Alternatively, instead of the control unit 311 controlling the entire PC310, multiple hardware components may share the processing to control the entire PC310. The control unit 311 receives the image acquired by the imaging unit 302 (captured image) and the attitude information acquired by the attitude sensor unit 304 from the HMD300. The control unit 311 performs image processing on the captured image to cancel out aberrations in the optical system of the imaging unit 302 and the optical system of the image display unit 303. Then, the control unit 311 combines the captured image with an arbitrary CG to generate a composite image. 11 transmits the composite image to the HMD control unit 301 in the HMD300.

[0088] Furthermore, the control unit 311 determines the number of controllers 320 included in the captured image. The control unit 311 also uses the information obtained via the communication unit 314 to perform processing to recognize the mounting position of each controller 320. Then, based on the recognized result, the control unit 311 controls each controller to change the operation content for the input information of each controller 320.

[0089] The control unit 311 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 300. For example, when the control unit 311 places a virtual object represented by CG near a specific object that exists in real space within the space represented by the composite image, it increases the size of the virtual object (CG) as the distance between the specific object and the imaging unit 302 decreases. By controlling the position, orientation, and size of the CG in this way, the control unit 311 can generate a composite image in which CG objects that are not actually located in real space appear as if they were.

[0090] Furthermore, the control unit 311 receives information estimated by the HMD control unit 301 of the HMD 300 (coordinate information of each joint point of the hand). The received information is temporarily stored in the working memory 313.

[0091] Furthermore, the control unit 311 receives information on the position or orientation change of the controller 320 from the communication unit 323 of the controller 320. The control unit 311 superimposes display items indicating the indicated positions corresponding to the position or orientation change information of the controller 320 onto the composite image. The control unit 311 may also superimpose display items indicating the indicated positions corresponding to the position and orientation change information of the controller 320 onto the composite image.

[0092] The non-volatile memory 312 is an electrically erasable and recordable non-volatile memory. The non-volatile memory 312 stores information such as the program executed by the control unit 311 (described later) and CG. The control unit 311 can switch the CG read from the non-volatile memory 312 (i.e., the CG used to generate the composite image).

[0093] The working memory 313 is used as a buffer memory to temporarily hold image data captured by the imaging unit 302 and time-series information of the estimated coordinate positions of each joint point of the hand. The working memory 313 is also used as the image display memory for the image display unit 303 and as a work area for the control unit 311.

[0094] Alternatively, wrist joint estimation may be performed by the PC310. In this case, after the captured image is output from the imaging unit 302 to the PC310, the control unit 311 of the PC310 estimates the position or orientation of each joint point of the hand. The control unit 311 then uses this information to process the image and outputs it to the HMD300. Alternatively, the control unit 311 may estimate the position and orientation of each joint point of the hand, use this information to process the image, and output it to the HMD300.

[0095] (Internal configuration of the controller) Referring to Figure 4, the internal configuration of the controller 320 will be explained. The controller 320 includes a controller control unit 321, an operation unit 322, a communication unit 323, a controller attitude sensor unit 324, and an output unit 325.

[0096] The controller control unit 321 is a CPU that controls each component of the controller 320. Instead of the controller control unit 321 controlling the entire controller 320, multiple The entire controller 320 may be controlled by hardware that shares the processing.

[0097] The operation unit 322 includes a button. The operation unit 322 detects whether or not a button has been pressed and transmits the detection information to the PC 310 via the communication unit 323. The operation unit 322 may have multiple types of input formats.

[0098] The communication unit 323 communicates wirelessly with the PC 310 via Bluetooth. If multiple controllers 320 are connected to the PC 310, each controller 320's communication unit 323 communicates wirelessly with the PC 310 via Bluetooth.

[0099] The controller attitude sensor unit 324 has an inertial measurement unit (IMU) consisting of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 320. The detected position and attitude change information is communicated from the communication unit 323 to the PC 310 via the controller control unit 321.

[0100] The output unit 325 consists of an LED light source, a speaker, and a vibration element, among other things.

[0101] (An example of a mixed reality (MR) space) Referring to Figure 5, an example of an MR space experienced by a user wearing the HMD300 in Embodiment 1 will be described. The MR space 500 contains a user 501, the HMD300 worn by user 501, a PC310 that communicates with the HMD300, and a camera 100 that communicates with the PC310. The MR space 500 also contains real objects 502, virtual objects 503, and a virtual window 510.

[0102] The virtual window 510 is an example of the UI of a shooting application. The virtual window 510 displays the live view image 511, a virtual object 512, and an operating element 513. The live view image 511 is an image acquired by the imaging unit 211 of the camera 100. The virtual object 512 is a virtual object whose form corresponds to the position and orientation of the camera 100.

[0103] (Example of HMD300 screen display) Referring to Figure 6, an example of the display on the image display unit 303 of the HMD300 will be explained. The screen 600 in Figure 6 shows an example of the display on the image display unit 303 when the imaging unit 302 of the HMD300 worn by the user 501 captures an image of the MR space.

[0104] Screen 600 displays a real object 502, a virtual object 503, and a virtual window 510. The virtual window 510 also displays a live view image 511, a virtual object 512, and an operating component 513. Additionally, screen 600 displays the user's right hand 601 (the hand holding the camera 100) and left hand 602 (the hand holding a lens, not shown).

[0105] The virtual window 510 is described in an example where it is positioned as if it were virtually existing at any three-dimensional position and orientation in space, as shown in the MR space 500 in Figure 5, but it is not limited to this. The virtual window 510 may also be positioned at a two-dimensional position within the display area of ​​the screen 600.

[0106] (Live view processing for Camera 100) The processing of camera 100 in Embodiment 1 will be explained with reference to the flowchart in Figure 7.

[0107] In step S701, the system control unit 50 controls the communication unit 221 to connect with the PC 310 so that communication is possible. The type of connection method with the PC 310 is not limited. The connection with the PC 310 may be achieved by either wireless communication or wired communication.

[0108] In step S702, the system control unit 50 determines whether or not the PC 310 has requested the acquisition of a live view image (LV image; real-time image). If it determines that the acquisition of a live view image has been requested, it proceeds to step S703. If it determines that the acquisition of a live view image has not been requested, it proceeds to step S704.

[0109] In step S703, the system control unit 50 sends the live view image (live view information) to the PC 310 in response to a request to acquire a live view image. In addition to the live view image information, the system control unit 50 also sends lens optical information.

[0110] In step S704, the system control unit 50 determines whether or not a photo shoot has been requested (either a photo shoot requested from the PC 310 or a photo shoot requested by the user operating the camera 100). If it is determined that a photo shoot has been requested, the process proceeds to step S705. If it is determined that a photo shoot has not been requested, the process proceeds to step S708.

[0111] In step S705, the system control unit 50 performs the image capture process. The system control unit 50 transmits the image (captured image) acquired through the image capture process to the HMD300.

[0112] In step S706, the system control unit 50 receives the composite image generated in the HMD300.

[0113] In step S707, the system control unit 50 saves the composite image received in step 706 to the recording medium 227.

[0114] In step S708, the system control unit 50 determines whether or not communication with PC310 has been disconnected. If it is determined that communication with PC310 has been disconnected, the process in this flowchart ends. If it is determined that communication with PC310 has not been disconnected, the process proceeds to step S702.

[0115] (Live View Processing) The live view processing by the PC310 in Embodiment 1 will be explained with reference to the flowchart in Figure 8. Note that instead of the PC310, the HMD300 (HMD control unit 301) may perform all or part of the processing shown in this flowchart.

[0116] In step S801, the control unit 311 controls the communication unit 314 to connect with the camera 100 so that communication is possible.

[0117] In step S802, the control unit 311 launches the shooting application. The UI (user interface) of the launched shooting application is displayed on the image display unit 303.

[0118] In step S803, the control unit 311 requests the camera 100 to acquire a live view image.

[0119] In step S804, the control unit 311 receives the live view image and lens optical information from the camera 100.

[0120] In step S805, the control unit 311 calculates the position and orientation (position and orientation information) of the camera 100 based on the live view image and lens optical information. For example, SLAM (Simultaneous Localization and Mapping) can be used to calculate the position and orientation. Alternatively, the control unit 311 may calculate the distance from the imaging unit 302 by triangulation using stereo matching with two live view images obtained by the imaging unit 302. The control unit 311 may then calculate (detect) the position and orientation of the camera 100. Furthermore, the control unit 311 may use the orientation information obtained by the orientation detection unit 222 inside the camera 100 as the orientation information of the camera 100.

[0121] In step S806, the control unit 311 renders the object in the virtual space displayed on the HMD 300 so that it appears as if it were an object viewed from the camera 100, based on the position and orientation of the camera 100.

[0122] In step S807, the control unit 311 composites the objects in the virtual space displayed on the HMD300 into the live view image based on the rendering results from step S806. This generates an LV composite image. The control unit 311 then displays the LV composite image in the window of the shooting application.

[0123] In step S808, the control unit 311 determines whether or not a photo shoot has been requested (either by the user operating the photo shoot application or by the user operating the camera 100). If it is determined that a photo shoot has been requested, the process proceeds to step S809. If it is determined that a photo shoot has not been requested, the process proceeds to step S813.

[0124] In step S809, the control unit 311 receives the captured image from the camera 100.

[0125] In step S810, the control unit 311 renders the object in the virtual space displayed on the HMD 300 so that it appears as if it were an object viewed from the camera 100, based on the position and orientation of the camera 100.

[0126] In step S811, the control unit 311 composites the objects in the virtual space displayed on the HMD300 with the captured image based on the rendering results from step S810. Through this process, the control unit 311 generates a composite image.

[0127] In step S812, the control unit 311 transmits the composite image generated in step S811 to the camera 100.

[0128] In step S813, the control unit 311 determines whether or not to terminate the live view display (display of the live view image). If it is determined that the live view display should be terminated, the process in this flowchart ends. If it is determined that the live view display should not be terminated, the process proceeds to step S803.

[0129] (UI display processing) Referring to the flowchart in Figure 9, the UI display process during live view by the PC310 in Embodiment 1 will be explained. Note that instead of the PC310, the HMD300 (HMD control unit 301) may perform all or part of the processing in this flowchart.

[0130] In step S901, the control unit 311 calculates the distance between the HMD 300 (=user's head) and the camera 100 based on the image acquired by the imaging unit 302 (an image of the space captured from the user's viewpoint). The control unit 311 then performs SLAM calculations based on the image, for example. The position and orientation of the HMD300 are calculated. Then, the control unit 311 calculates the distance between the HMD300 and the camera 100 based on the position and orientation of the HMD300 and the position and orientation of the camera 100 calculated in step S805. The control unit 311 may use the distance information calculated in step S805 as the distance information between the HMD300 and the camera 100. Note that once the processing of step S901 is completed, the process may proceed to step S903 instead of step S902.

[0131] In step S902, the control unit 311 determines whether the distance between the camera 100 and the HMD 300 (=user's head) is greater than or equal to the threshold Th1. If it is determined that the distance between the camera 100 and the HMD 300 is greater than or equal to the threshold Th1, the process proceeds to step S903. If it is determined that the distance between the camera 100 and the HMD 300 is less than the threshold Th1 (below a specific threshold), the process proceeds to step S904.

[0132] In step S903, the control unit 311 determines, based on the image acquired by the imaging unit 302, whether the user's hand is away from the camera 100 (or whether the distance between the user's hand and the camera 100 is less than a specific value). If it is determined that the user's hand is away from the camera 100 (the hand is not gripping the camera 100), the process proceeds to step S905. If it is determined that the user's hand is not away from the camera 100 (the hand is gripping the camera 100), the process proceeds to step S906.

[0133] In step S904, the control unit 311 determines whether the user's hand is away from the camera 100 based on the image acquired by the imaging unit 302. If it is determined that the user's hand is away from the camera 100 (the hand is not gripping the camera 100), the process proceeds to step S907. If it is determined that the user's hand is not away from the camera 100 (the hand is gripping the camera 100), the process proceeds to step S908.

[0134] In each of the following steps S905 to S908, a composite image is generated by combining one image with the shooting application 1001 (the image captured by the camera 100), and the composite image is displayed on the image display unit 303.

[0135] In step S905, the control unit 311 generates a composite image in which the shooting application 1001 is positioned so as not to overlap with the camera 100 in the image captured from the user's viewpoint (the viewpoint of the HMD 300), as shown in Figure 10A. At this point, the user is not holding the camera 100, so it is unlikely that the user will perform any operations such as pressing a physical button on the camera 100. For this reason, the control unit 311 may overlay the shooting application 1001 on top of the camera 100 in the image captured from the user's viewpoint (the viewpoint of the HMD 300). At this time, the control unit 311 also places (displays) the operation UI 1002 for controlling the settings (shooting settings) of the camera 100 in the composite image. The control unit 311 then displays the composite image on the image display unit 303. In this case, the control unit 311 estimates the reach of the fingers based on the position and orientation of the finger joints, and if the shooting application 1001 is located outside the reach of the fingers in the MR space, it may display the shooting application 1001 so that a part of the camera 100 overlaps it. In addition, the control unit 311 may display the operation UI 1002 at a predetermined distance from the hand that has released the camera 100, for ease of operation by the user.

[0136] In step S906, the control unit 311 generates a composite image in which the shooting application 1001 is positioned so as not to overlap with the camera 100 in the image captured from the user's viewpoint of the MR space, as shown in Figure 10B. At this time, the control unit 311 does not place the operation UI 1002 in the composite image (it is hidden). The control unit 311 then displays the composite image on the image display unit 303. At this time, the control unit 311 also determines the position of the joints of the fingers. Based on posture, the reach of the fingers is estimated, and if the shooting application 1001 is located outside the reach of the fingers in the MR space, the shooting application 1001 may be displayed so that a part of the camera 100 overlaps it.

[0137] In step S907, the control unit 311 generates a composite image in which the shooting application 1001 is placed in the center of the background image (composite image) (center of the HMD 300's field of view), as shown in Figure 11A, and all UI elements other than the shooting application 1001 are hidden. The control unit 311 also places (displays) the operation UI 1002 of the shooting application in the composite image. The control unit 311 then displays the composite image on the image display unit 303. The background image may be an image of the MR space captured from the user's viewpoint, or it may be a black image. Instead of making the background image black, the control unit 311 may use an image acquired by the imaging unit 302 that has undergone at least defocusing, monochrome, and brightness reduction processing. In this case, the control unit 311 does not need to place the shooting application 1001 in the exact center of the background image (composite image), but rather should place it in a position closer to the center on the image display unit 303 (composite image) compared to steps S905 and S906. The control unit 311 may display (position) the shooting application 1001 larger than in the cases of steps S905 and S906, so that the user can easily see the shooting application 1001.

[0138] In step S908, the control unit 311 generates a composite image in which the shooting application 1001 is placed in the center of the background image (composite image) (center of the HMD 300's field of view), as shown in Figure 11B, and all UI elements other than the shooting application 1001 are hidden. The control unit 311 also does not place (hide) the operation UI 1002 of the shooting application in the composite image. At this time, the control unit 311 does not need to place the shooting application 1001 in the exact center of the background image (composite image), but rather can place it in a position closer to the center of the image display unit 303 (composite image) compared to steps S905 and S906. The control unit 311 may also display (place) the shooting application 1001 larger than in steps S905 and S906. The background image may be an image of the MR space captured from the user's perspective, or it may be a black image.

[0139] In step S909, the control unit 311 determines whether or not to terminate the live view display (LV display). If it is determined that the live view display should be terminated, the process in this flowchart ends. If it is determined that the live view display should not be terminated, the process proceeds to step S901.

[0140] In this way, the position where the shooting application 1001 is displayed is switched depending on the distance between the camera 100 and the HMD 300. Specifically, when the distance between the camera 100 and the HMD 300 is short, the camera 100 occupies a large proportion of the image captured from the user's viewpoint of the MR space, and the need to view that image decreases. Therefore, in such cases, it can be assumed that the user will want to refer to the shooting application 1001, so the shooting application 1001 is placed in the center of the composite image. On the other hand, when the distance between the camera 100 and the HMD 300 is long, the camera 100 occupies a small proportion of the image captured from the user's viewpoint of the MR space, and the user is likely to want to view that image. Therefore, in such cases, the shooting application 1001 is placed in a position that does not overlap with the camera 100, which is an important element in that image.

[0141] For example, when a user is wearing an HMD and taking a picture while holding the imaging device, they may adopt a style where they hold the imaging device with both hands and look through the viewfinder. This style has the advantages of "being able to take pictures at almost the same position as one's own line of sight" and "being able to hold the camera firmly with one's elbows tucked in for stable shooting." For this reason, it is considered a useful shooting style even when wearing the HMD goggles. However, adopting such a style... In the previous configuration, the imaging device occupied most of the image displayed on the HMD, making it difficult for the user to concentrate on taking pictures. To address this issue, this embodiment allows the user to perform the shooting process while viewing the live view image acquired by the camera 100 using the HMD 300, thereby enabling the generation of images that allow for greater concentration on shooting.

[0142] Furthermore, the control unit 311 positions the shooting application (the image captured by camera 100) in a location that does not overlap with camera 100 in the image captured from the user's viewpoint of the MR space, at least when the user's hand is not away from camera 100. This prevents the shooting application from interfering when the user operates the physical buttons of camera 100. In other words, according to this embodiment, it is possible to perform physical operations such as pressing buttons by the user while providing a virtual display.

[0143] <Embodiment 2> Embodiment 2 shows an example in which the UI is displayed only when the gaze detection unit 307 detects the user's gaze and determines that the camera 100 (external imaging device) is not located in the extension of the user's gaze. This prevents the display of unintended UI because the user may have taken their hands off the camera 100's touch panel to operate it.

[0144] Referring to the flowchart in Figure 12, the UI display process during live view by the PC310 in Embodiment 2 will be explained. Note that instead of the PC310, the HMD300 (HMD control unit 301) may perform all or part of the processing in this flowchart. Steps S901 to S909 in the flowchart of Figure 12 are the same as in Embodiment 1 and therefore will not be explained.

[0145] The process in step S1201 begins if it is determined in step S903 that the user's hand has moved away from the camera 100. In step S1201, the control unit 311 acquires the user's gaze information from the gaze imaging unit 307. The control unit 311 then determines whether the camera 100 is located on the extension of the user's gaze. If it is determined that the camera 100 is not located on the extension of the user's gaze, the process proceeds to step S905. If it is determined that the camera 100 is located on the extension of the user's gaze, the process proceeds to step S906.

[0146] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" may be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, "greater than or equal to A" may be rephrased as "greater than (higher; longer; more) than A," and "less than or equal to A" may be rephrased as "less than (lower; shorter; fewer) than A." And "greater than (higher; longer; more) than A" may be rephrased as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be rephrased as "less than or equal to A."

[0147] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0148] Furthermore, the above-mentioned processors are processors in a broad sense, and include general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Pro). Examples include a processing unit (GPU), MPU (Micro Processing Unit), and DSP (Digital Signal Processor). Dedicated processors include, for example, a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), and PLD (Programmable Logic Device). Programmable logic devices include, for example, an FPGA (Field Programmable Gate Array) and a CPLD (Complex Programmable Logic Device).

[0149] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0150] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0151] The above-disclosed embodiments include the following configurations, methods, and programs. (Composition 1) An information processing device that is communicatively connected to an imaging device. A first acquisition means for acquiring a first image of the space including the imaging device, captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A generation means for generating a composite image obtained by combining the first image and the second image, Having The generation means, if the user has the imaging device, generates the composite image in which the second image is arranged so as not to overlap the imaging device in the first image. An information processing device characterized by the following: (Configuration 2) If the generating means places the second image in the space shown by the composite image in a position out of the user's reach, it places the imaging device on top of the second image. The information processing device according to configuration 1, characterized by the above. (Composition 3) The information processing apparatus according to configuration 1 or 2, wherein the generation means, if the user does not have the imaging device, places an operation UI (user interface) for controlling the settings of the imaging device on the composite image. (Composition 4) The generation means, when the user does not possess the imaging device and the imaging device is not located in the extension of the user's line of sight, places an operation UI (user interface) for controlling the settings of the imaging device in the composite image. An information processing device according to configuration 1 or 2, characterized by the above. (Composition 5) The generation means, in the event that the user is no longer holding the imaging device, generates a composite image in which an operation UI (user interface) for controlling the settings of the imaging device is positioned within a predetermined distance from the hand that released the imaging device. An information processing device according to configuration 1 or 2, characterized in that... (Composition 6) The system includes a determination means for determining whether the user's hand is holding the imaging device based on the first image. An information processing device according to any one of configurations 1 to 5, characterized in that... (Composition 7) The second image includes a real-time image acquired from the imaging device. An information processing device according to any one of configurations 1 to 6, characterized in that... (Composition 8) The second image includes an area for receiving user input, An information processing device according to any one of configurations 1 to 7, characterized in that... (Composition 9) The imaging device is controlled based on the user's instructions. An information processing device according to any one of configurations 1 to 8, characterized in that... (Composition 10) The generation means, in the first case where the distance between the user's head and the imaging device is less than or equal to a threshold, places the second image closer to the center of the composite image than in the second case where the distance is greater than the threshold. An information processing device according to any one of configurations 1 to 9, characterized by the above. (Composition 11) The system further includes an acquisition means for acquiring information on the distance between the user's head and the imaging device. The information processing apparatus according to configuration 10, characterized by the above. (Composition 12) The acquisition means acquires information about the distance between the user's head and the imaging device based on the first image. The information processing apparatus according to configuration 11, characterized in that... (Composition 13) In the first case, the generating means generates the composite image by combining the black image and the second image. An information processing device according to any one of configurations 10 to 12, characterized in that... (Composition 14) In the first case, the generating means generates the composite image by combining the first image with an image obtained by applying at least one of defocusing, monochrome processing, and brightness reduction processing to the first image, and the second image. An information processing device according to any one of configurations 10 to 12, characterized in that... (method) A control method for an information processing device that is communicatively connected to an imaging device. A first acquisition step involves acquiring a first image of the space including the imaging device, captured according to the user's viewpoint. A second acquisition step involves acquiring a second image of the space captured by the imaging device, The method includes a generation step of generating a composite image by combining the first image and the second image. In the generation step, if the user has the imaging device, the composite image is generated such that the second image does not overlap the imaging device in the first image. A control method for an information processing device characterized by the following features. (program) The computer functions as one of the means of the information processing device described in any of configurations 1 to 14. A program to make that happen. [Explanation of Symbols]

[0152] 100: Camera (imaging device), 300: HMD, 310: PC (Information Processing Unit), 302: Imaging Unit, 311: Control Unit, 314: Communication Unit

Claims

1. An information processing device that is communicatively connected to an imaging device. A first acquisition means for acquiring a first image of the space including the imaging device, captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A generation means for generating a composite image obtained by combining the first image and the second image, Having The generation means, if the user has the imaging device, generates the composite image in which the second image is arranged so as not to overlap the imaging device in the first image. An information processing device characterized by the following:

2. If the generation means places the second image in the space shown by the composite image in a position out of the user's reach, it places the imaging device on top of the second image. The information processing apparatus according to feature 1.

3. The information processing apparatus according to claim 1, characterized in that the generation means, if the user does not have the imaging device, places an operation UI (user interface) for controlling the settings of the imaging device on the composite image.

4. The generation means, when the user does not possess the imaging device and the imaging device is not located in the extension of the user's line of sight, places an operation UI (user interface) for controlling the settings of the imaging device in the composite image. The information processing apparatus according to feature 1.

5. The generation means, in the event that the user is no longer holding the imaging device, generates a composite image in which an operation UI (user interface) for controlling the settings of the imaging device is positioned within a predetermined distance from the hand that released the imaging device. The information processing apparatus according to claim 1, characterized in that...

6. The imaging device has a determination means for determining whether or not the user's hand is holding the imaging device, based on the first image. The information processing apparatus according to claim 1, characterized in that...

7. The second image includes a real-time image acquired from the imaging device. The information processing apparatus according to claim 1, characterized in that...

8. The second image includes an area for receiving user input, The information processing apparatus according to claim 1, characterized in that...

9. The imaging device is controlled based on the user's instructions. The information processing apparatus according to claim 1, characterized in that...

10. The generation means, in the first case where the distance between the user's head and the imaging device is less than or equal to a threshold, places the second image closer to the center of the composite image than in the second case where the distance is greater than the threshold. The information processing apparatus according to feature 1.

11. The system further includes an acquisition means for acquiring information on the distance between the user's head and the imaging device. The information processing apparatus according to feature 10.

12. The acquisition means acquires information about the distance between the user's head and the imaging device based on the first image. The information processing apparatus according to claim 11, characterized in that...

13. In the first case, the generation means generates the composite image by combining the black image and the second image. The information processing apparatus according to claim 10, characterized in that...

14. In the first case, the generating means generates the composite image by combining the second image with an image obtained by applying at least one of the following processes to the first image: defocusing, monochrome, and brightness reduction. The information processing apparatus according to claim 10, characterized in that...

15. A control method for an information processing device that is communicatively connected to an imaging device. A first acquisition step involves acquiring a first image of the space including the imaging device, captured according to the user's viewpoint. A second acquisition step involves acquiring a second image of the space captured by the imaging device, The method includes a generation step of generating a composite image by combining the first image and the second image. In the generation step, if the user has the imaging device, the composite image is generated such that the second image does not overlap the imaging device in the first image. A control method for an information processing device characterized by the following features.

16. A program for causing a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 14.

Images