Information processing device, control method for information processing device, program

Abstract

When using an imaging device to capture images of a specific space, the system generates images to assist the user in concentrating on the capture process. [Solution] An information processing device that is communicatively connected to an imaging device includes: a first acquisition means for acquiring a first image of space captured according to the user's viewpoint; a second acquisition means for acquiring a second image of space captured by the imaging device; and a generation means for generating a composite image by combining the second image and a third image. In the first case, where the distance between the user's head and the imaging device is greater than a threshold, the third image is the first image. In the second case, where the distance between the user's head and the imaging device is less than or equal to the threshold, the generation means places the second image closer to the center of the composite image than in the first case.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] As a technology that enables experiencing a virtual space, VR (Virtual Reality) technology is known. Also, as a technology that fuses the real space and the virtual space in real time and seamlessly, so-called MR (Mixed Reality) technology is known. For a device that can experience such technologies, for example, a head-mounted device typified by an HMD (Head Mounted Display) is used.

[0003] Patent Document 1 describes a method of determining whether to display a virtual object or a real object based on whether the distance between the position of the viewpoint and the position of the virtual object exceeds a set threshold when the virtual object and the real object overlap.

[0004] In order to photograph a space (MR space) including the real space and virtual objects at an arbitrary viewing angle, a screen shot function of an HMD equipped with MR technology is used. In this case, in the HMD, due to weight and other factors, there is a limit to the photographing function, so the quality of the image that can be photographed is not high compared to an imaging device that images a real space with high photographing performance.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] To capture high-quality MR images, it was necessary to use an external imaging device with high imaging performance, and to capture the MR space while checking the field of view and preview image from the perspective of the external imaging device. However, until now, capturing images with an external imaging device while viewing the MR space while wearing an HMD had not been considered. Applying conventional imaging methods, which are used without an HMD, to a state where an HMD is worn can reduce the operability of the imaging device due to the HMD, making it difficult to concentrate on capturing images.

[0007] Therefore, the objective is to generate images that assist the user in concentrating on capturing images in an MR space using an imaging device. [Means for solving the problem]

[0008] One aspect of the present invention is, An information processing device that is communicatively connected to an imaging device. A first acquisition means for acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A generation means for generating a composite image obtained by combining the second image and the third image, Having In the first case where the distance between the user's head and the imaging device is greater than the threshold, the third image is the first image. The generation means is such that the distance between the user's head and the imaging device is less than or equal to the threshold. In the second case, the second image is positioned closer to the center of the composite image than in the first case. This is an information processing device characterized by the following features. [Effects of the Invention]

[0009] According to the present invention, when an imaging device is used to photograph a specific space, it is possible to generate images that assist the user in concentrating on the shooting process. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing the system configuration according to Embodiment 1. [Figure 2] This is an external view of the camera according to Embodiment 1. [Figure 3] This is an internal configuration diagram of the camera according to Embodiment 1. [Figure 4] This is an internal configuration diagram of the HMD, etc., according to Embodiment 1. [Figure 5] This is a diagram illustrating the MR space according to Embodiment 1. [Figure 6] This figure illustrates an example of the display on the HMD according to Embodiment 1. [Figure 7] This is a flowchart of the live view processing of the camera according to Embodiment 1. [Figure 8] This is a flowchart of the live view processing on a PC according to Embodiment 1. [Figure 9] This is a flowchart of the UI display process of the PC according to Embodiment 1. [Figure 10] This is a flowchart of the live view processing of the camera according to Embodiment 2. [Figure 11] This is a flowchart of the UI display process of the PC according to Embodiment 2. [Figure 12] This is a diagram illustrating the composite image according to Embodiment 1. [Figure 13] This is a diagram illustrating the composite image according to Embodiment 1. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] <Embodiment 1> (Configuration of the entire system) Referring to FIG. 1, an example of the configuration of the entire system according to Embodiment 1 will be described. The information processing system 1 includes a camera 100, an HMD 300, a PC (personal computer) 310, and a controller 320.

[0013] The camera 100 is connected to the PC 310 in a communicable state, either wired or wirelessly. The camera 100 transmits and receives various data (such as live view image data and captured image data). Note that, for example, instead of the camera 100, an imaging device (such as a smartphone or a tablet terminal) that can realize the functions described below may be used. Note that the camera 100 may communicate with the HMD 300, not limited to the PC 310.

[0014] The HMD 300 is a display device (a head-mounted electronic device) that can be worn on the user's head. The HMD 300 displays a composite image in which "an imaging image captured by the HMD 300 within the range in front of the user" and "content such as CG in a form corresponding to the position and orientation of the HMD 300" are combined.

[0015] The PC 310 is an information processing device that controls the HMD 300. The PC 310 is connected to the HMD 300 by wire such as a USB cable or wirelessly such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). For example, the PC 310 generates a composite image by combining an imaging image and CG, and transmits the composite image to the HMD 300. In this case, when the PC 310 receives a live view image or a captured image from the camera 100, it generates a composite image in which the received image and CG in a form corresponding to the position and orientation of the camera 100 are combined. The PC 310 transmits the composite image to the HMD 300.

[0016] A smartphone or tablet device may be used instead of the PC310. Furthermore, each component of the PC310 may also be present in the HMD300. While Embodiment 1 shows an example where the PC310 and camera 100 are connected wirelessly, the PC310 and camera 100 may also be connected via a wired connection.

[0017] The controller 320 performs various controls on the HMD 300. When the PC 310 is in a specific control mode, the HMD 300 is controlled according to the user's operation when the user operates the controller 320. As shown in Figure 1, the controller 320 is an operating component that is "ring-shaped and can be worn and supported on the user's finger" or "handheld and can be held in the hand". The controller 320 also has physical buttons for making decisions and selections on the display.

[0018] Controller 320 communicates with PC 310 wirelessly via Bluetooth. Controller 320 may also communicate with HMD 300, not just PC 310. The user can change the indicated position on the display by moving Controller 320. The indicated position may be represented as a point, or as a virtual ray connecting the point and the controller with a straight line (line segment) or dotted line. The user can perform menu selection and confirmation operations by pressing physical buttons.

[0019] The controller 320 is ring-shaped or handheld. However, the controller 320 may be any shape as long as it can be supported by a finger, hand, or arm. The buttons on the controller 320 are described as physical buttons, but they may be operable in the form of a trackpad, touch panel, wheel, or trackball. In addition to button presses, the controller 320 may also accept slide, flick, and touch operations. The controller 320 may be wearable on at least one of the fingers, hand, or arm. The controller 320 may also be attached to an object held in the hand, and positional and orientation information of the attached position may be obtained from sensors. Examples of such objects include objects that mimic tools.

[0020] (External structure of a digital camera) Figures 2A and 2B show an example of the external configuration of the camera 100, which is an imaging device. Figure 2A is a perspective view of the camera 100 from the front. Figure 2B is a perspective view of the camera 100 from the rear.

[0021] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface.

[0022] The shutter button 101 is an operation unit for preparing to take a picture or giving a shooting command. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode selector switch 103 is an operation unit for switching between various modes.

[0023] The main electronic dial 104 is used to change settings such as shutter speed and aperture. This is a rotary control unit. The sub-electronic dial 105 is a rotary control unit used for moving the selection frame (cursor) and advancing images.

[0024] The video button 106 is an operation unit for instructing the start and stop of video recording. The viewfinder external display unit 107 displays various settings such as shutter speed and aperture.

[0025] The camera 100 also includes a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back.

[0026] The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108.

[0027] The directional keys 110 are an operation unit consisting of keys that can be pressed in the up, down, left, and right directions (4-way keys). The camera 100 can be controlled according to the position where the directional keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed when confirming a selection item.

[0028] The AE lock button 112 is an operation button that is pressed to fix the exposure state when the camera is in shooting standby mode.

[0029] The zoom button 113 is an operation unit used to switch the zoom mode on and off in the live view display (LV display) of the shooting mode. When the zoom mode is on, the live view image (LV image) will be enlarged or reduced when the main electronic dial 104 is operated. The zoom button 113 is also used in playback mode to enlarge the playback image or increase the magnification ratio.

[0030] The playback button 114 is an operation unit for switching between shooting mode and playback mode. When the playback button 114 is pressed in shooting mode, the device switches to playback mode, and the most recent image recorded on the recording medium 227 (described later) is displayed on the display unit 108.

[0031] The menu button 115 is an operation button that is pressed to display a menu screen on the display unit 108 that allows for various settings. The user can intuitively make various settings using the menu screen displayed on the display unit 108, the directional keys 110, and the SET button 111.

[0032] The eyepiece section 116 is the part that allows the user to bring their eye close to (eye-at) the eyepiece viewfinder (a type of viewfinder) 117. The user can view the image displayed on the EVF 217 (Electronic View Finder) through the eyepiece section 116.

[0033] The eyepiece detection unit 118 is a sensor that detects whether or not a user is looking into the eyepiece unit 116.

[0034] The touch bar 119 is a line-shaped touch operation area (line touch sensor) that can accept touch operations. The touch bar 119 is positioned so that it can be touched with the right thumb when the grip section 120 is held with the right hand (with the little finger, ring finger, and middle finger of the right hand), allowing the shutter button 101 to be pressed with the right index finger. In other words, the touch bar 119 can be operated when the user is looking through the eyepiece viewfinder 117 with their eyepiece in the eyepiece section 116 and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can be operated by tapping (touching and releasing without moving within a predetermined period of time) and sliding (touching and then moving the touched area) on the touch bar 119. It can accept operations such as moving the touch position. The touch bar 119 is a different operation unit from the touch panel 109 and does not have a display function. The touch bar 119 in this embodiment is a multifunction bar and functions as, for example, an M-Fn bar.

[0035] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, and the like.

[0036] The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. When the camera 100 is held with the grip section 120 held by the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand.

[0037] The thumb rest section 121 (thumb waiting position) is a grip section located on the back of the camera 100, in a position where it is easy to rest the thumb of the right hand holding the grip section 120 when no controls are being operated. The thumb rest section 121 is made of rubber material or the like to enhance the holding power (grip feel).

[0038] The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 227 and the slot for storing the recording medium 227 by closing the slot.

[0039] The communication terminal 124 is a terminal for communicating with the lens unit 200.

[0040] (Internal structure of a digital camera) Figure 3 shows an example of the internal configuration of camera 100. In Figure 3, components identical to those in Figure 2 are denoted by the same reference numerals, and their descriptions are omitted as appropriate. A lens unit 200 is attached to camera 100.

[0041] First, let's describe the lens unit 200. The lens unit 200 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens unit 200 is a single-lens reflex lens and is an example of a normal lens. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0042] The aperture diameter of aperture 201 is adjustable. Lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of aperture 201. The AF drive circuit 204 drives lens 202 to focus.

[0043] The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc., based on instructions from the system control unit 50. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203. The lens system control circuit 205 focuses by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication takes place via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100.

[0044] Next, we will explain the camera 100. The camera 100 consists of a shutter 210 and an imaging unit 2 11, it includes an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0045] The shutter 210 is a focal-plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50.

[0046] The imaging unit 211 is an image sensor composed of a CCD or CMOS element, which converts an optical image into an electrical signal. The imaging unit 211 may also have an image plane phase difference sensor that outputs defocus amount information to the system control unit 50.

[0047] The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal.

[0048] The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing, and color conversion) on data from the A / D converter 212 or data from the memory control unit 213. The image processing unit 214 also performs predetermined calculations using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. This process enables TTL (through-the-lens) AF processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data and performs TTL AWB (auto white balance) processing based on the obtained calculation results. Image data from the A / D converter 212 is written to memory 215 via the image processing unit 214 and memory control unit 213. Alternatively, image data from the A / D converter 212 is written to memory 215 via memory control unit 213 without going through the image processing unit 214.

[0049] Memory 215 stores "image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212" and "image data for display on the display unit 108 and EVF 217". Memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. Memory 215 also serves as memory for image display (video memory).

[0050] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies the analog signal to the display unit 108 and EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 and EVF 217 via the D / A converter 216. The display unit 108 and EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and EVF 217 are displays such as LCDs or OLEDs. The digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216. The analog signal is sequentially transferred to the display unit 108 and EVF 217, enabling live view display, which shows an image representing the space in real time.

[0051] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements each process of the flowchart described later by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0052] Furthermore, camera 100 has system memory 218, non-volatile memory 219, and system type It has a motor 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118.

[0053] System memory 218 may include, for example, RAM. System memory 218 stores constants and variables for the operation of the system control unit 50, programs read from non-volatile memory 219, and so on.

[0054] The non-volatile memory 219 is an electrically erasable and recordable memory. For example, an EEPROM is used for the non-volatile memory 219. Constants for the operation of the system control unit 50, programs, etc., are stored in the non-volatile memory 219. The program here refers to a program for executing the flowchart processing described later.

[0055] The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals or audio signals to and from external devices connected by wireless or wired cables.

[0056] The communication unit 221 can connect to both a wireless LAN (Local Area Network) and the internet. Furthermore, the communication unit 221 can communicate with external devices via Bluetooth® and Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227. The communication unit 221 can also receive image data and other various types of information from external devices.

[0057] The attitude detection unit 222 detects the attitude of the camera 100 relative to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image before recording. For example, an acceleration sensor or a gyroscope sensor can be used for the attitude detection unit 222. The attitude detection unit 222 can also be used to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.).

[0058] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece section 116 of the "eyepiece viewfinder 117 with built-in EVF 217". For example, an infrared proximity sensor can be used for the eyepiece detection unit 118. When an object approaches the eyepiece section 116, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece section 116 to the object can be determined by the amount of infrared light received (=sensor value). In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece section 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eye-to-eye contact) and departure (eye-away) of an eye (object) to the eyepiece section 116 of the eyepiece viewfinder 117. When an object approaches the eyepiece 116 within a predetermined distance from the non-eyepiece state (not close state), it is detected that the eye has been placed. Conversely, when the eyepiece is in the close state, it is detected that the eye has been removed when the object that was detected approaching moves beyond a predetermined distance. The threshold for detecting eye placement and the threshold for detecting eye removal may be different, for example, by providing hysteresis. Furthermore, after eye placement is detected, the eyepiece is considered to be in the eyepiece state until eye removal is detected. After eye removal is detected, the eyepiece is considered to be in the non-eyepiece state until eye placement is detected again.

[0059] The system control unit 50 switches the display (display state) / hide (hide state) on the display unit 108 and EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, at least when in shooting standby mode, and the display destination switching setting is set to automatic switching, In this case, the system control unit 50 turns on the display to the display unit 108 when not using the eyepiece, and hides the EVF 217. Also, when using the eyepiece, the system control unit 50 turns on the display to the EVF 217, and hides the display unit 108. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as eyepiece use may be used.

[0060] The camera 100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium interface 226, an operation unit 228, and the like.

[0061] The external viewfinder display unit 107 displays various settings of the camera 100 (such as shutter speed and aperture) via the external viewfinder display drive circuit 223.

[0062] The power control unit 224 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and the like. The power control unit 224 detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, supplying the necessary voltage to each part (including the recording medium 227) for the required period.

[0063] The power supply unit 225 is a primary battery (such as an alkaline battery or a lithium battery), a secondary battery (such as a NiCd battery, a NiMH battery or a Li battery), or an AC adapter.

[0064] The recording medium I / F 226 is an interface to the recording medium 227. The recording medium 227 is a memory card or the like for recording captured images. The recording medium 227 is composed of semiconductor memory or a magnetic disk or the like.

[0065] The recording medium 227 may be detachable from the camera 100, or it may be built into the camera 100.

[0066] The operation unit 228 is an input unit that receives user input (user operation). The operation unit 228 is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode selector switch 103, a touch panel 109, and other operation units 229.

[0067] Other control units 229 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a play button 114, a menu button 115, a touch bar 119, and the like.

[0068] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231.

[0069] The first shutter switch 230 turns on during the operation of the shutter button 101, specifically when it is half-pressed (indicating preparation for shooting), and generates the first shutter switch signal SW1. Upon generation of the first shutter switch signal SW1, the system control unit 50 starts the shooting preparation process (AF processing, AE processing, AWB processing, EF processing, etc.).

[0070] The second shutter switch 231 turns on when the shutter button 101 is fully pressed (shooting instruction), generating the second shutter switch signal SW2. Upon generation of the second shutter switch signal SW2, the system control unit 50 performs a series of shooting processes (from reading the signal from the imaging unit 211 to generating and recording an image file containing the captured image). Start writing to media 227.

[0071] The mode switch 103 switches the operating mode of the system control unit 50 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. Modes included in the still image shooting mode include auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), program AE mode (P mode), etc. Modes included in the still image shooting mode include various scene modes and custom modes, which are shooting settings for different shooting scenes. The user can switch directly to any of the above shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0072] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is mounted on top of the display surface of the display unit 108 so that the light transmittance of the touch panel 109 does not interfere with the display of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109. Either method is acceptable.

[0073] The system control unit 50 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). The finger or pen that was touching the touch panel 109 is lifted (released), meaning the touch action ends (hereinafter referred to as "Touch-Up"). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0074] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0075] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 50 via the internal bus. The system control unit 50 receives the notified information Based on the information, it is determined what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If it is detected that a touch move has been made beyond a predetermined distance, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109 and then quickly moved a certain distance and then released is called a flick. In other words, a flick is an operation in which the finger is quickly traced across the touch panel 109 as if flicking it. If it is detected that a touch move has been made beyond a predetermined distance and at a predetermined speed or faster, and a touch-up is then detected, it is determined that a flick has been performed (it can be determined that a flick followed a slide operation). Furthermore, a touch operation in which multiple locations (for example, two points) are touched together (multitouch) and the touch positions are brought closer together is called a pinch-in, and a touch operation in which the touch positions are moved further apart is called a pinch-out. The actions of pinching out and pinching in are collectively referred to as pinch operations (or simply pinch).

[0076] (HMD configuration) Referring to Figure 4, an example of the configuration of the HMD300 will be described. The HMD300 includes an HMD control unit 301, an imaging unit 302, an image display unit 303, an attitude sensor unit 304, a non-volatile memory 305, a working memory 306, and a gaze imaging unit 307.

[0077] The HMD control unit 301 is a CPU that controls each component of the HMD 300. When the HMD control unit 301 acquires a composite image (an image created by combining an image captured by the imaging unit 302 of the space in front of the user with computer graphics) from the PC 310, it displays the composite image on the image display unit 303. Alternatively, instead of the HMD control unit 301 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0078] The imaging unit 302 includes two cameras (imaging devices). The two cameras are for capturing images used for synthesis with images in a virtual space and for generating positional orientation information, and have an imaging unit for the left eye and an imaging unit for the right eye. The left eye imaging unit captures a moving image of the real space corresponding to the left eye of the HMD300 wearer, and outputs an image (captured image) of each frame in the moving image from the left eye imaging unit. The right eye imaging unit captures a moving image of the real space corresponding to the right eye of the HMD300 wearer, and outputs an image (captured image) of each frame in the moving image from the right eye imaging unit. In other words, the imaging unit 302 acquires captured images as stereo images having parallax that substantially coincides with the positions of the left and right eyes of the HMD300 wearer. In addition, distance information from the two cameras to the subject can be acquired as distance information by measuring distance with the stereo cameras. In the case of an HMD for an MR system, it is preferable that the central optical axis of the imaging range of the imaging unit is arranged to substantially coincide with the line of sight of the HMD wearer.

[0079] Each of the left and right eye imaging units includes an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incident light as an image. The images captured by the two cameras of the subject (the area directly in front of the user) are output to the PC 310 and the HMD control unit 301. The imaging unit 302 may output video instead of an image.

[0080] The image display unit 303 displays a composite image. The image display unit 303 has a liquid crystal panel or an organic EL panel, etc. When the user is wearing the HMD 300, the image display unit 303 is positioned in front of each of the user's eyes. It is also possible to use a device with a semi-transparent half-mirror for the image display unit 303. In this case, for example, the image display unit 303 superimposes CG directly onto the real space visible through the half-mirror using a technology generally known as AR (Augmented Reality). The image may be displayed in a visible manner. Alternatively, the image display unit 303 may display an image of a completely virtual space without using captured images, using a technology commonly known as VR (Virtual Reality).

[0081] The attitude sensor unit 304 acquires attitude (and position) information of the HMD 300. The attitude sensor unit 304 may also acquire user (the user wearing the HMD 300) attitude information that corresponds to the attitude (and position) of the HMD 300. For example, the attitude sensor unit 304 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 304 is used to acquire user attitude information, and the HMD control unit 301 outputs the user's attitude information to the PC 310. The attitude information may also be acquired from one or more of the following: a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, or an angular velocity sensor.

[0082] The HMD control unit 301 estimates the position or orientation of each joint point of the user's hand and fingers based on images obtained from the two cameras of the imaging unit 302. The joint points include characteristic points of various parts of the hand, such as the finger joints, fingertips, back of the hand (palm), and arm. Each joint point represents a coordinate position. Based on information from multiple joint points, the hand's orientation can be estimated. Methods for estimating the position or orientation of the hand and each joint point can include, for example, known machine learning methods for object recognition and pose estimation using convolutional neural networks. Furthermore, the depth-direction position information of each joint point of the hand can be obtained, for example, by calculating the distance from the imaging unit 302 to each joint point through triangulation using stereo matching with images obtained from the two cameras of the imaging unit 302. The estimated coordinate information of each joint point of the hand is output from the HMD control unit 301 to the PC 310.

[0083] The non-volatile memory 305 is an electrically erasable and recordable non-volatile memory that stores programs, such as those executed by the control unit 311 (described later).

[0084] The working memory 306 is used as a buffer memory for temporarily holding image data captured by the imaging unit 302, as well as as an image display memory for the image display unit 303 and as a working area for the HMD control unit 301.

[0085] The gaze-tracking unit 307 is a camera that acquires images to detect the user's gaze. The gaze-tracking unit 307 is mounted inside the HMD 300 to capture images of the user's eyes when the user wears the HMD 300. The image captured by the camera of the subject (user's eyes) is output to the control unit 311 of the PC 310 via the HMD control unit 301. The control unit 311 detects the gaze of the user wearing the HMD 300 from the image captured by the gaze-tracking unit 307 and identifies the area the user is fixated on in the image display unit 303.

[0086] (Internal configuration of the PC) Referring to Figure 4, the internal configuration of PC310 will be described. PC310 includes a control unit 311, a non-volatile memory 312, a working memory 313, a communication unit 314, and a recording medium 315.

[0087] The control unit 311 is a CPU that controls various parts of the PC310 according to the input signals and the program described later. Alternatively, instead of the control unit 311 controlling the entire PC310, multiple hardware components may share the processing to control the entire PC310. The control unit 311 receives the image acquired by the imaging unit 302 (captured image) and the attitude information acquired by the attitude sensor unit 304 from the HMD300. The control unit 311 performs image processing on the captured image to cancel out aberrations in the optical system of the imaging unit 302 and the optical system of the image display unit 303. The control unit 311 combines the captured image with an arbitrary CG to generate a composite image. The control unit 311 then transmits the composite image to the HMD control unit 301 in the HMD 300.

[0088] Furthermore, the control unit 311 determines the number of controllers 320 included in the captured image. The control unit 311 also uses the information obtained via the communication unit 314 to perform processing to recognize the mounting position of each controller 320. Then, based on the recognized result, the control unit 311 controls each controller to change the operation content for the input information of each controller 320.

[0089] The control unit 311 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 300. For example, when the control unit 311 places a virtual object represented by CG near a specific object that exists in real space within the space represented by the composite image, it increases the size of the virtual object (CG) as the distance between the specific object and the imaging unit 302 decreases. By controlling the position, orientation, and size of the CG in this way, the control unit 311 can generate a composite image in which CG objects that are not actually located in real space appear as if they were.

[0090] Furthermore, the control unit 311 receives information estimated by the HMD control unit 301 of the HMD 300. The received information is temporarily stored in the working memory 313.

[0091] Furthermore, the control unit 311 receives information on the position or orientation change of the controller 320 from the communication unit 323 of the controller 320. The control unit 311 superimposes display items indicating the indicated positions corresponding to the position or orientation change information of the controller 320 onto the synthesized image. The control unit 311 may also superimpose display items indicating the indicated positions corresponding to the position and orientation change information of the controller 320 onto the synthesized image.

[0092] The non-volatile memory 312 is an electrically erasable and recordable non-volatile memory. The non-volatile memory 312 stores information such as the program executed by the control unit 311 (described later) and CG. The control unit 311 can switch the CG read from the non-volatile memory 312 (i.e., the CG used to generate the composite image).

[0093] The working memory 313 is used as a buffer memory to temporarily hold image data captured by the imaging unit 302 and time-series information of the estimated coordinate positions of each joint point of the hand. The working memory 313 is also used as the image display memory for the image display unit 303 and as a work area for the control unit 311.

[0094] Alternatively, wrist joint estimation may be performed by the PC310. In this case, after the captured image is output from the imaging unit 302 to the PC310, the control unit 311 of the PC310 estimates the position or orientation of each joint point of the hand. The control unit 311 then uses this information to process the image and outputs it to the HMD300. Alternatively, the control unit 311 may estimate the position and orientation of each joint point of the hand, use this information to process the image, and output it to the HMD300.

[0095] (Internal configuration of the controller) Referring to Figure 4, the internal configuration of the controller 320 will be explained. The controller 320 includes a controller control unit 321, an operation unit 322, a communication unit 323, a controller attitude sensor unit 324, and an output unit 325.

[0096] The controller control unit 321 is a CPU that controls each component of the controller 320. Instead of the controller control unit 321 controlling the entire controller 320, multiple The entire controller 320 may be controlled by hardware that shares the processing.

[0097] The operation unit 322 includes a button. The operation unit 322 detects whether or not a button has been pressed and transmits the detection information to the PC 310 via the communication unit 323. The operation unit 322 may have multiple types of input formats.

[0098] The communication unit 323 communicates wirelessly with the PC 310 via Bluetooth. If multiple controllers 320 are connected to the PC 310, each controller 320's communication unit 323 communicates wirelessly with the PC 310 via Bluetooth.

[0099] The controller attitude sensor unit 324 has an inertial measurement unit (IMU) consisting of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 320. The detected position and attitude change information is communicated from the communication unit 323 to the PC 310 via the controller control unit 321.

[0100] The output unit 325 consists of an LED light source, a speaker, and a vibration element, among other things.

[0101] (An example of a mixed reality (MR) space) Referring to Figure 5, an example of an MR space experienced by a user wearing the HMD300 in Embodiment 1 will be described. The MR space 500 contains a user 501, the HMD300 worn by user 501, a PC310 that communicates with the HMD300, and a camera 100 that communicates with the PC310. The MR space 500 also contains real objects 502, virtual objects 503, and a virtual window 510.

[0102] The virtual window 510 is an example of the UI of a shooting application. The virtual window 510 displays the live view image 511, a virtual object 512, and an operating element 513. The live view image 511 is an image acquired by the imaging unit 211 of the camera 100. The virtual object 512 is a virtual object whose form corresponds to the position and orientation of the camera 100.

[0103] Additionally, arrow 504 indicates the direction in which the imaging unit 302 of the HMD 300 worn by user 501 is capturing images. Arrow 505 indicates the direction in which the imaging unit 211 of camera 100 is capturing images.

[0104] (Example of HMD300 screen display) Referring to Figure 6, an example of the display on the image display unit 303 of the HMD300 will be explained. The screen 600 in Figure 6 shows an example of the display on the image display unit 303 when the imaging unit 302 of the HMD300 worn by the user 501 captures an image of the range in the direction of the arrow 504 in Figure 5.

[0105] Screen 600 displays a real object 502, a virtual object 503, and a virtual window 510. The virtual window 510 also displays a live view image 511, a virtual object 512, and an operating component 513.

[0106] The position and orientation of the virtual object 503 and the virtual window 510 are adjusted to correspond to the position and orientation when the imaging unit 302 captures an image within the range indicated by arrow 504 in Figure 5. The orientation of the virtual object 512 displayed in the virtual window 510 is adjusted to correspond to the position and orientation when the imaging unit 211 of the camera 100 captures an image within the range indicated by arrow 505 in Figure 5. The virtual window 510 is described in an example where it appears to be virtually located at any three-dimensional position and orientation in space, as shown in the MR space 500 in Figure 5, but it is not limited to this. The virtual window 510 is displayed on the screen 600. It may be placed at any two-dimensional position within the display area.

[0107] (Live view processing for Camera 100) The processing of camera 100 in Embodiment 1 will be explained with reference to the flowchart in Figure 7.

[0108] In step S701, the system control unit 50 controls the communication unit 221 to connect with the PC 310 so that communication is possible. The type of connection method with the PC 310 is not limited. The connection with the PC 310 may be achieved by either wireless communication or wired communication.

[0109] In step S702, the system control unit 50 determines whether or not the PC 310 has requested the acquisition of a live view image (LV image; real-time image). If it determines that the acquisition of a live view image has been requested, it proceeds to step S703. If it determines that the acquisition of a live view image has not been requested, it proceeds to step S704.

[0110] In step S703, the system control unit 50 sends the live view image (live view information) to the PC 310 in response to a request to acquire a live view image. In addition to the live view image information, the system control unit 50 also sends lens optical information.

[0111] In step S704, the system control unit 50 determines whether or not a photo shoot has been requested (either a photo shoot requested from the PC 310 or a photo shoot requested by the user operating the camera 100). If it is determined that a photo shoot has been requested, the process proceeds to step S705. If it is determined that a photo shoot has not been requested, the process proceeds to step S708.

[0112] In step S705, the system control unit 50 performs the image capture process. The system control unit 50 transmits the image (captured image) acquired through the image capture process to the HMD300.

[0113] In step S706, the system control unit 50 receives the composite image generated in the HMD300.

[0114] In step S707, the system control unit 50 saves the composite image received in step 706 to the recording medium 227.

[0115] In step S708, the system control unit 50 determines whether or not communication with PC310 has been disconnected. If it is determined that communication with PC310 has been disconnected, the process in this flowchart ends. If it is determined that communication with PC310 has not been disconnected, the process proceeds to step S702.

[0116] (Live View Processing) The live view processing by the PC310 in Embodiment 1 will be explained with reference to the flowchart in Figure 8. Note that instead of the PC310, the HMD300 (HMD control unit 301) may perform all or part of the processing shown in this flowchart.

[0117] In step S801, the control unit 311 controls the communication unit 314 to connect with the camera 100 so that communication is possible.

[0118] In step S802, the control unit 311 launches the shooting application. The UI (user interface) of the launched shooting application is displayed on the image display unit 303.

[0119] In step S803, the control unit 311 requires the camera 100 to acquire a live view image. To seek.

[0120] In step S804, the control unit 311 receives the live view image and lens optical information from the camera 100.

[0121] In step S805, the control unit 311 calculates the position and orientation (position and orientation information) of the camera 100 based on the live view image and lens optical information. For example, SLAM (Simultaneous Localization and Mapping) can be used to calculate the position and orientation.

[0122] In step S806, the control unit 311 renders the object in the virtual space displayed on the HMD 300 so that it appears as if it were an object viewed from the camera 100, based on the position and orientation of the camera 100.

[0123] In step S807, the control unit 311 composites the objects in the virtual space displayed on the HMD300 into the live view image based on the rendering results from step S806. This generates an LV composite image. The control unit 311 then displays the LV composite image in the window of the shooting application.

[0124] In step S808, the control unit 311 determines whether or not a photo shoot has been requested (either by the user operating the photo shoot application or by the user operating the camera 100). If it is determined that a photo shoot has been requested, the process proceeds to step S809. If it is determined that a photo shoot has not been requested, the process proceeds to step S813.

[0125] In step S809, the control unit 311 receives the captured image from the camera 100.

[0126] In step S810, the control unit 311 renders the object in the virtual space displayed on the HMD 300 so that it appears as if it were an object viewed from the camera 100, based on the position and orientation of the camera 100.

[0127] In step S811, the control unit 311 composites the objects in the virtual space displayed on the HMD300 with the captured image based on the rendering results from step S810. Through this process, the control unit 311 generates a composite image.

[0128] In step S812, the control unit 311 transmits the composite image generated in step S811 to the camera 100.

[0129] In step S813, the control unit 311 determines whether or not to terminate the live view display (display of the live view image). If it is determined that the live view display should be terminated, the process in this flowchart ends. If it is determined that the live view display should not be terminated, the process proceeds to step S803.

[0130] (UI display processing) Referring to the flowchart in Figure 9, the UI display process during live view by the PC310 in Embodiment 1 will be explained. Note that instead of the PC310, the HMD300 (HMD control unit 301) may perform all or part of the processing in this flowchart.

[0131] In step S901, the control unit 311, based on the image acquired by the imaging unit 302 (an image of the space captured from the user's viewpoint), determines the HMD 300 (=user's head) and camera 10 The distance between 0 and 100 is calculated. The control unit 311 calculates the position and orientation of the HMD 300 by performing a SLAM calculation based on the image, for example. Then, the control unit 311 calculates the distance between the HMD 300 and the camera 100 based on the position and orientation of the HMD 300 and the position and orientation of the camera 100 calculated in step S805.

[0132] In step S902, the control unit 311 determines whether the distance between the camera 100 and the HMD 300 (=user's head) is greater than or equal to the threshold Th1. If it is determined that the distance between the camera 100 and the HMD 300 is greater than or equal to the threshold Th1, the process proceeds to step S903. If it is determined that the distance between the camera 100 and the HMD 300 is less than the threshold Th1, the process proceeds to step S904.

[0133] In step S903, the control unit 311 determines, based on the image acquired by the imaging unit 302, whether the user's hand is away from the camera 100 (or whether the distance between the user's hand and the camera 100 is less than a specific value). If it is determined that the user's hand is away from the camera 100, the process proceeds to step S905. If it is determined that the user's hand is not away from the camera 100, the process proceeds to step S906.

[0134] In step S904, the control unit 311 determines whether the user's hand is away from the camera 100 based on the image acquired by the imaging unit 302. If it is determined that the user's hand is away from the camera 100, the process proceeds to step S907. If it is determined that the user's hand is not away from the camera 100, the process proceeds to step S908.

[0135] In each of the following steps S905 to S908, a composite image is generated by combining one image with the shooting application 1201 (the image captured by the camera 100), and the composite image is displayed on the image display unit 303.

[0136] In step S905, as shown in Figure 12A, the control unit 311 generates a composite image in which the shooting application 1201 is positioned in a location that does not overlap with the camera 100 in the image captured from the user's viewpoint (the viewpoint of the HMD 300) in the MR space. At this time, the control unit 311 also places (displays) the operation UI 1202 for controlling the settings (shooting settings) of the camera 100 in the composite image. Then, the control unit 311 displays the composite image on the image display unit 303.

[0137] In step S906, the control unit 311 generates a composite image in which the shooting application 1201 is positioned in a location that does not overlap with the camera 100 in the image captured from the user's viewpoint of the MR space, as shown in Figure 12B. At this time, the control unit 311 does not place the operation UI 1202 in the composite image (it is hidden). Then, the control unit 311 displays the composite image on the image display unit 303.

[0138] In step S907, the control unit 311 generates a composite image in which the shooting application 1201 is placed in the center of the background image (composite image) (center of the HMD 300's field of view), as shown in Figure 13A, and all UI elements other than the shooting application 1201 are hidden. The control unit 311 also places (displays) the operation UI 1202 of the shooting application in the composite image. Then, the control unit 311 displays the composite image on the image display unit 303. The background image may be an image of the MR space captured from the user's perspective, or it may be a black image. Instead of making the background image black, the control unit 311 may use an image acquired by the imaging unit 302 that has undergone at least defocusing, monochrome, and brightness reduction processing. In this case, the control unit 311 does not need to place the shooting application 1201 in the exact center of the background image (composite image), but rather in a position closer to the center on the image display unit 303 (composite image) compared to steps S905 and S906. In section 311, the shooting application 1201 may be displayed (positioned) larger than in the cases of steps S905 and S906, so that the user can easily see the shooting application 1201.

[0139] In step S908, the control unit 311 generates a composite image in which the shooting application 1201 is placed in the center of the background image (composite image) (center of the HMD 300's field of view), as shown in Figure 13B, and all UI elements other than the shooting application 1201 are hidden. The control unit 311 also does not place (hide) the operation UI 1202 of the shooting application in the composite image. At this time, the control unit 311 does not need to place the shooting application 1201 in the exact center of the background image (composite image), but rather can place it in a position closer to the center of the image display unit 303 (composite image) compared to steps S905 and S906. The control unit 311 may also display (place) the shooting application 1201 larger than in steps S905 and S906. The background image may be an image of the MR space captured from the user's perspective, or it may be a black image.

[0140] In step S909, the control unit 311 determines whether or not to terminate the live view display (LV display). If it is determined that the live view display should be terminated, the process in this flowchart ends. If it is determined that the live view display should not be terminated, the process proceeds to step S901.

[0141] In this way, the position where the shooting application 1201 is displayed is switched depending on the distance between the camera 100 and the HMD 300. Specifically, when the distance between the camera 100 and the HMD 300 is short, the camera 100 occupies a large proportion of the image captured from the user's viewpoint of the MR space, and the need to view that image decreases. Therefore, in such cases, it can be assumed that the user will want to refer to the shooting application 1201, so the shooting application 1201 is placed in the center of the composite image. On the other hand, when the distance between the camera 100 and the HMD 300 is long, the camera 100 occupies a small proportion of the image captured from the user's viewpoint of the MR space, and the user is likely to want to view that image. Therefore, in such cases, the shooting application 1201 is placed in a position that does not overlap with the camera 100, which is an important element in that image.

[0142] For example, when a user holds the imaging device while wearing an HMD and taking a picture, they may adopt a style where they hold the imaging device with both hands and look through the viewfinder. This style has the advantages of "being able to take pictures at almost the same position as one's own line of sight" and "being able to hold the camera firmly with one's elbows tucked in for stable shooting." For this reason, it is considered a useful shooting style even when wearing the HMD goggles. However, when adopting this style, the imaging device occupies most of the image displayed on the HMD, making it difficult for the user to concentrate on shooting while looking at that image. To address this problem, according to this embodiment, when the user performs the shooting process while checking the live view image acquired by the camera 100 using the HMD 300, it becomes possible to generate an image that allows for greater concentration on shooting.

[0143] <Embodiment 2> Embodiment 2 will be described below. In Embodiment 2, the distance between the camera 100 and the HMD 300 is estimated using the value from the eyepiece sensor of the camera 100.

[0144] (Live view processing for Camera 100) The processing of camera 100 in Embodiment 2 will be explained with reference to the flowchart in Figure 10.

[0145] Steps S701 to S703 are the same as in Embodiment 1, so their explanation will be omitted.

[0146] In step S1003, the system control unit 50 acquires sensor values ​​from the eyepiece detection unit 118 (eyepiece sensor) and transmits the sensor values ​​to the HMD300. The sensor values ​​become smaller as the distance between the camera 100 and the HMD300 increases.

[0147] Steps S704 to S708 are the same as in Embodiment 1, so their explanation will be omitted.

[0148] (UI display processing) The process during live view in Embodiment 2 will be explained with reference to the flowchart in Figure 11.

[0149] In step S1101, the control unit 311 acquires the sensor value obtained by the eyepiece detection unit 118 from the camera 100.

[0150] In step S1102, the control unit 311 determines whether the sensor value acquired in step S1102 is less than or equal to the threshold Th2. If it is determined that the sensor value is less than or equal to the threshold Th2 (less than or equal to a specific threshold), it is determined that the distance between the camera 100 and the HMD 300 is greater than or equal to the threshold TH1, and the process proceeds to step S903. If it is determined that the sensor value is not less than or equal to the threshold Th2, it is determined that the distance between the camera 100 and the HMD 300 is greater than the threshold TH1, and the process proceeds to step S904.

[0151] Steps S903 to S909 are the same as in Embodiment 1, so their explanation will be omitted.

[0152] In Embodiment 2, the control unit 311 obtains sensor values ​​acquired by the eyepiece detection unit 118 via communication and acquires distance information between the camera 100 and the HMD 300 (=user's head) according to those sensor values. However, the control unit 311 may acquire distance information between the camera 100 and the HMD 300 by other means. For example, the control unit 311 may acquire distance information between the camera 100 and the HMD 300 based on the sensor values ​​of the distance sensor provided in the HMD 300. For example, the control unit 311 may acquire images captured by the camera 100 of the HMD 300 via communication and acquire distance information between the camera 100 and the HMD 300 based on the acquired images. Alternatively, if the camera 100 and the HMD 300 are communicating wirelessly with each other, the control unit 311 may acquire (estimate) distance information between the camera 100 and the HMD 300 based on the communication strength of that wireless communication. Furthermore, the control unit 311 may obtain information about the distance between the camera 100 and the HMD 300 via communication from the camera 100.

[0153] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" may be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, "greater than or equal to A" may be rephrased as "greater than (higher; longer; more) than A," and "less than or equal to A" may be rephrased as "less than (lower; shorter; fewer) than A." And "greater than (higher; longer; more) than A" may be rephrased as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be rephrased as "less than or equal to A."

[0154] Note that the various controls described above are performed by a single piece of hardware (e.g., a processor or circuit). This is acceptable, or it is not. Multiple pieces of hardware (for example, multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0155] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0156] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0157] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0158] The above-disclosed embodiments include the following configurations, methods, and programs. (Composition 1) An information processing device that is communicatively connected to an imaging device. A first acquisition means for acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A generation means for generating a composite image obtained by combining the second image and the third image, Having In the first case where the distance between the user's head and the imaging device is greater than the threshold, the third image is the first image. In the second case, where the distance between the user's head and the imaging device is less than or equal to the threshold, the generation means places the second image closer to the center of the composite image than in the first case. An information processing device characterized by the following: (Configuration 2) If the generation means determines that the user's hands have left the imaging device, it places an operation UI (user interface) for controlling the settings of the imaging device onto the composite image. The information processing apparatus according to configuration 1, characterized in that... (Composition 3) The system includes a determination means for determining whether the user's hand has left the imaging device based on the first image. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) The second image includes a real-time image acquired from the imaging device. An information processing device according to any one of configurations 1 to 3, characterized in that... (Composition 5) The imaging device is controlled based on the user's instructions. An information processing device according to any one of configurations 1 to 4, characterized in that... (Composition 6) The system further includes an acquisition means for acquiring information on the distance between the user's head and the imaging device. An information processing device according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The acquisition means acquires information about the distance between the user's head and the imaging device based on the first image. The information processing apparatus according to configuration 6, characterized in that... (Composition 8) The acquisition means acquires information about the distance between the user's head and the imaging device based on the value of the eyepiece sensor of the imaging device. The information processing apparatus according to configuration 6, characterized in that... (Composition 9) The acquisition means acquires information about the distance between the user's head and the imaging device through communication from the imaging device. The information processing apparatus according to configuration 6, characterized in that... (Composition 10) In the second case, the third image is a black image. An information processing apparatus according to any one of configurations 1 to 9, characterized in that... (Composition 11) In the second case, the generating means generates the composite image by using as the third image an image obtained by applying at least one of the following processes to the first image: defocusing, monochrome processing, and brightness reduction processing. An information processing apparatus according to any one of configurations 1 to 9, characterized in that... (Composition 12) In the first case, the generating means places the second image at a position in the first image that does not overlap with the imaging device. An information processing device according to any one of configurations 1 to 11, characterized by the above. (method) A control method for an information processing device that is communicatively connected to an imaging device. A first acquisition step involves acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition step involves acquiring a second image of the space captured by the imaging device, A generation step of generating a composite image by combining the second image and the third image, Having In the first case where the distance between the user's head and the imaging device is greater than the threshold, the third image is the first image. In the generation step, in the second case where the distance between the user's head and the imaging device is less than or equal to the threshold, the second image is positioned closer to the center of the composite image than in the first case. A control method for an information processing device characterized by the following features. (program) A program for causing a computer to function as one of the means of an information processing device described in any of configurations 1 to 12. [Explanation of symbols]

[0159] 100: Camera (imaging device), 300: HMD, 310: PC (Information Processing Unit), 302: Imaging Unit, 311: Control Unit, 314: Communication Unit

Claims

1. An information processing device that is communicatively connected to an imaging device. A first acquisition means for acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A generation means for generating a composite image obtained by combining the second image and the third image, Having In the first case where the distance between the user's head and the imaging device is greater than the threshold, the third image is the first image. In the second case, where the distance between the user's head and the imaging device is less than or equal to the threshold, the generation means places the second image closer to the center of the composite image than in the first case. An information processing device characterized by the following:

2. If the generation means determines that the user's hands have left the imaging device, it places an operation UI (user interface) for controlling the settings of the imaging device onto the composite image. The information processing apparatus according to claim 1, characterized in that...

3. The device has a determination means for determining whether the user's hand has left the imaging device based on the first image. The information processing apparatus according to claim 2, characterized in that...

4. The second image includes a real-time image acquired from the imaging device. The information processing apparatus according to claim 1, characterized in that...

5. The imaging device is controlled based on the user's instructions. The information processing apparatus according to claim 1, characterized in that...

6. The system further includes an acquisition means for acquiring information on the distance between the user's head and the imaging device. The information processing apparatus according to feature 1.

7. The acquisition means acquires information about the distance between the user's head and the imaging device based on the first image. The information processing apparatus according to claim 6, characterized in that...

8. The acquisition means acquires information about the distance between the user's head and the imaging device based on the value of the eyepiece sensor of the imaging device. The information processing apparatus according to claim 6, characterized in that...

9. The acquisition means acquires information about the distance between the user's head and the imaging device through communication from the imaging device. The information processing apparatus according to claim 6, characterized in that...

10. In the second case, the third image is a black image. The information processing apparatus according to claim 1, characterized in that...

11. In the second case, the generating means uses an image obtained by applying at least one of the following processes to the first image as the third image: defocusing, monochrome processing, and brightness reduction processing. , generate the composite image, The information processing apparatus according to claim 1, characterized in that...

12. In the first case, the generating means places the second image at a position in the first image that does not overlap with the imaging device. The information processing apparatus according to feature 1.

13. A control method for an information processing device that is communicatively connected to an imaging device. A first acquisition step involves acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition step involves acquiring a second image of the space captured by the imaging device, A generation step of generating a composite image by combining the second image and the third image, Having In the first case where the distance between the user's head and the imaging device is greater than the threshold, the third image is the first image. In the generation step, in the second case where the distance between the user's head and the imaging device is less than or equal to the threshold, the second image is positioned closer to the center of the composite image than in the first case. A control method for an information processing device characterized by the following features.

14. A program for causing a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 12.

Images