Information processing device, control method for information processing device, program

Abstract

These images allow users to see what kinds of photos they can take, and enable the generation of high-quality images that include both real space and virtual objects with less overhead. [Solution] An information processing device that is communicatively connected to an imaging device includes a first acquisition means for acquiring a first image of a space captured according to the user's viewpoint, a second acquisition means for acquiring a second image of the space captured by the imaging device, and a control means for generating a composite image by combining a virtual object, the first image, and the second image. The control means combines the virtual object into either the region of the first image or the region of the second image in the composite image, depending on whether the user is viewing the region of the second image in the composite image.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] As a technology that enables experiencing a virtual space, VR (Virtual Reality) technology is known. Also, as a technology that seamlessly fuses the real space and the virtual space in real time, so-called MR (Mixed Reality) technology (technology of composite reality) is known. For a device that can experience such technology, for example, a head-mounted device typified by an HMD (Head Mounted Display) is used.

[0003] In order to photograph a space (MR space) including the real space and virtual objects at an arbitrary viewing angle, a screen shot function of an HMD equipped with MR technology is used. In this case, in order to capture a high-quality MR image, it is conceivable that the user captures the MR space while using an external imaging device with high imaging performance and checking the shooting angle of view and preview image from the viewpoint of the external imaging device. At this time, for example, the HMD performs self-position and orientation estimation processing and 3D object drawing processing based on the captured image received from the external imaging device, and synthesizes the virtual object image and the captured image generated by these processes to generate a preview image.

[0004] In Patent Document 1, a first imaging unit (such as a global shutter sensor represented by a CCD) is analyzed to estimate the position and orientation of the MR device. Then, a CG object is drawn on the image generated using a second imaging unit (such as a rolling shutter sensor represented by a CMOS) based on the estimated position and orientation of the MR device. This reduces processing costs. In Patent Document 2, one of several types of devices accesses an environment map, estimates the position and orientation of the device in relation to that environment map, and renders virtual content at a specified position. This also reduces processing costs. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2017-55397 [Patent Document 2] Special Publication No. 2022-551734 [Overview of the project] [Problems that the invention aims to solve]

[0006] In this context, virtual objects are sometimes placed on images of the real world captured from the viewpoints of both the HMD and the external imaging device, allowing users to view virtual objects from various perspectives. However, in HMDs, where comfortable wearability and portability are essential, it is difficult to secure the processing power necessary to handle self-position and orientation estimation and 3D object rendering for each viewpoint of both the HMD and the external imaging device. As a result, it has been difficult for users to understand what kind of imaging is possible by viewing the MR image (an image containing both the real world and virtual objects) displayed on the HMD.

[0007] Therefore, the present invention enables the generation of high-quality images, including real space and virtual objects, with lower processing load, for users to confirm what kind of photography is possible. The purpose is to provide technology. [Means for solving the problem]

[0008] One aspect of the present invention is, An information processing device that is communicatively connected to an imaging device, A first acquisition means for acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A control means for generating a composite image by combining a virtual object, the first image, and the second image, It has, The control means is In the first case, where it is determined that the user is not viewing the region of the second image in the composite image, the virtual object is composited into the region of the first image in the composite image, and the virtual object is not composited into the region of the second image in the composite image. In the second case, when it is determined that the user is viewing the region of the second image in the composite image, the virtual object is composited into the region of the second image in the composite image, and the virtual object is not composited into the region of the first image in the composite image. This is an information processing device characterized by the following features. [Effects of the Invention]

[0009] According to the present invention, high-quality images including real space and virtual objects, which are used by users to confirm what kind of photography is possible, can be generated with less overhead. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing the system configuration according to Embodiment 1. [Figure 2] It is an external view of the camera according to Embodiment 1. [Figure 3] It is an internal configuration diagram of the camera according to Embodiment 1. [Figure 4] It is an internal configuration diagram of an HMD or the like according to Embodiment 1. [Figure 5] It is a diagram for explaining the MR space according to Embodiment 1. [Figure 6] It is a diagram showing a display example of the HMD according to Embodiment 1. [Figure 7] It is a flowchart of the processing of the camera according to Embodiment 1. [Figure 8] It is a flowchart of the processing of the PC according to Embodiment 1.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] <Embodiment 1> (Configuration of the Whole System) Referring to FIG. 1, an example of the configuration of the whole system according to Embodiment 1 will be described. The information processing system 1 includes a camera 100, an HMD 300, a PC (personal computer) 310, and a controller 320.

[0013] The camera 100 is connected to the PC 310 in a communicable state, either wired or wirelessly. The camera 100 transmits and receives various data (such as live view image data and captured image data). Perform this. Note that, for example, instead of the camera 100, an imaging device (such as a smartphone or a tablet terminal) that can realize the functions described below may be used. Note that the camera 100 is not limited to the PC 310 and may communicate with the HMD 300.

[0014] The HMD 300 is a display device (head-mounted electronic device) that can be worn on the user's head. The HMD 300 displays a composite image in which "the captured image obtained by the HMD 300 capturing the range in front of the user" and "contents such as CG in a form corresponding to the position and orientation of the HMD 300" are combined.

[0015] The PC 310 is an information processing device that controls the HMD 300. The PC 310 is connected to the HMD 300 by wire such as a USB cable or wirelessly such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). For example, the PC 310 generates a composite image by combining the captured image and the CG, and transmits the composite image to the HMD 300. In this case, when the PC 310 receives the live view image or the captured image from the camera 100, the PC 310 generates a composite image in which the received image and the CG in a form corresponding to the position and orientation of the camera 100 are combined. The PC 310 transmits the composite image to the HMD 300.

[0016] Note that a smartphone or a tablet terminal may be used instead of the PC 310. Also, each component of the PC 310 may be possessed by the HMD 300. Note that in Embodiment 1, an example in which the PC 310 and the camera 100 are connected wirelessly is shown, but the PC 310 and the camera 100 may be connected by wire.

[0017] The controller 320 performs various controls on the HMD 300. When the PC 310 is in a specific control mode, the HMD 300 is controlled according to the user's operation when the user operates the controller 320. As shown in Figure 1, the controller 320 is an operating component that is "ring-shaped and can be worn and supported on the user's finger" or "handheld and can be held in the hand". The controller 320 also has physical buttons for making decisions and selections on the display.

[0018] Controller 320 communicates with PC 310 wirelessly via Bluetooth. Controller 320 may also communicate with HMD 300, not just PC 310. The user can change the indicated position on the display by moving Controller 320. The indicated position may be represented as a point, or as a virtual ray connecting the point and the controller with a straight line (line segment) or dotted line. The user can perform menu selection and confirmation operations by pressing physical buttons.

[0019] The controller 320 is ring-shaped or handheld. However, the controller 320 may be any shape as long as it can be supported by a finger, hand, or arm. The buttons on the controller 320 are described as physical buttons, but they may be operable in the form of a trackpad, touch panel, wheel, or trackball. In addition to button presses, the controller 320 may also accept slide, flick, and touch operations. The controller 320 may be wearable on at least one of the fingers, hand, or arm. The controller 320 may also be attached to an object held in the hand, and positional and orientation information of the attached position may be obtained from sensors. Examples of such objects include objects that mimic tools.

[0020] (External structure of a digital camera) Figures 2A and 2B show an example of the external configuration of the camera 100, which is an imaging device. Figure 2A is a perspective view of camera 100 from the front. Figure 2B is a perspective view of camera 100 from the rear.

[0021] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface.

[0022] The shutter button 101 is an operation unit for preparing to take a picture or giving a shooting command. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode selector switch 103 is an operation unit for switching between various modes.

[0023] The main electronic dial 104 is a rotary control unit for changing settings such as shutter speed and aperture. The sub-electronic dial 105 is a rotary control unit for moving the selection frame (cursor) and advancing images.

[0024] The video button 106 is an operation unit for instructing the start and stop of video recording. The viewfinder external display unit 107 displays various settings such as shutter speed and aperture.

[0025] The camera 100 also includes a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back.

[0026] The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108.

[0027] The directional keys 110 are an operation unit consisting of keys that can be pressed in the up, down, left, and right directions (4-way keys). The camera 100 can be controlled according to the position where the directional keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed when confirming a selection item.

[0028] The AE lock button 112 is an operation button that is pressed to fix the exposure state when the camera is in shooting standby mode.

[0029] The zoom button 113 is an operation unit used to switch the zoom mode on and off in the live view display (LV display) of the shooting mode. When the zoom mode is on, the live view image (LV image) will be enlarged or reduced when the main electronic dial 104 is operated. The zoom button 113 is also used in playback mode to enlarge the playback image or increase the magnification ratio.

[0030] The playback button 114 is an operation unit for switching between shooting mode and playback mode. When the playback button 114 is pressed in shooting mode, the device switches to playback mode, and the most recent image recorded on the recording medium 227 (described later) is displayed on the display unit 108.

[0031] The menu button 115 is an operation button that is pressed to display a menu screen on the display unit 108 that allows for various settings. The user can intuitively make various settings using the menu screen displayed on the display unit 108, the directional keys 110, and the SET button 111.

[0032] The eyepiece section 116 is the part that allows the user to bring their eye close to (eye-at) the eyepiece viewfinder (a type of viewfinder) 117. The user can view the image displayed on the EVF 217 (Electronic View Finder) through the eyepiece section 116.

[0033] The eyepiece detection unit 118 is a sensor that detects whether or not a user is looking into the eyepiece unit 116.

[0034] The touch bar 119 is a line-shaped touch operation area (line touch sensor) capable of accepting touch operations. The touch bar 119 is positioned so that it can be touched with the right thumb when the grip section 120 is held with the right hand (with the little finger, ring finger, and middle finger of the right hand), allowing the shutter button 101 to be pressed with the right index finger. In other words, the touch bar 119 can be operated when the user is looking through the eyepiece viewfinder 117 with their eyepiece in the eyepiece section 116 and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (touching and releasing without moving within a predetermined period of time), left and right sliding operations (touching and then moving the touch position while keeping the touch on the surface), etc. The touch bar 119 is a different operation area from the touch panel 109 and does not have a display function. The touch bar 119 in this embodiment is a multifunction bar and functions, for example, as an M-Fn bar.

[0035] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, and the like.

[0036] The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. When the camera 100 is held with the grip section 120 held by the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand.

[0037] The thumb rest section 121 (thumb waiting position) is a grip section located on the back of the camera 100, in a position where it is easy to rest the thumb of the right hand holding the grip section 120 when no controls are being operated. The thumb rest section 121 is made of rubber material or the like to enhance the holding power (grip feel).

[0038] The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 227 and the slot for storing the recording medium 227 by closing the slot.

[0039] The communication terminal 124 is a terminal for communicating with the lens unit 200.

[0040] (Internal structure of a digital camera) Figure 3 shows an example of the internal configuration of camera 100. In Figure 3, components identical to those in Figure 2 are denoted by the same reference numerals, and their descriptions are omitted as appropriate. A lens unit 200 is attached to camera 100.

[0041] First, let's describe the lens unit 200. The lens unit 200 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens unit 200 is a single-lens reflex lens and is an example of a normal lens. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0042] The aperture diameter of the aperture 201 is adjustable. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The F drive circuit 204 drives the lens 202 to focus.

[0043] The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc., based on instructions from the system control unit 50. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203. The lens system control circuit 205 focuses by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication takes place via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100.

[0044] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0045] The shutter 210 is a focal-plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50.

[0046] The imaging unit 211 is an image sensor composed of a CCD or CMOS element, which converts an optical image into an electrical signal. The imaging unit 211 may also have an image plane phase difference sensor that outputs defocus amount information to the system control unit 50.

[0047] The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal.

[0048] The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing, and color conversion) on data from the A / D converter 212 or data from the memory control unit 213. The image processing unit 214 also performs predetermined calculations using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. This process enables TTL (through-the-lens) AF processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data and performs TTL AWB (auto white balance) processing based on the obtained calculation results. Image data from the A / D converter 212 is written to memory 215 via the image processing unit 214 and memory control unit 213. Alternatively, image data from the A / D converter 212 is written to memory 215 via memory control unit 213 without going through the image processing unit 214.

[0049] Memory 215 stores "image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212" and "image data for display on the display unit 108 and EVF 217". Memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. Memory 215 also serves as memory for image display (video memory).

[0050] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies the analog signal to the display unit 108 and EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 and EVF 217 via the D / A converter 216. The display unit 108 and EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and EVF 217 are displays such as LCDs or OLEDs. The digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216. The analog signal is then sequentially transmitted to the display unit 108 and the EVF 217, enabling a live view display that shows an image representing the real-time space.

[0051] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements each process of the flowchart described later by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0052] The camera 100 also includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118.

[0053] System memory 218 may include, for example, RAM. System memory 218 stores constants and variables for the operation of the system control unit 50, programs read from non-volatile memory 219, and so on.

[0054] The non-volatile memory 219 is an electrically erasable and recordable memory. For example, an EEPROM is used for the non-volatile memory 219. Constants for the operation of the system control unit 50, programs, etc., are stored in the non-volatile memory 219. The program here refers to a program for executing the flowchart processing described later.

[0055] The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals or audio signals to and from external devices connected by wireless or wired cables.

[0056] The communication unit 221 can connect to both a wireless LAN (Local Area Network) and the internet. Furthermore, the communication unit 221 can communicate with external devices via Bluetooth® and Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227. The communication unit 221 can also receive image data and other various types of information from external devices.

[0057] The attitude detection unit 222 detects the attitude of the camera 100 relative to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image before recording. For example, an acceleration sensor or a gyroscope sensor can be used for the attitude detection unit 222. The attitude detection unit 222 can also be used to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.).

[0058] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece section 116 of the "eyepiece finder 117 with built-in EVF 217". For example, an infrared proximity sensor can be used in the eyepiece detection unit 118. When an object approaches the eyepiece section 116, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece section 116 to the object can be determined by the amount of infrared light received (=sensor value). In this way, the eyepiece detection unit 118 can determine the proximity distance of an object to the eyepiece section 116. Eyepiece detection is performed to detect when an eye (object) approaches (eyepiece) and moves away (eye-away) from the eyepiece section 116 of the eyepiece finder 117. When an object is detected approaching the eyepiece section 116 from a non-eyepiece state (non-approach state), it is detected that an eyepiece has been placed. On the other hand, when the object that was detected approaching moves away from the eyepiece state (approach state) beyond a predetermined distance, it is detected that an eye has been removed. The threshold for detecting eyepiece placement and the threshold for detecting eye-away may be different, for example, by providing hysteresis. After eyepiece placement is detected, the eyepiece is considered to be in an eyepiece state until eye-away is detected. After eye-away is detected, the eyepiece is considered to be in a non-eyepiece state until eyepiece placement is detected again.

[0059] The system control unit 50 switches the display (display state) / hidden (hidden state) on the display unit 108 and EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, at least in the shooting standby state and when the display destination switching setting is set to automatic switching, the system control unit 50 turns on the display on the display unit 108 and hides the EVF 217 when the user is not using an eyepiece. Also, when the user is using an eyepiece, the system control unit 50 turns on the display on the EVF 217 and hides the display unit 108. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as eyepiece use may be used.

[0060] The camera 100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium interface 226, an operation unit 228, and the like.

[0061] The external viewfinder display unit 107 displays various settings of the camera 100 (such as shutter speed and aperture) via the external viewfinder display drive circuit 223.

[0062] The power control unit 224 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and the like. The power control unit 224 detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, supplying the necessary voltage to each part (including the recording medium 227) for the required period.

[0063] The power supply unit 225 is a primary battery (such as an alkaline battery or a lithium battery), a secondary battery (such as a NiCd battery, a NiMH battery or a Li battery), or an AC adapter.

[0064] The recording medium I / F 226 is an interface to the recording medium 227. The recording medium 227 is a memory card or the like for recording captured images. The recording medium 227 is composed of semiconductor memory or a magnetic disk or the like.

[0065] The recording medium 227 may be detachable from the camera 100, or it may be built into the camera 100.

[0066] The operation unit 228 is an input unit that receives user input (user operation). The operation unit 228 is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode selector switch 103, a touch panel 109, and other operation units 229.

[0067] Other control units 229 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a play button 114, a menu button 115, a touch bar 119, and the like.

[0068] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231.

[0069] The first shutter switch 230 turns on during the operation of the shutter button 101, specifically when it is half-pressed (indicating preparation for shooting), and generates the first shutter switch signal SW1. Upon generation of the first shutter switch signal SW1, the system control unit 50 starts the shooting preparation process (AF processing, AE processing, AWB processing, EF processing, etc.).

[0070] The second shutter switch 231 turns on when the shutter button 101 is fully pressed (a shooting instruction), generating the second shutter switch signal SW2. Upon generation of the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processes (from reading the signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227).

[0071] The mode switch 103 switches the operating mode of the system control unit 50 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. Modes included in the still image shooting mode include auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), program AE mode (P mode), etc. Modes included in the still image shooting mode include various scene modes and custom modes, which are shooting settings for different shooting scenes. The user can switch directly to any of the above shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0072] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is mounted on top of the display surface of the display unit 108 so that the light transmittance of the touch panel 109 does not interfere with the display of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109. Either method is acceptable.

[0073] The system control unit 50 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). - The finger or pen that was touching the touch panel 109 has left (released) the touch panel 109, that is, the end of the touch (hereinafter referred to as Touch-Up and say). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0074] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0075] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 50 via the internal bus. Based on the notified information, the system control unit 50 determines what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation was performed. An operation in which a finger is touched on the touch panel 109 and then quickly moved a certain distance and then released is called a flick. In other words, a flick is an operation in which the finger is quickly traced across the touch panel 109 as if flicking it. If a touch move of a predetermined distance or more at a predetermined speed or faster is detected, and a touch-up is then detected, it is determined that a flick was performed (it can be determined that a flick followed a slide operation). Furthermore, touching multiple points (for example, two points) simultaneously (multitouch) to bring them closer together is called pinch-in, and touching them further apart is called pinch-out. Pinch-out and pinch-in are collectively referred to as pinch operations (or simply pinch).

[0076] (HMD configuration) Referring to Figure 4, an example of the configuration of the HMD300 will be described. The HMD300 includes an HMD control unit 301, an imaging unit 302, an image display unit 303, an attitude sensor unit 304, a non-volatile memory 305, a working memory 306, and a gaze imaging unit 307.

[0077] The HMD control unit 301 is a CPU that controls each component of the HMD 300. When the HMD control unit 301 acquires a composite image (an image created by combining an image captured by the imaging unit 302 of the space in front of the user with computer graphics) from the PC 310, it displays the composite image on the image display unit 303. Alternatively, instead of the HMD control unit 301 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0078] The imaging unit 302 includes two cameras (imaging devices). The two cameras are for capturing images used for synthesis with images in a virtual space and for generating positional orientation information, and have an imaging unit for the left eye and an imaging unit for the right eye. The left eye imaging unit captures a moving image of the real space corresponding to the left eye of the HMD300 wearer, and outputs an image (captured image) of each frame in the moving image from the left eye imaging unit. The right eye imaging unit captures a moving image of the real space corresponding to the right eye of the HMD300 wearer, and outputs an image (captured image) of each frame in the moving image from the right eye imaging unit. In other words, the imaging unit 302 acquires captured images as stereo images having parallax that substantially coincides with the positions of the left and right eyes of the HMD300 wearer. In addition, distance information from the two cameras to the subject can be acquired as distance information by measuring distance with the stereo cameras. In the case of an HMD for an MR system, it is preferable that the central optical axis of the imaging range of the imaging unit is arranged to substantially coincide with the line of sight of the HMD wearer.

[0079] Each of the left and right eye imaging units includes an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incident light as an image. The images captured by the two cameras of the subject (the area directly in front of the user) are output to the PC 310 and the HMD control unit 301. The imaging unit 302 may output video instead of an image.

[0080] The image display unit 303 displays a composite image. The image display unit 303 has a liquid crystal panel or an organic EL panel, etc. When the user is wearing the HMD 300, the image display unit 303 is positioned in front of each of the user's eyes. It is also possible to use a device with a semi-transparent half-mirror for the image display unit 303. In this case, for example, the image display unit 303 may display an image so that the CG is superimposed directly onto the real space visible through the half-mirror, using a technology generally known as AR (Augmented Reality). Alternatively, the image display unit 303 may display an image of a completely virtual space without using captured images, using a technology generally known as VR (Virtual Reality).

[0081] The attitude sensor unit 304 acquires attitude (and position) information of the HMD 300. The attitude sensor unit 304 may also acquire user (the user wearing the HMD 300) attitude information that corresponds to the attitude (and position) of the HMD 300. For example, the attitude sensor unit 304 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 304 is used to acquire user attitude information, and the HMD control unit 301 outputs the user's attitude information to the PC 310. The attitude information may also be acquired from one or more of the following: a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, or an angular velocity sensor.

[0082] The HMD control unit 301 estimates the position or orientation of each joint point of the user's hand and fingers based on images obtained from the two cameras of the imaging unit 302. The joint points include characteristic points of various parts of the hand, such as the finger joints, fingertips, back of the hand (palm), and arm. Each joint point represents a coordinate position. Based on information from multiple joint points, the hand's orientation can be estimated. Methods for estimating the position or orientation of the hand and each joint point can include, for example, known machine learning methods for object recognition and pose estimation using convolutional neural networks. Furthermore, the depth-direction position information of each joint point of the hand can be obtained, for example, by calculating the distance from the imaging unit 302 to each joint point through triangulation using stereo matching with images obtained from the two cameras of the imaging unit 302. The estimated coordinate information of each joint point of the hand is output from the HMD control unit 301 to the PC 310.

[0083] The non-volatile memory 305 is an electrically erasable and recordable non-volatile memory that stores programs, such as those executed by the control unit 311 (described later).

[0084] The working memory 306 is used as a buffer memory for temporarily holding image data captured by the imaging unit 302, as well as as an image display memory for the image display unit 303 and as a working area for the HMD control unit 301.

[0085] The gaze-tracking unit 307 is a camera that acquires images to detect the user's gaze. The gaze-tracking unit 307 is mounted inside the HMD 300 to capture images of the user's eyes when the user wears the HMD 300. The image captured by the camera of the subject (user's eyes) is output to the control unit 311 of the PC 310 via the HMD control unit 301. The control unit 311 detects the gaze of the user wearing the HMD 300 from the image captured by the gaze-tracking unit 307 and identifies the area the user is fixated on in the image display unit 303. ru.

[0086] (Internal configuration of the PC) Referring to Figure 4, the internal configuration of PC310 will be described. PC310 includes a control unit 311, a non-volatile memory 312, a working memory 313, a communication unit 314, and a recording medium 315.

[0087] The control unit 311 is a CPU that controls various parts of the PC310 according to the input signals and the program described later. Alternatively, instead of the control unit 311 controlling the entire PC310, multiple hardware components may share the processing to control the entire PC310. The control unit 311 receives the image acquired by the imaging unit 302 (captured image) and the attitude information acquired by the attitude sensor unit 304 from the HMD300. The control unit 311 performs image processing on the captured image to cancel out aberrations in the optical system of the imaging unit 302 and the optical system of the image display unit 303. Then, the control unit 311 combines the captured image with an arbitrary CG to generate a composite image. The control unit 311 transmits the composite image to the HMD control unit 301 in the HMD300.

[0088] Furthermore, the control unit 311 determines the number of controllers 320 included in the captured image. The control unit 311 also uses the information obtained via the communication unit 314 to perform processing to recognize the mounting position of each controller 320. Then, based on the recognized result, the control unit 311 controls each controller to change the operation content for the input information of each controller 320.

[0089] The control unit 311 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 300. For example, when the control unit 311 places a virtual object represented by CG near a specific object that exists in real space within the space represented by the composite image, it increases the size of the virtual object (CG) as the distance between the specific object and the imaging unit 302 decreases. By controlling the position, orientation, and size of the CG in this way, the control unit 311 can generate a composite image in which CG objects that are not actually located in real space appear as if they were.

[0090] Furthermore, the control unit 311 receives information estimated by the HMD control unit 301 of the HMD 300. The received information is temporarily stored in the working memory 313.

[0091] Furthermore, the control unit 311 receives information on the position or orientation change of the controller 320 from the communication unit 323 of the controller 320. The control unit 311 superimposes display items indicating the indicated positions corresponding to the position or orientation change information of the controller 320 onto the synthesized image. The control unit 311 may also superimpose display items indicating the indicated positions corresponding to the position and orientation change information of the controller 320 onto the synthesized image.

[0092] The non-volatile memory 312 is an electrically erasable and recordable non-volatile memory. The non-volatile memory 312 stores information such as the program executed by the control unit 311 (described later) and CG. The control unit 311 can switch the CG read from the non-volatile memory 312 (i.e., the CG used to generate the composite image).

[0093] The working memory 313 is used as a buffer memory to temporarily hold image data captured by the imaging unit 302 and time-series information of the estimated coordinate positions of each joint point of the hand. The working memory 313 is also used as the image display memory for the image display unit 303 and as a work area for the control unit 311.

[0094] Furthermore, wrist joint estimation may be performed by PC310. In this case, after the captured image is output from the imaging unit 302 to PC310, the control unit 311 of PC310 estimates each joint point of the hand. The control unit 311 estimates the position or orientation of the hand. Then, it uses this information to process the image and outputs it to the HMD 300. Alternatively, the control unit 311 may estimate the position and orientation of each joint point of the hand, use this information to process the image, and output it to the HMD 300.

[0095] (Internal configuration of the controller) Referring to Figure 4, the internal configuration of the controller 320 will be explained. The controller 320 includes a controller control unit 321, an operation unit 322, a communication unit 323, a controller attitude sensor unit 324, and an output unit 325.

[0096] The controller control unit 321 is a CPU that controls each component of the controller 320. Alternatively, instead of the controller control unit 321 controlling the entire controller 320, multiple hardware components may share the processing to control the entire controller 320.

[0097] The operation unit 322 includes a button. The operation unit 322 detects whether or not a button has been pressed and transmits the detection information to the PC 310 via the communication unit 323. The operation unit 322 may have multiple types of input formats.

[0098] The communication unit 323 communicates wirelessly with the PC 310 via Bluetooth. If multiple controllers 320 are connected to the PC 310, each controller 320's communication unit 323 communicates wirelessly with the PC 310 via Bluetooth.

[0099] The controller attitude sensor unit 324 has an inertial measurement unit (IMU) consisting of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 320. The detected position and attitude change information is communicated from the communication unit 323 to the PC 310 via the controller control unit 321.

[0100] The output unit 325 consists of an LED light source, a speaker, and a vibration element, among other things.

[0101] (An example of a mixed reality (MR) space) Referring to Figure 5, an example of an MR space experienced by a user wearing the HMD300 in Embodiment 1 will be described. The MR space 500 contains a user 501, the HMD300 worn by user 501, a PC310 that communicates with the HMD300, and a camera 100 that communicates with the PC310. The MR space 500 also contains real objects 502, virtual objects 503, and a virtual window 510.

[0102] The virtual window 510 is an example of the UI of a shooting application. The virtual window 510 displays the live view image 511, a virtual object 512, and an operating element 513. The live view image 511 is an image acquired by the imaging unit 211 of the camera 100. The virtual object 512 is a virtual object whose form corresponds to the position and orientation of the camera 100.

[0103] Additionally, arrow 504 indicates the direction in which the imaging unit 302 of the HMD 300 worn by user 501 is capturing images. Arrow 505 indicates the direction in which the imaging unit 211 of camera 100 is capturing images.

[0104] (Example of display on HMD300) Referring to Figures 6A to 6C, an example of the display in the image display unit 303 of the HMD300 according to Embodiment 1 will be described.

[0105] The screen 600 shown in Figure 6A shows an example of the display on the image display unit 303 of the HMD 300 when the imaging unit 302 of the HMD 300 worn by the user 501 captures the MR space in the direction of the arrow 504 in Figure 5. The screen 600 displays a real object 502, a virtual object 503, and a virtual window 510.

[0106] In addition, the virtual window 510 displays the live view image 511, the virtual object 512, and the operating member 513. The live view image 511 is an image acquired by the imaging unit 211 of the camera 100. The virtual object 512 is a virtual object whose shape corresponds to the position and orientation of the camera 100.

[0107] The position and orientation of the virtual object 503 and the virtual window 510 are adjusted to correspond to the position and orientation (the position and orientation corresponding to the user 501's viewpoint) when the imaging unit 302 of the HMD 300 worn by user 501 takes an image in the direction of arrow 504 in Figure 5. The position and orientation of the virtual object 512 displayed in the virtual window 510 are adjusted to correspond to the position and orientation (the position and orientation corresponding to the camera 100's viewpoint) when the imaging unit 211 of the camera 100 takes an image in the direction of arrow 505 in Figure 5.

[0108] In Embodiment 1, the description assumes that the user 501 in Figure 5 can perform the operation of sending a shooting command to the camera 100 using the operating member 513, but it is not limited to this. For example, multiple operating members may be arranged in the virtual window 510, and each operating member can be assigned operations such as sending a shooting command as well as commands to set various shooting conditions. Also, in Embodiment 1, the description assumes that the virtual window 510 is arranged as if it virtually exists at any three-dimensional position and in any three-dimensional orientation in space, as shown in the MR space 500 in Figure 5, but it is not limited to this. The virtual window 510 may be arranged at any two-dimensional position in the display area of ​​the screen 600.

[0109] The screen 600 shown in Figure 6B shows an example of the display on the image display unit 303 of the HMD 300 when the user 501 shown in Figure 5 is not paying attention to the virtual window 510. A virtual object 503 with a shape based on the position and orientation of the HMD 300 is displayed, but a virtual object 512 with a shape based on the position and orientation of the camera 100 (the position and orientation of the user's head) is not displayed.

[0110] The screen 600 shown in Figure 6C shows an example of the display on the image display unit 303 of the HMD 300 when user 501 in Figure 5 is looking at the virtual window 510. The virtual object 503, whose shape is based on the position and orientation of the HMD 300, is not displayed, but the virtual object 512, whose shape is based on the position and orientation of the camera 100, is displayed.

[0111] (Processing related to live view images in camera 100) Referring to the flowchart in Figure 7, the processing using the live view image from camera 100 according to Embodiment 1 will be explained.

[0112] In step S701, the system control unit 50 controls the communication unit 221 to connect with the PC 310 so that communication is possible. The connection with the PC 310 can be made using any connection method. The connection with the PC 310 may be achieved by either wireless communication or wired communication.

[0113] In step S702, the system control unit 50 determines whether or not the PC 310 has requested the acquisition of a live view image (LV image). If it determines that the acquisition of a live view image has been requested, the system proceeds to step S703. If it determines that the acquisition of a live view image has not been requested, the system proceeds to step S704.

[0114] In step S703, the system control unit 50 sends the live view image (live view information) to the PC 310 in response to the request to acquire the live view image. In addition to the live view image, the system control unit 50 also sends lens optical information (optical characteristic information including lens aberration information).

[0115] In step S704, the system control unit 50 determines whether or not a photo shoot has been requested (either a photo shoot requested from the PC 310 or a photo shoot requested by the user operating the camera 100). If it is determined that a photo shoot has been requested, the process proceeds to step S705. If it is determined that a photo shoot has not been requested, the process proceeds to step S708.

[0116] In step S705, the system control unit 50 performs the shooting process. The system control unit 50 transmits the image acquired by shooting (shot image) to the PC 310.

[0117] In step S706, the system control unit 50 receives the composite image from the PC 310.

[0118] In step S707, the system control unit 50 saves the composite image received in step 706 to the recording medium 227.

[0119] In step S708, the system control unit 50 determines whether or not communication with PC310 has been disconnected. If it is determined that communication with PC310 has been disconnected, the process in this flowchart ends. If it is determined that communication with PC310 has not been disconnected, the process proceeds to step S702.

[0120] (Example of processing flow in PC310) Referring to the flowchart in Figure 8, the processing at PC310 in Embodiment 1 will be described. Note that instead of PC310, HMD300 (HMD control unit 301) may perform all or part of the processing shown in this flowchart.

[0121] In step S801, the control unit 311 controls the communication unit 314 to connect the camera 100 and the PC 310 so that communication is possible. The type of connection method to the camera 100 is not limited. The connection to the camera 100 may be achieved by either wireless communication or wired communication.

[0122] In step S802, the control unit 311 launches an application to enable live view display. The window of the launched application is displayed on the image display unit 303, as shown in the virtual window 510 in screen 600 shown in Figure 6A.

[0123] In step S803, the control unit 311 requests the camera 100 to acquire a live view image.

[0124] In step S804, the control unit 311 receives the live view image and lens optical information from the camera 100.

[0125] In step S805, the control unit 311 determines whether user 501 in Figure 5 is paying attention to (looking at) the virtual window 510 (=the area where the live view image is displayed). If it is determined that user 501 is paying attention to the virtual window 510, the process proceeds to step S806. If it is determined that user 501 is not paying attention to the virtual window 510, the process proceeds to step S810.

[0126] For example, the control unit 311 places a virtual window 51 on the extension of the user 501's line of sight. If the control unit 311 determines that a certain area including a region of 0 is located, it determines that user 501 is paying attention to the virtual window 510 (looking at the virtual window 510). The control unit 311 may also determine that user 501 is paying attention to the virtual window 510 if it determines that the ratio (occupancy rate) of the display area of ​​the virtual window 510 on the screen 600 (the entire LV composite image described later) exceeds a certain threshold. In addition, the control unit 311 may determine that user 501 is paying attention to the virtual window 510 if it determines that user 501 is operating an operating member 513 (an operating member corresponding to the virtual window 510) placed in the virtual window 510.

[0127] In step S806, the control unit 311 controls the system to stop the "estimation process of the position and orientation of the virtual object 503 based on the viewpoint of the HMD 300, and the display process of the virtual object 503 on the screen 600 (3D rendering process and CG drawing process)." At this time, the control unit 311 may stop only the display process of the virtual object 503. Details of these processes will be described later in the explanation of step S810.

[0128] In step S807, the control unit 311 calculates the position and orientation of the camera 100 based on the live view image and lens optical information. In this embodiment, a method based on continuous image information, such as SLAM (Simultaneous Localization and Mapping), may be used to calculate the position and orientation. This means that the position and orientation of the camera 100 can be calculated even if the camera 100 is equipped only with general imaging functions. On the other hand, if the camera 100 is equipped with a mechanism capable of acquiring depth information, such as a ToF (Time of Flight) sensor, the position and orientation may be calculated based on that information.

[0129] In step S808, the control unit 311 performs 3D rendering based on the position and orientation of the camera 100 calculated in step S807. Here, the control unit 311 generates an image of the virtual object 512. Furthermore, by performing 3D rendering that takes into account the lens optical information received from the camera 100 in step S804, the control unit 311 can generate a more natural image of the virtual object that reflects the imaging characteristics of the camera 100.

[0130] In step S809, the control unit 311 places an image created by combining the live view image 511 and the image of the virtual object 512 generated in step S808 into the virtual window 510. The control unit 311 then generates an LV composite image (see Figure 6C) by combining the virtual window 510 with the image captured by the HMD 300 of the real space. In the generated LV composite image, the virtual object 512 is placed in the live view image 511, but the virtual object 503 is not placed in the image captured by the HMD 300 of the real space (an image of the real space captured to correspond to the viewpoint of the HMD 300).

[0131] The LV composite image is an image that allows user 501 to check in real time what angle of view the MR image, which is a composite of the real space and virtual objects, will be captured by camera 100. For this reason, the processing in steps S808 to S809 should be less resource-intensive than the virtual object compositing processing in steps S813 to S814, which will be described later. For example, in the processing in steps S808 to S809, "reduction in the quality of the 3D model to be rendered (such as the number of polygons and texture quality)" or "simplification of shading and reflection effects applied to the 3D model" may be performed.

[0132] In step S810, the control unit 311 performs a process (estimation process) to estimate the position and orientation of the virtual object 503 based on the viewpoint of the HMD 300 (user). Furthermore, the control unit 311 performs a display process (3D rendering process and CG drawing) of the virtual object 503. The system is controlled to perform the following process. This generates an LV composite image, including a virtual window 510 and a virtual object 503, as shown in screen 600 in Figure 6B. In the generated LV composite image, the virtual object 512 is not placed in the live view image 511, but the virtual object 503 is placed in the image captured by the HMD 300 of the real space (an image of the real space captured to correspond to the viewpoint of the HMD 300).

[0133] In step S811, the control unit 311 determines whether or not a photo capture has been requested. A photo capture has been requested if the user 501 requests a photo capture by operating the operating member 513 located in the virtual window 510, or by operating the camera 100, etc. If it is determined that a photo capture has been requested, the process proceeds to step S812. If it is determined that a photo capture has not been requested, the process proceeds to step S816.

[0134] In step S812, the control unit 311 receives the captured image from the camera 100.

[0135] In step S813, the control unit 311 performs 3D rendering based on the position and orientation of the camera 100 calculated in step S807. Here, the control unit 311 generates an image of the virtual object 512. At this time, the control unit 311 may also perform 3D rendering that takes into account the lens optical information received from the camera 100 in step S804. This makes it possible to generate a more natural virtual object image that reflects the imaging characteristics of the camera 100.

[0136] In step S814, the control unit 311 generates a composite image by combining the captured image received from the camera 100 in step S812 with the image of the virtual object 512 generated in step S813.

[0137] In step S815, the control unit 311 transmits the composite image generated in step S814 to the camera 100.

[0138] In step S816, the control unit 311 determines whether or not to terminate the live view display (display of the live view image). If it is determined that the live view display should be terminated, the process in this flowchart ends. If it is determined that the live view display should not be terminated, the process proceeds to step S803.

[0139] In step S809, the control unit 311 composites a virtual object onto the live view image to generate an LV composite image as shown in screen 600 of Figure 6C. However, instead of the process in step S809, the control unit 311 may send an image of the virtual object after 3D rendering (conversion) to the camera 100. Then, when the control unit 311 obtains the image from the camera 100 which is a composite of the live view image and the converted image of the virtual object, it may generate an LV composite image by compositing that image with an image of the real space captured from the viewpoint of the HMD 300.

[0140] According to Embodiment 1, the virtual object is placed only in the image viewed by the user, out of two images corresponding to the real space. Therefore, the processing steps of the PC310 can be reduced compared to the case where the virtual object is placed in both images. In addition, since more processing can be spent on rendering a single virtual object, higher quality MR images can be generated. Thus, according to Embodiment 1, high-quality images including the real space and virtual objects, which are used by the user to confirm what kind of shooting is possible, can be generated with less overhead.

[0141] (Variation 1) In Embodiment 1, the control unit 311 places a virtual object 512 in the virtual window 510 (the area of ​​the image captured by the camera 100). At this time, the control unit 311 does not place a virtual object 503 in the area of ​​the image captured according to the user's viewpoint. On the other hand, the control unit 311 does not place a virtual object 512 in the virtual window 510 when it is not looking at the virtual window 510 (the area of ​​the image captured by the camera 100). At this time, the control unit 311 places a virtual object 503 in the area of ​​the image captured according to the user's viewpoint.

[0142] However, if the rendering or compositing load of virtual objects can be reduced compared to the case of generating screen 600 shown in Figure 6A, then an advantageous effect can be obtained compared to conventional technology. For example, when the user is viewing the virtual window 510, the control unit 311 places virtual objects 512 in the virtual window 510. At this time, the control unit 311 places virtual objects 503 with a lower resolution in the area of ​​the image captured of the space according to the user's viewpoint compared to when the user is not viewing the virtual window 510. On the other hand, when the user is not viewing the virtual window 510, the control unit 311 places virtual objects 503 in the area of ​​the image captured of the space according to the user's viewpoint. At this time, the control unit 311 places virtual objects 512 with a lower resolution in the virtual window 510 compared to when the user is viewing the virtual window 510. This also makes it possible to reduce the rendering and compositing load of virtual objects.

[0143] Furthermore, not only virtual objects with reduced resolution, but also virtual objects with arbitrarily reduced image quality (such as virtual objects with reduced rendering accuracy or without coloring) may be used. In other words, any method can be adopted as long as the rendering or compositing load is reduced compared to the case of generating screen 600 shown in Figure 6A.

[0144] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" may be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, "greater than or equal to A" may be rephrased as "greater than (higher; longer; more) than A," and "less than or equal to A" may be rephrased as "less than (lower; shorter; fewer) than A." And "greater than (higher; longer; more) than A" may be rephrased as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be rephrased as "less than or equal to A."

[0145] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0146] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Logic Devices). Examples include programmable gate arrays (GATE Arrays) and CPLDs (Complex Programmable Logic Devices).

[0147] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0148] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0149] The above-disclosed embodiments include the following configurations, methods, and programs. (Composition 1) An information processing device that is communicatively connected to an imaging device, A first acquisition means for acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A control means for generating a composite image by combining a virtual object, the first image, and the second image, It has, The control means is In the first case, where it is determined that the user is not viewing the region of the second image in the composite image, the virtual object is composited into the region of the first image in the composite image, and the virtual object is not composited into the region of the second image in the composite image. In the second case, when it is determined that the user is viewing the region of the second image in the composite image, the virtual object is composited into the region of the second image in the composite image, and the virtual object is not composited into the region of the first image in the composite image. An information processing device characterized by the following: (Configuration 2) In the second case, the control means is Optical characteristic information, including lens aberration information, is acquired from the imaging device. The virtual object based on the position and orientation of the imaging device is transformed based on the optical characteristic information and then composited. The information processing device according to configuration 1, characterized by the above. (Composition 3) In the second case, the control means is Based on the position and orientation of the imaging device, the virtual object is transformed, The converted image of the virtual object is transmitted to the imaging device. When a third image, which is a composite of the second image and the image of the virtual object after conversion, is obtained from the imaging device, the composite image is generated by combining the first image and the third image. An information processing device according to configuration 1 or 2, characterized by the above. (Composition 4) The control means determines that the user is looking at the region of the second image when it determines that a certain region including the region of the second image is located in the direction of the user's line of sight. An information processing device according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The control means determines that the user is viewing a region of the second image when it determines that the occupancy rate of the second image in the composite image exceeds a certain threshold. An information processing device according to any one of configurations 1 to 3, characterized by the above. (Composition 6) When the control means determines that the user is operating the operating member corresponding to the second image, it determines that the user is viewing the region of the second image. An information processing device according to any one of configurations 1 to 3, characterized by the above. (Composition 7) An information processing device that is communicatively connected to an imaging device, A first acquisition means for acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A control means for generating a composite image by combining a virtual object, the first image, and the second image, It has, The control means is The virtual object is composited into the region of the first image and the region of the second image of the composite image, respectively. In the first case, where it is determined that the user is not viewing the region of the second image in the composite image, the image quality of the virtual object composited into the region of the second image is reduced compared to the second case, where it is determined that the user is viewing the region of the second image in the composite image. In the second case, compared to the first case, the image quality of the virtual object composited into the region of the first image is reduced. An information processing device characterized by the following: (Method 1) A control method for an information processing device that is communicatively connected to an imaging device, A first acquisition step involves acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition step involves acquiring a second image of the space captured by the imaging device, A control step of generating a composite image by combining a virtual object, the first image, and the second image, It has, In the control step described above, In the first case, where it is determined that the user is not viewing the region of the second image in the composite image, the virtual object is composited into the region of the first image in the composite image, and the virtual object is not composited into the region of the second image in the composite image. In the second case, when it is determined that the user is viewing the region of the second image in the composite image, the virtual object is composited into the region of the second image in the composite image, and the virtual object is not composited into the region of the first image in the composite image. A control method for an information processing device characterized by the following features. (Method 2) A control method for an information processing device that is communicatively connected to an imaging device, A first acquisition step involves acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition step involves acquiring a second image of the space captured by the imaging device, A composite image is generated by combining a virtual object, the first image, and the second image. A control step, It has, In the control step described above, The virtual object is composited into the region of the first image and the region of the second image of the composite image, respectively. In the first case, where it is determined that the user is not viewing the region of the second image in the composite image, the image quality of the virtual object composited into the region of the second image is reduced compared to the second case, where it is determined that the user is viewing the region of the second image in the composite image. In the second case, compared to the first case, the image quality of the virtual object composited into the region of the first image is reduced. A control method for an information processing device characterized by the following features. (program) A program for causing a computer to function as one of the means of the information processing device described in any of configurations 1 to 7. [Explanation of symbols]

[0150] 100: Camera (imaging device), 300: HMD, 310: PC (Information Processing Unit), 302: Imaging Unit, 311: Control Unit, 314: Communication Unit

Claims

1. An information processing device that is communicatively connected to an imaging device, A first acquisition means for acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A control means for generating a composite image by combining a virtual object, the first image, and the second image, It has, The control means is In the first case, where it is determined that the user has not viewed the region of the second image in the composite image, the virtual object is composited into the region of the first image in the composite image, and the virtual object is not composited into the region of the second image in the composite image. In the second case, when it is determined that the user is viewing the region of the second image in the composite image, the virtual object is composited into the region of the second image in the composite image, and the virtual object is not composited into the region of the first image in the composite image. An information processing device characterized by the following:

2. In the second case, the control means is Optical characteristic information, including lens aberration information, is acquired from the imaging device. The virtual object based on the position and orientation of the imaging device is transformed based on the optical characteristic information and then composited. The information processing apparatus according to feature 1.

3. In the second case, the control means is Based on the position and orientation of the imaging device, the virtual object is transformed, The converted image of the virtual object is transmitted to the imaging device. When a third image, which is a composite of the second image and the image of the converted virtual object, is obtained from the imaging device, the composite image is generated by combining the first image and the third image. The information processing apparatus according to feature 1.

4. The control means determines that the user is looking at the region of the second image when it determines that a certain region including the region of the second image is located in the direction of the user's line of sight. The information processing apparatus according to feature 1.

5. The control means determines that the user is viewing a region of the second image when it determines that the occupancy rate of the second image in the composite image exceeds a certain threshold. The information processing apparatus according to feature 1.

6. When the control means determines that the operating member corresponding to the second image is being operated, it determines that the user is viewing the region of the second image. The information processing apparatus according to feature 1.

7. An information processing device that is communicatively connected to an imaging device, A first acquisition means for acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition means for acquiring a second image of the space captured by the imaging device, A composite image is generated by combining a virtual object, the first image, and the second image. Control means and It has, The control means is The virtual object is composited into the region of the first image and the region of the second image of the composite image, respectively. In the first case, where it is determined that the user is not viewing the region of the second image in the composite image, the image quality of the virtual object composited into the region of the second image is reduced compared to the second case, where it is determined that the user is viewing the region of the second image in the composite image. In the second case, compared to the first case, the image quality of the virtual object composited into the region of the first image is reduced. An information processing device characterized by the following:

8. A control method for an information processing device that is communicatively connected to an imaging device, A first acquisition step involves acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition step involves acquiring a second image of the space captured by the imaging device, A control step to generate a composite image by combining a virtual object, the first image, and the second image, It has, In the control step described above, In the first case, where it is determined that the user has not viewed the region of the second image in the composite image, the virtual object is composited into the region of the first image in the composite image, and the virtual object is not composited into the region of the second image in the composite image. In the second case, when it is determined that the user is viewing the region of the second image in the composite image, the virtual object is composited into the region of the second image in the composite image, and the virtual object is not composited into the region of the first image in the composite image. A control method for an information processing device characterized by the following features.

9. A control method for an information processing device that is communicatively connected to an imaging device, A first acquisition step involves acquiring a first image in which the space is captured according to the user's viewpoint, A second acquisition step involves acquiring a second image of the space captured by the imaging device, A control step to generate a composite image by combining a virtual object, the first image, and the second image, It has, In the control step described above, The virtual object is composited into the region of the first image and the region of the second image of the composite image, respectively. In the first case, where it is determined that the user is not viewing the region of the second image in the composite image, the image quality of the virtual object composited into the region of the second image is reduced compared to the second case, where it is determined that the user is viewing the region of the second image in the composite image. In the second case, compared to the first case, the image quality of the virtual object composited into the region of the first image is reduced. A control method for an information processing device characterized by the following features.

10. The computer is used as one of the means in the information processing apparatus described in any one of claims 1 to 7. A program to make something work.

Images