Information processing apparatus, method for controlling information processing apparatus, system, and program

Abstract

To provide a technique that can improve the operability in operations for a virtual object arranged at a position where it is difficult to be operated from the position of a user in a virtual space.SOLUTION: An information processing apparatus has: display means that displays, in a space in which a virtual object is arranged, a virtual display displaying viewpoint information indicating the position and posture of a virtual viewpoint and an image of the virtual object seen from the virtual viewpoint; and control means that, according to an operation for the virtual display, controls to select the virtual object arranged in the space through the virtual display.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus.

Background Art

[0002] In recent years, Virtual Reality (VR) and Mixed Reality (MR) have been widely used. In VR and MR, a user can perform operations such as selecting, moving, duplicating, and controlling the display / non-display of virtual 3D models (hereinafter referred to as virtual objects).

[0003] When a user controls virtual objects, there may be a case where there are multiple virtual objects, and the virtual objects are arranged overlapping in the depth direction from the user's position, making it difficult for the user to visually recognize them. Also, there may be a case where the virtual objects are arranged far from the user's position, making it difficult for the user to visually recognize them.

[0004] Patent Document 1 discloses a technique of creating a light ray for selecting a virtual object from the user's dominant hand, controlling the length of the light ray based on the distance between the dominant hand and the non-dominant hand, and enabling selection of any of the virtual objects overlapping in the depth direction.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in Patent Document 1, for a virtual object arranged behind and hidden by a virtual object in the foreground, it is difficult for the user to visually recognize the virtual object in the back, so it is difficult to perform an operation of selecting the virtual object arranged in the back.

[0007] Therefore, an object of the present invention is to provide an information processing apparatus capable of improving the operability of an operation on a virtual object arranged at a position difficult to operate from the position of a user in a virtual space.

Means for Solving the Problems

[0008] One aspect of the present invention includes display means for displaying a virtual display that displays viewpoint information indicating the position and orientation of a virtual viewpoint and an image of the virtual object viewed from the virtual viewpoint on a space where the virtual object is arranged, and control means for controlling to select the virtual object arranged on the space via the virtual display in response to an operation on the virtual display. The information processing apparatus is characterized by having the above.

Effects of the Invention

[0009] According to the present invention, it is possible to provide an information processing apparatus capable of improving the operability of an operation on a virtual object arranged at a position difficult to operate from the position of a user in a virtual space.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same reference numerals are assigned to the same or similar configurations, and duplicate explanations are omitted.

[0012] <First Embodiment> With reference to FIG. 1, the information processing system 1 according to the first embodiment will be described. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0013] The HMD 100 is a head-mounted display device (electronic device) that can be worn on a user's head. A composite image in which an imaging image obtained by the HMD 100 imaging the front range of the user and content such as CG in a form corresponding to the posture of the HMD 100 are combined is displayed on the HMD 100.

[0014] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 by wire such as a USB cable or wirelessly such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by synthesizing a captured image and CG, and transmits the composite image to the HMD 100. Here, although a PC is described as an example of the information processing apparatus, the information processing apparatus is not limited to this. For example, the information processing apparatus may be a smartphone or a tablet terminal, and each configuration of the PC 110 may be possessed by the HMD 100.

[0015] The controller 120 performs various controls of the HMD 100. If the PC 110 is in a specific control mode, when a user operation is performed on the controller 120, the HMD 100 is controlled according to the user operation. As shown in FIG. 1, the controller 120 may have a ring-shaped (ring type) shape that can be worn and supported on the user's finger or a hand-held shape that can be held by hand. Further, the controller 120 has physical buttons for performing a determination operation or a selection operation on the display. The controller 120 performs wireless communication with the PC 110 by Bluetooth. Note that the controller is not limited to a system that communicates with the PC 110, and may be a system that communicates with the HMD 100.

[0016] The user can change the indicated position on the display according to the movement of the controller 120 by moving the controller 120. The indicated position may be represented by a point, or may be represented by a virtual ray (ray) connecting the point of the indicated position and the controller with a straight line (line segment) or a dotted line. By pressing a physical button, a menu determination operation or a selection operation can be performed. Note that although the shape of the controller 120 is assumed to be a ring type or a hand-held type, it is not limited thereto as long as it can be supported by a finger, a hand, or an arm. Further, although the button is assumed to be a physical button, it may be operable like a track pad, a touch panel, a wheel, or a track ball, and in addition to pressing the button, a slide operation, a flick operation, or a touch operation may be performed.

[0017] Note that the controller may be worn on at least one of a finger, a hand, or an arm.

[0018] Note that the controller may be attached to an object held by a hand and used to obtain position information and attitude information of the attached position from a sensor. Examples of such an object include an object imitating a tool.

[0019] <Internal Structure of HMD> Referring to FIG. 2, the internal structure of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, a working memory 206, and a gaze imaging unit 207.

[0020] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by synthesizing an imaging image captured by the imaging unit 202 of the space in front of the user and a CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, a plurality of hardware may share the processing to control the entire device.

[0021] The imaging unit 202 includes two cameras (imaging devices). The two cameras are for capturing imaging images used for synthesis with the image in the virtual space and generation of position and orientation information, and have an imaging unit for the left eye and an imaging unit for the right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the wearer of the HMD 101, and an image (imaging image) of each frame in the moving image is output from the imaging unit for the left eye. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the wearer of the HMD 101, and an image (imaging image) of each frame in the moving image is output from the imaging unit for the right eye. That is, the imaging unit 202 acquires an imaging image as a stereo image having a parallax substantially coinciding with the positions of the left and right eyes of the wearer of the HMD 101. Also, by distance measurement using a stereo camera, information on the distance from the two cameras to the subject can be acquired as distance information. In the HMD for the MR system, it is preferable that the central optical axis of the imaging range of the imaging unit is arranged to substantially coincide with the line-of-sight direction of the wearer of the HMD.

[0022] Each of the imaging unit for the left eye and the imaging unit for the right eye has an optical system and an imaging device. Light incident from the outside enters the imaging device through the optical system, and the imaging device outputs an image corresponding to the incident light as an imaging image. Images captured by the two cameras of the subject (the range in front of the user) are output to the PC 110 and the HMD control unit 201. Note that the imaging unit 202 may capture and output a video instead of the imaging image.

[0023] The image display unit 203 displays the composite image. The image display unit 203 includes a liquid crystal panel, an organic EL panel, or the like. When the user wears the HMD 100, the image display unit 203 is arranged in front of each of the user's eyes. Note that a device using a semi-transmissive half mirror can also be used for the image display unit 203. In this case, for example, the image display unit 203 may display an image such that CG is directly superimposed on the real space visible through the half mirror by a technology generally called AR (Augmented Reality). Also, the image display unit 203 may display an image of a complete virtual space without using a captured image by a technology generally called VR (Virtual Reality).

[0024] The attitude sensor unit 204 acquires the attitude (and position) information of the HMD 100. Note that the attitude sensor unit 204 may acquire the attitude information of the user (the user wearing the HMD 100) corresponding to the attitude (and position) of the HMD 100. The attitude sensor unit 204 includes an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 204 is used when acquiring the information (attitude information) of the user's attitude, and the HMD control unit 201 outputs the information (attitude information) of the user's attitude to the PC 110. Note that the attitude information may be acquired from any one or more of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor.

[0025] The HMD control unit 201 estimates the positions or postures of the joints of the user's hand and fingers from the two camera images obtained by the imaging unit 202. The joint points include the characteristic points of parts such as finger joints, fingertips, the back of the hand (palm), and the arm. Each joint point indicates a coordinate position, and the posture can be estimated from the information of multiple joint points. As a method for estimating the positions or postures of the hand and each joint point of the hand, for example, known object recognition or pose estimation methods of machine learning using a convolutional neural network can be used. Also, the position information of each joint point of the hand in the depth direction can be obtained by calculating the distance from the imaging unit 202 to each joint point by triangulation using stereo matching with the two camera images obtained by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the HMD control unit 201 to the PC 110.

[0026] Note that the HMD control unit 201 may estimate the position or posture of the controller from the two camera images obtained by the imaging unit 202. The shape of the controller may be directly detected, or the infrared sensor of the controller may be detected.

[0027] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory, and stores programs and the like to be executed by the HMD control unit 201, which will be described later.

[0028] The working memory 206 is used as a buffer memory for temporarily holding the image data captured by the imaging unit 202, an image display memory for the image display unit 203, a working area of the HMD control unit 201, and the like.

[0029] The imaging unit 207 for the line of sight is a camera that acquires an image for detecting the user's line of sight, and is attached inside the HMD to image the user's eyes when the user wears the HMD 100. The image of the subject (the user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the user's line of sight from the image captured by the imaging unit 207 for the line of sight, and identifies the location on which the user is gazing on the image display unit 203.

[0030] <Internal Configuration of the Controller> Referring to FIG. 2, the internal configuration of the controller 120 will be described. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0031] The controller control unit 221 is a CPU that controls each component of the controller 120. Note that instead of the controller control unit 221 controlling the entire apparatus, a plurality of hardware components may share the processing to control the entire apparatus.

[0032] The operation unit 222 includes buttons. The operation unit 222 detects whether a button has been operated, and transmits detection information to the PC 110 via the communication unit 223. Note that the operation unit 222 may have a plurality of types of input formats.

[0033] The communication unit 223 performs wireless communication with the PC 110 via Bluetooth. When there are a plurality of controllers, each performs wireless communication with the PC 110 via Bluetooth.

[0034] The controller attitude sensor unit 224 includes an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 120. The detected change information on the position and attitude is communicated from the communication unit 223 to the PC 110 via the controller control unit 221.

[0035] The output unit 225 is composed of a light source of an LED, a speaker, a vibration element, and the like.

[0036] <Internal Configuration of the PC> With reference to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 includes a control unit 211, a non-volatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0037] The control unit 211 is a CPU that controls each part of the PC 110 according to the input signal and the program described later. Note that instead of the control unit 211 controlling the entire apparatus, a plurality of hardware components may share the processing to control the entire apparatus. The control unit 211 receives the image (captured image) acquired by the imaging unit 202 and the attitude information acquired by the attitude sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image so as to cancel the aberration in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 synthesizes the captured image and an arbitrary CG to generate a synthesized image. The control unit 211 transmits the synthesized image to the HMD control unit 201 in the HMD 100.

[0038] Note that the control unit 211 controls the position, orientation, and size of the CG in the synthesized image based on the information (distance information and attitude information) acquired by the HMD 100. For example, when the control unit 211 arranges a virtual object indicated by the CG near a specific object existing in the real space in the space represented by the synthesized image, the closer the distance between the specific object and the imaging unit 202, the larger the virtual object (CG). By controlling the position, orientation, and size of the CG in this way, the control unit 211 can generate a synthesized image such that the CG object not arranged in the real space appears as if it is arranged in the real space.

[0039] In addition, the control unit 211 receives the information estimated by the HMD control unit 201 of the HMD 100. The received information is temporarily stored in the working memory 213.

[0040] Also, in the control unit 211, the communication unit 214 receives from the communication unit 223 of the controller 120 information on changes in the position or orientation of the controller 120 and operation information of the operation unit 222 by the user. The control unit 211 superimposes and displays an instruction position corresponding to the change information of the position or orientation of the controller 120 on the synthesized image. Note that the control unit 211 may superimpose and display an instruction position corresponding to the change information of the position and orientation of the controller 120 on the synthesized image. Further, the control unit 211 executes processing according to the operation information of the operation unit 222 by the user. For example, based on the input information of the button, processing related to selection and determination is performed.

[0041] Note that the control unit 211 may control the output unit 225 of the controller 120 via the communication unit 214.

[0042] Also, the control unit 211 respectively performs control of virtual objects, acquisition of instructions for virtual camera windows, generation of virtual camera window images, generation of instructions, and generation of synthesized images. The virtual camera window is a virtual display and is hereinafter expressed as a window.

[0043] As control of virtual objects, the control unit 211 generates an image depicting the virtual objects constituting the virtual space. More specifically, the control unit 211 generates an image based on the data of the virtual objects stored in the non-volatile memory 212 or the recording medium 215. Further, the control unit 211 acquires instruction information input by the user to the virtual objects via the controller 120, and controls the position, shape, color information, display / non-display, etc. of the virtual objects based on the instruction information.

[0044] In addition, the control unit 211 acquires the user's instructions regarding the position and orientation and the viewing angle of the drone camera in the virtual space. The control unit 211 acquires the user's instructions regarding the position and orientation of the window in the virtual space. The drone camera is arranged at an arbitrary position in the virtual space and is a viewpoint different from the user's viewpoint for observing virtual objects therefrom, and the window displays the viewpoint from the drone camera. The drone camera and the window are controlled so as to be able to freely move in the virtual space by operating the controller 120.

[0045] In addition, the control unit 211 generates and displays a drone camera as viewpoint information representing the position and orientation of a virtual viewpoint. Further, the control unit 211 generates a window according to the position and orientation of the drone camera. The control unit 211 generates by superimposing the window on the virtual space that the user is viewing, and displays an image of a virtual object observable from the drone camera in the window. At this time, the control unit 211 determines the range observable from the drone camera based on the instruction of the viewing angle of the drone camera acquired by the control unit 211, and displays the image of the virtual object in the window. The user can observe the virtual object arranged in the virtual space from a free viewpoint through the window. Also, when the user gives an instruction to a virtual object displayed in the window, the control unit 211 controls the virtual object based on the instruction, and displays the image of the virtual object in the window according to the instruction. Thereby, the user can operate the virtual object in the virtual space through the window. For example, when the user gives an instruction to move a virtual object through the window, the virtual object actually arranged in the virtual space is moved according to the instruction, and the virtual object is also moved on the window according to the viewpoint of the drone camera. Specific examples will be described later. Although the position and orientation of a viewpoint different from the user's viewpoint are displayed by the drone camera, the position of the viewpoint may be displayed as a point, showing only the position. In this case, the orientation of the viewpoint can be represented by the viewing angle displayed in the window. In addition, when the drone camera is hidden behind a virtual object that can be moved by the user through the window, since the position and posture of another viewpoint cannot be displayed by a virtual object directly showing the drone camera, the position of another viewpoint may be displayed in coordinates. The coordinates may be numerical values or coordinate information indicating a position on a map. When the drone camera is arranged at a position outside the viewing angle, since it may not be possible to display the position of the drone camera, a display may be made to show only the orientation of the viewpoint, such as by an arrow. That is, the position and orientation of another viewpoint can be represented by the display of the drone camera, the position of another viewpoint only can be represented by the display of a point or coordinates, and the orientation of another viewpoint only can be represented by displaying the direction without limiting the position, such as by an arrow.Note that not only the position and orientation of the drone camera but also the zoom ratio may be displayed. The above specific examples will be described later with reference to FIG. 9.

[0046] In addition, in the case of VR, the control unit 211 generates a virtual object representing the controller in the virtual space based on the position information or attitude information of the controller. Further, the control unit 211 generates a path (ray) from the virtual object representing the controller generated in the virtual space to the coordinates pointed to by the user in the virtual space. The path is a tool used by the user to select the virtual object, and in this embodiment, it is a virtual object in the shape of a laser. When the user is pointing at a virtual object in the virtual space, the control unit 211 generates one path from the operation unit to the virtual object. On the other hand, when the user is pointing at a virtual object on the window, the control unit 211 generates two paths, one from the operation unit to the virtual object on the window and the other from the drone camera to the virtual object in the virtual space. Note that although the control unit 211 of this embodiment generates a path, a pointer may be generated at the coordinates pointed to by the user in the virtual space. At this time, when the user is pointing at a virtual object on the window, two pointers are generated for the virtual object on the window and the virtual object in the virtual space. Note that the starting point of the ray may be in contact with or separated from the controller. Note that the ray and the pointer may be displayed simultaneously. In the case of MR, the control unit 211 controls to generate a path (ray) from the controller to the coordinates pointed to in the mixed reality space based on the position or attitude of the controller in the captured image.

[0047] In addition, the control unit 211 synthesizes the virtual objects constituting the virtual space, the drone camera and the window, and the rays and pointers indicating the instruction positions indicated by the controller, and generates an image of the virtual space. The control unit 211 sends the image of the virtual space to the image display unit 203.

[0048] The non-volatile memory 212 is a non-volatile memory that can be electrically erased and recorded, and stores programs described later executed by the control unit 211, information such as CG, etc. Note that the control unit 211 can switch the CG read from the non-volatile memory 212 (that is, the CG used for generating the composite image).

[0049] The working memory 213 is used as a buffer memory that temporarily holds the image data captured by the imaging unit 202 and the time-series information of the coordinate positions of the estimated joint points of the hand, the image display memory of the image display unit 203, the working area of the control unit 211, etc.

[0050] The communication unit 214 performs wireless communication with the controller 120 via Bluetooth. When there are multiple controllers, wireless communication with each controller is performed via Bluetooth.

[0051] An information processing program is recorded on the recording medium 215. The information processing program is read from the recording medium 215 and expanded into the working memory 213, and is executed by the control unit 211. Note that the information processing program may be stored in the non-volatile memory 212. In addition, data related to virtual objects constituting the virtual space (shape information, position and orientation information, etc.) and data of the virtual space such as data related to light sources irradiated in the virtual space are recorded on the recording medium 215.

[0052] Also, the estimation of the hand joints may be performed in the PC 110. In that case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or orientation of each joint point of the hand, processes the image using the information, and outputs it to the HMD 100. Note that the control unit 211 may estimate the position and orientation of each joint point of the hand, process the image using the information, and output it to the HMD 100.

[0053] In addition to the above components of the MR system, for example, in the control unit of either the HMD 100 or the PC 110, various image processes are performed on the captured image acquired by the imaging unit 202 and the display image displayed on the image display unit 203. However, since this is not the main focus of the present invention, the description will be omitted.

[0054] FIG. 3 is a flowchart for explaining the operation processes of the drone camera, window, and virtual object when the PC 110 of the first embodiment controls the virtual object displayed in the window. FIG. 4 is an explanatory diagram of the operation of the drone camera, window, and virtual object when the PC 110 of the first embodiment controls the virtual object displayed in the window. Hereinafter, the operation of the PC 110 of the first embodiment will be described with reference to the flowchart of FIG. 3 and the explanatory diagram of FIG. 4.

[0055] The process shown in the flowchart of FIG. 3 is realized by the control unit 211 executing the information processing program according to this embodiment.

[0056] In FIG. 4, the user 401 wears the controller 120 as an operation device in the hand and the HMD 100 as a display device on the head. The operation information, position, and attitude information of the user 401's controller 120 are transmitted to the PC 110, which is an information processing device (not shown). Based on the position and attitude information of the HMD 100, the operation information of the controller 120, and the position and attitude information, the PC 110 generates CG. In the case of VR, the PC 110 also generates the CG serving as the background image. In the case of MR, the PC 110 transmits to the HMD 100 a composite image obtained by superimposing the generated CG on the image captured by the HMD 100. The HMD 100 displays the CG image or the composite image acquired from the PC 110. Note that the position and attitude information of the HMD 100 may be based on the information acquired by the attitude sensor unit 204 of the HMD 100 or may be based on the captured image captured by the imaging unit 202 of the HMD 100.

[0057] FIG. 3 explains the flowchart of the process of the control unit 211 in the virtual space.

[0058] In step S301, in response to an instruction to generate a virtual object being input, for example, by a user via a GUI such as a model creation button (not shown), the control unit 211 acquires data from the non-volatile memory 212 and arranges the virtual object in the virtual space. The control unit 211 generates an image in which the virtual object is arranged and transmits it to the HMD 100.

[0059] FIG. 4(a) is a diagram showing virtual objects 421, 422, and 423 arranged. Since the virtual object 421 is arranged behind the virtual object 422 and is hidden from the position of the user 401, it is difficult for the user 401 to visually recognize the virtual object 421. In the present embodiment, the case where the user 401 operates the virtual object 421 will be described.

[0060] In step S302, the control unit 211 arranges the drone camera 411 in the virtual space. Here, the drone camera 411 may be arranged at the initial position. The initial position is near the user and may be any position that is easily visible from the position of the user. The control unit 211 arranges the drone camera 411 in the virtual space and displays the composite image.

[0061] In step S303, the control unit 211 arranges a window 413 for displaying an image of the virtual object observed from the drone camera 411 arranged at the initial position in the virtual space. Here, the window 413 may be arranged at the initial position. Here, the initial position is near the user and may be any position that is easily visible from the position of the user. The control unit 211 arranges the window 413 in the virtual space and displays the composite image.

[0062] Note that the process of step S302 may be performed when the user instructs the placement of the drone camera, or may be performed in response to the placement of the virtual object in the virtual space. Also, the process of step S303 may be performed in response to the placement of the drone camera. Note that the order of the processes of step S302 and step S303 may be reversed. In this case, the drone camera may be placed in response to the placement of the window.

[0063] Figure 4(b) is a diagram in which the drone camera 411 and the window 413 are placed at the initial positions near the user 401. The drone camera 411 has a lens unit 412, and the user 401 can confirm the posture of the drone camera 411 based on the orientation of the lens unit 412. In Figure 4(b), assume a scene in which the lens unit 412 of the drone camera 411 is directed from near the user 401 toward the virtual objects 421, 422, and 423. Images of the virtual objects 431, 432, and 433 observed from the drone camera 411 are displayed on the window 413 placed in front of the user 401. Here, the virtual object 431 corresponds to the virtual object 421, the virtual object 432 corresponds to the virtual object 422, and the virtual object 433 corresponds to the virtual object 423, respectively. Since the virtual object 431 displayed on the window 413 is placed behind the virtual object 432 and hidden, it is difficult for the user 401 to visually recognize the virtual object 431.

[0064] In step S304, the control unit 211 determines whether there is an instruction to change the position and orientation, field of view angle of the drone camera 411, or the position and orientation of the window 413. Here, for example, the user 401 operates the drone camera or the window via a GUI (Graphical User Interface) such as a virtual button (not shown) or a UI (User Interface) such as a controller. As a result of the determination in step S304, if there is no instruction to change the position and orientation, field of view angle of the drone camera 411, or the position and orientation of the window 413, the process proceeds to step S313; if there is an instruction, the process proceeds to step S305. When the user wants to move the virtual object further away than the range displayed in the window, the user moves the position of the drone camera 411 to a position farther from the virtual object or gives an instruction to widen the field of view angle from the drone camera 411. Thereby, the range of the virtual space displayed in the window 413 can be widened, and the virtual object can be moved farther away.

[0065] In step S305, the control unit 211 obtains an instruction from the user regarding the position or orientation of the window 413 based on the operation input via the controller 120, and proceeds to step S306.

[0066] In step S306, the control unit 211 changes the position or orientation based on the obtained instruction and arranges the window 413 in the virtual space, and proceeds to step S307.

[0067] In step S307, the control unit 211 obtains an instruction from the user regarding the position or orientation of the drone camera 411 based on the operation input via the controller 120, and proceeds to step S308.

[0068] In step S308, the control unit 211 changes the position or orientation based on the obtained instruction and arranges the drone camera 411 in the virtual space, and proceeds to step S309.

[0069] In addition, in step S305 and step S307, instructions to change the position and orientation of the drone camera 411 and the window 413 may be obtained. Note that the order of the processes from step S305 to step S308 may be changed to the order of step S307, step S308, step S305, and step S306.

[0070] FIG. 4(c) is a diagram showing a scene in which the position and orientation of the drone camera 411 and the window 413 are changed. Assume a scene in which the drone camera 411 is moved from the initial position above the virtual objects 421, 422, and 423, and the lens unit 412 is reoriented (perpendicular to the XZ plane) to face the virtual objects 421, 422, and 423. The window 413 is moved from the initial position to the right side of the user 401, and an image including the virtual objects 431, 432, and 433 obtained by observing the virtual objects 421, 422, and 423 from above is displayed. In this way, the user 401 can visually recognize the virtual object 421, which was difficult to visually recognize because it was hidden behind another virtual object in the virtual space, as the virtual object 431 through the window.

[0071] In step S309, the control unit 211 obtains an instruction regarding the viewing angle of the drone camera 411 of the user 401 based on an operation input via the controller 120.

[0072] In step S310, the control unit 211 changes the image of the virtual object to be displayed on the window 413 according to the instructed viewing angle and displays it within the window 413.

[0073] FIG. 4(d) is a diagram in which the viewing angle of the drone camera 411 is changed from FIG. 4(c). The virtual objects 431, 432, and 433 displayed on the window 413 are displayed smaller than in FIG. 4(c) because the user 401 widened the viewing angle of the drone camera 411. In addition, a display indicating the current zoom ratio of the drone camera and a GUI for changing the zoom ratio may be provided.

[0074] In step S311, the control unit 211 determines whether there is an instruction to display a path representing the direction indicated by the controller, that is, a ray (light ray). For example, the user can input an instruction to display the ray by pressing a button (not shown) on the controller 120. As a result of the determination in step S311, if there is no instruction to display the path, the process returns to step S304; if there is an instruction, the process proceeds to step S312.

[0075] In step S312, the control unit 211 generates a path from the controller 120 drawn in the virtual space to the coordinates pointed by the user in the virtual space. For example, when the user 401 points to the virtual object 421 in the virtual space, the path is generated from the operation unit to the virtual object 421. When the user 401 points to the virtual object 431 on the window, the path from the controller 120 to the virtual object 431 on the window and the path from the drone camera 411 to the virtual object 421 in the virtual space are generated. The control unit 211 displays the composite image with the path generated in the virtual space. Note that the control unit 211 may switch the display or non-display of the path according to the user's instruction, or may make the path non-displayed when there is no user operation for a predetermined time. Instead of displaying the path, a pointer indicating the position indicated by the controller may be displayed.

[0076] In step S313, the control unit 211 determines whether the user has selected a virtual object on the window. For example, after the user 401 points to the virtual object 431 on the window, the user can select the virtual object 421 by pressing a button (not shown) on the controller 120. When the virtual object is selected, the virtual object may be made distinguishable as selected by changing its color or contour or by adding a mark outside the virtual object. As a result of the determination in step S313, if there is no selection of the virtual object, the process proceeds to step S320; if there is a selection of the virtual object, the process proceeds to step S314.

[0077] Figure 4(e) is a diagram in which user 401 has selected virtual object 421. A path 441 from controller 120 displayed in the virtual space to virtual object 421 selected by user 401 is shown. Also, a path 442 as seen from drone camera 411 is shown in the window. Also, the contour lines forming the respective shapes are changed so that it can be seen that virtual object 421 selected by user 401 and virtual object 431 have been selected.

[0078] Figure 4(f) is a diagram in which user 401 has selected virtual object 431. A path 443 from controller 120 displayed in the virtual space to virtual object 431 on window 413 selected by user 401 and a path 444 from lens unit 412 of drone camera 411 to virtual object 421 in the virtual space are shown. The contour lines forming the respective shapes are changed so that it can be seen that virtual object 421 and virtual object 431 selected by user 401 have been selected. In this way, the user is notified that a virtual object is being selected. Note that any display may be used as long as the selection can be notified, for example, a balloon or the like may be used to notify the selection.

[0079] In step S314, control unit 211 acquires the instruction information of user 401 for the selected virtual object, and controls the position, shape, color information, display / non-display, etc. of the virtual object based on the instruction information. For example, when user 401 changes the position while pointing while selecting a virtual object, control unit 211 moves the virtual object, and when user 401 changes it via a GUI such as a menu where the shape and color can be changed after selecting the virtual object, control unit 211 changes the virtual object to the changed form.

[0080] In step S315, control unit 211 generates an image of the virtual object reflected based on the instruction information of user 401. Also, control unit 211 generates an image of the virtual object reflected based on the instruction information of user 401 as seen from drone camera 411, and displays it on window 413.

[0081] FIG. 4(g) is a diagram in which the user 401 moves the virtual object 421. When the user 401 moves the virtual object 431 displayed in the window 413 from a position above the virtual object 432 to a position above the virtual object 433, the virtual object 421 in the virtual space is moved from behind the virtual object 422 to behind the virtual object 423 as viewed by the user 401.

[0082] FIG. 4(h) is a diagram in which the user 401 changes the color of the virtual object 421. When the user 401 selects the virtual object 421 and then displays a GUI of a menu where color selection is possible and the user 401 selects another color, the virtual object 421 and the virtual object 421 are changed to the selected color.

[0083] In step S320, when an end instruction for this process is input by the user or a condition for ending this process is satisfied, the control unit 211 ends this process. On the other hand, when an end instruction for this process has not been input and the condition for ending this process is not satisfied, the control unit 211 proceeds to step S304.

[0084] According to the first embodiment, the user can place a drone camera in the virtual space and control the virtual objects observed therefrom via a window provided near the user. The user can indirectly select and operate the virtual objects arranged in the space by selecting and moving the virtual objects displayed in the virtual display. That is, virtual objects that are hidden behind other virtual objects, arranged far away and displayed small, and difficult to visually recognize from the user's position can be quickly controlled.

[0085] <Second Embodiment> In the first embodiment, the case where the movement of the virtual object is performed within the window has been described. In particular, when it is desired to move the virtual object far away, it has been described to change the position and field angle of the drone camera and expand the range of the virtual space displayed in the window. In this embodiment, an example of moving the virtual object far away without expanding the range of the virtual space displayed in the window will be described.

[0086] Note that the configuration of the PC 110 according to the second embodiment is the same as that according to the first embodiment. Therefore, hereinafter, only the operation process of moving the virtual object, which is a process different from the process performed in the first embodiment, will be described.

[0087] FIG. 5 is a flowchart for explaining the operation processes of the drone camera, the window, and the virtual object when the PC 110 according to the second embodiment controls the virtual object displayed in the window by the user. Since steps S301 to S315 and step S320 are the same as those in FIG. 3, detailed descriptions thereof are omitted.

[0088] In step S516, the control unit 211 determines whether an instruction to move the virtual object outside the display area of the window has been given. For example, when the virtual object being moved by the user 401 touches the frame of the window, or touches the frame of the window for a predetermined time, the control unit 211 determines that it will move outside the window. If not determined, the process proceeds to step S320, and if determined, the process proceeds to step S517.

[0089] In step S517, the control unit 211 moves the drone camera 411 on a plane orthogonal to the orientation of the lens unit 412 in accordance with the position of the frame of the window touched by the virtual object that is about to move outside the display area of the window. For example, as shown in FIG. 6(a), when the virtual object 431 touches the position of the left side of the frame of the window 413, the drone camera 411 moves in the left direction. That is, the drone camera 411 moves planar while maintaining the posture toward the virtual object 431. The control unit 211 moves the virtual object 421 in accordance with the movement of the drone camera 411 while fixing the positional relationship between the drone camera 411 and the virtual object 421 in the virtual space. That is, until the virtual object 431 touches the frame of the window 413, the virtual object 431 and the virtual object 421 move within the window, and the drone camera 411 does not move. After the virtual object 431 touches the frame of the window 413, the virtual object 431 does not move within the window, and the drone camera 411 and the virtual object 421 move. Note that after the virtual object 431 touches the frame of the window 413, that is, when the drone camera moves, the inside of the window is displayed so as to move in the direction opposite to the moving direction of the drone camera. Note that the drone camera 411 may also be moved when the virtual object 431 approaches within a predetermined distance from the frame of the window 413.

[0090] In addition, when the virtual object 431 touches the frame of the window 413, the position of the drone camera may be moved so that the virtual object 431 comes to a predetermined position of the window 413. In addition, when a predetermined time has elapsed while the virtual object 431 is touching the frame of the window 413, the moving speed of the drone camera 411 may be adjusted. In addition, the moving speed of the drone camera 411 when the virtual object 431 touches the frame of the window 413 may be set to an arbitrary value by the user. In addition, the range within which the drone camera 411 can move may be limited to within the field of view of the HMD 100 worn by the user, or it may be moved not only within the field of view of the HMD 100. In addition, when the drone camera 411 moves, only the position is moved without changing the posture, but only the posture may be changed to change the display area of the window 413, or the posture may be changed while moving the position. In addition, when the virtual object 431 is moved so as to contact the corner of the upper left frame of the window 413, the control unit 211 may control to move the virtual object 421 and the drone camera 411 according to the direction from the center of the window 413 to the corner of the upper left frame. In addition, for preventing misoperation, when the speed at which the virtual object 431 is selected and moved is equal to or higher than the threshold value, the position of the drone camera 411 may not be moved even if it touches the frame of the window 413.

[0091] FIG. 6 is an explanatory diagram of the operations of the drone camera, the window, and the virtual object when the PC 110 according to the second embodiment controls the virtual object displayed in the window by the user 401. In the present embodiment, a case where the user 401 further moves the virtual object 431 on the window 413 from FIG. 4(g) in which the virtual object 431 is moved will be described.

[0092] FIG. 6(a) is a diagram in which the virtual object 431 has moved further from the position in FIG. 4(f) and touched the left (x-direction) of the frame of the window 413. After the virtual object 431 touches the frame of the window 413 and a predetermined time has elapsed, the drone camera 411 moves in the X direction while maintaining the posture directed at the virtual object 421. At the same time, the control unit 211 moves the virtual object 421 in the virtual space in accordance with the moving direction of the drone camera 411. The control unit 211 displays an image observable from the moving drone camera 411 while the virtual object 431 touches the frame of the window 413.

[0093] FIG. 6(b) is a diagram in which the drone camera 411 has moved in the X direction. The positions of the drone camera 411 and the virtual object 421 remain fixed while the drone camera 411 moves in the X direction, and the virtual object 421 displayed in the window 413 also moves in the x direction in the same manner.

[0094] According to the second embodiment, the user can quickly control a virtual object that is difficult to visually recognize from the user's position without changing the position or viewing angle of the drone camera 411. In particular, it can be easily moved outside the display area of the window 413 that is observing the virtual object.

[0095] Note that, as an example of means for the control unit 211 to determine whether the user moves the virtual object outside the window, it has been mentioned that the virtual object has touched the window frame for a predetermined time, but it is not limited to this. For example, it may be determined when the path 443 indicating the virtual object touches the window frame, or it may be determined based on whether the speed when the virtual object touches the window frame is greater than a predetermined value.

[0096] <Third Embodiment> In the second embodiment, the case where the drone camera is moved planar in accordance with the position of the window frame touched by the virtual object has been described. In this embodiment, an example of the case where the drone camera and the virtual object are moved three-dimensionally will be described.

[0097] Note that the configuration of the PC 110 according to the third embodiment is the same as that according to the first embodiment. Therefore, hereinafter, only the operation process of moving the drone camera and the virtual object, which is a process different from the process performed in the first embodiment, will be described. In this embodiment, the case where the virtual object 421 is moved from FIG. 4(e) in which the user 401 has selected the virtual object 421 will be described.

[0098] FIG. 7 is a flowchart for explaining the operation process of the drone camera, the window, and the virtual object when the user controls the virtual object displayed in the window, which is executed by the PC 110 according to the third embodiment. Since steps S301 to S313 and step S320 are the same as those in FIG. 3, detailed description thereof will be omitted.

[0099] In step S714, the control unit 211 determines whether the user 401 has fixed the positional relationship between the drone camera 411 and the virtual object 421 on the virtual space that the user has selected. For example, after the user 401 selects the virtual object on the window as shown in FIG. 4(e), the user can fix the positional relationship between the drone camera 411 and the selected virtual object 421 by pressing a button (not shown) on the controller 120.

[0100] In step S715, the control unit 211 acquires an instruction from the user 401 regarding the position or orientation of the drone camera 411 based on the operation input via the controller 120. For example, the user 401 can input an instruction to change the position or orientation of the drone camera 411 by pressing a button (not shown) on the controller 120.

[0101] In step S716, the control unit 211 moves the drone camera 411 based on the instruction acquired via the controller 120. At the same time, the control unit 211 moves the virtual object whose positional relationship with the drone camera 411 is fixed in accordance with the moving direction of the drone camera 411. Further, the control unit 211 displays the drone camera 411, the virtual object, and the window 413 on which the moved virtual object is displayed.

[0102] FIG. 8 is an explanatory diagram of the drone camera, window, and operation of the virtual object when the user 401 controls the virtual object displayed in the window, which is executed by the PC 110 according to the third embodiment.

[0103] FIG. 8 shows the drone camera moved in the X and Y directions from the position in FIG. 4(f). With the positions of the drone camera 411 and the virtual object 421 fixed, the user 401's operation causes it to move in the X and Y directions from the position of the drone camera 414 to the position of the drone camera 411. Also, the virtual object 431 displayed in the window 413 similarly moves in the x and y directions (the depth direction of the window). Here, since the drone camera 411 moves in the X direction together with the virtual object 421 and moves far away from the virtual objects 422 and 423 in the virtual space in the Y direction, the virtual objects 432 and 433 in the window 413 are displayed smaller than in FIG. 4(f). Also, the positions of the virtual objects 432 and 433 are changing within the display area of the window 413. However, since the virtual object 421 has a fixed positional relationship with the drone camera 411, its position and size do not change within the display area of the window.

[0104] According to the third embodiment, the user can three-dimensionally move a virtual object placed in a position that is difficult to view from the user's position by changing the position and orientation of the drone camera having a fixed positional relationship with the virtual object.

[0105] In each of the above-described embodiments, an example of an image representing a virtual space (VR space) composed of virtual objects has been given, but it can also be implemented for an image representing a real space captured by an imaging device mounted on an HMD, with a virtual object superimposed thereon, representing a virtual space (MR space).

[0106] <Modification Example> Note that in the first to third embodiments, the drone camera and rays have been described, but it is not limited thereto. Modification examples of the drone camera and rays are shown below.

[0107] FIG. 9(a) is a diagram showing an arrow 911 indicating the position and orientation of a virtual viewpoint instead of the drone camera, and displaying a pointer instead of a ray. In FIG. 9(a), a pointer 941 indicating the position instructed by the controller 120 by the user 401 and a pointer 942 indicating the position instructed by the controller 120 by the user 401 via the window 413 are displayed.

[0108] FIG. 9(b) shows the drone camera 411, a ray 443 indicating the position instructed by the controller 120 by the user 401, and a pointer 942 indicating the position instructed by the controller 120 by the user 401 via the window 413. Thus, the displays indicating the position instructed by the controller and the position instructed by the controller via the window may be different. For example, a pointer indicating the position instructed by the controller 120 by the user 401 and a ray indicating the position instructed by the controller 120 by the user 401 via the window 413 may be displayed.

[0109] FIG. 9(c) is a diagram showing a frame 912 indicating a display area to be displayed in the window instead of the drone camera. Further, when the virtual object 421 is indirectly selected by selecting the virtual object 431 in the window 413 with the controller 120, the unselected virtual objects 922 and 923 are displayed transparently to emphasize that the virtual object 421 has been selected. Note that the virtual objects 932 and 933 in the window 413 corresponding to the virtual objects 922 and 923 are also displayed transparently.

[0110] Note that not only the controller but also the user's operations may be received based on the position and posture of the user's hand. FIG. 9(d) is a diagram assuming a scene in which the user 401's hand indicates the virtual object 431 by hand tracking without using the controller 120. The user 401 selects the virtual object by pointing the hand in the instruction direction without holding the controller. Here, the user's hand may be recognized by the imaging unit 202 of the HMD 100 or may be recognized by a camera arranged at a predetermined position (not shown).

[0111] Note that the drone camera may be rotated by the user's operation. In that case, the image displayed in the virtual window rotates according to the rotation of the drone camera.

[0112] <Other Embodiments> Note that the present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and a computer (or a control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0113] As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the present invention. Some of the above-described embodiments may be appropriately combined.

[0114] Note that each functional unit in each of the above embodiments (each modification example) may be individual hardware, or may not be. The functions of two or more functional units may be realized by common hardware. Each of the multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Also, each functional unit may be realized by hardware such as an ASIC, FPGA, or DSP, or may not be. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. And the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0115] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

[0116] [Configuration 1] Display means for displaying a virtual display that displays viewpoint information indicating the position and orientation of a virtual viewpoint and an image of the virtual object as viewed from the virtual viewpoint on a space in which the virtual object is arranged; Control means for controlling to select the virtual object arranged on the space via the virtual display in response to an operation on the virtual display; An information processing apparatus characterized by comprising:

[0117] [Configuration 2] The control means controls to move the virtual object arranged on the space via the virtual display in response to an operation on the virtual display to move the virtual object. The information processing apparatus according to Configuration 1, characterized by the above.

[0118] [Configuration 3] It further has acquisition means for acquiring the position or posture of the operation means, Based on the position or posture of the operation means acquired by the acquisition means, the display means displays the instruction position indicated by the operation means. The information processing apparatus according to Configuration 1 or 2, characterized in that.

[0119] [Configuration 4] The operation means is the user's hand or a controller. The information processing apparatus according to Configuration 3, characterized in that.

[0120] [Configuration 5] The instruction position is a pointer, When the operation means indicates within the virtual display, the display means displays the pointer both within the virtual display and at the position indicated via the virtual display. The information processing apparatus according to Configuration 3 or 4, characterized in that.

[0121] [Configuration 6] The instruction position is a virtual ray, When the operation means indicates within the virtual display, the display means displays the ray from the operation means towards the virtual display and the ray from the virtual viewpoint towards the position indicated via the virtual display. The information processing apparatus according to Configuration 3 or 4, characterized in that.

[0122] [Configuration 7] The display means displays a virtual camera as the viewpoint information, and the position and posture of the virtual camera indicate the position and posture of the virtual viewpoint. The information processing apparatus according to any one of Configurations 1 to 6, characterized in that.

[0123] [Configuration 8] When the control means selects the virtual object, it controls to notify the user that the virtual object is being selected. The information processing apparatus according to any one of Configurations 1 to 7, characterized in that.

[0124] [Configuration 9] When the control means moves the virtual object in the virtual display and makes it approach the frame of the image of the virtual object viewed from the virtual viewpoint, it controls to move the virtual viewpoint. The information processing apparatus according to any one of Configurations 2 to 8, characterized in that.

[0125] [Configuration 10] When the user moves the virtual object in the virtual display, the control means controls to move the virtual viewpoint so as to maintain the positional relationship between the virtual viewpoint and the virtual object being moved. The information processing apparatus according to any one of Configurations 2 to 9, characterized in that.

[0126] [Configuration 11] It further has a second acquisition means for acquiring an imaging image that images the real space. The display means displays the viewpoint information and the virtual display on the imaging image acquired by the second acquisition means. The information processing apparatus according to any one of Configurations 1 to 10, characterized in that.

[0127] [Configuration 12] In response to an operation by the user to change the viewing angle of the virtual viewpoint, the display means changes the viewing angle of the image and displays it on the virtual display. The information processing apparatus according to any one of Configurations 1 to 11, characterized in that.

[0128] [Control method] A display step of displaying a virtual display that displays viewpoint information indicating the position and orientation of a virtual viewpoint and an image of the virtual object viewed from the virtual viewpoint on a space where the virtual object is arranged. A control step of controlling to select the virtual object arranged in the space via the virtual display in response to an operation on the virtual display. A control method for an information processing apparatus, characterized by the above.

[0129] [Program] A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 12.

[0130] [System] A display device that displays a virtual display that displays viewpoint information indicating the position and orientation of a virtual viewpoint and an image of the virtual object viewed from the virtual viewpoint on a space where the virtual object is arranged. A control device that controls to select the virtual object arranged in the space via the virtual display in response to an operation on the virtual display. An information processing system, characterized by comprising the above.

Claims

1. Display means for displaying a virtual display that displays viewpoint information indicating the position and orientation of a virtual viewpoint and an image of the virtual object viewed from the virtual viewpoint on a space in which the virtual object is arranged; Control means for controlling to select the virtual object arranged on the space via the virtual display in response to an operation on the virtual display; An information processing apparatus comprising the same.

2. The control means controls to move the virtual object arranged on the space via the virtual display in response to an operation on the virtual display for moving the virtual object. The information processing apparatus according to claim 1, characterized in that.

3. Further comprising acquisition means for acquiring the position or orientation of the operation means, The display means displays an instruction position indicated by the operation means based on the position or orientation of the operation means acquired by the acquisition means. The information processing apparatus according to claim 1 or 2, characterized in that.

4. The operation means is a user's hand or a controller. The information processing apparatus according to claim 3, characterized in that.

5. The instruction position is a pointer, When the operation means indicates within the virtual display, the display means displays the pointer in the virtual display and at a position indicated via the virtual display. The information processing apparatus according to claim 3, characterized in that.

6. The instruction position is a virtual ray, When the display means is instructed within the virtual display by the operation means, the display means displays a light ray from the operation means toward the virtual display and a light ray from the virtual viewpoint toward the position indicated through the virtual display. The information processing apparatus according to claim 3, characterized in that.

7. The display means displays a virtual camera as the viewpoint information, and the position and orientation of the virtual camera indicate the position and orientation of the virtual viewpoint. The information processing apparatus according to claim 1, characterized in that.

8. When the control means selects the virtual object, the control means controls to notify the user that the virtual object is being selected. The information processing apparatus according to claim 1, characterized in that.

9. When the control means moves the virtual object within the virtual display and makes it close to the frame of the image of the virtual object viewed from the virtual viewpoint, the control means controls to move the virtual viewpoint. The information processing apparatus according to claim 2, characterized in that.

10. When the user moves the virtual object within the virtual display, the control means controls to move the virtual viewpoint so as to maintain the positional relationship between the virtual viewpoint and the virtual object being moved. The information processing apparatus according to claim 2, characterized in that.

11. The apparatus further includes a second acquisition means for acquiring an imaging image that images the real space. The display means displays the viewpoint information and the virtual display on the imaging image acquired by the second acquisition means. The information processing apparatus according to claim 1, characterized in that.

12. In response to an operation by the user to change the viewing angle of the virtual viewpoint, the display means changes the viewing angle of the image and displays it on the virtual display. The information processing apparatus according to claim 1, characterized in that...

13. A display step of displaying a virtual display that displays viewpoint information indicating the position and orientation of a virtual viewpoint and an image of the virtual object as seen from the virtual viewpoint on the space where the virtual object is arranged; A control step of controlling to select the virtual object arranged on the space via the virtual display in response to an operation on the virtual display; having... A control method for an information processing apparatus, characterized in that...

14. A program for causing a computer to function as each means of the information processing apparatus according to claim 1.

15. A display device that displays a virtual display that displays viewpoint information indicating the position and orientation of a virtual viewpoint and an image of the virtual object as seen from the virtual viewpoint on the space where the virtual object is arranged; A control device that controls to select the virtual object arranged on the space via the virtual display in response to an operation on the virtual display; An information processing system, characterized by comprising...

Images