Information processing device, control method for information processing device, and program

Abstract

The objective is to provide an information processing device, a control method for the information processing device, and a program that can stably move virtual objects. [Solution] The HMD100, which is an information processing device, comprises a generation means (control unit 101) for generating a virtual image, a display control means (control unit 101) for controlling the display of the virtual image, a first acquisition means (imaging unit 104) for acquiring first information relating to a first part of the user viewing the virtual image, a second acquisition means (sensor unit 105) for acquiring second information relating to a second part of the user, and a processing means (control unit 101) for executing a first movement process for moving a virtual object based on the first information and a second movement process for moving a virtual object based on the second information, and for switching between the first movement process and the second movement process depending on the acquisition status of the first information or the acquisition status of the second information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] In recent years, technologies related to XR (Extended Reality) such as VR (Virtual Reality), MR (Mixed Reality), and AR (Augmented Reality) are known. In XR, an HMD (Head Mounted Display) is used. Further, in video see-through using an HMD for MR, a video in which virtual objects by CG (Computer Graphics) are superimposed on a video of the real space imaged by a camera (imaging unit) mounted on the HMD is generated. Then, a user wearing the HMD can operate as if actually touching the virtual objects included in this generated video and moving them with the user's own hand. In this case, it is necessary to detect operation means such as the user's hand or a controller held by the user's hand as a reference for moving the virtual object. When the operation means cannot be detected, an alternative means to replace the operation means may be presented to the user, or the user may be prompted to change the movement of their own hand. For example, Patent Document 1 discloses a configuration for notifying the cause of an event where an operation object (virtual object) cannot be moved. Further, Patent Document 2 discloses a configuration for changing the operation state of system processing when an object existing within a predetermined detection range is not detected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

[0004] Here, for example, consider a scenario where, while a user is grasping and moving a virtual object with their hand, the HMD loses the ability to detect the user's hand due to hand tremors, changes in the camera's field of view, etc., causing the virtual object to be left behind without following the hand. In the configuration disclosed in Patent Document 1, it is possible to notify the user of the cause of the event that prevents the virtual object from moving, but each time such notification occurs, the user may have to go and grab the left-behind virtual object again. In the configuration disclosed in Patent Document 2, it is possible to keep the virtual object following by changing the operating state of the system processing, but the user cannot control the change in the operating state of the system processing. Therefore, it may be difficult for the user to interrupt or resume the movement operation of the virtual object at the timing they intend.

[0005] This invention has been made in view of the above-mentioned problems. The object of this invention is to provide an information processing device, a control method for the information processing device, and a program that can stably move virtual objects. [Means for solving the problem]

[0006] To achieve the above objective, the information processing apparatus of the present invention is characterized by comprising: generation means for generating a virtual image of a virtual space including a movable virtual object; display control means for controlling the display of the virtual image; first acquisition means for acquiring first information relating to a first part, which is a predetermined part of a user viewing the virtual image displayed by the control of the display control means; second acquisition means for acquiring second information relating to a second part, which is a predetermined part of the user different from the first part; and processing means for executing a first movement process for moving the virtual object based on the first information acquired by the first acquisition means, and a second movement process for moving the virtual object based on the second information acquired by the second acquisition means, and for switching between the first movement process and the second movement process depending on the acquisition state of the first information by the first acquisition means or the acquisition state of the second information by the second acquisition means. [Effects of the Invention]

[0007] According to the present invention, when moving a virtual object, the movement can be performed stably. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the hardware configuration of an information processing device according to the first embodiment. [Figure 2A] This diagram shows the chronological changes in the mixed reality image displayed on the HMD's display. [Figure 2B] This diagram shows the chronological changes in the mixed reality image displayed on the HMD's display. [Figure 2C] This diagram shows the chronological changes in the mixed reality image displayed on the HMD's display. [Figure 2D] This diagram shows the chronological changes in the mixed reality image displayed on the HMD's display. [Figure 2E] This diagram shows the chronological changes in the mixed reality image displayed on the HMD's display. [Figure 2F]It is a diagram showing the temporal change of the composite reality image displayed on the display unit of the HMD in order. [Figure 3A] It is an x-y plane view and a y-z plane view corresponding to FIG. 2A. [Figure 3B] It is an x-y plane view and a y-z plane view corresponding to FIG. 2B. [Figure 3C] It is an x-y plane view and a y-z plane view corresponding to FIG. 2C. [Figure 3D] It is an x-y plane view and a y-z plane view corresponding to FIG. 2D. [Figure 3E] It is an x-y plane view and a y-z plane view corresponding to FIG. 2E. [Figure 3F] It is an x-y plane view and a y-z plane view corresponding to FIG. 2F. [Figure 4A] It is a diagram showing the temporal change of the information stored in the RAM of the HMD in order. [Figure 4B] It is a diagram showing the temporal change of the information stored in the RAM of the HMD in order. [Figure 4C] It is a diagram showing the temporal change of the information stored in the RAM of the HMD in order. [Figure 4D] It is a diagram showing the temporal change of the information stored in the RAM of the HMD in order. [Figure 4E] It is a diagram showing the temporal change of the information stored in the RAM of the HMD in order. [Figure 5] It is a timing chart showing the execution timing of the process executed by the HMD. [Figure 6] It is a flowchart showing the process executed by the HMD. [Figure 7] It is a flowchart showing the detailed process in step S604, which is a subroutine of the flowchart shown in FIG. 6. [Figure 8] It is a flowchart showing the detailed process in step S606, which is a subroutine of the flowchart shown in FIG. 6. [Figure 9] It is a flowchart showing the detailed process in step S607, which is a subroutine of the flowchart shown in FIG. 6. [Figure 10] It is a flowchart showing the process (selection object determination process) executed by the HMD according to the second embodiment. [Figure 11] It is a flowchart showing the process (moving means determination process) executed by the HMD.

Mode for Carrying Out the Invention

[0009] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can exhibit the same function. Also, an arbitrary component may be added. Further, any two or more configurations (features) in each embodiment can be combined.

[0010] <First Embodiment> Hereinafter, the first embodiment will be described with reference to FIGS. 1 to 9. FIG. 1 is a block diagram showing an example of the hardware configuration when an information processing apparatus according to the first embodiment is applied to a head-mounted display (HMD). Note that, as an apparatus to which the information processing apparatus can be applied, in this embodiment, it is an HMD, but it is not limited thereto. For example, the information processing apparatus can also be applied to other image-displayable apparatuses such as a handheld display, a notebook personal computer, a tablet terminal, a smartphone, and a digital camera. Further, the information processing apparatus can also be applied to a desktop personal computer. In this case, the desktop personal computer is used in a state of being communicably connected to the image-displayable apparatus. As shown in FIG. 1, the HMD 100 has a control unit 101, a ROM 102, a RAM 103, an imaging unit 104, a sensor unit 105, and a display unit 106, and these are connected to each other so as to be communicable via a system bus 107.

[0011] The control unit 101 is a computer having at least one processor such as a CPU, MPU, or GPU. The control unit 101 also functions as a generation means for generating virtual images of a virtual space (generation process). The virtual image (virtual space) includes, for example, virtual objects that can be moved within the virtual image. As described later, the virtual object in this embodiment is a cube, but it is not limited to this, and may be a sphere, for example. The ROM 102 stores various programs, etc. These programs are not particularly limited, and include, for example, programs that cause the control unit 101 to execute each process (control method of the information processing device), which will be described later. When the main power supply (not shown) of the HMD 100 is turned on, the control unit 101 reads the program from the ROM 102 and starts controlling various hardware connected via the system bus 107. The ROM 102 is not particularly limited, and for example, it is composed of electrically erasable and recordable non-volatile memory such as flash memory. The RAM 103 is used as a work area for programs executed by the control unit 101. The RAM 103 is not particularly limited and can be composed of, for example, a volatile memory having semiconductor elements such as DRAM.

[0012] The imaging unit 104 includes, for example, an optical lens unit, an optical system for controlling aperture, zoom, focus, etc., and an image sensor for converting light (image) introduced through the optical lens unit into an electrical image signal. The image sensor is not particularly limited, and for example, a CMOS image sensor (CMOS image sensor) with a CMOS, a CCD image sensor (CCD image sensor) with a CCD, etc., can be used. The imaging unit 104 is also a stereo camera, consisting of two cameras positioned on the left and right of the HMD 100. The color images captured by these stereo cameras are output as video signals to the system bus 107. The video signals are then processed by the control unit 101 and stored in the RAM 103 as data for the left and right background images, i.e., real images. The control unit 101 can then generate a composite reality image by combining the real image and the virtual image. The sensor unit 105 is composed of various sensors. The sensors are not particularly limited and include, for example, an accelerometer for detecting changes in the position of the HMD 100, a gyroscope for detecting changes in the attitude of the HMD 100, and a geomagnetic sensor for detecting the orientation of the HMD 100. The display unit 106 consists of two display devices, such as displays, positioned on the left and right sides of the HMD 100, and displays the augmented reality image generated by the control unit 101. This enables stereoscopic viewing of the augmented reality image. The control (display control process) for displaying the augmented reality image on the display unit 106 is performed by the control unit 101. Thus, in this embodiment, the control unit 101 also functions as a display control means for controlling the display of the augmented reality image on the display unit 106. In addition, the HMD 100 may have a separate component that functions as a display control means from the control unit 101.

[0013] Figures 2A to 2F sequentially show the changes in the mixed reality image displayed on the HMD's display unit over time. The display unit 106 displays the solid lines in Figures 2A to 2F, while the dashed lines are not displayed. In this embodiment, the mixed reality image shown in Figures 2A to 2F is assigned a three-dimensional coordinate system (left-handed coordinate system) consisting of x, y, and z axes, with all coordinate units in "cm". Furthermore, in this embodiment, unless otherwise specified, the coordinate system is the reference coordinate system for the entire virtual space, i.e., the world space coordinate system. If the coordinate system is based on a specific object, i.e., a local space coordinate system, the specific local space coordinate system is indicated.

[0014] The mixed reality image 200 shown in Figure 2A is an image processed by the control unit 101 at time t=t50, which will be described later. The mixed reality image 210 shown in Figure 2B is an image processed by the control unit 101 at time t=t51, which will be described later. The mixed reality image 220 shown in Figure 2C is an image processed by the control unit 101 at time t=t52, which will be described later. The mixed reality image 230 shown in Figure 2D is an image processed by the control unit 101 at time t=t53, which will be described later. The mixed reality image 240 shown in Figure 2E is an image processed by the control unit 101 at time t=t54, which will be described later. The mixed reality image 250 shown in Figure 2F is an image processed by the control unit 101 at time t=t56, which will be described later. Furthermore, mixed reality images 200 to 250 include a virtual object 201. The virtual object 201 is a cube generated based on CG (Computer Graphics) data such as video footage or 3D model data. The virtual object 201, which is a cube, has eight vertices. In the mixed reality image 200, that is, at time t=t50, points p202 to p209 are displayed as the eight vertices, with point p202=(x1,y1,z2)=(3,5,12). Also, point p203=(x1,y1,z1)=(3,5,5). Point p204=(x1,y2,z1)=(3,15,5). Point p205=(x1,y2,z2)=(3,15,12). Point p206=(x2,y2,z1)=(9,15,5). Point p207=(x2,y2,z2)=(9,15,12). Point p208 = (x2, y1, z2) = (9, 5, 12). Point p209 = (x2, y1, z1) = (9, 5, 5).

[0015] Mixed reality images 210 to 250 include the hand 211 of a user who is wearing the HMD 100 on their head and viewing the mixed reality images 210 to 250. The hand 211 includes a point p213 indicating the tip of the thumb and a point p214 indicating the tip of the index finger. In mixed reality image 210, i.e., at time t=t51, point p213=(x3,y3,z3)=(3.5,13.5,10.5). Also, point p214=(x5,y5,z5)=(4.5,14.5,11.5). The hand 211 also includes a point p215 calculated based on points p213 and p214. Point p215 is the reference point for finger movement when moving the hand 211, and in this embodiment, it is the midpoint between points p213 and p214. In the mixed reality image 210, point p215 = (x4, y4, z4) = (4, 14, 11). In this embodiment, the finger movement reference point is the midpoint between the tip of the thumb and the tip of the index finger, but it is not limited to this, and for example, it may be either the tip of the thumb or the tip of the index finger. In this way, the HMD 100 can acquire positional information regarding the position of the user's fingers captured by the imaging unit 104 (first acquisition step). In this embodiment, the positional information of the fingers (first part, which is a predetermined part of the user) is used as first information, and the imaging unit 104 functions as a first acquisition means to acquire this first information. Positional information of the fingers can also be acquired by hand tracking. When the thumb and index finger are touching, for example, it forms a pinching shape for picking up a virtual object. When the thumb and index finger separate from the pinching shape, it forms a separated shape different from the pinching shape.

[0016] Point p222 of the virtual object 201 in the mixed reality image 220 shown in Figure 2C corresponds to point p202 at time t=t52, with (x6,y6,z6)=(12,22,12). Point p225 is the finger movement reference point at time t=t52, with (x7,y7,z7)=(4,31,11). Point p232 of the virtual object 201 in the mixed reality image 230 shown in Figure 2D corresponds to point p202 at time t=t53, with (x8,y8,z8)=(5,29,11). Point p242 of the virtual object 201 in the mixed reality image 240 shown in Figure 2E corresponds to point p202 at time t=t54, with (x9,y9,z9)=(approximately 8,20,approximately 18). In the mixed reality image 250 shown in Figure 2F, point p253 on hand 211 represents the tip of the thumb at time t=t56, with (x10,y10,z10)=(1,16,2). Also, point p254 represents the tip of the index finger at time t=t56, with (x11,y11,z11)=(3,30,15).

[0017] Figure 3A is the xy-plane and yz-plane corresponding to Figure 2A. Figure 3B is the xy-plane and yz-plane corresponding to Figure 2B. Figure 3C is the xy-plane and yz-plane corresponding to Figure 2C. Figure 3D is the xy-plane and yz-plane corresponding to Figure 2D. Figure 3E is the xy-plane and yz-plane corresponding to Figure 2E. Figure 3F is the xy-plane and yz-plane corresponding to Figure 2F. Coordinate 301 in Figures 3A to 3F corresponds to the x-coordinate (x1) in Figure 2. Also, coordinate 302 corresponds to the x-coordinate (x2) in Figure 2. Coordinate 311 corresponds to the x-coordinate (x3) in Figure 2. Coordinate 312 corresponds to the x-coordinate (x4) in Figure 2. Coordinate 313 corresponds to the x-coordinate (x5) in Figure 2. Coordinate 321 corresponds to the x-coordinate (x6) in Figure 2. Coordinate 322 corresponds to the x-coordinate (x7) in Figure 2. Coordinate 331 corresponds to the x-coordinate (x8) in Figure 2. Coordinate 341 corresponds to the x-coordinate (x9) in Figure 2. Coordinate 351 corresponds to the x-coordinate (x10) in Figure 2. Coordinate 352 corresponds to the x-coordinate (x11) in Figure 2.

[0018] In Figures 3A to 3F, coordinate 303 corresponds to the y-coordinate (y1) in Figure 2. Also, coordinate 304 corresponds to the y-coordinate (y2) in Figure 2. Coordinate 314 corresponds to the y-coordinate (y3) in Figure 2. Coordinate 315 corresponds to the y-coordinate (y4) in Figure 2. Coordinate 316 corresponds to the y-coordinate (y5) in Figure 2. Coordinate 323 corresponds to the y-coordinate (y6) in Figure 2. Coordinate 324 corresponds to the y-coordinate (y7) in Figure 2. Coordinate 332 corresponds to the y-coordinate (y8) in Figure 2. Coordinate 342 corresponds to the y-coordinate (y9) in Figure 2. Coordinate 353 corresponds to the y-coordinate (y10) in Figure 2. Coordinate 354 corresponds to the y-coordinate (y11) in Figure 2.

[0019] In Figures 3A to 3F, coordinate 305 corresponds to the z-coordinate (z1). Coordinate 306 corresponds to the z-coordinate (z2). Coordinate 317 corresponds to the z-coordinate (z3). Coordinate 318 corresponds to the z-coordinate (z4). Coordinate 319 corresponds to the z-coordinate (z5). Coordinate 325 corresponds to the z-coordinate (z6). Coordinate 326 corresponds to the z-coordinate (z7). Coordinate 333 corresponds to the z-coordinate (z8). Coordinate 343 corresponds to the z-coordinate (z9). Coordinate 355 corresponds to the z-coordinate (z10). Coordinate 356 corresponds to the z-coordinate (z11). Note that the x-coordinates are x1=3, x2=5, x3=3.5, x4=4, x5=4.5, x6=12, x7=4, x8=5, x9=approximately 8, x10=1, and x11=3. Furthermore, the y-coordinates are y1=5, y2=15, y3=13.5, y4=14, y5=14.5, y6=22, y7=31, y8=29, y9=20, y10=16, and y11=40. The z-coordinates are z1=5, z2=12, z3=10.5, z4=11, z5=11.5, z6=11, z7=12, z8=11, z9=approximately 18, z10=2, and z11=15.

[0020] Figures 4A to 4E sequentially show the changes in information stored in the HMD's RAM. The selected object management tables 400A shown in Figure 4A to 400E shown in Figure 4E are selected object management tables that manage information on virtual objects 201 selected as targets for movement on a mixed reality image (virtual image). Selected object management table 400A is the table at time t=t51. Selected object management table 400B is the table at time t=t52. Selected object management table 400C is the table at time t=t53. Selected object management table 400D is the table at time t=t54. Selected object management table 400E is the table at time t=t56. The selected object management tables 400A to 400E include, but are not limited to, ID 401, finger movement reference point 402, head reference position information 403, and head movement flag 404 as items. ID401 is identification information for identifying the virtual object 201 selected as the target for movement on the mixed reality image. The finger movement reference point 402 is the coordinate of the reference point when moving the virtual object 201 together with the hand 211 on the mixed reality image. As mentioned above, in this embodiment, the finger movement reference point 402 is the midpoint between the tip of the thumb and the tip of the index finger. The HMD 100 can then execute a first movement process to move the virtual object 201 on the mixed reality image based on the position information of this midpoint, i.e., the position information of the fingers (first information). Note that data 402a in Figure 4A is the coordinate of the finger movement reference point 402 when ID401 is "1". Similarly, data 402b in Figure 4B is the coordinate of the finger movement reference point 402 when ID401 is "1".

[0021] The head reference position information 403 is the coordinate of each vertex of the virtual object 201 in a local spatial coordinate system (head local spatial coordinates) based on a predetermined position of the user's head, i.e., with the origin. In Figure 4B, data 403b is the coordinate of the head reference position information 403 with ID 401 "1". Similarly, in Figure 4D, data 403d is the coordinate of the head reference position information 403 with ID 401 "1". The control unit 101 calculates the coordinate of each point in the local spatial coordinate system based on the information acquired by the sensor unit 105 and the world spatial coordinate information of the virtual object 201. The coordinate of each vertex of the virtual object 201 is the coordinate of the period prior to the time when the coordinate is calculated. This period will be described later with reference to Figure 5. The HMD 100 can also acquire position information related to a predetermined position of the user's head (second acquisition step). In this embodiment, the position information of the head (a predetermined part different from the first part of the user, which is the second part) is used as second information, and the sensor unit 105 functions as a second acquisition means for acquiring this second information. Based on this head position information (second information), the HMD 100 can then perform a second movement process to move the virtual object 201 on the mixed reality image. Therefore, it is not necessary for the head reference position information 403 to be acquired until the second movement process is performed.

[0022] The head movement flag 404 is flag data for switching between the first movement process and the second movement process. The process (processing step) for switching between the first movement process and the second movement process is executed by the control unit 101. Thus, in this embodiment, the control unit 101 also functions as a processing means for switching between the first movement process and the second movement process. In the HMD 100, the part that functions as a processing means may be provided separately from the control unit 101. When the first movement process is executed, "False" is entered into the head movement flag 404. When the second movement process is executed, "True" is entered into the head movement flag 404. In Figure 4C, data 404c is the flag data for the head movement flag 404 with ID 401 "1". In addition, the selected object management tables 400A to 400E may include selected object element data as an item. The selected object element data is the column when the virtual object 201 is registered in the selected object management table.

[0023] Figure 5 is a timing chart showing the execution timing of processes performed on the HMD. One cycle in the timing chart shown in Figure 5 is the period during which the program based on the flowchart shown in Figure 6 is repeatedly executed, and can be, for example, the image sampling interval during which images are sampled. Times t50 to t56 are the times when various processes were executed based on the image sampling interval. Times t50 to t56 will be described later.

[0024] As mentioned above, the HMD100 can perform a first movement process based on finger position information (first information), that is, while acquiring finger position information. Furthermore, the HMD100 can perform a second movement process based on head position information (second information), that is, while acquiring head position information. In the HMD100, the first movement process takes precedence over the second movement process. For example, if the level of acquisition of first information decreases during the execution of the first movement process, the first movement process may be interrupted. Therefore, the HMD100 is configured to allow switching to the second movement process in such cases. This configuration and operation will be explained below. In this embodiment, "moving based on a reference point" refers to movement using coordinate space transformation with APIs such as Unity. Such movement is publicly known. Furthermore, movement using coordinate space implemented in Unity, for example, when following a hand, uses the local coordinates of a hand tracking object as a base to map the local coordinates of a virtual object to world space coordinates. When following a head, the world coordinates of a virtual object are calculated based on the camera coordinate space for movement.

[0025] Figure 6 is a flowchart showing the processes executed by the HMD. The program based on the flowchart shown in Figure 6 is executed by the control unit 101 of the HMD 100 controlling other hardware of the HMD 100. The control unit 101 also repeats the processes of this flowchart according to the cycle described with reference to Figure 5. As shown in Figure 6, in step S601, the control unit 101 controls the imaging unit 104 to acquire a video signal. Then, the control unit 101 generates data for the left and right background images, i.e., real-world images, based on the video signal. This background image data is stored in RAM 103. At this time, the control unit 101 also generates a virtual image containing virtual objects and generates a composite reality image by combining the virtual image and the real-world image. This composite reality image data is also stored in RAM 103. The composite reality image is then displayed on the display unit 106.

[0026] In step S602, the control unit 101 detects the user's hand, the fingertips of each finger, and the joint points of the wrist from the left and right background image data held in RAM 103 in step S601, that is, it obtains finger position information. The method for detecting finger position is not particularly limited and includes methods using a trained model, methods using algorithms according to known rule bases, etc. Specifically, the hand position detection is performed step by step, detecting the region in the background image that contains the user's hand and then detecting the hand position again from that region. The detection result of the user's hand position, i.e., finger position information, is then held in RAM 103. If the detection result of the user's hand position cannot be obtained, information to that effect is held in RAM 103.

[0027] In step S603, the control unit 101 controls the sensor unit 105 to acquire the orientation information of the HMD 100 as position information of the user's head. The orientation information of the HMD 100 (head position information) is represented by three-dimensional coordinates in the world space coordinate system and orientation in terms of angle. Specifically, it is represented by the variables Position and Rotation. The orientation information of the HMD 100 is stored in the RAM 103.

[0028] In step S604, the control unit 101 performs a selection object determination process to determine the virtual object on the mixed reality image (virtual image) selected by the user as the virtual object to be moved. The control unit 101 registers the virtual object determined to be the virtual object to be moved in the selection object management table, as explained with reference to Figures 4A to 4E. The detailed processing in step S604 will be described later with reference to Figure 7.

[0029] In step S605, the control unit 101 determines whether or not information (element data) related to the virtual object exists in the selected object management table. If, based on the determination in step S604, the control unit 101 determines that an element exists in the selected object management table, the process proceeds to step S606. On the other hand, if, based on the determination in step S604, the control unit 101 determines that no element exists in the selected object management table, the process proceeds to step S608.

[0030] In step S606, the control unit 101 executes a movement means determination process to determine the means of movement when moving a virtual object registered in the selected object management table. The detailed processing in step S606 will be described later with reference to Figure 8.

[0031] In step S607, the control unit 101 executes a virtual object movement process to move the virtual object using the movement means determined in step S606. The details of the process in step S607 will be described later with reference to Figure 9.

[0032] In step S608, the control unit 101 generates augmented reality image data by superimposing the rendered virtual objects onto the background image data held in RAM 103 in step S601. Then, the control unit 101 controls the display unit 106 to display this augmented reality image. After step S608 is executed, the process ends.

[0033] Figure 7 is a flowchart showing the detailed processing in step S604, which is a subroutine of the flowchart shown in Figure 6. As shown in Figure 7, in step S701, the control unit 101 determines whether or not it was able to obtain the finger position information held in RAM 103 in step S602. If the control unit 101 determines that it was able to obtain the finger position information as a result of the determination in step S701, the process proceeds to step S702. On the other hand, if the control unit 101 determines that it was not able to obtain the finger position information as a result of the determination in step S701, the process ends. For example, the control unit 101 may determine that it was not able to obtain the finger position information at time t50 and from time t53 to t54, but that it was able to obtain the finger position information at time t51 to t52 and from time t55 to t56.

[0034] In step S702, the control unit 101 determines whether the distance between the thumb tip and the index fingertip is less than a predetermined threshold, based on the finger position information that it determined was obtained from the RAM 103 in step S701. If, as a result of the determination in step S702, the control unit 101 determines that the distance between the thumb tip and the index fingertip is less than the predetermined threshold, the process proceeds to step S703. On the other hand, if, as a result of the determination in step S702, the control unit 101 determines that the distance between the thumb tip and the index fingertip is not less than the predetermined threshold, the process proceeds to step S707. Here, as an example, the predetermined threshold is set to "2 cm". At time t51, as mentioned above, the coordinates of the thumb tip and the index fingertip are point p213 (3.5, 13.5, 10.5) and point p214 (4.5, 14.5, 11.5), respectively. In this case, the distance between the thumb tip and the index fingertip is 1.5 cm (<2 cm). Therefore, the control unit 101 determines that the distance between the thumb tip and the index finger tip at time t51 is less than a predetermined threshold, and the process proceeds to step S703. At time t56, the coordinates of the thumb tip and the index finger tip are point p253=(1,16,2) and point p254=(3,30,15), respectively. In this case, the distance between the thumb tip and the index finger tip is approximately 19.2cm (>2cm). Therefore, the control unit 101 determines that the distance between the thumb tip and the index finger tip at time t56 is not less than a predetermined threshold, and the process proceeds to step S707.

[0035] In step S703, the control unit 101 calculates the finger movement reference point based on the finger position information. This calculation result is stored in the RAM 103. As mentioned above, for example, the finger movement reference point at time t51 is point p215(4,14,11).

[0036] In step S704, the control unit 101 determines whether the finger movement reference point calculated in step S703 overlaps with a virtual object on the composite reality image. This determination is made, for example, based on whether the virtual object contains the finger movement reference point (coordinates). If the virtual object contains the finger movement reference point, it is determined that the finger movement reference point overlaps with the virtual object. If the virtual object does not contain the finger movement reference point, it is determined that the finger movement reference point does not overlap with the virtual object. Also, this determination is executed for all virtual objects arranged in the composite reality image. And as a result of the determination in step S704, if the control unit 101 determines that the finger movement reference point overlaps with the virtual object, the process proceeds to step S705. On the other hand, as a result of the determination in step S704, if the control unit 101 determines that the finger movement reference point does not overlap with the virtual object, the process proceeds to step S709. For example, at time t51, the minimum and maximum values of the coordinates of points p202 to p209 of the virtual object 201 are 3 < x < 9, 5 < y < 15, and 5 < z < 12, respectively. In this case, the control unit 101 can determine that point p215 is included in the virtual object 201 on the assumption that the coordinates of the finger movement reference point fall within the range from the minimum value to the maximum value of each coordinate of the virtual object 201.

[0037] In step S705, the control unit 101 determines whether the same virtual object (ID) has already been registered in the selection object management table. As a result of the determination in step S705, if the control unit 101 determines that the same virtual object has already been registered, the process proceeds to the process of step S709. On the other hand, as a result of the determination in step S705, if the control unit 101 determines that the same virtual object has not already been registered, the process proceeds to step S706.

[0038] In step S706, the control unit 101 adds the virtual object that was determined to overlap with the finger movement reference point in step S704 to the selected object management table. For example, the control unit 101 adds the virtual object 201 to the selected object element data in the selected object management table 400A. At this time, it registers the coordinates of point p215 calculated in step S703 to the finger movement reference point 402 corresponding to ID 401 "1" (see data 402a), and initializes the head movement flag 404 to "False". After step S706 is executed, the process ends.

[0039] In step S707, the control unit 101 determines whether or not an element exists in the selected object management table. If the determination in step S707 determines that an element exists in the selected object management table, the process proceeds to step S708. On the other hand, if the determination in step S707 determines that an element does not exist in the selected object management table, the process terminates.

[0040] In step S708, the control unit 101 deletes a virtual object registered in the selected object management table. For example, the control unit 101 deletes a virtual object with ID 401 and "1" that was registered at time t51 at time t56. After step S708 is executed, the process ends.

[0041] In step S709, the control unit 101 updates the head reference position information 403 in the selected object management table based on the finger position information. For example, suppose that at time t52, the head posture information is obtained as position [cm]=(-1,30,2) and angle [rad]=(0,0,0). In this case, the coordinates of each vertex of the virtual object 201 are the same as the result of applying the movement process at time t51, so point p212 is (3,5,12). Then, from this information, the head local space coordinates of point p212 at time t51 are calculated to be (4,-25,10). Similarly, the coordinates of the other vertices are calculated and the head reference position information 403 is updated (see data 403b in Figure 4B). Also, suppose that at time t54, the head posture information is obtained as position [cm]=(-1,30,2) and angle [rad]=(0,5,0). In this case, the coordinates of each vertex of the virtual object 201 remain as they were when the translation process was applied at time t53, so point p232 is (5,29,11). The local space coordinates of the head of point p232 are calculated to be (5.56,-10,16.5). After step S709 is executed, the process ends.

[0042] As described above, in the HMD100, the control unit 101 can determine the virtual object 201 to be moved on the mixed reality image based on the position information (first information) of the fingers. Thus, in this embodiment, the control unit 101 also functions as a determination means for determining the virtual object 201 to be moved. In addition, in the HMD100, the part that functions as a determination means may be provided separately from the control unit 101. Furthermore, in the HMD100, it is also possible to change the virtual object 201 that has been determined as the target of movement to another virtual object 201, that is, to cancel the decision. This makes it possible to make a desired virtual object 201 the target of movement.

[0043] Figure 8 is a flowchart showing the detailed processing in step S606, which is a subroutine of the flowchart shown in Figure 6. As shown in Figure 8, in step S801, the control unit 101 determines whether or not it was able to obtain the finger position information held in RAM 103 in step S602. If the control unit 101 determines that it was able to obtain the finger position information as a result of the determination in step S801, the process ends. On the other hand, if the control unit 101 determines that it was not able to obtain the finger position information as a result of the determination in step S801, the process proceeds to step S802.

[0044] In step S802, the control unit 101 updates the head movement flag 404 by inputting "True" for the virtual object registered in the selected object management table. As mentioned above, "True" is input to the head movement flag 404 when the second movement process is to be executed (see data 404c in Figure 4C). After step S608 is executed, the process ends.

[0045] Figure 9 is a flowchart showing the detailed processing in step S607, which is a subroutine of the flowchart shown in Figure 6. As shown in Figure 9, in step S901, the control unit 101 determines whether an element of the selected object management table is moving based on head position information, that is, whether the second movement process is being executed. This determination is made based on the head movement flag 404 of the selected object management table. For example, if the head movement flag 404 is "True", it is determined that the object is moving based on head position information, and if the head movement flag 404 is "False", it is determined that the object is not moving based on head position information. If, as a result of the determination in step S901, the control unit 101 determines that the object is moving based on head position information, the process proceeds to step S902. On the other hand, if, as a result of the determination in step S901, the control unit 101 determines that the object is not moving based on head position information, the process proceeds to step S903.

[0046] In step S902, the control unit 101 calculates (determines) the coordinates of the virtual object after movement (destination) based on the head reference position information 403 in the selected object management table. This calculation result, i.e., the calculated coordinates, is stored in RAM 103. The control unit 101 then moves the virtual object toward these calculated coordinates, that is, it can place the virtual object at the calculated coordinates. The method for calculating the coordinates after movement is not particularly limited. For example, first, the world space coordinates are calculated in the head attitude information at the calculation time so that they match the head local space coordinates of the previous cycle. As an example, let's consider the case where the coordinates of point p242 of virtual object 201 at time t54 are calculated. In this case, the coordinate point before movement is point p232(5,29,11) at time t53. The control unit 101 obtains (5,56,-10,16,5) as the head local space coordinates from the data 403d in the selected object management table 400D. At this time, the head's orientation information at time t54 is, as mentioned above, position [cm] = (-1, 30, 2) and angle [rad] = (0, 5, 0). Therefore, the control unit 101 can calculate the world space coordinates of point p242 at time t54 as (approximately 7.977, 20, approximately 17.9526). Alternatively, a reference point can be calculated, similar to the fingers, and the virtual object can be translated. In this case, if Unity is used, the coordinates of the virtual object in the camera coordinate space at time t53 are obtained. Then, at time t54, by setting the world space coordinates of the virtual object's coordinates relative to the camera coordinate space, it becomes possible to visually perceive that the virtual object has moved in accordance with the head position.

[0047] In step S903, the control unit 101 calculates the amount of movement (destination) of the virtual object based on the position information of the fingers. This calculation result is stored in RAM 103. For example, as mentioned above, in step S703 at time t52, the control unit 101 obtains point p225(4,31,11), which is the finger movement reference point, stored in RAM 103. The control unit 101 obtains point p215(4,14,11) as the finger movement reference point at time t51 from the selected object management table at the time step S903 is executed. Then, based on these obtained coordinates, the control unit 101 calculates the position coordinates of each vertex of the virtual object at time t52 as relative coordinates to the coordinates at time t51. The relative coordinates are obtained from point p225(4,31,11) - point p215(4,14,11), resulting in (0,17,0). As a result, the control unit 101 calculates the coordinates of the virtual object after translating it by the amount of (x,y,z)=(0,17,0) as the position coordinates of each vertex of the virtual object at time t52. This calculation result is stored in RAM 103. In this embodiment, the relative coordinates between the finger movement reference point of the previous cycle and the current finger movement reference point are used, but the embodiment is not limited to this. For example, the virtual object may be linked to a hand object in the virtual space that corresponds to the user's hand movements.

[0048] In step S904, the control unit 101 updates the finger movement reference point 402 for the virtual object registered in the selected object management table (see data 402b in Figure 4B). This update causes the virtual object to move.

[0049] As described above, in the HMD100, the control unit 101 can determine the degree to which finger position information has been acquired (acquisition status), that is, whether or not finger position information has been acquired. Thus, in this embodiment, the control unit 101 also functions as a means for determining the degree to which finger position information has been acquired. In the HMD100, the part that functions as a means for determining the degree may be provided separately from the control unit 101. If finger position information has been acquired, a first movement process based on the finger position information is executed. On the other hand, if finger position information has not been acquired, the head movement flag 404 is set to "True", head position information is acquired, and a second movement process based on the head position information is executed. In this case as well, the control unit 101 can determine the degree to which head position information has been acquired (acquisition status).

[0050] Furthermore, in the HMD100, the control unit 101 can switch between the first movement process and the second movement process depending on the degree to which finger position information or head position information is acquired. If, for example, the fingers move out of the detection range (sensor range) of the sensor unit 105 during the execution of the first movement process, and it is determined that the acquisition state of finger position information has deteriorated below a predetermined state, the system switches to the second movement process. This allows the virtual object to continue moving stably even if finger position information cannot be acquired while the user is moving the virtual object using their fingers. Note that if the amount of hand movement in a video frame is relatively large, i.e., if there is relatively large hand shake, it may also be determined that the acquisition state of finger position information has deteriorated below a predetermined state.

[0051] Furthermore, if, during the execution of the second movement process, it is determined that the head position information has changed excessively (e.g., the head has been shaken excessively), or that the acquisition state of the head position information has fallen below a predetermined state, the second movement process may be stopped. This prevents, for example, unnecessary head movements for the user. In addition, if, during the execution of the second movement process after the first movement process, it is determined that the acquisition state of the finger position information has risen above a predetermined state, the system may switch back to the first movement process. This allows the execution of the first movement process to be resumed. Since the first movement process was originally performed, it is considered that the first movement process tends to be more in line with the user's intentions than the second movement process. Also, upon switching back to the first movement process, a virtual object may be placed on the display unit 106 at a position based on the finger position information. This allows the virtual object to be placed at a position in line with the user's intentions by moving the finger back into the detection range of the sensor unit 105 if the finger moves out of the detection range of the sensor unit 105 due to user inattention or other circumstances not in line with the user's intentions.

[0052] Furthermore, the control unit 101 may make the texture (surface color) of the virtual object during the first movement process different from the texture of the virtual object during the second movement process. This allows the user to understand whether the movement of the virtual object is based on the position information of the fingers or the position information of the head. In this embodiment, the texture of the virtual object during the first movement process is set to the state shown in Figures 2B and 2C, and the texture of the virtual object during the second movement process is set to the state shown in Figures 2D and 2E. Alternatively, a message indicating whether the movement of the virtual object is based on the position information of the fingers or the position information of the head may be added instead of, or along with, a texture change. In addition, sound or vibration may be emitted to indicate whether the movement is based on the position information of the fingers or the position information of the head.

[0053] Furthermore, if the shape of the fingers is the pinch shape (first shape) described above, the control unit 101 can execute the first movement process based on the first information. Suppose that while this first movement process is being executed, the shape of the fingers changes from the pinch shape to the separated shape described above. If the control unit 101 is unable to acquire the first information without acquiring the separated shape, it can switch to the second movement process. If the pinch shape is acquired after switching to the second movement process, the control unit 101 can switch back to the first movement process. In this way, with the HMD 100, if the acquisition of the pinch shape fails and the pinch shape is acquired again, it can return to the first movement process. On the other hand, if the separated shape is acquired after switching to the second movement process, the control unit 101 terminates the second movement process without switching back to the first movement process. In this way, with the HMD 100, if the acquisition of the pinch shape fails and the separated shape is acquired, either the first movement process or the second movement process can be terminated. Furthermore, if a detached shape is acquired after switching to the second movement process, the control unit 101 places a virtual object at the finger position on the hidden section 106 and terminates the second movement process. In this way, with the HMD 100, if the acquisition of the thumb shape fails and a detached shape is acquired, the position where the detached shape was acquired can be assumed to be the position where the user wanted to place the virtual object.

[0054] <Second Embodiment> The second embodiment will be described below with reference to Figures 10 and 11, focusing on the differences from the previously described embodiment, and omitting explanations of similar matters. In this embodiment, motion information related to gesture movements is acquired as first information. Figure 10 is a flowchart of the process (selection object determination process) executed by the HMD according to the second embodiment. In the flowchart shown in Figure 10, unlike the flowchart shown in Figure 7, steps S1000 and S1001 are added instead of step S702. Also, in the flowchart shown in Figure 10, step S1002 is added between steps S703 and S704. Also, in the flowchart shown in Figure 10, step S1003 is added instead of step S707. As shown in Figure 10, if the control unit 101 determines, as a result of the determination in step S701, that it has been able to acquire the finger position information, the process proceeds to step S1000.

[0055] In step S1000, the control unit 101 recognizes a hand gesture. This recognition result is stored in the RAM 103. For example, the method for recognizing a hand gesture is not particularly limited, and one example is to use a learning model that takes background image data acquired from the imaging unit 104 as input and outputs a hand gesture. Alternatively, it may be a method based on the shape information of the fingers included in the finger position information. In this embodiment, the control unit 101 recognizes the shape of the hand 211 at time t51 as the gesture "grasp (holding motion)" (see Figure 2B). The control unit 101 also recognizes the shape of the hand 211 at time t56 as the gesture "open hand" (see Figure 2F). After step S1000 is executed, the process proceeds to step S1001.

[0056] In step S1001, similar to step S707, the control unit 101 determines whether or not an element exists in the selected object management table. If the determination in step S1001 determines that an element exists in the selected object management table, the process proceeds to step S703. On the other hand, if the determination in step S1001 determines that an element does not exist in the selected object management table, the process terminates.

[0057] In step S1002, following step S703, the control unit 101 determines whether the gesture recognized in step S1000 is "grab". If the control unit 101 determines in step S1002 that the gesture is "grab", the process proceeds to step S704 and the subsequent steps are executed in order. On the other hand, if the control unit 101 determines in step S1002 that the gesture is not "grab", the process proceeds to step S1003. In this embodiment, the control unit 101 can determine, for example, that the gesture is "grab" at time t51 (see Figure 2B).

[0058] In step S1003, the control unit 101 determines whether the gesture recognized in step S1001 is "paper". If the control unit 101 determines that the gesture is "paper" as a result of the determination in step S1003, the process proceeds to step S708. On the other hand, if the control unit 101 determines that the gesture is not "paper" as a result of the determination in step S1003, the process ends. In this embodiment, the control unit 101 can determine that the gesture is "paper" at time t56, for example (see Figure 2F).

[0059] As described above, in this embodiment, if the gesture is determined to be "grabbing," that is, if the gesture movement is a movement to grasp a virtual object, then that virtual object can be determined as the virtual object to be moved. In this way, the virtual object to be moved can be determined based on the gesture.

[0060] Figure 11 is a flowchart of the process (movement method determination process) performed by the HMD. Unlike the flowchart shown in Figure 8, steps S1101 to S1104 are added after step S801 in the flowchart shown in Figure 11. As shown in Figure 11, if the control unit 101 determines that it has been able to acquire the finger position information as a result of the determination in step S801, the process proceeds to step S1101.

[0061] In step S1101, the control unit 101 determines whether the gesture recognized in step S1001 is a "peck". If the control unit 101 determines that the gesture is a "peck", the process proceeds to step S802. On the other hand, if the control unit 101 determines that the gesture is not a "peck", the process proceeds to step S1102.

[0062] In step S1102, similar to step S901, the control unit 101 determines whether or not the unit is moving based on the head position information, that is, whether or not the second movement process is being executed. If, as a result of the determination in step S1102, the control unit 101 determines that the unit is moving based on the head position information, the process proceeds to step S1103. On the other hand, if, as a result of the determination in step S1102, the control unit 101 determines that the unit is not moving based on the head position information, the process ends.

[0063] In step S1103, the control unit 101 determines whether the gesture recognized in step S1001 is a "peace" gesture. If the control unit 101 determines that the gesture is a "peace" gesture as a result of the determination in step S1103, the process proceeds to step S1104. On the other hand, if the control unit 101 determines that the gesture is not a "peace" gesture as a result of the determination in step S1103, the process ends.

[0064] In step S1104, the control unit 101 updates the head movement flag 404 for the virtual object registered in the selected object management table by inputting "False". After step S1104 is executed, the process ends.

[0065] As described above, in this embodiment, when moving a virtual object, it is possible to determine whether to use finger position information or head position information, that is, the means of moving the virtual object can be determined by hand gestures. This allows the user to actively switch the means of moving the virtual object depending on the situation while moving the virtual object with their fingers. In this embodiment, the gestures given are "grab," "open hand," "poke," and "peace sign," but the embodiment is not limited to these.

[0066] While preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. The present invention provides a program that implements one or more functions of the embodiments described above to a system or device via a network or storage medium. It can also be implemented by one or more general-purpose processors (ASICs) of the computer of the system or device reading and executing the program. Furthermore, the present invention can also be implemented by a dedicated processor (e.g., an ASIC or FPGA) that implements one or more functions. Moreover, the present invention can also be implemented by a combination of a general-purpose processor and a dedicated processor. Here, "processor" refers to a processor in a broad sense and includes both general-purpose processors and dedicated processors. Furthermore, the process of implementing the present invention may be executed by only one processor, or by the cooperation of multiple processors located at physically separate locations. As first information, examples include finger position information and gesture information, but it is not limited to these, and it can also be, for example, leg position information. As second information, examples include head position information, but it is not limited to these, and it can also be, for example, eye movement information, i.e., gaze information.

[0067] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Configuration 1) A generation means for generating a virtual image of a virtual space containing movable virtual objects, A display control means that controls the display of the virtual image, A first acquisition means for acquiring first information relating to a first part, which is a predetermined part of the user that views the virtual image displayed by the control means of the display control means, A second acquisition means for acquiring second information relating to a second body part which is a predetermined body part different from the first body part of the user, An information processing apparatus characterized by comprising: a processing means that performs a first movement process for moving the virtual object based on the first information acquired by the first acquisition means, and a second movement process for moving the virtual object based on the second information acquired by the second acquisition means, and performs a process of switching between the first movement process and the second movement process according to the acquisition status of the first information by the first acquisition means or the acquisition status of the second information by the second acquisition means. (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that when the processing means executes the first movement process, it determines the destination of the virtual object in the virtual image based on the first information and moves the virtual object to the destination, and when the processing means executes the second movement process, it determines the destination of the virtual object in the virtual image based on the second information and places the virtual object at the destination. (Configuration 3) comprising a determination means for determining the acquisition status of the first information by the first acquisition means and the acquisition status of the second information by the second acquisition means, The information processing apparatus according to configuration 1 or 2, characterized in that the processing means performs a process of switching between the first movement process and the second movement process according to the determination result of the determination means. (Configuration 4) The information processing apparatus according to Configuration 3, wherein the processing means switches to the second movement process when the determination means determines that the acquisition state of the first information has fallen below a predetermined state during the execution of the first movement process. (Configuration 5) The information processing apparatus according to Configuration 3 or 4, characterized in that the processing means stops the second movement process if the determination means determines that the second information acquired by the second acquisition means has changed or that the acquisition state of the second information has deteriorated below a predetermined state during the execution of the second movement process. (Configuration 6) The information processing apparatus according to any one of Configurations 3 to 5, characterized in that the processing means switches to the first movement process when the determination means determines that the acquisition state of the first information has risen to a predetermined state or higher during the execution of the second movement process. (Configuration 7) The information processing apparatus according to Configuration 6, characterized in that, when the determination means determines that the acquisition state of the first information has risen to a predetermined state or higher during the execution of the second movement process, the processing means switches to the first movement process and places the virtual object at the position based on the first information. (Configuration 8) The first acquisition means acquires, as the first information, position information relating to the position of the user's fingers or motion information relating to the movement of gestures, The information processing device according to any one of configurations 1 to 7, characterized in that the second acquisition means acquires positional information relating to the position of the user's head as the second information. (Configuration 9) The information processing apparatus according to Configuration 8, characterized in that the processing means executes the first movement process with priority over the second movement process. (Configuration 10) The information processing apparatus according to Configuration 8 or 9, characterized in that it comprises a determination means for determining a virtual object to be moved in the first move process based on the first information prior to the first move process. (Configuration 11) The information processing apparatus according to Configuration 10, characterized in that, when the first information is location information, if the virtual object includes the coordinates contained in the location information, the determination means can determine the virtual object as the virtual object to be moved, and can also cancel the determination. (Configuration 12) The information processing apparatus according to Configuration 10 or 11, characterized in that, when the first information is motion information, if the gesture movement is a movement of grasping a virtual object, the determination means can determine the virtual object to be moved as the virtual object to be moved, and can also cancel the determination. (Configuration 13) The information processing apparatus according to any one of Configurations 1 to 12, characterized in that the generation means makes the texture of the virtual object during the first movement process different from the texture of the virtual object during the second movement process. (Configuration 14) The generation means generates a composite reality image by combining the virtual image and the real image, The information processing apparatus according to any one of configurations 1 to 13, characterized in that the display control means performs control to display the augmented reality image. (Configuration 15) The processing means executes the first movement process based on the first information acquired by the first acquisition means when the shape of the first part is the first shape. An information processing device according to any one of configurations 1 to 14, characterized in that, during the execution of the first movement process, if the first information becomes unavailable without acquiring a second shape in which the shape of the first part differs from the first shape, the device switches to the second movement process. (Configuration 16) The information processing apparatus according to Configuration 15, characterized in that, while executing the first movement process, if the first shape of the first part is obtained after switching to the second movement process, the processing means switches back to the first movement process, and if the second shape of the first part is obtained, the processing means terminates the second movement process without switching back to the first movement process. (Configuration 17) The information processing apparatus according to Configuration 16, characterized in that, when the processing means switches to the second movement process while executing the first movement process, and then obtains the second shape of the first part, it places the virtual object at the position of the first part and terminates the second movement process. (Method 1) A method for controlling an information processing device, A generation process that generates a virtual image of a virtual space containing movable virtual objects, A display control step that performs control to display the virtual image, A first acquisition step involves acquiring first information relating to a first part, which is a predetermined part of the user that views the virtual image displayed by the control in the display control step, A second acquisition step of acquiring second information relating to a second body part which is a predetermined body part different from the first body part of the user, A control method for an information processing apparatus, characterized by comprising: a processing step that performs a first movement process to move the virtual object based on the first information acquired in the first acquisition step, and a second movement process to move the virtual object based on the second information acquired in the second acquisition step, and a processing step that switches between the first movement process and the second movement process depending on the acquisition status of the first information in the first acquisition step or the acquisition status of the second information in the second acquisition step. (Program 1) A program characterized by causing a computer to execute the control method described in Method 1. [Explanation of Symbols]

[0068] 100 HMD 101 Control Unit 104 Imaging Unit 105 Sensor section 106 Display section 200-250 Mixed Reality Images 201 Virtual Objects

Claims

1. A generation means for generating a virtual image of a virtual space containing movable virtual objects, A display control means that controls the display of the virtual image, A first acquisition means for acquiring first information relating to a first part, which is a predetermined part of the user that views the virtual image displayed by the control means of the display control means, A second acquisition means for acquiring second information relating to a second body part, which is a predetermined body part different from the first body part of the user, An information processing apparatus characterized by comprising: a processing means that performs a first movement process for moving the virtual object based on the first information acquired by the first acquisition means, and a second movement process for moving the virtual object based on the second information acquired by the second acquisition means, and performs a process of switching between the first movement process and the second movement process according to the acquisition status of the first information by the first acquisition means or the acquisition status of the second information by the second acquisition means.

2. The information processing apparatus according to claim 1, characterized in that when the processing means executes the first movement process, it determines the destination of the virtual object in the virtual image based on the first information and moves the virtual object to the destination, and when the second movement process executes the second information, it determines the destination of the virtual object in the virtual image based on the second information and places the virtual object at the destination.

3. The system includes a determination means for determining the acquisition status of the first information by the first acquisition means and the acquisition status of the second information by the second acquisition means, The information processing apparatus according to claim 1, characterized in that the processing means performs a process of switching between the first movement process and the second movement process according to the determination result of the determination means.

4. The information processing apparatus according to claim 3, characterized in that the processing means switches to the second movement process when the determination means determines that the acquisition state of the first information has fallen below a predetermined state during the execution of the first movement process.

5. The information processing apparatus according to claim 3, wherein the processing means stops the second movement process if the determination means determines that the second information acquired by the second acquisition means has changed or that the acquisition state of the second information has deteriorated below a predetermined state during the execution of the second movement process.

6. The information processing apparatus according to claim 3, characterized in that the processing means switches to the first movement process when the determination means determines that the acquisition state of the first information has risen to a predetermined state or higher during the execution of the second movement process.

7. The information processing apparatus according to claim 6, characterized in that, when the determination means determines that the acquisition state of the first information has risen to a predetermined state or higher during the execution of the second movement process, the processing means switches to the first movement process and places the virtual object at a position based on the first information.

8. The first acquisition means acquires, as the first information, positional information relating to the position of the user's fingers or motion information relating to the movement of gestures, The information processing apparatus according to claim 1, characterized in that the second acquisition means acquires positional information relating to the position of the user's head as the second information.

9. The information processing apparatus according to claim 8, characterized in that the processing means executes the first movement process with priority over the second movement process.

10. The information processing apparatus according to claim 8, further comprising a determination means for determining, based on the first information, a virtual object to be moved in the first move process prior to the first move process.

11. The information processing apparatus according to claim 10, characterized in that, when the first information is location information, if the virtual object encompasses the coordinates included in the location information, the determination means can determine the virtual object to be moved as the virtual object to be moved, and can also cancel the determination.

12. The information processing apparatus according to claim 10, characterized in that, if the first information is motion information, the determination means can determine the virtual object as the virtual object to be moved if the gesture movement is a movement of grasping a virtual object, and can also cancel the determination.

13. The information processing apparatus according to claim 1, characterized in that the generation means makes the texture of the virtual object during the first movement process different from the texture of the virtual object during the second movement process.

14. The generation means generates a composite reality image by combining the virtual image and the real image, The information processing apparatus according to claim 1, characterized in that the display control means performs control to display the augmented reality image.

15. The processing means executes the first movement process based on the first information acquired by the first acquisition means when the shape of the first part is the first shape. The information processing device according to claim 1, characterized in that, during the execution of the first movement process, if the first information becomes unavailable without obtaining a second shape in which the shape of the first part differs from the first shape, the device switches to the second movement process.

16. The information processing apparatus according to claim 15, characterized in that, while executing the first movement process, if the first shape of the first part is obtained after switching to the second movement process, the processing means switches back to the first movement process, and if the second shape of the first part is obtained, the processing means terminates the second movement process without switching back to the first movement process.

17. The information processing apparatus according to claim 16, characterized in that, if the processing means switches to the second movement process while executing the first movement process and then obtains the second shape of the first part, it places the virtual object at the position of the first part and terminates the second movement process.

18. A method for controlling an information processing device, A generation process that generates a virtual image of a virtual space containing movable virtual objects, A display control step that performs control to display the virtual image, A first acquisition step involves acquiring first information relating to a first part, which is a predetermined part of the user that views the virtual image displayed by the control in the display control step, A second acquisition step of acquiring second information relating to a second body part which is a predetermined body part different from the first body part of the user, A control method for an information processing apparatus, characterized by comprising: a processing step that performs a first movement process to move the virtual object based on the first information acquired in the first acquisition step, and a second movement process to move the virtual object based on the second information acquired in the second acquisition step, and a processing step that switches between the first movement process and the second movement process depending on the acquisition status of the first information in the first acquisition step or the acquisition status of the second information in the second acquisition step.

19. A program characterized by causing a computer to execute the control method described in claim 18.

Images