Information processing device, information processing method, and storage medium

Abstract

[Problem] To make it possible to reduce a mounting load and realize a movement operation of a user viewpoint in a virtual space. [Solution] An information processing device comprising an acquisition unit that acquires information indicating a position and a posture of an operation body (410), and a display control unit that determines a reference position of an operation on the basis of the position of the operation body, and performs control to display an operation reference region image (450) indicating a predetermined range serving as a reference for operation at the reference position. The display control unit performs control to change a user viewpoint (U-eye) in a virtual space (V) on the basis of a change in the position or posture of the operation body relative to the operation reference region image.

Description

Information processing device, information processing method, and storage medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a storage medium.

[0002] Recently, various virtual objects such as 3D models or 2D images are presented in VR spaces realized by AR (Augmented Reality) technology or immersive displays, and users can perform various operational inputs on the virtual objects using a controller.

[0003] As an example of technology related to user operation input, Patent Document 1 discloses a game program that prevents players from becoming confused when using a pointing device to imitate a joystick as an example of operation input to a touch panel.

[0004] Patent No. 3734820

[0005] In a space that blends virtual and reality, such as XR (cross reality), a controller (real object) held in the user's hand, gloves worn on the hand, and devices worn on the user's arms, legs, or head are used to detect the user's real-world movements and reflect them in the virtual world, thereby realizing a space that more fully blends virtual and reality. However, wearing these devices requires effort and burden for the user, and there is a need to reduce the burden of wearing them by leaving parts of the body, such as the palms and fingertips, free. On the other hand, operating to move one's own position in an XR space requires various operational inputs, such as up / down, left / right, front / back, and yaw rotation, which allows for horizontal looking around.

[0006] Therefore, the present disclosure proposes an information processing device, an information processing method, and a storage medium that can reduce the burden of wearing the device and enable the user to move the viewpoint in a virtual space.

[0007] According to the present disclosure, there is provided an information processing device including: an acquisition unit that acquires information indicating the position and posture of an operating body; and a display control unit that determines a reference position for operation based on the position of the operating body and controls displaying an operation reference area image that indicates a predetermined range that serves as a reference for operation at the reference position, wherein the display control unit controls changing a user's viewpoint in a virtual space based on a change in the position or posture of the operating body relative to the operation reference area image.

[0008] Furthermore, according to the present disclosure, there is provided an information processing method including: a processor acquiring information indicating the position and posture of an operating body; determining a reference position for operation based on the position of the operating body; and performing control to display an operation reference area image indicating a predetermined range serving as a reference for operation at the reference position; and performing control to change the user's viewpoint in a virtual space based on a change in the position or posture of the operating body relative to the operation reference area image.

[0009] Furthermore, according to the present disclosure, there is provided a storage medium having stored thereon a program that causes a computer to function as an acquisition unit that acquires information indicating the position and posture of an operating body, and a display control unit that determines a reference position for operation based on the position of the operating body and controls displaying an operation reference area image indicating a predetermined range that serves as a reference for operation at the reference position, and the display control unit controls changing the user's viewpoint in a virtual space based on a change in the position or posture of the operating body relative to the operation reference area image.

[0010] 1 is a diagram illustrating an overall configuration of an information processing system 1 according to an embodiment of the present disclosure. FIG. 1 is a block diagram illustrating an example of the configuration of an HMD 20 according to this embodiment. FIG. 2 is a diagram illustrating display of an operation reference area according to this embodiment. FIG. 3 is a flowchart illustrating an example of the flow of a user's viewpoint movement control process according to this embodiment. FIG. 4 is a diagram illustrating movement in the XY directions (horizontal direction) according to this embodiment. FIG. 5 is a diagram illustrating movement in the Z direction (vertical direction) according to this embodiment. FIG. 6 is a diagram illustrating rotation according to this embodiment. FIG. 7 is a diagram illustrating an operation input for moving a user's viewpoint on the XY plane according to this embodiment. FIG. 8 is a diagram illustrating an example of a display screen when an operation input for moving a user's viewpoint on the XY plane according to this embodiment has been made. FIG. 9 is a diagram illustrating another operation input for moving a user's viewpoint on the XY plane according to this embodiment. FIG. 10 is a diagram illustrating an example of a display screen when an operation input for moving a user's viewpoint on the XY plane according to this embodiment has been made. FIG. 11 is a diagram illustrating an operation input for moving a user's viewpoint in the Z direction according to this embodiment. FIG. 12 is a diagram illustrating an operation input for rotating a user's viewpoint in the Yaw direction according to this embodiment. FIG. 13 is a diagram illustrating an example of a display screen when an operation input for moving a user's viewpoint for rotation in the Yaw direction according to this embodiment has been made. 1 is a diagram illustrating an example of deformation of the shape of an operation reference area image 450 in accordance with a user operation according to this embodiment. FIG. 2 is a diagram illustrating an example of deformation of the shape of an operation reference area image 450 in accordance with a user operation according to this embodiment. FIG. 3 is a diagram illustrating a visible area depending on a difference in FoV according to this embodiment.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The explanation will be given in the following order: 1. Configuration 1-1. System configuration 1-2. Configuration of the HMD 20 2. Operation processing 3. Movement and rotation of the user's viewpoint 3-1. Control of each movement direction and rotation direction 3-2. Movement on the XY plane and in the Z direction 3-3. Movement in the Yaw direction 3-4. Determination of movement direction 4. Modified examples 4-1. Adjustment of movement speed and look-around angular velocity 4-2. Indication of coordinate information in virtual space 4-3. Setting of operation reference area image 450 4-4. Handling unstable detection results 4-5. Changing movement speed and rotation angular velocity according to the field of view of the HMD 20 4-6. Usefulness in mechanical CAD 4-7. Other 5. Supplementary information

[0013] 1. Configuration An information processing system according to an embodiment of the present disclosure will be specifically described with reference to the drawings.

[0014] 1 is a diagram showing the overall configuration of an information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 according to this embodiment includes a ring device 10 and an HMD (Head Mounted Display) 20.

[0015] The ring device 10 is worn on any finger of the user. The ring device 10 is equipped with a clickable button (switch unit 110). The ring device 10 transmits press information (click information) of the switch unit 110 to the HMD 20 in real time. The position and orientation of the ring device 10 can be detected by various sensors provided in the ring device 10. In the HMD 20, the position and orientation of the ring device 10 may be detected as the position and orientation of the user's hand.

[0016] The HMD 20 is a display device (an example of an information processing device) that is worn on the user's head and has a display placed in front of the user's eyes. The HMD 20 may be configured to cover the user's entire field of vision when worn. The HMD 20 may display an image from the user's viewpoint in a virtual space. The HMD 20 may also be a video see-through type, in which case a virtual object is superimposed on a captured image of real space.

[0017] In this embodiment, the user can input operations into the virtual space, particularly operations for moving the user's viewpoint in three dimensions, with one hand, thereby reducing the burden on the user when wearing the device.

[0018] The overview of the information processing system 1 according to an embodiment of the present disclosure has been described above. Next, the configuration of the HMD 20 included in the information processing system 1 according to this embodiment will be described with reference to the drawings.

[0019] 2 is a block diagram showing an example of the configuration of the HMD 20 according to this embodiment. As shown in Fig. 2, the HMD 20 includes a communication unit 210, a ToF sensor 220, a camera 230, an inertial sensor 240, a control unit 250, a display unit 260, and a storage unit 270.

[0020] (Communication Unit 210) The communication unit 210 has a transmission unit that transmits data to an external device and a reception unit that receives data from an external device. The communication unit 210 according to this embodiment may be communicatively connected to an external device or the Internet using, for example, a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution)), 5G (fifth generation mobile communication system)), or the like.

[0021] (ToF Sensor 220) The ToF (Time of Flight) sensor 220 is a sensor that detects the distance to an object using infrared light. The ToF sensor 220 detects the distance to an object present around the user wearing the HMD 20. A plurality of ToF sensors 220 may be provided. For example, the position and orientation of the user's fingers and the distance to an obstacle present around the user can be calculated from the information output from the ToF sensor 220.

[0022] (Camera 230) Camera 230 is an imaging unit that captures images of the surroundings. Camera 230 may be a stereo camera. Multiple cameras 230 may be provided. Information output from camera 230 may be displayed as a through image on display unit 260, or may be used to calculate the position and orientation of the user's fingers.

[0023] (Inertial Sensor 240) The inertial sensor 240 is a sensor that detects three-dimensional inertial movement. The inertial sensor 240 has a three-axis gyro sensor and a three-axis acceleration sensor. The orientation of the HMD 20 (i.e., the orientation of the user's head) can be calculated from the information output from the inertial sensor 240.

[0024] (Control Unit 250) The control unit 250 functions as an arithmetic processing unit and a control device, and controls the overall operation of the HMD 20 in accordance with various programs. The control unit 250 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. The control unit 250 may also include a ROM (Read Only Memory) that stores programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0025] The control unit 250 can also function as a hand position / posture calculation unit 251 and a display control unit 252 .

[0026] The hand and finger position and orientation calculation unit 251 calculates the position and orientation of the user's hand and fingers based on information acquired from at least one of the ToF sensor 220 and the camera 230. For example, the hand and finger position and orientation calculation unit 251 estimates information on the joint points of the fingers and the skeletal structure based on the information output from the ToF sensor 220 and the camera 230. The hand and finger position and orientation calculation unit 251 is an example of an acquisition unit that acquires information indicating the position and orientation of the user's fingers (an example of an operating object). Note that sensing data of the user's fingers may be transmitted by the ring device 10, and the hand and finger position and orientation calculation unit 251 may calculate the hand and finger position and orientation based on this data. Alternatively, the ring device 10 may calculate the position and orientation of the user's fingers, and the hand and finger position and orientation calculation unit 251 may acquire this data.

[0027] The display control unit 252 performs control to display a predetermined image on the display unit 260 .

[0028] For example, the display control unit 252 may display in real time on the display unit 260 (through display) a captured image (video) captured by the camera 230 in the user's line of sight (the front direction of the HMD 20), causing the display unit 260 to function as a video see-through display. Furthermore, the display control unit 252 may superimpose a virtual object on a captured image of real space displayed on the display unit 260 to provide an augmented reality (AR) space. While performing through display, the display control unit 252 may also superimpose an image of the user's viewpoint in a virtual space with a hidden background (virtual space video) on the through image. For example, the display control unit 252 may display a virtual object to be worked on, allowing the user to change the position or orientation of the virtual object or edit the virtual object.

[0029] The display unit 260 may be an optical see-through display. In this case, the display control unit 252 may also superimpose an image of the user's viewpoint in the virtual space on the scenery in the real space.

[0030] The display control unit 252 can also cause the display unit 260 to function as a fully immersive display by stopping the through display and displaying an image of a virtual space. In this way, the display control unit 252 can provide a VR (Virtual Reality) space.

[0031] The display control unit 252 can reflect operation inputs made by the user's finger movements in display control of virtual objects. The user can operate virtual objects by raising their arms in front of their bodies and moving their fingers in the air. For example, if the display unit 260 is a video see-through type, a captured image of real space in the front direction of the user's face is displayed on the display unit 260 in real time. Therefore, the user can make movements such as grasping a virtual object or moving the grasped virtual object in any direction while viewing an image of their own finger and a virtual object superimposed on the image. The user's finger movements are acquired by the finger position and orientation calculation unit 251.

[0032] In this embodiment, fingers are given as an example of the operating object, but the operating object is not limited to this and may be, for example, a controller held by the user. In the case of a controller, the position and orientation of the controller are detected by an inertial sensor provided in the controller and transmitted in real time to the HMD 20. In the HMD 20, the position and orientation of the controller may be used as the position and orientation of the user's fingers.

[0033] Here, the operations that the user can perform include operations on virtual objects as described above, as well as operations to move the user's viewpoint in the virtual space (i.e., moving the user's own position). Operations to move the user's own position in the virtual space require various operations such as up / down, left / right, front / back, and yaw rotation, which is looking around horizontally.

[0034] The display control unit 252 according to this embodiment can reflect operation inputs based on the movement of the user's fingers in the movement control of the user's viewpoint in the virtual space (i.e., the display control of the virtual space image). Specifically, when a predetermined trigger is detected, the display control unit 252 displays an operation reference area image centered on the position of the user's hand (specifically, for example, the second and third joints of the index finger (so-called proximal joints)).

[0035] The predetermined trigger can be considered to be an operation that invokes the user's viewpoint movement control function according to this embodiment. The predetermined trigger may be, for example, the detection of a predetermined finger movement (e.g., snapping fingers twice, tapping the tips of the index and thumb together twice, etc.), or the pressing of the switch unit 110 provided on the ring device 10 worn on the user's finger (e.g., double-clicking). The display control unit 252 continues to display the operation reference area image as long as the predetermined trigger is active. For example, the operation reference area image continues to be displayed in the following states: a pause continues after snapping fingers twice, the tips of the index and thumb are in contact, and the pressing of the switch unit 110 of the ring device 10 for the second click continues. Here, the display of the operation reference area will be specifically described with reference to FIG. 3 .

[0036] 3 is a diagram illustrating the display of the operation reference area according to this embodiment. As shown in FIG. 3, a captured image 310 of the real space is displayed in real time on the display unit 260, and a virtual object 400 is superimposed on the captured image 310. The user's hand 320 viewed on the display unit 260 may be a virtual hand (CG image) superimposed on the user's hand shown in the captured image 310. The virtual hand is displayed following the position of the user's hand. Furthermore, a virtual object (CG image) corresponding to the ring device 10 may be worn on the fingers of the virtual hand.

[0037] The display control unit 252 sets, for example, the second and third joints (so-called proximal joints) of the index finger of the user's hand as the reference position, and displays the operation reference area image 450 with the reference position as the center. The shape of the operation reference area image 450 is not particularly limited, and may be, for example, a spherical virtual object (3DCG image).

[0038] As described above, the operation reference area image 450 continues to be displayed at the reference position as long as the predetermined trigger state continues. In other words, the operation reference area image 450 continues to be displayed at the reference position even when the user's hand moves up, down, left, right, forward, or backward while the predetermined trigger state is in effect (for example, when the switch unit 110 of the ring device 10 is pressed with a second click).

[0039] When a predetermined trigger is triggered and the user's hand leaves the operation reference area image 450 (i.e., when the hand moves a predetermined distance from the reference position), the display control unit 252 controls the user's viewpoint to move up, down, left, right, forward, or backward in accordance with the direction of the user's hand movement. The threshold value (e.g., 10 cm) for starting up, down, left, right, forward, or backward movement is the same as the radius of the spherical virtual object displayed as the operation reference area image 450, and when the hand leaves the operation reference area image 450, that is, when the movement distance from the reference position of the hand exceeds the threshold value. The display of the operation reference area image 450 clearly indicates the threshold value (distance from the reference position) for starting up, down, left, right, forward, or backward movement, allowing the user to intuitively grasp how far the hand needs to move to move the user's viewpoint. Details of the operation of moving the user's viewpoint in the up, down, left, right, forward, and backward directions will be described later with reference to FIGS. 5 to 12 .

[0040] Furthermore, the display control unit 252 performs control to rotate the user's viewpoint in the Yaw direction when the user's hand rotates by a predetermined angle (e.g., 30 degrees) or more around the forearm as an axis with respect to the user's hand posture (reference posture) when the reference position was determined within the operation reference area image 450. By making the operation reference area image 450 a three-dimensional image (sphere, cube, etc.), space for inputting a look-around operation can be secured. Details of the rotation operation of the user's viewpoint in the Yaw direction (look-around operation) will be described later with reference to FIGS. 13 to 15 .

[0041] The display control unit 252 may provide feedback on the operation by changing the display mode of the operation reference area image 450 in accordance with the movement of the user's hand (an example of an operating object), such as when the user's hand moves out of the operation reference area image 450 or when the user's hand rotates by more than a predetermined angle within the operation reference area image 450. Examples of changes in the display mode include changes in the color, density, transparency, pattern, shape, animation, or the like of the operation reference area image 450. For example, the color may be "green" when the hand is neither moving nor rotating, "orange" when the hand moves out of the operation reference area image 450, "blue" when the hand rotates by more than a predetermined angle within the operation reference area image 450, and "purple" when the hand moves out of the operation reference area image 450 and rotates by more than a predetermined angle.

[0042] The moving speed and rotation speed of the user's viewpoint by the display control unit 252 may be specified in advance. For example, the moving speed may be 1.2 m / sec and the rotation angular speed may be 60 degrees / sec.

[0043] As described above, in this embodiment, the input of viewpoint movement can be realized by simply using the user's finger movements, or by using a device with as few operating parts as possible, such as the ring device 10 having the switch unit 110, thereby reducing the burden of wearing the device on the user's body.

[0044] (Display Unit 260) The display unit 260 has a function of displaying an image. The display unit 260 is formed, for example, so as to cover the user's field of vision. The display unit 260 may also be a video see-through display or an optical see-through display. When the HMD 20 is realized as a glasses-type device, the display unit 260 may also be a transmissive display provided in the lens portion of the glasses.

[0045] (Storage Unit 270) The storage unit 270 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 250, and a RAM that temporarily stores parameters that change as appropriate.

[0046] Although the configuration of the HMD 20 has been specifically described above, the configuration of the HMD 20 according to the present disclosure is not limited to the example shown in FIG. 2 . For example, the HMD 20 does not necessarily have all of the components shown in FIG. 2 . Furthermore, the ToF sensor 220, the camera 230, and the inertial sensor 240 have been given as examples of sensors for acquiring the position and orientation of the user's fingers and the position and orientation of the user's head, but this embodiment is not limited to these. The HMD 20 may acquire the position and orientation of the user's fingers and the position and orientation of the user's head using other sensors. Furthermore, the HMD 20 may be realized by multiple devices.

[0047] 2. Operational Processing Next, the operational processing of the information processing system according to this embodiment will be described.

[0048] FIG. 4 is a flowchart showing an example of the flow of the user viewpoint movement control process according to this embodiment.

[0049] As shown in Fig. 4, first, the HMD 20 checks whether the switch unit 110 of the ring device 10 has been pressed (step S103). When the switch unit 110 of the ring device 10 has been pressed, a switch-on signal is sent from the ring device 10 to the HMD 20. Here, it is assumed that the trigger for accepting the operation to move the user's viewpoint is a double-click on the switch unit 110. Therefore, the HMD 20 checks whether the switch unit 110 of the ring device 10 has been pressed a second time. Furthermore, the ring device 10 continuously sends a switch-on signal to the HMD 20 while the switch unit 110 continues to be pressed. The HMD 20 performs the user's viewpoint movement control process described below while the second press on the switch unit 110 continues.

[0050] While the switch on the ring device 10 continues to be pressed, the HMD 20 continuously calculates the position and posture of the fingers using the finger position and posture calculation unit 251 based on various data detected by the ToF sensor 220 and the camera 230 provided in the HMD 20 (step S106).

[0051] Next, the display control unit 252 of the HMD 20 determines the position of the index finger (more specifically, the second joint point, etc.) as a reference position based on the position and orientation information of the fingers, and displays an operation reference area image 450 centered on the reference position (step S109). Note that the reference position is just an example, and the present embodiment is not limited to this. For example, the reference position may be the center position of the hand.

[0052] Next, the display control unit 252 checks whether the user's hand has moved outside the operation reference area image 450 (whether the distance traveled from the reference position of the hand has exceeded a threshold) based on the position and posture of the fingers continuously calculated by the finger position and posture calculation unit 251 (step S112).

[0053] If the hand goes outside the operation reference area image 450 (step S112 / Yes), the display control unit 252 performs control to move the user's viewpoint up, down, left, right, forward, or backward according to the direction of the hand movement (step S115). More specifically, the movement of the user's viewpoint up, down, left, right, forward, or backward is movement on the XY plane or movement in the Z direction. Details will be described later.

[0054] On the other hand, if the hand does not move out of the operation reference area image 450 (step S112 / No) and rotates within the operation reference area image 450 (around the forearm as an axis) by a predetermined angle or more (step S118 / Yes), the display control unit 252 performs control to rotate the user's viewpoint in the Yaw direction in accordance with the rotation of the hand (step S121). The Yaw direction rotation control of the user's viewpoint will also be described in detail later.

[0055] Then, the display control unit 252 reflects the movement and rotation of the user's viewpoint in the display control of the virtual space, and updates the display of the virtual space (step S124).

[0056] The above-described processing shown in steps S106 to S124 can be repeated while the switch section 110 of the ring device 10 continues to be pressed. When the pressing is released, the processing for controlling the movement of the user's viewpoint ends.

[0057] 4 is merely an example, and the present embodiment is not limited thereto. For example, it is conceivable that the hand moves out of the operation reference area image 450 and rotates by a predetermined angle or more. In this case, the display control unit 252 may perform control to move the user's viewpoint up, down, left, right, forward, and backward in accordance with the movement of the hand, and to rotate the viewpoint in the Yaw direction.

[0058] 3. Movement and Rotation of User's Viewpoint Next, movement and rotation of the user's viewpoint according to this embodiment will be specifically described.

[0059] <<3-1. Control of Each Movement Direction and Rotation Direction>> Movement and rotation in three-dimensional space have three degrees of freedom: forward, backward, left, right, up, and down, and three degrees of freedom of rotation: yaw, roll, and pitch. However, these directions of movement and rotation are not necessarily equivalent from the perspective of user movement. For example, horizontal and diagonal movement is common in everyday life, but diagonal movement, including vertical movement, is difficult to perform freely and is performed only in limited situations in everyday life, such as on slopes, stairs, or escalators. Furthermore, with regard to rotation direction, yaw rotation is a commonly performed motion, as it involves turning one's head from side to side. On the other hand, roll and pitch rotations resemble cartwheels or flips, which are rarely performed and can be quite unnatural because they change the angle between the user and the horizontal plane. Therefore, in this embodiment, the movement direction and rotation direction are restricted as follows.

[0060] 5 is a diagram illustrating movement in the XY directions (horizontal direction) according to this embodiment. As shown in FIG. 5, this embodiment allows the user's viewpoint to move on the XY plane, including diagonal movement (e.g., arrow directions D1, D2, etc.), in accordance with the movement of the user's hand 410 in the XY directions (horizontal direction).

[0061] 6 is a diagram illustrating movement in the Z direction (vertical direction) according to this embodiment. As shown in Fig. 6, in this embodiment, only movement of the user's viewpoint in the Z direction (vertical direction) is permitted in response to movement of the user's hand 410 in the Z direction (vertical direction), and diagonal movement of the user's viewpoint that includes a horizontal movement component is not permitted.

[0062] FIG. 7 is a diagram illustrating rotation according to this embodiment. As shown in FIG. 7 , in the rotation control of the user's viewpoint according to this embodiment, rotation in the Yaw direction is possible (and can coexist with movement), but rotation including a Roll / Pitch component (diagonal rotation) is not possible. Furthermore, rotation in the Roll / Pitch direction is not possible in the user's viewpoint rotation. Here, look-around rotation of the user's viewpoint is possible only in the Yaw direction, which is horizontal rotation. However, it is difficult to rotate a human hand in the Yaw direction in an operation input due to the structure of the human body. Therefore, for the operation input of look-around rotation according to this embodiment, a rotational motion (twist) of the hand around the forearm is used. Although the operation input uses a rotation axis different from the rotation direction (Yaw direction) that the user wants to look around, the operation input of look-around rotation is possible with a natural movement due to the structure of the human body.

[0063] <<3-2. Movement on the XY Plane and Movement in the Z Direction>> Next, movement on the XY plane and movement in the Z direction will be specifically described. The above-mentioned "movement of the user's viewpoint up, down, left, right, forward, backward" refers more specifically to movement on the XY plane or movement in the Z direction.

[0064] (Control of movement on the XY plane) Fig. 8 is a diagram illustrating an operation input for moving the user's viewpoint on the XY plane according to this embodiment. The upper left and right diagrams of Fig. 8 show a virtual object Obj placed in a virtual space V and the user's viewpoint (U-eye). The lower left and right diagrams of Fig. 8 show a user's hand 410 and an operation reference area image 450 displayed at a reference position. Both diagrams are views from above. Here, of the user's gaze movements in the forward and backward directions on the XY plane, an operation input for moving the user's gaze backward will be described as an example.

[0065] As shown in FIG. 8 , when the user pulls the hand 410 back from the operation reference area image 450, i.e., moves the hand 410 backward from the reference position and moves out of the operation reference area image 450, the display control unit 252 controls the movement of the user's viewpoint (U-eye) backward. Note that the display control unit 252 continues to move the user's viewpoint (U-eye) backward as long as the hand 410 that has moved out of the operation reference area image 450 remains positioned outside the operation reference area image 450. The display control unit 252 can continue to control the movement of the user's viewpoint (U-eye) until the switch unit 110 of the ring device 10 is released from being pressed. Furthermore, if there is a limit to the movable area, such as the area of ​​a virtual space, the display control unit 252 can continue to control the movement of the user's viewpoint (U-eye) up to that limit.

[0066] 9 is a diagram showing an example of a display screen when an operation input for moving the user's viewpoint on the XY plane is performed according to this embodiment. As described with reference to FIG. 8 , the user's viewpoint moves back due to the movement of pulling the hand 410 toward the user, and therefore, as shown in FIG. 9 , the virtual object 400 appears to move relatively far away on the display unit 260. At this time, the display mode of the operation reference area image 450 is also changed. The change in the display mode of the operation reference area image 450 can continue while the user's viewpoint (U-eye) is moving (i.e., while the user's viewpoint is being changed).

[0067] 10 is a diagram illustrating another operation input for moving the user's viewpoint on the XY plane according to this embodiment. The upper part of FIG. 10 shows a virtual object Obj placed in a virtual space V and the user's viewpoint (U-eye). The lower part of FIG. 10 shows a user's hand 410 and an operation reference area image 450 displayed at a reference position. Both are views from above. Here, an operation input for moving the user's viewpoint to the left as seen from the user, among the user's viewpoint movements to the left and right as seen from the user in the X direction, will be described.

[0068] 10 , when the user moves the hand 410 to the left of the operation reference area image 450, i.e., moves the hand 410 leftward from the reference position and moves outside the operation reference area image 450, the display control unit 252 controls the movement of the user's viewpoint (U-eye) to the left. Note that the display control unit 252 continues to move the user's viewpoint (U-eye) to the left as long as the hand 410 that has moved outside the operation reference area image 450 remains positioned outside the operation reference area image 450. The display control unit 252 can continue to control the movement of the user's viewpoint (U-eye) until the switch unit 110 of the ring device 10 is released from being pressed. Furthermore, if there is a limit to the movable area, such as the area of ​​a virtual space, the display control unit 252 can continue to control the movement of the user's viewpoint (U-eye) up to that limit.

[0069] 11 is a diagram showing an example of a display screen when an operation input for moving the user's viewpoint on the XY plane is performed according to this embodiment. As described with reference to FIG. 10 , by moving the hand 410 to the left, the user's viewpoint moves leftward, and therefore, as shown in FIG. 11 , the virtual object 400 appears to move relatively to the right on the display unit 260. At this time, the display mode of the operation reference area image 450 is changed. The change in the display mode of the operation reference area image 450 can continue while the user's viewpoint (U-eye) is moving (i.e., while the user's viewpoint is being changed).

[0070] (Z-direction movement control) Fig. 12 is a diagram illustrating an operation input for moving the user's viewpoint in the Z direction according to this embodiment. The upper part of Fig. 12 shows a virtual object Obj placed in a virtual space V, and the user's viewpoint (U-eye). The lower part of Fig. 12 shows a user's hand 410 and an operation reference area image 450 displayed at a reference position. Both are diagrams in which the user is viewed from the horizontal direction, and the X-axis indicates the front direction of the user. Here, of the user's gaze movements in the up and down directions in the Z direction, an operation input for moving the user's gaze upward will be described as an example.

[0071] 12 , when the user moves their hand 410 from a state in which the hand 410 is raised up to around the chest in front of the body (reference position) to the upper side of the operation reference area image 450, that is, moves the hand 410 upward from the reference position and goes outside the operation reference area image 450, the display control unit 252 controls the movement of the user's viewpoint (U-eye) upward. Note that the display control unit 252 continues to move the user's viewpoint (U-eye) upward as long as the hand 410 that has gone outside the operation reference area image 450 is positioned outside the operation reference area image 450. The display control unit 252 can control the movement of the user's viewpoint (U-eye) up to the limit of the virtual space area or until the switch unit 110 of the ring device 10 is released from being pressed.

[0072] <<3-3. Movement in the Yaw Direction>> Next, rotation in the Yaw direction will be specifically described.

[0073] Fig. 13 is a diagram illustrating an operation input for rotating the user's viewpoint in the Yaw direction according to this embodiment. The upper part of Fig. 13 shows a virtual object Obj placed in a virtual space V and the user's viewpoint (U-eye). The lower part of Fig. 13 shows a user's hand 410 and an operation reference area image 450 displayed at a reference position. Both are views from above. Here, of the user's gaze movements of clockwise rotation and counterclockwise rotation as viewed from above in the vertical direction in the Yaw direction, an operation input for counterclockwise user gaze movement will be described as an example.

[0074] 13 , when the user rotates the hand 410 counterclockwise as seen from the user within the operation reference area image 450 around the forearm as an axis, i.e., when the user twists the hand counterclockwise as seen from the user relative to the reference posture and rotates the hand by more than a predetermined angle, the display control unit 252 controls the user's viewpoint (U-eye) to rotate counterclockwise in the Yaw direction as seen from above in the vertical direction. Note that the display control unit 252 continues to rotate the user's viewpoint (U-eye) counterclockwise as long as the hand 410 remains twisted by more than the predetermined angle within the operation reference area image 450. The display control unit 252 can control the movement of the user's viewpoint (U-eye) until the switch unit 110 of the ring device 10 is released from being pressed.

[0075] 14 is a diagram showing an example of a display screen when an operation input for moving the user's viewpoint in a yaw direction rotation is performed according to this embodiment. As described with reference to FIG. 13 , by twisting the hand 410 counterclockwise, the user's viewpoint rotates counterclockwise in the yaw direction (looking around horizontally), and therefore, as shown in FIG. 14 , the virtual object 400 appears to move relatively clockwise on the display unit 260. At this time, the display mode of the operation reference area image 450 is changed. The change in the display mode of the operation reference area image 450 can continue while the user's viewpoint (U-eye) is rotating and moving (i.e., while the user's viewpoint is being changed).

[0076] <<3-4. Determination of Movement Direction>> The display control unit 252 acquires information about the movement direction of the hand based on the information about the position and orientation of the fingers calculated by the hand position and orientation calculation unit 251, and reflects this information in the above-mentioned movement control of the user's viewpoint. However, whether to move the user's viewpoint in the direction on the XY plane or the Z direction may be determined by decomposing the hand movement vector into components of the direction on the XY plane and the Z direction, and using the larger Euclidean norm. Figure 15 is a diagram for explaining determination of the movement direction according to this embodiment.

[0077] 15 , when a movement vector V1 of the user's hand is detected, the display control unit 252 decomposes the movement vector V1 into an XY component vector V2 and a Z component vector V3. In the example shown in FIG. 15 , the Euclidean norm of the Z component vector V3 is greater than that of the XY component vector V2, so the display control unit 252 determines to move the user's viewpoint only in the Z direction. Then, if the user's hand is outside the operation reference area image 450, the display control unit 252 starts movement control of the user's viewpoint, but does not perform movement control if the user's hand is not outside the operation reference area image 450.

[0078] <4. Modified Examples> <<4-1. Adjustment of Movement Speed ​​and Look Angular Velocity>> The display control unit 252 changes the movement speed and look angular velocity in accordance with the distance the hand has moved from the center (reference position) of the operation reference area image 450, the angle of rotation from the reference posture, and the duration of the operation input state.

[0079] For example, the display control unit 252 increases the speed of the user's viewpoint movement as the distance the hand moves from the reference position increases. Furthermore, the display control unit 252 increases the speed of the user's viewpoint movement as the angle of rotation of the hand from the reference posture increases. Furthermore, the display control unit 252 increases the speed of the user's viewpoint movement as the time the hand remains outside the operation reference area image 450 increases (the longer the operation input state continues).

[0080] This makes it possible to dynamically and easily switch between requirements for moving quickly and far, and for moving while adjusting the distance a little at a time.

[0081] <<4-2. Explicit Display of Coordinate Information in Virtual Space>> When a user's viewpoint (self-position) is moved or rotated in an augmented reality (AR) HMD such as a video see-through or optical see-through HMD, it may be difficult for the user to tell whether the virtual object 400 has moved or whether the user's own position has moved or rotated in the virtual space.

[0082] 16 is a diagram showing an example of how coordinate information is displayed when moving or rotating according to this embodiment. When moving or rotating, the display control unit 252 can make the coordinate information in the virtual space clear by additionally displaying grid lines G on the display screen, as shown in FIG. 16, thereby improving usability.

[0083] <<4-3. Setting the Operation Reference Area Image 450>> The size of the operation reference area image 450 (threshold for movement operation from the reference position) may be set appropriately for the user. The larger the size of the operation reference area image 450, the less likely it is that the user will accidentally move their hand unintentionally. However, if the size is too large, the user will need to move outside the operation reference area image 450 for a long time, which will increase the operational load.

[0084] Therefore, for example, physical characteristics such as the length of the user's arms may be input in advance, and adjustments may be made such as increasing the size of the operation reference area image 450 since people with long arms tend to move their hands more widely.

[0085] Furthermore, the display control unit 252 may recognize whether the user is in front of a desk, whether there are any obstacles (real objects) in the vicinity, and so on. Since it is difficult to move the hand downward in front of a desk, the display position of the operation reference area image 450 may be moved slightly upward in advance, or the area below the operation reference area image 450 may be reduced. Because the operation input method involves actually moving the hand in the air, for example, when working in front of a desk, the actual range available for operation varies depending on the situation because real objects such as the desk and a drink placed on the desk are present. In this embodiment, the shape of the operation reference area image 450 may be changed depending on the range in which the hand can be moved. Changing the size of the operation reference area image 450 depending on the surrounding situation improves the user experience.

[0086] In addition, assuming that the user will become tired of moving their hands as the user continues to use the device for a long time, the display control unit 252 may reduce the size of the operation reference area image 450 and change the threshold value for starting movement.

[0087] Furthermore, the display control unit 252 may acquire the user's usage history and, if the usage time becomes long, may assume that the user has become accustomed to the operation and reduce the size of the operation reference area image 450 .

[0088] In addition, the display control unit 252 may acquire the user's usage history, and if the number of times the user has looked around is large, the size of the operation reference area image 450 may be increased to make the looking around operation easier and to make it less likely that the user will move up, down, left, right, forward, or backward.

[0089] Furthermore, the display control unit 252 may change the size of the operation reference area image 450 depending on the performance (hand tracking detection performance) of the hand position and orientation calculation unit 251 based on the sensor, CPU, etc. used. For example, when the hand tracking detection performance is high, the display control unit 252 may reduce the size of the operation reference area image 450 because it can detect even minute hand movements. On the other hand, when the detection performance is low, the display control unit 252 may increase the size of the operation reference area image 450 so that the user can move their hand more widely.

[0090] Furthermore, the display control unit 252 may not only change the color of the operation reference area image 450 but also change its shape, flash the display, or provide sound or vibration as feedback on the user's operation input. This allows the user to easily perceive the direction and amount of movement, and the amount of rotation of the hand. FIG. 17 is a diagram illustrating an example of deformation of the shape of the operation reference area image 450 in response to a user operation according to this embodiment. As shown in FIG. 17 , when the user's hand moves outside the operation reference area image 450, a portion of the operation reference area image 450 is protruded as if pulled to the position of the user's finger, making it easier for the user to perceive the movement distance and direction of the operation. FIG. 18 is a diagram illustrating an example of deformation of the shape of the operation reference area image 450 in response to a user operation according to this embodiment. As shown in FIG. 18 , when the user's hand is twisted within the operation reference area image 450, the operation reference area image 450 is changed into a twisted shape, making it easier for the user to perceive the rotation angle of the operation.

[0091] <<4-4. Dealing with Unstable Detection Results>> When tracking the user's fingers in the HMD 20 (continuous calculation of the finger position and orientation by the finger position and orientation calculation unit 251), if there is an obstruction between the finger and the various sensors of the HMD 20 (the ToF sensor 220 and the camera 230), the position of the finger may be lost, resulting in unstable detection results.

[0092] In this embodiment, if the tracking result is lost or the detection result is unstable and has changed significantly from the previous value, the following measures are taken to prevent operation from going against the user's intention.

[0093] First, the display control unit 252 hides the operation reference area image 450 even if the switch unit 110 of the ring device 10 is pressed. Furthermore, the display control unit 252 stops the movement and rotation of the user's viewpoint when an unstable detection result is obtained. Then, when a stable tracking result is obtained, the display control unit 252 returns to normal processing.

[0094] <<4-5. Changing the Movement Speed ​​and Rotational Angular Velocity According to the Field of View of the HMD 20>> The HMD 20 has a field of view in which the user can view the display, which generally tends to be narrower than the field of view when the user is not wearing the HMD 20. If the FoV (Field of View) is wide, there is no problem even if the rotational angular velocity for looking around in the Yaw direction is somewhat fast, but if the FoV is narrow and the rotational angular velocity in the Yaw direction is fast, the range that can be viewed at one time is narrow, making it difficult to adjust the direction you want to view. The same applies to movement on the XY plane and movement in the Z direction.

[0095] Figure 19 is a diagram illustrating the visual recognition area depending on the FoV according to this embodiment. The upper part of Figure 19 shows a visual recognition area 265 when the FoV is wide, and the lower part of Figure 19 shows a visual recognition area 266 when the FoV is narrow. The size of the visual recognition area is such that visual recognition area 265 is larger than visual recognition area 266.

[0096] As shown in the upper part of Figure 19, when the FoV is wide, a state in which a portion of viewing area 265-1 immediately before the viewpoint shift and a portion of viewing area 265-2 after the viewpoint shift overlap easily occurs. If a portion of viewing area 265-1 immediately before the viewpoint shift is reflected in viewing area 265-2 after the viewpoint shift, even if the look rotation angular velocity is somewhat fast, it does not cause a problem, and the user can easily recognize where they are looking. On the other hand, as shown in the lower part of Figure 19, when the FoV is narrow, a state in which a portion of viewing area 266-1 immediately before the viewpoint shift and a portion of viewing area 266-2 after the viewpoint shift overlap less easily occurs. Because viewing area 266-2 after the viewpoint shift is not continuous with viewing area 266-1 immediately before the viewpoint shift, a fast look rotation angular velocity makes it difficult for the user to recognize where they are looking.

[0097] Therefore, the display control unit 252 can improve the user experience by adjusting the yaw-direction rotation angular velocity and the movement speed on the XY plane and in the Z direction depending on the width of the FoV of the HMD 20. Specifically, when the width of the FoV of the HMD 20 is narrower than a predetermined value, the display control unit 252 may adjust the yaw-direction rotation angular velocity and the movement speed on the XY plane and in the Z direction to be slower than the initial values.

[0098] <<4-6. Usefulness in Mechanical CAD>> In existing 3D games, three-dimensional movement and Yaw rotation are achieved by operating two controllers with the left and right hands. However, in the case of a 3D drawing creation application for mechanical drawing design, operations such as selecting a part after moving or looking around, creating a cross section to check the cross section, and measuring the distance between parts are required, so it is preferable to be able to perform such operations with the other hand after moving or looking around. In this embodiment, three-dimensional movement and Yaw rotation operations can be input using only one hand, making this application useful for 3D drawing creation applications for mechanical drawing design.

[0099] <<4-7. Others>> The display control unit 252 may not allow diagonal movement on the XY plane in the X and Y axis directions either, and may allow movement only in either the X or Y axis direction, similar to the Z axis.

[0100] The display control unit 252 may display a guide indicating in which direction the user's hand will move and look around in the virtual space when moving the hand, when displaying the operation reference area image 450. The guide display can improve usability.

[0101] Since continuous movement and looking around may induce sickness, the display control unit 252 may move and look around intermittently, for example, every 0.5 seconds, instead of continuously, in order to reduce sickness. Similarly, in order to reduce sickness when moving or looking around, the display control unit 252 may make the surrounding field of view darker and add an effect such as a convergence line, making it easier to visually recognize the speed of movement.

[0102] The distance traveled and the angle of rotation may be limited to a numerical value such as 1 m for movement and 5 degrees for rotation, assuming that the device is used for mechanical CAD applications, for example.

[0103] The display control unit 252 may change the movement speed and the look angular velocity depending on the position in the virtual space and the positional relationship with the virtual object. For example, the display control unit 252 may temporarily stop or slow down the movement when the movement reaches the same height as the floor during vertical movement. Furthermore, when the user approaches a virtual object, it is assumed that the user may want to look around the virtual object in detail, so the display control unit 252 may also temporarily stop or slow down the movement at that point.

[0104] The movement and look-around functions can be invoked not only by double-clicking on the ring device 10, but also by other input methods such as a specific gesture such as opening your hand, or a voice command.

[0105] The display control unit 252 may store the state before movement and looking around are started, and may be configured to be able to return to that state even after one movement or looking around has been performed. Since it is possible to immediately return to the previous state, restoration in the event of an operation failure or comparison with the previous viewpoint can be easily performed.

[0106] The upper limits of the movement speed and look angular velocity may be changed depending on the size of the virtual object and the spatial sizes of the real world and the virtual world.

[0107] In the above-described embodiment, the user's viewpoint is rotated in the Yaw direction in response to a rotation about the user's forearm, but this embodiment is not limited to this. The viewpoint may be rotated about another direction or may be rotated in response to an operation on a touchpad provided on the ring device 10.

[0108] 5. Supplementary Information Although preferred embodiments of the present disclosure have been described above in detail with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0109] It is also possible to create one or more computer programs for causing the hardware, such as the CPU, ROM, and RAM, built into the HMD 20 to perform the functions of the HMD 20. A computer-readable storage medium storing the one or more computer programs is also provided.

[0110] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0111] Note that the present technology can also be configured as follows. (1) An information processing device including: an acquisition unit that acquires information indicating a position and posture of an operating body; and a display control unit that determines a reference position for operation based on the position of the operating body and controls displaying an operation reference area image indicating a predetermined range serving as a reference for operation at the reference position, wherein the display control unit controls changing a user's viewpoint in a virtual space based on a change in the position or posture of the operating body with respect to the operation reference area image. (2) The information processing device according to (1), wherein, when the operating body moves outside the operation reference area image, the display control unit controls moving the user's viewpoint in accordance with a moving direction from the reference position. (3) The information processing device according to (1) or (2), wherein, when the operating body rotates by a predetermined angle or more in the operation reference area image from a reference posture when the reference position is determined, the display control unit controls rotating the user's viewpoint in accordance with a rotation direction. (4) The information processing device according to any one of (1) to (3), wherein the display control unit limits changes in the movement direction or rotation direction of the user's viewpoint to some directions. (5) The information processing device according to (4), wherein the display control unit allows diagonal movement of the user's viewpoint on an XY plane but prohibits diagonal movement in the Z direction. (6) The information processing device according to (4) or (5), wherein the display control unit allows rotation of the user's viewpoint only in the Yaw direction but prohibits rotation in the Roll direction and the Pitch direction. (7) The information processing device according to (6), wherein the display control unit performs control to rotate the user's viewpoint in the Yaw direction in virtual space in accordance with rotation of the operating body around the user's forearm as an axis. (8) The information processing device according to any one of (1) to (7), wherein the display control unit continues moving the user's viewpoint as long as the operating body remains positioned outside the operation reference area image. (9) The information processing device according to any one of (1) to (8), wherein the display control unit continues to rotate the user's viewpoint as long as the operating object continues to be rotated by a predetermined angle or more in the operation reference area image.(10) The information processing device according to any one of (1) to (9), wherein the display control unit changes the display mode of the operation reference area image while changing the user's viewpoint. (11) The information processing device according to any one of (1) to (10), wherein the display control unit changes the display mode of the operation reference area image in accordance with a movement of the operating object. (12) The information processing device according to any one of (1) to (11), wherein the display control unit changes the display mode of the operation reference area image in accordance with a real object present in the periphery. (13) The information processing device according to any one of (1) to (12), wherein the display control unit starts displaying the operation reference area image when a predetermined trigger is detected. (14) The information processing device according to (13), wherein the predetermined trigger is a specific movement of the operating object. (15) The information processing device according to (13), wherein the operating object is a user's finger, and wherein the predetermined trigger is a click operation on a ring device worn on the user's finger. (16) The information processing device according to any one of (1) to (15), wherein the operation reference area image is a spherical virtual object. (17) The information processing device according to any one of (1) to (16), wherein the information processing device is an HMD that is worn on a user's head and has a display placed in front of the user's eyes. (18) An information processing method including: a processor acquiring information indicating a position and attitude of an operating body, determining a reference position for operation based on the position of the operating body, and performing control to display an operation reference area image that indicates a predetermined range that serves as a reference for operation at the reference position, and performing control to change a user's viewpoint in a virtual space based on a change in the position or attitude of the operating body with respect to the operation reference area image.(19) A storage medium having stored therein a program that causes a computer to function as: an acquisition unit that acquires information indicating the position and posture of an operating body; and a display control unit that determines a reference position for operation based on the position of the operating body and controls displaying an operation reference area image indicating a predetermined range that serves as a reference for operation at the reference position, wherein the display control unit controls changing of a user's viewpoint in a virtual space based on a change in the position or posture of the operating body relative to the operation reference area image.

[0112] REFERENCE SIGNS LIST 10 Ring device 110 Switch unit 20 HMD 210 Communication unit 220 ToF sensor 230 Camera 240 Inertial sensor 250 Control unit 251 Hand position and orientation calculation unit 252 Display control unit 260 Display unit 270 Storage unit 400 Virtual object 410 Hand 450 Operation reference area image

Claims

1. An information processing device comprising: an acquisition unit that acquires information indicating the position and posture of an operating body; and a display control unit that determines a reference position for operation based on the position of the operating body and controls displaying an operation reference area image indicating a predetermined range serving as the reference for operation at the reference position, wherein the display control unit controls changing a user's viewpoint in a virtual space based on a change in the position or posture of the operating body relative to the operation reference area image.

2. The information processing device according to claim 1, wherein the display control unit performs control to move the user's viewpoint in accordance with a moving direction from the reference position when the operating object moves outside the operation reference area image.

3. The information processing device of claim 1, wherein the display control unit controls the user's viewpoint to rotate in accordance with the direction of rotation when the operating body rotates by a predetermined angle or more in the operation reference area image from the reference posture when the reference position was determined.

4. The information processing device according to claim 1, wherein the display control unit limits changes in a movement direction or a rotation direction of the user's viewpoint to some directions.

5. The information processing device according to claim 4, wherein the display control unit enables diagonal movement when the user's viewpoint moves on an XY plane, but disables diagonal movement when the user's viewpoint moves in the Z direction.

6. The information processing device according to claim 4, wherein the display control unit allows rotation of the user's viewpoint only in the Yaw direction, and does not allow rotation in the Roll direction or Pitch direction.

7. The information processing device according to claim 6, wherein the display control unit performs control to rotate the user's viewpoint in the virtual space in a Yaw direction in response to rotation of the operating body around the user's forearm as an axis.

8. The information processing device according to claim 1, wherein the display control unit continues to move the user's viewpoint as long as the operating object remains positioned outside the operation reference area image.

9. The information processing device according to claim 1, wherein the display control unit continues to rotate the user's viewpoint as long as the operating object remains rotated by a predetermined angle or more in the operation reference area image.

10. The information processing device according to claim 1, wherein the display control unit changes a display mode of the operation reference area image while the user's viewpoint is being changed.

11. The information processing device according to claim 1, wherein the display control unit changes a display mode of the operation reference area image in accordance with the movement of the operating object.

12. The information processing device according to claim 1, wherein the display control unit changes a display mode of the operation reference area image in accordance with a real object existing in the vicinity.

13. The information processing device according to claim 1, wherein the display control unit starts displaying the operation reference area image when a predetermined trigger is detected.

14. The information processing device according to claim 13, wherein the predetermined trigger is a specific movement of the operating object.

15. The information processing device according to claim 13, wherein the operating object is a user's finger, and the predetermined trigger is a click operation on a ring device worn on the user's finger.

16. The information processing device according to claim 1, wherein the operation reference area image is a spherical virtual object.

17. The information processing device according to claim 1, wherein the information processing device is an HMD that is worn on a user's head and has a display positioned in front of the user's eyes.

18. An information processing method including: a processor acquiring information indicating the position and posture of a control body; determining a reference position for operation based on the position of the control body, and controlling the display of an operation reference area image indicating a predetermined range serving as a reference for operation at the reference position; and controlling the change of a user's viewpoint in a virtual space based on a change in the position or posture of the control body relative to the operation reference area image.

19. A storage medium having stored therein a program that causes a computer to function as: an acquisition unit that acquires information indicating the position and posture of an operating body; and a display control unit that determines a reference position for operation based on the position of the operating body and controls displaying an operation reference area image indicating a predetermined range serving as the reference for operation at the reference position, wherein the display control unit controls changing a user's viewpoint in a virtual space based on a change in the position or posture of the operating body relative to the operation reference area image.

Images