Information processing apparatus, information processing method, and storage medium

Abstract

[Problem] To make it possible to reduce the burden on wearing and to achieve a user viewpoint movement operation in a virtual space. [Solution] An information processing device is provided with: an acquisition unit (410) that acquires information indicating the position and orientation of an operation body; and a display control unit that determines a reference position of the operation in accordance with the position of the operation body, and performs control to display, at the reference position, an operation reference region image (450) indicating a predetermined range serving as a reference for the operation. The display control unit performs control to change the user viewpoint (U-eye) in the virtual space (V) on the basis of a change in the position or orientation of the operation body with respect to the operation reference region image.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing methods, and storage media. Background Technology

[0002] In recent years, various virtual objects, such as 3D models or 2D images, have been presented in VR spaces displayed through AR (Augmented Reality) technology or on immersive displays, and users can use controllers to perform various types of operation inputs on the virtual objects.

[0003] As a technology related to user input, for example, as an example of input via a touch panel, Patent Document 1 discloses a game program that prevents confusion caused by the player's use of pointing devices and analog joysticks.

[0004] Reference List

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent No. 3734820 Summary of the Invention

[0007] Technical issues

[0008] Here, spaces such as XR (Cross-Reality), which integrate virtual and reality, realize a space where a user's actions in the real world are detected and reflected in the virtual world by controllers (real objects) held by the user's hands, gloves worn on the hands, and devices worn on the user's arms, feet, and head, thus achieving a space with enhanced virtual-reality fusion. However, this requires additional work and places a burden on users who need to wear such devices. There is a need to reduce the burden of wearing them, so that physical parts such as the palms or fingertips are not restricted. On the other hand, operations for self-positional movement in XR space require various operational inputs, including movement in the up-down, left-right, or forward-backward directions, as well as yaw rotation as a circumferential movement in the horizontal direction.

[0009] In view of this, the present disclosure proposes an information processing apparatus, an information processing method, and a storage medium that can reduce the burden of wearing and enable user viewpoint movement operations in virtual space.

[0010] Solution to the problem

[0011] This disclosure provides an information processing apparatus, including: an acquisition unit that acquires information indicating the position and posture of an operating component; and a display control unit that determines a reference position for operation based on the position of the operating component, the display control unit performs control to display an operation reference area image indicating a predetermined area used as an operation reference at the reference position, and the display control unit performs control to change the user's viewpoint in virtual space based on changes in the position or posture of the operating component relative to the operation reference area image.

[0012] Furthermore, this disclosure provides an information processing method executed by a processor, the information processing method comprising: acquiring information indicating the position and orientation of an operating component; determining a reference position for operation based on the position of the operating component, and performing control to display an operation reference area image indicating a predetermined area used as an operation reference at the reference position; and performing control to change the user's viewpoint in virtual space based on changes in the position or orientation of the operating component relative to the operation reference area image.

[0013] Furthermore, this disclosure provides a storage medium storing a program that causes a computer to operate as follows: an acquisition unit that acquires information indicating the position and orientation of an operating component; a display control unit that determines a reference position for the operation based on the position of the operating component, the display control unit performing control to display an operation reference area image indicating a predetermined area used as an operation reference at the reference position, and the display control unit performing control to change the user's viewpoint in virtual space based on changes in the position or orientation of the operating component relative to the operation reference area image. Attached Figure Description

[0014] Figure 1 The overall configuration of the information processing system 1 according to an embodiment of the present disclosure is shown.

[0015] Figure 2 This is a block diagram illustrating an example of the structure of the HMD 20 according to this embodiment.

[0016] Figure 3 This is a diagram used to illustrate the display of the operation reference area according to this embodiment.

[0017] Figure 4 This is a flowchart illustrating an example of the user viewpoint movement control process according to this embodiment.

[0018] Figure 5 This is a diagram used to illustrate the movement in the XY direction (horizontal direction) according to this embodiment.

[0019] Figure 6 This is a diagram used to illustrate the movement in the Z direction (vertical direction) according to this embodiment.

[0020] Figure 7 This is a diagram used to illustrate the rotation according to this embodiment.

[0021] Figure 8 This is a diagram illustrating the operation input for moving the user's viewpoint in the XY plane according to this embodiment.

[0022] Figure 9 An example of a display screen is shown in which operation input for user viewpoint movement is performed in the XY plane according to this embodiment.

[0023] Figure 10 This is a diagram illustrating another operational input for user viewpoint movement in the XY plane according to this embodiment.

[0024] Figure 11 An example of a display screen is shown in which operation input for user viewpoint movement is performed in the XY plane according to this embodiment.

[0025] Figure 12 This is a diagram illustrating the operation input for moving the user's viewpoint in the Z direction according to this embodiment.

[0026] Figure 13 This is a diagram illustrating the operation input for rotating the user's viewpoint in the yaw direction according to this embodiment.

[0027] Figure 14 An example of a display screen is shown in which operation input for user viewpoint movement is performed during yaw direction rotation according to this embodiment.

[0028] Figure 15 This is a diagram used to illustrate the determination of the moving direction according to this embodiment.

[0029] Figure 16 An example of the indication of coordinate information during movement or rotation according to this embodiment is shown.

[0030] Figure 17 An example of the deformation of the shape of the operation reference area image 450 caused by user operation according to this embodiment is shown.

[0031] Figure 18 An example of the deformation of the shape of the operation reference area image 450 caused by user operation according to this embodiment is shown.

[0032] Figure 19 This is a diagram used to illustrate the different observation areas according to the FoV difference in this embodiment. Detailed Implementation

[0033] In the following, advantageous embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components with substantially the same functional structure are labeled with the same reference numerals, and thus repeated descriptions thereof are omitted.

[0034] In addition, instructions are provided in the following order.

[0035] 1. Configuration

[0036] 1-1. System Configuration

[0037] 1-2. HMD 20 Configuration

[0038] 2. Operation Processing

[0039] 3. Regarding the movement and rotation of the user's viewpoint

[0040] 3-1. Control in each direction of movement and rotation

[0041] 3-2. Movement in the XY plane and the Z direction

[0042] 3-3. Movement in the yaw direction

[0043] 3-4. Determining the direction of movement

[0044] 4. Modified Example

[0045] 4-1. Adjustment of movement speed and circumferential view speed

[0046] 4-2. Indication of coordinate information in virtual space

[0047] 4-3. Setting the reference area image 450

[0048] 4-4. Measures for handling unstable test results

[0049] 4-5. Based on the changes in movement speed and rotational angular velocity within the HMD 20's field of view.

[0050] 4-6. Practicality in Mechanical CAD

[0051] 4-7. Other

[0052] 5. Supplementary Explanation

[0053] <1. Configuration>

[0054] The information processing system according to embodiments of this disclosure is explained in detail with reference to the accompanying drawings.

[0055] <<1-1. System Configuration>>

[0056] Figure 1 The overall configuration of an information processing system 1 according to an embodiment of the present disclosure is shown. The information processing system 1 according to this embodiment includes a ring device 10 and an HMD (head-mounted display) 20.

[0057] The ring device 10 is worn on any of the user's fingers. A clickable button (switch 110) is mounted on the ring device 10. The ring device 10 transmits press information (click information) regarding the switch 110 to the HMD 20 in real time. The position / posture of the ring device 10 can be detected by various types of sensors installed on the ring device 10. In the HMD 20, the position / posture of the ring device 10 can be detected as the position / posture of the user's hand.

[0058] HMD 20 is a display device (example of an information processing device) worn on a user's head and having a display positioned in front of the user's eyes. HMD 20 can be configured to cover the user's entire field of vision when worn. HMD 20 can display video from the user's viewpoint in virtual space. Furthermore, HMD 20 can be a video perspective HMD, in which case virtual objects are overlaid on images captured in real space.

[0059] In this embodiment, users can perform operations on the virtual space with one hand, especially three-dimensional user viewpoint movement operations, and the burden of wearing a mask can be reduced.

[0060] Up to this point, an overview of the information processing system 1 according to embodiments of the present disclosure has been explained. Next, the structure of the HMD 20 included in the information processing system 1 according to the present embodiment will be described with reference to the accompanying drawings.

[0061] <<1-2. Structure of HMD 20>>

[0062] Figure 2 This is a block diagram illustrating an example of the structure of the HMD 20 according to this embodiment. Figure 2 As shown, the HMD 20 includes a communication unit 210, a ToF sensor 220, a camera device 230, an inertial sensor 240, a control unit 250, a display unit 260, and a storage device unit 270.

[0063] (Communications Department 210)

[0064] The communication unit 210 includes a transmitting unit for sending data to external devices and a receiving unit for receiving data from external devices. For example, the communication unit 210 according to this embodiment can communicate with external devices or the Internet using wired or wireless LAN (local area network), Wi-Fi, Bluetooth, mobile communication networks (LTE (Long Term Evolution), 5G (Fifth Generation Mobile Communications System)), etc.

[0065] (ToF sensor 220)

[0066] The Time-of-Flight (ToF) sensor 220 is a sensor that uses infrared light to detect the distance to a target object. The ToF sensor 220 detects the distance to objects around the user wearing the HMD 20. Multiple ToF sensors 220 can also be configured. Based on the information output from the ToF sensor 220, for example, the position / pose of the user's hand / finger and the distance to obstacles near the user can be calculated.

[0067] (Camera device 230)

[0068] The camera device 230 is an image capturing unit that captures images of the area around it. The camera device 230 can be a stereo camera device. Multiple camera devices 230 can be installed. The information output from the camera device 230 can be displayed as a perspective image on the display unit 260, or it can be used to calculate the position / posture of the user's hand / finger.

[0069] (Inertial sensor 240)

[0070] Inertial sensor 240 is a sensor that detects three-dimensional inertial motion. Inertial sensor 240 has a three-axis gyroscope sensor and a three-axis accelerometer sensor. Based on the information output from inertial sensor 240, the posture of HMD 20 (i.e., the posture of the user's head) can be calculated.

[0071] (Control Unit 250)

[0072] The control unit 250 serves as a computing and control device, and controls the overall operation of the HMD 20 according to various types of programs. For example, the control unit 250 is implemented by electronic circuitry such as a CPU (Central Processing Unit) or a microprocessor. In addition, the control unit 250 may include ROM (Read-Only Memory) for storing programs to be used, operation parameters, etc., and RAM (Random Access Memory) for temporarily storing parameters that are changed appropriately.

[0073] In addition, the control unit 250 can also be used as a hand / finger position / posture calculation unit 251 and a display control unit 252.

[0074] The hand / finger position / pose calculation unit 251 calculates the user's hand / finger position / pose based on information acquired from at least one of the ToF sensor 220 and the camera device 230. For example, the hand / finger position / pose calculation unit 251 estimates information about the joints and skeletal structure of the hand / finger based on information output from the ToF sensor 220 and the camera device 230. The hand / finger position / pose calculation unit 251 is an example of an acquisition unit that acquires information representing the position and pose of the user's hand / finger (an example of an operating component). Note that the sensing data of the user's hand / finger can be transmitted at the ring device 10, and based on this, the hand / finger position / pose can be calculated at the hand / finger position / pose calculation unit 251. Furthermore, the user's hand / finger position / pose can be calculated at the ring device 10, and the hand / finger position / pose calculation unit 251 can acquire this position / pose.

[0075] The display control unit 252 performs control to display a predetermined image on the display unit 260.

[0076] For example, the display control unit 252 can display (perspective display) a captured image (video) taken by the camera device 230 in the user's line of sight (the direction in front of the HMD 20) on the display unit 260 in real time, and use the display unit 260 as a video perspective display. Furthermore, the display control unit 252 can overlay virtual objects onto the captured image in real space on the display unit 260 to provide an AR (augmented reality) space. Alternatively, the display control unit 252 can also overlay a video (virtual space video) from the user's viewpoint in a virtual space with a hidden background onto a perspective image, instead of performing perspective display. For example, the display control unit 252 can display a virtual object representing a work target, and allow the user to change the position or orientation of the virtual object or edit the virtual object.

[0077] Note that the display unit 260 can be an optically perceptible display. In this case, the display control unit 252 can also overlay the video from the user's viewpoint in the virtual space onto the real space view.

[0078] Furthermore, the display control unit 252 can enable the display unit 260 to function as a fully immersive display by stopping the perspective display and displaying an image of the virtual space. Thus, the display control unit 252 can provide a VR (virtual reality) space.

[0079] The display control unit 252 can reflect the operation input performed by the user's hand / finger movements in the display control of the virtual object. The user can perform operations on the virtual object by raising their arm in front of their body and moving their hand / finger in the air. For example, if the display unit 260 is a video perspective type display unit, a real-space image of the user's face in frontal direction is displayed in real time on the display unit 260. Therefore, while watching a video of their hand / finger and the overlaid virtual object, the user can perform actions such as grasping the virtual object, moving the virtual object in any direction while the user is grasping it. Information about the user's hand / finger movements is obtained by the hand / finger position / pose calculation unit 251.

[0080] Note that although an example of a hand / finger as an operating component is described in this embodiment, the operating component is not limited to this. For example, the operating component can be a controller held by the user. In the case where the operating component is a controller, the position / pose of the controller is detected by an inertial sensor mounted on the controller and transmitted to the HMD 20 in real time. In the HMD 20, the position / pose of the controller can be used as the position / pose of the user's hand / finger.

[0081] Here, the operations that the user can perform include operations for moving the user's viewpoint in the virtual space (i.e., self-positioning movement), as well as operations on virtual objects similar to those described above. Self-positioning movement in the virtual space requires various operations, including movement in the up-down, left-right, or forward-backward directions, and yaw rotation as a circumferential movement in the horizontal direction.

[0082] The display control unit 252 according to this embodiment can reflect the operation input of the user's hand / finger movements in the user's viewpoint movement control in virtual space (i.e., virtual space video display control). Specifically, when a predetermined trigger is detected, the display control unit 252 displays an operation reference area image centered on the position of the user's hand (specifically, for example, the second and third joints of the index finger (so-called proximal joints)).

[0083] A predetermined trigger can be considered as an operation that invokes the user viewpoint movement control function according to this embodiment. For example, a predetermined trigger may be the detection of a predetermined hand / finger movement (e.g., snapping fingers twice, tapping the fingertips of the index finger and thumb twice, etc.) or a press (e.g., double-clicking) provided to the switch portion 110 of the ring device 10 worn on the user's finger. As long as the state of executing the predetermined trigger continues, the display control unit 252 continues to display the operation reference area image. For example, the operation reference area image continues to be displayed in the state of the posture continuing after snapping fingers twice, in the state of the fingertips of the index finger and thumb touching each other, or in the state of the press continuing after the second click on the switch portion 110 of the ring device 10. Here, refer to Figure 3 Please provide a detailed explanation of the operation reference area display.

[0084] Figure 3 This is a diagram used to illustrate the display of the operation reference area according to this embodiment. For example... Figure 3 As shown, the display unit 260 displays a real-time captured image 310 of the real space, and overlays a virtual object 400 onto the captured image 310. The user's hand 320 viewed on the display unit 260 can be a virtual hand (CG image) overlaid on the captured image 310 of the user's hand. The virtual hand is displayed to track the position of the user's hand. Alternatively, a virtual object (CG image) corresponding to the ring device 10 can be worn on the fingers of the virtual hand.

[0085] For example, the display control unit 252 displays an operation reference area image 450 centered on the second and third joints (so-called proximal joints) of the user's index finger as a reference position. The shape of the operation reference area image 450 is not particularly limited. For example, the operation reference area image 450 can be a spherical virtual object (3D CG image).

[0086] As described above, as long as the state of executing the predetermined trigger continues, the operation reference area image 450 continues to be displayed at the reference position. That is, when the state of executing the predetermined trigger (for example, the state of pressing the switch part 110 of the ring device 10 a second time) is in effect, the operation reference area image 450 continues to be displayed at the reference position while the user's hand moves in the up-down, left-right, or back-forward direction.

[0087] When a user's hand moves out of the operation reference area image 450 while a predetermined trigger is being executed (i.e., the user's hand moves a predetermined distance from the reference position), the display control unit 252 executes control to move the user's viewpoint in the vertical, horizontal, or forward / backward directions based on the direction of the user's hand movement. The threshold (e.g., 10 cm) used to initiate movement in the vertical, horizontal, or forward / backward directions is the same as the radius of the spherical virtual object displayed as the operation reference area image 450. The movement of the hand out of the operation reference area image 450 is equivalent to the hand moving more than the threshold distance from the reference position. The display of the operation reference area image 450 indicates the threshold (distance from the reference position) used to initiate movement in the vertical, horizontal, or forward / backward directions, and the user can intuitively understand how far the hand must move to perform the user viewpoint movement operation. Details of the user viewpoint movement operation in the vertical, horizontal, or forward / backward directions will be discussed later. Figures 5 to 12 Describe it.

[0088] Furthermore, if, in the operation reference area image 450, the user's hand has rotated around the forearm by a predetermined angle (e.g., 30 degrees) or more relative to the posture of the user's hand at a predetermined reference position (reference posture), the display control unit 252 executes control to rotate the user's viewpoint in the yaw direction. By displaying the operation reference area image 450 as a stereoscopic image (sphere, cube, etc.), space for panoramic operation input can be ensured. Details regarding the rotation operation of the user's viewpoint in the yaw direction (panoramic operation) will be referred to later. Figures 13 to 15 Describe it.

[0089] When the user's hand has moved out of the operation reference area image 450, when the user's hand has rotated within the operation reference area image 450 by a predetermined angle or more, or in other cases, the display control unit 252 may change the display mode of the operation reference area image 450 to provide operational feedback in response to the user's hand (an example of an operation component) movement. Examples of changing the display mode include changes to the color, density, transparency, pattern, shape, animation, etc., of the operation reference area image 450. For example, when the hand neither moves nor rotates, the operation reference area image 450 may be displayed as "green"; when the hand moves out of the operation reference area image 450, it may be displayed as "orange"; when the hand has rotated within the operation reference area image 450 by a predetermined angle or more, it may be displayed as "blue"; and when the hand moves out of the operation reference area image 450 and has rotated by a predetermined angle or more, it may be displayed as "purple".

[0090] The user viewpoint movement speed and rotation speed of the display control unit 252 can be predefined. For example, examples of movement speed include 1.2 meters per second, and examples of rotation angular velocity include 60 degrees per second.

[0091] As described above, in this embodiment, the operation input for viewpoint movement is achieved solely through the user's hand / finger movements or by combining a device such as the ring device 10 with a switch 110 with a minimum number of operating components, thereby reducing the physical burden on the user.

[0092] (Display unit 260)

[0093] The display unit 260 has the function of displaying images. For example, the display unit 260 is configured to cover the user's field of vision. Furthermore, the display unit 260 can be a video perspective display or an optical perspective display. In the case where the HMD 20 is implemented as an eyeglass-type device, the display unit 260 can be a transmissive display disposed on the lens portion of the eyeglasses.

[0094] (Storage device section 270)

[0095] The storage device unit 270 is implemented by ROM storing programs, operation parameters, etc. used in the processing executed by the control unit 250 and RAM temporarily storing parameters, etc., that need to be changed.

[0096] The configuration of the HMD 20 has been explained in detail so far. The configuration of the HMD 20 according to this disclosure is not limited to... Figure 2 The example shown. For example, HMD 20 does not necessarily have... Figure 2 All the components shown are illustrated. Furthermore, while the ToF sensor 220, camera 230, and inertial sensor 240 are examples of sensors for acquiring information about the position / pose of the user's hand / finger and the position / pose of the user's head, this embodiment is not limited to these. The HMD 20 can use other sensors to acquire information about the position / pose of the user's hand / finger and the pose of the user's head. Additionally, the HMD 20 can be implemented using multiple devices.

[0097] <2. Operation Processing>

[0098] Next, the operation processing performed by the information processing system according to this embodiment will be described.

[0099] Figure 4 This is a flowchart illustrating an example of the user viewpoint movement control process according to this embodiment.

[0100] like Figure 4As shown, HMD 20 first checks whether the switch section 110 of the ring device 10 has been pressed (step S103). If the switch section 110 of the ring device 10 has been pressed, an activation signal is sent from the ring device 10 to HMD 20. Here, it is assumed that the switch section 110 has been double-clicked as a trigger for receiving user viewpoint movement operations. Therefore, HMD 20 confirms whether the switch section 110 of the ring device 10 has been pressed twice. In addition, while the switch section 110 continues to be pressed, activation signals are continuously sent from the ring device 10 to HMD 20. HMD 20 performs the user viewpoint movement control processing described below while the switch section 110 is pressed a second time.

[0101] When the switch of the ring device 10 is pressed again, the HMD 20 causes the hand / finger position / posture calculation unit 251 to continuously calculate the hand / finger position / posture based on various types of data detected by the ToF sensor 220 and the camera device 230 installed on the HMD 20 (step S106).

[0102] Next, the display control unit 252 of the HMD 20 determines the position of the index finger (more specifically, the point of the second joint, etc.) as a reference position based on the position / posture information of the hand / finger, and displays an operation reference area image 450 centered on the reference position (step S109). Note that the reference position is an example and this embodiment is not limited thereto. For example, the reference position may be the center position of the hand.

[0103] Next, based on the hand / finger position / pose continuously calculated by the hand / finger position / pose calculation unit 251, the display control unit 252 confirms whether the user's hand has moved out of the operation reference area image 450 (whether the distance the hand moves from the reference position exceeds the threshold) (step S112).

[0104] Once the hand has moved out of the operation reference area image 450 (step S112 / Yes), the display control unit 252 executes control to move the user's viewpoint in the up / down, left / right, or forward / backward direction according to the direction of hand movement (step S115). The movement of the user's viewpoint in the up / down, left / right, or forward / backward directions is more specifically movement in the XY plane or in the Z direction. Details will be described later.

[0105] On the other hand, if the hand does not move out of the operation reference area image 450 (step S112 / No) and rotates by a predetermined angle or more (around the forearm) in the operation reference area image 450 (step S118 / Yes), the display control unit 252 performs control to rotate the user's viewpoint in the yaw direction in response to the hand rotation operation (step S121). Details of the yaw direction rotation control of the user's viewpoint will also be described later.

[0106] Then, the display control unit 252 moves or rotates the user's viewpoint and reflects it in the display control of the virtual space, and updates the display of the virtual space (step S124).

[0107] While continuing to press the switch section 110 of the ring device 10, the processing described in steps S106 to S124 above can be repeated. When the press is released, the user viewpoint movement control processing ends.

[0108] Notice, Figure 4 The operation process shown is an example, and this embodiment is not limited to it. For example, it is also possible to consider moving the hand out of the operation reference area image 450 and rotating it by a predetermined angle or more. In this case, the display control unit 252 can perform control to move the user's viewpoint in the up-down, left-right, or front-back directions in response to the hand's movement and rotate the user's viewpoint in the yaw direction.

[0109] <3. Regarding the movement and rotation of the user's viewpoint>

[0110] Next, the user viewpoint movement and rotation according to this embodiment will be described in detail.

[0111] <<3-1. Control in each direction of movement and rotation>>

[0112] In three-dimensional movement and rotation, there are degrees of freedom for movement in three directions: forward / backward, left / right, and up / down, and degrees of freedom for rotation in three directions: yaw / roll / pitch. However, in terms of user movement, the directions of movement and rotation are not necessarily equivalent. For example, diagonal movement on a horizontal plane is common in daily activities, but diagonal movement in directions including the vertical is difficult to perform freely and is often limited to situations such as ramps, stairs, or escalators in daily activities. Furthermore, regarding rotation, yaw rotation is based on commonly performed operations, as it is a panoramic operation of turning the head left / right. On the other hand, roll and pitch rotation are side-flip or back-support loop operations, which are rarely performed and involve significant changes in the angle between the user and the horizontal plane, undesirably leading to operations with significant discomfort. Therefore, in this embodiment, the directions of movement and rotation are restricted in the following manner.

[0113] Figure 5 This is a diagram used to illustrate the movement in the XY direction (horizontal direction) according to this embodiment. For example... Figure 5As shown, in this embodiment, in response to the movement of the user U's hand 410 in the XY direction (horizontal direction), the user's viewpoint in the XY plane is allowed to move, including oblique movements (e.g., arrow directions D1 and D2, etc.).

[0114] Figure 6 This is a diagram used to illustrate the movement in the Z-direction (vertical direction) according to this embodiment. For example... Figure 6 As shown, in this embodiment, in response to the movement of the user U's hand 410 in the Z direction (vertical direction), only the user's viewpoint movement in the Z direction is allowed, and oblique user viewpoint movement including horizontal movement components is prohibited.

[0115] Figure 7 This is a diagram used to illustrate the rotation according to this embodiment. For example... Figure 7 As shown, in the user viewpoint rotation control according to this embodiment, rotation control in the yaw direction is allowed (and the coexistence of movement in the yaw direction is also allowed); on the other hand, rotation including roll / pitch components (oblique rotation) is prohibited. Furthermore, roll / pitch rotation is prohibited in user viewpoint rotation. Here, user viewpoint panoramic rotation is only allowed in the yaw direction corresponding to rotation in the horizontal direction, and rotating a person's hand in the yaw direction during operation input is anatomically difficult. Therefore, regarding the operation input for panoramic rotation according to this embodiment, a rotation operation (twist) of the hand around the forearm is used. Although the operation input uses a rotation axis different from the rotation direction (yaw direction) in which the panoramic operation is desired, the operation input for panoramic rotation can be performed through anatomically natural manipulation.

[0116] <<3-2. Movement in the XY plane and movement in the Z direction>>

[0117] Next, we will explain in detail the movement in the XY plane and the movement in the Z direction. The "movement of the user's viewpoint in the vertical, horizontal, or forward / backward directions" described above is more specifically the movement in the XY plane or the movement in the Z direction.

[0118] (Control of movement in the XY plane)

[0119] Figure 8 This is a diagram illustrating the operation input for user viewpoint movement in the XY plane according to this embodiment. Figure 8 The upper left and right figures show the virtual object Obj configured in virtual space V and the user's viewpoint (U-eye). Figure 8The lower part of the left and right figures shows the user's hand 410 and the operation reference area image 450 displayed at the reference position. Both figures are top views. Here, in the user's gaze movement in the forward and backward directions in the XY plane, the operation input for the user's gaze movement in the backward direction is used as an example for illustration.

[0120] like Figure 8 As shown, when the user has pulled their hand 410 out of the operation reference area image 450, that is, has moved their hand 410 backward from the reference position and removed their hand from the operation reference area image 450, the display control unit 252 executes control to move the user's viewpoint (U-eye) backward. Note that as long as the hand 410 removed from the operation reference area image 450 remains outside the operation reference area image 450, the display control unit 252 continues to move the user's viewpoint (U-eye) backward. The display control unit 252 can continue to execute control to move the user's viewpoint (U-eye) until the pressure on the switch unit 110 of the ring device 10 is released. In addition, when there is a restriction on the area where the hand can move, such as a virtual space area, the display control unit 252 can continue to execute control to move the user's viewpoint (U-eye) to that restriction.

[0121] Figure 9 An example of a display screen is shown in which operation input for user viewpoint movement is performed in the XY plane according to this embodiment. (As already referred to...) Figure 8 As explained, since the user's viewpoint has moved backward due to the action of pulling the hand 410, the virtual object 400 appears to have moved relatively far on the display unit 260, such as... Figure 9 As shown. Additionally, at this time, the display mode of the operation reference area image 450 changes. This change in display mode of the operation reference area image 450 can continue even as the user's viewpoint (U-eye) is moving (i.e., the user's viewpoint is changing).

[0122] Figure 10 This is a diagram illustrating another operational input for user viewpoint movement in the XY plane according to this embodiment. Figure 10 The upper part shows the virtual object Obj configured in virtual space V and the user viewpoint (U-eye). Figure 10 The lower part shows the user's hand 410 and an operation reference area image 450 displayed at a reference position. Both figures are top views. Here, the operation input for the user's eye movement in the left-right direction observed by the user in the X direction is explained.

[0123] like Figure 10As shown, when the user moves their hand 410 to the left from the operation reference area image 450, that is, moves it to the left from the reference position, and the hand 410 moves out of the operation reference area image 450, the display control unit 252 executes control to move the user's viewpoint (U-eye) to the left. Note that as long as the hand 410, which has moved out of the operation reference area image 450, remains outside the operation reference area image 450, the display control unit 252 continues to move the user's viewpoint (U-eye) to the left. The display control unit 252 can continue to execute control to move the user's viewpoint (U-eye) until the pressure on the switch unit 110 of the ring device 10 is released. In addition, when there is a restriction on the area where the hand can move, such as a virtual space area, the display control unit 252 can continue to execute control to move the user's viewpoint (U-eye) to that restriction.

[0124] Figure 11 An example of a display screen is shown in which operation input for user viewpoint movement is performed in the XY plane according to this embodiment. (As already referred to...) Figure 10 As explained, because the user's viewpoint moves to the left due to the action of moving hand 410 to the left, the virtual object 400 appears to have moved relative to the right on the display unit 260, such as... Figure 11 As shown. Additionally, at this time, the display mode of the operation reference area image 450 changes. This change in display mode of the operation reference area image 450 can continue even as the user's viewpoint (U-eye) is moving (i.e., the user's viewpoint is changing).

[0125] (Movement control in the Z direction)

[0126] Figure 12 This is a diagram illustrating the operation input for user viewpoint movement in the Z direction according to this embodiment. Figure 12 The upper part shows the virtual object Obj configured in virtual space V and the user viewpoint (U-eye). Figure 12 The lower part shows the user's hand 410 and an operation reference area image 450 displayed at a reference position. Both are diagrams showing the user's view in the horizontal direction, with the X-axis representing the user's forward direction. Here, the operation input for the user's gaze moving in the upward direction is illustrated as an example of the user's gaze moving in the upward direction, within the vertical movement of the Z-axis.

[0127] When the user raises their hand 410 to a position near their chest in front of their body (reference position), the hand 410 then moves upward from the operation reference area image 450; that is, the hand 410 moves upward from the reference position, and the hand 410... Figure 12When the user moves out of the operating reference area image 450, the display control unit 252 performs control to move the user's viewpoint (U-eye) upwards. Note that as long as the hand 410, which has moved out of the operating reference area image 450, remains outside the operating reference area image 450, the display control unit 252 continues to move the user's viewpoint (U-eye) upwards. The display control unit 252 can perform control to move the user's viewpoint (U-eye) until the user's viewpoint reaches the limit of an area such as virtual space or the pressure on the switch 110 of the ring device 10 is released.

[0128] <<3-3. Movement in the yaw direction>>

[0129] Next, we will explain the rotation in the yaw direction in detail.

[0130] Figure 13 This is a diagram illustrating the operational input for rotating the user's viewpoint in the yaw direction according to this embodiment. Figure 13 The upper part shows the virtual object Obj configured in virtual space V and the user viewpoint (U-eye). Figure 13 The lower part shows the user's hand 410 and an operation reference area image 450 displayed at a reference position. Both figures are top views. Here, the operation input for counterclockwise user eye movement is illustrated as an example of clockwise and counterclockwise rotation of the user's gaze in the yaw direction when viewed from above in the vertical direction.

[0131] like Figure 13 As shown, when the user has rotated their hand 410 counterclockwise around the forearm observed in the operation reference area image 450, that is, twisted the hand counterclockwise relative to the reference posture, and rotated the hand by a predetermined angle or more, the display control unit 252 executes control to rotate the user's viewpoint (U-eye) counterclockwise in the yaw direction observed from above in the vertical direction. Note that as long as the hand 410 in the operation reference area image 450 remains in a state of twisting by a predetermined angle or more, the display control unit 252 continues to rotate the user's viewpoint (U-eye) counterclockwise. The display control unit 252 can execute control to move the user's viewpoint (U-eye) until the pressure on the switch 110 of the ring device 10 is released.

[0132] Figure 14 An example of a display screen is shown in which operation input for user viewpoint movement is performed during yaw direction rotation according to this embodiment. (As already referred to...) Figure 13As explained, since the user's viewpoint has already rotated counterclockwise in the yaw direction due to the counterclockwise twisting of the hand 410 (already looking around in the horizontal direction), the virtual object 400 appears to have moved relatively clockwise on the display unit 260, such as... Figure 14 As shown. Additionally, at this time, the display mode of the operation reference area image 450 changes. This change in display mode of the operation reference area image 450 can continue even as the user's viewpoint (U-eye) is rotating and moving (i.e., the user's viewpoint is changing).

[0133] <<3-4. Determining the Direction of Movement>>

[0134] The display control unit 252 obtains information about the hand's movement direction based on the hand / finger position / pose information calculated by the hand / finger position / pose calculation unit 251, and reflects this information in the user viewpoint movement control. The direction of user viewpoint movement, the direction in the XY plane, or the Z direction can be determined by decomposing the hand's movement vector into components in the XY plane and the Z direction, and selecting the component with a larger Euclidean norm. Figure 15 This is a diagram used to illustrate the determination of the moving direction according to this embodiment.

[0135] In such Figure 15 When the user's hand movement vector V1 is detected, the display control unit 252 decomposes the movement vector V1 into XY component vector V2 and Z component vector V3. Figure 15 In the example shown, since the Euclidean norm of the Z component vector V3 is greater than the Euclidean norm of the XY component vector V2, the display control unit 252 determines that the user's viewpoint will only be moved in the Z direction. Then, if the user's hand has moved out of the operation reference area image 450, the display control unit 252 starts user viewpoint movement control; otherwise, movement control is not performed.

[0136] <4. Modified Example>

[0137] <<4-1. Adjustment of Movement Speed ​​and Around-the-View Speed>>

[0138] The display control unit 252 changes the movement speed or the circumferential viewing speed based on the distance the hand moves from the center (reference position) of the operation reference area image 450, the rotation angle relative to the reference posture, and the duration of the operation input state.

[0139] For example, the display control unit 252 increases the speed of user viewpoint movement as the distance the hand moves relative to the reference position increases. Additionally, the display control unit 252 increases the speed of user viewpoint movement as the rotation angle of the hand relative to the reference posture increases. Furthermore, the display control unit 252 increases the speed of user viewpoint movement as the time since the hand left the operation reference area image 450 increases (the duration of the operation input state increases).

[0140] This allows for a dynamic and easy switching between the need for rapid movement over long distances and the need for fine-tuned movement over short distances.

[0141] <<4-2. Indication of Coordinate Information in Virtual Space>>

[0142] In augmented reality (AR) HMDs, such as those using video perspective or optical perspective schemes, when the user's viewpoint (position) moves or rotates, the user may have difficulty discerning whether the virtual object 400 has moved in the virtual space or whether its own position has moved or rotated.

[0143] Figure 16 An example of indicating coordinate information during movement or rotation according to this embodiment is shown. When movement or rotation is performed, the display control unit 252 enables the recognition of coordinate information in virtual space, and displays grid lines G additionally on the display screen, for example, as shown. Figure 16 As shown, this improves usability.

[0144] <<4-3. Setting the reference area image to 450>>

[0145] The size of the operation reference area image 450 (the threshold in a movement operation from the reference position) can be set to a value suitable for the user. The larger the size of the operation reference area image 450, the more effectively unintentional movement is reduced when the hand moves unintentionally. However, if the size is too large, the distance required to move the hand out of the operation reference area image 450 is longer, resulting in a greater operational burden.

[0146] In view of this, for example, physical characteristics such as the user's arm length can be pre-input, and adjustments such as increasing the size of the operation reference area image 450 for users with long arms can be made, since users with long arms tend to make larger hand movements.

[0147] Furthermore, the display control unit 252 can identify whether the user is at a desk and whether there are any obstacles (real objects) around them. The display control unit 252 can pre-position the operation reference area image 450 slightly upwards or reduce the size of the lower area of ​​the operation reference area image 450, because if the user is at a desk, it is difficult for the user to move their hand downwards. Since the hand actually moves in the air in this operation input method, for example, when the user is working at a desk, the actual area available for operation varies depending on the situation due to the presence of real objects such as the desk or a drink placed on it. In this embodiment, the shape of the operation reference area image 450 can be changed according to the area where the hand can move. By changing the size of the operation reference area image 450 according to the surrounding situation, user usability is improved.

[0148] Furthermore, assuming that the user experiences hand fatigue during continuous use, the display control unit 252 can reduce the size of the operation reference area image 450 and change the threshold for initiating movement.

[0149] Furthermore, the display control unit 252 can obtain the user's usage history in advance, and when the usage time is longer, the display control unit 252 can assume that the user has become accustomed to the operation and reduce the size of the operation reference area image 450.

[0150] In addition, the display control unit 252 can obtain the user's usage history in advance, and if the number of surround view operations is high, the display control unit 252 can increase the size of the operation reference area image 450, making it easier for the user to perform surround view operations and reducing the possibility of movement in the up-down, left-right, or forward-backward directions.

[0151] Furthermore, the display control unit 252 can change the size of the operation reference area image 450 based on the performance (hand tracking and detection performance) of the hand / finger position / pose calculation unit 251, which is based on the sensors, CPU, etc. used. For example, if the hand tracking and detection performance is high, the display control unit 252 can reduce the size of the operation reference area image 450 because even subtle hand movements can be detected. On the other hand, if the detection performance is low, the display control unit 252 can increase the size of the operation reference area image 450, allowing the user to make correspondingly larger hand movements.

[0152] Furthermore, as feedback to user-performed inputs, the display control unit 252 can not only change the color of the operation reference area image 450, but also change its shape, flash the display, or generate sound or vibration. This makes it easier for the user to perceive the direction, amount, and rotation of their hand movements. Figure 17An example of the shape deformation of the operation reference area image 450 caused by user operation according to this embodiment is shown. Figure 17 As shown, when the user's hand moves out of the operation reference area image 450, by making a portion of the operation reference area image 450 protrude as if it were being pulled toward the position of the user's finger, the user can more easily perceive the distance and direction of the operation. Figure 18 An example of shape deformation of the operation reference area image 450 caused by user operation according to this embodiment is shown. (By means such as...) Figure 18 As shown, when the user's hand is twisted in the operation reference area image 450, the operation reference area image 450 is changed into a twisted shape, making it easier for the user to perceive the rotation angle during operation.

[0153] <<4-4. Measures for handling unstable test results>>

[0154] If there are obstructions between the user's hand / finger and various types of sensors (ToF sensor 220 and camera device 230) of the HMD 20 while the HMD 20 is tracking the user's hand / finger (the hand / finger position / posture calculation unit 251 continuously calculates the hand / finger position / posture), the detection results may become unstable in some cases due to loss of hand / finger position or other reasons.

[0155] In this embodiment, if the tracking result is lost or an unstable detection result that fluctuates significantly relative to the previous value has been obtained, the following measures are taken to prevent operations that deviate from the user's intention.

[0156] First, even when the switch section 110 of the pressing ring device 10 is in the open state, the display control unit 252 hides the operation reference area image 450. Furthermore, the display control unit 252 stops the user's viewpoint movement and rotation at the point when the detection results become unstable. Then, the display control unit 252 returns to normal processing once stable tracking results have been obtained.

[0157] <<4-5. Movement speed and rotational speed change according to the HMD 20's field of view>>

[0158] In an HMD 20, there is a field of view that the user can see on the display, and this field of view is generally narrower than the field of view when the user is not wearing an HMD 20. Even if the yaw rotation angular velocity is slightly faster, there is no problem as long as the field of view (FoV) is wide; however, with a narrow FoV, if the yaw rotation angular velocity is fast, the immediately visible area is narrow, making it difficult to adjust in the direction the user wants to view. This also applies to movement in the XY plane and movement in the Z direction.

[0159] Figure 19 This is a diagram used to illustrate the different observation areas according to the FoV difference in this embodiment. Figure 19 The upper part shows the observation area 265 in the case of a wide FoV, and Figure 19 The lower part shows the observation area 266 in the case of a narrow FoV. The width of the observation area 265 is greater than the width of the observation area 266.

[0160] In such Figure 19 In the case of a wider FoV shown in the upper part, it is possible that a portion of the observed area 265-1 before the viewpoint movement and a portion of the observed area 265-2 after the viewpoint movement may overlap. If a portion of the observed area 265-1 before the viewpoint movement is captured in the observed area 265-2 after the viewpoint movement, there is no problem even if the panoramic rotation angular velocity is slightly faster, and the user can easily identify where they are looking. On the other hand, as... Figure 19 As shown in the lower part of the diagram, with a narrower FoV, it is unlikely that the observed area 266-1 before the viewpoint movement and the observed area 266-2 after the viewpoint movement will overlap. Because the observed area 266-2 after the viewpoint movement is discontinuous with the observed area 266-1 immediately preceding the viewpoint movement, if the panoramic rotation angular velocity increases, the user will have difficulty identifying where they are looking.

[0161] Therefore, the display control unit 252 can improve user usability by adjusting the yaw rotation angular velocity and the movement speed in the XY and Z directions according to the width of the FoV of the HMD 20. Specifically, when the width of the FoV of the HMD 20 is narrower than a predetermined value, the display control unit 252 can adjust the yaw rotation angular velocity and the movement speed in the XY and Z directions to be slower than the initial values.

[0162] <<4-6. Practicality in Mechanical CAD>>

[0163] In existing 3D games, three-dimensional movement and yaw rotation are achieved using two controllers, one for the left hand and one for the right. However, in 3D drafting applications for mechanical design, the operations that need to be performed include: selecting parts after performing movement or panning actions; creating cross-sections to inspect those cross-sections; and measuring distances between parts. Therefore, it is advantageous to allow the user to perform such operations with the other hand after performing movement, panning, etc. Since the input for three-dimensional movement and yaw rotation in this embodiment can be performed with only one hand, this embodiment is useful for 3D drafting applications in mechanical design.

[0164] <<4-7 Others>>

[0165] During movement in the XY axis direction, the display control unit 252 may also, similar to the Z axis, only allow movement in the X or Y axis direction, but not oblique movement in the XY plane.

[0166] When displaying the operation reference area image 450, the display control unit 252 can display guidance indicating the direction of movement and a virtual panoramic view corresponding to hand movements. This guidance display improves usability.

[0167] Since continuous movement or circling motions can induce nausea, the display control unit 252 can present continuous movement or circling motions as intermittent movements or intermittent circling motions with intervals of 0.5 seconds, instead of continuous movement, in order to reduce nausea during movement and circling motions. Similarly, to reduce nausea during movement and circling motions, the display control unit 252 can darken the peripheral area of ​​the field of view and add effects such as focus lines to facilitate visual recognition of movement speed.

[0168] For example, assuming it is used in mechanical CAD, the movement distance or the panoramic angle can be limited to values ​​such as 1-meter increments for movement and values ​​such as 5-degree increments for panoramic angle.

[0169] The display control unit 252 can change the movement speed and the circumferential viewing speed based on the user's position in the virtual space or their positional relationship with a virtual object. For example, the display control unit 252 can temporarily stop or slow down the vertical movement when the user's hand has reached the floor height. Furthermore, if the user wants to closely observe the area around the virtual object when their hand is close to it, the display control unit 252 can similarly stop or slow down the movement at that point.

[0170] The invocation of the movement and look-around action functions is not limited to double-clicking the ring device 10, and can be performed by another input method, such as a specific gesture or voice command such as opening the hand.

[0171] The display control unit 252 can pre-store the state before the movement and view actions begin, and can return to that state even after a movement or view action has been performed. Since it allows immediate return to the previous state, recovery in case of operation failure and comparison with the previous viewpoint can be easily performed.

[0172] The upper limits for movement speed and surround view speed can be changed based on the size of the virtual object or the spatial dimensions of the real and virtual worlds.

[0173] Furthermore, although in the above embodiment, the user's viewpoint's panoramic action (yaw direction rotation) is interpreted as being performed in response to rotation around the user's forearm, this embodiment is not limited to this. The panoramic action may be performed in response to rotation around a line parallel to another direction, or it may be performed in response to an operation on a touchpad provided on the ring device 10.

[0174] <5. Supplementary Explanation>

[0175] Although advantageous embodiments of this disclosure have been explained in detail with reference to the accompanying drawings, the technology is not limited to the examples. It will be apparent to those skilled in the art that various modifications or alterations can be conceived within the scope of the technical concept set forth in the claims, and it should be understood that such modifications or alterations are of course within the technical scope of this disclosure.

[0176] In addition, one or more computer programs can be created to enable the hardware built into the HMD 20, such as the CPU, ROM, or RAM, to perform the functions of the HMD 20. Furthermore, a computer-readable storage medium is provided on which one or more computer programs are stored.

[0177] Furthermore, the advantages described herein are for illustrative purposes only and are not intended to be limiting. That is, the technology according to this disclosure may, together with or in lieu of the foregoing advantages, exhibit other advantages that will be apparent to those skilled in the art based on the description herein.

[0178] Note that this technology can also be configured as follows.

[0179] (1) An information processing device, comprising:

[0180] The acquisition unit acquires information indicating the position and orientation of the operating component; and

[0181] The display control unit determines a reference position for operation based on the position of the operating component, and performs control to display an operation reference area image at the reference position, indicating a predetermined area to be used as an operation reference.

[0182] The display control unit performs control to change the user's viewpoint in the virtual space based on changes in the position or posture of the operating component relative to the operating reference area image.

[0183] (2) The information processing apparatus according to (1), wherein

[0184] When the operating component has been moved out of the operating reference area image, the display control unit performs control to move the user's viewpoint according to the direction of movement from the reference position.

[0185] (3) The information processing apparatus according to (1) or (2), wherein

[0186] When the operating component has rotated by a predetermined angle or more from the reference posture at the time of determining the reference position in the operating reference area image, the display control unit performs control to rotate the user's viewpoint according to the rotation direction.

[0187] (4) The information processing apparatus according to (1) to (3), wherein

[0188] The display control unit restricts changes in some directions of the user's viewpoint's movement or rotation.

[0189] (5) The information processing apparatus according to (4), wherein

[0190] The display control unit allows the user's viewpoint to move obliquely in the XY plane and prohibits the user's viewpoint from moving obliquely in the Z direction.

[0191] (6) The information processing apparatus according to (4) or (5), wherein

[0192] The display control unit only allows the user's viewpoint to rotate in the yaw direction, and prohibits the user's viewpoint from rotating in the roll and pitch directions.

[0193] (7) The information processing apparatus according to (6), wherein

[0194] The display control unit performs control to rotate the user's viewpoint in the virtual space in the yaw direction in response to the rotation of the operating component about the user's forearm.

[0195] (8) The information processing apparatus according to (1) to (7), wherein

[0196] As long as the operating component remains outside the operating reference area image, the display control unit continues to move the user's viewpoint.

[0197] (9) The information processing apparatus according to (1) to (8), wherein

[0198] As long as the state in which the operating component has been rotated by a predetermined angle or more in the operating reference area image continues, the display control unit continues to rotate the user's viewpoint.

[0199] (10) The information processing apparatus according to (1) to (9), wherein

[0200] The display control unit changes the display mode of the operation reference area image when the user's viewpoint is changed.

[0201] (11) The information processing apparatus according to (1) to (10), wherein

[0202] The display control unit changes the display mode of the operation reference area image in response to the operation of the operation component.

[0203] (12) The information processing apparatus according to (1) to (11), wherein

[0204] The display control unit changes the display mode of the operation reference area image based on real objects present in the vicinity.

[0205] (13) The information processing apparatus according to (1) to (12), wherein

[0206] Upon detecting a predetermined trigger, the display control unit begins to display the image of the operation reference area.

[0207] (14) The information processing apparatus according to (13), wherein

[0208] The predetermined trigger is a specific action of the operating component.

[0209] (15) The information processing apparatus according to (13), wherein

[0210] The operating components are the user's hand / fingers, and

[0211] The predetermined trigger is a click operation performed on the ring device worn on the user's finger.

[0212] (16) The information processing apparatus according to (1) to (15), wherein

[0213] The operational reference area image is a spherical virtual object.

[0214] (17) The information processing apparatus according to (1) to (16), wherein

[0215] The information processing device is an HMD worn on the user's head, and the HMD has a display positioned in front of the user's eyes.

[0216] (18) An information processing method executed by a processor, the information processing method comprising:

[0217] Acquire information about the position and orientation of the control components;

[0218] A reference position for operation is determined based on the position of the operating component, and control is executed to display an operation reference area image at the reference position, indicating a predetermined area to be used as an operation reference; and

[0219] The control is executed to change the user's viewpoint in the virtual space based on changes in the position or posture of the operating component relative to the operating reference area image.

[0220] (19) A storage medium storing a program that enables a computer to function as

[0221] The acquisition unit acquires information indicating the position and posture of the operating component;

[0222] The display control unit determines a reference position for operation based on the position of the operating component, and performs control to display an operation reference area image at the reference position, indicating a predetermined area to be used as an operation reference.

[0223] The display control unit performs control to change the user's viewpoint in the virtual space based on changes in the position or posture of the operating component relative to the operating reference area image.

[0224] Reference tag list

[0225] 10-ring device

[0226] 110 Switchgear

[0227] 20 HMD

[0228] 210 Ministry of Communications

[0229] 220 ToF sensor

[0230] 230 camera device

[0231] 240 Inertial Sensors

[0232] 250 Control Department

[0233] 251 Hand / Finger Position / Posture Calculation Section

[0234] 252 Display Control Unit

[0235] 260 Display Section

[0236] 270 Storage Devices Section

[0237] 400 Virtual Objects

[0238] 410 Hands

[0239] 450 Operational Reference Area Image

Claims

1. An information processing apparatus, comprising: The acquisition unit acquires information indicating the position and posture of the operating component; as well as The display control unit determines a reference position for operation based on the position of the operating component, and performs control to display an operation reference area image at the reference position, indicating a predetermined area to be used as an operation reference. The display control unit performs control to change the user's viewpoint in the virtual space based on changes in the position or posture of the operating component relative to the operating reference area image.

2. The information processing apparatus according to claim 1, wherein... When the operating component has moved out of the operating reference area image, the display control unit performs control to move the user's viewpoint according to the direction of movement from the reference position.

3. The information processing apparatus according to claim 1, wherein... When the operating component has rotated by a predetermined angle or more from the reference posture at the time of determining the reference position in the operating reference area image, the display control unit performs control to rotate the user's viewpoint according to the rotation direction.

4. The information processing apparatus according to claim 1, wherein... The display control unit restricts the user's viewpoint from changing in some directions, either in the direction of movement or rotation.

5. The information processing apparatus according to claim 4, wherein The display control unit allows the user's viewpoint to move obliquely in the XY plane and prohibits the user's viewpoint from moving obliquely in the Z direction.

6. The information processing apparatus according to claim 4, wherein The display control unit only allows the user's viewpoint to rotate in the yaw direction, and prohibits the user's viewpoint from rotating in the roll and pitch directions.

7. The information processing apparatus according to claim 6, wherein The display control unit performs control to rotate the user's viewpoint in the virtual space in the yaw direction in response to the rotation of the operating component about the user's forearm.

8. The information processing apparatus according to claim 1, wherein As long as the operating component remains outside the operating reference area image, the display control unit continues to move the user's viewpoint.

9. The information processing apparatus according to claim 1, wherein As long as the state in which the operating component has been rotated by a predetermined angle or more in the operating reference area image continues, the display control unit continues to rotate the user's viewpoint.

10. The information processing apparatus according to claim 1, wherein The display control unit changes the display mode of the operation reference area image when the user's viewpoint is changed.

11. The information processing apparatus according to claim 1, wherein The display control unit changes the display mode of the operation reference area image in response to the operation of the operation component.

12. The information processing apparatus according to claim 1, wherein The display control unit changes the display mode of the operation reference area image based on real objects present in the vicinity.

13. The information processing apparatus according to claim 1, wherein Upon detecting a predetermined trigger, the display control unit begins to display the image of the operation reference area.

14. The information processing apparatus according to claim 13, wherein The predetermined trigger is a specific action of the operating component.

15. The information processing apparatus according to claim 13, wherein The operating components are the user's hand / fingers, and The predetermined trigger is a click operation performed on the ring device worn on the user's finger.

16. The information processing apparatus according to claim 1, wherein The operational reference area image is a spherical virtual object.

17. The information processing apparatus according to claim 1, wherein The information processing device is an HMD worn on the user's head, and the HMD has a display positioned in front of the user's eyes.

18. An information processing method executed by a processor, the information processing method comprising: Acquire information about the position and orientation of the control components; A reference position for operation is determined based on the position of the operating component, and control is executed to display an operation reference area image at the reference position, indicating a predetermined area to be used as an operation reference. as well as The control is executed to change the user's viewpoint in the virtual space based on changes in the position or posture of the operating component relative to the operating reference area image.

19. A storage medium storing a program that causes a computer to: The acquisition unit acquires information indicating the position and posture of the operating component; The display control unit determines a reference position for operation based on the position of the operating component, and performs control to display an operation reference area image at the reference position, indicating a predetermined area to be used as an operation reference. The display control unit performs control to change the user's viewpoint in the virtual space based on changes in the position or posture of the operating component relative to the operating reference area image.

Images