Information processing apparatus, information processing system, information processing method, and program

Abstract

To provide an information processing apparatus that can improve operability by reducing a situation where a ray is directed to a direction not intended by a user.SOLUTION: An information processing apparatus of the present invention has: acquisition means that acquires the movement of a controller; and display control means that displays an indication position on the basis of the acquired movement. When the movement of the controller is a specific movement according to the operation of bending or extension of a finger, the display control means restricts the movement of the indication position based on the specific movement.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Conventionally, in cross-reality (XR) systems, when a user operates a UI via a head-mounted display (HMD), an operation method using virtual light rays called "rays" emitted from the hand or a controller has been utilized. Patent Document 1 also proposes a method in which a wearable device is worn on a finger and changes in the finger's posture are detected. The start point and direction of the ray can also be determined using the posture of the wearable device, which serves as a controller. Rays emitted from the hand and the controller can be used together, with the ray emitted from the hand being used when the hand can be photographed, and the ray emitted from the controller being used when the hand cannot be photographed. When operating using a ray, a ray can be emitted from a part of the palm of the hand as the start point in the direction of the palm, or a ray can be emitted from the controller as the start point with the thumb close to the controller worn on the finger. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-118929 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when operating using a ray, if the user changes posture or the starting point of the ray changes, the ray may point in a direction unintended by the user. Therefore, an object of the present invention is to provide an information processing device that can improve operability by reducing the possibility of the ray being pointed in a direction unintended by the user. [Means for solving the problem]

[0005] One aspect of the present invention is an information processing device having an acquisition means for acquiring a movement of a controller and a display control means for displaying an indication position based on the acquired movement, wherein the display control means restricts movement of the indication position based on the specific movement when the movement of the controller is a specific movement corresponding to the flexion or extension of a finger. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide an information processing device that can improve operability by reducing the likelihood of a ray being directed in a direction unintended by the user. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Figure 2] 1 is a diagram illustrating the internal configuration of an information processing device according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 5] 10 is a flowchart of a ray display process according to the first embodiment. [Figure 6] FIG. 3 is a diagram illustrating an example of motion detection according to the first embodiment. [Figure 7] 10 is a flowchart of a ray display process according to the second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of motion detection according to the second embodiment. [Figure 9] 10 is a flowchart of a ray display process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0009] (Embodiment 1) An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0010] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG (computer graphics) in a form that corresponds to the posture of the HMD 100.

[0011] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 via a wired connection such as a USB cable, or wirelessly via Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by combining a captured image with CG, and transmits the composite image to the HMD 100. Note that although a PC is described here as an example of an information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and the components of the PC 110 may be included in the HMD 100.

[0012] The controller 120 performs various controls of the HMD 100. If the PC 110 is in a specific control mode, when an operation is performed on the controller 120 by the user, the HMD 100 is controlled according to the user's operation. As shown in FIG. 1, the controller 120 can be considered to have a ring-shaped form that can be worn and supported on the user's finger. Further, the controller 120 has physical buttons (121, 122, 123) for performing determination operations and selection operations on the display. The controller 120 performs wireless communication with the PC 110 via Bluetooth.

[0013] The user can change the indicated position on the display according to the movement of the controller 120 by moving the controller 120. The indicated position may be represented by a point, or may be represented by a virtual ray (ray) connecting the point of the indicated position and the controller with a straight line (line segment) or a dotted line. By pressing any of the buttons 121, 122, 123, a menu determination operation or a selection operation can be performed. Although the buttons are physical buttons, they may be operable like a trackpad, a touch panel, a wheel, or a trackball, and in addition to pressing the buttons, a slide operation, a flick operation, or a touch-on operation may also be used.

[0014] <Internal Configuration of HMD> Referring to FIG. 2, the internal configuration of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, and a working memory 206.

[0015] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by synthesizing an imaging image captured by the imaging unit 202 of the space in front of the user and a CG) from the PC 110, the composite image is displayed on the image display unit 203. Instead of the HMD control unit 201 controlling the entire device, the entire device may be controlled by a plurality of hardware sharing the processing.

[0016] The imaging unit 202 includes two cameras (imaging devices). The two cameras are placed near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture video and images of a space similar to the space the user normally sees. Images of a subject (the area in front of the user) captured by the two cameras are output to the PC 110 and the control unit 201. Furthermore, the two cameras in the imaging unit 202 can acquire information on the distance from the two cameras to the subject as distance information by measuring distances using a stereo camera. Note that the imaging unit 202 may capture and output video.

[0017] The image display unit 203 displays a composite image. The image display unit 203 includes a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, an organic EL panel is disposed in front of each of the user's eyes. Note that a device using a semi-transmissive half mirror may also be used for the image display unit 203. In this case, for example, the image display unit 203 may use a technology generally called AR (Augmented Reality) to display an image so that CG appears to be directly superimposed on the real space visible through the half mirror. Furthermore, the image display unit 203 may use a technology generally called VR (Virtual Reality) to display an image of a completely virtual space without using captured images.

[0018] The orientation sensor unit 204 acquires orientation (and position) information of the HMD 100. Then, the orientation sensor unit 204 acquires orientation information of the user (the user wearing the HMD 100) that corresponds to the orientation (and position) of the HMD 100. The orientation sensor unit 204 has an inertial measurement unit (IMU) that is configured from an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The orientation sensor unit 204 is used when acquiring information on the user's orientation (orientation information), and the HMD control unit 201 outputs the information on the user's orientation (orientation information) to the PC 110.

[0019] The HMD control unit 201 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images acquired by the imaging unit 202. The joint points include characteristic points of parts such as the finger joints and fingertips, the back of the hand (palm), and the arm. Each joint point indicates a coordinate position, and the posture can be estimated from information on multiple joint points. Methods for estimating the position or posture of the hand and each joint point can include known object recognition and pose estimation techniques based on machine learning using, for example, a convolutional neural network. Depth position information of each joint point of the hand can be obtained by calculating the distance from the imaging unit 202 to each joint point, for example, by triangulation using stereo matching using the two camera images acquired by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the control unit 201 to the PC 110. The HMD control unit 201 may estimate the position and posture of each joint point of the user's hand and fingers.

[0020] The nonvolatile memory 205 is an electrically erasable and recordable nonvolatile memory, and stores programs to be executed by the control unit 101, which will be described later.

[0021] The volatile memory 206 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, an image display memory for the image display unit 203, a work area for the control unit 201, and the like.

[0022] The gaze imaging unit 207 is a camera that acquires an image for detecting the user's gaze, and is attached inside the HMD to capture an image of the user's eyes when the user wears the HMD 100. The image of the subject (user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the image captured by the gaze imaging unit 207, and identifies the location on the image display unit 203 that the user is gazing at.

[0023] <Internal configuration of the controller> Referring to FIG. 2, the internal configuration of the controller 120 will be described. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0024] The controller control unit 221 is a CPU that controls each component of the controller 120. Note that instead of the controller control unit 221 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0025] The operation unit 222 includes buttons. The operation unit 222 detects whether a button has been operated and transmits detection information to the PC 110 via the communication unit 223. Note that the operation unit 222 may have multiple input formats.

[0026] The communication unit 223 performs wireless communication with the PC 110 via Bluetooth. When there are multiple controllers, each performs wireless communication with the PC 110 via Bluetooth.

[0027] The controller attitude sensor unit 224 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 120. The detected position and attitude change information is communicated from the communication unit 223 to the PC 110 via the controller control unit 221.

[0028] The output unit 225 is composed of an LED light source, a speaker, a vibration element, etc.

[0029] <Internal Configuration of the PC> Referring to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 includes a control unit 211, a non-volatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0030] The control unit 211 is a CPU that controls each unit of the PC 110 in accordance with input signals and a program described below. Note that instead of the control unit 211 controlling the entire device, the entire device may be controlled by a plurality of hardware units sharing the processing. The control unit 211 receives, from the HMD 100, an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. The control unit 211 then composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0031] The control unit 211 also receives information estimated by the control unit 201 of the HMD 100. The received information is temporarily stored in the working memory 213.

[0032] The communication unit 214 receives information about a change in the position or posture of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 superimposes and displays an indication position according to the information about the change in the position or posture of the controller 120 on the combined image.

[0033] The nonvolatile memory 212 is an electrically erasable and recordable nonvolatile memory, and stores information such as programs to be described later and CG executed by the control unit 211. The control unit 211 can switch the CG read from the nonvolatile memory 212 (i.e., the CG used to generate a composite image).

[0034] The working memory 213 is used as a working area for the control unit 211, such as a buffer memory that temporarily stores image data captured by the imaging unit 202 and information on the estimated coordinate positions of each joint point of the hand, and a buffer memory that temporarily stores the starting point and inclination of the current ray.

[0035] The wrist joints may also be estimated in the PC 110. In this case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or posture of each joint point of the hand, processes the image using this information, and outputs it to the HMD 100. Note that the control unit 211 may also estimate the position and posture of each joint point of the hand, processes the image using this information, and outputs it to the HMD 100.

[0036] <About ray display processing> With reference to FIG. 3, the control and display of a ray using a user's hand will be described. Hereinafter, ray control using a user's hand will be referred to as a hand ray. The control unit 211 estimates (recognizes) the orientation of the user's hand, i.e., the posture of the user's hand, based on the color and shape of the user's hand shown in the captured image. At this time, the control unit 211 displays a CG image of a ray extending in the direction pointed by the index finger on the image display unit 203, as shown in FIG. 3. The control unit 211 then displays (moves) a pointer to the position where the ray is pointed. Therefore, the user can change the position and direction of the ray by changing the orientation of his or her hand. Note that the control unit 211 may display the ray based on a posture other than that of the index finger. For example, the ray may be projected from near the center of the palm. In this case, the user thrusts his or her palm forward and points it toward the object he or she wants to select.

[0037] Alternatively, the control unit 211 may estimate the position of a part of the user's body, such as the shoulder, from the position of the HMD 100, and display a ray in the direction connecting that position and the hand position.

[0038] By pressing button 121 of controller 120 while pointing the ray at a specific display item, a process of deciding to select the specific item indicated by the ray (decision process) is executed. Alternatively, for example, the decision operation may be performed by projecting the ray from between the index finger and thumb and bringing the index finger and thumb together.

[0039] FIG. 3 assumes that the subject, a hand 302, is within the imaging range of the HMD 100, i.e., the range of the image captured by the imaging unit 202 (hereinafter referred to as imaging range 301), and that the controller 120 is worn on the fingers of the user's hand 302. It is also assumed that the imaging range 301 is the screen viewed by the user on the image display unit 203 of the HMD 100. It is also assumed that on the screen viewed by the user, a ray 303 is superimposed as CG from the tip of the index finger of the hand 302, and multiple windows 304, 305, and 306 are displayed as if floating in space. The ray 303 extends from the tip of the finger as a reference point, i.e., a starting point. In FIG. 3, the ray 303 extends from the hand 302 toward the window 306 in the composite image, pointing to the position of the window 306. The user can control the direction of the ray by moving the hand 302, and can select a window by moving the tip of the ray 303. Specifically, the control unit 211 controls the direction of the CG ray 303 based on the captured image captured by the imaging unit 202. When the button 121 of the controller 120 is pressed with the ray 303 directed at one window 306, the control unit 211 performs processing corresponding to the window 306. For example, when window 306 is selected, an internet browser runs and specific search results are displayed. When window 304 is selected, an application capable of creating a document runs. When window 305 is selected, a video or music player application runs, allowing the user to listen to music, etc.

[0040] In Figure 3, the ray is controlled based on the part of the user's body that appears in the captured image, so it can be controlled using hand gestures without using a controller.

[0041] Ray control and display using a controller will be described with reference to Figure 4. Ray control using a controller will be referred to as controller ray hereinafter.

[0042] In FIG. 4, the ray 403 extends from the controller 120 toward the window 306. That is, the starting point of the ray 403 is determined by the position of the controller. The control unit 211 detects the position and orientation of the controller 120 based on inertial information (detection results of the controller orientation sensor unit 224). Furthermore, if the controller 120 is within the imaging range, the position and orientation of the controller may be detected using SLAM or the like. Note that if the controller is equipped with an infrared LED, the position and orientation of the controller may be detected by reading it with an infrared camera.

[0043] If the controller 120 moves from within the imaging range to outside the imaging range and the user has not changed the ray projection method, the start point of the ray is determined as follows. The time when the controller 120 is no longer included in the imaging range, that is, the time when the imaging unit 202 can no longer detect the controller 120, is defined as time t1, and the position that is the start point of the ray 403 at time t1 is defined as position A1. The time immediately before time t1 is defined as time t0, and the position that is the start point of the ray 403 at time t0 is defined as position A0. The immediately previous time t0 is the time when the user's hand 302 was included in the imaging range. Position A0 is also the position that is the start point of the ray 403 based on the position of the user's hand 302 at time t0.

[0044] The control unit 211 calculates the amount of movement of the controller 120 between time t0 and time t1 from the inertial information acquired from the controller attitude sensor unit 224, and obtains position A1, which is the starting point of the ray, by adding the amount of movement to position A0.

[0045] The display of the controller ray is updated as follows. The work memory 213 stores the position and orientation information of the controller 120 used to display the ray 403 on the image display unit 203 in the current frame, as well as the starting point and inclination of the ray 403. The control unit 211 acquires information from the inertial measurement unit provided in the controller 120 to display the ray 403 in the next frame. The control unit 211 calculates the difference in the position and orientation of the controller from the acquired information and the information stored in the work memory 213. The control unit 211 calculates the starting point and inclination of the ray 403 to be displayed in the next frame by adding the calculated difference to the starting point and inclination of the ray 403 stored in the work memory 213. After the calculation, the control unit 211 updates the position and orientation information of the controller 120 and the starting point and inclination of the ray 403 stored in the work memory 213 to the values ​​used to display the ray 403 in the next frame. The control unit 211 uses the starting point and gradient of the ray 403 stored in the working memory 213 to display the ray 403 on the image display unit 203 in the next frame.

[0046] Although the starting point of the ray 403 is determined by the position of the controller when the hand is outside the display area 301, ray control may be used in which the starting point of the ray 403 is fixed and only the inclination is changed. An example of a fixed position is the position where the hand 302 is outside the display area 301.

[0047] The ray display process will be described with reference to the flowchart in Fig. 5. It is assumed that before the process of this flowchart starts, the operation mode of the PC 110 is set to a mode in which the HMD 100 controls a ray (ray control mode). The PC 110 also periodically receives inertial information from the controller 120 via the communication unit. If the PC 110 is set to the ray control mode, the control unit 211 controls the HMD 100 according to the orientation of the user's hand, i.e., the position of a virtual ray extending in an extension of the posture of the user's hand.

[0048] In step S1001, the control unit 211 determines whether or not the user's hand 302 can be detected within the imaging range of the imaging unit 202. That is, it determines whether or not a hand is present in the image. If it is determined that the user's hand 302 can be detected within the imaging range of the imaging unit 202, the process proceeds to step S1002. If it is determined that the user's hand 302 cannot be detected within the imaging range of the imaging unit 202, the process proceeds to step S1004.

[0049] In step S1002, the control unit 211 acquires the position or posture of the user's hand 302 from the captured image. Note that the control unit 211 may also acquire the position and posture of the user's hand 302 from the captured image. In this case, a unique effect is achieved in that an operation closer to actual movement is possible compared to when the position or posture of the user's hand 302 is acquired.

[0050] In step S1003, the control unit 211 updates the starting point and inclination of the ray 303 stored in the work memory 213 based on the position or posture of the user's hand 302. After updating, the control unit 211 causes the image display unit 203 (HMD 100) to display the ray 303 using the starting point and inclination of the ray 303 stored in the work memory 213; that is, the control unit 211 controls the display of the ray. Note that the control unit 211 may also cause the image display unit 203 (HMD 100) to display the ray 303 based on the position and posture of the user's hand 302. The data area for the starting point and inclination of the ray 303 stored in the work memory 213 is also used in common for the ray 403, which is the controller ray.

[0051] In step S1004, the control unit 211 acquires the position or orientation of the controller 120 based on inertial information acquired from the controller 120, i.e., information related to the acceleration and angular velocity of the controller 120. The control unit 211 also acquires the position or orientation of the controller 120 based on the previous detection result of the controller 120 and the inertial information. Then, the control unit 211 acquires the distance between the PC 110 and the controller 120 and determines whether the distance is equal to or less than a threshold. In other words, it determines whether the distance is within a predetermined distance. The predetermined distance, which is the threshold, is, for example, 2 meters to determine the distance from the user's hand to the HMD 100. If the distance is within the predetermined distance, the process proceeds to step S1005. If the distance is greater than the predetermined distance, the process proceeds to step S1014. The threshold may be set for each user.

[0052] In step S1005, the control unit 211 determines whether the user is wearing the controller 120. There are two possible methods for determining whether the controller 120 is being worn. The first method is to make the determination based on inertial information transmitted from the controller. In this method, the control unit 211 acquires the amount of movement of the controller 120 within a predetermined time period based on the inertial information of the controller 120. If the acquired amount of movement is equal to or greater than a threshold, i.e., equal to or greater than a predetermined amount of movement, the control unit 211 determines that the user is wearing (moving) the controller, and proceeds to step S1006. If the acquired amount of movement is smaller than the predetermined amount of movement, the control unit 211 determines that the user is not wearing the controller, and proceeds to step S1014. Here, the threshold is set to an amount of movement that corresponds to an error that may be acquired even when the controller 120 is stationary. Note that the threshold may be set individually for each user. The second method is to make the determination based on information related to the wearing of the controller 120 transmitted from the controller. In this method, the control unit 211 acquires information related to the wearing of the controller 120 from the controller 120. Examples of information related to the wearing include pairing information for the controller and, if a contact sensor is provided on the controller, information indicating contact. Another example is a case where controller 120 is able to acquire its own position and transmits information that controller 120 is moving to control unit 211. If the acquired information indicates that controller 120 is being worn, the process proceeds to step S1006. If the acquired information indicates that controller 120 is not being worn, the process proceeds to step S1014.

[0053] In step S1006, the control unit 211 determines whether the reset flag for the ray by the controller held in the work memory 213 is ON. If it is ON, the process proceeds to S1007, where the reset flag is turned OFF. If it is not ON, the process proceeds to S1008.

[0054] In step S1007, the control unit 211 saves the orientation information of the controller 120 acquired in S1004 in the working memory 213 as the orientation information for calculating the tilt of the ray 403. Therefore, when the ray is updated in S1011 after S1007 is performed, the value saved in the working memory 213 and the value of the orientation information of the controller 120 acquired in S1004 are the same, so the amount of change in tilt will be 0. By setting the ray direction calculation standard in S1007, the ray is drawn based on the movement of the controller 120 so that the relationship between the position or orientation of the controller 120 and the position indicated by the ray is maintained.

[0055] In step S1008, control unit 211 performs hand motion detection processing based on inertial information of controller 120. As an example of hand motion detection, detection of clenching and opening of hands will be described with reference to Fig. 6. Changes in roll, pitch, and yaw can be calculated from the inertial information of controller 120.

[0056] As an example, FIG. 6 shows a graph of the changes over time in roll, pitch, and yaw obtained from the inertia information of the controller 120. A hand-clenching motion is performed from time T1 to time T2, during which the roll, pitch, and yaw change. This change pattern is determined in advance as a condition for the hand-clenching motion, and when the condition is met, the motion is detected as a gripping motion. The gripping motion here is the motion of transitioning from a posture with the palm facing forward to a posture with the fingers bent.

[0057] Similarly, from time T3 to T4, the hand is opened, and this change pattern is used to detect the opening movement (Embodiment 3). The opening movement here is the movement when the hand transitions from a position with the fingers bent to a position with the palm facing forward.

[0058] In step S1009, the control unit 211 determines whether or not a gripping motion (a motion of bending the fingers) has been detected. If a gripping motion has been detected, the process proceeds to S1013. As a result, the ray 403 is not updated, and the ray 403 displayed in the current frame will be displayed in the next frame at the same position and with the same inclination. If a gripping motion has not been detected, the process proceeds to S1010.

[0059] In step S1010, control unit 211 determines whether the end of the gripping motion has been detected. The end of the gripping motion is detected using data that manages the end of the gripping motion, which is stored in work memory 213. After a gripping motion is detected in S1009, when the gripping motion is no longer detected in S1009, control unit 211 sets a value indicating the end to the data that manages the end of the gripping motion. If the end of the gripping motion has been detected, the process proceeds to S1012, and if the end of the gripping motion has not been detected, the process proceeds to S1011.

[0060] In step S1011, the control unit 211 performs an update process for the ray 403. The update process is performed by the method described above. Here, calculation is performed on how to display the ray, and at the time of this process, the ray is not displayed to the user.

[0061] In step S1012, the control unit 211 turns on the reset flag of the ray.

[0062] In step S1013, the control unit 211 displays the ray 403 on the image display unit 203 of the HMD 100. Here, if the reset flag for the ray is turned ON in S1012 and the process proceeds to S1013, the update process for the ray 403 remains unchanged, and the ray is drawn so as to point to the same position as in the previous frame. Note that if the controller has moved, the ray will be different from the ray in the previous frame. For example, the length of the displayed ray from the controller to the pointed position may be different, but the direction of the ray may be the same.

[0063] In step S1014, the control unit 211 determines whether or not to end the ray display mode. The control unit 211 determines whether or not to end the ray display mode, for example, depending on whether or not the user has performed an operation to change the display mode. If it is determined that the ray display mode should be ended, the process proceeds to step S1015. If it is determined that the ray display mode should not be ended, the process returns to step S1001.

[0064] In step S1015, the control unit 211 removes the display of the ray 403 from the composite image, that is, makes the ray 403 invisible.

[0065] According to the first embodiment, the control unit 211 can detect the gripping motion of the hand during the gripping motion using inertia information from the controller 120 and restrict the movement of the ray, thereby reducing unintended changes in the direction of the ray by the user.

[0066] In the first embodiment, a method of not performing update processing is shown as a method of not updating the controller ray, but a method of adding an update difference as 0 may also be used as a method of not updating the ray.

[0067] (Embodiment 2) When using a hand ray, a decision process may be performed using an action called a pinch. A pinch is, for example, a movement in which the tips of the thumb and index finger touch together. When performing a pinch operation, the hand may move, and this hand movement may also move the hand ray. When performing a pinch to perform a decision process for an item, the hand may move, causing the hand ray to move away from the item, preventing the decision process from being performed. To reduce this difficulty in operation, a method can be considered in which a pinch action is detected using inertia information from the controller 120, and the ray is not updated while the pinch action is detected.

[0068] The ray display process when pinched will be described with reference to FIG.

[0069] In step S2001, the control unit 211 determines whether or not the user's hand 302 can be detected within the imaging range of the imaging unit 202. That is, it determines whether or not a hand is present in the image. If it is determined that the user's hand 302 can be detected within the imaging range of the imaging unit 202, the process proceeds to step S2002. If it is determined that the user's hand 302 cannot be detected within the imaging range of the imaging unit 202, the process proceeds to step S2006.

[0070] In step S2002, control unit 211 determines whether or not the user is wearing controller 120. The determination method is the same as in step S2005. If the user is wearing controller 120, the process proceeds to step S2003. If the user is not wearing controller 120, the process proceeds to step S2004.

[0071] In step S2003, the control unit 211 determines whether or not a pinch motion has been detected.

[0072] The detection of a pinch motion will be described with reference to Figure 8. Figure 8 shows, as an example, a graph of the changes over time in roll, pitch, and yaw obtained from the inertial information of the controller 120 during a pinch motion. A pinch motion is performed from time T1 to time T2 in Figure 8, and the roll and pitch change during this time. The pattern of these changes is determined in advance as a condition for a pinch motion, and a pinch motion is detected when the condition is met. If a pinch motion is detected, the process proceeds to step S2005. If a pinch motion is not detected, the process proceeds to step S2004.

[0073] In step S2004, the control unit 211 acquires from the captured image the position or posture of the user's hand 302. Note that the control unit 211 may acquire the position and posture of the user's hand 302 from the captured image.

[0074] In step S2005, the control unit 211 updates the starting point and inclination of the ray 303 stored in the work memory 213 based on the position or posture of the user's hand 302. After updating, the control unit 211 causes the image display unit 203 (HMD 100) to display the ray 303 using the starting point and inclination of the ray 303 stored in the work memory 213. Note that the control unit 211 may also cause the image display unit 203 (HMD 100) to display the ray 303 based on the position and posture of the user's hand 302.

[0075] In step S2006, the control unit 211 performs controller ray processing, which is the same as the processing from steps S1004 to S1013 in FIG.

[0076] In step S2007, the control unit 211 determines whether or not to end the ray display mode. The control unit 211 determines whether or not to end the ray display mode, for example, depending on whether or not the user has performed an operation to change the display mode. If it is determined that the ray display mode should be ended, the process proceeds to step S2008. If it is determined that the ray display mode should not be ended, the process returns to step S2001.

[0077] In step S2008, the control unit 211 erases the display of the ray 303 in the composite image, that is, makes the ray 303 non-displayable.

[0078] According to the second embodiment, the control unit 211 can detect a pinch action using inertia information from the controller 120 and not update the ray, thereby reducing the possibility of the ray 303 being moved away from the item due to a pinch action, resulting in a failure of the decision process.

[0079] (Embodiment 3) In the third embodiment, the ray operation uses the controller ray even when the user's hand can be detected within the shooting range. An operation (hereinafter referred to as "direct operation") may be performed to grab and move a virtual object displayed in a virtual space with the user's hand displayed in the virtual space. When a virtual object is located in an area out of reach, it is assumed that the direct operation is performed after using the ray to pull it to a position within reach. When using a ray to pull an object, the user first selects the object with the ray and then pulls the ray toward the user to pull the object within reach. Since an object is selected with the controller ray by pressing a controller button, one example of the operation is that the user operates the controller ray with a fist that is easy to press the button. After the object is pulled, it is assumed that the object is detached from the ray and direct operation is performed with the object's position fixed. One example of the operation to detach an object from the ray is to deselect the object by pressing the controller button again. The distance between the object and the hand is used to transition from the ray operation state to the direct operation state. When the object and the hand are closer than a predetermined distance, the state transitions from the ray operation state to the direct operation state. After bringing an object within reach, it is expected that the user will change from a closed hand to an open hand when performing direct manipulation. If the user forgets to deselect the object and switches to direct manipulation after opening their hand, the object will move in accordance with the movement of the ray when opening their hand. For a user who wants to operate the object at a position where it is pulled close, this movement of the object will make operation difficult. To reduce this difficulty in operation, a method can be considered in which the inertia information of the controller 120 is used to detect the movement of opening the hand (movement of extending the fingers), and the ray is not updated while the movement is detected.

[0080] Referring to FIG. 9, the display process of the controller ray when operating an object will be described.

[0081] Steps S3001 to S3012 are almost the same as steps S1004 to S1015, and only steps S3006 and S3007 are different, so only these two steps will be explained.

[0082] In step S3006, the control unit 211 determines whether or not an opening hand movement has been detected in the hand movement detection process of S3005. The method described in FIG. 6 of the first embodiment is used as an example for detecting the opening hand movement. If an opening hand movement has been detected, the process proceeds to step S3009. If an opening hand movement has not been detected, the process proceeds to step S3007.

[0083] In step S3007, control unit 211 determines whether the end of the hand opening motion has been detected. The end of the hand opening motion is detected using data that manages the end of the hand opening motion, which is stored in work memory 213. After the hand opening motion is detected in S3006, when the hand opening motion is no longer detected in S3006, control unit 211 sets a value representing the end to the data that manages the end of the hand opening motion. If the end of the hand opening motion has been detected, the process proceeds to S3009, and if the end of the hand opening motion has not been detected, the process proceeds to S3008.

[0084] According to the third embodiment, the control unit 211 can detect the hand opening motion from the inertia information of the controller 120 and not update the ray, thereby reducing the movement of the virtual object unintended by the user.

[0085] As in the first and third embodiments, by not updating the ray while the hand is bending or extending to change the posture of the hand when performing a ray operation, it is possible to reduce movement of the virtual object unintentionally by the user.

[0086] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0087] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0088] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0089] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0090] In addition, in each of the examples described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0091] [Configuration 1] an acquisition means for acquiring a movement of a controller; a display control means for displaying a pointing position based on the acquired movement, When the movement of the controller is a specific movement corresponding to the bending or extending movement of a finger, the display control means limits the movement of the pointing position based on the specific movement. 1. An information processing device comprising:

[0092] [Configuration 2] further comprising a determination means for determining whether or not the controller is performing the specific movement; When the determination means determines that the controller is in the middle of performing the specific movement, the display control means restricts movement of the pointing position based on the specific movement until the specific movement is completed. 3. The information processing device according to configuration 1 or 2.

[0093] [Configuration 3] The display control means restricts movement of the pointing position based on the specific movement from when it is determined by the determination means that the controller is in the middle of performing the specific movement until when it is determined that the specific movement has been completed. 3. The information processing device according to configuration 2.

[0094] [Configuration 4] The controller is a ring-shaped controller that is worn on the user's finger. 4. The information processing device according to any one of configurations 1 to 3.

[0095] [Configuration 5] The specific movement is a movement corresponding to a pinch operation. 5. The information processing device according to any one of configurations 1 to 4.

[0096] [Configuration 6] The specific movement is a movement when transitioning from a posture with the palm facing forward to a posture with the fingers bent. 6. The information processing device according to any one of configurations 1 to 5.

[0097] [Configuration 7] The specific movement is a movement when transitioning from a posture in which the fingers are bent to a posture in which the palm faces forward. 7. The information processing device according to any one of configurations 1 to 6.

[0098] [Configuration 8] The controller further includes a recording means for recording the movement of the controller acquired by the acquisition means. 8. The information processing device according to any one of configurations 1 to 7.

[0099] [Configuration 9] When the specific movement is completed, the display control means moves the designated position based on the movement of the controller so that the relationship between the position or orientation of the controller and the designated position at the time when the specific movement is completed is maintained. 9. The information processing device according to any one of configurations 1 to 8.

[0100] [Configuration 10] The pointing position is a virtual ray or pointer. 10. The information processing device according to any one of configurations 1 to 9.

[0101] [Control method] an acquisition step of acquiring a controller movement; a display control step of displaying the pointing position based on the acquired movement, When the movement of the controller is a specific movement corresponding to a flexion or extension movement of a finger, the display control step limits movement of the pointing position based on the specific movement. 2. A method for controlling an information processing apparatus comprising:

[0102] [program] 11. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 10.

[0103] [system] an acquisition device for acquiring a movement of the controller; a display control device that displays the pointing position based on the acquired movement, When the movement of the controller is a specific movement corresponding to a flexion or extension movement of a finger, the display control device limits movement of the pointing position based on the specific movement. An information processing system comprising:

Claims

1. an acquisition means for acquiring a movement of a controller; a display control means for displaying a pointing position based on the acquired movement, When the movement of the controller is a specific movement corresponding to the bending or extending movement of a finger, the display control means limits the movement of the pointing position based on the specific movement.

1. An information processing device comprising:

2. further comprising a determination means for determining whether or not the controller is performing the specific movement; When the determination means determines that the controller is in the middle of performing the specific movement, the display control means restricts movement of the pointing position based on the specific movement until the specific movement is completed.

2. The information processing apparatus according to claim 1, wherein:

3. The display control means restricts movement of the pointing position based on the specific movement from when it is determined by the determination means that the controller is in the middle of performing the specific movement until when it is determined that the specific movement has been completed.

3. The information processing apparatus according to claim 2, wherein:

4. The controller is a ring-shaped controller that is worn on the user's finger.

2. The information processing apparatus according to claim 1, wherein:

5. The specific movement is a movement corresponding to a pinch operation.

2. The information processing apparatus according to claim 1, wherein:

6. The specific movement is a movement when transitioning from a posture with the palm facing forward to a posture with the fingers bent.

2. The information processing apparatus according to claim 1, wherein:

7. The specific movement is a movement when transitioning from a posture in which the fingers are bent to a posture in which the palm faces forward.

2. The information processing apparatus according to claim 1, wherein:

8. The controller further includes a recording means for recording the movement of the controller acquired by the acquisition means.

2. The information processing apparatus according to claim 1, wherein:

9. When the specific movement is completed, the display control means moves the designated position based on the movement of the controller so that the relationship between the position or orientation of the controller and the designated position at the time when the specific movement is completed is maintained.

2. The information processing apparatus according to claim 1, wherein:

10. The pointing position is a virtual ray or pointer.

2. The information processing apparatus according to claim 1, wherein:

11. an acquisition step of acquiring a controller movement; a display control step of displaying the pointing position based on the acquired movement, When the movement of the controller is a specific movement corresponding to a flexion or extension movement of a finger, the display control step limits movement of the pointing position based on the specific movement.

2. A method for controlling an information processing apparatus comprising:

12. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

13. an acquisition device for acquiring a movement of the controller; a display control device that displays the pointing position based on the acquired movement, When the movement of the controller is a specific movement corresponding to a flexion or extension movement of a finger, the display control device limits movement of the pointing position based on the specific movement. An information processing system comprising:

Images