Information processing apparatus, method for controlling information processing apparatus, and program

Abstract

To provide an information processing apparatus, a method for controlling an information processing apparatus, and a program with which utilization of an input device can be promoted which is used for operations in cross reality (XR).SOLUTION: An information processing apparatus 101 comprises: a hand-switch detection unit 301 that, when a user holds an input device 102 allowing the user to perform operations in XR by one hand, detects a motion of the user in changing a holding hand that is the hand holding the input device 102, as a hand-switch motion; and a holding hand discrimination unit 302 that discriminates the holding hand on the basis of the passing motion detected by the passing detection unit 301.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, the development of "XR (cross reality)" has been remarkable. XR is a technology that realizes the fusion of the real world and the virtual world, and is a general term for "VR (virtual reality)", "AR (augmented reality)", and "MR (mixed reality)". Users mainly use a dedicated controller (hereinafter referred to as "XR controller") to operate XR. The shape of the XR controller is diverse, for example, a rod-shaped general-purpose type that is held by the user (hereinafter referred to as "holding type"), and a type that can be worn by the user such as a wristwatch, bracelet, or ring (hereinafter referred to as "wearing type"). In addition, XR controllers generally have a function to track the position and orientation of the XR controller using an acceleration sensor, a gyro sensor, and a geomagnetic sensor. Furthermore, there are two types of XR controllers: one that is held and worn by the user in a pair (hereinafter referred to as "two-handed controllers"), and one that is held and worn by the user in one hand (hereinafter referred to as "one-handed controller").

[0003] When using a one-handed controller, the hand that is not holding or wearing the one-handed controller is free, which has the advantage of allowing the user to use the free hand to perform hand gestures and touch objects in the real world. On the other hand, one-handed controllers have the disadvantage of having roughly half the number of physical input mechanisms, such as buttons, compared to two-handed controllers. In XR, there are fewer spatial constraints, and the use of XR controllers is widespread. For this reason, one-handed controllers among XR controllers can be used by changing the hand that holds or wears them (hereinafter referred to as "holds" them), unlike conventional controllers for non-XR applications that are operated with the dominant hand, such as mice. As such, one-handed controllers are designed to be used with either the right or left hand, with a top and bottom and symmetrical left and right sides.

[0004] In the following description, the change of hand that the user uses to hold the one-handed controller is referred to as "changing hands." The hand that the user uses to hold the one-handed controller is referred to as "holding hand." In the case of changing hands, for example, if an input function according to the holding hand or dominant hand can be assigned to the input mechanism of the one-handed controller at the timing of changing hands, it will lead to the promotion of the use of the one-handed controller. For this purpose, at least the holding hand needs to be automatically identified. In this regard, Patent Document 1 describes a technology for identifying the hand of a user holding a mobile information terminal based on the movement direction of the user's finger operating a touch panel. There are types of XR controllers that have a touchpad as an input mechanism. Therefore, the technology described in Patent Document 1 can also be applied to some XR controllers that have a touchpad. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2011-76521 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in order to distinguish the holding hand when changing grips with the technology described in Patent Document 1, the user needs to perform a specific operation to be notified of the timing of the change and to distinguish the holding hand. Therefore, with the technology described in Patent Document 1, it is difficult to automatically distinguish the holding hand when changing grips, and there is a problem that it does not lead to promotion of the use of one-handed controllers.

[0007] The present invention has been made in view of the above problems. It is an object of the present invention to provide an information processing device, a control method for the information processing device, and a program that can promote utilization of an input device used for operations in XR. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the information processing device of the present invention is characterized in that, when a user holds an input device that can be operated with one hand in XR (cross reality), the information processing device detects an action of the user changing the holding hand that holds the input device as a gripping action, and a first discrimination means for discriminating the holding hand based on the gripping action detected by the detection means. Effect of the Invention

[0009] According to the present invention, it is possible to promote utilization of input devices used for operations in XR. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system according to each embodiment. [Diagram 2] FIG. 11 is a diagram showing the relationship between an information processing device, an input device, and a display and imaging device when the information processing system provides an XR to a user in each embodiment. [Diagram 3]FIG. 1 is a block diagram showing a functional configuration of an information processing device in first to third embodiments. [Figure 4] 4 is a flowchart showing a process flow of the information processing device in the first to third embodiments. [Diagram 5] 13 is a flowchart showing the flow of detection processing for a change of gripping motion in the first embodiment. [Figure 6] 6 is a flowchart showing the flow of a holding hand discrimination process in the first embodiment. [Figure 7] FIG. 11 is a diagram showing an example of an image of how a gripping motion is detected and the holding hand is determined by tracking using image recognition for a grip-type one-handed controller in the first embodiment. [Figure 8] FIG. 11 is a diagram showing an example of an image of how a gripping motion is detected and the holding hand is determined by tracking using image recognition for a wearable one-handed controller in the first embodiment. [Figure 9] 13 is a flowchart showing the flow of detection processing for a change of gripping motion in the second embodiment. [Figure 10] 13 is a flowchart showing the flow of a holding hand discrimination process in the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of an image of how a gripping-type one-handed controller is detected and the holding hand is determined using information from an acceleration sensor in the second embodiment. [Figure 12] FIG. 11 is a diagram showing an example of an image of how a gripping motion is detected and the holding hand is determined using information from an acceleration sensor for a wearable one-handed controller in the second embodiment. [Figure 13] 13 is a graph showing an example of a change in acceleration of the one-handed controller when changing the grip in the second embodiment. [Figure 14] 13 is a graph showing an example of a change in speed of a one-handed controller when changing grip in the second embodiment. [Figure 15] 13 is a flowchart showing the flow of detection processing for a change of gripping motion in the third embodiment. [Figure 16]13 is a flowchart showing the flow of a holding hand discrimination process in the third embodiment. [Figure 17] FIG. 13 is a diagram showing an example of an image of how a gripping motion is detected and the holding hand is determined using information from a gyro sensor for a grip-type one-handed controller in the third embodiment. [Figure 18] FIG. 13 is a diagram showing an example of an image of how a gripping motion is detected and the holding hand is determined using information from a gyro sensor for a wearable one-handed controller in the third embodiment. [Figure 19] 13 is a graph showing an example of a change in angular velocity of a one-handed controller when changing the grip in the third embodiment. [Figure 20] 13 is a graph showing an example of a change in angle of a one-handed controller when changing the grip in the third embodiment. [Figure 21] FIG. 13 is a diagram showing an image of right rotation of a one-handed controller in the third embodiment. [Figure 22] FIG. 13 is a diagram showing an example of an image when a presentation is given in a virtual space in the fourth embodiment. [Diagram 23] FIG. 11 is a block diagram showing the functional configuration of an information processing device in the fourth and fifth embodiments. [Figure 24] 13 is a flowchart showing a process flow of the information processing device in the fourth and fifth embodiments. [Diagram 25] FIG. 13 is a diagram showing an example of a pointing function set and a support function set in the fourth embodiment. [Figure 26] FIG. 13 is a diagram showing an example of a dominant hand function set and a non-dominant hand function set in the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can perform the same function. Any configuration may be added. Any two or more configurations (features) of each embodiment can be combined. In each embodiment, the same configurations are described with the same reference symbols. Also, different threshold values ​​and predetermined times are used depending on the respective applications, as described below.

[0012] First Embodiment Hereinafter, the first embodiment will be described with reference to FIG. 1 to FIG. 8. FIG. 1 is a diagram showing a configuration example of an information processing system 100. The information processing system 100 provides a user with XR, which is a technology that realizes the fusion of the real world and the virtual world. As shown in FIG. 1, the information processing system 100 has an information processing device 101, an input device 102, and a display and imaging device 103. FIG. 2 is a diagram showing the relationship between the information processing device 101, the input device 102, and the display and imaging device 103 when the information processing system 100 provides the user with XR. As shown in FIG. 2, the information processing device 101 is a small box computer that can be carried on the shoulder of a user. Note that the information processing device 101 is not limited to this, and may be, for example, a tablet PC, a portable computer such as a smartphone, or the like.

[0013] The input device 102 is a one-handed controller. The display and image capturing device 103 is a head mounted display (HMD) that can be worn by a user on the head. In the first embodiment, the input device 102 and the display and image capturing device 103 are external components of the information processing device 101. However, if the display and image capturing device 103 has the information processing function of the information processing device 101 of the first embodiment described later, it can also be treated as the information processing device 101 of the present invention. This point is the same in the second to fifth embodiments.

[0014] Returning to the explanation of FIG. 1, the information processing device 101 has a CPU 111, a ROM 112, a RAM 113, a recording unit 114, an input / output I / F 115, a communication I / F 116, a GPU board 117, and a display driver 118. The input device 102 has an operation unit 119 and a sensor 120. The display / photographing device 103 has a photographing unit 121 and a display unit 122. In the information processing device 101, the CPU 111 is a system control unit, and controls the entire information processing device 101. Moreover, the CPU 111 realizes the information processing function of the first embodiment by executing an information processing program.

[0015] The ROM 112 is a read-only memory in which programs and parameters that do not require modification (for example, basic programs and initial data) are stored. The RAM 113 is a memory for temporarily storing input information, calculation results in information processing, calculation results in image processing, and the like. The recording unit 114 is a device capable of writing and reading various information. The recording unit 114 includes a hard disk, a memory card, a removable disk, an IC card, and the like. The hard disk can be built into or externally attached to the information processing device 101. The removable disk and the IC card can be detachably attached to the information processing device 101. The memory card can be built into, externally attached to, or detachably attached to the information processing device 101. The information processing program is recorded in the recording unit 114, read from the recording unit 114, expanded in the RAM 113, and executed by the CPU 111. The information processing program may be stored in the ROM 112. Furthermore, the recording unit 114 can record necessary data used by the information processing program executed by the CPU 111.

[0016] The input / output I / F 115 accepts input and output of required data. The required data includes controller input from the operation unit 119 of the input device 102, sensor input from the sensor 120 of the input device 102, photographing data from the photographing unit 121 of the display and photographing device 103, and display data to the display unit 122 of the display and photographing device 103. The input / output connection form by the input / output I / F 115 includes both wired connection by a USB cable or the like and wireless connection by Bluetooth or the like. The communication I / F 116 is an interface unit that allows the information processing device 101 to transmit and receive data to and from the outside, such as the cloud, via the Internet.

[0017] The GPU board 117 is a general-purpose graphic board, and performs processes such as image generation and synthesis. For example, the GPU board 117 can generate virtual objects and virtual spaces, and perform synthesis processes between virtual objects and virtual spaces and real images, by operating a one-handed controller. This allows the GPU board 117 to generate virtual images and virtual reality images. Note that the virtual objects and virtual spaces may be generated in advance and recorded in the recording unit 114. The display driver 118 is software for controlling a display device, which is the display unit 122 of the display and imaging device 103, via the input / output I / F 115.

[0018] In the input device 102, the operation unit 119 is an input mechanism of the one-handed controller, and includes a button, a trigger, a dial, a thumbstick, a trackpad, and the like, and can obtain operation instructions and commands from the user. The sensor 120 is a sensing component for obtaining the position and attitude of the one-handed controller and its changes, and includes an acceleration sensor and a gyro sensor. The acceleration sensor can detect the translational motion of the one-handed controller. The gyro sensor can detect the rotational motion of the one-handed controller. In the display and photographing device 103, the photographing unit 121 is a camera mounted on the HMD. The real image photographed by the photographing unit 121 is subjected to image recognition, and is used for identifying the center of gravity of the user's hand and the positions of the hand joints, tracking and analyzing the movements of the user's hand and the one-handed controller, and identifying the positions at which virtual objects and virtual spaces are superimposed. The display unit 122 is an electronic display mounted on the HMD as a display device. In addition, the display unit 122 displays virtual images and virtual reality images generated by the GPU board 117 of the information processing device 101.

[0019] FIG. 3 is a block diagram showing the functional configuration of the information processing device 101. The information processing device 101 has, as its functional configuration, a gripping detection unit 301 (detection means) and a holding hand discrimination unit 302 (first discrimination means). The gripping detection unit 301 detects a gripping motion. The gripping motion indicates the behavior of the one-handed controller and the behavior of the user's hand when the user changes the hand holding the one-handed controller. The holding hand discrimination unit 302 discriminates the holding hand. More specifically, the holding hand discrimination unit 302 discriminates whether the one-handed controller is held in the left or right hand of the user by analyzing the gripping motion detected by the gripping detection unit 301. The gripping detection unit 301 may detect a gripping motion by image recognition using a real image captured by a camera of the HMD or by using sensor information from the one-handed controller. The holding hand discrimination unit 302 may discriminate the holding hand by different methods depending on the gripping motion detection method by the gripping detection unit 301. Each detection and discrimination method will be described in detail later.

[0020] Fig. 4 is a flowchart showing a process flow of the information processing device 101 in the first embodiment. The process flow of the information processing device 101 (control method of the information processing device) shown in the flowchart of Fig. 4 is realized by the CPU 111 (computer) expanding the information processing program recorded in the recording unit 114 into the RAM 113 and executing it. In step S401, the grip change detection unit 301 performs a process of detecting a grip change motion (detection step). In this process, the grip change detection unit 301 detects a grip change motion. Note that the process of step S401 will be described in detail when explaining the subroutines described later.

[0021] In step S402, it is determined whether the gripping detection unit 301 has detected a gripping motion. If the gripping detection unit 301 determines that a gripping motion has been detected, the process proceeds to step S403. On the other hand, if the gripping detection unit 301 determines that a gripping motion has not been detected, the process returns to step S401. In step S403, the holding hand determination unit 302 performs a holding hand determination process (first determination step). In this process, the holding hand is determined by the holding hand determination unit 302. The process of step S403 will be described in detail when describing a subroutine to be described later. After the process of step S403 has been performed, the flowchart in FIG. 4 ends.

[0022] The process of detecting a gripping motion performed in step S401 and the process of determining a holding hand performed in step S403 will be described below with reference to Figs. 5 to 8. In the first embodiment, the process of detecting a gripping motion in step S401 and the process of determining a holding hand in step S403 are performed by image recognition that tracks both hands of the user and the one-handed controller based on a real image from the user's viewpoint captured by the camera of the HMD. Fig. 5 is a flowchart showing the flow of the subroutine of step S401, that is, the process of detecting a gripping motion in the first embodiment. Fig. 6 is a flowchart showing the flow of the subroutine of step S403, that is, the process of determining a holding hand in the first embodiment. Fig. 7 is a diagram showing an example of an image when a gripping motion is detected and a holding hand is determined by tracking using image recognition for a gripping type one-handed controller (hereinafter sometimes abbreviated as "grip type"). Fig. 8 is a diagram showing an example of an image when a gripping motion is detected and a holding hand is determined by tracking using image recognition for a wearable type one-handed controller (hereinafter sometimes abbreviated as "wearable type").

[0023] First, the detection process of the gripping motion performed in step S401 will be described with reference to Figs. 5, 7, and 8. As shown in Fig. 5, in the subroutine of the gripping motion detection process in step S401, in step S501, the gripping detection unit 301 acquires the position coordinates of the center of gravity of the one-handed controller by image recognition. This allows the position of the input device 102 to be estimated. Specifically, in the case shown in Fig. 7, that is, when the input device 102 is a grip type, the gripping detection unit 301 acquires the position coordinates of the center of gravity 701 of the grip type one-handed controller by image recognition. In the case shown in Fig. 8, that is, when the input device 102 is a wearable type, the gripping detection unit 301 acquires the position coordinates of the center of gravity 801 of the wearable type one-handed controller by image recognition.

[0024] Returning to the description of FIG. 5, in step S502, the gripping detection unit 301 acquires the position coordinates of the user's hands by image recognition. Specifically, in the case shown in FIG. 7, that is, when the input device 102 is a gripping type, the gripping detection unit 301 acquires the position coordinates of the center of gravity 703 of the user's right hand 702 and the position coordinates of the center of gravity 705 of the user's left hand 704 by image recognition. This allows the positions of both hands of the user to be estimated when the input device 102 is a gripping type. In the case shown in FIG. 8, that is, when the input device 102 is a wearable type, the gripping detection unit 301 acquires the position coordinates of the first joint 803 of the index finger of the user's right hand 802 and the position coordinates of the first joint 805 of the index finger of the user's left hand 804 by image recognition. This allows the positions of both hands of the user to be estimated when the input device 102 is a wearable type. The grip change detection unit 301 may obtain, by image recognition, the position coordinates of each joint of the user's right hand 802 and left hand 804. The grip change detection unit 301 may also obtain, by image recognition, the position coordinates of a specific joint among the joints of the user's right hand 802 and left hand 804, other than the first joints 803 and 805 of the index fingers.

[0025] Returning to the description of FIG. 5, in step S503, the grip change detection unit 301 detects a grip change action by comparing the distance from the one-handed controller to the user's hand with a threshold value. At that time, the grip change detection unit 301 calculates the distance from the one-handed controller to the user's hand using each position coordinate acquired in steps S501 and S502. Specifically, in the case shown in FIG. 7, that is, when the input device 102 is a grip type, the grip change detection unit 301 detects a grip change action when the distance from the center of gravity 701 of the one-handed controller to the center of gravity 703 of the user's right hand 702 becomes equal to or greater than the threshold value from less than the threshold value. On the other hand, the grip change detection unit 301 does not detect a grip change action unless the distance from the center of gravity 701 of the one-handed controller to the center of gravity 703 of the user's right hand 702 becomes equal to or greater than the threshold value from less than the threshold value. Note that the same applies to the center of gravity 705 of the user's left hand 704.

[0026] In the case shown in FIG. 8, that is, when the input device 102 is a wearable type, the grip change detection unit 301 detects a grip change action when the distance from the center of gravity 801 of the one-handed controller to the first joint 803 of the index finger of the user's right hand 802 changes from less than the threshold to equal to or greater than the threshold. On the other hand, the grip change detection unit 301 does not detect a grip change action when the distance from the center of gravity 801 of the one-handed controller to the first joint 803 of the index finger of the user's right hand 802 does not change from less than the threshold to equal to or greater than the threshold. Note that the same applies to the first joint 805 of the index finger of the user's left hand 804. In addition, when the grip change detection unit 301 acquires the position coordinates of each joint of the user's right hand 802 and left hand 804, the grip change detection unit 301 may detect a grip change action based on the positions of joints other than the first joints 803 and 805 of the index fingers. Furthermore, when the grip change detection unit 301 acquires the position coordinates of a specific joint other than the first joints 803 and 805 of the index fingers among the joints of the user's right hand 802 and left hand 804, the grip change detection unit 301 may detect the grip change motion based on the position of the specific joint. Returning to the explanation of Fig. 5. After the process of step S503 is performed, the flowchart of Fig. 5 ends, and the process returns to the flowchart of Fig. 4.

[0027] Next, the holding hand discrimination process performed in step S403 will be described with reference to Fig. 6, Fig. 7, and Fig. 8. As shown in Fig. 6, in the subroutine of the holding hand discrimination process performed in step S403, in step S601, the holding hand discrimination unit 302 discriminates the holding hand by comparing the distance from the one-handed controller to the user's hand with a threshold value. At that time, the holding hand discrimination unit 302 uses the distance calculated in step S503 as the distance from the one-handed controller to the user's hand.

[0028] Specifically, in the case shown in FIG. 7, that is, when the input device 102 is a grip type, if the distance from the center of gravity 701 of the one-handed controller to the center of gravity 703 of the user's right hand 702 is less than a threshold value, the holding hand determination unit 302 determines that the user's right hand 702 is the holding hand. The same applies to the center of gravity 705 of the user's left hand 704. However, if the distance from the one-handed controller to the center of gravity of the user's hand is less than the threshold value for both the user's right hand 702 and left hand 704, the holding hand determination unit 302 determines that the hand with the shorter distance is the holding hand. As a result, even if it is possible to determine that both of the user's hands are the holding hands, the holding hand determination unit 302 determines that either the user's right hand 702 or the left hand 704 is the holding hand. Note that in the case shown in FIG. 7, the user's right hand is determined to be the holding hand.

[0029] In the case shown in FIG. 8, that is, when the input device 102 is a wearable type, the holding hand determination unit 302 determines that the user's right hand 802 is the holding hand if the distance from the center of gravity 801 of the one-handed controller to the first joint 803 of the index finger of the user's right hand 802 is less than the threshold value. This is also true for the first joint 805 of the index finger of the user's left hand 804. However, if the distance from the one-handed controller to the first joint of the index finger of the user's hand is less than the threshold value for both the user's right hand 802 and the left hand 804, the holding hand determination unit 302 determines that the hand with the shorter distance is the holding hand. As a result, even if it is possible to determine that both of the user's hands are the holding hands, the holding hand determination unit 302 determines that either the user's right hand 802 or the left hand 804 is the holding hand. Note that in the case shown in FIG. 8, the user's right hand is determined to be the holding hand. Return to the description of FIG. 6. After the process of step S601 is performed, the flow chart of FIG. 6 ends, and the process returns to the flow chart of FIG.

[0030] As described above, the information processing device 101 according to the first embodiment tracks the one-handed controller and the user's hand by image recognition using a real image from the user's viewpoint, and acquires the position coordinates of the one-handed controller and the user's hand. Furthermore, the information processing device 101 calculates the distance from the one-handed controller to the user's hand using the position coordinates of the one-handed controller and the position coordinates of the user's hand, and detects a gripping action and identifies the holding hand by comparing the calculated distance with a threshold. As a result, the information processing device 101 automatically identifies the holding hand at the timing of gripping, which can promote the use of the input device 102 used for XR operations.

[0031] The position coordinates used in the first embodiment may be two-dimensional position coordinates obtained by simply recognizing a planar image, or three-dimensional position coordinates obtained by using stereo image recognition or radar technology in combination. The one-handed controller and the user's hand may be tracked by infrared light other than visible light using image recognition. In this case, an infrared light emitting device is provided in the one-handed controller, which is the input device 102, and an infrared camera is provided in the camera of the HMD, which is the display and photographing device 103. Image recognition may be performed using a real image captured by a camera with a third-party viewpoint. In this case, the camera is installed around the user or held by a third party, for example, to ensure a third-party viewpoint.

[0032] Furthermore, when the user's hand or the one-handed controller is outside the angle of view of the HMD camera, the information processing device 101 may maintain the current holding hand discrimination result in order to avoid a situation in which the holding hand is unknown, since the image recognition cannot detect the change of gripping action and the holding hand cannot be determined. Furthermore, the information processing device 101 may improve the accuracy of the holding hand discrimination by excluding the user's hand that cannot be the holding hand from the discrimination target of the holding hand based on the relative position of the user's hand and the HMD camera. Furthermore, the information processing device 101 may determine that the one hand of the user is the holding hand when it is confirmed by image recognition that the one-handed controller is overlapped only by one hand of the user. Furthermore, the information processing device 101 may estimate that the other hand of the user is the holding hand when it is confirmed by image recognition that the one-handed controller and one hand of the user are not overlapped.

[0033] <Second embodiment> Hereinafter, the second embodiment will be described with reference to FIG. 9 to FIG. 14. Here, the differences from the first embodiment will be mainly described. In the second embodiment, the detection process of the gripping change action in step S401 and the determination process of the holding hand in step S403 are performed using information from an acceleration sensor of the one-handed controller. Specifically, when the one-handed controller is gripped, the gripping change action is detected and the holding hand is determined based on the characteristics of the change in acceleration or speed of the one-handed controller, which occurs when the one-handed controller collides with the palm or fingers of the user and is captured by the acceleration sensor. Note that the input device 102 can be treated as the information processing device 101 of the present invention if it has the information processing function of the information processing device 101 of the second embodiment described later. This point is the same in the third to fifth embodiments.

[0034] FIG. 9 is a flowchart showing the flow of the subroutine of step S401, that is, the flow of the detection process of the gripping motion in the second embodiment. FIG. 10 is a flowchart showing the flow of the subroutine of step S403, that is, the flow of the discrimination process of the holding hand in the second embodiment. FIG. 11 is a diagram showing an example of an image when the gripping motion is detected and the holding hand is discriminated using information from an acceleration sensor for a grip-type one-handed controller. Note that FIG. 11 shows an image of the grip-type one-handed controller, which is the input device 102, colliding with the right hand 1101 of the user when the controller is changed. FIG. 12 is a diagram showing an example of an image when the gripping motion is detected and the holding hand is discriminated using information from an acceleration sensor for a wearable one-handed controller. Note that FIG. 12 shows an image of the wearable one-handed controller, which is the input device 102, colliding with the index finger of the right hand 1201 of the user when the controller is changed.

[0035] FIG. 13 is a graph showing an example of a change in acceleration of the one-handed controller when changing its grip. That is, the graph in FIG. 13 shows an example of the relationship between the acceleration of the one-handed controller acquired by the acceleration sensor and the passage of time when changing its grip. FIG. 14 is a graph showing an example of a change in the speed of the one-handed controller when changing its grip. That is, the graph in FIG. 14 shows an example of the relationship between the speed of the one-handed controller acquired by the acceleration sensor and the passage of time when changing its grip. In both the graphs in FIG. 13 and FIG. 14, the positive (upward) direction of the vertical axis represents the rightward direction of the one-handed controller, the negative (downward) direction of the vertical axis represents the leftward direction of the one-handed controller, and the positive (rightward) direction of the horizontal axis represents the passage of time. Note that the graph in FIG. 13 shows an example of the relationship between the acceleration of the one-handed controller and the passage of time when the user changes the hand holding the one-handed controller to the right hand 1101 or 1201 as shown in FIG. 11 and FIG. 12. Similarly, the graph in FIG. 14 shows an example of the relationship between the speed of the one-handed controller and the passage of time when the user changes the hand holding the one-handed controller to the right hand 1101, 1201, as shown in FIGS. 11 and 12.

[0036] First, the grip change detection process performed in step S401 will be described with reference to Fig. 9 and Fig. 11 to Fig. 14. The grip change detection process performed in step S401 is performed in the same manner in the case shown in Fig. 11, i.e., when the input device 102 is a grip type, and in the case shown in Fig. 12, i.e., when the input device 102 is a wearable type. As shown in Fig. 9, in the grip change detection subroutine in step S401, in step S901, the grip change detection unit 301 acquires the acceleration or speed of the one-handed controller from the acceleration sensor. In step S902, the grip change detection unit 301 detects the grip change by analyzing a change in the acceleration or speed of the one-handed controller.

[0037] 13, that is, when the acceleration of the one-handed controller is acquired from the acceleration sensor, the grip change detection unit 301 detects the point 1301 as a feature of the change in acceleration of the one-handed controller that occurs when changing grip. This detects the grip change action. Note that 1301 is the point indicating that the user has stopped holding the controller with one hand, and is the point where the acceleration of the one-handed controller changes from 0 to an absolute value equal to or greater than a threshold within a predetermined time and then returns to 0.

[0038] In the graph of FIG. 13, at the point 1302, that is, at the point indicating the start of holding the controller with one hand by the user, the acceleration of the one-handed controller changes to be similar to that at the point 1301. Therefore, the grip change detection unit 301 distinguishes the point 1302 from the point 1301 by using an evaluation item different from the point 1301, for example, by adjusting a predetermined time. In this way, the grip change detection unit 301 can distinguish between the point (point 1302) indicating the start of holding the controller with one hand by the user and the point (point 1301) indicating the end of holding the controller with one hand by the user. Therefore, the grip change detection unit 301 may improve the detection accuracy by detecting the grip change action when the start and end of holding the controller with one hand by the user occur within a certain time.

[0039] 14, that is, when the speed of the one-handed controller is acquired from the acceleration sensor, the grip change detection unit 301 detects the point indicated by 1401 as a characteristic of the change in the speed of the one-handed controller that occurs when changing grips. This detects the grip change action. Note that 1401 is a point indicating that the user has stopped holding the controller with one hand, and is a point where the speed of the one-handed controller changes from an absolute value equal to or greater than a threshold to 0 within a predetermined time.

[0040] In the graph of FIG. 14, at the point 1402, that is, at the point indicating the start of holding the controller with one hand by the user, the speed of the one-handed controller changes from 0 to an absolute value equal to or greater than the threshold within a predetermined time, contrary to the point 1401. Therefore, the grip change detection unit 301 can distinguish between the point indicating the start of holding the controller with one hand by the user (point 1402) and the point indicating the end of holding the controller with one hand by the user (point 1401). Therefore, even when acquiring the speed of the one-handed controller from the acceleration sensor, the grip change detection unit 301 may improve the detection accuracy by detecting the grip change action by detecting that the start and end of holding the controller with one hand by the user occur within a certain time. Return to the explanation of FIG. 9. After the process of step S902 is performed, the flowchart of FIG. 9 ends, and the process returns to the flowchart of FIG. 4.

[0041] Note that the acceleration or speed change in a gripping motion is a unique change. Therefore, even if a user attempts to reproduce the acceleration or speed change in a gripping motion at a time other than the gripping motion, for example, a slight shaking of the user's hand will cause a change different from the acceleration or speed change in the gripping motion. Therefore, the gripping detection unit 301 can distinguish between the acceleration or speed change in the gripping motion and the acceleration or speed change when reproduced by the user.

[0042] Next, the holding hand discrimination process performed in step S403 will be described with reference to Figs. 10 to 14. The holding hand discrimination process performed in step S403 is performed in the same manner in the case shown in Fig. 11, i.e., when the input device 102 is of the grip type, and in the case shown in Fig. 12, i.e., when the input device 102 is of the wearable type. As shown in Fig. 10, in the subroutine of the holding hand discrimination process performed in step S403, in step S1001, the holding hand discrimination unit 302 discriminates the holding hand according to the directional component of the acceleration or the directional component of the velocity. At that time, the grip change detection unit 301 obtains the directional component of the acceleration or the directional component of the velocity from the detection result of step S902, i.e., from the point indicating the end of the user's holding of the controller with one hand.

[0043] Specifically, in the case shown in FIG. 13, that is, when the acceleration of the one-handed controller is acquired from the acceleration sensor, if the acceleration of the one-handed controller is a negative value, that is, a directional component in the left direction, as in the portion 1301, it is determined that the user's right hand is the holding hand. Note that, at a portion indicating the end of the user's holding of the one-handed controller, if the acceleration of the one-handed controller is a positive value, that is, a directional component in the right direction, as opposed to the portion 1301, it is determined that the user's left hand is the holding hand. On the other hand, in the case shown in FIG. 14, that is, when the velocity of the one-handed controller is acquired from the acceleration sensor, if the velocity of the one-handed controller is a positive value, that is, a directional component in the right direction, as in the portion 1401, it is determined that the user's right hand is the holding hand. Note that, at a portion indicating the end of the user's holding of the one-handed controller, if the velocity of the one-handed controller is a negative value, that is, a directional component in the left direction, as opposed to the portion 1401, it is determined that the user's left hand is the holding hand. Return to the explanation of FIG. 10. After the process of step S1001 is performed, the flowchart in FIG. 10 ends, and the process returns to the flowchart in FIG.

[0044] The change in acceleration or speed when the user places the one-handed controller on a desk is similar to the change in acceleration or speed when the user stops holding the one-handed controller. However, even if the holding hand is identified based on the change in acceleration or speed when the user places the one-handed controller on a desk and some setting is made, the setting will ultimately be the setting for when the controller is not in use and can be ignored.

[0045] As described above, the information processing device 101 according to the second embodiment acquires the acceleration or speed of the one-handed controller. Furthermore, the information processing device 101 detects a gripping action and identifies the holding hand by detecting the characteristics of the change in acceleration or speed when the one-handed controller collides with the palm or fingers of the user from the acquisition result of the acceleration or speed of the one-handed controller. As a result, the information processing device 101 automatically identifies the holding hand at the timing of gripping, which can promote the use of the input device 102 used for XR operations.

[0046] In addition, when a user performs a gripping action without moving the one-handed controller, only a slight change in acceleration or speed of the one-handed controller occurs in the gripping action. Therefore, the manufacturer notifies the user in a user's manual or the like that it is difficult to detect the gripping action and to identify the holding hand when the user performs a gripping action without moving the one-handed controller. In addition, the acceleration and speed changes in an actual gripping action are not represented by a simple combination of line segments as shown in FIG. 13 and FIG. 14. There are individual differences in the tendency of acceleration and speed changes for each user performing a gripping action. Therefore, the information processing device 101 may accumulate data indicating acceleration and speed changes in the gripping action and apply machine learning and deep learning to improve the accuracy of the detection of the gripping action performed by the gripping action detection unit 301 and the identification of the holding hand performed by the holding hand identification unit 302.

[0047] <Third embodiment> Hereinafter, the third embodiment will be described with reference to Figs. 15 to 21. Here, the differences from the first embodiment will be mainly described. In the third embodiment, the detection process of the gripping change action in step S401 and the determination process of the holding hand in step S403 are performed using information from a gyro sensor of the one-handed controller. Specifically, under the rule that the user intentionally rotates the one-handed controller before holding it when gripping it, the gripping change action is detected and the holding hand is determined based on the characteristics of the change in angular velocity or angle of the rotation of the one-handed controller captured by the gyro sensor. Note that, according to the rule in the third embodiment, when the user changes the hand holding the one-handed controller from the left hand to the right hand, the user rotates the one-handed controller to the right, and when the hand holding the one-handed controller is changed from the right hand to the left hand, the user rotates the one-handed controller to the left.

[0048] FIG. 15 is a flowchart showing the flow of the subroutine of step S401, that is, the flow of the detection process of the gripping motion in the third embodiment. FIG. 16 is a flowchart showing the flow of the subroutine of step S403, that is, the flow of the discrimination process of the holding hand in the third embodiment. FIG. 17 is a diagram showing an example of an image when the gripping motion is detected and the holding hand is discriminated using information from a gyro sensor for a grip-type one-handed controller. Note that FIG. 17 shows an image of the grip-type one-handed controller, which is the input device 102, rotating to the right according to the rules when changing the grip and then colliding with the right hand 1701 of the user. FIG. 18 is a diagram showing an example of an image when the gripping motion is detected and the holding hand is discriminated using information from a gyro sensor for a wearable one-handed controller. Note that FIG. 18 shows an image of the wearable one-handed controller, which is the input device 102, rotating to the right according to the rules when changing the grip and then colliding with the index finger of the right hand 1801 of the user.

[0049] FIG. 19 is a graph showing an example of a change in the angular velocity of the one-handed controller when changing its grip. That is, the graph in FIG. 19 shows an example of the relationship between the angular velocity of the one-handed controller acquired by the gyro sensor and the passage of time when changing its grip. FIG. 20 is a graph showing an example of a change in the angle of the one-handed controller when changing its grip. That is, the graph in FIG. 20 shows an example of the relationship between the angle of the one-handed controller acquired by the gyro sensor and the passage of time when changing its grip. In both the graphs in FIG. 19 and FIG. 20, the positive (upward in the figure) direction of the vertical axis represents the right rotation of the one-handed controller, the negative (downward in the figure) direction of the vertical axis represents the left rotation of the one-handed controller, and the positive (rightward in the figure) direction of the horizontal axis represents the passage of time.

[0050] Furthermore, in the graph of angle change in FIG. 20, the limit for right rotation is 180 degrees, and the limit for left rotation is -180 degrees, and if the angle changes beyond the limit, a displacement occurs. Also, in the graph of angle change in FIG. 20, the state in which the one-handed controller is upright is considered to be no rotation (0 degrees). Note that the graph in FIG. 19 shows an example of the relationship between the angular velocity of the one-handed controller and the passage of time when the user changes the hand holding the one-handed controller to the right hand 1701, 1801 as shown in FIG. 17 and FIG. 18. Similarly, the graph in FIG. 20 shows an example of the relationship between the angle of the one-handed controller and the passage of time when the user changes the hand holding the one-handed controller to the right hand 1701, 1801 as shown in FIG. 17 and FIG. 18.

[0051] FIG. 21 is a diagram showing an image of right rotation of a one-handed controller in the third embodiment. FIG. 21(A) is a diagram showing an image of right rotation of a one-handed controller when the input device 102 is a grip type. FIG. 21(B) is a diagram showing an image of right rotation of a one-handed controller when the input device 102 is a wearable type. As shown in FIG. 21(A) and (B), in the third embodiment, right rotation of a one-handed controller refers to clockwise rotation as viewed from the user when the user is holding the one-handed controller. Therefore, although not shown, in the third embodiment, left rotation of a one-handed controller refers to counterclockwise rotation as viewed from the user when the user is holding the one-handed controller.

[0052] First, the grip-changing motion detection process performed in step S401 will be described with reference to Fig. 15 and Figs. 17 to 20. The grip-changing motion detection process performed in step S401 is performed in the same manner in the case shown in Fig. 17, i.e., when the input device 102 is a grip type, and in the case shown in Fig. 18, i.e., when the input device 102 is a wearable type. As shown in Fig. 15, in the grip-changing motion detection subroutine in step S401, in step S1501, the grip-changing detection unit 301 acquires the angular velocity or angle of the one-handed controller from the gyro sensor. In step S1502, the grip-changing detection unit 301 detects the grip-changing motion by analyzing a change in the angular velocity or angle of the one-handed controller.

[0053] Specifically, in the case shown in FIG. 19, that is, when the angular velocity of the one-handed controller is acquired from the gyro sensor, the grip change detection unit 301 detects the location 1901 as a feature of the change in angular velocity of the one-handed controller that occurs when the controller is gripped. This detects the grip change operation. Note that 1901 is a location that indicates the rotation of the one-handed controller defined by the rules, and is a location where the angular velocity of the one-handed controller is maintained at a value equal to or greater than a threshold value for a longer period of time than a predetermined period of time. On the other hand, in the case shown in FIG. 20, that is, when the angle of the one-handed controller is acquired from the gyro sensor, the grip change detection unit 301 detects the location 2001 as a feature of the change in angle of the one-handed controller that occurs when the controller is gripped. This detects the grip change operation. Note that 2001 is a location that indicates the rotation of the one-handed controller defined by the rules, and is a location where the angle change of the one-handed controller in a specific direction is maintained for a longer period of time than a predetermined period of time. Return to the description of FIG. 15. After the process of step S1502 is performed, the flowchart of FIG. 15 ends, and the process returns to the flowchart of FIG. 4.

[0054] Next, the holding hand discrimination process performed in step S403 will be described with reference to Fig. 16 to Fig. 20. The holding hand discrimination process performed in step S403 is performed similarly in the case shown in Fig. 17, i.e., when the input device 102 is of the grip type, and in the case shown in Fig. 18, i.e., when the input device 102 is of the wearable type. As shown in Fig. 16, in the subroutine of the holding hand discrimination process performed in step S403, in step S1601, the holding hand discrimination unit 302 discriminates the holding hand according to the directional component of the angular velocity or the directional component of the angle.

[0055] At this time, the grip change detection unit 301 acquires the directional component of the angular velocity or the directional component of the angle from the detection result of step S1502, that is, from the location indicating the rotation of the one-handed controller defined by the rule. Specifically, in the case shown in Fig. 19, that is, when the angular velocity of the one-handed controller is acquired from the gyro sensor, as in the case of location 1901, if the angular velocity of the one-handed controller is a positive value, that is, a directional component of a right rotation, then it is determined that the user's right hand is the holding hand. Note that, in the location indicating the rotation of the one-handed controller defined by the rule, contrary to location 1901, if the angular velocity of the one-handed controller is a negative value, that is, a directional component of a left rotation, then it is determined that the user's left hand is the holding hand.

[0056] On the other hand, in the case shown in Fig. 20, that is, when the angle of the one-handed controller is acquired from the gyro sensor, if the angle of the one-handed controller changes in the positive direction, that is, in the direction of right rotation, as in the portion 2001, it is determined that the user's right hand is the holding hand. Note that, in the portion indicating the rotation of the one-handed controller defined by the rule, if the angle of the one-handed controller changes in the negative direction, that is, in the direction of left rotation, opposite to the portion 2001, it is determined that the user's left hand is the holding hand. Return to the explanation of Fig. 16. After the process of step S1601 is performed, the flowchart of Fig. 16 ends, and the process returns to the flowchart of Fig. 4.

[0057] As described above, the information processing device 101 according to the third embodiment acquires the angular velocity or angle of the one-handed controller. Furthermore, the information processing device 101 detects a gripping action and identifies the holding hand by detecting the characteristics of the change in angular velocity or angle when the one-handed controller collides with the palm or fingers of the user from the acquisition result of the angular velocity or angle of the one-handed controller. As a result, the information processing device 101 automatically identifies the holding hand at the timing of gripping, which can promote the use of the input device 102 used for XR operations.

[0058] Note that, if the user holds the one-handed controller firmly while using it, the angular velocity change or angle change of the one-handed controller due to the rotation defined by the rules will not occur except during a grip-changing action. However, depending on how the user uses the one-handed controller, there is a possibility that a change similar to the angular velocity change or angle change of the one-handed controller due to the rotation defined by the rules may occur during a grip-changing action. Therefore, the manufacturer notifies the user in the user's manual or the like that the angular velocity change or angle change of the one-handed controller due to the rotation defined by the rules will be detected as the angular velocity change or angle change of the one-handed controller during a grip-changing action. Furthermore, the manufacturer also mentions the possibility of erroneous detection of a grip-changing action in the user's manual or the like.

[0059] Furthermore, even if the correspondence between the rotation direction of the one-handed controller and the holding hand is opposite to the above-mentioned rule, the holding hand determination unit 302 can determine the holding hand. Therefore, the information processing device 101 may allow the user to select the correspondence between the rotation direction of the one-handed controller and the holding hand in advance, and may predetermine a rule according to the selection. Furthermore, the angular velocity change and angle change in the actual gripping motion are not represented by a simple combination of line segments as shown in FIG. 19 and FIG. 20. There are individual differences in the tendency of the angular velocity change and angle change for each user who performs the gripping motion. Therefore, the information processing device 101 may accumulate data indicating the angular velocity change and angle change in the gripping motion and apply machine learning and deep learning to improve the accuracy of the detection of the gripping motion performed by the gripping motion detection unit 301 and the discrimination of the holding hand performed by the gripping hand determination unit 302.

[0060] <Fourth embodiment> Hereinafter, the fourth embodiment will be described with reference to Figs. 22 to 25. Here, the differences from the first to third embodiments will be mainly described. In the fourth embodiment, a case will be described in which the information processing device 101 determines the usage status of the one-handed controller in addition to the holding hand, and further assigns an input function to the input mechanism of the one-handed controller according to the determined usage status of the holding hand and the one-handed controller. For example, when a presentation is made in a virtual space, various scenes are conceivable, such as the avatar of the user who is the presenter pointing to a virtual object on a screen that is a virtual object, or sending a display page that is a virtual object. Therefore, the information processing device 101 assigns an input function optimal for each scene to the input mechanism of the one-handed controller according to the holding hand and the usage status of the one-handed controller.

[0061] Fig. 22 is a diagram showing an example of an image when a presentation is made in a virtual space. In Fig. 22, a screen 2201, which is a virtual object, is in the right hand of an avatar 2202 of a user who is a presenter. In such a case, when the presenter holds the one-handed controller in his right hand, it is preferable that an input function specialized for pointing, such as magnification and highlighting, is assigned to the input mechanism of the one-handed controller. On the other hand, when the presenter holds the one-handed controller in his left hand, the presenter performs pointing with his right hand on the screen 2201 side, so it is preferable that an input function specialized for support, such as page forward and page back, is assigned to the input mechanism of the one-handed controller.

[0062] FIG. 23 is a block diagram showing the functional configuration of the information processing device 101. The information processing device 101 has, as its functional configuration, a gripping detection unit 301, a holding hand determination unit 302, a use status determination unit 2301, and a function allocation unit 2302. In FIG. 23, the gripping detection unit 301 and the holding hand determination unit 302 are similar to the gripping detection unit 301 and the holding hand determination unit 302 shown in FIG. 3 described above, so their description will be omitted. The use status determination unit 2301 (second determination means) determines the use status of the one-handed controller. The use status of the one-handed controller determined by the use status determination unit 2301 includes the user's personal settings related to the one-handed controller, information on the use form of the one-handed controller in XR, and the like. The function allocation unit 2302 (allocation means) allocates an input function to the input mechanism of the one-handed controller according to the information on the holding hand determined by the holding hand determination unit 302 and the use status of the one-handed controller determined by the use status determination unit 2301.

[0063] FIG. 24 is a flowchart showing a process flow of the information processing device 101 in the fourth embodiment. The process flow of the information processing device 101 (control method of the information processing device) shown in the flowchart of FIG. 24 is realized by the CPU 111 (computer) expanding the information processing program recorded in the recording unit 114 in the RAM 113 and executing it. In FIG. 24, steps S401 to S403 are similar to steps S401 to S403 shown in FIG. 4 described above, and therefore their description will be omitted. In step S2401, the use status determination unit 2301 determines the use status of the one-handed controller (second determination step). In the case of the presentation example shown in FIG. 22, it is determined that the screen 2201, which is a virtual object, is on the right hand side of the avatar 2202 of the user who is the presenter, as the use mode of the one-handed controller in XR.

[0064] In step S2402, the function allocation unit 2302 allocates an input function optimal for the current usage situation of the one-handed controller to the input mechanism of the one-handed controller according to the holding hand and usage situation determined in steps S403 and S2401 (allocation process). In the presentation example shown in Fig. 22, if it is determined that both of the following (1) and (2) are satisfied, the function allocation unit 2302 allocates an input function optimal for pointing to the input mechanism of the one-handed controller using a pointing function set described later. (1) The user holds a one-handed controller in his right hand. (2) A screen 2201, which is a virtual object, is located to the right of an avatar 2202 of the presenter user.

[0065] On the other hand, unlike (1), if it is determined that the user is holding the one-handed controller in the left hand, the function assignment unit 2302 assigns an input function that is optimal for support to the input mechanism of the one-handed controller using a support function set described below.

[0066] FIG. 25 is a diagram showing an example of a pointing function set and a support function set. FIG. 25(A) is a diagram showing an example of a pointing function set. FIG. 25(B) is a diagram showing an example of a support function set. When the pointing function set shown in FIG. 25(A) is used, the function allocation unit 2302 assigns the input functions of enlarge, highlight, and . . . to the input mechanisms of the buttons A, B, C, . . . of the one-handed controller. On the other hand, when the support function set shown in FIG. 25(B) is used, the function allocation unit 2302 assigns the input functions of page forward, page back, and . . . to the input mechanisms of the buttons A, B, C, . . . of the one-handed controller. Return to the description of FIG. 24. After the process of step S2402 is performed, the flowchart of FIG. 24 ends.

[0067] As described above, the information processing device 101 according to the fourth embodiment smoothly assigns an input function that is optimal for the current usage situation of the one-handed controller to the input mechanism of the one-handed controller depending on the holding hand and the usage pattern of the one-handed controller in XR. As a result, the information processing device 101 automatically identifies the holding hand and assigns the input function of the one-handed controller at the timing of changing the grip, which can promote the use of the input device 102 used for XR operations.

[0068] <Fifth embodiment> Hereinafter, the fifth embodiment will be described with reference to FIG. 26. Here, the differences from the fourth embodiment will be mainly described. In the fifth embodiment, a case will be described in which the information processing device 101 further assigns an input function to the input mechanism of the one-handed controller according to the user's handedness. For example, when training using tools is performed in a virtual space, various situations can be considered, such as performing practical exercises using the tools in accordance with the instructor's explanation, or taking notes with real paper and pencil while listening to the instructor's explanation. Therefore, the information processing device 101 assigns an input function optimal for each situation to the input mechanism of the one-handed controller according to the holding hand and the usage situation of the one-handed controller including the user's handedness.

[0069] In the case of the training example in the fifth embodiment, the usage status of the one-handed controller determined in the above step S2401 includes, as the usage form of the one-handed controller in XR, that the tool used in training is an impact driver. Furthermore, the usage status of the one-handed controller determined in the above step S2401 includes information on the user's dominant hand as the user's personal setting related to the one-handed controller. Furthermore, when it is determined that the one-handed controller is held in the dominant hand of the user, the function allocation unit 2302 assigns an input function optimal for tool operation using the one-handed controller to the input mechanism of the one-handed controller in the above step S2402. At that time, the function allocation unit 2302 uses a dominant hand function set described later. On the other hand, when it is determined that the one-handed controller is held in the non-dominant hand of the user, the function allocation unit 2302 assigns an input function unrelated to tool operation using the one-handed controller to the input mechanism of the one-handed controller in the above step S2402. At that time, the function allocation unit 2302 uses a non-dominant hand function set described later.

[0070] FIG. 26 is a diagram showing an example of a dominant hand function set and a non-dominant hand function set. FIG. 26(A) is a diagram showing an example of a dominant hand function set. In the dominant hand function set of FIG. 26(A), an impact driver is assumed as a tool used in the training of the example in the fifth embodiment. FIG. 26(B) is a diagram showing an example of a non-dominant hand function set. When using the dominant hand function set shown in FIG. 26(A), the function allocation unit 2302 assigns the input functions of tip rotation, tip replacement, and . . . to the input mechanisms of buttons A, B, C, . . . of the one-handed controller. On the other hand, when using the non-dominant hand function set shown in FIG. 26(B), the function allocation unit 2302 assigns the input functions of volume up, volume down, and . . . to the input mechanisms of buttons A, B, C, . . . of the one-handed controller.

[0071] As described above, the information processing device 101 according to the fifth embodiment smoothly assigns an input function that is optimal for the current usage situation of the one-handed controller to the input mechanism of the one-handed controller, depending on the holding hand, the usage pattern of the one-handed controller in XR, and further the dominant hand of the user. As a result, the information processing device 101 automatically identifies the holding hand and assigns the input function of the one-handed controller at the timing of changing hands, thereby promoting the use of the input device 102 used for XR operations.

[0072] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) A detection means for detecting, when a user holds an input device that can be operated with one hand in XR (cross reality), an action of the user changing the hand that holds the input device as a change-of-hand action; a first discrimination means for discriminating the holding hand based on the gripping change motion detected by the detection means. (Configuration 2) The information processing device described in Configuration 1, characterized in that the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand based on the position of the input device and the position of the user's hand tracked by image recognition using real-world images from a user's perspective or a third-party perspective. (Configuration 3) The information processing device according to configuration 2, wherein the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand by comparing the distance from the input device to the user's hand with a threshold value. (Configuration 4) The information processing device described in Configuration 3, characterized in that when it is possible to determine that both of the user's hands are the holding hand, the hand that is closer to the input device is the holding hand. (Configuration 5) An information processing device described in any one of configurations 2 to 4, characterized in that the detection means and the first discrimination means use the position of the center of gravity of the user's hand or the position of the joints of the user's hand as the position of the user's hand to be tracked by the image recognition. (Configuration 6) The information processing device according to any one of configurations 2 to 5, wherein the detection means and the first discrimination means perform the image recognition using infrared rays. (Configuration 7) The information processing device according to configuration 1, wherein the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand based on the acquired acceleration or speed of the input device. (Configuration 8) The information processing device described in Configuration 7, wherein the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand by detecting characteristics of the change in acceleration or velocity of the input device when the input device collides with the palm or fingers of the user from the acquired results of the acceleration or velocity of the input device. (Configuration 9) The information processing device according to Configuration 8, wherein the detection means and the first discrimination means detect characteristics of changes in acceleration or velocity of the input device when the input device collides with a palm or finger of a user using machine learning or deep learning. (Configuration 10) The information processing device described in Configuration 8 or 9, characterized in that the detection means and the first discrimination means treat a change in acceleration or velocity of the input device when the user stops holding the input device as a characteristic of the acceleration or velocity of the input device when the input device collides with the user's palm or fingers. (Configuration 11) The information processing device according to configuration 8 or 9, characterized in that the detection means treats a change in acceleration or velocity of the input device from when the user starts holding the input device to when the user stops holding the input device as a characteristic of the acceleration or velocity of the input device when the input device collides with the user's palm or fingers. (Configuration 12) The information processing device according to configuration 1, wherein the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand based on the acquired results of the angular velocity or angle of the input device. (Configuration 13) The information processing device described in Configuration 12, characterized in that the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand by detecting the angular velocity or angle characteristics of the input device when the user rotates the input device from the acquired acceleration or velocity results of the input device. (Configuration 14) The information processing device described in Configuration 13, characterized in that the detection means and the first discrimination means detect the angular velocity or angle characteristics of the input device when the user rotates the input device using machine learning or deep learning. (Configuration 15) A second determination means for determining a usage status of the input device; and an assignment means for assigning an input function to an input mechanism of the input device in accordance with the holding hand determined by the first determination means and the usage status determined by the second determination means. (Configuration 16) The information processing device according to configuration 15, wherein the usage status includes information regarding a usage pattern of the input device in XR. (Configuration 17) The information processing device according to configuration 16, wherein the usage status further includes information regarding the user's dominant hand. (Method 1) A detection step of detecting, when a user holds an input device that can be operated with one hand in XR (cross reality), an action of the user changing the hand that holds the input device as a change-of-hand action; a first determination step of determining the holding hand based on the gripping change motion detected in the detection step. (Method 2) a second determination step of determining a usage status of the input device; and an assignment process for assigning an input function to an input mechanism of the input device depending on the holding hand determined in the first determination process and the usage status determined in the second determination process. (Program 1) A program for causing a computer to execute each means of the information processing device according to any one of configurations 1 to 14. (Program 2) A program for causing a computer to execute each of the means of the information processing device according to any one of configurations 15 to 17.

[0073] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0074] 101 Information processing device 102 Input Device 301 Change of hand detection unit (detection means) 302 Holding hand discrimination unit (first discrimination means)

Claims

1. a detection means for detecting, when a user holds an input device that can be operated with one hand in XR (cross reality), an action of the user changing a holding hand that is a hand holding the input device as a change-of-hand action; a first determination means for determining the holding hand based on the gripping movement detected by the detection means.

2. The information processing device according to claim 1, characterized in that the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand based on the position of the input device and the position of the user's hand tracked by image recognition using real images from a user's perspective or a third-party perspective.

3. 3. The information processing apparatus according to claim 2, wherein the detection means and the first determination means detect the gripping motion and determine the holding hand by comparing a distance from the input device to the user's hand with a threshold value.

4. 4. The information processing device according to claim 3, wherein the first discrimination means, when it is possible to discriminate both of the user's hands as the holding hand, discriminates that the hand that is closer to the input device is the holding hand.

5. 3 . The information processing apparatus according to claim 2 , wherein the detection means and the first determination means use the position of the center of gravity of the user's hand or the positions of the joints of the user's hand as the position of the user's hand to be tracked by the image recognition.

6. 3. The information processing apparatus according to claim 2, wherein the detection means and the first discrimination means perform the image recognition using infrared rays.

7. 2 . The information processing apparatus according to claim 1 , wherein the detection means and the first determination means detect the gripping action and determine the holding hand based on an acquired result of an acceleration or a speed of the input device.

8. The information processing device according to claim 7, characterized in that the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand by detecting characteristics of a change in acceleration or velocity of the input device when the input device collides with the palm or fingers of the user from the acquired results of the acceleration or velocity of the input device.

9. The information processing device according to claim 8 , characterized in that the detection means and the first discrimination means detect characteristics of a change in acceleration or velocity of the input device when the input device collides with a palm or a finger of the user using machine learning or deep learning.

10. The information processing device according to claim 8, characterized in that the detection means and the first discrimination means treat a change in acceleration or velocity of the input device when the user stops holding the input device as a characteristic of the acceleration or velocity of the input device when the input device collides with the palm or fingers of the user.

11. 10. The information processing device according to claim 8, wherein the detection means treats a change in acceleration or velocity of the input device from when the user starts holding the input device to when the user stops holding the input device as a characteristic of the acceleration or velocity of the input device when the input device collides with the palm or fingers of the user.

12. 2 . The information processing apparatus according to claim 1 , wherein the detection means and the first determination means detect the gripping action and determine the holding hand based on an acquired result of an angular velocity or an angle of the input device.

13. The information processing device according to claim 12, characterized in that the detection means and the first discrimination means detect the gripping motion and discriminate the holding hand by detecting characteristics of the angular velocity or angle of the input device when the user rotates the input device from the acquired results of the acceleration or velocity of the input device.

14. The information processing device according to claim 13 , wherein the detection means and the first discrimination means detect characteristics of an angular velocity or an angle of the input device when the input device is rotated by a user by using machine learning or deep learning.

15. A second determination means for determining a usage status of the input device; 2. The information processing apparatus according to claim 1, further comprising: an assignment means for assigning an input function to an input mechanism of the input device in accordance with the holding hand determined by the first determination means and the usage status determined by the second determination means.

16. The information processing apparatus according to claim 15 , wherein the usage status includes information regarding a usage pattern of the input device in XR.

17. The information processing apparatus according to claim 16 , wherein the usage status further includes information regarding a user's handedness.

18. a detection step of detecting, when a user holds an input device that can be operated with one hand in XR (cross reality), an action of the user changing the hand that holds the input device as a change-of-hand action; a first determination step of determining the holding hand based on the gripping change motion detected in the detection step.

19. a second determination step of determining a usage status of the input device; 20. The method of claim 18, further comprising an assignment step of assigning an input function to an input mechanism of the input device in accordance with the holding hand determined in the first determination step and the usage status determined in the second determination step.

20. 2. A program for causing a computer to execute each of the means of the information processing apparatus according to claim 1.

21. A program for causing a computer to execute each of the means of the information processing apparatus according to claim 15.

Images