Information processing apparatus, control method, storage medium, and computer program product

Abstract

The present invention provides an information processing apparatus, a control method, a storage medium, and a computer program product capable of facilitating utilization of an input device for operation in XR. The information processing apparatus includes: a detection unit configured to detect an action of a user for changing a holding hand holding an input device as a re-holding action when the user holds the input device enabling a single-hand operation in cross-reality; and a first determination unit configured to determine the holding hand based on the re-holding action detected by the detection unit.

Description

Technical Field

[0001] The present invention relates to an information processing device, a control method, a storage medium and a computer program product. Background Art

[0002] In recent years, the development of XR (cross reality) has been amazing. XR is a technology for realizing the fusion of the real world and the virtual world, and is a general term for VR (virtual reality), AR (augmented reality) and MR (mixed reality). Users mainly use dedicated controllers (hereinafter referred to as XR controllers) to perform operations in XR. There are various types of XR controllers, such as a general rod-shaped type held by a user (hereinafter referred to as a holding type), a type that can be worn by a user such as a wristwatch, a wristband or a ring (hereinafter referred to as a wearable type), etc. The XR controller generally has a function of tracking its position and orientation by using an acceleration sensor, a gyroscope sensor or a geomagnetic sensor. In addition, the XR controller includes a type consisting of a pair of components held or worn by the user's hands respectively (hereinafter referred to as a two-hand controller) and a type consisting of a single component held or worn by one hand of the user (hereinafter referred to as a one-hand controller).

[0003] When a user uses a one-handed controller, the hand that is not holding or wearing the one-handed controller becomes a free hand. Therefore, there is an advantage that the user can use the free hand to perform gesture operations or touch objects in the real world. On the other hand, compared with a two-handed controller, the one-handed controller has the disadvantage that physical input members such as buttons are roughly halved. In XR, there are few space restrictions, and the XR controller is used in a wide range of situations. Therefore, when a user uses the one-handed controller in the XR controller, unlike traditional controllers for non-XR purposes (such as a mouse, etc.) operated by the user's dominant hand, the user can change the hand that holds or wears (hereinafter referred to as "holding") the one-handed controller. As described above, since it is assumed that the one-handed controller is used by the user's right hand or left hand, the one-handed controller has a top and a bottom and is designed to be bilaterally symmetrical.

[0004] In the following description, the change of the hand holding the one-hand controller is referred to as "re-holding". The hand of the user holding the one-hand controller is represented by the "holding hand". In the case of re-holding, for example, if the input function corresponding to the holding hand or the dominant hand can be assigned to the input member of the one-hand controller at the re-holding timing, the utilization of the one-hand controller is facilitated. For this reason, it is necessary to automatically judge the holding hand. In this regard, Japanese Patent Laid-Open No. 2011-76521 (a family application of U.S. Patent No. 10042386B2) discloses a technology for judging the hand of a user holding a personal digital assistant based on the moving direction of the finger of the user operating the touch panel. There is an XR controller of the type having a touchpad as an input member. Therefore, the technology described in the publication text is applicable to an XR controller having a touchpad.

[0005] However, the technology described in the above publication requires the user to perform a notification of the re-hold timing and a specific operation for determining the holding hand, so as to determine the holding hand at the re-hold timing. Therefore, the technology described in the above publication is difficult to automatically determine the holding hand at the re-hold timing, and thus cannot promote the use of the one-hand controller. Summary of the invention

[0006] The present invention provides an information processing apparatus capable of facilitating utilization of an input device for operation in XR, a control method thereof, and a storage medium storing a control program thereof.

[0007] Therefore, one aspect of the present invention provides an information processing device, comprising: a detection unit, which is configured to detect the user's action of changing the holding hand holding the input device as a re-holding action when the user holds the input device that enables one-handed operation in cross reality; and a first judgment unit, which is configured to judge the holding hand based on the re-holding action detected by the detection unit.

[0008] According to the present invention, utilization of an input device for operations in XR is facilitated.

[0009] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram showing a configuration example of an information processing system in each embodiment.

[0011] Figure 2 1 is a diagram showing the relationship between the information processing apparatus, the input device, and the imaging / display device when the information processing system provides XR to the user in each embodiment.

[0012] Figure 3is a block diagram showing the functional configuration of the information processing apparatus in the first embodiment, the second embodiment, and the third embodiment.

[0013] Figure 4 is a flowchart showing the flow of processing by the information processing apparatus in the first embodiment, the second embodiment, and the third embodiment.

[0014] Figure 5 1 is a flowchart showing the flow of the re-holding motion detection process in the first embodiment.

[0015] Figure 6 1 is a flowchart showing the flow of the holding hand determination process in the first embodiment.

[0016] Figure 7 1 is a diagram showing an example of a situation in which detection of a re-holding motion of a grip-type one-hand controller and determination of a holding hand are performed by tracking using image recognition in the first embodiment.

[0017] Figure 8 1 is a diagram showing an example of a situation in which detection of a re-holding motion of a wearable type one-hand controller and determination of a holding hand are performed by tracking using image recognition in the first embodiment.

[0018] Fig. 9 : is a flowchart showing the flow of the re-holding action detection process in the second embodiment.

[0019] Fig.10 : is a flowchart showing the flow of the holding hand determination process in the second embodiment.

[0020] Fig.11 1 is a diagram showing an example of a situation in which detection of a re-holding motion of a grip-type one-hand controller and determination of a holding hand are performed by using information from an acceleration sensor in the second embodiment.

[0021] Fig.12 1 is a diagram showing an example of a situation in which information from an acceleration sensor is used to detect a re-holding motion of a wearable one-hand controller and determine whether the hand is being held in the second embodiment.

[0022] Fig.13 is a graph showing an example of a change in acceleration of the one-hand controller at the time of re-holding in the second embodiment.

[0023] Fig.14 is a graph showing an example of a change in speed of the one-hand controller at the time of re-holding in the second embodiment.

[0024] Fig.15 : is a flowchart showing the flow of the re-holding action detection process in the third embodiment.

[0025] Fig.16 : is a flowchart showing the flow of the holding hand determination process in the third embodiment.

[0026] Fig.17 1 is a diagram showing an example of a situation in which detection of a re-holding motion of a grip-type one-hand controller and determination of a holding hand are performed by using information from a gyro sensor in the third embodiment.

[0027] Fig.18 1 is a diagram showing an example of a case in which detection of a re-holding motion of a wearable type one-hand controller and determination of holding the hand are performed by using information from a gyro sensor in the third embodiment.

[0028] Fig.19 is a graph showing an example of a change in the angular velocity of the one-handed controller at the time of re-holding in the third embodiment.

[0029] Fig. 20 is a graph showing an example of a change in the angle of the one-hand controller upon re-holding in the third embodiment.

[0030] Fig.21A and Fig. 21B 1 is a diagram showing a situation in which the one-hand controller in the third embodiment is rotated rightward.

[0031] Fig. 22 is a diagram showing an example of a situation in which presentation is performed in a virtual space in the fourth embodiment.

[0032] Fig.23 is a block diagram showing the functional configuration of the information processing apparatus in the fourth embodiment and the fifth embodiment.

[0033] Fig.24 is a flowchart showing the flow of processing by the information processing apparatus in the fourth and fifth embodiments.

[0034] Fig.25A and Fig.25B 1 and 2 are diagrams respectively showing an example of a pointing function set and an example of a supporting function set in the fourth embodiment.

[0035] Fig.26A and Fig.26B 2 are diagrams respectively showing an example of a dominant-hand function set and an example of a non-dominant-hand function set in the fifth embodiment. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments. For example, each unit constituting the present invention can be replaced by any configuration that can show the same function. In addition, any components can be added. Any two or more than two configurations (features) in the embodiments can be combined. In an embodiment, the same components are represented by the same reference numerals. As described below, different threshold values ​​and different predetermined times are used according to the purpose.

[0037] In the following, reference will be made to Figures 1 to 8 A first embodiment is described. Figure 1 1 is a diagram showing a configuration example of an information processing system 100. The information processing system 100 provides users with cross reality (XR), which is a technology for achieving fusion of the real world and the virtual world. Figure 1 As shown, the information processing system 100 includes an information processing apparatus 101 , an input device 102 , and an imaging / display device 103 . Figure 2 1 is a diagram showing the relationship between the information processing device 101, the input device 102, and the imaging / display device 103 when the information processing system 100 provides XR to the user in each embodiment. Figure 2 As shown, the information processing device 101 is a small box-type computer that a user can shoulder-mount. The information processing device 101 is not limited thereto, and may be a portable computer such as a tablet PC or a smartphone.

[0038] The input device 102 is a one-handed controller that allows one-handed operation in XR. The camera / display device 103 is an HMD (head mounted display) that can be mounted on the user's head. In the first embodiment, the input device 102 and the camera / display device 103 are components outside the information processing device 101. However, as long as the camera / display device 103 has the information processing function of the information processing device 101 in the first embodiment described later, the camera / display device 103 can be regarded as the information processing device 101 of the present invention. This is the same in the second to fifth embodiments.

[0039] Description Back to Figure 1The information processing apparatus 101 includes a CPU 111, a ROM 112, a RAM 113, a storage unit 114, an input / output I / F (interface) 115, a communication I / F 116, a GPU board 117, and a display driver 118. The input device 102 includes an operation unit 119 and a sensor 120. The camera / display device 103 includes a camera unit 121 and a display unit 122. In the information processing apparatus 101, the CPU 111 is a system controller for controlling the entire information processing apparatus 101. The CPU 111 implements the information processing function of the first embodiment by executing an information processing program.

[0040] ROM 112 is a read-only memory storing programs and parameters (e.g., basic programs, initial date, etc.) that do not need to be changed. RAM 113 is a memory for temporarily storing input information, calculation results in information processing, calculation results in image processing, etc. Storage unit 114 is a device capable of writing and reading various information. Storage unit 114 may be a hard disk, a memory card, a removable disk, an IC card, etc. The hard disk may be built-in or externally attached to information processing device 101. A removable disk or an IC card may be attached to and removed from information processing device 101. A memory card may be built-in or externally attached to information processing device 101, and attached to and removed from information processing device 101. An information processing program is recorded in storage unit 114, read from storage unit 114, expanded into RAM 113, and executed by CPU 111. The information processing program may be stored in ROM 112. The storage unit 114 may also store necessary data utilized by the information processing program executed by the CPU 111 .

[0041] The input / output I / F 115 receives input and output of required data. The required data includes controller input data from the operation unit 119 of the input device 102, sensor input data from the sensor 120 of the input device 102, image data from the image pickup unit 121 of the image pickup / display device 103, and display data to the display unit 122 of the image pickup / display device 103. The input / output connection form through the input / output I / F 115 includes both wired connection through a USB cable or the like and wireless connection through Bluetooth or the like. The communication I / F 116 is an interface for enabling the information processing apparatus 101 to transmit and receive data with the outside such as a cloud via the Internet.

[0042] The GPU board 117 is a general-purpose graphic board, and performs processing such as image generation and image synthesis. For example, the GPU board 117 can generate a virtual object and a virtual space according to an operation using a one-hand controller, and perform synthesis processing of the virtual object, the virtual space, and the real image. Therefore, the GPU board 117 can generate a virtual image or a virtual reality image. The virtual object and the virtual space can be generated in advance and recorded in the storage unit 114. The display driver 118 is software for controlling the display device of the display unit 122 as the camera / display device 103 via the input / output I / F 115.

[0043] In the input device 102, the operation unit 119 is an input member of the one-hand controller, and includes buttons, triggers, dials, thumb sticks, touchpads, etc., and can obtain operation instructions, commands, etc. from the user. The sensor 120 is a sensing component for obtaining the position and orientation of the one-hand controller and its changes, and includes an acceleration sensor, a gyroscope sensor, etc. The acceleration sensor can sense the translational movement of the one-hand controller. The gyroscope sensor can sense the rotational movement of the one-hand controller. In the camera / display device 103, the camera unit 121 is a camera mounted on the HMD. The real image captured by the camera unit 121 is image recognized and used to specify the center of gravity of the user's hand and the position of the joint of the hand, track and analyze the movement of the user's hand and the movement of the one-hand controller, and specify the position of the virtual object and the virtual space superposition. The display unit 122 is an electronic display mounted on the HMD as a display device. The display unit 122 displays the virtual image and virtual reality image generated by the GPU board 117 of the information processing device 101.

[0044] Figure 3 1 is a block diagram showing the functional configuration of the information processing device 101. The information processing device 101 includes a re-holding detection unit 301 (detection unit) and a holding hand judgment unit 302 (first judgment unit) as the functional configuration. The re-holding detection unit 301 detects a re-holding action. The re-holding action refers to the behavior of the one-hand controller and the behavior of the user's hand when the user changes the hand holding the one-hand controller. The holding hand judgment unit 302 judges the holding hand. More specifically, the holding hand judgment unit 302 judges whether the one-hand controller is held by the left hand or the right hand of the user by analyzing the re-holding action detected by the re-holding detection unit 301. The re-holding detection unit 301 can detect the re-holding action by a method using image recognition of a real image captured by a camera of the HMD, a method using sensor information from the one-hand controller, and the like. In addition, the method of judging the holding hand by the holding hand judgment unit 302 depends on the method of detecting the re-holding action by the re-holding detection unit 301. The detection and judgment methods will be described in detail later.

[0045] Figure 4 is a flowchart showing the flow of processing by the information processing apparatus 101 in the first embodiment. Figure 4 The process of the processing of the information processing device 101 shown in the flowchart (the control method of the information processing device) is realized by the CPU 111 (computer) expanding the information processing program stored in the storage unit 114 to the RAM 113 and executing the information processing program. In step S401, the re-holding detection unit 301 performs re-holding action detection processing (detection step). In this processing, the re-holding action is detected by the re-holding detection unit 301. The processing in step S401 will be described in detail in the description of the subroutine.

[0046] In step S402, it is determined whether the re-holding detection unit 301 has detected a re-holding action. When it is determined that the re-holding detection unit 301 has detected a re-holding action, the processing proceeds to step S403. On the other hand, when it is determined that the re-holding detection unit 301 has not detected a re-holding action, the processing returns to step S401. In step S403, the holding hand judgment unit 302 performs a holding hand judgment process (first judgment process). In this process, the holding hand judgment unit 302 judges the holding hand. The processing in step S403 will be described in detail in the description of the subroutine. After performing the processing in step S403, Figure 4 The flowchart in ends.

[0047] In the following, reference will be made to Figures 5 to 8 The re-holding action detection process performed in step S401 and the holding hand judgment process performed in step S403 are described. In the first embodiment, the re-holding action detection process in step S401 and the holding hand judgment process in step S403 are performed by image recognition that tracks both the user's hand and the one-hand controller based on a real image captured from the user's viewpoint by a camera of the HMD. Figure 5 4 is a flowchart showing the flow of the subroutine in step S401 (ie, the re-holding action detection process in the first embodiment). Figure 6 4 is a flowchart showing the flow of the subroutine in step S403 (ie, the holding hand determination process in the first embodiment). Figure 7 : is a diagram showing an example of a case where detection of a re-holding motion of a grip-type one-hand controller (hereinafter may be abbreviated as “grip type”) and judgment of a holding hand are performed by tracking using image recognition.

[0048] Figure 8 : is a diagram showing an example of a case where detection of a re-holding motion of a wearable type one-hand controller (hereinafter may be abbreviated as “wearable type”) and judgment of a holding hand are performed by tracking using image recognition.

[0049] First, refer to Figure 5 , Figure 7 and Figure 8 The re-holding action detection process performed in step S401 is described below. Figure 5 As shown, in the subroutine of the re-holding action detection processing in step S401, in step S501, the re-holding detection unit 301 obtains the position coordinates of the center of gravity of the one-handed controller through image recognition. Thus, the position of the input device 102 is estimated. Specifically, when the input device 102 is Figure 7 When the holding type is shown, the re-holding detection unit 301 obtains the position coordinates of the center of gravity 701 of the one-handed controller of the holding type through image recognition. Figure 8 When the wearing type is shown, the detection unit 301 is held again to obtain the position coordinates of the center of gravity 801 of the one-handed controller of the wearing type through image recognition.

[0050] Description Back to Figure 5 In step S502, the re-holding detection unit 301 obtains the position coordinates of the user's hand through image recognition. Specifically, Figure 7 As shown in FIG. 1 , when the input device 102 is of the grip type, the re-holding detection unit 301 obtains the position coordinates of the center of gravity 703 of the user's right hand 702 and the position coordinates of the center of gravity 705 of the user's left hand 704 through image recognition. Thus, the positions of the user's hands are estimated when the input device 102 is of the grip type. Figure 8 As shown, when the input device 102 is a wearable type, the re-holding detection unit 301 obtains the position coordinates of the first joint 803 of the index finger of the user's right hand 802 and the position coordinates of the first joint 805 of the index finger of the user's left hand 804 through image recognition. Thus, the positions of the user's hands are estimated when the input device 102 is a wearable type. The re-holding detection unit 301 can obtain the position coordinates of the joints of the user's right hand 802 and the user's left hand 804 through image recognition. In addition, the re-holding detection unit 301 can obtain the position coordinates of specific joints of the joints of the user's right hand 802 and the user's left hand 804 except the first joints 803 and 805 of the index finger through image recognition.

[0051] Description Back to Figure 5 In step S503, the re-holding detection unit 301 detects the re-holding action by comparing the distance from the one-handed controller to the user's hand with a threshold. At this time, the re-holding detection unit 301 uses the position coordinates obtained in steps S501 and S502 to calculate the distance from the one-handed controller to the user's hand. Specifically, Figure 7As shown, in the case where the input device 102 is of the grip type, when the distance from the center of gravity 701 of the one-hand controller to the center of gravity 703 of the user's right hand 702 increases from a value less than a threshold value to equal to or greater than a threshold value, the re-holding detection unit 301 should detect a re-holding action. On the other hand, unless the distance from the center of gravity 701 of the one-hand controller to the center of gravity 703 of the user's right hand 702 increases from a value less than a threshold value to equal to or greater than a threshold value, the re-holding detection unit 301 should not detect a re-holding action. These points are the same for the center of gravity 705 of the user's left hand 704.

[0052] like Figure 8 As shown, in the case where the input device 102 is a wearable type, when the distance from the center of gravity 801 of the one-hand controller to the first joint 803 of the index finger of the user's right hand 802 increases from a value less than a threshold value to equal to or greater than the threshold value, the re-hold detection unit 301 should detect the re-hold action. On the other hand, unless the distance from the center of gravity 801 of the one-hand controller to the first joint 803 of the index finger of the user's right hand 802 increases from a value less than a threshold value to equal to or greater than the threshold value, the re-hold detection unit 301 should not detect the re-hold action. These points are the same for the first joint 805 of the index finger of the user's left hand 804. In addition, when obtaining the position coordinates of the joints of the user's right hand 802 and the user's left hand 804, the re-hold detection unit 301 can detect the re-hold action based on the positions of the joints other than the first joints 803 and 805 of the index finger. In addition, when the position coordinates of specific joints other than the first joints 803 and 805 of the index fingers among the joints of the user's right hand 802 and the user's left hand 804 are obtained, the re-holding detection unit 101 can detect the re-holding action based on the positions of the specific joints. Figure 5 After the processing in step S503, the Figure 5 The process returns to Figure 4 Flowchart in .

[0053] Now refer to Figure 6 , Figure 7 and Figure 8 The holding hand determination process performed in step S403 is described below. Figure 6 As shown, in the subroutine of the hand holding judgment processing performed in step S403, in step S601, the hand holding judgment unit 302 judges the hand holding by comparing the distance from the one-hand controller to the user's hand with the threshold. At this time, the hand holding judgment unit 302 uses the distance calculated in step S503 as the distance from the one-hand controller to the user's hand.

[0054] Specifically, Figure 7As shown, in the case where the input device 102 is of the holding type, when the distance from the center of gravity 701 of the one-hand controller to the center of gravity 703 of the user's right hand 702 is less than a threshold value, the holding hand judgment unit 302 judges that the user's right hand 702 is the holding hand. This point is the same for the center of gravity 705 of the user's left hand 704. However, when the distance from the one-hand controller to the center of gravity of both the user's right hand 702 and the user's left hand 704 is less than a threshold value, the holding hand judgment unit 302 judges that the hand with the shorter distance is the holding hand. Therefore, even when both hands of the user can be judged as holding hands, the holding hand judgment unit 302 judges that one of the user's right hand 702 and the user's left hand 704 is the holding hand. Figure 7 In the illustrated case, the user's right hand 702 is determined to be the holding hand.

[0055] like Figure 8 As shown, in the case where the input device 102 is of a wearable type, when the distance from the center of gravity 801 of the one-hand controller to the first joint 803 of the index finger of the user's right hand 802 is less than a threshold value, the holding hand judgment unit 302 judges that the user's right hand 802 is the holding hand. This point is the same for the first joint 805 of the index finger of the user's left hand 804. However, when the distance from the one-hand controller to the first joints of the index fingers of the user's right hand 802 and the user's left hand 804 are both less than a threshold value, the holding hand judgment unit 302 judges that the hand with the shorter distance is the holding hand. Therefore, even when both hands of the user can be judged as holding hands, the holding hand judgment unit 302 judges that one of the user's right hand 802 and the user's left hand 804 is the holding hand. Figure 8 In the case shown, the user's right hand 802 is determined to be the holding hand. Figure 6 After the processing in step S601, the Figure 6 The process returns to Figure 4 Flowchart in .

[0056] The information processing device 101 according to the first embodiment tracks the one-hand controller and the user's hand by using image recognition of a real image from the user's viewpoint, and obtains the position coordinates of the one-hand controller and the position coordinates of the user's hand. In addition, the information processing device 101 uses the position coordinates of the one-hand controller and the position coordinates of the user's hand to calculate the distance from the one-hand controller to the user's hand, and compares the calculated distance with a threshold value, thereby detecting the re-holding action and judging the holding hand. Thus, the information processing device 101 automatically judges the holding hand at the re-holding timing, which facilitates the utilization of the input device 102 for operations in XR.

[0057] The position coordinates used in the first embodiment may be two-dimensional position coordinates obtained by recognizing a simple plane image, or may be three-dimensional position coordinates obtained by recognizing a stereoscopic image or by using radar technology or a combination of the two. In addition, the tracking of the one-handed controller and the user's hand by image recognition may be performed by using infrared light in addition to visible light. In this case, the one-handed controller as the input device 102 is provided with an infrared light emitting device, and the camera of the HMD as the camera / display device 103 is provided with an infrared camera. In addition, image recognition may be performed using a real image captured from a third person's viewpoint using a camera. In this case, for example, in order to ensure the viewpoint of a third person, the camera is installed around the user or held by a third person.

[0058] In addition, when the user's hand or the one-hand controller is outside the field of view of the camera of the HMD, the information processing device 101 cannot detect the re-holding action, and cannot judge the holding hand through image recognition. In this case, the information processing device 101 can maintain the current judgment result related to the holding hand to avoid the situation where the holding hand is unknown. In addition, the information processing device 101 can improve the accuracy of the judgment of the holding hand by excluding the user's hand that cannot be used as the holding hand from the judgment object of the holding hand based on the relative position between the user's hand and the camera of the HMD. In addition, when it is confirmed through image recognition that the one-hand controller overlaps only with one of the user's hands, the information processing device 101 can judge that the overlapping user's hand is the holding hand. In addition, when it is confirmed through image recognition that the one-hand controller does not overlap with one of the user's hands, the information processing device 101 can estimate that the other hand of the user is the holding hand.

[0059] In the following, reference will be made to Figures 9 to 14 The second embodiment is described. Hereinafter, the differences from the first embodiment will be mainly described. In the second embodiment, information from an acceleration sensor included in the one-hand controller is used to perform a re-holding action detection process in step S401 and a holding hand judgment process in step S403. Specifically, based on the characteristics of the acceleration change or speed change of the one-hand controller generated when the one-hand controller collides with the palm or finger of the user and detected by the acceleration sensor when re-holding, the re-holding action is detected and the holding hand is judged. The input device 102 can be regarded as the information processing device 101 of the present invention as long as it has the information processing function of the information processing device 101 in the second embodiment described later. This point is the same in the third to fifth embodiments.

[0060] Fig. 9 4 is a flowchart showing the flow of the subroutine in step S401 (ie, the re-holding motion detection process in the second embodiment). Fig.104 is a flowchart showing the flow of the subroutine in step S403 (ie, the holding hand determination process in the second embodiment). Fig.11 1 is a diagram showing an example of a situation in which a re-holding motion of a grip-type one-hand controller and a holding hand determination are performed by using information from an acceleration sensor. Fig.11 A situation in which a grip-type one-hand controller as the input device 102 collides with the user's right hand 1101 when being re-held is shown. Fig.12 1 is a diagram showing an example of a case where the detection of a re-holding motion of a wearable one-hand controller and the determination of holding the hand are performed by using information from an acceleration sensor. Fig.12 A situation is shown in which a wearable one-hand controller as the input device 102 collides with the index finger of the user's right hand 1201 when being held again.

[0061] Fig.13 is a graph showing an example of the change in acceleration of a one-handed controller when it is re-held. That is, Fig.13 The graph in shows an example of the relationship between the acceleration of the one-hand controller obtained by the acceleration sensor and the passage of time. Fig.14 is a graph showing an example of a change in speed of a one-handed controller when it is re-held. That is, Fig.14 The graph in FIG. 1 shows an example of the relationship between the velocity of a one-handed controller obtained by an acceleration sensor and the passage of time. Fig.13 and Fig.14 In the curve graph, the positive direction (upward) of the vertical axis indicates the right direction of the one-hand controller, the negative direction (downward) of the vertical axis indicates the left direction of the one-hand controller, and the positive direction (rightward) of the horizontal axis indicates the passage of time. Fig.13 The curve diagram in the figure shows that when the user Fig.11 or Fig.12 An example of the relationship between acceleration and time when the holding hand of the one-handed controller is changed to the user's right hand 1101 or 1201 is shown. In the same way, Fig.14 The curve diagram in the figure shows that when the user Fig.11 or Fig.12 An example of the relationship between the speed and the passage of time when the holding hand of the one-hand controller is changed to the user's right hand 1101 or 1201 is shown.

[0062] First, refer to Fig. 9 and Figures 11 to 14 The re-holding action detection processing in step S401 in the second embodiment is described. Fig.11 The type of holding shown is still the input device 102 is Fig.12 For the wearing types shown, the re-holding action detection process in step S401 is similarly performed. Fig. 9As shown, in the subroutine of the re-holding action detection processing in step S401, in step S901, the re-holding detection unit 301 obtains the acceleration or speed of the one-hand controller from the acceleration sensor. In step S902, the re-holding detection unit 301 detects the re-holding action by analyzing the acceleration change or speed change of the one-hand controller.

[0063] Specifically, Fig.13 As shown, when the acceleration of the one-handed controller is obtained from the acceleration sensor, the re-holding detection unit 301 detects a spot 1301 as a characteristic of the acceleration change of the one-handed controller that occurs when re-holding. As a result, the re-holding action is detected. Spot 1301 indicates that the user's holding of the one-handed controller ends. The acceleration of the one-handed controller that changes from 0 to a value equal to or greater than a threshold value in absolute value within a predetermined time period returns to 0 in spot 1301.

[0064] exist Fig.13 In the graph of , the acceleration of the one-handed controller changes in the part 1302 indicating the start of holding of the one-handed controller by the user similarly to the change in the part 1301. The re-holding detection unit 301 distinguishes the part 1302 from the part 1301 by using other evaluation items different from the evaluation items for the part 1301 (for example, by adjusting the predetermined time period). In this way, the re-holding detection unit 301 distinguishes between the part 1302 indicating the start of holding of the one-handed controller by the user and the part 1301 indicating the end of holding of the one-handed controller by the user. Therefore, the re-holding detection unit 301 can improve the detection accuracy by detecting the re-holding action when the start and the end of holding of the one-handed controller by the user occur within a certain time period.

[0065] On the other hand, Fig.14 As shown, when the speed of the one-handed controller is obtained from the acceleration sensor, the re-holding detection unit 301 detects the portion 1401 as a characteristic of the change in speed of the one-handed controller that occurs when re-holding. Thus, the re-holding action is detected. The portion 1401 indicates that the user has ended holding the one-handed controller. The speed of the one-handed controller changes from a value whose absolute value is equal to or greater than the threshold value to 0 in the portion 1401.

[0066] exist Fig.14In the curve graph of , in portion 1402 indicating the start of holding of the one-handed controller, the speed of the one-handed controller changes from 0 to a value whose absolute value is equal to or greater than a threshold value within a predetermined time period, which is opposite to the change in portion 1401. Therefore, the re-holding detection unit 301 distinguishes between portion 1402 indicating the start of holding of the one-handed controller by the user and portion 1401 indicating the end of holding of the one-handed controller by the user. Therefore, even when the re-holding detection unit 301 obtains the speed of the one-handed controller from the acceleration sensor, the re-holding detection unit 101 can improve the detection accuracy by detecting the re-holding action when the start and end of holding of the one-handed controller by the user occur within a certain time period. Description Return to Fig. 9 After the processing in step S902, the Fig. 9 , and processing returns to Figure 4 Flowchart in .

[0067] It should be noted that the acceleration change or speed change due to the re-holding action is the only change. This is different from the acceleration change or speed change due to actions other than the re-holding action (e.g., shaking of the user's hand). Therefore, the re-holding detection unit 301 distinguishes between the acceleration change or speed change of the re-holding action and the acceleration change or speed change of other actions.

[0068] Next, we will refer to Figures 10 to 14 The holding hand determination process in the second embodiment performed in step S403 is described. Fig.11 The type of holding shown is still the input device 102 is Fig.12 For the wearing types shown, the holding hand determination process in step S403 is similarly performed. Fig.10 As shown, in the subroutine of the holding hand judgment process performed in step S403, in step S1001, the holding hand judgment unit 302 judges the holding hand based on the direction component of the acceleration or the direction component of the velocity. At this time, the re-holding detection unit 301 obtains the direction component of the acceleration or the direction component of the velocity from the detection result in step S902 (i.e., from the part indicating that the user has ended holding the one-hand controller).

[0069] Specifically, Fig.13 As shown in the portion 1301 in FIG. 1 , when the acceleration of the one-hand controller obtained from the acceleration sensor is a negative value (i.e., when the direction component is in the left direction), it is determined that the user's right hand is the holding hand. When the acceleration of the one-hand controller in the portion indicating that the user has ended holding the one-hand controller is a positive value (which is opposite to the value in portion 1301) (i.e., when the direction component is in the right direction), it is determined that the user's left hand is the holding hand. On the other hand, Fig.14As shown in the portion 1401 in FIG. 14 , when the velocity of the one-hand controller obtained from the acceleration sensor is a positive value (i.e., when the direction component is in the right direction), it is determined that the user's right hand is the holding hand. When the velocity of the one-hand controller in the portion indicating that the user has ended holding the one-hand controller is a negative value (which is opposite to the value in portion 1401) (i.e., when the direction component is in the left direction), it is determined that the user's left hand is the holding hand. Description Return to Fig.10 After the processing in step S1001, terminate Fig.10 , and processing returns to Figure 4 Flowchart in .

[0070] The acceleration change or speed change when the user puts the one-handed controller on the table is similar to the acceleration change or speed change when the user ends holding the one-handed controller. However, even if holding the hand is judged based on the acceleration change or speed change when the user puts the one-handed controller on the table and a certain setting is made, the setting is ignored because the setting is ultimately a non-use setting.

[0071] As described above, the information processing device 101 according to the second embodiment obtains the acceleration or velocity of the one-hand controller. In addition, the information processing device 101 detects the re-holding action and judges the holding hand by detecting the characteristics of the acceleration change or velocity change when the one-hand controller collides with the palm or finger of the user based on the obtained result of the acceleration or velocity of the one-hand controller. Therefore, the information processing device 101 automatically judges the holding hand at the re-hold timing, which promotes the utilization of the input device 102 for operation in XR.

[0072] When the user re-holds the one-hand controller without moving the one-hand controller, the acceleration change or speed change of the one-hand controller in the re-holding action is slight. Therefore, the manufacturer informs the user through the user manual that it is difficult to detect the re-holding action and judge the holding hand when the user performs the re-holding action without moving the one-hand controller. In addition, the acceleration change and speed change in the actual re-holding action are not determined by the user's Fig.13 and Fig.14 The simple combination of the line segments shown is represented. The trend of acceleration change and speed change also varies between users who perform the re-holding action. Therefore, the information processing device 101 can improve the accuracy of the re-holding detection unit 301 detecting the re-holding action and the accuracy of the holding hand judgment unit 302 judging the holding hand by accumulating data indicating the acceleration change and speed change in the re-holding action and subjecting the data to machine learning and deep learning.

[0073] In the following, reference will be made to Figures 15 to 20 as well as Fig.21A and Fig. 21BThe third embodiment is described. Hereinafter, the differences from the first embodiment will be mainly described. In the third embodiment, information from a gyro sensor included in the one-handed controller is used to perform a re-holding action detection process in step S401 and a holding hand judgment process in step S403. Specifically, at the time of re-holding, under the rule that the user holds the one-handed controller after rotating the one-handed controller, the re-holding action is detected and the holding hand is judged based on the characteristics of the angular velocity change or angle change of the one-handed controller captured by the gyro sensor. According to the rule in the third embodiment, when the user changes the hand holding the one-handed controller from the left hand to the right hand, the user rotates the one-handed controller in the right direction, and when the user changes the hand holding the one-handed controller from the right hand to the left hand, the user rotates the one-handed controller in the left direction.

[0074] Fig.15 4 is a flowchart showing the flow of the subroutine in step S401 (ie, the re-holding motion detection process in the third embodiment). Fig.16 4 is a flowchart showing the flow of the subroutine in step S403 (ie, the holding hand determination process in the third embodiment). Fig.17 1 is a diagram showing an example of a situation in which detection of a re-holding motion of a grip-type one-hand controller and determination of a holding hand are performed by using information from a gyro sensor in the third embodiment. Fig.17 The following situation is shown: a grip-type one-hand controller as the input device 102 collides with the user's right hand 1701 after being rotated right according to the rule when being re-held. Fig.18 1 is a diagram showing an example of a case in which detection of a re-holding motion of a wearable type one-hand controller and determination of holding the hand are performed by using information from a gyro sensor in the third embodiment. Fig.18 A situation is shown in which a wearable one-hand controller as the input device 102 collides with the index finger of the user's right hand 1801 after being rotated right according to a rule when being held again.

[0075] Fig.19 is a graph showing an example of the change in angular velocity of the one-handed controller when re-holding. That is, Fig.19 The graph in shows an example of the relationship between the angular velocity of the one-handed controller obtained by the gyro sensor at the time of re-holding and the passage of time. Fig. 20 is a graph showing an example of the angle change of the one-handed controller when re-holding. That is, Fig. 20 The graph in FIG. 1 shows an example of the relationship between the angle of the one-handed controller obtained by the gyro sensor and the passage of time when the controller is re-held. Fig.19 and Fig. 20In the curve graph, the positive (upward) direction of the vertical axis represents the right rotation of the one-handed controller, the negative (downward) direction of the vertical axis represents the left rotation of the one-handed controller, and the positive (right) direction of the horizontal axis represents the passage of time.

[0076] In addition, Fig. 20 In the angle change graph in , the right rotation is limited to 180 degrees, and the left rotation is limited to -180 degrees. When the angle change exceeds the limit, the left and right should be reversed. Fig. 20 In the angle change graph in , the correct state for the top and bottom of a one-handed controller should be no rotation (0 degrees). Fig.19 The curve diagram in Fig.17 or Fig.18 1701 or 1801. Similarly, Fig. 20 The curve diagram in Fig.17 or Fig.18 An example of the relationship between the angle of the one-hand controller and the passage of time when the user changes the hand holding the one-hand controller to the right hand 1701 or 1801 is shown.

[0077] Fig.21A and Fig. 21B 1 is a diagram showing a situation in which the one-hand controller in the third embodiment is rotated rightward. Fig.21A 1 is a diagram showing a situation where a one-hand controller is rotated right when the input device 102 is of a grip type. Fig. 21B 1 is a diagram showing a situation where a one-hand controller is rotated right when the input device 102 is a wearable type. Fig.21A and Fig. 21B As shown, in the third embodiment, the right rotation of the one-handed controller refers to a clockwise rotation viewed from the user while the user is holding the one-handed controller. Therefore, in the third embodiment, although not shown, the left rotation of the one-handed controller refers to a counterclockwise rotation viewed from the user while the user is holding the one-handed controller.

[0078] First, refer to Fig.15 and Figures 17 to 20 The re-holding action detection processing in the third embodiment performed in step S401 is described. Fig.17 The type of holding shown is still the input device 102 is Fig.18 For the wearing types shown, the re-holding action detection process in step S401 is similarly performed. Fig.15As shown, in the subroutine of the re-holding action detection processing in step S401, in step S1501, the re-holding detection unit 301 obtains the angular velocity or angle of the one-hand controller from the gyro sensor. In step S1502, the re-holding detection unit 301 detects the re-holding action by analyzing the angular velocity change or angle change of the one-hand controller.

[0079] Specifically, Fig.19 As shown, when the angular velocity of the one-handed controller is obtained from the gyro sensor, the re-holding detection unit 301 detects the portion 1901 as a characteristic of the change in the angular velocity of the one-handed controller that occurs when re-holding. Thus, the re-holding action is detected. The portion 1901 shows the rotation of the one-handed controller defined by the rule. In the portion 1901, the angular velocity of the one-handed controller is maintained at a value equal to or greater than the threshold value for a time period longer than the predetermined time period. On the other hand, as Fig. 20 As shown, when obtaining the angle of the one-handed controller from the gyro sensor, the re-holding detection unit 301 detects the portion 2001 as a characteristic of the angle change of the one-handed controller that occurs when re-holding. Thus, the re-holding action is detected. The portion 2001 shows the rotation of the one-handed controller defined by the rule. In the portion 2001, the angle change of the one-handed controller in the specified direction is maintained for a time longer than a predetermined period of time. Description Return to Fig.15 After the processing in step S1502, the Fig.15 , and processing returns to Figure 4 Flowchart in .

[0080] Next, we will refer to Figures 16 to 20 The holding hand determination process in the third embodiment performed in step S403 is described. Fig.17 The type of holding shown is still the input device 102 is Fig.18 For the wearing types shown, the holding hand determination process in step S403 is similarly performed. Fig.16 As shown, in the subroutine of the holding hand judgment processing performed in step S403, in step S1601, the holding hand judgment unit 302 judges the holding hand based on the direction component of the angular velocity or the direction component of the angle.

[0081] At this time, the re-hold detection unit 301 obtains the directional component of the angular velocity or the directional component of the angle from the detection result in step S1502 (ie, from the part indicating the rotation of the one-handed controller as defined in the rule). Fig.19As shown, when the angular velocity of the one-hand controller is obtained from the gyro sensor, as in part 1901, when the angular velocity of the one-hand controller is a positive value (that is, when the direction component is right rotation), it is determined that the user's right hand is the holding hand. In the part of the rotation of the one-hand controller defined by the indication rule, when the angular velocity of the one-hand controller is a negative value (which is opposite to the value in part 1901) (that is, when the direction component is left rotation), the user's left hand is determined to be the holding hand.

[0082] On the other hand, Fig. 20 As shown, in the case where the angle of the one-hand controller is obtained from the gyro sensor, as in part 2001, when the angle of the one-hand controller changes in the positive direction (i.e., in the direction of right rotation), it is determined that the user's right hand is the holding hand. In the part of the rotation of the one-hand controller defined by the indication rule, when the angle of the one-hand controller changes in the negative direction (which is opposite to the direction in part 2001) (i.e., in the direction of left rotation), it is determined that the user's left hand is the holding hand. Description Return to Fig.16 After the processing in step S1601, the Fig.16 , and processing returns to Figure 4 Flowchart in .

[0083] As described above, the information processing device 101 according to the third embodiment obtains the angular velocity or angle of the one-hand controller. In addition, the information processing device 101 detects the re-holding action and judges the holding hand by detecting the characteristics of the angular velocity change or angle change when the one-hand controller collides with the palm or finger of the user from the angular velocity or angle of the one-hand controller as the obtained result. Therefore, the information processing device 101 automatically judges the holding hand at the re-hold timing, which facilitates the utilization of the input device 102 for operation in XR.

[0084] Typically, a change in angular velocity or angle of a one-handed controller due to a rotation defined in the rules will not occur due to actions other than the re-holding action, as long as the user securely holds and uses the one-handed controller. However, when the user does not securely hold and use the one-handed controller, a change similar to the change in angular velocity or angle of the one-handed controller due to the rotation defined in the rules may occur due to actions other than the re-holding action. Therefore, the manufacturer informs the user through the user manual that the change in angular velocity or angle of the one-handed controller due to the rotation defined in the rules is detected as the change in angular velocity or angle of the one-handed controller due to the re-holding action. In addition, the manufacturer mentions the possibility of false detection of the re-holding action in the user manual.

[0085] Furthermore, even if the correspondence between the rotation direction of the one-handed controller and the holding hand is opposite to the above rule, the holding hand judgment unit 302 can judge the holding hand. Therefore, the information processing device 101 can enable the user to pre-select the correspondence between the rotation direction of the one-handed controller and the holding hand, and can pre-define the rule corresponding to the selection. In addition, the angular velocity change and angle change in the actual re-holding action are not determined by the above rule. Fig.19 and Fig. 20 The simple combination of the line segments shown is represented. The trend of the angular velocity change and the angle change also varies between each user who performs the re-holding action. Therefore, the information processing device 101 can improve the accuracy of the re-holding detection unit 301 detecting the re-holding action and the accuracy of the holding hand judgment unit 302 judging the holding hand by accumulating data indicating the angular velocity change and the angle change in the re-holding action and subjecting the data to machine learning and deep learning.

[0086] In the following, reference will be made to Figure 22 to Figure 24 as well as Fig.25A and Fig.25B The fourth embodiment is described. Hereinafter, the differences from the first to third embodiments will be mainly described. In the fourth embodiment, in addition to the holding hand, the information processing device 101 also determines the usage status of the one-handed controller, and also assigns the input function to the input member of the one-handed controller according to the determined usage status of the holding hand and the one-handed controller. For example, when presenting in a virtual space, various scenarios can be imagined, such as a scenario in which an avatar of a user as a presenter points to a virtual object on a screen as a virtual object, and a scenario in which the avatar sends a display page as a virtual object. Therefore, the information processing device 101 assigns the optimal input function for each scenario to the input member of the one-handed controller according to the usage status of the holding hand and the one-handed controller.

[0087] Fig. 22 is a diagram showing an example of a situation in which presentation is performed in a virtual space. Fig. 22 In the example, the screen 2201 as a virtual object is located on the right side of the avatar 2202 of the user who is the poster. In this case, when the poster holds the one-hand controller with the right hand, it is preferable to assign input functions dedicated to pointing (such as zooming in and highlighting) to the input member of the one-hand controller. On the other hand, when the poster holds the one-hand controller with the left hand, the poster points with the right hand on the side of the screen 2201, and therefore, it is preferable to assign input functions dedicated to support (such as page feed and page return) to the input member of the one-hand controller.

[0088] Fig.232 is a block diagram showing the functional configuration of the information processing device 101. The information processing device 101 includes a re-holding detection unit 301, a holding hand judgment unit 302, a use status judgment unit 2301, and a function allocation unit 2302 as its functional configuration. Fig.23 In the process, the re-holding detection unit 301 and the holding hand judgment unit 302 are connected with Figure 3 The re-holding detection unit 301 and the holding hand judgment unit 302 shown are the same, so their description is omitted. The usage status judgment unit 2301 (second judgment unit) judges the usage status of the one-handed controller. The usage status of the one-handed controller judged by the usage status judgment unit 2301 includes the user's personal settings related to the one-handed controller, information related to the usage form of the one-handed controller in XR, etc. The function allocation unit 2302 (allocation unit) allocates the input function to the input component of the one-handed controller based on the information related to the holding hand judged by the holding hand judgment unit 302 and the usage status of the one-handed controller judged by the usage status judgment unit 2301.

[0089] Fig.24 : is a flowchart showing the flow of processing by the information processing apparatus 101 in the fourth embodiment. Fig.24 The flow of processing of the information processing device 101 (control method of the information processing device) shown in the flowchart is realized by the CPU 111 (computer) expanding the information processing program stored in the storage unit 114 to the RAM 113 and executing the information processing program. Fig.24 Steps S401 to S403 in the above Figure 4 Steps S401 to S403 shown in FIG. 2 are the same, and thus their description will be omitted. In step S2401, the use condition determination unit 2301 determines the use condition of the one-handed controller (second determination step). Fig. 22 In the case of the presentation shown, it is determined that the screen 2201 as a virtual object is located on the right hand side of the avatar 2202 of the user who is the presenter, as a usage form of a one-hand controller in XR.

[0090] In step S2402, the function allocation unit 2302 allocates the input function that best suits the current usage status of the one-handed controller to the input member of the one-handed controller according to the holding hand and usage status determined in steps S403 and S2401 (allocation processing). Fig. 22 In the example of presentation shown, when it is determined that both of the following conditions (1) and (2) are satisfied, the function assignment unit 2302 assigns an input function optimal for pointing to the input member of the one-hand controller using a pointing function set described later.

[0091] (1) The user holds the one-hand controller with his right hand.

[0092] (2) A screen 2201 as a virtual object is located on the right hand side of an avatar 2202 of a user who is a presenter.

[0093] On the other hand, unlike condition (1), when it is determined that the user is holding the one-hand controller with the left hand, the function assignment unit 2302 assigns the input function best supported to the input member of the one-hand controller using the support function set described later.

[0094] Fig.25A and Fig.25B are diagrams respectively showing an example of a pointing function set and an example of a supporting function set. Fig.25A is a diagram showing an example of a pointing function set. Fig.25B is a diagram showing an example of a supported function set. Fig.25A When pointing to the function set shown, the function assignment unit 2302 assigns input functions such as magnification and highlighting to input members such as buttons A and B of the one-handed controller, respectively. Fig.25B When the supported function set is shown, the function allocation unit 2302 allocates input functions such as page feed and page return to input members such as buttons A and B of the one-hand controller, respectively. Fig.24 After the processing in step S2402, the Fig.24 Processing of the flowchart in .

[0095] As described above, the information processing device 101 according to the fourth embodiment smoothly allocates the input function that is optimal for the current use condition of the one-hand controller to the input member of the one-hand controller according to the holding hand and the use form of the one-hand controller in XR. Thus, the information processing device 101 automatically performs the judgment of the holding hand and the allocation of the input function in the one-hand controller at the re-holding timing, which promotes the utilization of the input device 102 for the operation in XR.

[0096] In the following, reference will be made to Fig.26A and Fig.26B The fifth embodiment is described. Here, the differences from the fourth embodiment will be mainly described. In the fifth embodiment, the information processing device 101 also assigns input functions to the input components of the one-hand controller according to the user's dominant hand. For example, when training on the use of tools in a virtual space, various scenarios can be envisioned, such as a scenario in which the user performs actual techniques for using the tool according to the speaker's instructions, and a scenario in which the user takes notes with real paper and a pen while listening to the speaker's instructions. Therefore, the information processing device 101 assigns the optimal input functions for each scenario to the input components of the one-hand controller according to the holding hand and the usage status of the one-hand controller including the user's dominant hand.

[0097] In the training scenario of the fifth embodiment, the usage condition of the one-handed controller determined in step S2401 includes the case where the tool used in the training is an impact drill as the usage form of the one-handed controller in XR. In addition, the usage condition of the one-handed controller determined in the above step S2401 includes information related to the user's dominant hand as the user's personal settings related to the one-handed controller. In addition, in the case where it is determined that the one-handed controller is held by the user's dominant hand, in step S2402, the function allocation unit 2302 allocates the input function that is optimal for the tool operation using the one-handed controller to the input member of the one-handed controller. At this time, the function allocation unit 2302 uses the dominant hand function set described later. On the other hand, when it is determined that the one-handed controller is held by the user's non-dominant hand, in step S2402, the function allocation unit 2302 allocates the input function that is not related to the tool operation using the one-handed controller to the input member of the one-handed controller. At this time, the function allocation unit 2302 uses the non-dominant hand function set described later.

[0098] Fig.26A is a diagram showing an example of a dominant hand function set. Fig.26A In the dominant hand function focus, it is assumed that an impact drill is used as the tool used in the training scenario in the fifth embodiment. Fig.26B is a diagram showing an example of a non-dominant hand function set. Fig.26A When the dominant hand function set is shown, the function allocation unit 2302 allocates input functions such as rotation of the drill bit and replacement of the drill bit to input members such as buttons A and B of the one-hand controller, respectively. Fig.26B When the non-dominant hand function set is shown, the function assignment unit 2302 assigns input functions such as volume up and volume down to input members such as buttons A and B of the one-hand controller, respectively.

[0099] As described above, the information processing device 101 according to the fifth embodiment smoothly allocates the input function that is optimal for the current use condition of the one-hand controller to the input member of the one-hand controller according to the holding hand, the use form of the one-hand controller in XR, and the user's dominant hand. Thus, the information processing device 101 automatically performs the judgment of the holding hand and the allocation of the input function in the one-hand controller at the re-holding timing, which promotes the utilization of the input device 102 for the operation in XR.

[0100] Other embodiments

[0101] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0102] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0103] This application claims the benefit of Japanese Patent Application No. 2023-118312, filed on Jul. 20, 2023, which is hereby incorporated by reference herein in its entirety.

Claims

1. An information processing device, comprising: a detection unit configured to detect, when the user holds an input device that enables a one-handed operation in a cross reality, a user's action of changing a holding hand holding the input device as a re-holding action; as well as A first determination unit is configured to determine the holding hand based on the re-holding action detected by the detection unit.

2. The information processing device according to claim 1, wherein: The detection unit and the first judgment unit respectively detect the re-holding action and judge the holding hand based on the position of the input device and the position of the user's hand tracked by image recognition using a real image from the user's viewpoint or a third person's viewpoint.

3. The information processing device according to claim 2, wherein: The detection unit and the first judgment unit respectively detect the re-holding action and judge the holding hand by comparing the distance from the input device to the user's hand with a threshold.

4. The information processing device according to claim 3, wherein: When both hands of the user can be determined as the holding hand, the first determination unit determines that the hand at a shorter distance from the input device is the holding hand.

5. The information processing device according to any one of claims 2 to 4, wherein: The detection unit and the first judgment unit use the position of the center of gravity of the user's hand or the positions of the joints of the user's hand as the position of the user's hand tracked by the image recognition.

6. The information processing device according to any one of claims 2 to 4, wherein: The detecting unit and the first determining unit perform the image recognition by using infrared light.

7. The information processing device according to claim 1, wherein: The detection unit and the first judgment unit respectively detect the re-holding action and judge the holding hand based on the obtained result of the acceleration or speed of the input device.

8. The information processing device according to claim 7, wherein: The detection unit and the first judgment unit respectively detect the re-holding action and judge the holding hand by detecting the characteristics of the acceleration change or speed change that occurs when the input device collides with the user's palm or finger from the results of the acceleration or speed of the input device.

9. The information processing device according to claim 8, wherein: The detection unit and the first judgment unit use machine learning or deep learning to detect characteristics of an acceleration change or a speed change of the input device that occurs when the input device collides with a palm or a finger of a user.

10. The information processing device according to claim 8 or 9, wherein: The detection unit and the first judgment unit treat the acceleration change or speed change of the input device when the user ends holding the input device as a characteristic of the acceleration change or speed change of the input device that occurs when the input device collides with the user's palm or finger.

11. The information processing device according to claim 8 or 9, wherein: The detection unit handles the acceleration change or speed change of the input device from the start to the end of the holding of the input device by the user as a characteristic of the acceleration change or speed change of the input device that occurs when the input device collides with the palm or finger of the user.

12. The information processing device according to claim 1, wherein: The detection unit and the first judgment unit respectively detect the re-holding action and judge the holding hand based on the obtained result of the angular velocity or angle of the input device.

13. The information processing device according to claim 12, wherein: The detection unit and the first judgment unit respectively detect the re-holding action and judge the holding hand by detecting the characteristics of the angular velocity change or angle change of the input device that occurs when the user rotates the input device from the obtained angular velocity or angle results.

14. The information processing device according to claim 13, wherein: The detection unit and the first determination unit detect characteristics of an angular velocity change or an angle change of the input device that occurs when a user rotates the input device, using machine learning or deep learning.

15. The information processing device according to claim 1, further comprising: a second determining unit configured to determine a usage status of the input device; as well as An allocating unit is configured to allocate an input function to an input member of the input device according to the holding hand determined by the first determining unit and the use status determined by the second determining unit.

16. The information processing device according to claim 15, wherein: The usage status includes information on a usage form of the input device in the cross reality.

17. The information processing device according to claim 16, wherein: The usage conditions also include information related to the user's dominant hand.

18. A control method for an information processing device, comprising: a detection step for detecting, when the user holds an input device that enables one-handed operation in cross reality, an action of the user for changing the holding hand holding the input device as a re-holding action; as well as The first judging step is used to judge the holding hand based on the re-holding action detected in the detecting step.

19. The control method for an information processing device according to claim 18, further comprising: A second determination step is used to determine the usage status of the input device; as well as An allocating step is used to allocate an input function to an input member of the input device according to the holding hand determined in the first determining step and the use status determined in the second determining step. 20 . A computer program product comprising a control program for causing a computer to execute each unit of the information processing device according to claim 1 . 21 . A non-transitory computer-readable storage medium storing a control program for causing a computer to execute each unit of the information processing device according to claim 1 .

Images