Information processing apparatus and information processing method

Abstract

An information processing device is provided with a control unit for displaying an object including a virtual object and a real object in an XR space so as to be manipulatable by a user. Further, the control unit determines a position and a gesture of the user's hand with respect to the object, and activates a function relating to manipulation of the object, the function having been assigned to a combination of the position and the gesture of the hand corresponding to a result of the determination.

Description

Technical Field

[0001] The present disclosure relates to an information processing device and an information processing method. Background Art

[0002] In the related art, there is a technology known as extended reality (XR) or cross reality, which is an image processing technology for fusing a virtual space and a real space, such as virtual reality (VR) and augmented reality (AR).

[0003] As an information processing device for providing XR to users, there are known glasses-type or hat-type head mounted displays (HMDs) etc. The HMD causes a display in front of the user's eyes to display virtual objects in a manner similar to real objects existing in a real space.

[0004] In addition, the HMD recognizes various operations of the user with respect to the virtual object, such as rotation, movement, resizing, or design change, through the installed sensor, and activates a function corresponding to the recognized operation to reflect the operation in the virtual object. In recent years, in addition to entertainment fields such as games, such XR has been significantly spread in fields such as design support of computer-aided design (CAD).

[0005] Note that as a method of recognizing a user's operation and reflecting the user's operation in an object, for example, there is a known technology in which a predetermined gesture such as a click operation or a click and hold operation by a hand or a finger is assigned to each of the functions related to the object operation, and the function corresponding to the gesture is activated when the gesture is recognized using sensor information (for example, see patent document 1).

[0006] Reference List

[0007] Patent Literature

[0008] Patent Document 1: JP 2018-180840 A Summary of the invention

[0009] Technical issues

[0010] However, there is still room for further improvement in the above-described related technologies in terms of improving the operability with respect to objects in the XR space.

[0011] For example, in the related art described above, since a gesture that is unlikely to remind the user of a desired function is assigned, there is a problem that it is difficult for the user to remember the operation. In addition, since the number of gestures is limited, there is also a problem that many functions cannot be assigned.

[0012] Therefore, the present disclosure proposes an information processing device and an information processing method that can further improve the operability with respect to an object in the XR space.

[0013] Solution to the problem

[0014] In order to solve the above problem, one aspect of an information processing device according to the present disclosure includes a control unit that displays an object including a virtual object or a real object in an XR space in a manner that enables operation by a user. The control unit determines a position and a gesture of a user's hand relative to the object; and activates a function related to the operation of the object, the function being assigned to a combination of the position and the gesture of the hand corresponding to the determination result. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a diagram illustrating an exemplary schematic configuration of an information processing device according to an embodiment of the present disclosure.

[0016] Figure 2 This is a conceptual diagram when using an information processing device.

[0017] Figure 3 1 is a schematic explanatory diagram of an information processing method according to an embodiment of the present disclosure (#1).

[0018] Figure 4 It is a schematic explanatory diagram (#2) of an information processing method according to an embodiment of the present disclosure.

[0019] Figure 5 is a block diagram showing a configuration example of an information processing device according to an embodiment of the present disclosure.

[0020] Figure 6 is a diagram (#1) showing a specific example of a function activation control process according to an embodiment of the present disclosure.

[0021] Figure 7 2 is a diagram (#2) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0022] Figure 8 3 is a diagram (#3) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0023] Fig. 9 4 is a diagram showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0024] Fig.10 is a diagram (#5) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0025] Fig.11 is a diagram (#6) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0026] Fig.12 7 is a diagram showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0027] Fig.13 is a diagram (#8) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0028] Fig.14 is a diagram (#9) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0029] Fig.15 10 is a diagram showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0030] Fig.16 is a diagram (#11) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0031] Fig.17 is a diagram (#12) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0032] Fig.18 is a diagram (#13) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0033] Fig.19 is a flowchart showing a processing procedure performed by the information processing device according to an embodiment of the present disclosure.

[0034] Fig. 20 is a diagram (#1) showing a modification of the function activation control process.

[0035] Fig.21 is a diagram (#2) showing a modification of the function activation control process.

[0036] Fig. 22 is a diagram (#3) showing a modification of the function activation control process.

[0037] Fig.23 is a diagram (#4) showing a modification of the function activation control process.

[0038] Fig.24 is a diagram (#5) showing a modification of the function activation control process.

[0039] Fig.25is a diagram (#6) showing a modification of the function activation control process.

[0040] Fig.26 is a hardware configuration diagram showing an example of a computer that realizes the function of the information processing device. DETAILED DESCRIPTION

[0041] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. Note that in each of the following embodiments, the same parts are denoted by the same symbols, and redundant descriptions will be omitted.

[0042] Incidentally, hereinafter, description will be made taking as an example a case where the information processing apparatus according to an embodiment of the present disclosure (hereinafter, referred to as “the present embodiment” where appropriate) is a glasses-type HMD.

[0043] In addition, in the following text, "XR space" is a general term for various spaces in image processing technologies that merge the virtual world and reality, such as virtual reality space in VR, augmented reality space in AR, mixed reality space in mixed reality (MR), and alternate reality space in alternate reality (SR).

[0044] The present disclosure will be described in the order of the following items.

[0045] 1. Overview

[0046] 1-1. Example of schematic configuration of information processing device

[0047] 1-2. Overview of the information processing method according to the present embodiment

[0048] 2. Configuration of information processing equipment

[0049] 3. Specific examples of function activation control processing

[0050] 4. Processing of information processing devices

[0051] 5. Modifications

[0052] 6. Hardware Configuration

[0053] 7. Conclusion

[0054] <<1.Overview>>

[0055] <1-1. Example of schematic configuration of information processing device>

[0056] Figure 1 is a diagram showing an exemplary schematic configuration of the information processing device 10 according to an embodiment of the present disclosure. Figure 2 1 is a conceptual diagram of a case where the information processing device 10 is used. Figure 1As shown in , the information processing device 10 according to the embodiment of the present disclosure is implemented by, for example, a glasses-type HMD worn on the head of a user U. The information processing device 10 includes a sensor unit 11 and a display unit 12 .

[0057] The sensor unit 11 is a sensor group that acquires various types of sensor information about, for example, the situation around the user U or the state of the user U. Figure 1 The example of exemplifies a stereoscopic configuration in which two sensor units 11 are shown, but the number of sensor units 11 is not limited.

[0058] For example, the sensor unit 11 has a function as a camera for capturing an image of the space in front of the eyes of the user U or the movement of the user U. The sensor unit 11 also has a function as an inertial measurement unit (IMU) for measuring, for example, the acceleration and angular velocity of the information processing device 10. The information processing device 10 can recognize the movement state or posture of the user U, a gesture performed by the user U with a hand or a finger, and the like based on the sensor information of the sensor unit 11 having such a function.

[0059] The display unit 12 corresponds to the lens portion of glasses positioned in front of the eye of the user U when worn, and has optical transparency. The display unit 12 includes a right-eye display 121 that displays a right-eye image and a left-eye display 122 that displays a left-eye image.

[0060] When the information processing device 10 is worn by the user U, Figure 1 As shown in , the right-eye display 121 is positioned in front of the right eye of the user U, and the left-eye display 122 is positioned in front of the left eye of the user U.

[0061] Incidentally, the display unit 12 is not necessarily divided into the right eye display 121 and the left eye display 122, and a right eye image may be displayed on the right side of the integrally formed display, and a left eye image may be displayed on the left side of the display.

[0062] The information processing device 10 can realize stereoscopic vision of various objects O in the XR space by displaying the right eye image and the left eye image on the display unit 12, including Figure 2 A virtual object in front of the user U in the line of sight direction is shown in .

[0063] Note that in the following description, the object O will be described as having Figure 2 , but the shape of the object O to be operated by the user U is not limited. In addition, in the following description, reference numerals such as objects O1, O2, O3, ... may be assigned according to differences in types of the objects O. In the case where there is no need to consider the difference in type, the objects are collectively referred to as objects O.

[0064] In addition, the shape of the information processing device 10 is not limited to Figure 1 and Figure 2 For example, the information processing device 10 may be a headband type HMD using a band around the entire circumference of the temple or a band passed through the top of the head, a helmet type HMD in which the visor portion is used as a display, or the like.

[0065] <1-2. Overview of Information Processing Method According to the Present Embodiment>

[0066] The information processing device 10 recognizes various operations of the user U with respect to the object O, such as rotation, movement, size change, or design change, based on the sensor information of the sensor unit 11, and activates a function corresponding to the recognized operation to reflect the operation in the object O. As described above, in recent years, such XR has been significantly spread not only in the entertainment field such as games but also in the design support field such as CAD.

[0067] As a prior art for recognizing the operation of user U and reflecting the operation in object O, as described above, there is the following known technology: a predetermined gesture by a hand or finger is assigned to each of the functions related to the object operation, and the function corresponding to the gesture is activated when the gesture is recognized using sensor information of the sensor unit 11.

[0068] However, in the prior art, since a gesture that is unlikely to remind the user U of the function desired by the user U is assigned, there is a problem that it is difficult for the user U to remember the operation. In addition, there is also a problem that many functions cannot be assigned due to the limited number of gestures. That is, the prior art has room for further improvement in operability with respect to the object O in the XR space.

[0069] Therefore, in the information processing method according to the present embodiment, the information processing device 10 displays an object O including a virtual object or a real object in the XR space in a manner that enables it to be operated by a user U, determines the position and gesture of the user U's hand relative to the object O, and activates a function related to the operation of the object O, which is assigned to the combination of the hand position and gesture corresponding to the determination result.

[0070] Figure 3 1 is a schematic explanatory diagram of an information processing method according to an embodiment of the present disclosure (#1). Figure 4 is a schematic diagram (#2) of an information processing method according to an embodiment of the present disclosure. Figure 3 As shown in , the information processing apparatus 10 first sets a direction in the object O1 to be operated by the user U (step S1 ).

[0071] More specifically, the information processing device 10 divides the surrounding environment of the object O1. Figure 3 As shown in , in the case where the object O1 has a cubic shape, the information processing device 10 divides the surrounding environment into six directions, such as up, down, left, right, front, and back.

[0072] Note that in the following description, for object O, the surface corresponding to the upward direction can be called the top surface SU, the surface corresponding to the downward direction can be called the bottom surface SD, the surface corresponding to the left direction can be called the left surface SL, the surface corresponding to the right direction can be called the right surface SR, the surface corresponding to the forward direction can be called the front surface SF, and the surface corresponding to the backward direction can be called the back surface SB.

[0073] In addition, the division into six parts shown herein is only an example, and the number of divided parts is not limited. The number of divided parts may be, for example, ten or sixteen.

[0074] Then, the information processing device 10 selects a function related to the operation to be performed on the object O1 from among the functions through the position and gesture of one of the two hands (e.g., the left hand HL) of the user U relative to the object O1. In addition, the information processing device 10 causes the object O1 to be actually operated by the other hand (e.g., the right hand HR) of the two hands.

[0075] Specifically, Figure 3 As shown in , the information processing device 10 determines the position and gesture of one of the two hands (in this case, the left hand HL) relative to the object O1 based on the sensor information of the sensor unit 11 (step S2). Then, the information processing device 10 activates a function related to the object operation corresponding to the determination result (step S3).

[0076] Then, the information processing device 10 operates the object O1 according to the movement of the other hand (in this case, the right hand HR) through the activated function (step S4 ). Figure 3 The following example is shown, in which, when the user U makes a gesture of positioning the left hand HL at a predetermined position on the left side of the object O1 and pointing the finger to the left side SL, the information processing device 10 sets a rotation axis axR1 in the left-right direction passing through the left side SL and the right side SR for the object O1, and activates the function of rotating the object O1.

[0077] In this case, when the user U performs a motion of pinching the object O1 with the right hand HR and moves the object O1 in the up-down direction, the information processing device 10 rotates the object O1 around the rotation axis axR1 (see the arrow a1 in the figure). Note that when the rotation function is activated in step S3, the information processing device 10 may display the rotation axis axR1 and the arrow a1 for guidance or operation support for the user U.

[0078] at the same time, Figure 4 The following example is shown, in which, when the user U makes a gesture of positioning the left hand HL at a predetermined position on the left side of the object O1 and pointing to the left surface SL with two fingers, the information processing device 10 sets a movement axis axM1 in the left-right direction passing through the left surface SL and the right surface SR and activates the function of moving the object O1.

[0079] In this case, when the user U performs a motion of pinching the object O1 with the right hand HR and moves the object O1 in the left-right direction, the information processing device 10 moves the object O1 along the movement axis axM1 (see arrow a2 in the figure). Figure 3 Similar to the case of , when the movement function is activated in step S3 , the information processing device 10 may display the movement axis axM1 and the arrow a2 for guidance or operation support for the user U.

[0080] exist Figure 3 and Figure 4 In the two examples, it can be seen that the information processing device 10 sets the rotation axis axR1 or the movement axis axM1 according to the direction pointed by the left hand HL. That is, in the information processing method according to the present embodiment, the direction of the finger of the left hand HL of the user U corresponds to the axial direction of the operation to be performed on the object O1, so that the user U can easily recall the function desired by the user U.

[0081] As a result, the information processing method according to the present embodiment can facilitate the user U to easily remember various gestures for various operations on the object O1.

[0082] In addition, in the information processing method according to this embodiment, since each function related to object operation is activated according to the position and gesture of the left hand HL relative to the object O1, more functions can be assigned to the combination of at least the position and gesture of the left hand HL.

[0083] Therefore, according to the information processing method based on the present embodiment, the operability with respect to the object O in the XR space can be further improved.

[0084] Note that in the following, the information processing device 10 activates the object O according to the determination result of the position and gesture of one of the two hands of the user U relative to the object O. Figure 3 and Figure 4 The processing of such object operation-related functions shown in the figure can be referred to as "function activation control processing". Figure 6 and subsequent figures to describe Figure 3 and Figure 4Specific Examples and Various Modifications of Function Activation Control Processes Other Than Illustrated Hereinafter, a configuration example of the information processing apparatus 10 to which the information processing method according to the embodiment is applied will be described more specifically.

[0085] <<2. Configuration of information processing device>>

[0086] Figure 5 1 is a block diagram showing a configuration example of an information processing device 10 according to an embodiment of the present disclosure. Figure 5 In the present invention, only components necessary for describing the features of the embodiments are shown, and description of general components is omitted.

[0087] In other words, Figure 5 Each component shown in the figure is conceptual in terms of function and is not necessarily physically configured as shown in the drawings. For example, the specific form of distribution or integration of the frame is not limited to the form shown in the drawings, and all or part of it can be functionally or physically distributed or integrated in any unit according to various loads, usage status, etc.

[0088] Note that when using Figure 5 In the description of the present invention, the description of the components that have already been described may be simplified or omitted.

[0089] like Figure 5 As shown in , the information processing device 10 includes a sensor unit 11 , a display unit 12 , a communication unit 13 , a storage unit 14 , and a control unit 15 .

[0090] As described above, the sensor unit 11 is a sensor group that acquires various types of sensor information about the situation around the user U and the state of the user U. The sensor unit 11 includes an imaging device 11 a and an inertial sensor 11 b .

[0091] The camera 11 a captures an image of a subject (ie, a real object located in a real space) located in front of the user U. The subject located in front of the user U includes the user U's hands and fingers.

[0092] The inertial sensor 11b corresponds to the IMU described above. The inertial sensor 11b includes an acceleration sensor and an angular velocity sensor (not shown). The inertial sensor 11b measures the acceleration and angular velocity of the information processing device 10. The information processing device 10 recognizes the orientation of the head or body of the user U, the posture, the movement of the body, the direction of the field of view, the speed of viewpoint movement, etc. based on the sensor information of the inertial sensor 11b.

[0093] Note that the sensor unit 11 does not necessarily need to be mounted on the information processing device 10. The sensor unit 11 may be, for example, an external sensor group connected to the information processing device 10 in a wired or wireless manner.

[0094] Since the display unit 12 has been described, its description will be omitted here. The communication unit 13 is implemented by a network adapter, etc. The communication unit 13 is connected to a communication network such as the Internet in a wired or wireless manner, and transmits and receives information to and from another communication device (e.g., an external sensor group or a server device) via the communication network.

[0095] The storage unit 14 is implemented by, for example, a storage device such as a random access memory (RAM), a read only memory (ROM), or a flash memory. Figure 5 In the example of FIG. 1 , the storage unit 14 stores function activation control information 14 a.

[0096] The function activation control information 14 a is information related to function activation control processing, and includes, for example, definition information of various functions related to object operations assigned to each combination of the position of the hand of the user U relative to the object O and a gesture.

[0097] Furthermore, although not shown, the storage unit 14 stores various programs operated in the information processing device 10 including the program of the present disclosure.

[0098] The control unit 15 is a controller and is implemented by, for example, a central processing unit (CPU), a microprocessing unit (MPU), or the like that executes various programs stored in the storage unit 14 that uses the RAM as a work area. In addition, the control unit 15 can be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0099] The control unit 15 includes an acquisition unit 15a, an estimation unit 15b, a position determination unit 15c, a gesture determination unit 15d, a function activation control unit 15e, and an output control unit 15f, and implements or executes functions or actions of information processing described below.

[0100] The acquisition unit 15a acquires various types of sensor information from the sensor unit 11. The estimation unit 15b estimates the position and gesture of the hand of the user U based on the sensor information acquired by the acquisition unit 15a.

[0101] The position determination unit 15c determines the position of one of the two hands of the user U (in the present embodiment, the left hand HL) relative to the object O based on the estimation result of the estimation unit 15b. In addition, the position determination unit 15c determines the position of the other hand (in the present embodiment, the right hand HR) for the activated function related to the object operation. Note that the position determination unit 15c determines whether the position of the hand is at a predetermined position; however, the predetermined position does not strictly refer to the position of one point, but may refer to a predetermined area.

[0102] The gesture determination unit 15d determines a gesture performed with one hand by the user U based on the estimation result of the estimation unit 15b. Furthermore, the gesture determination unit 15d determines the movement of the other hand with respect to the activation function related to the object operation.

[0103] The function activation control unit 15e selects a function to be activated regarding the object operation based on the function activation control information 14a and the determination results of the position determination unit 15c and the gesture determination unit 15d. The function activation control unit 15e further activates the selected function.

[0104] The output control unit 15f controls output of display, sound, tactile sensation, etc. with respect to the XR space. For example, the output control unit 15f changes the display of the object O on the display unit 12 according to the movement of the other hand relative to one hand through the function activated by the function activation control unit 15e.

[0105] <<3. Specific example of function activation control processing>>

[0106] Next, we will refer to Figures 6 to 18 Various specific examples of the function activation process performed by the information processing device 10 are described. Figures 6 to 18 1 are diagrams (#1) to (#13) showing a specific example of the function activation control process according to an embodiment of the present disclosure.

[0107] In the already shown Figure 3 and Figure 4 , an example has been described in which, in a case where the user U positions the left hand HL on the left side of the object O1 and performs a gesture pointing to the object O1 with one or two fingers, the rotation function or the movement function of the object O1 is activated.

[0108] At the same time, if Figures 6 to 9 As shown in , the function to be activated can be determined by the position of the left hand HL relative to the object O1, and the axis of the determined function can be determined by the direction of the finger at this position.

[0109] In this case, specifically, Figure 6 and Figure 7As shown, for example, in a case where the user U positions the left hand HL under the object O1 , the function activation control unit 15 e selects the rotation function of the object O1 as the function to be activated.

[0110] Then, if Figure 6 As shown in FIG. 1 , for example, when the user U performs a gesture pointing in the right direction with the left hand HL positioned at a predetermined position below the object O1, the function activation control unit 15e sets a rotation axis axR1 in the left-right direction for the object O1. In addition, the function activation control unit 15e rotates the object O1 around the rotation axis axR1 by the movement of the right hand HR (see arrow a1 in the figure).

[0111] At the same time, if Figure 7 As shown in, for example, when the user U performs an upward pointing gesture with the left hand HL positioned at a predetermined position below the object O1, the function activation control unit 15e sets a rotation axis axR2 in the up-down direction for the object O1. In addition, the function activation control unit 15e rotates the object O1 around the rotation axis axR2 by the movement of the right hand HR (see arrow a3 in the figure).

[0112] In addition, if Figure 8 and Fig. 9 As shown in , for example, in a case where the user U positions the left hand HL on the left side of the object O1 , the function activation control unit 15 e selects the movement function of the object O1 as the function to be activated.

[0113] Then, if Figure 8 As shown in FIG. 1 , for example, when the user U performs a gesture pointing in the right direction with the left hand HL positioned at a predetermined position on the left side of the object O1, the function activation control unit 15e sets a movement axis axM1 in the left-right direction for the object O1. In addition, the function activation control unit 15e moves the object O1 along the movement axis axM1 by the movement of the right hand HR (see arrow a4 in the figure).

[0114] At the same time, if Fig. 9 As shown in FIG. 1 , for example, when the user U performs an upward pointing gesture with the left hand HL positioned at a predetermined position on the left side of the object O1, the function activation control unit 15e sets a movement axis axM2 in the up-down direction for the object O1. In addition, the function activation control unit 15e moves the object O1 along the movement axis axM2 by the movement of the right hand HR (see arrow a5 in the figure).

[0115] Furthermore, the part to be operated in the object O1 can be made selectable according to the orientation of the hand. A specific example of this is shown in Fig.10 and Fig.11 Shown in.

[0116] First, if Fig.10 As shown in , in a case where the left hand HL is positioned on the left side of the object O1 with the palm facing the object O1 , the function activation control unit 15 e activates the function of selecting the right face SR.

[0117] At the same time, if Fig.11 As shown in , in a case where the left hand HL is positioned on the left side of the object O1 with the back of the hand facing the object O1 , the function activation control unit 15 e activates the function of selecting the left side SL.

[0118] In these examples, in other examples, when the palm is regarded as the first surface and the back of the hand is regarded as the second surface, the function activation control unit 15e selects a surface on the side corresponding to the first surface from the surfaces of the object O1 that are substantially parallel to the first surface and the second surface, according to the orientation of the hand when the hand of the user U is compared to the object O1. Note that in this example, a surface on one side corresponding to the second surface can be selected. In this way, the function activation control unit 15e can determine the surface to be operated in the object O1 according to the orientation of the hand.

[0119] In addition, regarding Fig.10 and Fig.11 In the example, you can use not only the left-hand HL but also the right-hand HR to specify the detailed part of the selected surface. Fig.12 A specific example of this situation is shown in . Fig.12 The example of is based on the assumption that the object O2 has two protrusions on the right surface SR.

[0120] like Fig.12 As shown in Fig.10 Similar to the example of , in a case where the left hand HL is positioned on the left side of the object O1 with the palm facing the object O1, the function activation control unit 15e activates the function of selecting the right face SR.

[0121] Furthermore, for example, when the right hand HR is positioned at a predetermined position on the right side of the object O2 and points to one of the two protrusions, the function activation control unit 15e selects the face SR1 of the protrusion. In the related art, if only one hand is used, it has been difficult to specify a desired face, side, or point to be selected; however, by Fig.12 By preselecting the right surface SR with the left hand HL as in the example of FIG. 1 , it is possible to easily specify the surface SR1 which can be said to be a part of the right surface SR.

[0122] In addition, the guide display for operation support for the user U described above is not limited to Figure 3 and Figure 4 A specific example of this situation is shown in Figures 13 to 16 Shown in.

[0123] exist Fig.13 In the example of , based on the premise that in a case where the left hand HL is positioned at a predetermined position on the left side of the object O1 and a gesture pointing upward is performed, the movement function of moving the object O1 is activated.

[0124] In this case, when the movement function is activated, the function activation control unit 15e causes the output control unit 15f to display arrows a6 to a11 corresponding to the six directions described above as guides, such as: Fig.13 As a result, the user U can easily grasp the movement function that has been activated by the gesture of the left hand HL, and can move the object O1 along each of the arrows a6 to a11.

[0125] In addition, if Fig.14 As shown in , in the case where the user U positions the left hand HL in any of the upward, downward, left, right, forward and backward directions of the object O1, a menu M1 including a list of gesture candidates may be displayed around the left hand HL as a guide.

[0126] In this case, the gesture determination unit 15d determines that a gesture has been performed at a point when the state of the left hand HL changes from a state when the left hand HL moves in one of the upward, downward, leftward, rightward, forward, and backward directions relative to the object O1. Note that a function to be activated may be assigned to a combination of a position, a first gesture, and a second gesture, such as first gesture->movement of the position of the left hand HL->second gesture.

[0127] In addition, despite Fig.14 1 shows an example in which the menu M1 is displayed as a guide in the case where the user U positions the left hand HL in any of the upward, downward, leftward, rightward, forward, and backward directions relative to the object O1, but as Fig.15 As shown in , an expected position with respect to the user U may be estimated before the user U moves the left hand HL, and a menu M1 corresponding to the estimated position may be displayed in advance.

[0128] In this case, the estimation unit 15b estimates the orientation of the information processing device 10, the movement of the user U's line of sight, the indicative movement before the left hand HL is moved, etc. based on the sensor information acquired by the acquisition unit 15a, and estimates the expected position of the user U based on the estimation results.

[0129] exist Fig.14 and Fig.15 In the example shown in , for example, even if the user U does not remember all gestures that can be performed for the operation of the object O1 in each of the directions relative to the object O1, the user U can easily grasp the gestures that can be performed while referring to the menu M1.

[0130] Note that in XR, there is a case where a real object existing in the XR space is set as object O and an operation such as color change is performed.

[0131] In this case, if Fig.16 As shown in , when the user U positions the left hand HL in any direction of the upward, downward, left, right, forward and backward directions of the object O3 which is a real object, the function activation control unit 15e activates, for example, a color change function and displays a color palette CP around the left hand HL.

[0132] The position of the left hand HL relative to the object O for activating the function related to the object operation may include not only the position at a single point in time but also the trajectory information of the left hand HL before it moves to a specific position. A specific example of this case is Fig.17 and Fig.18 Shown in.

[0133] exist Fig.17 In the example, Fig.13 is similar to the example shown in , which is based on the premise that, in the case where the left hand HL is positioned at a predetermined position on the left side of the object O1 and performs a gesture pointing upward, the movement function of moving the object O1 is activated. Fig.17 In the example of , the position determination unit 15 c determines the position of the left hand HL including the trajectory information before the left hand HL reaches the predetermined position with respect to the predetermined position on the left side of the object O1.

[0134] Specifically, if Fig.17 As shown in , in this case, the position determination unit 15c activates the movement function of the object O1 when the left hand HL moves from the near side of the object O1 toward the left side along the trajectory indicated by the arrow a12 and reaches a predetermined position on the left side of the object O1.

[0135] At the same time, Fig.18 In the example of , based on the premise that in the case where the left hand HL is positioned at a predetermined position on the left side of the object O1 and performs a gesture pointing upward, the size change function of enlarging or reducing the object O1 is activated. Fig.18 In the example, Fig.17 Similar to the example of , the position determining unit 15 c determines the position of the left hand HL including the trajectory information before the left hand HL reaches the predetermined position relative to the predetermined position on the left side of the object O1.

[0136] Specifically, Fig.18As shown in , in this case, the position determination unit 15c activates the size changing function of the object O1 when the left hand HL moves from a predetermined position on the left side of the object O1 along the trajectory indicated by the arrow a13 from a position further to the left and reaches the predetermined position on the left side of the object O1.

[0137] In this way, by setting the position of the left hand HL to include the trajectory information before the left hand HL reached the position, the combination patterns of the position of the left hand HL and the gesture increase, so that more functions can be assigned.

[0138] <<4. Processing process of information processing device>>

[0139] Next, we will refer to Fig.19 The processing procedure executed by the information processing apparatus 10 is described. Fig.19 : is a flowchart showing a processing procedure performed by the information processing device 10 according to an embodiment of the present disclosure.

[0140] like Fig.19 As shown in , the control unit 15 of the information processing device 10 determines whether a hand exists at a predetermined position relative to the object O based on the sensor information from the sensor unit 11 (step S101).

[0141] If there is no hand at the predetermined position relative to the object O (step S101 , No), the control unit 15 repeats the process from step S101 .

[0142] On the other hand, if a hand exists at a predetermined position relative to the object O (step S101 , Yes), the control unit 15 determines whether a predetermined gesture has been performed at the position based on sensor information from the sensor unit 11 (step S102 ).

[0143] If the predetermined gesture is not performed at the position (step S102 , No), the control unit 15 repeats the process from step S101 .

[0144] On the other hand, if a predetermined gesture has been performed at the position (step S102, Yes), the control unit 15 activates a function corresponding to the position and gesture (step S103). Then, the control unit 15 repeats the process from step S101.

[0145] <<5. Modification>>

[0146] Meanwhile, the embodiments of the present disclosure described above may have several modifications.

[0147] Figure 20 to Figure 251 to 6 are diagrams showing modifications of the function activation control process. In the present embodiment described above, a case is described where the distance between the user U and the object O is short and the user U can position the hand at a predetermined position in any of the upward, downward, leftward, rightward, forward, and backward directions of the object O; however, it is also conceivable that the distance between the user U and the object O is long and remote operation is required.

[0148] In this case, the function activation control unit 15e can assign a function related to the operation of the object O to, for example, the position and gesture of the hand relative to the individual (user U). In addition, a copy of the remote object O can be generated and displayed at a position in front of the user U where the hand can reach, and a function related to the operation of the remote object O can be assigned to the position and gesture of the hand relative to the copy. A specific example of this case is shown in FIG. Figure 20 to Figure 22 Shown in.

[0149] In the case of remote operation, such as Fig. 20 As shown by way of example in FIG. 1 , the user U performs a gesture of moving the left hand HL to the left side of the user's body and raising the index finger while facing the object O1. Then, the function activation control unit 15e performs a gesture with, for example, Fig. 9 In a manner similar to that shown in , the object O1 is set with a movement axis axM2 in the up-down direction, and a movement function that enables the object O1 to move along the movement axis axM2 by movement of the right hand HR (see arrow a5 in the figure) is activated.

[0150] In addition, if Fig.21 As another example shown in FIG. 1 , in the case of remote operation, the user U performs a gesture of moving the left hand HL to the right side of the user's body and raising the index finger while facing the object O1. Then, the function activation control unit 15e performs a gesture with, for example, Figure 7 In a manner similar to that shown in , the object O1 is set with a rotation axis axR2 in the up-down direction, and a rotation function that enables the object O1 to rotate around the rotation axis axR2 by movement of the right hand HR (see arrow a3 in the figure) is activated.

[0151] As a result, functions related to the operation of the object O can also be assigned to the remote object O through the position and gesture of the user U's hand.

[0152] In addition, if Fig. 22As another example shown in FIG. 1 , in the case of remote operation, the function activation control unit 15e generates a copy of the remote object O11, for example, linked to the object O11, and displays the copy in front of the user U at a position where the user U can reach with his hand. Note that the scale of the object O12 as a copy is adjusted and displayed as needed. Then, the function activation control unit 15e activates the function related to the operation of the object O11 according to the position and gesture of the left hand HL of the user U relative to the object O12.

[0153] exist Fig. 22 In the example of Fig. 20 In a manner similar to that shown in , the object O11 is set with a movement axis axM2 in the up and down directions, and a movement function is activated that enables the object O11 to move along the movement axis axM2 (see arrow a5 in the figure) through movement of the right hand HR relative to the object O11.

[0154] As a result, functions related to the operation of object O can also be assigned to the remote object O through the position and gesture of the user U's hand relative to the copy of the remote object O.

[0155] Furthermore, even in the case of the same gesture, different functions can be assigned to be activated if the orientation of the hand is different. Fig.23 As shown in, for example, different orientation modes of the hand can be set, such as the palm of a hand facing "down", "up", "left", "right", "forward" and "backward", and even the same gesture can activate different functions according to the orientation of the hand.

[0156] In this case, for example, the function activation control unit 15e activates the rotation function when the index finger is raised with the palm facing “downward”, or activates the movement function when the index finger is raised with the palm facing “upward”. This makes it possible to assign more functions.

[0157] The orientation of the hand may be detected by a hand tracking function of the camera 11 a or an externally mounted camera, or may be detected by an inertial sensor or the like on a device held by the user U or mounted on the hand.

[0158] Furthermore, in the present embodiment described above, an example has been described in which a function to be activated is selected by the position and gesture of the left hand HL of the user U relative to the object O, and the object O is actually operated by the right hand HR. However, the roles of the left hand HL and the right hand HR may be reversed.

[0159] In addition, in the present embodiment described above, functions are assigned to the position of the hand and the gesture. However, the position of the device held or mounted to the hand by the user U may be used instead of the position of the hand, and input from a button or sensor on the device may be used instead of the gesture.

[0160] For example, Fig.24 As shown in , the function activation control unit 15 e replaces the position of the hand with the position of the XR controller 100 held by the user U, and replaces the gesture with the input from the button or sensor on the XR controller 100 .

[0161] In addition, if Fig.24 As shown in FIG. 1 , when the XR controller 100 is positioned at a predetermined position on the left side of the object O1 and a specific button on the XR controller 100 is pressed, the function activation control unit 15 e activates the object O1 in a manner similar to that shown in FIG. Fig.13 The movement function of the object O1 is activated in a manner similar to that shown in FIG.

[0162] In addition, in the present embodiment described above, in step S1 described above, the surroundings of the cube object O1 are divided into six directions such as up, down, left, right, front and back, and these directions are set. However, the number of divisions and the division positions can be modified within the range where the user U's hand can reach the object O.

[0163] Specifically, Fig.25 As shown in the left figure, for example, each face of the cube object includes a predetermined area for the user to place his hand (see the closed curve area in the figure), whereas if the same object is positioned at a higher position as shown in the right figure, it becomes difficult for the user U's hand to reach the top side, etc.

[0164] In this case, the predetermined area where the user places the hand can be changed according to the height position of the object, for example, at any number of divisions and division positions around the bottom surface side of the object. In addition, a set of valid gestures in each of the predetermined areas can be changed so that the user U can easily perform gestures.

[0165] Furthermore, the gesture determination unit 15d can determine whether the motion of the hand is intended to activate a function or to simply move the hand. In this case, for example, in the case where the user U moves a part other than fingers such as the back of the hand or the wrist, the gesture determination unit 15d determines that the user U intends to simply move the hand, and the function will not be activated even if the gesture is detected. On the other hand, in the case where the back of the hand or the wrist does not move and the hand is at a predetermined position, the gesture determination unit 15d determines that the user intends to activate the function and activates the function when the gesture is detected.

[0166] Furthermore, the mode may be switched between a mode in which a function is activated by the position and gesture of the hand relative to the object O and a mode in which the mode is disabled.

[0167] It is also possible to keep the function activated while a gesture is being performed with the left hand HL (for example, the left hand HL is placed on the left side of the object O and the rotation operation is continued with the right hand HR while the finger pointing gesture is continued). It is also possible to keep the function activated even if the user puts down the hand after performing a gesture with the left hand HL once (for example, after placing the left hand HL on the left side of the object O and performing the finger pointing gesture, the rotation operation can be performed with the right hand HR even if the left hand HL is separated from the object O).

[0168] In the latter case, for example, to fix a function, the function may be fixed or canceled by the user U verbally saying “lock”, performing a specific input from the XR controller 100 , mouse, keyboard, etc., or performing a specific gesture.

[0169] Furthermore, when using the same function as a previously used function, the previous function can be activated only by moving the hand to a predetermined position without performing a gesture.

[0170] In addition, when the user places the hand at a predetermined position such as the upper side, lower side, left side, right side, front side or back side of the object O, the shape, color or transparency of the virtual object (e.g., a 3D model of the hand) following the hand can be changed to notify the user of the gesture standby state.

[0171] Furthermore, in the present embodiment described above, an example has been described in which the information processing device 10, i.e., the HMD, is the main execution subject of the information processing of the present embodiment; however, a server device (not shown) provided so as to enable it to communicate with the HMD may be the main execution subject. In this case, the control unit of the server device includes, for example, processing units each corresponding to one of the estimation unit 15b, the position determination unit 15c, the gesture determination unit 15d, and the function activation control unit 15e, and performs information processing performed by these processing units based on the sensor information transmitted from the HMD.

[0172] Between the types of processing in the embodiments described above, all or part of the processing described as being performed automatically may be performed manually, or all or part of the processing described as being performed manually may be performed automatically by known methods. In addition, unless otherwise specified, the processing procedures, specific names, and information including various types of data or parameters shown above or in the accompanying drawings may be modified as needed. For example, the various types of information shown in the accompanying drawings are not limited to the information shown.

[0173] In addition, each component of each device shown in the drawings is conceptual in function and is not necessarily physically configured as shown in the drawings. That is, the specific forms of distribution and integration of the devices are not limited to those shown in the drawings, and all or part of them may be functionally or physically distributed or integrated in any unit according to various loads, usage status, etc.

[0174] Furthermore, the above embodiments may be appropriately combined as long as the processing contents do not contradict each other.

[0175] <<6. Hardware Configuration>>

[0176] The information processing device 10 according to the present embodiment described above is composed of, for example, Fig.26 The configuration shown in FIG. 1 is implemented by a computer 1000 . Fig.26 10 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device 10. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an auxiliary storage device 1400, a communication interface 1500, and an input and output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0177] The CPU 1100 operates and controls each of the components according to the program stored in the ROM 1300 or the auxiliary storage device 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the auxiliary storage device 1400 in the RAM 1200 and executes processing corresponding to the various programs.

[0178] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on the hardware of the computer 1000 , and the like.

[0179] The auxiliary storage device 1400 is a computer-readable recording medium that non-transitorily records a program to be executed by the CPU 1100 , data used by such a program, etc. Specifically, the auxiliary storage device 1400 is a recording medium that records a program according to the present disclosure, which is an example of the program data 1450 .

[0180] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.

[0181] The input and output interface 1600 is an interface for connecting the input and output device 1650 to the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input and output interface 1600. The CPU 1100 also sends data to an output device such as a display, a speaker, or a printer via the input and output interface 1600. In addition, the input and output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium, etc. The medium refers to, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase change rewritable disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), a magnetic tape medium, a magnetic recording medium, or a semiconductor memory.

[0182] For example, in the case where the computer 1000 is used as the information processing device 10 according to the present embodiment, the CPU 1100 of the computer 1000 realizes the function of the control unit 15 by executing the program loaded on the RAM 1200. In addition, the auxiliary storage device 1400 stores the program or data according to the present disclosure in the storage unit 14. Note that although the CPU 1100 reads the program data 1450 from the auxiliary storage device 1400 and executes the program data 1450, as another example, these programs may be acquired from another device via the external network 1550.

[0183] <<7. Conclusion>>

[0184] As described above, according to an embodiment of the present disclosure, the information processing device 10 includes a control unit 15 that displays an object including a virtual object or a real object in an XR space so that a user can operate the object. In addition, the control unit 15 determines the position and gesture of the hand of the user U relative to the object O, and activates a function related to the operation of the object O, which is assigned to a combination of the position and gesture of the hand corresponding to the determination result. As a result, the operability relative to the object O in the XR space can be further improved.

[0185] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above embodiments as they are, and various modifications may be made without departing from the gist of the present disclosure. In addition, components of different embodiments and modifications may be appropriately combined.

[0186] Furthermore, the effects of the embodiments described herein are merely examples and not limitations, and other effects may be achieved.

[0187] Note that the present technology may also have the following configurations. (1)

[0189] An information processing device includes a control unit that displays an object including a virtual object or a real object in an XR space in a manner that enables operation by a user, wherein:

[0190] The control unit:

[0191] determining a position and gesture of the user's hand relative to the object; and

[0192] A function related to the operation of the object is activated, the function being assigned to a combination of the position of the hand and the gesture corresponding to the determination result. (2)

[0194] The information processing device according to (1), wherein:

[0195] The control unit

[0196] The function is activated based on the position of the hand and the gesture of a first hand of the user's two hands. (3)

[0198] The information processing device according to (2), wherein:

[0199] The control unit

[0200] The object is caused to be physically manipulated by movement of a second hand of the two hands relative to the first hand. (4)

[0202] The information processing device according to (2) or (3), wherein:

[0203] The control unit

[0204] The function is activated in case the user performs a gesture pointing to the object such that the pointing direction forms an axial direction of an operation to be performed on the object. (5)

[0206] The information processing device according to (4), wherein:

[0207] The control unit

[0208] In case a rotation function of rotating the object is activated as the function, a rotation axis in the pointing direction is set for the object. (6)

[0210] The information processing device according to (4) or (5), wherein:

[0211] The control unit

[0212] In case a movement function of moving the object is activated as the function, a movement axis in the pointing direction is set for the object. (7)

[0214] The information processing device according to any one of (1) to (6), wherein:

[0215] The control unit

[0216] The position of the hand is determined, including the trajectory of the hand of the user before reaching the position of the hand. (8)

[0218] The information processing device according to (7), wherein:

[0219] The control unit

[0220] The function assigned to the combination of the position of the hand and the gesture is activated, the position also comprising the trajectory of the hand. (9)

[0222] The information processing device according to any one of (1) to (8), wherein:

[0223] The control unit:

[0224] In case of remote manipulation of the object, determining the position and gesture of the hand relative to the user; and

[0225] The function assigned to a combination of the position of the hand relative to the user and the gesture, the position and the gesture corresponding to the determination result, is activated. (10)

[0227] The information processing device according to any one of (1) to (9), wherein:

[0228] the control unit,

[0229] In a case where the palm of the user is regarded as the first surface and the back of the hand is regarded as the second surface, a selection function is activated: when the hand of the user is compared to the object, a surface corresponding to the first surface or the second surface is selected from the surfaces of the object that are substantially parallel to the first surface and the second surface according to the orientation of the hand. (11)

[0231] The information processing device according to (10), wherein:

[0232] The control unit

[0233] The selection function is activated by a first hand of the user's two hands, and a detailed portion of the surface of the object selected by the selection function is allowed to be further specified by a second hand relative to the first hand. (12)

[0235] An information processing method, performed by an information processing device, which displays objects including virtual objects or real objects in an XR space in a manner that enables operation by a user, the information processing method comprising:

[0236] determining a position and gesture of the user's hand relative to the object; and

[0237] A function related to the operation of the object is activated, the function being assigned to a combination of the position of the hand and the gesture corresponding to the determination result. (13)

[0239] A program for causing a computer, which is an information processing device that displays an object including a virtual object or a real object in an XR space in a manner that enables operation by a user, to execute the following:

[0240] determining a position and gesture of the user's hand relative to the object; and

[0241] A function related to the operation of the object is activated, the function being assigned to a combination of the position of the hand and the gesture corresponding to the determination result.

[0242] Reference numerals list

[0243] 10Information processing device

[0244] 11 sensor units

[0245] 11a Camera device

[0246] 11b Inertial Sensor

[0247] 12 display units

[0248] 13 Communication Unit

[0249] 14 Storage Units

[0250] 14a Function activation control information

[0251] 15Control Unit

[0252] 15a Acquisition Unit

[0253] 15b Estimation Unit

[0254] 15c Position determination unit

[0255] 15d gesture determination unit

[0256] 15e function activation control unit

[0257] 15f output control unit

[0258] HL Left Hand

[0259] HR right hand

[0260] O object

[0261] UUser

Claims

1. An information processing device, comprising a control unit that displays an object including a virtual object or a real object in an XR space in a manner that enables operation by a user, wherein: The control unit: determining a position and gesture of the user's hand relative to the object; as well as A function related to the operation of the object is activated, the function being assigned to a combination of the position of the hand and the gesture corresponding to the determination result.

2. The information processing device according to claim 1, wherein: The control unit The function is activated based on the position of the hand and the gesture of a first hand of the user's two hands.

3. The information processing device according to claim 2, wherein: The control unit The object is caused to be physically manipulated by movement of a second hand of the two hands relative to the first hand.

4. The information processing device according to claim 2, wherein: The control unit The function is activated in case the user performs a gesture pointing to the object such that the pointing direction forms an axial direction of an operation to be performed on the object.

5. The information processing device according to claim 4, wherein: The control unit In case a rotation function of rotating the object is activated as the function, a rotation axis in the pointing direction is set for the object.

6. The information processing device according to claim 4, wherein: The control unit In case a movement function of moving the object is activated as the function, a movement axis in the pointing direction is set for the object.

7. The information processing device according to claim 1, wherein: The control unit The position of the hand is determined, including the trajectory of the hand of the user before reaching the position of the hand.

8. The information processing device according to claim 7, wherein: The control unit The function assigned to the combination of the position of the hand and the gesture is activated, the position also comprising the trajectory of the hand.

9. The information processing device according to claim 1, wherein: The control unit: in case of remote manipulation of the object, determining the position and the gesture of the hand relative to the user; as well as The function assigned to a combination of the position of the hand relative to the user and the gesture, the position and the gesture corresponding to the determination result, is activated.

10. The information processing device according to claim 1, wherein: the control unit, In a case where the palm of the user is regarded as the first surface and the back of the hand is regarded as the second surface, a selection function is activated: when the hand of the user is compared to the object, a surface corresponding to the first surface or the second surface is selected from the surfaces of the object that are substantially parallel to the first surface and the second surface according to the orientation of the hand.

11. The information processing device according to claim 10, wherein: The control unit The selection function is activated by a first hand of the user's two hands, and a detailed portion of the surface of the object selected by the selection function is allowed to be further specified by a second hand relative to the first hand.

12. An information processing method, performed by an information processing device, the information processing device displaying an object including a virtual object or a real object in an XR space in a manner that enables operation by a user, the information processing method comprising: determining a position and gesture of the user's hand relative to the object; as well as A function related to the operation of the object is activated, the function being assigned to a combination of the position of the hand and the gesture corresponding to the determination result.

Images