Control device, control method, and program

Abstract

To provide technology that enables combined use of an input operations using an operation device and an input operation using hand tracking.SOLUTION: A control device includes: reception means capable of receiving an input operation using an operation device and an input operation using hand tracking; and determination means for determining, when the input operation by the operation device and the input operation by the hand tracking are received at same time, on the basis of a predetermined priority, which of the input operation by the operating device and the input operation by the hand tracking is to be adopted as an input operation to be used for controlling an information processing system.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] In a cross reality (XR) system that allows a user to experience virtual reality, there are a method of using an operating device such as a hand controller and a method of using hand tracking as a means for operating the display of a head mounted display (HMD). In the former, a user inputs to the HMD by changing the position and posture of the operating device or by operating an operating member such as a button on the operating device. Operating devices that are held in the hand (called grip type, handheld type, etc.) are common, but small operating devices that can be attached to the hand or finger (called wearable type, etc.) have also been proposed (see Patent Document 1). In the latter, hand tracking, a user's hand is recognized from an image captured by a camera, and the position, posture, gesture, etc. of the hand are captured as input operations. For example, Patent Document 2 discloses a method of controlling an HMD by gesture input. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2015 / 153803 [Patent Document 2] International Publication No. 2012 / 135553 Summary of the Invention [Problem to be solved by the invention]

[0004] The method using a control device has the advantage of reducing operational errors (making reliable operation easier), while the method using hand tracking has the advantage of allowing intuitive operation using hands and fingers. Some conventional XR systems allow users to switch between an operation mode using a control device and an operation mode using hand tracking.

[0005] The inventor came up with the idea that the convenience and operability of the XR system could be further improved if it were possible to accept both operations by the operation device and operations by hand tracking, rather than switching between the operation device and hand tracking modes. In particular, in the case of a wearable type, the operation device does not hide the hand (part of the hand is in the camera's blind spot), and the hand and fingers can be moved freely even while using the operation device, making it easier to use the operation device and hand tracking together.

[0006] However, in the process of considering the combined use of an operating device and hand tracking, the inventors faced the following problem. For example, when a user operates an operating device to input a command, the hand holding the operating device (or the hand on which the operating device is attached) moves, and this may be accidentally interpreted as a command input by hand tracking. Conversely, when a user performs a hand gesture, the position or posture of the operating device held in the hand (or attached to the hand) may change, and this may be accidentally interpreted as a command input by the operating device. In such a case, the XR system may recognize (detect) a command input by operating the operating device and a command input by hand tracking at the same time, which may cause control malfunctions or unintended behavior.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and has an object to provide a technique that enables input operations using an operating device and input operations using hand tracking to be used in combination. [Means for solving the problem]

[0008] The present disclosure relates to Worn on the user's first hand Input operation using a wearable device, The first hand having the operation device attached thereto A receiving means capable of receiving an input operation by hand tracking; worn on the first hand An input operation by the operation device; The first hand When the input operation by the hand tracking is simultaneously accepted, based on a predetermined priority, worn on the first hand An input operation by the operation device; The first hand A determination means for determining which of the input operations by the hand tracking is to be adopted as an input operation to be used for controlling an information processing system; providing means for providing an application with the input operation used to control the information processing system determined by the determining means; The control device includes a controller having a The present disclosure also includes a receiving means capable of receiving an input operation by an operating device that is a wearable type device and an input operation by hand tracking, and a determining means for determining, when an input operation by the operating device and an input operation by the hand tracking are received simultaneously, which of the input operation by the operating device and the input operation by the hand tracking to adopt as an input operation to be used for controlling an information processing system, based on a predetermined priority, wherein the determining means includes a control device that performs a process of determining a right-hand input operation to be adopted from a right-hand input operation by an operating device held or worn in the right hand and a right-hand input operation by hand tracking of the right hand, and a process of determining a left-hand input operation to be adopted from a left-hand input operation by an operating device held or worn in the left hand and a left-hand input operation by hand tracking of the left hand. Effect of the Invention

[0009] According to the present invention, it is possible to use both an input operation using an operating device and an input operation using hand tracking. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an overall configuration of an information processing system. [Diagram 2] FIG. 1 is a diagram illustrating the internal configuration of an information processing system. [Diagram 3] FIG. 13 is a diagram showing a display example of an HMD. [Figure 4] FIG. 2 is a diagram showing the software configuration of the image processing apparatus. [Diagram 5] FIG. 11 is a diagram showing an example of information used for processing an input operation. [Figure 6] FIG. 13 is a diagram showing an example of a priority table. [Figure 7] 13 is a flowchart of input operation processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] (Overall system configuration) An information processing system 1 according to an embodiment of the present invention will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, an image processing device 110, and an operation device 120. The information processing system 1 is an XR system for providing a user with a simulated experience (Cross Reality: XR) that combines the real world and the virtual world. XR includes VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), SR (Substitute Reality), etc., but the information processing system 1 is applicable to any type of XR content.

[0013] The HMD 100 is a head-mounted display device (electronic device) that is worn on a user's head 130. The HMD 100 displays a composite image that combines, for example, a captured image of the area in front of the user captured by the HMD 100 with content such as CG (computer graphics) in a form that corresponds to the posture of the HMD 100.

[0014] The image processing device 110 is a control device (electronic device) that controls the HMD 100, processes captured images captured by a camera mounted on the HMD 100, generates content such as CG, and generates composite images to be displayed on the HMD 100. The image processing device 110 is configured by a computer equipped with a CPU (processor) and memory. For example, a smartphone, a tablet terminal, a PC (personal computer), a game machine, or the like can be used as the image processing device 110. The image processing device 110 is connected to the HMD 100 wirelessly or by wire. The image processing device 110 generates a composite image by combining a captured image and CG, and transmits the composite image to the HMD 100. Note that each component of the image processing device 110 is It may be built into the MD100.

[0015] The operation device 120 is a device for inputting instructions (commands) to the HMD 100. When the information processing system 1 is in a specific control mode, when a user performs a predetermined input operation (action) using the operation device 120, a command corresponding to the input operation is received by the information processing system 1, and the HMD 100 and the image processing device 110 are controlled.

[0016] The operation device 120 is also called a "hand controller" or simply a "controller." A controller that is shaped to be held by the user's hand is also called a grip type controller or a handheld type controller, and a controller that is used while being worn on the user's hand or finger is also called a wearable type controller. In this embodiment, for example, as shown in FIG. 1, a ring-shaped operation device 120 is used so that it can be worn on the user's finger. If the operation device 120 can be worn on the user's finger, there is an advantage that the user can freely move his or her hand or fingers while holding the operation device 120, and the hand is less likely to be hidden by the operation device 120. The operation device 120 has a button with a built-in optical track pad (hereinafter referred to as "OTP"). The operation device 120 performs wireless communication with the image processing device 110 by Bluetooth (registered trademark).

[0017] Although the shape of the operation device 120 is a ring type, the shape is not limited to this. For example, the shape of the operation device 120 may be a glove type that can be worn on the hand, or a wristwatch type (bracelet type) that can be worn on the wrist. In this way, the operation device 120 may be in a form that can be held by the user's hand or worn on the hand or wrist so that the user can use it easily. Although only one operation device 120 is shown in FIG. 1, multiple operation devices for operating the HMD 100 may be provided. For example, an operation device for the right hand and an operation device for the left hand may be provided, or the operation devices may be worn on multiple fingers (for example, the thumb and index finger).

[0018] The operation device 120 may include any operation member instead of the OTP, as long as the user can operate it by physical contact. For example, the operation device 120 may include any of a touch pad, a touch panel, a cross key, a button, a joystick, and a track pad device instead of the OTP. Alternatively, if only a change in the position and / or attitude of the operation device 120 itself is used as the operation by the operation device 120, the OTP and other operation members may not be required.

[0019] (Internal structure of HMD) The internal configuration of the HMD 100 will be described with reference to Fig. 2. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, and a posture sensor unit 204.

[0020] The HMD control unit 201 controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 with CG) from the image processing device 110, the HMD control unit 201 displays the composite image on the image display unit 203. By wearing the HMD 100, the user can view the composite image displayed on the image display unit 203. The user can experience various mixed realities, such as CG blended into real space.

[0021] The imaging unit 202 may include two cameras (imaging devices). The two cameras are disposed near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture an image of a space similar to the space the user normally sees. Images of a subject (the area in front of the user) captured by the two cameras are output to the image processing device 110. The two cameras can obtain distance information from the two cameras to the subject by measuring distance using a stereo camera.

[0022] The image display unit 203 displays the composite image. The image display unit 203 has a display panel formed of, for example, a liquid crystal panel or an organic EL panel. When the user is wearing the HMD 100, the display panels are disposed in front of the left and right eyes of the user.

[0023] The orientation sensor unit 204 detects the orientation (and position) of the HMD 100. Then, the orientation sensor unit 204 detects (acquires) the orientation of the user (the user wearing the HMD 100) that corresponds to the orientation (and position) of the HMD 100. The orientation sensor unit 204 has an inertial measurement unit (IMU). The orientation sensor unit 204 outputs information on the user's orientation (orientation information) to the image processing device 110.

[0024] (Internal configuration of image processing device) The internal configuration of the image processing device 110 will be described with reference to Fig. 2. The image processing device 110 is a control device having a control unit 211, a content database (DB) 212, and a communication unit 213.

[0025] The control unit 211 receives an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0026] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 is closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG virtual object that does not exist in real space appears as if it were placed in real space.

[0027] Furthermore, the control unit 211 receives an input operation by the operation device 120, receives an input operation by hand tracking using a captured image obtained from the HMD 100, and controls the display of the HMD 100 based on the input operation. The details of the processing related to the input operation will be described later.

[0028] The content DB 212 is a storage unit that stores information such as CG, etc. The control unit 211 can switch the CG read from the content DB 212 (that is, the CG used to generate a composite image).

[0029] (Internal configuration of the operation device) 2, the internal configuration of the operation device 120 will be described. The operation device 120 includes a control unit 221, an operation unit 222, a communication unit 223, and a motion detection unit 224.

[0030] The control unit 221 controls each component of the operation device 120. The control unit 221 may be configured with a small general-purpose processor, a memory, and a program, or may be configured with a dedicated microprocessor or ASIC.

[0031] The operation unit 222 includes a button with an OTP built in. Information (operation information) regarding pressing or sliding a finger on the OTP on the operation unit 222 is transmitted to the image processing device 110 via the communication unit 223.

[0032] For example, the user can move a pointer displayed on the HMD 100 to a specified position by sliding a finger on the OTP. Also, the user can instruct the HMD 100 and the image processing device 110 to perform a specific process by pressing a button on the OTP. In this way, the user can control the HMD 100 by a combination of sliding a finger on the OTP and pressing a button.

[0033] The communication unit 223 performs wireless communication with the image processing device 110 (the communication unit 213).

[0034] The motion detection unit 224 has an IMU (inertial measurement unit; inertial sensor) capable of detecting inertial information (spatial movement amount and angle) of the operation device 120. In this embodiment, an IMU is used that has a built-in 3-axis (x, y, z) acceleration sensor and a 3-axis gyro sensor and is capable of acquiring inertial information of six degrees of freedom, namely, movement in the x, y, and z directions and rotation around the x, y, and z axes. The inertial information is transmitted to the image processing device 110 via the communication unit 223. Note that the motion detection unit 224 may have any device as long as it is a device that does not hinder the miniaturization of the operation device 120 and is a device that can detect inertial information (information such as positional displacement, speed, or acceleration).

[0035] (Hand tracking, operation with a control device) The information processing system 1 of this embodiment supports both operations using the operating device 120 and operations using hand tracking.

[0036] The operation by the operation device 120 means that the user changes the position or posture of the operation device 120 or operates an operation member such as an OPT provided on the operation device 120 to input a command (instruction) to the HMD 100. The types of input operations by the operation device 120 include, for example, the following variations. Moving the operation device 120 to a predetermined position or area Moving the operation device 120 in a predetermined direction (translation) Moving the operation device 120 along a predetermined path Moving the operation device 120 at a predetermined speed condition (for example, moving it slowly, tilting it quickly, etc.) Rotating the operation device 120 to a predetermined orientation Pressing a button (pressing it fully, pressing it halfway, pressing and holding it, double-clicking it, pressing multiple buttons at the same time, etc.) Specifying a direction using operation devices such as the OPT or cross keys

[0037] An operation by hand tracking is a method of inputting commands (instructions) to the HMD 100 by the position, posture, and movement of the user's own hand or fingers. The hand of the user wearing the HMD 100 is detected from an image captured by an imaging unit 202 provided in the HMD 100, and the hand's position, posture, and movement (gesture) are captured, thereby recognizing the input operation by hand tracking. There are, for example, the following variations in the types of input operations by hand tracking. Moving your hand to a specific location or area Moving your hand in a certain direction - Moving your hand along a specific path - Moving your hand at a specific speed (e.g. moving slowly, tilting quickly, etc.) - Put your hands in a specific position Making certain shapes with your hands or fingers (e.g. thumbs up, peace sign, etc.) Making specific movements with your hands and fingers (e.g. pinching, clenching, waving, etc.)

[0038] When the information processing system 1 (image processing device 110) receives (recognizes) an "input operation" by the operating device 120 or hand tracking, it converts the input operation into a "command" and executes control of the HMD 100 based on the command. A command is a control instruction for the information processing system 1 (HMD 100). For example, it is preferable to prepare commands that are abstract level instructions (such as "SELECT", "NEXT", and "MOVE") used for operating a GUI displayed on the HMD 100 or for operating a virtual object.

[0039] For example, an operation can be realized in which the user presses and holds a button on the operation device 120 to display a menu on the HMD 100, operates the OTP to move the pointer to a desired item, and then presses the button on the OTP to confirm the selection of the item. It is also possible to realize a similar operation using only hand tracking. Furthermore, it is also possible to combine operations using the operation device 120 and operations using hand tracking, such as displaying the menu by operating the operation device 120 and selecting or confirming an item by hand tracking.

[0040] As described below, the correspondence (mapping) between the input operation and the command is set in advance in the information processing system 1, and the information processing system 1 converts (translates) the received input operation into a command according to this setting. The correspondence between the input operation and the command may be freely set by the user. Also, the same movement can be assigned to different commands when performed with the right hand, the left hand, and both hands. As mentioned above, there are various types of input operations using the operating device 120 and input operations using hand tracking, but the information processing system 1 does not need to support all of them. Acceptable (recognizable) input operations can be implemented as appropriate according to the purpose, function, and content.

[0041] (Example of operation using hand tracking) First, an example of controlling the HMD 100 using hand tracking according to the position of the user's hand will be described.

[0042] The control unit 211 of the image processing device 110 determines (recognizes) the position, posture, and finger movement of the hand based on the color and shape of the user's hand shown in the captured image. Then, the control unit 211 controls the HMD 100 according to the position of the hand in the space represented by the composite image (for example, the distance between the fingers of the hand and the CG). Any algorithm may be used for the method of recognizing the fingers from the image. For example, a classical machine learning method represented by a support vector machine may be used, an algorithm based on deep learning such as R-CNN may be used, or a rule-based algorithm may be used.

[0043] For example, as shown in FIG. 3A, the control unit 211 displays CG of panels 301-303 showing three options in a composite image, and calculates the distance between the three panels 301-303 and the tip of the index finger in the space represented by the composite image. Then, the control unit 211 determines that the panel has been selected when the distance between any of the three panels 301-303 and the fingertip is equal to or less than a predetermined distance and continues for a predetermined time. Then, the control unit 211 performs a determination process according to the selected panel. In the example of FIG. 3A, the tip of the index finger of the user's right hand is on the middle panel 302. When this state continues for, for example, one second or more, the control unit 211 interprets that a command (instruction) to select the panel 302 has been input, and executes the process B assigned to the panel 302.

[0044] 3B, an example of controlling the HMD 100 using a virtual ray according to the direction of the user's hand will be described. The control unit 211 controls the HMD 100 according to the position of the virtual ray extending on an extension line of the direction of the user's hand.

[0045] The control unit 211 of the image processing device 110 determines (recognizes) the orientation of the hand based on the color and shape of the user's hand shown in the captured image. At this time, the control unit 211 displays a CG of a ray extending in the direction pointed by the hand on the image display unit 203, as shown in FIG. 3B. Then, the control unit 211 displays (moves) a pointer to the position where the ray is pointed. Therefore, the user can change the position and direction of the ray, as well as the position of the pointer, by changing the orientation of the hand. In the example of FIG. 3B, the pointer 311 of the ray 310 of the right hand points to the panel 302, and the pointer 321 of the ray 320 of the left hand points to the panel 301.

[0046] 3C, when the user makes a pinch gesture (a motion of pinching the hand) with the left hand while pointer 321 is displayed on panel 301, control unit 211 determines that panel 301 has been selected. Then, control unit 211 performs a determination process according to the selected panel 301. Although the pinch gesture has been described as an example, a selection operation may be performed by a hand gesture other than a pinch.

[0047] Information obtained from an IMU (inertial measurement unit; inertial sensor) provided on the operation device 120 worn on the user's hand may be used to control the direction of the ray. For example, the control unit 211 may determine the direction of the ray using information on the direction of the operation device 120 obtained from the IMU instead of information on the direction of the hand obtained by hand tracking. Alternatively, the control unit may improve the rate of agreement between the direction in which the user is pointing and the direction in which the ray extends, based on both information on the tilt and acceleration of the operation device 120 obtained from the IMU and information such as the direction of the fingers determined from an image.

[0048] (Configuration of the software platform of the image processing device 110) Next, the software configuration of the image processing device 110 according to the embodiment will be described with reference to FIG.

[0049] The platform 402 can be configured on an OS 401 such as Google's Android (registered trademark) or Linux (registered trademark). The platform 402 includes a group of device drivers for controlling various hardware, and provides an API for using various hardware to an application 406 running on the platform 402. The device driver group includes a Bluetooth (registered trademark) control unit 403. The platform 402 also includes a function for providing various types of input operation information generated based on data detected by an operation device control unit 404 and a hand tracking control unit 405 to the application 406. Various applications 406 can be installed in the image processing device 110 and run on the OS 401 and platform 402.

[0050] The application 406 includes an input operation mapping unit 407, an event processing unit 408, and a content DB 409. The input operation mapping unit 407 is a conversion unit that converts an input operation (action) by the operation device 120 or hand tracking into abstracted information (command) representing a control instruction for the information processing system 1. When a command is notified from the input operation mapping unit 407, the event processing unit 408 provides a function of executing a process associated with the command. The content DB 409 is the same as the content DB 212 shown in FIG. 2, and stores CG, GUI, and the like of virtual objects to be displayed on the HMD 100.

[0051] Input operation information notified from the operation device control unit 404 and / or the hand tracking control unit 405 is notified to an input operation mapping unit 407, and converted into predetermined abstracted information (commands). For example, information that the A button of a first operation device has been pressed is abstracted into information called "SELECT". The abstracted information is notified to an event processing unit 408 of the application. Furthermore, when supporting a second operation device different from the first operation device, information that the B button of the second operation device has been pressed can also be assigned to the information called "SELECT". This makes it easy for an application to support multiple and diverse operation devices.

[0052] 5 shows an example of input mapping information 412 held by the input operation mapping unit 407, an example of operation device information 410 held by the operation device control unit 404, and an example of hand tracking information 411 held by the hand tracking control unit 405. An example will be described below in which the right hand performs a ray operation using the operation device 120, and the left hand does not hold an operation device and performs a ray operation according to the orientation of the hand, as shown in FIG. 3B.

[0053] The operation device control unit 404 has the name of the operation device 120, the hand (right hand or left hand) holding / wearing it, the button press state, the operation state of the OTP, a three-dimensional position and orientation for ray display, and a three-dimensional position and orientation for displaying the operation device, as operation device information 410. Information such as the button press state, the operation state of the OTP, and the three-dimensional position and orientation is updated at the timing when the operation device 120 is operated or at a fixed time interval (for example, the output rate of the IMU). The position and orientation for ray display is used for ray display, and the position and orientation for displaying the operation device is used when displaying the operation device in CG. If the operation device 120 is different for the right hand and the left hand, the hand holding / wearing it is based on that. If the operation device 120 is common to the right hand and the left hand, it may be based on information manually set by the user, or may be automatically determined based on a captured image. The operation device information held differs depending on the configuration of the operation members, sensors, etc. mounted on the operation device 120.

[0054] The hand tracking control unit 405 has three-dimensional positions and orientations for ray display, hand gestures, and three-dimensional positions of joint positions for each of the right and left hands as hand tracking information 411. The positions and orientations for ray display are used for ray display. The hand tracking information 411 is updated at regular time intervals (for example, the frame rate of the imaging unit 202).

[0055] The input operation mapping unit 407 has, as input mapping information 412, a table that defines the correspondence between the input operation information received from the operation device 120 and hand tracking, and commands, for each of the right hand and the left hand.

[0056] Here, it is assumed that two types of operation devices 120, an XX controller and a YY controller, are available. "XX controller (RIGHT)" means that the XX controller is held or attached to the user's right hand, and "XX controller (LEFT)" means that the XX controller is held or attached to the user's left hand. The same applies to the YY controller.

[0057] The operation device information 410 in Fig. 5 represents a state in which the user is wearing the XX controller on his right hand and is not operating any buttons or OTP. Since the YY controller is not being used by the user, information regarding the YY controller is not included in the operation device information 410. Moreover, the hand tracking information 411 in Fig. 5 represents a state in which both the user's right and left hands are recognized and both the right and left hands are making an open gesture (hands in an open shape).

[0058] First, the input mapping information 412 for the left hand will be described. When the input operation mapping unit 407 receives any of pressing button A of the XX controller (LEFT), pressing button B of the YY controller (LEFT), or a pinch gesture of the hand tracking (left hand), it converts it into a command called "SELECT". The "SELECT" command is notified to the event processing unit 408. When the event processing unit 408 receives the "SELECT" command, it executes a process of selecting an item or virtual object pointed to by the pointer 321 of the left hand. The three-dimensional position and orientation for ray display of the XX controller (LEFT), the three-dimensional position and orientation for ray display of the YY controller (LEFT), and the three-dimensional position and orientation for ray display of the hand tracking (left hand) are converted into a command called "POINTER POSE". The "POINTER POSE" command is notified to the event processing unit 408 together with the three-dimensional position and orientation information, and is used to control the ray 320 and pointer 321 of the left hand.

[0059] In the example of the operation device information 410 in FIG. 5, no input operation related to the left hand occurs (because there is only an operation device held / worn by the right hand). In the example of the hand tracking information 411 in FIG. 5, a hand tracking (left hand) gesture and a three-dimensional position and orientation for ray display can be detected as an input operation related to the left hand. For example, the three-dimensional position and orientation for ray display of the left hand in the hand tracking information 411 is updated according to the position and orientation of the left hand. Then, the input operation mapping unit 407 issues a "POINTER POSE" command, and the event processing unit 408 updates the drawing positions of the ray 320 and the pointer 321 of the left hand. Then, when it is detected that the user has made a pinch gesture with the left hand, the information of the left hand gesture in the hand tracking information 411 is updated to "pinch". Then, the input operation mapping unit 407 issues a "SELECT" command, and the event processing unit 408 performs a process of selecting an item or virtual object at the position of the pointer 321 of the left hand.

[0060] Next, the input mapping information 412 for the right hand will be described. When any of the following occurs: pressing button A of the XX controller (RIGHT), pressing button B of the YY controller (RIGHT), or a pinch gesture of the hand tracking (right hand), it is converted into a "SELECT" command. The "SELECT" command is notified to the event processing unit 408. When the event processing unit 408 receives the "SELECT" command, it executes a process of selecting an item or virtual object pointed to by the pointer 311 of the right hand. The three-dimensional position and orientation for ray display of the XX controller (RIGHT), the three-dimensional position and orientation for ray display of the YY controller (RIGHT), and the three-dimensional position and orientation for ray display of the hand tracking (right hand) are converted into a "POINTER POSE" command. The "POINTER POSE" command is notified to the event processing unit 408 together with the three-dimensional position and orientation information, and is used to control the ray 310 and pointer 311 of the right hand.

[0061] In the example of the operation device information 410 in Fig. 5, the pressing of button A on the XX controller and the three-dimensional position and orientation for ray display can be detected as an input operation related to the right hand. In the example of the hand tracking information 411 in Fig. 5, the gesture of the hand tracking (right hand) and the three-dimensional position and orientation for ray display can be detected.

[0062] In this embodiment, in the input mapping information 412, the three-dimensional position and orientation for ray display of the XX controller (RIGHT) and the three-dimensional position and orientation for ray display of the hand tracking (right hand) are both assigned to the "POINTER POSE" of the right hand. For example, when a user moves his / her right hand, a change in the three-dimensional position and orientation for ray display of the XX controller held / attached to the right hand and a change in the three-dimensional position and orientation for ray display of the hand tracking (right hand) can be detected simultaneously. At this time, it is sufficient if the three-dimensional position and orientation value obtained from the operation device information 410 of the XX controller and the three-dimensional position and orientation value obtained from the hand tracking information 411 of the right hand match, but in reality, there is usually a difference between the two. For this reason, two "POINTER POSE" with different three-dimensional position and orientation values ​​corresponding to the control amount are assigned to the "POINTER POSE" of the right hand. This could result in a situation where a "conflicting OSE" command, that is, two commands (control instructions) with conflicting content, are issued simultaneously. Such "multiple input operations that may occur (be detected) simultaneously and may be interpreted as mutually contradictory commands" are hereinafter referred to as "conflicting input operations." If multiple conflicting input operations are accepted at the same time, it could cause control malfunctions or unintended behavior.

[0063] Therefore, in the information processing system 1 of the present embodiment, a priority (order of precedence) is set in advance for a set of competing input operations. When the information processing system 1 simultaneously receives multiple competing input operations, it determines which of the received input operations to adopt based on the preset priority. Simply, it is sufficient to adopt one input operation with the highest priority among the multiple received input operations. However, this is not limited to this method, and the input operation to be adopted may be selected using multiple evaluation indexes including the priority, rather than determining based only on the priority, or multiple input operations may be combined with a weight according to the priority.

[0064] Fig. 6 shows an example of a priority table that defines the correspondence between priorities, input operations, and commands, as an example of priority setting. Priorities 1 to 3 are set for the three input operations assigned to the "POINTER POSE" of the right hand. If there are other sets of input operations that are in a competing relationship, a priority table is prepared for each set.

[0065] This priority table may be set by the user or may be set by the application creator when creating the application. Alternatively, instead of using such a priority table, it may be defined by a rule that an input operation by the operation device 120 is always given priority over an input operation by hand tracking. This determination based on the priority may be made by the platform 402 or the application 406 (the input operation mapping unit 407 or the event processing unit 408).

[0066] (Example of input operation processing) Next, an example of input operation processing in this embodiment will be described with reference to the flowcharts of Figures 7A and 7B. The processing shown in Figures 7A and 7B is repeatedly executed by the image processing device 110 in a predetermined cycle.

[0067] In step S701, the operation device control unit 404 updates the operation device information 410 based on information acquired from the operation device 120. At this time, the operation device control unit 404 estimates the position and orientation of the operation device 120 from inertial information acquired from the operation device 120, and calculates a three-dimensional position and orientation for ray display and a three-dimensional position and orientation for operation device display. The process of step S701 corresponds to accepting an input operation by the operation device 120.

[0068] In step S702, the hand tracking control unit 405 estimates the positions, postures, joint positions, gestures, and the like of the right and left hands of the user (person wearing the HMD 100) from the images captured by the imaging unit 202 of the HMD 100. Then, the hand tracking control unit 405 updates the hand tracking information 411 based on the estimation results. The process of step S702 corresponds to accepting an input operation by hand tracking.

[0069] In step S703, the input operation mapping unit 407 performs exclusive control of the input operations by the operation device 120 and the input operations by hand tracking received in steps S701 and S702. The exclusive control of the input operations corresponds to determining which of the multiple input operations to adopt when multiple input operations that conflict with each other are received simultaneously.

[0070] In step S704, the input operation mapping unit 407 refers to the input mapping information 412 and performs step S 703 The input operation decided (adopted) in step 2 is converted into a command (a control instruction for the information processing system 1).

[0071] In step S705, the input operation mapping unit 407 notifies the event processing unit 408 of the command converted in step S704. At this time, information on the three-dimensional position and orientation of the operation device 120 or the hand is also notified to the event processing unit 408 together with the command, as necessary. The event processing unit 408 controls the display of the HMD 100 by executing processing according to the notified command.

[0072] Next, the details of the exclusive control process for input operations in step S703 will be described with reference to FIG. 7B.

[0073] In step S706, the input operation mapping unit 407 extracts an input operation performed with the right hand from among the multiple input operations received in steps S701 and S702. For example, input operations such as the three-dimensional position and orientation of the operation device 120 held or worn on the right hand, button and OTP operations, and the three-dimensional position and orientation and gestures by hand tracking of the right hand are extracted. Here, when an input operation by the operation device 120 (called a first input operation) and an input operation by hand tracking (called a second input operation) are extracted, the input operation mapping unit 407 checks whether the first input operation and the second input operation conflict with each other.

[0074] In the example of Figure 5, "Pressing Button A on XX controller (RIGHT)", "Pressing Button B on YY controller (RIGHT)", and "Pinch gesture on hand tracking (right hand)" are assigned to "SELECT" and are in a conflicting relationship. In addition, "3D position and orientation for ray display on XX controller (RIGHT)", "3D position and orientation for ray display on YY controller (RIGHT)", and "3D position and orientation for ray display on hand tracking (right hand)" are also in a conflicting relationship.

[0075] When the first input operation and the second input operation conflict with each other, the input operation mapping unit 407 adopts either the first input operation or the second input operation according to the priority defined in the priority table. In the example of Fig. 6, when "XX controller (RIGHT) three-dimensional position and posture for ray display" and "hand tracking (right hand) three-dimensional position and posture for ray display" are received, "XX controller (RIGHT) three-dimensional position and posture for ray display" is adopted. Note that the input operation "hand tracking (right hand) three-dimensional position and posture for ray display" that was not adopted is discarded.

[0076] In step S707, the input operation mapping unit 407 extracts an input operation performed with the left hand from among the multiple input operations received in steps S701 and S702. For example, input operations such as the three-dimensional position and orientation of the operation device 120 held or worn on the left hand, button and OTP operations, and the three-dimensional position and orientation and gestures by hand tracking of the left hand are extracted. As with the exclusive control of the right hand, when a first input operation and a second input operation that are in a competitive relationship are included, the input operation mapping unit 407 adopts either the first input operation or the second input operation according to the priority defined in the priority table.

[0077] According to the embodiment described above, when a plurality of input operations that are in a conflicting relationship are received, it is determined which of the plurality of input operations is to be adopted based on a predetermined priority. Therefore, it is possible to prevent a plurality of mutually contradictory commands (control instructions) from being generated, and to prevent control malfunctions and abnormalities from occurring. This allows input operations using an operating device and input operations using hand tracking to be used together without any problems. As a result, it is possible to provide an XR system that is highly convenient and easy to operate.

[0078] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0079] In addition, each functional unit in the above embodiment may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of the multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. In addition, each functional unit may or may not be realized by hardware such as ASIC, FPGA, DSP, etc. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0080] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0081] The disclosure of this specification includes the following configurations, methods, and programs. (Configuration 1) A receiving means capable of receiving an input operation by an operation device and an input operation by hand tracking; a determination means for determining, when an input operation by the operation device and an input operation by the hand tracking are simultaneously received, which of the input operation by the operation device and the input operation by the hand tracking is to be adopted as an input operation to be used for control of an information processing system, based on a predetermined priority; A control device having the above configuration. (Configuration 2) The input operation adopted by the determination means is further converted into a command based on mapping information that defines a correspondence between an input operation that can be accepted by the acceptance means and a command that is a control instruction for the information processing system. 2. The control device according to configuration 1. (Configuration 3) When a first input operation by the operating device and a second input operation by the hand tracking are simultaneously received, and when the first input operation and the second input operation are associated with the same command in the mapping information, the determination means determines which of the first input operation and the second input operation to adopt based on the priority. The control device according to configuration 2. (Configuration 4) The determining means is A process of determining an input operation of the right hand to be adopted from an input operation of the right hand using an operation device held or worn on the right hand and an input operation of the right hand using hand tracking of the right hand; and determining which left-hand input operation to adopt from a left-hand input operation using an operation device held or worn on the left hand and a left-hand input operation using hand tracking of the left hand. The control device according to any one of configurations 1 to 3. (Configuration 5) the input operation by the operation device includes an operation of changing an orientation of the operation device, the hand tracking input operation includes a motion of changing the orientation of the hand; The control of the information processing system includes control of changing a direction of a CG image displayed on a display device included in the information processing system in accordance with a direction of the operation device or a direction of the hand. The control device according to any one of configurations 1 to 4. (Configuration 6) The CG image shows the direction in which the user's hand is pointing with a ray of light. The control device according to configuration 5. (Configuration 7) The operating device is a wearable type device. The control device according to any one of configurations 1 to 6. (Configuration 8) The operating device is a ring-shaped device. The control device according to any one of configurations 1 to 7. (Configuration 9) The operating device is a wristwatch type device. The control device according to any one of configurations 1 to 7. (Configuration 10) The priority is set so that the input operation by the operation device is given priority over the input operation by the hand tracking. The control device according to any one of configurations 1 to 9. (Configuration 11) The priority is user configurable The control device according to any one of configurations 1 to 9. (Method 12) A method for controlling an information processing system, comprising: receiving an input operation by an operation device; Receiving an input operation by hand tracking; a step of determining, when an input operation by the operation device and an input operation by the hand tracking are received simultaneously, which of the input operation by the operation device and the input operation by the hand tracking is to be adopted as an input operation to be used for controlling the information processing system, based on a predetermined priority; A control method comprising: (Program 13) A program for causing a processor to execute each step of the control method described in Method 12. [Explanation of symbols]

[0082] 1: Information processing system 110: Image processing device (control device) 120:Operation device

Claims

1. A receiving means capable of receiving an input operation by an operation device which is a wearable type device worn on a first hand of a user and an input operation by hand tracking of the first hand wearing the operation device; a determination means for determining, when an input operation by the operation device worn on the first hand and an input operation by the hand tracking of the first hand are simultaneously received, which of the input operation by the operation device worn on the first hand and the input operation by the hand tracking of the first hand is to be adopted as an input operation to be used for controlling an information processing system, based on a predetermined priority; providing means for providing an application with the input operation used to control the information processing system determined by the determining means; A control device having the above configuration.

2. a conversion unit that converts the input operation adopted by the determination unit into a command based on mapping information that defines a correspondence between an input operation that can be accepted by the acceptance unit and a command that is a control instruction for the information processing system, The mapping information assigns different commands to the same movement when it is performed with the right hand, the left hand, and both hands. The control device according to claim 1 .

3. When a first input operation by the operation device and a second input operation by the hand tracking are simultaneously received, and when the first input operation and the second input operation are associated with the same command in the mapping information, the determination means determines which of the first input operation and the second input operation to adopt based on the priority. The control device according to claim 2.

4. A receiving means capable of receiving an input operation by an operating device that is a wearable type device and an input operation by hand tracking; The input operation by the operation device and the input operation by the hand tracking are performed simultaneously. a determination means for determining, when the input operation is received, which of the input operation by the operation device and the input operation by the hand tracking is to be adopted as an input operation to be used for controlling an information processing system, based on a predetermined priority; having The determining means is A process of determining an input operation of the right hand to be adopted from an input operation of the right hand using an operation device held or worn on the right hand and an input operation of the right hand using hand tracking of the right hand; and determining which left-hand input operation to adopt from a left-hand input operation using an operation device held or worn on the left hand and a left-hand input operation using hand tracking of the left hand. Control device.

5. the input operation by the operation device includes an operation of changing an orientation of the operation device, the hand tracking input operation includes a motion of changing the orientation of the hand; The control of the information processing system includes control of changing the orientation of CG displayed on a display device included in the information processing system in accordance with the orientation of the operation device or the orientation of the hand. The control device according to claim 1 .

6. The CG is a CG that represents the direction in which the user's hand is pointing with a ray of light. The control device according to claim 5.

7. The operating device is a ring-shaped device. The control device according to claim 1 .

8. The operating device is a wristwatch type device. The control device according to claim 1 .

9. The priority is set so that the input operation by the operation device is given priority over the input operation by the hand tracking. The control device according to claim 1 .

10. The priority is user configurable The control device according to claim 1 .

11. The present invention further comprises a determination means for determining whether or not an input operation by the operating device worn on the first hand and an input operation by the hand tracking of the first hand are in a relationship that can be interpreted as mutually contradictory commands when an input operation by the operating device worn on the first hand and an input operation by the hand tracking of the first hand are simultaneously received, When the determination means determines that the input operation by the operating device worn on the first hand and the input operation by the hand tracking of the first hand are in a relationship that can be interpreted as mutually contradictory commands, the decision means determines, based on a preset priority order, which of the input operation by the operating device worn on the first hand and the input operation by the hand tracking of the first hand will be used as the input operation to be used for controlling the information processing system. The control device according to claim 1 .

12. The determination means determines whether or not an input operation by the operating device attached to the first hand and an input operation by the hand tracking of the first hand are in a relationship in which they can be detected simultaneously and can be interpreted as mutually contradictory commands. The control device according to claim 11.

13. The present invention further comprises an extraction means for extracting an input operation performed by the first hand from among the plurality of input operations accepted by the acceptance means, When the extraction means extracts the input operation by the operation device worn on the first hand and the input operation by the hand tracking of the first hand, the determination means determines whether or not the input operation by the operation device worn on the first hand and the input operation by the hand tracking of the first hand are in a relationship that can be interpreted as mutually contradictory commands. The control device according to claim 11.

14. A method for controlling an information processing system, comprising: receiving an input operation by an operation device which is a wearable type device worn on a first hand of a user; receiving an input operation by hand tracking of the first hand wearing the operation device; a step of determining, when an input operation by the operation device worn on the first hand and an input operation by the hand tracking of the first hand are simultaneously received, which of the input operation by the operation device worn on the first hand and the input operation by the hand tracking of the first hand is to be adopted as an input operation to be used for controlling the information processing system, based on a predetermined priority; a providing step of providing an application with the input operation used to control the information processing system determined in the determining step; A control method comprising:

15. A method for controlling an information processing system, comprising: Receiving an input operation by an operation device which is a wearable type device; Receiving an input operation by hand tracking; a step of determining, when an input operation by the operation device and an input operation by the hand tracking are simultaneously received, which of the input operation by the operation device and the input operation by the hand tracking is to be adopted as an input operation to be used for control of an information processing system, based on a predetermined priority; Including, In the determining step, A process of determining an input operation of the right hand to be adopted from an input operation of the right hand using an operation device held or worn on the right hand and an input operation of the right hand using hand tracking of the right hand; and determining which left-hand input operation to adopt from a left-hand input operation using an operation device held or worn on the left hand and a left-hand input operation using hand tracking of the left hand. Control methods.

16. A program for causing a processor to execute each step of the control method according to claim 14 or 15.

Images