Information processing device, information processing method, and program

Abstract

To provide an information processing device that allows a user to manipulate a virtual object by moving a finger, arm or the like, and is less likely to cause manipulation or display not intended by a user.SOLUTION: An information processing device is provided, comprising image capturing means for acquiring a captured image by capturing an image of real space, tracking means configured to respond to a first case where a first body part of a user is present in the field of view of the image capturing means by tracking the first body part based on the captured image to detect the position and orientation of the first body part, and display control means for controlling the display means to display a first UI in the first case. The first UI is for manipulating a virtual object based on the position and orientation of the first body part detected by the tracking means. When the first body part moves outside the field of view of the image capturing means, the display control means controls display for manipulating the virtual object in accordance with a result of determination as to whether the user intends to manipulate the virtual object.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Mixed reality (MR) and augmented reality (AR) are well-known technologies that fuse real space with virtual space in real time. These technologies seamlessly fuse real space with 3D CG space created by computers. These technologies are not only used in the entertainment field such as games, but are also being used in industrial applications such as map navigation apps, design review, and 3D confirmation before prototyping.

[0003] Applications using AR and MR technologies can be divided into 3DoF applications, which only allow you to look around, and 6DoF applications, which reflect not only your posture but also your movement in the CG space.Since humans mainly use their hands to operate the device, 6DoF applications often reflect the position of your hands in the CG space, enabling operation.

[0004] Patent Document 1 describes changing the color of the gaze pointer display in an MR space. In a configuration where hand tracking is performed using an HMD and an HMD camera, a technique is described in which, if a finger is outside the field of view of the HMD camera, the display of the gaze cursor is changed to notify the user. Patent Document 2 describes a technique in which, in a configuration where a device worn on the arm and an HMD with a camera are worn, the position of the arm continues to be tracked even if the arm is outside the field of view of the camera.

[0005] According to these techniques, even if a finger, arm, or the like goes out of the field of view of the camera, for example, the user can continue to operate a displayed virtual object (CG). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 10,409,443 [Patent Document 2] US Patent Application Publication No. 2023 / 0041519 Summary of the Invention [Problem to be solved by the invention]

[0007] However, there are various reasons why a hand, finger, arm, or the like may go outside the field of view of the camera. For this reason, when a hand, finger, arm, or the like goes outside the field of view of the camera, the user may want to stop manipulating a virtual object using the hand, finger, arm, or the like. Therefore, with conventional techniques, operations or displays that the user did not intend may be performed.

[0008] The present invention aims to provide a technique for reducing the possibility of an operation or display not intended by a user in an information processing device that allows a virtual object to be operated by movements of the fingers, arms, or the like. [Means for solving the problem]

[0009] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. An information processing device that enables a user to operate a virtual object displayed on a display means by using a UI (User Interface) displayed on the display means, an imaging means for capturing an image by imaging a real space; a tracking means for detecting a position and orientation of a first part of the user by tracking the first part based on the captured image in a first case where the first part of the user is located within a field of view of the imaging means; a display control means for controlling the display means to display a first UI in the first case; and the first UI is a UI for manipulating the virtual object based on the position and orientation of the first part detected by the tracking means, and in a second case where the first part has gone out of the field of view of the imaging means, the display control means controls a display for operating the virtual object on the display means in accordance with a determination result of whether or not the user has an operation intention, which is an intention to operate the virtual object. The information processing device is characterized by the above.

[0010] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. 1. An information processing method for enabling a user to operate a virtual object displayed on a display means by using a UI (User Interface) displayed on the display means, comprising: an imaging step of capturing an image of a real space by an imaging means; a tracking step of detecting a position and orientation of a first part of the user by tracking the first part based on the captured image in a first case where the first part of the user is located within a field of view of the imaging means; a display control step of controlling the display means to display a first UI in the first case; and the first UI is a UI for operating the virtual object based on the position and orientation of the first part detected in the tracking step, In the display control step, in a second case where the first part has gone out of the field of view of the imaging means, a display for manipulating the virtual object on the display means is controlled in accordance with a determination result of whether or not the user has an operation intention that is an intention to operate the virtual object. The information processing method is characterized by the above. [Effects of the Invention]

[0011] According to the present invention, in an information processing device that allows a virtual object to be operated by the movement of a finger or an arm, the possibility of an operation or display not intended by a user can be reduced. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a hardware configuration diagram of an information processing device according to a first embodiment. [Figure 2] 1 is an external view of an information processing device according to a first embodiment. [Figure 3] 1 is a logical configuration diagram of an information processing device according to a first embodiment. [Figure 4] 10A and 10B are diagrams for explaining the reason why fingers appear outside the field of view according to the first embodiment. [Figure 5] 1 is a flowchart of the overall process according to the first embodiment. [Figure 6] 10 is a flowchart of a process when a finger goes out of the field of view according to the first embodiment. [Figure 7] 3A and 3B are diagrams illustrating a display on a display unit according to the first embodiment. [Figure 8] 10A and 10B are diagrams illustrating a display on a display unit according to a second embodiment. [Figure 9] 10A and 10B are diagrams illustrating a display on a display unit according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0014] <Embodiment 1> FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device 100 according to the first embodiment. The information processing device 100 includes a first device 101 and a second device 111.

[0015] (Hardware configuration of first device 101) The first device 101 is a gripping device that can be worn on a user's finger. The first device 101 includes a measurement unit 102, a control unit 103, a storage unit 104, and a bus 105. The first device 101 and the second device 111 are connected to each other via a network 106.

[0016] The measurement unit 102 includes a device capable of measuring the attitude of the first device 101. The device that measures the attitude is, for example, an inertial measurement unit (IMU). Generally, an inertial measurement unit can measure data including three-axis acceleration and angular velocity at a rate (update frequency per second) higher than the frame rate of image acquisition (imaging) by a camera.

[0017] Control unit 103 corresponds to a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). Control unit 103 executes programs stored in storage unit 104 or other storage media to operate various functions of first device 101, which will be described later.

[0018] The storage unit 104 has a storage medium such as a semiconductor memory, a hard disk, or a solid state drive. The storage unit 104 stores programs and data for processing by the first device 101. Note that some of the programs and data may be obtainable from an external data source (for example, a data server, network storage, or external memory) without being stored in the storage unit 104.

[0019] The bus 105 interconnects the measurement unit 102, the control unit 103, and the storage unit 104. The bus 105 does not necessarily need to connect these components in close physical proximity. The two components may be connected not only by wiring inside the device, but also by a transmission path such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. Information may also be exchanged directly between the two components. Alternatively, one of the two components may temporarily store (buffer) information in a file or memory and pass it to the other component at a desired timing by the control unit 103.

[0020] The network 106 is a network that interconnects the first device 101 and the second device 111. The first device 101 and the second device 111 may communicate with each other via a wired LAN or other wired connection. The first device 101 and the second device 111 may also communicate with each other via wireless communication such as WiFi (registered trademark) or Bluetooth (registered trademark). The first device 101 and the second device 111 mutually identify the device IDs of the other devices and communicate while recognizing the individuality of the other devices.

[0021] (Hardware configuration of the second device) The second device 111 is a head-mounted display (HMD) and includes an imaging unit 112, a display unit 113, an input unit 114, a storage unit 115, a control unit 116, and a bus 117.

[0022] The imaging unit 112 is an imaging device that uses images captured continuously in real space as digital information. The imaging unit 112 may be a monocular camera, a stereo camera, a multi-lens camera, or a camera equipped with a depth sensor. The imaging unit 112 may be an RGB camera or a grayscale camera. The imaging unit 112 may be configured inside the second device 111, or may be configured inside a separate entity that is physically separated from the second device 111.

[0023] The display unit 113 is a display. If the display unit 113 is a device that realizes MR, it displays an image acquired by the imaging unit 112 and draws a CG (Computer Graphics; virtual object) read from the storage unit 115 at a desired position. In addition, the display unit 113 can also display various information controlled by the control unit 116. The display unit 113 can also display information that has been processed from the information stored in the storage unit 115.

[0024] The input unit 114 is a device (operation member) with which the user gives instructions to the control unit 116. Specifically, the input unit 114 includes at least one of a keyboard, a keypad, a button, a switch, a mouse, a touchpad, and a gesture input device.

[0025] The storage unit 115 is a storage medium such as a semiconductor memory, a hard disk, or a solid state drive. The storage unit 115 stores programs and data for processing performed by the second device 111. Note that some of the programs and data may not be stored in the storage unit 115 and may be obtainable from an external data source (for example, a data server, network storage, or external memory).

[0026] The control unit 116 corresponds to a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The control unit 116 executes programs stored in the storage unit 115 or other storage media to realize various functions of the second device 111. Furthermore, the control unit 116 does not need to be a single physical component, and may be composed of multiple control units that cooperate with each other. For example, the control unit 116 may have a control unit in an HMD (Head Mounted Display) and a control unit of a personal computer, a mobile terminal, or the like.

[0027] The bus 117 interconnects the imaging unit 112, the display unit 113, the input unit 114, the storage unit 115, and the control unit 116. The bus 117 has the same characteristics as the bus 105. Therefore, for example, the bus 117 does not necessarily need to connect multiple components in close physical proximity.

[0028] Referring to FIG. 2, an example of the appearance of the information processing device 100 is shown.

[0029] 2, the second device 111 is worn on the head of a user 121. The second device 111 is, for example, a head-mounted display (HMD). The second device 111 is equipped with an imaging device 123 corresponding to the imaging unit 112. The imaging device 123 may be either a monocular camera or a stereo camera. Furthermore, the imaging device 123 may be a visible light (RGB) camera, or any of various other cameras such as a depth camera, an infrared camera, or an event camera.

[0030] The imaging device 123 can capture an image within a field of view 124. If the imaging device 123 has multiple cameras, the field of view 124 is typically the sum of the fields of view of the multiple cameras, or the field of view 124 may be the effective field of view of only a portion of the fields of view of the multiple cameras.

[0031] Furthermore, fingers 125 are the fingers of user 121. First device 101 is a gripping device attached to fingers 125. Graphic 127 indicates that fingers 125 and their joints are recognized by hand tracking technology.

[0032] (Functional configuration of information processing device) 3 shows an example of the configuration of logical functions realized by the information processing device 100 shown in FIG. 1. The information processing device 100 has a first device 101 and a second device 111. The first device 101 has a posture measurement unit 131. The second device 111 has an imaging unit 141, a hand tracking unit 142, a receiving unit 143, a tracking result integration unit 144, a motion source determination unit 145, It has a display control unit 146 .

[0033] The posture measurement unit 131 measures the posture of the first device 101. The posture measurement unit 131 has, for example, an inertial measurement unit (IMU) as a sensor for measuring the posture. It can be said that the posture measurement unit 131 continuously measures the posture of the fingers wearing the first device 101. The posture measurement unit 131 transmits information on the posture of the first device 101 (posture information) to the second device 111 (receiving unit 143).

[0034] The imaging unit 141 has one or more cameras capable of capturing images of real space. The camera of the imaging unit 141 is, for example, a general RGB camera or grayscale camera. However, the imaging unit 141 may be any device capable of acquiring digital images. The imaging unit 141 may be a depth camera, an infrared camera, or an event camera. The imaging unit 141 outputs an image (captured image) captured of real space to the hand and finger tracking unit 142. The imaging unit 141 may store the captured image in on-memory on a computer so that the hand and finger tracking unit 142 can refer to it, for example. The imaging unit 141 may write the captured image to file storage once and provide the captured image to the hand and finger tracking unit 142 while playing it back.

[0035] When a hand or finger is detected from the captured image, the hand tracking unit 142 performs processing including image processing on the captured image to track the hand or finger (performs hand tracking). An overview of hand tracking technology is known, so details will be omitted. The hand tracking unit 142 calculates the position and posture (position and posture) of one or more detected hands, and outputs the hand position and posture information to the tracking result integration unit 144.

[0036] The receiving unit 143 acquires the posture information of the first device 101 from the posture measuring unit 131 of the first device 101. For example, communication between the first device 101 and the second device 111 may be performed using Bluetooth. Alternatively, other communication methods may be used for communication between the first device 101 and the second device 111.

[0037] The tracking result integration unit 144 is an estimation unit that estimates the position and orientation (position and orientation) of the user's fingers. The tracking result integration unit 144 estimates the position and orientation of the user's fingers based on the orientation information of the first device 101 and the position and orientation information calculated by the finger tracking unit 142. For example, the orientation can be estimated by a method of simply averaging two orientations or a method of weighting two orientations. The estimated position and orientation information is output to the display unit 113, and the display unit 113 continuously displays a CG (virtual object) of the hand in that position and orientation.

[0038] The motion source determination unit 145 determines whether or not the fingers that are the target of tracking have gone out of the field of view of the imaging unit 141. Furthermore, if the fingers have gone out of the field of view, the motion source determination unit 145 determines whether or not the reason the fingers have gone out of the field of view is due to an "intention to operate (continue to operate) CG by moving the fingers" (hereinafter simply referred to as "operation intention"). The motion source determination unit 145 outputs to the display control unit 146 a determination result indicating whether or not the fingers have gone out of the field of view of the imaging unit 141 and whether or not the reason the fingers have gone out of the field of view is due to an operation intention.

[0039] Here, a typical cause of fingers going out of the field of view will be described with reference to Figure 4. Figure 4 shows a case where fingers 125 go out of the field of view 124 due to movement. Here, if user 121 is performing some kind of operation on the CG, there is a high possibility that fingers 125 have gone out of the field of view 124 due to an intention to operate. Furthermore, even if fingers 125 remain at a fixed spatial position, if fingers 125 go out of the field of view 124 due to user 121 turning their head, there is a high possibility that fingers 125 have gone out of the field of view 124 regardless of an intention to operate.

[0040] The display control unit 146 controls the display unit 113 based on the determination result obtained from the motion source determination unit 145. When a finger is outside the field of view of the imaging unit 141 and it is determined that this is due to the user's intention to operate, the display control unit 146 controls the display on the display unit 113 so that the user can continue operating the second device 111. Specifically, even if the finger is outside the field of view, the display control unit 146 performs tracking of the finger (detection of finger movement) by a method other than tracking of the finger by the finger tracking unit 142. Furthermore, the display control unit 146 changes the UI (User Interface) displayed on the display unit 113 to a UI for operating a CG (virtual object) in accordance with the finger movement tracked by the tracking method.

[0041] On the other hand, if it is determined that the movement of the fingers out of the field of view is not due to an intention to operate, the display control unit 146 notifies (displays) to the user that the CG cannot be operated with the fingers (that the CG cannot be operated). Furthermore, the display control unit 146 changes the UI for operation with the fingers to a display that is not obstructive. Specifically, the display control unit 146 erases the CG that has been displayed in conjunction with the position and posture of the fingers, makes the CG semi-transparent, or returns (resets) the position of the CG to the center of the screen.

[0042] (Overall processing) An example of the overall processing in the first embodiment will be described with reference to the flowchart in FIG.

[0043] In step S151, the imaging unit 112 captures an image of the real space to acquire the captured image. Here, the imaging unit 112 can acquire a captured image (digital image) of the user's fingers by continuously capturing images of the user's fingers in the real space. The imaging unit 112 is, for example, a pair of stereo RGB cameras. Furthermore, for example, the imaging unit 112 outputs the captured image of the fingers to the display unit 113, allowing the user wearing the HMD to observe the captured image (see-through image) of the fingers.

[0044] In step S152, when fingers are located within the field of view of the imaging unit 112 (when fingers are detected from the captured image), the finger tracking unit 142 tracks the fingers based on the captured image. In this way, the finger tracking unit 142 detects (estimates) the position and posture of the fingers. Tracking can be achieved using known technology. The field of view of the imaging unit 112 may include multiple hands, such as left and right hands. For this reason, when the finger tracking unit 142 is tracking multiple hands within the field of view of the imaging unit 112, it needs to be able to identify each hand individually. Identification of each of the multiple hands can be achieved using known technology.

[0045] In step S153, the receiving unit 143 acquires the orientation information of the first device 101 from the orientation measurement unit 131. In the first embodiment, the orientation information is continuously transmitted from the first device 101 to the second device 111 via Bluetooth. The frequency of transmitting the orientation information is determined based on the degree of real-timeness required.

[0046] In step S154, the tracking result integrating unit 144 estimates the position and posture of the fingers based on the posture information of the first device 101 and the posture positions of the fingers detected by the finger tracking unit 142. In the first embodiment, the tracking result integrating unit 144 trusts the posture information output by the posture measurement unit 131 with regard to the posture and uses it as the estimation result of the posture of the fingers. With regard to the position, the tracking result integrating unit 144 trusts the position calculation result by the finger tracking unit 142 and uses it as the estimation result of the position of the fingers. In addition, various methods may be used to estimate the position and posture of the fingers, such as integrating two pieces of information by weighting them according to certain conditions.

[0047] In order to estimate the position and orientation of the fingers, the first device 101 must be attached to the fingers being tracked. In the first embodiment, for example, the tracking result integration unit 144 integrates the angular velocity measured by the orientation measurement unit 131 of the first device 101 and the position and orientation of the fingers calculated based on the captured image. The tracking result integration unit 144 compares the angular velocity with the angular velocity of the first device 101. If the tracking result integration unit 144 determines, as a result of comparing the two angular velocities, that the first device 101 and the fingers are making the same movement, it can determine that the fingers shown in the captured image are wearing the first device 101. Whether or not the fingers are wearing the first device 101 can be determined using various methods, such as a method of adding a wearing sensor to the first device 101. Note that, if it is determined that the tracked fingers are not wearing the first device 101, the tracking result integration unit 144 may estimate the posture of the fingers detected by the finger tracking unit 142 as the position and posture of the fingers as they are.

[0048] In step S155, the display control unit 146 updates the display on the display unit 113 based on the current position and posture of the fingers estimated by the tracking result integration unit 144. In the first embodiment, the display control unit 146 draws a CG image showing the fingers in the estimated posture at the estimated position of the fingers. Furthermore, if the user is operating the second device 11, the display control unit 146 changes the display on the display unit 113 to a UI that indicates that the user is operating the second device 11, as shown in FIG. 7A.

[0049] (What to do if your fingers go out of view) The processing to be performed when the fingers go out of the field of view of the image capturing unit 112 (when the fingers are not detected in the captured image) will be described with reference to the flowchart of Fig. 6. The processing of the flowchart of Fig. 6 is performed instead of the processing of step S153 and subsequent steps in the flowchart of Fig. 5 when the finger tracking unit 142 determines in step S152 that the fingers have gone out of the field of view. That is, when it is determined in step S152 that the fingers are located within the field of view, the processing of step S153 and subsequent steps in the flowchart of Fig. 5 is performed, and when it is determined that the fingers have gone out of the field of view of the image capturing unit 112, the processing of the flowchart of Fig. 6 is performed.

[0050] For example, in step S152, the finger tracking unit 142 determines that a finger has gone outside the field of view of the imaging unit 112 if "a certain finger was able to be tracked up to the previous frame" and "the finger cannot be tracked in the current frame or cannot be found even after searching."

[0051] FIG. 7A shows an example of a free-form CG bounding box (hereinafter referred to as a "BB cube"), which is a cube that surrounds the outside, displayed in the experiential space. FIG. 7A shows an example of a UI displayed in step S155. A visual field area 171 indicates an area where hand tracking is possible. The fingers 172 are wearing the first device 101. The fingers 172 may be displayed as a see-through image, or a CG representing a hand may be drawn at the position and orientation of the fingers 172. The user may be able to enlarge or reduce the size of a specific CG using the fingers 172. The operation point 174 is one or more highlighted points that exist on the sides or corners of the BB cube. By moving the fingers 172 while pinching the operation point 174, the user can enlarge or reduce the CG surrounded by the BB cube according to the amount of movement.

[0052] 7B shows a state in which the CG is enlarged by pinching the operation point 174, and the fingers 172 go out of the field of view of the imaging unit 112. To continue tracking the fingers using the camera (hand tracking), the fingers 172 need to be brought back into the field of view of the imaging unit 112.

[0053] In step S163, the motion source determination unit 145 calculates the speed of the fingers (speed of the finger motion) based on the posture information of the first device 101 (information acquired from the posture measurement unit 131).

[0054] In the first embodiment, the motion source determination unit 145 calculates a composite component ω1 of the angular velocity of the fingers by referring to Equation 1. Here, ωx, ωy, and ωz are the angular velocities of the fingers on the X, Y, and Z axes. The motion source determination unit 145 stores the calculated composite angular velocity ω1.

number

[0055] In step S164, the motion source determination unit 145 calculates the position and orientation of the second device 111 (HMD) based on the captured image acquired by the second device 111. The motion source determination unit 145 calculates the velocity of the user's head (the velocity of head movement) based on the calculated position and orientation of the second device 111.

[0056] In the first embodiment, the motion source determination unit 145 calculates a resultant angular velocity ω2, which is a resultant component of the angular velocity of the second device 111. Specifically, the calculation of the resultant component ω2 can use Equation 2, which is the same as Equation 1 above. In Equation 2, ω'x, ω'y, and ω'z can be calculated from the difference between the position and orientation of the second device 111 in the previous frame and the position and orientation of the second device 111 in the current frame. The motion source determination unit 145 stores the calculated resultant angular velocity ω2.

[0057] Note that the calculation of the position and orientation of the second device 111 can be realized by a known technique. For example, the calculation of the position and orientation of the second device 111 can be performed based on an image of a marker fixed in real space. The calculation of the position and orientation of the second device 111 may use a SLAM (Simultaneous Localization and Mapping) technique that tracks one or more features (features that become intersections) present in a captured image by image processing.

[0058] In step S165, the motion source determination unit 145 compares the finger velocity with the head velocity. In the first embodiment, the motion source determination unit 145 compares the resultant angular velocity ω1 of the fingers calculated in step S163 with the resultant angular velocity ω2 of the second device 111 calculated in step S164 to determine which value has a larger absolute value. If it is determined that the finger velocity is greater than the head velocity, the process proceeds to step S166. If it is determined that the head velocity is greater than the finger velocity, the process proceeds to step S167.

[0059] In step S166, the motion source determination unit 145 determines that the user has an intention to operate. Then, the display control unit 146 controls (changes settings of) the operation method of the CG displayed on the second device 111 and the display of the display unit 113 so that the user can continue the operation using his or her fingers. In the first embodiment, the display control unit 146 updates to "a UI that enables operation of a specific CG based only on the posture information of the first device 101."

[0060] FIG. 7C shows a state in which the display control unit 146 has changed the UI for enlarging and reducing the CG. In the UI shown in FIG. 7C, instead of enlarging the CG by pinching the operation points of the BB cubes surrounding the CG, the CG can be continuously enlarged and reduced using a meter UI 177. The meter UI 177 displays the current magnification of the CG numerically and indicates the magnitude of the current magnification of the CG by the direction of an arrow. The direction of the arrow corresponds to the orientation information of the first device 101. Furthermore, when the fingers return to the field of view of the imaging unit 112, the display control unit 146 may return to the operation method using the operation points of the BB cubes. Changing the CG operation method every time the user enters or leaves the boundary between the field of view and the outside of the field of view may be irritating to the user, so a condition for returning the operation method may be set separately.

[0061] In step S167, since the speed of the head is faster, the motion source determination unit 145 determines that the user has no intention of operation. Therefore, the display control unit 146 prevents the CG from being enlarged or reduced in accordance with the movement of the user's fingers (changes in posture, etc.). Then, the display control unit 146 erases the CG display (such as a CG model of the fingers) based on the position and orientation of the fingers, and notifies the user that tracking has been lost.

[0062] FIG. 7D illustrates a case where the user wearing the HMD turns their head and moves their fingers out of the field of view of the image capture unit 112. In this case, in the first embodiment, it is determined that the user does not intend to operate, and the display control unit 146 hides (makes invisible) the CG 179 of the fingers that was displayed at the position of the fingers. Even if the user is enlarging or reducing a specific CG, the operation mode is canceled and the specific CG is displayed at the magnification before the operation started. By hiding the fingers, the user can intuitively understand that the fingers are not being tracked and that the CG cannot be operated. Therefore, normally, if the user wishes to operate the CG using their fingers, they will perform an action to bring their fingers back into the field of view of the image capture unit 112. Furthermore, the display control unit 146 may display a display item (character or icon) on the display unit 113 indicating that the fingers are out of the field of view of the image capture unit 112.

[0063] According to the first embodiment, when the user's fingers go out of the field of view of the imaging unit 112, the information processing device 100 maintains a state in which the CG can be operated by finger movement if it is determined that the user has an intention to operate (an intention to continue operating the CG). If the information processing device 100 determines that the user does not have an intention to operate, it sets a state in which the CG cannot be operated by finger movement and does not display an obstructive UI. This makes it easier for the user to understand that if the user wishes to operate the CG using their fingers, they need to bring their fingers into the field of view of the imaging unit 112.

[0064] That is, when the fingers go out of the field of view of the imaging unit 112, the information processing device 100 switches whether or not the CG can be operated by finger movement and the display (UI) of the display unit 113, depending on whether or not the user intends to operate. Therefore, in the information processing device 100, which can operate the CG by finger movement when the fingers are located within the field of view of the imaging unit 112, it is possible to reduce the possibility of an operation or display that is not intended by the user being performed when the fingers go out of the field of view of the imaging unit 112.

[0065] <Embodiment 2> In the second embodiment, a specific example of specific display and display control different from that of the first embodiment will be described with reference to the flowchart of FIG.

[0066] An example of a UI (a UI displayed in step S155) according to the second embodiment will be described with reference to FIG. 8A. FIG. 8A shows an example in which a three-dimensional button UI is displayed in an experiential space. Buttons 182, such as [Yes] and [No], are displayed at specific positions in the experiential space. When hand tracking is being performed, a user can generate an event when pressing a button in the three-dimensional space by aligning their fingers with the button and pressing the buttons.

[0067] 6, a process to be performed when a finger goes out of the field of view of the image capturing unit 112 will be described. Note that, for steps not described below, the same process as in the first embodiment is executed.

[0068] In step S166, the display control unit 146 controls the operation method of the CG displayed on the second device 111 and the display of the display unit 113 so that the user can continue to operate the button 182 with his or her fingers. In the second embodiment, the display control unit 146 updates the UI to one that enables the CG to be operated based only on the posture information of the first device 101.

[0069] FIG. 8B shows display unit 113 after finger movement has caused the finger to move out of field of view 181.

[0070] The UI 184 is a UI whose display is controlled by the display control unit 146. In the UI 184, a pointer 185 is arranged in a circular shape. The orientation of the pointer 185 is based on the current orientation information of the first device 101. Depending on whether the pointer 185 is tilted left or right, it is possible to select either [Yes] or [No] on the button 182. This allows the user to operate the CG even in a situation where a movement corresponding to the original finger pressing cannot be detected.

[0071] In step S167, since it is determined that the speed of the head is faster than the speed of the fingers, the display control unit 146 erases the CG display (CG model of the fingers, etc.) based on the position and posture of the fingers, and notifies the user that the fingers cannot be tracked. The display control unit 146 disables the selection (operation) of the button 182 according to the position and posture of the fingers (movement of the fingers).

[0072] FIG. 8C shows a case where the user's neck, wearing the HMD, turns, causing the fingers to move out of the field of view of the imaging unit 112. In this case, it is determined that the user has no intention of operating, and the CG 187 of the fingers displayed at the position of the fingers is hidden. The display control unit 146 displays the button 182 in a released state, even if the button 182 is pressed. By hiding the fingers, the user intuitively understands that the tracking of the fingers is not possible, and if the user wants to operate using the fingers, the user takes an action to bring the fingers back into the field of view. The display control unit 146 may display on the display unit 113 a character or icon indicating that the fingers have moved out of the field of view.

[0073] According to the second embodiment, in a UI for pressing a button in space, even if the user's finger goes out of the field of view, the user can continue to operate the button if the user intends to operate it. If the user does not intend to operate it, an intrusive UI is not displayed, so the user can understand that if he or she wants to operate the button, he or she needs to put his or her finger into the field of view of the image capture unit 112.

[0074] <Embodiment 3> The third embodiment will be described using the UI shown in FIGS. 9A and 9B (the UI displayed in step S155). FIG. 9A shows an example in which a ray (line segment; light beam) 193 is drawn in the experiential space, extending from a CG 192 representing a user's fingers in the direction the fingers are pointing. The ray 193 can be used for menu selection, etc. The ray 193 is used, for example, for pointing to a menu 194 arranged in the experiential space. The ray extends based on the position and posture of the fingers, and the menu pointed to by the ray 193 is selected from among a group of menu objects. FIG. 9B shows a state in which the position of the fingers has gone out of the field of view of hand tracking due to finger movement. The fingers have gone out of the field of view 191, and the position and posture of the fingers, which are the root of the ray 193, cannot be tracked.

[0075] 6, a process to be performed when a finger goes out of the field of view of the image capturing unit 112 will be described. Note that, for steps not described below, the same process as in the first embodiment is executed.

[0076] In step S166, the display control unit 146 changes the method of tracking fingers and the display of the UI so that the user can continue to operate the menu object by moving the fingers. In the third embodiment, the display control unit 146 updates the UI so that the menu object can be operated based only on the user's gaze information.

[0077] As shown in FIG. 9C, when the finger moves out of the field of view 191, the ray 193 is hidden. , the display is switched to enable pointer operation by gaze pointing. The gaze pointer UI 198 indicates the position where the user is looking and selects one menu. If the second device 111 is configured with a gaze sensor, the position where the user is looking can be detected by the gaze sensor. If the second device 111 is not configured with a gaze sensor, the display of the gaze pointer UI 198 can be similarly realized by treating the position corresponding to the center of the field of view of the imaging unit 112 of the second device 111 as the position of the gaze pointer.

[0078] In step S167, since the speed of the head is faster than the speed of the hand, the display control unit 146 erases the CG display (CG model of the fingers, etc.) based on the position and posture of the fingers, and notifies the user that tracking has been lost. The display control unit 146 disables selection (operation) of the menu 194 according to the position and posture of the fingers (movement of the fingers).

[0079] FIG. 9D shows a case where the user wearing the second device 111 turns their head, causing their fingers to move out of the field of view 191 of the imaging unit 112. In this case, it is determined that the user has no intention of operation, and the CG 200 of the fingers that was displayed at the position of the fingers is hidden. Even if a pointing operation using a ray was in progress, the operation-in-progress mode is canceled. Therefore, the CG 200 indicating the ray and the position and orientation of the fingers is hidden. By hiding these, the user intuitively knows that their fingers are not being tracked, and if they want to perform an operation using their fingers, they will take an action to bring their fingers into the field of view.

[0080] According to the third embodiment, in a UI that performs a pointing operation using a ray, even if the user's finger goes out of the field of view, the user can continue the pointing operation if the user intends to perform the operation. If the user does not intend to perform the operation, an intrusive UI is not displayed, so the user can understand that if he or she wants to perform a pointing operation, he or she needs to bring his or her finger into the field of view of the image capture unit 112.

[0081] <Variation 1> In each embodiment, when the fingers go out of the field of view of the image capture unit 112, the display control unit 146 displays a UI that enables CG to be operated based only on the posture of the fingers without using the position of the fingers, or hides the CG based on the position and posture of the fingers. However, various display controls may be performed without being limited to these examples. For example, the display control unit 146 may place the CG of the fingers at the center of the screen of the display unit 113 and prompt the user to "put the fingers back into the field of view of the image capture unit 112 and perform an operation by resuming tracking of the position and posture of the fingers from the center of the screen." In addition, various display controls may be used to change the UI to one that corresponds to a currently available tracking method without bothering the user.

[0082] <Variation 2> In each embodiment, the motion source determination unit 145 determines whether or not the user has an intention to operate based on the motion of the fingers and the motion of the head (neck). However, the user's intention to operate may also be determined based on any motion related to the user's motion. For example, if there is a controller that is larger than can be held by fingers and has a function for measuring its position and orientation, the motion source determination unit 145 calculates the motion of the controller from the position and orientation. Then, the motion source determination unit 145 may determine whether or not the user has an intention to operate based on the motion of the controller. Furthermore, the motion source determination unit 145 may determine whether or not the user has an intention to operate based on the motion of a specific body part, such as the motion of the feet or the motion of the torso, instead of the motion of the fingers.

[0083] <Variation 3> In each embodiment, the motion source determination unit 145 determines the amount of angular velocity of the fingers and the angular velocity of the head (neck). The motion source determination unit 145 determines whether the user has an intention to operate based on a comparison of the amounts of movement of the fingers or head. However, the motion source determination unit 145 may also estimate whether the fingers or head are "moving roughly," "moving little by little and precisely," or "resting" by analyzing posture information of the fingers or head using, for example, an IMU or the like. The motion source determination unit 145 may then determine the intention to operate based on the estimated state. For example, if the motion source determination unit 145 estimates that the fingers or head are "moving roughly" or "resting," it determines that the user does not have an intention to operate. For example, if the motion source determination unit 145 estimates that the fingers or head are "moving little by little and precisely," it determines that the user has an intention to operate.

[0084] Furthermore, in an example in which a user enlarges or reduces CG as shown in the first embodiment, it may be determined whether or not the user has an intention to operate based on the information itself that the enlargement or reduction is in progress. Alternatively, whether or not the user has an intention to operate may be determined by combining various information such as measurement of the user's line of sight movement, tendency of body movement, status of operation mode, distance between the first device 101 and the second device 111, communication status, etc.

[0085] <Variation 4> In addition, in each embodiment, "outside the field of view" has been described as being outside the field of view of the imaging unit 112. The imaging unit 112 has been described as having a stereo camera configuration. However, the imaging unit 112 may have a multi-camera configuration or a mixed configuration of different types of cameras.

[0086] Furthermore, the "field of view" in each embodiment is basically the sum of the fields of view of each of the multiple cameras (imaging devices) that make up the imaging unit 112. In other words, the "field of view" is the range included in the field of view of at least one of the multiple cameras that make up the imaging unit 112. However, the "field of view" may also be the common range of the fields of view of the multiple cameras (effective field of view, which is an overlapping field of view area). Here, the definition of the "field of view" that is suitable for tracking fingers varies depending on the camera configuration. For this reason, definition information for the "field of view" may be stored in advance, and for a specific camera configuration, "going outside the effective field of view" may be considered as "going outside the field of view" described in each embodiment.

[0087] The "field of view" may be set in advance as an area specific to the second device 111, or may be set adaptively according to the imaging settings (shutter speed, etc.) of the second device 111.

[0088] <Variation 5> In each embodiment, a configuration has been described in which both the first device 101, which is a gripping device, and the second device 111, which is an HMD, are present. However, if hand tracking is possible using the second device 111, the first device 101 may not be present. In this case, the second device 111 cannot acquire posture information of fingers outside the hand tracking field of view. On the other hand, for example, the display control unit 146 can perform display control such as returning (resetting) the CG position of the fingers to the center of the screen of the display unit 113 without using the posture information of the fingers. Alternatively, sensors may be added to the second device 111 or installed in the surrounding environment, and these sensors may replace the measurement of the posture of the fingers of the first device 101.

[0089] Also, in the above, "If A is greater than or equal to B, proceed to step S1, and if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1, and if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1, and if A is less than or equal to B, proceed to step S2." "Proceed to step S2" may be read as "If A is greater than or equal to B, proceed to step S1, and if A is less than (lower than) B, proceed to step S2." Therefore, unless a contradiction arises, "greater than or equal to A" may be read as "greater than (higher; longer; more) than A," and "less than or equal to A" may be read as "smaller than (lower; shorter; fewer) than A." Furthermore, "greater than (higher; longer; more) than A" may be read as "greater than or equal to A," and "smaller than (lower; shorter; fewer) than A" may be read as "less than or equal to A."

[0090] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0091] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0092] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0093] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0094] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An information processing device that enables a user to operate a virtual object displayed on a display means by using a UI (User Interface) displayed on the display means, an imaging means for capturing an image by imaging a real space; a tracking means for detecting a position and orientation of a first part of the user by tracking the first part based on the captured image in a first case where the first part of the user is located within a field of view of the imaging means; a display control means for controlling the display means to display a first UI in the first case; and the first UI is a UI for manipulating the virtual object based on the position and orientation of the first part detected by the tracking means, and in a second case where the first part has gone out of the field of view of the imaging means, the display control means controls a display for operating the virtual object on the display means in accordance with a determination result of whether or not the user has an operation intention, which is an intention to operate the virtual object. 1. An information processing device comprising: (Configuration 2) the display control means controls the display means to display a second UI when it is determined that the user has the intention to perform the operation in the second case; the second UI is a UI for operating the virtual object regardless of a result of tracking of the first part by the tracking means; 2. The information processing device according to configuration 1, (Configuration 3) The display control means In the first case, controlling the display means to display an object corresponding to the first part; In the second case, when it is determined that the user does not have the intention to operate, the display means is controlled to make the virtual object inoperable and not to display the object corresponding to the first part. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The device further includes a determination means for determining whether the user has the intention to operate the device based on the user's movement. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The user's movement is at least one of a finger movement and a head movement. 5. The information processing device according to configuration 4. (Configuration 6) the imaging means is included in the head-mounted device, the determination means determines whether the user has the intention to perform the operation by comparing the speed of the finger movement with the speed of the head movement. 6. The information processing device according to configuration 5. (Configuration 7) The determination means If the speed of the finger movement is faster than the speed of the head movement, it is determined that the user has the intention to perform the operation; If the speed of the finger movement is slower than the speed of the head movement, it is determined that the user does not have the intention to perform the operation. 7. The information processing device according to configuration 6. (Configuration 8) the imaging means has a plurality of imaging devices, The field of view of the captured image is a common range of the fields of view of the multiple imaging devices. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) the imaging means has a plurality of imaging devices, The field of view of the captured image is a range included in at least one of the fields of view of the multiple imaging devices. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 10) In the first case, when a specific device is attached to the first part, the system further comprises an estimation means for estimating a position and orientation of the first part based on an orientation of the specific device detected by the specific device and the position and orientation detected by the tracking means, the display control means, in the first case, operates the virtual object based on the position and orientation estimated by the estimation means. 10. The information processing device according to any one of configurations 1 to 9. (method) 1. An information processing method for enabling a user to operate a virtual object displayed on a display means by using a UI (User Interface) displayed on the display means, comprising: an imaging step of capturing an image of a real space by an imaging means; a tracking step of detecting a position and orientation of a first part of the user by tracking the first part based on the captured image in a first case where the first part of the user is located within a field of view of the imaging means; a display control step of controlling the display means to display a first UI in the first case; and the first UI is a UI for operating the virtual object based on the position and orientation of the first part detected in the tracking step, In the display control step, in a second case where the first part has gone out of the field of view of the imaging means, a display for manipulating the virtual object on the display means is controlled in accordance with a determination result of whether or not the user has an operation intention that is an intention to operate the virtual object. 1. An information processing method comprising: (program) 11. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0095] 100: information processing device, 141: imaging unit, 142: Finger tracking unit, 146: Display control unit

Claims

1. An information processing device that enables a user to operate a virtual object displayed on a display means by using a UI (User Interface) displayed on the display means, comprising: an imaging means for capturing an image by imaging a real space; a tracking means for detecting a position and orientation of a first part of the user by tracking the first part based on the captured image in a first case where the first part of the user is located within a field of view of the imaging means; a display control means for controlling the display means to display a first UI in the first case; and the first UI is a UI for operating the virtual object based on the position and orientation of the first part detected by the tracking means, and in a second case where the first part has gone out of the field of view of the imaging means, the display control means controls a display for operating the virtual object on the display means in accordance with a determination result of whether or not the user has an operation intention, which is an intention to operate the virtual object.

1. An information processing device comprising:

2. the display control means controls the display means to display a second UI when it is determined that the user has the intention to perform the operation in the second case; the second UI is a UI for operating the virtual object without depending on a result of tracking of the first part by the tracking means; 2. The information processing apparatus according to claim 1, wherein:

3. The display control means In the first case, controlling the display means to display an object corresponding to the first part; In the second case, when it is determined that the user does not have the intention to operate, the display means is controlled to make the virtual object inoperable and not to display the object corresponding to the first part.

2. The information processing apparatus according to claim 1, wherein:

4. The device further includes a determination means for determining whether the user has the intention to operate the device based on the user's movement.

2. The information processing apparatus according to claim 1, wherein:

5. The user's movement is at least one of a finger movement and a head movement.

5. The information processing apparatus according to claim 4,

6. the imaging means is included in the head-mounted device, the determination means determines whether the user has the intention to perform the operation by comparing the speed of the finger movement with the speed of the head movement.

6. The information processing apparatus according to claim 5,

7. The determination means If the speed of the finger movement is faster than the speed of the head movement, it is determined that the user has the intention to perform the operation; If the speed of the finger movement is slower than the speed of the head movement, it is determined that the user does not have the intention to perform the operation.

7. The information processing apparatus according to claim 6,

8. the imaging means has a plurality of imaging devices, The field of view of the captured image is a common range of the fields of view of the multiple imaging devices.

2. The information processing apparatus according to claim 1, wherein:

9. the imaging means has a plurality of imaging devices, The field of view of the captured image is a range included in at least one of the fields of view of the multiple imaging devices.

2. The information processing apparatus according to claim 1, wherein:

10. In the first case, when a specific device is attached to the first part, the system further comprises an estimation means for estimating a position and orientation of the first part based on an orientation of the specific device detected by the specific device and the position and orientation detected by the tracking means, the display control means, in the first case, operates the virtual object based on the position and orientation estimated by the estimation means.

2. The information processing apparatus according to claim 1, wherein:

11. 1. An information processing method for enabling a user to operate a virtual object displayed on a display means by using a UI (User Interface) displayed on the display means, comprising: an imaging step of capturing an image of a real space by an imaging means; a tracking step of detecting a position and orientation of a first part of the user by tracking the first part based on the captured image in a first case where the first part of the user is located within a field of view of the imaging means; a display control step of controlling the display means to display a first UI in the first case; and the first UI is a UI for operating the virtual object based on the position and orientation of the first part detected in the tracking step, In the display control step, in a second case where the first part has gone out of the field of view of the imaging means, a display for manipulating the virtual object on the display means is controlled in accordance with a determination result of whether or not the user has an operation intention, which is an intention to operate the virtual object. An information processing method comprising:

12. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10.

Images