Point-and-select system and method

Abstract

To provide an input device that directly digitizes and transforms minute hand movements and gestures into machine commands, such as a pointer or cursor ray, without interfering with the normal use of one's hands.SOLUTION: According to an example aspect of the present invention, there is provided an input device and corresponding method which digitizes and transforms minute hand movements and gestures into a directive ray without interfering with the normal use of a user's hands. The device and method may, for example, at least participate in computing the directive ray based on received information from a plurality of sensors, the sensors being preferably of different types.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] Various exemplary embodiments of the present disclosure relate to a system, at least one wearable device, and a method, which can be used to control a device, and in particular, can be used to control a device in the field of computing and extended reality user interface applications. Extended reality (XR) includes augmented reality (AR), virtual reality (VR), and mixed reality (MR).

Background Art

[0002] Digital devices have traditionally been controlled by dedicated physical controllers. For example, a computer can be operated with a keyboard and a mouse, a game console can be operated with a handheld controller, and a smartphone can be operated with a touch screen. Usually, such physical controllers include sensors and / or buttons that receive input from a user based on the user's actions. Although such separate controllers are widespread, they slow down human-machine interaction by adding an extra technology layer between the user's hand and the computing device. In addition, such dedicated devices are typically only suitable for controlling a specific device. Also, such devices can be in the way of the user, for example, the user cannot use their hand for other purposes while using the controller.

Summary of the Invention

Problems to be Solved by the Invention

[0003] In the field of XR user interfaces, improvements to the above problems are needed. A suitable input device according to the present invention disclosed in this specification digitizes and converts fine hand movements and gestures directly into machine commands such as pointer rays and cursor rays without disturbing the normal use of a person's hand. Embodiments of the present disclosure can, for example, calculate an indication ray based on data received from a plurality of sensors, and it is preferable that there are various types of sensors.

Means for Solving the Problems

[0004] The present invention is defined by the features described in the independent claims. Some specific embodiments are defined in the dependent claims.

[0005] According to a first aspect of the present invention, a system is provided, the system including a wrist wearable device, the wrist wearable device including a mounting component and a controller including a processing core and at least one memory including computer program code, and a wrist wearable IMU configured to perform measurements of a user, the system being configured to perform steps of receiving data including gravity information from the wrist wearable IMU, receiving data (including, for example, the azimuth of the user's head) from at least one head sensor configured to perform measurements of the user, calculating a yaw component of an indication ray based on the gravity information and at least to some extent on the data received from the head sensor, calculating a pitch component of the indication ray based at least to some extent on the gravity information, and calculating an indication ray based on a combination of the calculated yaw component and the calculated pitch component.

[0006] According to a second aspect of the present invention, there is provided a method for calculating an indication ray, the method comprising receiving data including gravity information from at least one wrist-wearable IMU, receiving data from at least one head sensor configured to measure a user (specifically, the orientation of the user's head), calculating a yaw component of the indication ray based on the gravity information and at least to some extent on the data received from the head sensor, calculating a pitch component of the indication ray based at least to some extent on the received gravity information, and calculating an indication ray based on a combination of the calculated yaw component and the calculated pitch component.

[0007] According to a third aspect of the present invention, there is provided a non-transitory computer-readable medium storing a set of computer-readable instructions that, when executed on a processor, cause the second aspect to be implemented or cause an apparatus including the processor to be configured according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0008]

Figure 1A

Figure 1B

Figure 2A

Figure 2B

Figure 3A

Figure 3C

Figure 4A

Figure 4B

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0009] When interacting with an AR / VR / MR / XR application, for example, when using a device such as a smartwatch or an extended reality headset, the user needs to be able to perform various actions (also called user actions), such as selection, drag & drop, rotate & drop, slider use, and zoom. This embodiment improves the detection of user actions, which can lead to an improvement in responsiveness when implemented in a controller or system, or by a controller or system. The above actions may be performed with respect to one or more interactive elements (e.g., interactables). An interactable may include, for example, a UI slider, a UI switch, or a UI button in a user interface.

[0010] This specification describes a system, a wearable device, and a method related to at least one of calculating and / or displaying an indication light beam. The system, the wearable device, and the method may be used, for example, for at least one of measurement, detection, signal acquisition, analysis, and each task of a user interface. Such a system or at least one device is preferably suitable for being worn by a user. The user can control one or more external devices and / or separate devices using the above system or at least one device. The external device and / or separate device may be, for example, a personal computer (PC), a server, a mobile phone, a smartphone, a tablet device, a smartwatch, or any type of suitable electronic device. The control may be in the form of a user interface (UI) or a human-machine interface (HMI). The system may include at least one device. The at least one device may include one or more sensors. The system may be configured to generate user interface data based on data from the one or more sensors. At least a part of the user interface data may be used to enable the user to control the system or at least one secondary device. The secondary device may be, for example, at least one of a personal computer (PC), a server, a mobile phone, a smartphone, a tablet device, a smartwatch, or any type of suitable electronic device. The device to be controlled or controllable may implement at least one of an application, a game, and / or an operating system, and all of these may be controlled by a multimodal device.

[0011] The user may perform at least one user action. Typically, the user performs an action to affect a controllable device and / or an AR / VR / MR / XR application. The user action may include, for example, at least one of movement, gesture, interaction with an object, interaction with a body part of the user, and null action. An example of a user action is a "pinch" gesture, which is a gesture where the user touches the tip of the index finger and the tip of the thumb. Another example is a "thumbs up" gesture, which is a gesture where the user extends the thumb, curls the other fingers, and rotates the hand so that the thumb points upward. The present embodiment is configured to identify (specify) at least one user action based at least to some extent on sensor input. By reliably identifying user actions, action-based control (e.g., using the embodiments disclosed herein) becomes possible.

[0012] The user action may include at least one characteristic, referred to as a user action characteristic and / or an event. Such characteristics may be, for example, any of temporal characteristics, positional characteristics, spatial characteristics, physiological characteristics, and / or kinematic characteristics. The characteristics may include a display of a body part of the user (e.g., a finger of the user). The characteristics may include a display of the movement of the user (e.g., the trajectory of the user's hand). For example, the characteristic may be the movement of the middle finger. In another example, the characteristic may be a movement that draws a circle. In yet another example, the characteristic may be the time until an action and / or the time after an action. In at least some of the present embodiments, the characteristics of the user action may be specified (e.g., by a neural network) based at least to some extent on sensor data.

[0013] In at least some of the present embodiments, the system includes at least one device including a mounting component configured to be worn by a user. Accordingly, the at least one device is a wearable device. Such a component may be any of a strap, a band, a wristband, a bracelet, a glove, glasses, goggles, a helmet, a cap, a hat, a headband, or similar headgear. In the case of an embodiment mounted on the hand, a strap may be preferred. The mounting component may be attached to and / or formed by another device such as a smartwatch, or may form part of a larger device such as a glove or a headset. In some embodiments, the strap, band, and / or wristband has a width of 2 to 5 cm, preferably 3 to 4 cm, and more preferably 3 cm. The mounting component may be attached to and / or formed by another device, which may be, for example, a head-mounted display, a virtual reality (VR) headset, an extended reality (XR) headset, an augmented reality (AR) headset, or a mixed reality (MR) headset.

[0014] In the present embodiment, a signal is measured by at least one sensor. The sensor may include a processor and a memory, or may be connected to a processor and a memory, and the processor may be configured to perform the measurement. The measurement may be referred to as detection or sensing. The measurement includes, for example, detecting a change in at least one of a user, an environment, and the physical world. The measurement may further include, for example, attaching at least one timestamp to the sensed data and transmitting the sensed data (regardless of the presence or absence of the at least one timestamp).

[0015] Embodiments of the present disclosure may be configured to measure a user's position, surroundings, and movement using an inertial measurement unit (IMU). The IMU may be configured to output position information of at least a part of the system or a part of the user's body to which the IMU is mounted, attached, or worn. The IMU may be referred to as an inertial measurement unit sensor. The IMU includes a gyroscope. Further, the IMU may include at least one of a multi-axis accelerometer, a magnetometer, an altimeter, and a barometer. Preferably, the IMU includes a magnetometer because the magnetometer provides an absolute reference for the IMU. The barometer can be used as an altimeter and can give the IMU additional degrees of freedom.

[0016] A sensor (e.g., an inertial measurement unit (IMU) sensor) may be configured to output a sensor data stream. This output may be within the device, e.g., to a controller, a digital signal processor (DSP), a memory. Alternatively or additionally, this output may be to an external device. The sensor data stream may include one or more raw signals measured by the sensor. Additionally, the sensor data stream may include at least one of synchronization data, configuration data, and / or identification data. Such data may be used, for example, for a controller to compare or combine data from various sensors.

[0017] In embodiments, the IMU as well as other sensors may output data representing the orientation, position, and / or movement of the user's body to the system and / or the wearable device. The system and / or the wearable device may calculate at least one indication ray based on the above data. For example, the system may include a head-mounted device (e.g., an HMD) and / or a wrist wearable device (e.g., a smartwatch).

[0018] Devices such as wrist wearable devices or head devices may include controllers 103, 163 within housings 102, 162. The controllers 103, 163 include at least one processor, at least one memory, and a communication interface. The memory may contain instructions that, when executed by the processor, enable communication with other devices (e.g., computing devices) via the communication interface. The controllers 103, 163 may be configured to communicate with a head sensor and a wrist wearable IMU (e.g., head sensor 60 and wrist wearable IMU 104), at least for receiving data streams from the head sensor and the wrist wearable IMU. That is, the controllers 103, 163 may be configured to cause the controller to receive at least one sensor data stream from at least one sensor. The controller may be configured to perform preprocessing on at least one of those sensor data streams, and the preprocessing may include the preprocessing disclosed herein. The controller may be configured to process at least one sensor data stream received from the at least one sensor. The controllers 103, 163 may be configured to generate at least one user interface (UI) event and / or command based on the characteristics of the processed sensor data stream. Further, the controllers 103, 163 may include a model, and this model may include at least one neural network. In some embodiments, the head device controller 163 may be located within the head device or mounted on the head device. In some other embodiments, the wrist wearable device controller 103 may be located within the wrist wearable device or mounted on the wrist wearable device.

[0019] The devices within the present disclosure (e.g., the wrist wearable device 170) may be configured to communicate with at least one head device 160 that includes head sensors 60, 61, 164. It should be understood that the head device is a device that can be worn or mounted on a user's head. Such a head device may be, for example, glasses, goggles, a helmet, a cap, a hat, a headband, or similar headgear. The head device 160 may be, for example, at least one of a head-mounted device (HMD), a headset, a head-mounted display, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or an augmented reality (AR) headset. The head device may include a processor, a memory, and a communication interface, and the memory may include instructions that, when executed by the processor, enable communication with other devices (e.g., computing devices) via the communication interface and / or communication with other devices within the same system that includes the head device.

[0020] In the pointing ray calculation, the orientation of the pointing rays 21, 27, particularly sensor drift, is corrected and / or compensated for by head sensor data. Such head sensor data may be, for example, data from a head-mounted device (HMD), a headset, a virtual reality (VR) headset, an extended reality (XR) headset, an augmented reality (AR) headset, or a video camera, and from such data, data representing, for example, a user's gaze, eye tracking, field of view, head position, and / or head orientation may be obtained.

[0021] The head sensor may be configured to measure the position, orientation, and / or movement of the user's head. The head sensor 60 may be housed within the head-mounted device 160 (e.g., a head-mounted device), or may be disposed on its surface. The head sensor 60 may include an inertial measurement unit (IMU) 164, and / or the head sensor may include detection of the orientation of the user's head by a camera (e.g., gaze tracking). The IMU-based head sensor 164 may be configured to output orientation information of the user's head. This is because the IMU-based head sensor 164 can be attached or mounted to the head-mounted device and thus to the user's head. The IMU-based head sensor 164 may preferably include a gyroscope. Further, the IMU-based head sensor may include at least one of a multi-axis accelerometer, a magnetometer, an altimeter, and a barometer. The head sensor 60 (e.g., in the form of an IMU and / or a camera system) may be configured to output a sensor data stream to the controller 103. Devices such as the head-mounted device 160 and / or the wrist-wearable device 170 may include such a controller. The head sensor 60 data can output data with zero or nearly zero drift, and this data is used to correct the drift of the wrist-wearable IMU 104 data. The wrist-wearable IMU 104 may be connected to or configured to communicate with at least one controller. The head sensor may be connected to or configured to communicate with at least one controller.

[0022] The camera-based head sensor 60 may include detecting the orientation of the user's head by the camera. In some embodiments, the head sensor 60 data may include camera data, or data received from multiple cameras. The camera data may be used to calculate the position and / or orientation of the user's head. For example, multiple cameras disposed in an environment external to the user and / or the head device may be used to track the position of the head device. Alternatively or additionally, the head device may include multiple cameras configured to track and / or image the environment to obtain the position and / or orientation of the user's head relative to the head device and / or the user's surroundings.

[0023] FIG. 1A shows a system 100 including a wrist wearable device 170 and a head sensor 60. The system 100 is capable of calculating and / or displaying a homomorphic indication ray 21. The wrist wearable device 170 includes a wrist wearable IMU 104. In homomorphic ray casting, when the position of the user's hand, and thus the position of the wrist wearable IMU 104, shifts, a linearly proportional change occurs in the indication ray 21. An additional device such as a display unit may be used to display the indication ray 21.

[0024] The "selection plane" is a plane (e.g., a two-dimensional plane) in three-dimensional space where interactive or other user interface (UI) elements may exist, and is understood to be the plane where the indication rays 21, 27 end, i.e., the plane where the end points of the indication rays 21, 27 are located. Note that the above selection plane may be, for example, a curved surface. Alternatively, the above indication rays may extend to the tip of the interactive. That is, the indication rays 21, 27 and the interactive may intersect in the selection plane. The selection plane may include different positions and orientations depending on the position and / or orientation of the indication rays.

[0025] As shown in FIG. 1A, due to the movement of the hand and the resulting hand trajectory 34, a linear movement 35 of the end point of the homomorphic indication ray 21 occurs in the selection plane 30. That is, the movement of the end point of the homomorphic indication ray 21 in the selection plane 30 is directly proportional to the movement of the hand of the user wearing the wrist wearable device 170 (and the wrist wearable IMU therein). Therefore, the wrist wearable IMU's, and as a result, the user's hand trajectory 34 is proportionally reflected within the selection plane. The end point position of the homomorphic indication ray 21 on the selection plane 30 has shifted from the previous end point 31 of the indication ray to the end point at the homomorphic end point 32 in FIG. 1A, and is linearly dependent on the hand trajectory 34. Therefore, in homomorphic ray casting, the homomorphic cursor point 32 is the end point of the indication ray 21. Note that the desired cursor point represented as the heteromorphic cursor point 33 in FIG. 1A may be missed by the cursor due to the homomorphic behavior of the homomorphic indication ray 21.

[0026] FIG. 1B shows that the system 100 is capable of calculating and / or displaying a heteromorphic indication ray 27. That is, when the position of the user's hand and the wrist wearable device 170 worn on that hand shifts, a non-linearly proportional change (e.g., according to context logic) can occur in the heteromorphic indication ray 27. An additional device such as a display unit may be used to display the heteromorphic indication ray 27.

[0027] In FIG. 1B, the movement 36 of the end point of the heteromorphic indication ray 27 may be non-linearly proportional to the movement of the wrist wearable IMU 104, and thus to the hand trajectory 34 at least partially measured by the wrist wearable IMU 104. The end point position of the heteromorphic indication ray 27 on the selection plane 30 has shifted from the previous end point 31 of the indication ray to the end point at the heteromorphic cursor point 33 in FIG. 1B, and is non-linearly dependent on the hand trajectory 34.

[0028] In the example shown in FIG. 1B, a desired cursor point (e.g., an interactive such as a UI slider, UI switch, or UI button in a user interface, or other contextual information in a user interface) affects the movement speed and / or positioning of the end point of the deformed indication ray 27, and such a change in the movement of the indication ray 27 is depicted as a curve, a bent indication ray. If a regular indication ray were used as in the case of FIG. 1A, the regular cursor point 32 would be the end point of such a regular indication ray, and thus, the interactive that is at the desired cursor point (at the deformed cursor point 33 in FIG. 1B) would not be considered when calculating the indication ray and thus the cursor point. In embodiments where the deformed indication ray 27 is used, such an indication ray can be visualized as a bent curve, in other words, a non-straight trajectory, at least between the interactive element and the wrist wearable IMU 104. The interactive element may be an interactive. Such visualization can be achieved, for example, by a head-mounted display.

[0029] FIGS. 2A and 2B respectively show the system 100 and the systems 100, 101 according to at least some embodiments of the present invention. The wrist wearable device 170 includes a mounting component 105, which is shown in the form of a strap in FIG. 2A. The wrist wearable device 170 includes a housing 102 attached to the mounting component 105. The wrist wearable device 170 includes a controller 103 and an IMU 104 within the housing 102. The controller 103 includes a processor, a memory, and a communication interface. The user 20 may wear the wrist wearable device 170, for example, on the user 20's wrist or other part of the arm such that the wrist wearable IMU 104 represents the movement data of the user's hand.

[0030] In at least some embodiments, a wrist wearable inertial measurement unit (IMU) 104, or a sensor (such as a gyroscope) or multiple sensors housed in a single device or multiple devices, is installed and / or fixed to the user's arm. The fixing may be performed, for example, by a mounting component 105. It should be understood that the arm is the upper limb of the user's body, including the hand, forearm, and upper arm. The wrist connects the forearm and the hand. In the present disclosure, the hand, arm, and forearm may be used interchangeably.

[0031] In at least some embodiments, the wrist wearable device 170 may include a smartwatch. In at least some embodiments, the wrist wearable device 170 may include at least one of the following components, for example, a tactile device, a screen, a touch screen, a speaker, a heart rate sensor (such as an optical sensor), a Bluetooth communication device.

[0032] The wrist wearable device 170 may be configured to at least participate in providing data for calculating and / or displaying the indication light ray 21.

[0033] Referring to FIG. 2B, the systems 100, 101 include a wrist wearable device 170 and a head device 160. The systems 100, 101 may be suitable for detecting the user's movement and / or action and operating as a user interface device and / or an HMI device. The head device may include a head sensor 60 which may be a wrist wearable IMU.

[0034] Systems 100, 101, and / or wrist-wearable device 170 include a wrist-wearable inertial measurement unit (IMU) 104. The wrist-wearable IMU 104 is configured to transmit a sensor data stream (e.g., to controller 103). The sensor data stream includes, for example, at least one of multi-axis accelerometer data, gyroscope data, and / or magnetometer data. One or more of the components of the wrist-wearable device 170 may be combined together. For example, the wrist-wearable IMU 104 and the controller 103 may be placed on the same printed circuit board (PCB). Changes in the wrist-wearable IMU 104 data reflect, for example, movement and / or actions by the user, so such movement and / or actions are detectable by using the wrist-wearable IMU data. The wrist-wearable IMU 104 may be directly or indirectly connected to the processor and / or memory of the controller 103 to output sensor data to the controller.

[0035] In some embodiments, the system is further configured to reduce disturbances caused by measurement errors in the user's pointing posture by implementing a filter that improves the accuracy of the pointing rays 21, 27.

[0036] The hand trajectory is to be understood as the spatio-temporal path of a part of the arm. The hand trajectory may be measured and / or calculated with a suitable device, for example, measured and / or calculated with a wrist-wearable device including a wrist-wearable IMU. And the above device may be used for the calculation of the indication light rays 21, 27. The arm further includes an elbow. The elbow connects the upper arm and the forearm. In some embodiments, it is possible to assist in the formation and / or accuracy of the indication light rays 21, 27 by using an estimated value obtained by calculating the elbow position (i.e., the so-called elbow point 23). The indication light rays 21, 27 may be aligned in a straight line with the elbow point 23 and the position of the wrist-wearable IMU 104. For example, it is possible to assist in predicting the elbow height of the hand in interaction by using a wrist-wearable device including a wrist-wearable IMU with six degrees of freedom (6-DOF) and having a barometer, and using the predicted elbow height as the anchor point of the indication light rays 21, 27. Alternatively or additionally, the horizontal position of the elbow point may be estimated, for example, estimated by using at least a part of the head sensor data. In some embodiments, the elbow position of the arm may be estimated. Further, the indication light rays may be aligned in a straight line with the estimated elbow position and the position of the wrist-wearable IMU 104. Arranging them in such a straight line may include adjusting the indication light rays 21, 27, and this adjustment is made such that the starting point is roughly the user's elbow and the light rays proceed after intersecting the position of the IMU 104 on the user's wrist.

[0037] The indication light rays 21, 27 may be visualized in a point-and-select user interface that is in a head-mounted device 160 including a display unit and / or in a display unit separate from the head-mounted device (for example, an external screen, a computer monitor, or a display). The head-mounted device 160 may be configured to communicate with the system 100 and transmit and receive data streams, or alternatively, the system (for example, system 100 or system 101) may include the head-mounted device 160. In some embodiments, the indication light rays 21, 27 may be visualized on a display, for example, on a display of a head-mounted device or on an external display such as a computer monitor. In embodiments where the indication light rays 21, 27 are displayed to the user, the indication light rays 21, 27 may be visualized as curves rather than straight lines extending from the user's hand to a selection plane in three-dimensional space. In other words, if the position of the wrist-wearable IMU 104 and the cursor point on the selection plane are not aligned on the same straight line path generated from the estimated value of the hand trajectory, the indication light rays 21, 27 may be displayed as bent curves.

[0038] In some embodiments, the indication light rays 21, 27 are calculated immediately after receiving the wrist-wearable IMU 104 data stream and the head sensor 60 data stream in a wrist-wearable device including a controller (for example, the wrist-wearable device 170). In contrast, in another embodiment, the indication light rays 21, 27 are calculated by a head-mounted device 160 including a controller, and the head-mounted device is configured to receive data streams from the head sensor and the wrist-wearable IMU. The calculation of each component of the indication light rays 21, 27 may be performed in separate devices. For example, one component of the indication light ray may be calculated in the wrist-wearable device 170, and another component may be calculated in the head-mounted device 160 (for example, a head-mounted device (HMD)). Such each component may be, for example, the pitch component and the yaw component of the indication light ray.

[0039] Figure 3A shows an exemplary system 100 that includes a wrist wearable device 170. The device 170 includes a controller 103 and a wrist wearable IMU 104. Further, the memory may include instructions that, when executed by a processor, enable the wrist wearable device 170 to calculate an indication ray 21 based on a data stream 116 from a head sensor 60 and a data stream 114 from the wrist wearable IMU 104. The wrist wearable device 170 is configured to receive a head sensor data stream 116 from the head sensor 60 at the controller 103. The controller 103 is configured to receive a wrist wearable IMU data stream 114 from the IMU 104. The indication ray may be calculated based at least on the wrist wearable IMU data stream and the head sensor data stream. The controller 103 may be configured to calculate indication rays 21, 27. The calculated indication rays 21, 27 may be transmitted to the head device 160 as an indication ray data stream 114, and the head device 160 may include a display unit that can visualize, for example, the indication rays 21, 27. The head sensor 60 and the head device 160 may be included in a single housing and / or device.

[0040] Figure 3B shows an exemplary system 101 that includes a head device 160, where the head device includes a controller 163 that includes a processor, a memory, and a communication interface. The memory may include instructions that, when executed by the processor, enable the head device 160 and / or the controller 163 to calculate indication rays 21, 27 based on a data stream 116 from a head sensor 60 and a data stream 115 from the wrist wearable device 170. At least a portion of the data stream 115 from the wrist wearable device 170 may be obtained as a data stream 114 from the wrist wearable IMU 104. The head sensor 60 may be separate from the head device 160, or the head device 160 may include a head sensor (e.g., IMU 164, etc.).

[0041] The controller 163 may be configured to calculate the indication light rays 21, 27. At least a part of the information related to the calculated indication light rays 21, 27 may be provided to the user via the head-mounted display. Such provision may include displaying, visualizing, or presenting the indication light rays 21, 27 in an extended reality (XR) environment (e.g., augmented reality (AR), mixed reality (MR), or virtual reality (VR)). The controller 163 may be configured to output the indication light rays 21, 27 as a data stream 210.

[0042] FIG. 3C shows an exemplary system 399 that includes a wrist wearable device and a head device 160, the head device including a controller 163 that includes a processor, a memory, and a communication interface. At least a portion of the information associated with the calculated pointing rays 21, 27 may be provided to the user via a head-mounted display. The controller 163 may be configured to output the pointing rays 21, 27 as a data stream 210. In the case of system 399, the head sensor 60 may include or be a head sensor 61 capable of detecting the user's gaze. The head sensor 61 may include a gaze tracking function and, for example, may include a camera configured to capture eye movements. In one embodiment, gaze tracking may be extended to fixation tracking, in which case information related to the position of the user's fixation in a three-dimensional environment is obtained. In one embodiment, the user's gaze may be used to correct or assist in correcting the drift of the estimated hand orientation by the IMU and thus the estimated yaw component of the pointing rays 21, 27. In one embodiment, fixation tracking is used to obtain information related to context logic associated with the direction and / or orientation of the pointing rays 21, 27. In other words, fixation tracking provides information about where the user is looking and this can be used to estimate and / or correct the direction and / or orientation of the pointing rays 21, 27. Similarly, the orientation of the pointing rays 21, 27 may be corrected using gaze tracking information, for example, by orienting the pointing rays towards the point the user is looking at.

[0043] In one embodiment, the system (e.g., system 399) may further be configured to align the pointing rays 21, 27 with respect to an origin, the coordinates of the origin being determined by the orientation of the user's head or gaze (POV center), for example, the origin being equal to the center of the user's field of view. Such alignment may include adjustment of the orientation and / or position of the pointing rays 21, 27 (e.g., directing the pointing rays towards the origin). Further, the extent, i.e., the length, of the pointing rays 21, 27 may be adjusted based on the origin.

[0044] Systems included in the present disclosure, such as System 100, System 101, or System 399, and devices included therein (e.g., the head device 160 and / or the wrist wearable device 170), may be configured to change the operation mode, for example, among FIGS. 3A, 3B, and 3C, based on the received operation mode change command. In other words, the system may first pass the data stream 115 to the head device, and then, in the same or another application, the operation mode may be changed to the one where the light beam is calculated on the wrist wearable device. Such a configuration realizes the flexibility that the user can use this wrist wearable device with multiple different systems. The system may be configured to communicate with at least one computing device, and this communication may include transmitting and / or receiving a data stream. The computing device may be at least one of a computer, a server, and a mobile device (e.g., a smartphone or a tablet). The above device may be configured to be at least involved in (e.g., by calculation) providing a VR / XR / AR / MR experience to the user. The computing device may include a processor, a memory, and a communication interface, and the memory may include instructions, which, when executed by the processor, enable communication with other devices (e.g., the wrist wearable device 170 or the head device 160) via the communication interface. Alternatively, the head device may include the above computing device.

[0045] The calculated indication light beams 21 depend at least on the position and movement of the user's arm and changes thereof, which may be measured, for example, by the device 170. Euler angles may be used to represent the orientation of the indication light beams 21, 27, and the Euler angles include components of yaw (ψ), pitch (θ), and roll (φ), which are also called heading, elevation, and bank. In some embodiments, the roll component may be omitted or its contribution may be suppressed because the influence of the roll component on the general direction of the indication light beams 21, 27 may be minimal.

[0046] Using the wrist wearable IMU 104 data stream, the head sensor 60 data stream, and gravity information (e.g., the calculated gravity vector), the yaw and pitch components of the indication light rays 21, 27 can be calculated, whereby the indication light rays 21, 27 can be obtained. The indication light ray 21 may be represented by a vector v, which combines the yaw component ψ and the pitch component θ according to Equation 1 below.

[0047]

Number

[0048] According to Equation 1 above, the forward directions of the indication light rays 21, 27 can be defined as v = (0, 0, -1).

[0049] The sensor data stream from a wrist wearable inertial measurement unit (IMU) 104 located in contact with the user's hand is obtained by a system or device that calculates the indication light rays 21, 27, i.e., a system or device for ray casting. The wrist wearable IMU 104 data stream may include gyroscope data and / or gravity information.

[0050] Gravity information describes information regarding the orientation of a sensor or sensors with respect to gravity. For example, the gravity information may be obtained from an inertial measurement unit (IMU).

[0051] Gravity information may include a gravity vector. The term "gravity vector" means a three-dimensional vector representing the direction and magnitude of gravity with respect to the orientation of the sensor from which the data used in the calculation is obtained. Gravity vector

Number

[0052] The gravity information may include an accelerometer data stream and a gyroscope data stream received from the wrist wearable IMU 104, which may be the case where a Madgwick filter is applied to at least one of those data streams. Such a Madgwick filter provides orientation information of the wrist wearable IMU 104, which may be used in the calculation of the pitch component and yaw component of the indicating light rays 21, 27, and may also be used in the calculation of the above components using the gravity vector.

[0053] The obtained gravity vector may be normalized by the norm or length of the gravity vector, thereby obtaining a normalized gravity vector. The normalized gravity vector

Number

Number

[0054]

Number

[0055] The yaw component ψ of the indication light beam 21 may be calculated from the calculated normalized gravity vector and at least one sensor data stream (e.g., the yaw component of gyroscope data obtained from an IMU 104 wearable on the user's hand).

[0056] Furthermore, head sensor data is used to account for drift in the wrist-wearable IMU 104 sensors in the yaw component. The head sensor data may be received from a head-mounted device as described above, and may be received, for example, from a head-mounted device configured to obtain information regarding the head orientation and to display a VR, MR, AR, or XR application to the user. The information obtained may be pre-processed by a head-mounted device including head sensors to obtain the orientation of the head-mounted device and thus the head orientation of the user. Alternatively, the head sensor data stream may be transmitted, for example, to a controller included in a wrist-wearable device, or the head sensor data may be pre-processed and / or processed at the head-mounted device. The raw data streams from the head sensors 60 and the wrist-wearable IMU 104 may be processed, for example, by the head-mounted device 60. The head sensor 60 data is useful in providing information regarding the orientation of the user and / or the head orientation of that user, the direction of the indication light beams 21, and possible intentions regarding user actions associated with the indication light beams 21.

[0057] The head sensor 60 data can provide information regarding the user's line of sight, which can be useful in assessing the direction and orientation of the indication light beams 21, 27. The head sensor data may be output as a data stream including information regarding (or related to) the head orientation.

[0058] An exemplary process 903B for obtaining the yaw components ψ of the indication light beams 21, 27, implemented by a system (e.g., any of systems 100, 101, and 399), is as follows. A vertical direction vector derived from a normalized gravity vector

Number

Number

Number

Number

[0059]

Number

[0060] The yaw component of the angular velocity

Number

Number

Number

Number

Number

[0061]

Number

[0062] And the yaw components of the indication light beams 21 and 27

Number

[0063]

Number

[0064] Since there is sensor drift, a scalar c may be used to position around the indication light beam toward yaw obtained from a head sensor (for example, a head sensor included in a head-mounted device (HMD) or the like). This yaw is the headset yaw ψ h which is called. Then, as shown by Equation 6 below, by integrating the product of the scalar c and the yaw component of the angular velocity with respect to time t, a corrected yaw component of the gyroscope data is obtained.

[0065]

Number

[0066] The yaw angle of the gyroscope

Number

[0067]

Number

[0068] It is possible to obtain the value of the scalar c by using the difference Δψ between the yaw component of the gyroscope and the yaw component of the headset. An example of the value of the scalar c is shown in Equation 8 below.

[0069]

Equation

[0070] In other words, c may depend on the difference between the yaw component of the gyroscope

Equation

[0071] The advantage of using the head sensor and the yaw component of the head sensor is that it can correct and / or compensate for particularly noticeable sensor drift in the yaw component obtained from the wrist wearable IMU. In other words, since the drift of the yaw component of the head sensor is extremely small or non-existent, it is possible to correct the sensor drift using the head sensor data, for example, as shown in Equations 7 and 8 above. The head sensor data, and its calculated yaw component, are obtained in a comparable coordinate system, in other words, with reference to the yaw component from the wrist wearable IMU104.

[0072] The pitch component θ of the indicating light ray 21 may be calculated at least to some extent from the normalized gravity vector and at least one sensor data stream (for example, gyroscope data obtained from the IMU104). This at least one sensor data stream may be high-pass filtered, and the gravity vector data may be low-pass filtered.

[0073] An exemplary process 903A for obtaining the pitch components of the indication light rays 21, 27, which is implemented by a system (e.g., any one of systems 100, 101, and 399), is shown below.

[0074] To obtain the pitch components of the indication light rays 21, 27,

Number

Number

Number

Number

[0075]

Number

[0076] The pitch component of the angular velocity

Number

Number

Number

[0077]

Number

[0078] According to the following Equation 11, the pitch component of the angular velocity

Equation

Equation

[0079]

Equation

[0080] For example, due to sensor drift, the pitch angle may be corrected by the normalized gravity vector. Next, the angle of the normalized gravity vector

Equation

Equation

Equation

Equation

[0081]

Equation

[0082] The angle

Equation

Equation

[0083]

Number

[0084] Gravity angle

Number

Number

Number

Number

Number

Number

[0085]

Number

[0086] Furthermore, this filtering may be performed such that the yaw component of the gyroscope data or the yaw component of the gravity vector data is emphasized in the yaw components of the reference beams 21, 27. Such low-pass filtering and high-pass filtering may be performed, for example, using an infinite impulse response (IIR) filter according to the following equations 15 and 16, respectively.

[0087] low t =α*in t +(1-α)*low t-1 (Equation 15)

[0088] high t =in t -low t (Equation 16)

[0089] For example, the α value (i.e., the attenuation parameter) of the low-pass filter in Equation 15 above may be selected such that the yaw component of the gyroscope data or the yaw component of the gravity vector data is emphasized in the yaw component of the reference beam 21. Alternatively, the α value may be selected such that approximately half of the signal is derived from the yaw component of the gyroscope data and approximately half of the signal is derived from the yaw component of the gravity vector data. Alternatively or additionally, the low-pass filtering and high-pass filtering may be performed, for example, using Chebyshev, Butterworth, and / or various other filter designs.

[0090] The cut-off frequencies of the low-pass filter and the high-pass filter may be the same or approximately the same. That is, the frequency limit of the low-pass filter may be the same or approximately the same frequency as the frequency limit of the high-pass filter.

[0091] In some embodiments, the apparatus is configured to recalculate the yaw component of the indicating light rays 21, 27 based at least in part on the normalized gravity vector and at least in part on data obtained from at least one IMU 104 when the indicating light rays 21, 27 are substantially parallel or anti-parallel to the normalized gravity vector. That is, when the user points their hand with the wrist-wearable IMU 104 mounted downward or substantially downward, the yaw component of the indicating light rays 21, 27 is reset. When the hand is lifted from the downward position, the indicating light rays are directed or pointed in the direction of the user's head, and the information in that direction is obtained using the head sensor 60. In such a situation, the yaw component obtained from the wrist-wearable IMU may be omitted until reliable information about the orientation of the wrist-wearable IMU relative to the head sensor information is obtained. A similar reset of the indicating light rays may also occur when the user's hand is pointing straight up or substantially up.

[0092] In some embodiments, the sensitivity of gesture recognition is adjusted by information based on the interactive element and / or the trajectory of the indicating light rays 21, 27.

[0093] Figure 4A shows, in flowchart form, an exemplary process that can support at least some embodiments of the present invention. The flowchart of Figure 4A includes, by way of example and not limitation, phases 801, 802, 803A, 803B, 804, and 805. Further, the actions within phases 801, 802, 803A, 803B, 804, and 805 need not be performed simultaneously.

[0094] In the top phase 801 of FIG. 4A, data is acquired from at least one wrist wearable IMU 104 and at least one head sensor 60. The actions and phases of the process shown in FIG. 4A may be at least partially implemented by a device (e.g., the device disclosed herein (e.g., a controller (e.g., controller 103))). At least some actions (e.g., actions that provide context information) may be implemented by another device (e.g., a computing device or a controllable system, etc.). The data of at least one wrist wearable IMU 104 includes gyroscope data. Such wrist wearable IMU 104 data may be acquired, for example, as part of a data stream. The data of at least one head sensor 60 may be, for example, the head orientation.

[0095] In 802, data is acquired from at least one wrist wearable IMU 104. This data includes gyroscope data and gravity information. The gravity information may include information received from an IMU mounted on the hand (e.g., accelerometer data and / or gyroscope data). The gravity information may be normalized, for example, if the normalization includes, for example, scaling the information. And the gravity information is used in the calculation of the pitch component of the indicated light ray in phase 803A.

[0096] In some embodiments, the pitch component may be directly obtained from the gravity information, for example, obtained as in Equation 12 or Equation 13 regarding calculating the pitch from the normalized gravity vector and its respective components. That is, in such embodiments, for example, in phase 802, the gyroscope data may be omitted from the calculation of the pitch component in phase 803A.

[0097] Regarding the calculation of the yaw component, in phase 803B, head sensor data is used together with the yaw component (e.g., gyroscope data) of the wrist wearable IMU to calculate the yaw components of the indication light rays 21 and 27. The drift of the yaw component calculated from the wrist wearable IMU 104 is corrected using the head sensor data.

[0098] Regarding the calculation of the pitch component in 903A, for example, the calculation may be performed using a Madgwick filter adjusted to "excessively" capture the gyroscope. That is, this filter may be configured to prioritize the accuracy of the angular velocity over the accuracy of the orientation.

[0099] In phase 804, the calculated yaw component and the calculated pitch component are combined to obtain the indication light rays 21 and 27 (e.g., Equation 1 above).

[0100] In phase 805, a collision with the obtained indication light ray can be detected. Such a collision can provide context information for calculating a continuous or subsequent indication light ray and / or the position, location, and / or orientation of such a continuous or subsequent indication light ray. Further, such a collision may be, for example, an intersection of the indication light rays 21 and 27 with the surrounding environment (e.g., an interactive object (e.g., a UI button, a UI switch, and a UI slider, or, for example, a graspable object)) or a part thereof. The collision with the indication light rays 21 and 27 may be displayed and / or visualized in extended reality (XR) (e.g., virtual reality (VR), augmented reality (AR), or mixed reality (MR)).

[0101] Figure 4B shows an exemplary process that can support at least some embodiments of the present invention in the form of a flowchart. The flowchart of Figure 4B includes, by way of example and not limitation, phases 901, 902, 903A, 903B, 904, and 905. Further, the actions within phases 901, 902, 903A, 903B, 904, and 905 do not have to be performed simultaneously.

[0102] Using data obtained from at least one wrist wearable IMU 104, a gravity vector may be calculated. And that gravity vector is normalized in phase 902. And that normalized gravity vector is used in the calculation of the yaw component and pitch component of the indication light rays.

[0103] For the pitch components of the indication light rays 21, 27, as shown in phase 903A, low-pass filtering of the gravity vector data and high-pass filtering of the gyroscope data may be applied. The filtering may be performed, for example, using equations 15 and 16 described above. And the pitch components of the indication light rays 21, 27 may be calculated, which is done by combining the low-pass filtered gravity data and the high-pass filtered gyroscope data, for example, using equation 14 described above.

[0104] For the calculation of the yaw component, in phase 903B, the head sensor data is utilized together with the yaw component of the wrist wearable IMU to calculate the yaw components of the indication light rays 21, 27. The drift of the yaw component calculated from the wrist wearable IMU 104 is corrected using the head sensor data.

[0105] In phase 904, the calculated yaw component and the calculated pitch component are combined to obtain the indication light rays 21, 27 (for example, equation 1 described above).

[0106] In phase 905, a collision with the acquired indicator light beam can be detected. Such a collision can provide context information used to calculate a continuous or subsequent indicator light beam and / or the position, location, and / or orientation of such continuous or subsequent indicator light beam. Further, such a collision can be, for example, an intersection of the indicator light beams 21, 27 with the surrounding environment (e.g., an interactive (e.g., UI button, UI switch, and UI slider, or, e.g., a graspable object)) or a part thereof. The collision with the indicator light beams 21, 27 can be displayed and / or visualized in extended reality (XR) (e.g., virtual reality (VR), augmented reality (AR), or mixed reality (MR)).

[0107] The controller 103 may be configured to perform the above-described phases including phases 903A and 903B. Further, the controller 103 may be configured to supply the calculated indicator light beams 21, 27 to at least one of another device, a head-mounted device, and a computing device via a communication interface. The indicator light beam may be supplied as a data stream or data packet (including, for example, a bitstream), and such an indicator light beam data stream may be supplied to a head-mounted device (e.g., a head-mounted display device including a display).

[0108] FIG. 5 shows a system 300 that can support at least some embodiments of the present invention. This system is the same as system 100 unless otherwise specified.

[0109] FIG. 5 shows an exemplary schematic diagram of system 300. As can be seen from FIG. 5, system 300 includes a wrist wearable IMU 304, a head sensor 306, and a controller 303. Controller 303 is shown in a diagram represented by a dotted line. As shown in that figure, controller 303 may include at least preprocessing blocks 350 and 370, and model 330 (e.g., within the memory of the controller). Controller 303 may include an event interpreter and / or classifier 380. System 300 may be configured such that event interpretation and UI command generation are performed within system 300.

[0110] Sensors 304 and 306 are each configured to transmit a sensor data stream (e.g., sensor data streams 314 and 316 respectively). The sensor data streams may be received by controller 303. Sensor data streams 314 and 316 may be preprocessed, and based at least in part on sensor data streams 314 and 316, an indication ray may be calculated using model 330. The calculated indication ray is provided and / or applied to scene information 361. The calculated indication ray may be provided as data stream 310. Using context information 312 (e.g., scene-based context information and / or gesture-related information), indication ray 310 may be updated to correspond to likely user actions and / or user intentions (e.g., a location within a selection plane in 3D space, or an intended pointing location and / or selection location acquired in parallel). Such information may be obtained indirectly from event interpreter or classifier 380 via scene 361, or directly. Context logic and scene information 361 may be provided to model 330 from the scene within head-mounted device 360 as data stream 312.

[0111] Scene information 361 (e.g., the scene information disclosed with respect to system 300) may include, for example, interactables (e.g., user interface (UI) buttons, UI switches, or UI sliders). The scene information may include one or more previous locations, orientations, and / or positions of the pointing ray, or may be associated with the user's viewpoint and / or gaze. Within the controller and / or using controller 303, the received and pre-processed sensor data stream is directed to at least one model (e.g., model 330).

[0112] The controller (e.g., controller 103 or controller 303) may include a neural network. The neural network may be, for example, a feedforward neural network, a convolutional neural network, or a recurrent neural network, or a graph neural network. The neural network may include a classifier and / or regression. The neural network may apply a supervised learning algorithm. In supervised learning, samples of inputs for which the output is known are used, from which the network learns to generalize. Alternatively, an unsupervised learning algorithm or a reinforcement learning algorithm may be used to build the model. In some embodiments, the neural network has been trained such that certain signal characteristics correspond to certain user action characteristics (e.g., the trajectory of the user's hand).

[0113] The model may include at least one of an algorithm, a heuristic, and / or a mathematical model. For example, model 330 may include an algorithm that uses the wrist wearable IMU data stream 314 along with the head sensor data stream 316 to calculate orientation, and this algorithm is included in the calculation of the pointing ray. The wrist wearable IMU may output data from a triaxial accelerometer, a gyroscope, and a magnetometer. The pointing ray may further be provided to a classifier or event interpreter 380, and the pointing ray may be classified based on the scene information 361. Such a classifier 380 may be, for example, a ballistic classifier / correction phase classifier.

[0114] A model (e.g., model 330) may be implemented to improve the ray casting and the orientation of the pointing ray 21, as well as the direction of the pointing ray 21, by identifying or classifying the user's gestures. Based at least to some extent on the identification result and / or classification result, at least one confidence value, preferably a series of confidence values, may be output from the model, device, or system. At least one sensor data stream (e.g., an optical sensor data stream and / or an IMU data stream) may be directed into at least one model. Such a model may be a machine learning model, which may include, for example, at least one neural network (e.g., a feed-forward neural network, a convolutional neural network, a recurrent neural network, or a graph neural network). The model may include, additionally or alternatively, at least one of a supervised learning algorithm, an unsupervised learning algorithm, or a reinforcement learning algorithm. Feature extraction from the input data may be performed using a feature extraction module, which may include a neural network (e.g., a convolutional neural network). User actions (e.g., a pinch action to select an interactive element) may be identified, for example, using a long short-term memory (LSTM) recurrent neural network (RNN). The system may further include a feature extraction module configured to analyze the sensor data stream to identify additional user actions beyond the selection gesture. The additional user actions may be, for example, gestures used for pointing, such as gestures used for pointing with various fingers (e.g., pointing with the index finger or pointing with the middle finger). Additionally or alternatively, the additional user actions may include turning the wrist or arm.

[0115] The sensitivity of a gesture recognition model using a neural network may be further adjusted based on a user's previous selections and / or pointing history. Such user-specific adjustment of the gesture recognition model may be performed, for example, by adjusting some of the initial and / or later layers of the neural network. The history of a user's previous selections, including at least one selection made by the user, may be stored in a device such as a head-mounted device or a wrist-wearable device, or, for example, in a cloud-based storage system. Such a system may be configured to adjust the sensitivity of a machine learning model based on a user's previous history (e.g., selection history).

[0116] The model may include, for example, at least one convolutional neural network (CNN) that performs inferences on input data (e.g., signals received from sensors and / or preprocessed data). The convolution may be performed in the spatial or temporal dimension. Features (calculated from sensor data fed into the CNN) may be selected algorithmically or manually. The model may further include an RNN (recurrent neural network), which may be used in combination with a neural network that supports user action characteristic identification based on sensor data reflecting user behavior.

[0117] According to the present disclosure, training of the model may be performed, for example, using a labeled data set that includes multi-modal motion data from a plurality of subjects. This data set may be enhanced and augmented using synthetic data. The sequence of computational operations that make up the model may be derived by backpropagation, Markov decision processes, Monte Carlo methods, or other statistical methods, depending on the adopted model construction technique. Model construction may include dimensionality reduction techniques and clustering techniques.

[0118] Furthermore, context logic may be applied to improve the orientation and direction of the indication ray 21. Such context logic may be based on received context information (e.g., context information 312). The context information may include position information (e.g., coordinates) of user-interactable elements (e.g., sliders, buttons, etc.) implemented in (e.g., augmented reality (AR), mixed reality (MR), extended reality (XR), or virtual reality (VR)). Additionally or alternatively, the context information may include the proximity between the ray endpoint and at least one UI element calculated by the device providing the scene. In other words, the calculation of the indication ray 21 may be based at least in part on the context information. A confidence value may be used to remove ambiguity in context selection. Thus, the context information may be incorporated to generate an adaptation threshold for interactable elements or other context information. Furthermore, the sensitivity of gesture recognition may be adjusted based on the context information, and this adjustment may include adjusting a threshold of the confidence value. For example, if a UI element is near the ray endpoint, the threshold for detecting a pinch may be lowered (e.g.,) from 90% to 50%.

[0119] Furthermore, scene information (e.g., obtained from a head-mounted device including a display unit) may be incorporated into gesture recognition and / or pointing ray calculation. In other words, context information may be incorporated into information regarding the reliability of a specified gesture. For example, even if the reliability of a certain gesture is considered to be high enough to be recognized as a gesture, if there is no interactive element suitable for that gesture in the scene information where the pointing ray is located, such a gesture may not be regarded as a gesture. Context information including information related to the gesture may be incorporated into the pointing ray calculation. For example, when a gesture exists or is considered to be a specific gesture with a high probability, the orientation, position, and / or movement of the pointing ray may be adjusted based on the scene information. Such scene information may be, for example, gesture-specific interactive. The system may be configured to adjust the detection threshold of point interaction based on the end points of the pointing rays 21, 27, and this adjustment is based on the context logic received from the scene. Point interaction should be understood as the user pointing at something and may include, for example, the intersection or interaction between an element such as an interactive and the pointing rays 21, 27. Furthermore, the sensitivity of gesture identification may be adjusted based on the context information, and this adjustment may include adjusting the threshold value of the confidence value. Adjusting may include snapping the pointing ray to the interactive.

[0120] Context information (e.g., context information 312) may include interactive elements present in the scene (e.g., user interface (UI) sliders, UI switches, and UI buttons, etc. in the scene information of extended reality (XR)). To improve the orientation, direction, and movement of the pointing ray 21, information from both selection interactions and point interactions may be utilized. For example, when the pointing ray 21 directly points at a "pinchable" interactive element or points near that element, the pinch inference threshold may be adjusted. Similarly, changing the movement characteristics of the pointing ray 21 with respect to a certain interactive element may be done, for example, based on the proximity to that interactive element. Such movement characteristics may be, for example, the speed of the pointing ray 21 or the inertia or momentum experienced by the user of the pointing ray 21, which may be characterized or experienced by the user as the "stickiness" or "gravity" of the pointing ray 21. Further, the sensitivity of gesture recognition may be adjusted based on the trajectory and / or speed of the pointing ray, and this adjustment may include adjusting the threshold of the confidence value. For example, when the trajectory is irregular and indicates, for example, an uncontrolled movement of the user, the threshold for detecting a pinch may be increased, for example, from 90% to 97%.

[0121] At least one user action output from the event interpreter 380 is received by the scene within the device 360. The controller may be configured to generate a UI command based on the user action that is at least somewhat based on the identified user action. The controller 303 may be configured to send the user interface command to a device (e.g., device 360) and / or store the user interface command in the memory of the controller 303.

[0122] An example of the operation of system 300 including head sensor 60, wrist wearable inertial measurement unit (IMU) 104, and controller 103 is shown next. The user is using system 300 including wrist wearable IMU 104 worn on the hand, and the hand may be, for example, the wrist, forearm, palm, or back of the hand, and the head sensor can measure the head orientation. The user is looking at point H1, the user's head is oriented towards point H1, and the user's hand including wrist wearable IMU 104 points in the downward direction W1. The user performs an action, and this action includes turning the head from point H1 to direction H2, raising the hand to a horizontal position, extending the forearm, and pointing at point W2 with the index finger. The sensors of system 300 output data during the action as follows. · Head sensor 60 outputs data stream 316 including information regarding the orientation of the user's hand. Thus, the head sensor includes a measurement of the change in orientation from point H1 to point H2 that occurs while the orientation of the user's head changes. · Wrist wearable IMU 104 outputs a gravity vector and data stream 314 reflecting the orientation and movement of the user's wrist and / or hand (i.e., the trajectory of the user's hand). Thus, the wrist wearable IMU sensor measurements include a measurement of the change in orientation of the wrist wearable IMU sensor from point W1 to point W2.

[0123] Data streams 316 and 314 are acquired by system 300 (preferably, controller 303 included in the above system). The data stream 316 acquired from the head sensor 60 by the present system is used to calculate the yaw component indicating the position and orientation of the user's hand with respect to the wrist wearable IMU in physical world coordinates, and as a result, is used to correct the drift of the wrist wearable IMU sensor data stream. In parallel, the data stream 314 acquired from the wrist wearable IMU 104 by the present device is used to calculate the normalized gravity vector, yaw component, and pitch component of the wrist wearable IMU 104 with respect to physical world coordinates. The yaw component of the indication ray 21 is obtained from the data stream of the wrist wearable IMU 104 that has been corrected for drift by combining the yaw component of the head sensor 60 with the yaw component acquired from the wrist wearable IMU 104. The pitch component and the yaw component are combined, and the resulting indication ray may be used for interaction with the interactable, and the resulting indication ray 21 may be displayed to the user through a computer monitor or a computer display (for example, a head-mounted device worn by the user and including the display). By continuously and / or continuously updating the indication ray by recalculating the pitch component and the yaw component, real-time or near-real-time input from the user and real-time or near-real-time visualization of the indication ray can be obtained. When the user moves their hand, as the wrist wearable IMU 104 extends, the yaw component and the pitch component of the indication ray 21 may be recalculated from the data streams from the head sensor 60 and the wrist wearable IMU 104.

[0124] FIG. 6 shows a system 700 that can support at least some of the embodiments of the present invention. System 700 may include a wrist wearable device (e.g., device 170) and / or a head device 160.

[0125] System 700 includes a controller 702. The controller includes at least one processor and at least one memory including computer program code and optionally data. System 700 may further include a communication unit or communication interface. Such a unit may include, for example, a wireless and / or wired transceiver. System 700 may further include sensors (e.g., sensors 703, 704, 705) operatively connected to the controller. These sensors may include an IMU. System 700 may also include other elements not shown in FIG. 7.

[0126] System 700 is shown as including one processor, but may include two or more processors. In one embodiment, the memory is capable of storing instructions, for example, capable of storing at least any one of an operating system, various applications, models, neural networks, and / or preprocessing sequences. Further, the memory may include storage (which may be used to store at least some of the information and data used in the embodiments of the present disclosure).

[0127] Furthermore, the processor is capable of executing the stored instructions. In one embodiment, the processor may be implemented as a multi-core processor, or as a single-core processor, or as a combination of one or more multi-core processors and one or more single-core processors. For example, the processor may be implemented as one or more of various processing devices, and such processing devices include coprocessors, microprocessors, controllers, digital signal processors (DSPs), processing circuits with or without DSPs, or other various processing devices, and other various processing devices include, for example, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontroller units (MCUs), hardware accelerators, dedicated computer chips, and other integrated circuits. In one embodiment, the processor may be configured to execute hard-coded functions. In one embodiment, the processor is implemented as an executor of software instructions, and those instructions, when executed, may specifically configure the processor such that the processor implements at least any one of the models, sequences, algorithms, and / or operations described herein.

[0128] The memory may be implemented as one or more volatile memory devices, and / or as one or more non-volatile memory devices, and / or as a combination of one or more volatile memory devices and one or more non-volatile memory devices. For example, the memory may be implemented as semiconductor memory (e.g., mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory), etc.).

[0129] At least one memory and computer program code, together with at least one processor, · receiving data including gravity information from one wrist-wearable IMU 104; ·Receiving data (e.g., including the user's head orientation) from at least one head sensor 60 configured to measure the user (20); ·Calculating the yaw components of the indication light rays 21, 27 based on gravity information and at least to some extent on the data received from the head sensor 60; ·Calculating the pitch components of the indication light rays 21, 27 based at least to some extent on gravity information; ·Calculating the indication light rays 21, 27 based on the combination of the calculated yaw components and the calculated pitch components; It may be configured to cause the system 700 to perform at least the above.

[0130] At least one memory and computer program code, together with at least one processor, ·Receiving data including gravity information from one wrist-wearable IMU 104; ·Receiving data from at least one head sensor 60 configured to measure the user (20) (specifically, the orientation of that user's head); ·Calculating a normalized gravity vector based at least to some extent on gravity information; ·Calculating the yaw components of the indication light rays 21, 27 based on the normalized gravity vector and at least to some extent on the data received from the head sensor 60; ·Calculating the pitch components of the indication light rays 21, 27 based at least to some extent on the normalized gravity vector; ·Calculating the indication light rays 21, 27 based on the combination of the calculated yaw components and the calculated pitch components; It may be configured to cause the system 700 to perform at least the above.

[0131] At least one memory and computer program code, together with at least one processor, ·Receiving data including gravity information from one wrist-wearable IMU 104; ·Receiving data from at least one head sensor 60 configured to measure a user (20) (specifically, the head orientation of the user); ·Calculating yaw components of the indicating light rays 21, 27 based on gravity information and at least to some extent on the data received from the head sensor 60; ·Calculating pitch components of the indicating light rays 21, 27, the pitch components being based at least to some extent on gravity information and being calculated using a Madgwick filter configured to prioritize angular velocity accuracy over azimuth accuracy; ·Calculating the indicating light rays 21, 27 based on the combination of the calculated yaw components and the calculated pitch components; It may be configured to cause the system 700 to perform at least the above.

[0132] The system (e.g., system 700) may be configured to wirelessly (e.g., using Bluetooth) transmit data (e.g., indicating light ray data stream, user action, and / or user interface frame) to another device. The receiving side of such data transmission may be at least any one of an attached wearable device, any wearable device, and a smartphone.

[0133] The present disclosure may also be utilized by the following clauses.

[0134] Clause 1. A non - transitory computer - readable medium storing a set of computer - readable instructions, which, when executed by at least one processor, perform at least: - Measuring parallel signals corresponding to at least one of a user's movement, position, or orientation using at least one wrist - wearable inertial measurement unit and at least one head sensor; - In a controller including at least one processor and at least one memory including computer program code, the steps of receiving those measured parallel signals; - The step of combining those received parallel signals by the controller; - The step of generating an indicated light beam data stream by the controller based at least to some extent on the characteristics of those combined received signals; A non - transitory computer - readable medium for causing the system to perform.

[0135] Clause 2. A system including a wrist wearable device, where the wrist wearable device includes - A strap; - A wrist wearable inertial measurement unit (IMU) 104; - A controller including at least one processor and at least one memory including computer program code, where the at least one memory and the computer program code, together with the at least one processor, at least - The step of receiving parallel signals from a head sensor and the wrist wearable IMU 104; - The step of combining those received signals; - The step of generating an indicated light beam data stream based at least to some extent on the characteristics of those combined received signals; A controller configured to cause the controller to perform, A system including.

[0136] Clause 3. A method for calculating an indicated light beam, - The step of receiving data including gravity information from at least one wrist wearable IMU (104); - The step of receiving data from at least one head sensor (60) configured to measure a user (20) (specifically, the orientation of the user's head); - calculating a yaw component of the indicating light beams (21, 27) based at least in part on the received gravity information and at least in part on the data received from the head sensor (60); - calculating a pitch component of the indicating light beams (21, 27) based at least in part on the received gravity information; - calculating the indicating light beams (21, 27) based on a combination of the calculated yaw component and the calculated pitch component; A method comprising the steps of.

[0137] Clause 4. A computer program configured to implement the method according to clause 3, the computer program being storable on a non-transitory computer-readable medium.

[0138] As a further advantage of the present disclosure, the responsiveness of the indicating light beam 21 is improved (particularly in embodiments that use context logic to improve the light beam direction). Embodiments of the present disclosure achieve the technical effect of increasing the battery or user usage time until fatigue by increasing the user responsiveness of the light beam, thereby improving factors such as power, accuracy, ergonomics, comfort, and / or cognition. In other words, at least some embodiments achieve effective user-machine interaction by increasing the number of task actions and / or user actions achievable per unit time.

[0139] Embodiments of the disclosure provide a technical solution to a technical problem. One technical problem to be solved is to internally calculate an improved indicating light beam that achieves a high degree of usability in consideration of the application state (e.g., due to anisotropy). This has actually become a problem because the power storage and internal processing capabilities of such a device are limited because the device worn on the wrist needs to be very lightweight and not too rugged. On the other hand, in the case of external processing, there is a delay in the system, which may result in a decrease in the responsiveness of the indicating light beam.

[0140] In the embodiments of this specification, in order to overcome these limitations, the yaw of the light beam is calculated using gravity information and head orientation information received from another device. Further, the pitch is calculated based on the gravity information and gyroscope data described above. In this way, it is possible to more accurately and robustly achieve the calculation and provision of the pointing light beam. This brings several advantages. First, a highly responsive pointing light beam can be calculated and provided to the XR system in use. Second, since the light beam can be provided at least in part by a wrist wearable device, the user does not need to add a device for interacting with the XR system. Third, the wrist wearable device may be configured to further detect gestures performed by the user. Other technical improvements may also result from these embodiments, and other technical problems may also be solved.

[0141] Naturally, the disclosed embodiments of the present invention are not limited to the specific structures, processing procedures, or materials disclosed herein, but extend to equivalents that would be understood by those skilled in the art. Further, naturally, the terms used herein are used only for the purpose of describing specific embodiments and are not intended to be limiting.

[0142] References to one embodiment or an embodiment throughout this specification mean that the particular features, structures, or characteristics described in connection with that embodiment are included in at least one embodiment of the present invention. Accordingly, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. For example, when a numerical value is referenced using terms such as about or substantially, the exact numerical value is also disclosed.

[0143] A plurality of items, structural elements, compositional elements, and / or materials used in this specification may, for convenience, be present in a general list. However, these lists should be interpreted as if each element of the list is separately identified as a distinct and unique element. Thus, the individual elements of such a list should be interpreted as de facto equivalents of any other element of the same list only based on the fact that they are present in a common group, unless the opposite is indicated. Further, in this specification, various embodiments and examples of the present invention may be referred to in conjunction with alternative forms with respect to their various components. Of course, such embodiments, examples, and alternative forms should not be interpreted as de facto equivalents of each other, but rather as distinct and independent representations of the present invention.

[0144] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this description, various specific details such as examples of length, width, shape, etc. are shown so that a sufficient understanding of the embodiments of the present invention can be obtained. However, as will be understood by those skilled in the art, the present invention can be implemented without one or more of these specific details, or with other methods, components, materials, etc. In other examples, well-known structures, materials, or operations are not illustrated or described in detail, but this is to prevent the aspects of the present invention from being obscured.

[0145] Each of the above embodiments illustrates the principles of the present invention in one or more specific applications. However, as will be apparent to those skilled in the art, various changes in the form, usage, and details of the implementation may be made without exercising inventive faculty and without departing from the principles and concepts of the present invention. Accordingly, the present invention is not to be limited except as defined by the following claims.

[0146] In this document, the verbs "to comprise" and "to include" are used as open limitations that do not exclude the presence of features not described and do not require their presence either. Features described in dependent claims may be freely combined with each other, unless otherwise specified. Further, of course, the use of "a" or "an", i.e., the singular form, does not exclude plurality throughout this document.

Industrial Applicability

[0147] At least some embodiments of the present invention have industrial applications in providing a user interface (e.g., related to XR) to a controllable device (e.g., a personal computer). List of Acronyms ASIC Application-Specific Integrated Circuit AR Augmented Reality CNN Convolutional Neural Network DOF Degrees of Freedom DSP Digital Signal Processor EPROM Erasable PROM EQ Equalizer FPGA Field-Programmable Gate Array HID Human Interface Device HMD Head-Mounted Device HMI Human-Machine Interface IIR Infinite Impulse Response IMU Inertial Measurement Unit LRA Linear Resonant Actuator LSTM Long Short-Term Memory MCU Microcontroller Unit MEMS Micro-Electro-Mechanical System MR Mixed Reality NN Neural Network PCB Printed Circuit Board PROM Programmable ROM RNN Recurrent Neural Network RAM Random Access Memory ROM Read-Only Memory UI User Interface VR Virtual Reality XR Extended Reality

Explanation of Signs

[0148] 100, 101, 300, 399, 700 System 102, 162 Housing 103, 163, 303 Controller 104 Wrist Wearable Inertial Measurement Unit (IMU) 105 Mounting Component 160, 360 Headgear 361 Scene Information 170 Wrist Wearable Device 20 User 21 Homogeneous Indicating Ray 22 User Head Orientation 23 Estimated Value of User Elbow Point 24 Measured Point of User Wrist 27 Heterogeneous Indicating Ray 60, 360, 61 Head Sensor 30 Selection Plane 31 End Point of Previous Indicating Ray 32 Homogeneous Cursor Point 33 Heterogeneous Cursor Point 34 Hand Trajectory 35 Homogeneous Shift within Selection Plane 36 Heterogeneous Shift within Selection Plane 307 Classifier Data Stream 210, 310 Indicating Ray Data Stream 312 Scene Information Data Stream 114, 314 Wrist Wearable IMU Data Stream 115 Wrist Wearable Device Data Stream 116, 316 Head Sensor Data Stream 164 Head Sensor Inertial Measurement Unit (IMU) 330, 350, 370 models Phases of processes 801, 802, 803A, 803B, 804, 805 Phases of processes 901, 902, 903A, 903B, 904, 905

Claims

1. A wrist wearable device (170), comprising: A mounting component (105); A controller (103) including a processing core (701) and at least one memory (707) containing computer program code; a wrist-wearable IMU (104) configured to take measurements of a user (20); The wrist wearable device (170) comprising A system (100, 101) comprising: The system comprises: receiving data from said wrist-wearable IMU (104) including gravity information; receiving data (e.g., including a head orientation of the user) from at least one head sensor (60) configured to perform measurements of the user (20); calculating a yaw component of a pointing beam (21, 27) based on the gravity information and based at least in part on the data received from the head sensor (60); calculating a pitch component of said pointing beam (21, 27) based at least in part on said gravity information; calculating said pointing ray (21, 27) based on a combination of said calculated yaw component and said calculated pitch component; 4. The method according to claim 1, System (100, 101).

2. The system comprises: calculating a normalized gravity vector based at least in part on the gravity information; calculating the yaw component of the pointing ray (21, 27) based on the normalized gravity vector and based at least in part on the data received from the head sensor (60); calculating the pitch component of the pointing ray (21, 27) based at least in part on the calculated normalized gravity vector; The system (100) of claim 1 configured to implement:

3. The system comprises: calculating the pitch component of the pointing beam using a Madgwick filter configured to favor angular velocity accuracy over orientation accuracy; The system (100) of claim 1 configured to implement:

4. The system further includes a head device (160), the head device (160) comprising: A mounting component; A display unit; A head device controller (163); Including, At least one of the head device controller (163) or the wrist wearable device controller (103) is configured to receive the wrist wearable IMU data (114) and the head sensor data (116) and to calculate the yaw component, the pitch component, and the pointing ray (21, 27). The system (100) according to any one of claims 1 to 3.

5. The system (100) of any one of claims 1 to 4, wherein at least one of the head device controller (163) or the wrist wearable device controller (103) is configured to provide a data stream representing the indicator light (210), for example to a head device (160) and / or a computing device.

6. The system according to any one of claims 4 to 5, wherein the head device (160) comprises the head sensor (60, 61), the head device being configured to display the calculated indication rays (21, 27) using the display unit.

7. The system according to any one of claims 1 to 6, wherein the calculation of the pointing rays (21, 27) is further based on context information (312) (eg received from the scene (360)).

8. The system further comprises: receiving optical sensor data from the wrist wearable device; identifying a selection gesture based on at least one of the received optical data or the received IMU data; [0023] The present invention is configured to the identification is performed at least in part by a machine learning model (e.g., a neural network), and the system includes the machine learning model; The selection gesture may include, for example, a pinch. A system according to any one of claims 1 to 7.

9. The system further comprises: Enabling or disabling the gesture identification in response to a classification of the hand trajectory, the classification of the hand trajectory being performed, for example, by a trajectory classifier / correction phase classifier. The system of claim 8 configured to implement:

10. The system according to any one of claims 1 to 9, wherein the system is further configured to perform the steps of calculating a set of confidence values ​​for each selection gesture and outputting the confidence values.

11. The system is further configured to perform a step of adjusting a sensitivity of the gesture identification based on received context information (312), the received context information (312) including, for example, position information of an interactive element and / or proximity information regarding an end point of the indication light beam and a position of the interactive element, and the step of adjusting the sensitivity includes, for example, adjusting a confidence value threshold using the gesture identification result.

12. The system according to any one of claims 1 to 11, further configured to perform a step of adjusting a sensitivity of the gesture identification based on the trajectory of the indication light beam (21, 27), the step of adjusting the sensitivity comprising, for example, adjusting a confidence value threshold using the gesture identification result.

13. The system (399) is further configured to align the indication beam (21, 27) with respect to an origin, the coordinates of which are determined by the orientation of the user's head or gaze (POV center), e.g., the origin is equal to the center of the user's field of view.

14. The system according to any one of claims 1 to 13, further configured to perform a step of adjusting a detection threshold of at least one point interaction based on an end point of the indicating ray (21, 27), the adjusting step being based on context logic received from the scene.

15. The system according to any one of claims 1 to 14, further configured to perform a step of reducing disturbances due to measurement errors of the user's pointing pose by implementing a filter that improves the accuracy of the indication beam (21, 27).

16. The system of any one of claims 1 to 15, further configured to perform a step of adjusting a sensitivity of a machine learning model based on a previous selection history (e.g., selection history) of the user.

17. The system of any one of claims 1 to 16, further comprising a feature extraction module configured to analyze the sensor data stream to identify additional user actions beyond a selection gesture.

18. The system of any one of claims 1 to 17, wherein the instruction beam is visualized as a curved line at least between an interactive element and the wrist-wearable IMU (104).

19. The system of any one of claims 1 to 18, wherein an elbow position of an arm is estimated and the indicating light beam is aligned with the estimated elbow position and with the position of the wrist-wearable IMU (104).

20. The system of any one of claims 1 to 19, wherein the orientation of the pointing beam is corrected using eye-tracking information.

21. 21. The system of claim 1, further comprising: if the pointing ray (21, 27) is approximately parallel or anti-parallel to the normalized gravity vector, the system is configured to perform a step of recalculating a yaw component of the pointing ray (21, 27) based at least in part on the normalized gravity vector and based at least in part on the data obtained from at least one IMU (104).

22. 1. A method for calculating a pointed ray, comprising the steps of: receiving data from at least one wrist-wearable IMU (104) including gravity information; receiving data from at least one head sensor (60) configured to provide a measurement of the user (20), in particular the user's head orientation; calculating a yaw component of a pointing beam (21, 27) based on the gravity information and based at least in part on the data received from the head sensor (60); calculating a pitch component of said pointing beam (21, 27) based at least in part on said received gravity information; calculating said pointing ray (21, 27) based on a combination of said calculated yaw component and said calculated pitch component; The method includes:

23. calculating a normalized gravity vector based at least in part on the gravity information; calculating the yaw component of the pointing ray (21, 27) based on the normalized gravity vector and based at least in part on the data received from the head sensor (60); calculating the pitch component of the pointing ray (21, 27) based at least in part on the calculated normalized gravity vector; 23. The method of claim 22, further comprising:

24. calculating the pitch component of the pointing beam using a Madgwick filter configured to favor angular velocity accuracy over orientation accuracy; 23. The method of claim 22, further comprising:

25. A non-transitory computer readable medium storing a set of computer readable instructions, the set of instructions, when executed by at least one processor, to perform at least: receiving data from at least one wrist-wearable IMU (104), the data including gravity information; receiving data from at least one head sensor (60) configured to provide a measurement of a user (20), in particular a head orientation of said user; calculating a yaw component of the pointing beam (21, 27) based at least in part on the received gravity information and based at least in part on the data received from the head sensor (60); calculating a pitch component of said pointing beam (21, 27) based at least in part on said received gravity information; calculating said pointing ray (21, 27) based on a combination of said calculated yaw component and said calculated pitch component; A non-transitory computer readable medium for causing a system to implement the method.

Images