Click system and method

Abstract

According to an exemplary aspect of the present invention, there is provided an input device and a corresponding method that can digitize and convert subtle actions and gestures of a hand into directional light without interfering with normal use of the hand. For example, the apparatus and method may be used at least to calculate directional light from information received from a plurality of sensors, preferably of different types.

Description

Technical Field

[0001] Various exemplary embodiments of the present disclosure relate to a system, at least one wearable device, and a method, where the device and method can be used to control devices, particularly in the field of computing and extended reality user interface applications. Extended reality (XR) includes augmented reality (AR), virtual reality (VR), and mixed reality (MR). Background Art

[0002] Traditionally, digital devices have been controlled by dedicated physical controllers. For example, a keyboard and mouse can be used to operate a computer, a handheld controller can be used to operate a gaming console, and a smartphone can be operated through a touch screen. Generally, these physical controllers include sensors and / or buttons for receiving input from a user based on the user's actions. Such discrete controllers are ubiquitous, but they add an extra layer of technology between the user's hands and the computing device, thus hindering human-computer interaction. In addition, such dedicated devices are usually only suitable for controlling specific devices. Moreover, such devices may, for example, be in the way of the user, preventing the user from using their hands for other purposes when using the control device. Summary of the Invention

[0003] In view of the above problems, there is a need for improvement in the field of XR user interfaces. A suitable input device according to the present invention can directly digitize and convert subtle movements and gestures of the hand into machine commands, such as a pointer or cursor ray, without interfering with the normal use of the hands. For example, embodiments of the present disclosure can calculate a yaw component of a directional ray based on data received from a plurality of sensors, where the sensors are preferably of different types.

[0004] The present invention is defined by the features of the independent claims. Some specific embodiments are defined in the dependent claims.

[0005] According to a first aspect of the present invention, there is provided a system, which includes: a wrist-worn device, which includes a mounting component, a controller including a processing core, and at least one memory including computer program code; and a wrist-worn IMU configured to measure a user. Wherein, the system is configured to: receive data from the wrist-worn IMU, the data including gravity information; receive data from at least one head sensor configured to measure a user, the data including, for example, the azimuth of the user's head; calculate a yaw component of a directional ray based on the gravity information and at least partially based on the data received from the head sensor; calculate a pitch component of the directional ray, where the pitch component is calculated at least partially based on the gravity information; and calculate the directional ray based on a combination of the calculated yaw component and the calculated pitch component.

[0006] According to a second aspect of the present invention, there is provided a method for calculating a directional ray, the method comprising: receiving data from at least one wrist-worn IMU, the data including gravity information; receiving data from at least one head sensor configured to measure the orientation of a user, particularly the orientation of the user's head; calculating a yaw component of the directional ray based on the gravity information and at least in part based on the data received from the head sensor; calculating a pitch component of the directional ray, wherein the pitch is calculated at least in part based on the received gravity information; and calculating the directional ray based on a combination of the calculated yaw component and the calculated pitch component.

[0007] According to a third aspect of the present invention, there is provided a non-transitory computer-readable medium having stored thereon a set of computer-readable instructions that, when run on a processor, are capable of performing the second aspect described above, or cause a processor included in a device to be configured according to the first aspect described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A shows a schematic diagram of an isotropic directional ray presented to a user according to at least some embodiments of the present invention;

[0009] Figure 1B shows a schematic diagram of a heterogeneous directional ray presented to a user according to at least some embodiments of the present invention;

[0010] Figure 2A and 2B shows a schematic diagram of an exemplary system capable of supporting at least some embodiments of the present invention;

[0011] Figures 3A to 3C shows a schematic diagram of an exemplary device capable of supporting at least some embodiments of the present invention;

[0012] Figure 4A and 4B shows an exemplary process capable of supporting at least some embodiments of the present invention in the form of a flowchart;

[0013] Figure 5 shows a schematic diagram of a system capable of supporting at least some embodiments of the present invention; and

[0014] Figure 6 shows a schematic diagram of an exemplary device capable of supporting at least some embodiments of the present invention. DETAILED DESCRIPTION

[0015] When interacting with AR / VR / MR / XR applications, such as when using devices such as smartwatches or extended reality headsets, users need to be able to perform various actions (also known as user actions), such as select, drag and drop, rotate and lower, use sliders, and zoom. Embodiments of the present invention improve the detection of user actions, which can result in enhanced responsiveness when implemented by or in a controller or system. The actions can be performed on one or more interactive elements (such as interactive elements). Interactive elements can, for example, include user interface sliders, user interface switches, or user interface buttons in a user interface, etc.

[0016] A system, wearable device, and method related to at least one of computing and / or displaying directed light are described herein. The system, wearable device, and method can be used, for example, for at least one of the following: measurement, sensing, signal acquisition, analysis, user interface tasks. Such a system or at least one device is preferably suitable for a user to wear. The user can use the system or at least one device to control one or more external and / or stand-alone devices. The external and / or stand-alone devices can be, for example, a personal computer, server, mobile phone, smartphone, tablet device, smartwatch, or any type of suitable electronic device. The control can be in the form of a user interface (UI) or a human-machine interface (HMI). The system can include at least one device. The at least one device can include one or more sensors. The system can be configured to generate user interface data based on data from the one or more sensors. The user interface data can be at least partially used to allow the user to control the system or at least a second device. For example, the second device can be at least one of a personal computer, server, mobile phone, smartphone, tablet device, smartwatch, or any type of suitable electronic device. The controlled or controllable device can perform at least one of the following: application programs, games, and / or operating systems, any of which can be controlled by a multimodal device.

[0017] The user can perform at least one user action. Generally, the user will perform actions to affect the controllable device and / or the AR / VR / MR / XR application. For example, user actions can include at least one of the following: movement, gesture, interaction with an object, interaction with a user body part, empty action. An example of a user action is the "pinch" gesture, where the user touches the tip of the index finger with the tip of the thumb. Another example is the "thumbs up" gesture, where the user extends their thumb, bends their fingers, and rotates their palm so that the thumb is facing up. This embodiment is configured to identify (determine) at least one user action based at least in part on sensor input. Reliably identifying user actions enables action-based control, such as using the embodiments disclosed herein.

[0018] A user action may include at least one characteristic, referred to as a user action characteristic and / or an event. For example, such a characteristic may be any of the following: temporal, positional, spatial, physiological, and / or kinematic characteristics. The characteristic may include an indication of a user body part, such as a user's finger. The characteristic may include an indication of user movement, such as a user's hand trajectory. For example, the characteristic may be the movement of the middle finger. In another example, the characteristic may be a circular motion. In yet another example, the characteristic may be: the time before the action, and / or the time after the action. In at least some embodiments, the characteristics of the user action are determined at least in part by a neural network based on sensor data.

[0019] In at least some embodiments, the system includes at least one device that includes a mounting component configured to be worn by a user. Thus, the at least one device is a wearable device. Such a component may be any of the following: a strap, a band, a wristband, a bracelet, a glove, glasses, goggles, a helmet, a hat, a headband, or similar headgear. For hand-mounted embodiments, a strap is preferably used. The mounting component may be connected to and / or formed by another device (such as a smartwatch) or form part of a larger device (such as a glove or a head mount). In some embodiments, the width of the strap, band, and / or wristband is 2 to 5 centimeters, preferably 3 to 4 centimeters, more preferably 3 centimeters. The mounting component may be connected to and / or formed by another device (such as a head-mounted display, a virtual reality head mount, an extended reality head mount, an augmented reality head mount, or a mixed reality head mount).

[0020] In some embodiments, a signal is measured by at least one sensor. The sensor may include a processor and a memory or be connected to a processor and a memory, where the processor may be configured to perform the measurement. The measurement may be referred to as sensing or detection. The measurement includes, for example, detecting a change in at least one of the user, the environment, and the physical world. For example, the measurement may further include applying at least one timestamp to the sensed data and transmitting the sensed data (with or without the at least one timestamp).

[0021] Embodiments of the present disclosure are configured to measure a user's position, orientation, and movement by using an inertial measurement unit (IMU). The IMU may be configured to provide position information of at least a part of the system or a part of the user's body (on which the IMU is mounted, connected, or worn). The IMU may be referred to as an inertial measurement unit sensor. The IMU includes a gyroscope. In addition, the IMU may further include at least one of the following: a multi-axis accelerometer, a magnetometer, an altimeter, a barometer. The IMU preferably includes a magnetometer because the magnetometer can provide an absolute reference for the IMU. A barometer that can be used as an altimeter can provide an additional degree of freedom for the IMU.

[0022] Sensors, such as inertial measurement unit sensors, can be configured to provide sensor data streams. Such provision can occur within the device, e.g., to a controller, a digital signal processor (DSP), a memory. As an alternative or in addition, such provision can be made to an external device. The sensor data stream can include one or more raw signals measured by the sensor. Additionally, the sensor data stream can further include at least one of synchronization data, configuration data, and / or identification data. For example, a controller can use this data to compare or combine data from different sensors.

[0023] In some embodiments, an IMU and other sensors can provide data representative of the orientation, position, and / or movement of a user's body to a system and / or a wearable device. The system and / or the wearable device can calculate at least one directional ray based on the data. For example, the system can include: a head-mounted device, such as an HMD, and / or a wrist-worn device, such as a smartwatch.

[0024] Devices such as a wrist-worn device or a head-mounted device can include controllers 103, 163 located within enclosures 102, 162. The controllers 103, 163 include at least one processor, at least one memory, and a communication interface, where the memory can include instructions that, when executed by the processor, allow communication with other devices (e.g., computing devices) via the communication interface. The controllers 103, 163 can be configured to communicate with a head sensor and a wrist-worn IMU (e.g., head sensor 60 and wrist-worn IMU 104) to at least receive data streams from the head sensor and the wrist-worn IMU. In other words, the controllers 103, 163 can be configured to cause the controller to receive at least one sensor data stream from at least one sensor. The controller can be configured to perform preprocessing on the at least one sensor data stream, where the preprocessing can include the preprocessing disclosed herein. The controller can be configured to process the at least one sensor data stream received from at least one sensor. The controllers 103, 163 can be configured to generate at least one user interface (UI) event and / or command based on the characteristics of the processed sensor data stream. Additionally, the controllers 103, 163 can include a model that can include at least one neural network. In some embodiments, the head-mounted device controller 163 can be located within or mounted on the head-mounted device. In some other embodiments, the wrist-worn device controller 103 can be located within or mounted on the wrist-worn device.

[0025] The devices in this disclosure, such as the wrist-worn device 170, can be configured to communicate with at least one head device 160, which includes head sensors 60, 61, 164. The head device should be understood as a device that can be worn or mounted on the user's head. For example, such a head device can include glasses, goggles, helmets, hats, headbands, or similar headgear. The head device 160 can include, for example, at least one of a head-mounted device (HMD), a head mount, a head-mounted display, a virtual reality head mount, a mixed reality head mount, an extended reality head mount, or an augmented reality head mount. The head device can include a processor, a memory, and a communication interface, where the memory can include instructions that, when run by the processor, allow communication with other devices such as computing devices through the communication interface, and / or communication with other devices within the same system that includes the head device.

[0026] For the calculation of the directional light, the orientation of the directional light 21, 27 can be corrected and / or compensated by the head sensor data, especially sensor drift. Such head sensor data can include, for example, data from a head-mounted device (HMD), a head mount, a virtual reality head mount, an extended reality head mount, an augmented reality head mount, or a camera, from which data representing, for example, user gaze, eye tracking, field of view, head position, and / or head orientation can be obtained.

[0027] The head sensors are configured to measure the position, orientation, and / or movement of the user's head. The head sensor 60 can be installed inside or on the head device 160 (such as a head-mounted device). The head sensor 60 can include an inertial measurement unit (IMU) 164, and / or the head sensor can include camera-based user head orientation detection, such as gaze tracking. The IMU-based head sensor 164 can be configured to provide the orientation information of the user's head because the IMU-based head sensor 164 can be connected or mounted to the head device and thus to the user's head. The IMU-based head sensor 164 preferably includes a gyroscope. In addition, the IMU-based head sensor can include at least one of the following: a multi-axis accelerometer, a magnetometer, an altimeter, a barometer. The head sensor 60 (such as in the form of an IMU and / or a camera system) can be configured to provide a sensor data stream to the controller 103. Devices such as the head device 160 and / or the wrist-worn device 170 can include such a controller. The data of the head sensor 60 can provide zero-drift or near-zero-drift data, which can be used to correct the drift of the data of the wrist-worn IMU 104. The wrist-worn IMU 104 can be connected to at least one controller or configured to communicate with at least one controller. The head sensor can be connected to at least one controller or configured to communicate with at least one controller.

[0028] The camera-based head sensor 60 may include detection of the orientation of the user's head based on a camera. In some embodiments, the data of the head sensor 60 may include camera data, or data received from multiple cameras. The camera data may be used to calculate the position and / or orientation of the user's head. For example, multiple cameras positioned relative to the user and / or the head device in the external environment may be used to track the position of the head device. As an alternative or addition, the head device may also include multiple cameras configured to track and / or image the environment to obtain the position and / or orientation of the user's head relative to the head device and / or the user's surrounding environment.

[0029] Figure 1A System 100 is shown, which includes a wrist-worn device 170 and a head sensor 60. The system 100 is capable of calculating and / or displaying the isotropic orientation ray 21. The wrist-worn device 170 includes a wrist-worn IMU 104. In isotropic ray projection, the position of the user's hand and the movement of the wrist-worn IMU 104 produce a linear proportional change in the orientation ray 21. An additional device, such as a display unit, may be used to display the orientation ray 21.

[0030] The "selection plane" should be understood as a plane in three-dimensional space, such as a two-dimensional plane, on which an interactive element or other user interface (UI) element may be located, and the orientation rays 21, 27 terminate on this plane, that is, the endpoints of the orientation rays 21, 27 are located on this plane. It should be noted that the selection plane may be a curved plane, for example. Alternatively, the orientation ray may extend beyond the interactive element, in other words, the orientation rays 21, 27 and the interactive element intersect at the selection plane. Depending on the position and / or orientation of the orientation ray, the selection plane may include different positions and orientations.

[0031] As Figure 1A shown, the movement of the hand and the resulting hand trajectory 34 produce a linear movement 35 of the isotropic orientation ray 21 and its endpoint at the selection plane 30. In other words, the movement of the endpoint of the isotropic orientation ray 21 on the selection plane 30 is proportional to the movement of the user's hand on which the wrist-worn device 170 and the wrist-worn IMU are provided. Therefore, the movement trajectory 34 of the wrist-worn IMU and the movement trajectory of the user's hand are proportionally reflected on this selection plane. The endpoint position of the isotropic orientation ray 21 on the selection plane 30 moves from the previous orientation ray endpoint 31 to Figure 1A the isotropic endpoint 32 therein and has a linear relationship with the hand trajectory 34. Therefore, in isotropic ray projection, the isotropic light point 32 is the endpoint of the orientation ray 21. It should be noted that due to the isotropic behavior of the isotropic orientation ray 21, the desired light point, represented as the anisotropic light point 33 in Figure 1A may be missed by the cursor.

[0032] Figure 1B Disclosed is a system 100 capable of calculating and / or displaying heterogeneous directional light rays 27. In other words, the movement of the user's hand position and the wrist-worn device 170 thereon can provide a non-linear proportion change of the heterogeneous directional light rays 27, for example, according to the context logic. An additional device, such as a display unit, can be used to display the heterogeneous directional light rays 27.

[0033] In Figure 1B , the movement 36 of the endpoints of the heterogeneous directional light rays 27 can be non-linearly proportional to the movement of the wrist-worn IMU 104, so that the hand trajectory 34 can be measured at least in part by the wrist-worn IMU 104. The endpoint position of the heterogeneous directional light rays 27 on the selection plane 30 moves from the previous directional light ray endpoint 31 to Figure 1B the heterogeneous light point 33 in , and is non-linearly related to the hand trajectory 34.

[0034] In Figure 1B the example shown, the required light points (such as user interface sliders, user interface switches, or user interface buttons in the user interface or other context information) affect the movement speed and / or positioning of the endpoints of the heterogeneous directional light rays 27. This change in the movement of the directional light rays 27 is displayed as a curved line, i.e., a curved directional light ray. If homogeneous directional light rays are used, as in Figure 1A , the homogeneous light point 32 would be the endpoint of such homogeneous directional light rays. Therefore, when calculating the directional light rays and the light points, the interactive elements at the required light points ( Figure 1B the heterogeneous light point 33 in ) would not be considered. In embodiments using the heterogeneous directional light rays 27, such directional light rays can be displayed as a curved curve, i.e., a non-linear trajectory at least between the interactive element and the wrist-worn IMU 104. The interactive element can be an interactive element. For example, a head-mounted display can be used to obtain such a visualization effect.

[0035] Figure 2A And 2B respectively show the system 100 and the systems 100, 101 according to at least some embodiments of the present invention. The wrist-worn device 170 includes a mounting member 105, which is presented in the form of a strap in Figure 2A . The wrist-worn device 170 includes a housing 102 connected to the mounting member 105. The wrist-worn device 170 includes a controller 103 and an IMU 104 within the housing 102. The user 20 can wear the wrist-worn device 170, for example, on the user 20's wrist or other parts of the arm. Thus, the wrist-worn IMU 104 represents the movement data of the user's hand.

[0036] In at least some embodiments, a wrist-worn inertial measurement unit (IMU) 104 or a sensor (such as a gyroscope) or multiple sensors disposed in a single device or multiple devices are placed and / or fixed on a user's arm. For example, a mounting component 105 can be used for fixation. The arm should be understood as the upper limb of the user's body, including the hand, forearm, and upper arm. The wrist connects the forearm and the hand. In the context of the present disclosure, the hand, arm, and forearm can be used interchangeably.

[0037] In at least some embodiments, the wrist-worn device 170 can include a smartwatch. In at least some embodiments, the wrist-worn device 170 can include at least one of the following, for example: a tactile device, a screen, a touch screen, a speaker, a heart rate sensor (such as an optical sensor), a Bluetooth communication device.

[0038] The wrist-worn device 170 can be configured to at least participate in providing data for calculating and / or displaying the directional light 21.

[0039] See Figure 2B , the systems 100, 101 include a wrist-worn device 170 and a head-mounted device 160. The systems 100, 101 can be adapted to sense the movement and / or actions of a user and be used as a user interface device and / or a human-machine interface device. The head-mounted device includes a head sensor 60, which can be a wrist-worn IMU.

[0040] The systems 100, 101 and / or the wrist-worn device 170 include a wrist-mounted inertial measurement unit (IMU) 104. The wrist-worn IMU 104 is configured to, for example, send a sensor data stream to the controller 103. The sensor data stream includes, for example, at least one of the following: multi-axis accelerometer data, gyroscope data, and / or magnetometer data. One or more components of the wrist-worn device 170 can be combined together. For example, the wrist-worn IMU 104 and the controller 103 can be located on the same printed circuit board (PCB). Changes in the wrist-worn IMU 104 data reflect, for example, the movement and / or actions of the user, and thus such movement and / or actions can be detected by using the wrist-worn IMU data. The wrist-worn IMU 104 can be directly or indirectly connected to the processor and / or memory of the controller 103 to provide sensor data to the controller.

[0041] In some embodiments, the system is further configured to reduce interference caused by measurement errors of the user's pointing gesture by applying a filter to improve the accuracy of the directional lights 21, 27.

[0042] The trajectory of the hand should be understood as the spatio-temporal path of a part of the arm. The trajectory of the hand can be measured and / or calculated by a suitable device, such as a wrist-worn device including a wrist-worn IMU. The device can then be used to calculate the directional rays 21, 27. The arm also includes an elbow that connects the upper arm and the forearm. In some embodiments, a calculated estimate of the elbow position or a so-called elbow point 23 can be used to assist in the formation and / or accuracy of the directional rays 21, 27. The directional rays 21, 27 can be aligned with the elbow point 23 and the position of the wrist-worn IMU 104. For example, a wrist-worn device consisting of a wrist-worn IMU with six degrees of freedom (6-DOF) and a barometer can be used to assist in predicting the elbow height of the hand being interacted with and use the predicted elbow height as an anchor point for the directional rays 21, 27. As an alternative or addition, head sensor data can also be used at least in part to estimate the horizontal position of the elbow point. In some embodiments, the elbow position of the arm can be estimated. Additionally, the directional rays can be aligned with the estimated elbow position and the position of the wrist-worn IMU 104. Such alignment can include adjusting the directional rays 21, 27 so that the starting point is approximately at the user's elbow and the intersection of the ray with the position of the IMU 104 on the user's wrist, and then continuing.

[0043] The directional rays 21, 27 can be displayed in a point-and-click user interface of the head-mounted device 160, which includes a display unit, and / or a display unit separate from the head-mounted device, such as an external screen, a computer monitor, or a display screen. The head-mounted device 160 can be configured to communicate with the system 100, transmit / receive data streams, or the system (such as system 100 or system 101) can include the head-mounted device 160. In some embodiments, the directional rays 21, 27 can be visualized on a display, such as a display in a head-mounted device, or an external display (such as a computer monitor). In embodiments where the directional rays 21, 27 are displayed to the user, the directional rays 21, 27 can be displayed as curves rather than straight lines extending from the user's hand to a selection plane in three-dimensional space. In other words, in cases where the position of the wrist-worn IMU 104 and the light punctuation on the selection plane are not aligned on the same straight path resulting from the hand trajectory estimate, the directional rays 21, 27 can be displayed as curved lines.

[0044] In some embodiments, the directional rays 21, 27 are calculated on a wrist-worn device, such as the wrist-worn device 170 including a controller, when receiving the data streams of the wrist-worn IMU 104 and the head sensor 60. However, in other embodiments, the directional rays 21, 27 are calculated by a head device 160 including a controller, where the head device is configured to receive data streams from the head sensor and the wrist-worn IMU. The calculation of each component of the directional rays 21, 27 can be performed on different devices. For example, one component of the directional ray can be calculated on the wrist-worn device 170, while another component can be calculated on the head device 160 (such as a head-mounted device). These components can be, for example, the pitch component and the yaw component of the directional ray.

[0045] Figure 3A An exemplary system 100 including the wrist-worn device 170 is shown. The device 170 includes a controller 103 and a wrist-worn IMU 104. Additionally, the memory may include instructions that, when executed by the processor, may allow the wrist-worn device 170 to calculate the directional rays 21 based on the data stream 116 from the head sensor 60 and the data stream 114 from the wrist-worn IMU 104. The wrist-worn device 170 is configured to receive the head sensor data stream 116 from the head sensor 60 in the controller 103. The controller 103 is configured to receive the wrist-worn IMU data stream 114 from the IMU 104. The directional rays can be calculated at least based on the wrist-worn IMU data stream and the head sensor data stream. The controller 103 may be configured to calculate the directional rays 21, 27. Then, the calculated directional rays 21, 27 can be transmitted as a directional ray data stream 114 to the head device 160, which includes, for example, a display unit where the directional rays 21, 27 can be visualized. The head sensor 60 and the head device 160 may be included in one housing and / or device.

[0046] Figure 3B An exemplary system 101 including the head device 160 is shown, where the head device includes a controller 163, which includes a processor, a memory, and a communication interface. The memory may include instructions that, when executed by the processor, allow the head device 160 and / or the controller 163 to calculate the directional rays 21, 27 based on the data stream 116 from the head sensor 60 and the data stream 115 from the wrist-worn device 170. The data stream 115 from the wrist-worn device 170 can be obtained at least partially from the wrist-worn IMU 104 as the data stream 114. The head sensor 60 can be separate from the head device 160, or the head device 160 can include a head sensor, such as the IMU 164.

[0047] The controller 163 can be configured to calculate the directional rays 21, 27. At least partial information related to the calculated directional rays 21, 27 can be provided to the user via a head-mounted display. Such provision can include displaying, visualizing, or presenting the directional rays 21, 27 in an extended reality environment, such as augmented reality, mixed reality, or virtual reality. The controller 163 can be configured to provide the directional rays 21, 27 in the form of a data stream 210.

[0048] Figure 3C An exemplary system 399 is shown, which includes a wrist-worn device and a head device 160, where the head device includes a controller 163 that includes a processor, a memory, and a communication interface. At least partial information related to the calculated directional rays 21, 27 can be provided to the user via a head-mounted display. The controller 163 can be configured to provide the directional rays 21, 27 in the form of a data stream 210. For the system 399, the head sensor 60 can include a head sensor 61 capable of detecting the user's gaze, or it can be the head sensor 61. The head sensor 61 can include an eye-tracking function, such as a camera configured to capture eye movements. In one embodiment, eye tracking can be extended to gaze tracking, where information related to the user's gaze position in a three-dimensional environment is obtained. In one embodiment, the user's gaze can be used to correct or assist in correcting drift in IMU-based hand pose estimation, thereby correcting or assisting in correcting the yaw component estimation of the directional rays 21, 27. In one embodiment, gaze tracking is used to obtain context logic information related to the direction and / or orientation of the directional rays 21, 27. In other words, gaze tracking provides information on where the user is looking, and this information can be used to estimate and / or correct the direction and / or orientation of the directional rays 21, 27. Similarly, eye-tracking information can also be used to correct the orientation of the directional rays 21, 27, for example, by directing the directional rays towards the point where the user is looking.

[0049] In one embodiment, the system (such as system 399) can be further configured to align the directional rays 21, 27 to an origin, where the coordinates of the origin are determined by the orientation of the user's head or gaze (POV center), for example, the origin is equal to the center of the user's field of view. Such alignment can include adjusting the orientation and / or position of the directional rays 21, 27, such as directing the directional rays towards the origin. In addition, the range (i.e., length) of the directional rays 21, 27 can be adjusted according to the origin.

[0050] The systems in the present disclosure, such as system 100, system 101, or system 399, and the devices included therein, such as the head device 160 and / or the wrist-worn device 170, can be configured to, in accordance with received instructions for changing the mode, for example Figure 3A 、 3BChange the operation mode between 3C. In other words, the system can initially provide the data stream 115 to the head-mounted device, and then change to the operation mode of calculating light on the wrist-worn device in the same or different applications. This configuration provides flexibility because the user can use the wrist-worn device with multiple different systems. The system can be configured to communicate with at least one computing device, where the communication can include providing and / or receiving data streams. The computing device can include at least one of a computer, a server, a mobile device (such as a smartphone or a tablet), etc. The device can be configured to at least participate (e.g., by computing) in providing a VR / XR / AR / MR experience to the user. The computing device can include a processor, a memory, and a communication interface, where the memory can include instructions that, when executed by the processor, can communicate with other devices (such as the wrist-worn device 170 or the head-mounted device 160) through the communication interface. Alternatively, the head-mounted device can include the computing device.

[0051] The calculated directional light 21 depends at least on the position, movement, and changes of the user's arm, which can be measured by the device 170, for example. Euler angles can be used to represent the orientation of the directional lights 21, 27, including the yaw component ψ, the pitch component θ, and the roll component φ, or the heading angle, the elevation angle, and the inclination angle. In some embodiments, the roll component can be omitted or its contribution reduced because its impact on the total direction of the directional lights 21, 27 may be negligible.

[0052] Using the data stream of the wrist-worn IMU 104, the data stream of the head sensor 60, and the gravity information (such as the calculated gravity vector), the directional lights 21, 27 can be obtained by calculating the yaw component and the pitch component of the directional lights 21, 27. According to the following formula EQ.1, the directional light 21 can be represented as a vector v, that is, a combination of the yaw component ψ and the pitch component θ.

[0053]

[0054] According to the above formula EQ.1, the forward direction of the directional lights 21, 27 can be defined as v = (0, 0, -1).

[0055] The system or device can obtain the sensor data stream from the wrist-worn inertial measurement unit (IMU) 104 located on the user's hand for calculating the directional lights 21, 27, in other words, for light projection. The data stream of the wrist-worn IMU 104 can include gyroscope data and / or gravity information.

[0056] The gravity information describes the orientation information of the sensor or multiple sensors relative to gravity. For example, the gravity information can be obtained from an inertial measurement unit (IMU).

[0057] The gravity information may include a gravity vector. The term "gravity vector" refers to a three-dimensional vector that describes the direction and magnitude of gravity relative to the sensor orientation, and the data for calculation is obtained from this sensor. The gravity vector can be calculated at least from the accelerometer data and gyroscope data from the wrist-worn IMU 104. Additionally, in some embodiments, the gravity vector can be calculated by combining data streams obtained from multiple sensors and / or sensor types. Alternatively, for example, a pre-calculated gravity vector data stream can be obtained from the wrist-worn IMU 104 or a wrist-worn device configured to generate a data stream for calculating the gravity vector. Such a pre-calculated gravity vector and the corresponding data stream can be considered to be obtained from a "software sensor", where multiple sensor signals are processed and calculated to obtain the gravity vector.

[0058] The gravity information may include the accelerometer and gyroscope data streams received from the wrist-worn IMU 104, and a Madgwick filter can be applied to at least one of the data streams. Such a Madgwick filter provides the orientation information of the wrist-worn IMU 104, which can be used to calculate the pitch component and yaw component of the directed light rays 21, 27, similar to using the gravity vector to calculate the above components.

[0059] The obtained gravity vector can be normalized according to the normal or length of the gravity vector to provide a normalized gravity vector. The normalized gravity vector can be defined using the obtained gravity vector according to the following formula EQ.2 as follows.

[0060]

[0061] The yaw component ψ of the directed light ray 21 can be calculated according to the calculated normalized gravity vector and at least one sensor data stream, such as the yaw component of the gyroscope data obtained from the IMU 104 worn on the user's hand.

[0062] In addition, the head sensor data can be used to calculate the drift of the wrist-worn IMU 104 sensor in the yaw component. As previously mentioned, the head sensor data can be received from a head device, such as a head device configured to obtain head orientation information and display VR, MR, AR, or XR applications to the user. The information obtained can be pre-processed by the head device containing the head sensor to obtain the orientation of the head device and thus the head orientation of the user. Alternatively, the head sensor data stream can be transmitted to a controller, such as one included in the wrist-worn device, or the head sensor data stream can be pre-processed and / or processed at the head device. For example, the raw data streams of the head sensor 60 and the wrist-worn IMU 104 can be processed by the head device 160. The head sensor 60 data is useful for providing information about the orientation of the user and / or the head orientation of the user, as well as the possible intention with respect to the direction of the directional light 21 and related user actions.

[0063] The head sensor 60 data can provide information about the user's gaze position, which can help evaluate the direction and orientation of the directional lights 21, 27. The head sensor data can be provided as a data stream containing information related to the head orientation.

[0064] An exemplary process 903B for obtaining the yaw component ψ of the directional lights 21, 27 performed by a system (such as any of the systems 100, 101, and 399) is as follows: The vertical direction vector derived from the normalized gravity vector describes the downward direction relative to the sensor device coordinates, perpendicular to the user's arm or hand at the location of the sensor. In other words, represents the roll component of the gravity vector and thus further represents the normalized gravity vector projected onto the plane perpendicular to the arm. The vertical direction vector can be as described by the following formula EQ.3.

[0065]

[0066] The yaw component of the angular velocity describes the yaw in the world orientation, while describes the sensor orientation. The yaw component of the angular velocity can be calculated by the dot product of the gyroscope data and the vertical direction vector .

[0067]

[0068] Then, according to the following formula EQ.5, the yaw component of the directional lights 21, 27, which is the time integral of the yaw component of the angular velocity, can be obtained

[0069]

[0070] Due to sensor drift, a scalar c can be used to center the yaw orientation of the directed light beam obtained from a head sensor (which is included, for example, in a head-mounted device such as a headset). This yaw is referred to as the headset yaw ψ h . Then, as shown in Equation EQ.6, the product between the scalar c and the angular velocity yaw component is integrated with respect to time t to obtain the corrected yaw component of the gyroscope data.

[0071]

[0072] The yaw angle of the gyroscope corresponds to the headset yaw angle ψ h . The difference between the two yaw components (i.e., the gyroscope yaw component and the headset yaw component) can be used to calculate the scalar c. The sign of this difference is inverted to represent the direction of the directed light beam, as shown in Equation EQ.7.

[0073]

[0074] Using the difference Δψ between the gyroscope yaw component and the headset yaw component, the value of the scalar c can be obtained.

[0075] The following Equation EQ.8 gives an example of the value of the scalar c.

[0076]

[0077] In other words, c may depend on the gyroscope yaw component and the difference with the headset yaw component ψ h . Thus, the yaw component ψ of the directed light beams 21, 27 can be obtained through the scalar c.

[0078] One advantage of using the head sensor and the yaw component of the head sensor is the ability to correct and / or compensate for sensor drift, particularly in the yaw component obtained from a wrist-worn IMU. In other words, since the drift of the yaw component of the head sensor is minimal or non-existent, the head sensor data can be used to correct sensor drift, for example, as described in Equations EQ.7 and EQ.8. The head sensor data and its calculated yaw component are obtained relative to the yaw component of the wrist-worn IMU 104, in other words, in a comparable coordinate system.

[0079] The pitch component θ of the directed light 21 can be calculated at least in part by normalizing the gravity vector and at least one sensor data stream (e.g., gyroscope data obtained from the IMU 104). The at least one sensor data stream can be high-pass filtered, while the gravity vector data is low-pass filtered.

[0080] An example of a process 903A for obtaining the pitch components of the directed lights 21, 27 performed by a system (e.g., any of the systems 100, 101, and 399) is described below.

[0081] To obtain the pitch components of the directed lights 21, 27, one can calculate a vector. This vector is perpendicular to the normalized gravity vector and perpendicular to the arm. It can be obtained by using the following formula EQ.9

[0082]

[0083] The angular velocity pitch component is obtained by taking the dot product between the vector and the angular velocity of the obtained gyroscope data using formula EQ.10.

[0084]

[0085] Using the angular velocity pitch component and integrating with respect to time t according to the following formula EQ.11, the pitch angle component of the gyroscope can be obtained

[0086]

[0087] Due to sensor drift and other reasons, the normalized gravity vector can be used to correct the pitch angle. Next, the angle of the normalized gravity vector the normalized gravity vector the angle can be calculated, for example, by formula EQ.12.

[0088]

[0089] There are other ways to calculate the angle For example, formula EQ.12 may not be accurate, especially when is close to or equal to 1. Therefore, other components of the gravity vector can be combined. An example is to use the two-variable arctangent in formula EQ.13.

[0090]

[0091] By combining the gravity angle and the gyro pitch angle it is possible to obtain the pitch angle θ of the directional light rays 21, 27, which is the representation of the vertical component. In addition, it is also possible to filter the gravity angle and the gyro pitch angle . By correcting sensor defects and using such filtering to reduce noise, the accuracy of the directional light rays 21, 27 can be improved. At least in some embodiments, a low-pass filter L is performed on the gravity vector , while a high-pass filter H is performed on the gyro pitch component angle to obtain the pitch component θ of the directional light rays 21, 27, as shown in the following formula EQ.14.

[0092]

[0093] In addition, filtering can also be performed such that the yaw component of the gyro data or the yaw component of the gravity vector data is emphasized in the yaw component of the directional light rays 21, 27. Such low-pass filtering and high-pass filtering can be performed, for example, using an infinite impulse response filter according to the following formulas EQ.15 and EQ.16, respectively.

[0094] low t = α * in t + (1 - α) * low t-1 (EQ.15)

[0095] high t = in t - low t (EQ.16)

[0096] For example, the α value (i.e., the attenuation parameter) of the low-pass filter in the above formula EQ.15 can be selected such that the yaw component of the gyro data or the yaw component of the gravity vector data is emphasized in the yaw component of the directional light ray 21. Alternatively, the α value can also be selected such that approximately half of the signal comes from the yaw component of the gyro data and half of the signal comes from the yaw component of the gravity vector data. Using alternatively or additionally, low-pass filtering and high-pass filtering can also be performed, for example, using Chebyshev, Butterworth, and / or various other filter designs.

[0097] The cut-off frequencies of the low-pass filter and the high-pass filter can be the same or substantially the same. In other words, the frequency boundaries of the low-pass filter can be the same or substantially the same as the frequency boundaries of the high-pass filter.

[0098] In some embodiments, if the directional rays 21, 27 are substantially parallel or anti-parallel to the normalized gravity vector, the device may be configured to recalculate the yaw component of the directional rays 21, 27 based at least in part on the normalized gravity vector and at least in part on data obtained from at least one IMU 104. In other words, when the user points the hand with the wrist-worn IMU 104 downward or substantially downward, the yaw component of the directional rays 21, 27 will be reset. When the hand is lifted from the downward position, the directional rays will point or be oriented towards the direction of the user's head, and this information can be obtained by the head sensor 60. In this case, the yaw component obtained from the wrist-worn IMU can be omitted until reliable information about the orientation of the wrist-worn IMU relative to the head sensor information is obtained. A similar reset of the directional rays can also be performed when the user's hand points directly upward or substantially upward.

[0099] In some embodiments, the sensitivity of gesture recognition can be adjusted based on information based on the interactive element and / or the trajectory of the directional rays 21, 27.

[0100] Figure 4A An exemplary process that can support at least some embodiments of the present invention is shown in the form of a flowchart. Figure 4A The flowchart includes illustrative but non-limiting steps 801, 802, 803A, 803B, 804, and 805. In addition, the operations in steps 801, 802, 803A, 803B, 804, and 805 do not have to be performed simultaneously.

[0101] In Figure 4A In step 801 at the top, data is obtained from at least one wrist-worn IMU 104 and at least one head sensor 60. Figure 4A The operations and steps of the illustrated process can be performed at least in part by a device, such as the device disclosed herein, such as a controller, such as controller 103. At least some operations (such as providing context information) can be performed by a second device, such as a computing device or a controllable system. The data of at least one wrist-worn IMU 104 includes gyroscope data. Such data of the wrist-worn IMU 104 can be obtained, for example, as part of a data stream. The data of at least one head sensor 60 can be, for example, the head orientation.

[0102] In step 802, data is obtained from at least one wrist-worn IMU 104. This data includes gyroscope data and gravity information. The gravity information can include information received from an IMU mounted on the hand, such as accelerometer and / or gyroscope data. The gravity information can be normalized, for example, where the normalization process can include scaling the information. Then the gravity information can be used in step 803A to calculate the pitch component of the directional rays.

[0103] In some embodiments, the pitch component can be directly obtained from gravity information, such as according to Equation EQ.12 or EQ.13 for calculating pitch from the normalized gravity vector and its components. In other words, for example, in step 802, gyroscope data can be omitted from the calculation of the pitch component in step 803A.

[0104] For the calculation of the yaw component, head sensor data can be used together with the yaw component (such as gyroscope data) of the wrist-worn IMU to calculate the yaw component of the directional rays 21, 27 in step 803B. The head sensor data is used to correct the drift of the yaw component calculated from the wrist-worn IMU 104.

[0105] For the calculation of the pitch component in step 803A, for example, the Madgwick filter can be used for the calculation to absorb the "excessive" gyroscopes. In other words, the filter can be configured to prioritize angular velocity correctness over azimuth correctness.

[0106] In step 804, the calculated yaw component and the calculated pitch component are combined to obtain the directional rays 21, 27, for example, by Equation EQ.1 discussed above.

[0107] In step 805, collisions with the obtained directional rays can be detected. Such collisions can provide context information for calculating consecutive or subsequent directional rays and / or the position, location, and / or orientation of the consecutive or subsequent directional rays. In addition, such collisions can be intersections between the directional rays 21, 27 and the surrounding environment (such as interactive elements, such as user interface buttons, switches, and sliders, or graspable objects, etc.), or a part of these intersections. The collisions with the directional rays 21, 27 can be displayed and / or visualized in extended reality, which is, for example, virtual reality, augmented reality, or mixed reality.

[0108] Figure 4B An exemplary process that can support at least some embodiments of the present invention is illustrated in the form of a flowchart. Figure 4B The flowchart includes illustrative but non-limiting steps 901, 902, 903A, 903B, 904, and 905. In addition, the operations in steps 901, 902, 903A, 903B, 904, and 905 do not have to be performed simultaneously.

[0109] Use the data obtained from at least one wrist-worn IMU 104 to calculate the gravity vector. Then, in step 902, the gravity vector is normalized. After that, the normalized gravity vector is used to calculate the yaw component and the pitch component of the directional rays.

[0110] For the pitch components of the directional rays 21, 27, low-pass filtering of the gravity vector data and high-pass filtering of the gyroscope data can be applied, as shown in step 903A. For example, filtering can be performed using the formulas EQ.15 and EQ.16 as described above. Then, the pitch components of the directional rays 21, 27 can be calculated by combining the low-pass filtered gravity data and the high-pass filtered gyroscope data, for example using the formula EQ.14 as described above.

[0111] For the calculation of the yaw component, the head sensor data is used together with the yaw component of the wrist-worn IMU to calculate the yaw components of the directional rays 21, 27, as shown in step 903B. The head sensor data is used to correct the drift in the yaw component calculated from the wrist-worn IMU 104.

[0112] In step 904, the calculated yaw component and the calculated pitch component are combined to obtain the directional rays 21, 27, for example according to the formula EQ.1 as described above.

[0113] In step 905, collisions with the obtained directional rays can be detected. Such collisions can provide context information for calculating successive or subsequent directional rays and / or the position, location, and / or orientation of the successive or subsequent directional rays. Additionally, such collisions can be intersections between the directional rays 21, 27 and the surrounding environment (such as interactive elements, such as user interface buttons, switches, and sliders, or graspable objects, etc.), or a portion of these intersections. Collisions with the directional rays 21, 27 can be displayed and / or visualized in extended reality, which is, for example, virtual reality, augmented reality, or mixed reality.

[0114] The controller 103 can be configured to perform the above steps, including steps 903A and 903B. Further, the controller 103 can be configured to provide the calculated directional rays 21, 27 to at least one of a second device, a head device, and a computing device via a communication interface. The directional rays can be provided as a data stream (such as including a bit stream or data packet), and such a directional ray data stream can be provided to the head device, such as a head-mounted device including a display.

[0115] Figure 5 A system 300 is shown that can support at least some embodiments of the present invention. Unless otherwise stated, the system is the same as system 100.

[0116] In Figure 5 a, an exemplary schematic diagram of the system 300 is shown. From Figure 5As can be seen, system 300 includes a wrist-worn IMU 304, a head sensor 306, and a controller 303. The controller 303 is shown as a dashed line in the figure. As shown, the controller 303 may include, for example, preprocessing modules 350, 370, and a model 330 in the controller memory at least. The controller 303 may include an event interpreter and / or classifier 380. The system 300 may be configured to perform event interpretation and user interface command generation in the system 300.

[0117] Sensors 304 and 306 are respectively configured to transmit sensor data streams, such as sensor data streams 314 and 316. The sensor data streams may be received by the controller 303. The sensor data streams 314 and 316 may be preprocessed, and the directional light rays may be calculated at least in part based on the sensor data streams 314 and 316 and using the model 330. Then the calculated directional light rays are provided and / or applied to the scene information 361. The calculated directional light rays may be provided as a data stream 310. The context information 312 (such as scene-based context information and / or gesture-related information) may be used to update the directional light rays 310 to correspond to possible user actions and / or user intentions, such as the position of a selected plane in three-dimensional space, or the desired pointing position and / or selection position obtained simultaneously. This information may be obtained indirectly from the event interpreter or classifier 380 through the scene 361, or directly. The context logic and the scene information 361 may be provided to the model 330 as a data stream 312 from the scene within the head device 360.

[0118] The scene information 361, such as those disclosed for the system 300, may include interactive elements, such as user interface buttons, user interface switches, or user interface sliders, etc. The scene information may include one or more previous positions, orientations, and / or locations of the directional light rays, and may also be related to the user's viewpoint and / or gaze. Inside the controller and / or using the controller 303, the received and preprocessed sensor data streams are directed to at least one model, such as the model 330.

[0119] A controller, such as controller 103 or controller 303, may include a neural network. The neural network may be, for example, a feedforward neural network, a convolutional neural network, a recurrent neural network, or a graph neural network. The neural network may include a classifier and / or regression analysis. The neural network may apply a supervised learning algorithm. In supervised learning, input samples with known outputs are used, and the network learns from them and makes generalizations. Alternatively, unsupervised learning or reinforcement learning algorithms may also be used to build the model. In some embodiments, the neural network is trained such that certain signal features correspond to certain user action features (such as the user's hand trajectory).

[0120] The model may include at least one of algorithms, heuristics, and / or mathematical models. For example, model 330 may include an algorithm for calculating orientation that uses the wrist-worn IMU data stream 314 and the head sensor data stream 316 and is included in the directional ray calculation. The wrist-worn IMU may provide triaxial accelerometer, gyroscope, and magnetometer data. The directional ray may be further provided to a classifier or event interpreter 380, where the directional ray may be classified according to the scene information 361. Such a classifier 380 may be, for example, a ballistic / modified phase classifier.

[0121] The model (such as model 330) may be implemented to improve the ray casting, the orientation of the directional ray 21, and the direction of the directional ray 21 by recognizing or classifying the user's gestures. The model, device, or system may output at least one confidence value, preferably a series of confidence values, at least in part based on the recognition and / or classification. At least one sensor data stream (such as an optical sensor data stream and / or an IMU data stream) may be imported into at least one model. Such a model may be a machine learning model, for example, including at least one neural network, such as a feedforward, convolutional, recurrent, or graph neural network. As an addition or alternative, the model may also include at least one of supervised, unsupervised, or reinforcement learning algorithms. Feature extraction from the input data may be performed using a feature extraction module, which may include a neural network, such as a convolutional neural network. For example, a long short-term memory (LSTM) recurrent neural network (RNN) may be used to recognize user actions, such as a pinching action to select an interactive element. The system may also incorporate a feature extraction module configured to analyze the sensor data stream and recognize additional user actions other than the selection gesture. For example, the additional user action may be a gesture for pointing, such as pointing with a different finger, such as the index finger or the middle finger. As an alternative or addition, the additional user action may also include rolling of the wrist or arm.

[0122] The sensitivity of the gesture recognition model using a neural network may be further adjusted according to the user's previous selections and / or pointing history. Such user-specific adjustment of the gesture recognition model may be done, for example, by adjusting certain early layers and / or late layers of the neural network. The user's previous selection history (including at least one selection made by the user) may be stored in the device, such as a head-mounted device or a wrist-worn device, or, for example, in a cloud storage system. Such a system may be configured to adjust the sensitivity of the machine learning model according to the user's previous history (such as the selection history).

[0123] For example, the model may include at least one convolutional neural network (CNN) that performs inference on input data (such as signals received from sensors and / or preprocessed data). Convolution can be performed in the spatial or temporal dimension. Features (calculated from the sensor data fed into the CNN) can be selected algorithmically or manually. The model may also include an RNN (recurrent neural network), which can be used in combination with the neural network to support the recognition of user action features based on sensor data reflecting user activities.

[0124] According to the present disclosure, the training of the model can be performed, for example, using a labeled dataset containing multimodal motion data from multiple subjects. Synthetic data can be used to augment and expand the dataset. Depending on the model construction technique employed, the sequence of computational operations constituting the model can be derived through backpropagation, Markov decision processes, Monte Carlo methods, or other statistical methods. Model construction may involve dimensionality reduction and clustering techniques.

[0125] In addition, context logic can also be applied to improve the orientation and direction of the directional light 21. Such context logic can be based on the received context information, such as context information 312. The context information may include position information (such as coordinates) of user interactive elements (such as sliders, buttons, etc.), for example, position information implemented in augmented reality, mixed reality, extended reality, or virtual reality. As an addition or alternative, the context information may also include the proximity of the light endpoint calculated by the device providing the scene to at least one user interface element. In other words, the calculation of the directional light 21 can be at least partially based on the context information. For disambiguation of context selection, confidence values can be used. Thus, adaptive thresholds for interactive elements or other context information can be generated in combination with the context information. In addition, the sensitivity of gesture recognition can be adjusted according to the context information, where the adjustment may include adjusting the threshold of the confidence value. For example, if the user interface element is close to the light endpoint, the threshold for detecting a pinch can be reduced from, for example, 90% to 50%.

[0126] In addition, scene information (e.g., information obtained from a head-mounted device including a display unit) can also be incorporated into gesture recognition and / or directional light calculation. In other words, context information can be combined with information regarding the confidence level of determining a gesture. For example, if the confidence level of a gesture is considered high enough to be recognized as a gesture, but there is no interactive element suitable for the gesture in the scene information where the directional light is located, then such a gesture will be rejected in this scene. Context information containing gesture-related information can be incorporated into directional light calculation. For example, if a gesture appears or is considered a specific gesture with a high probability, the orientation, position, and / or movement of the directional light can be adjusted according to the scene information. For example, such scene information can be gesture-specific interactive elements. The system can be configured to adjust the detection threshold of point interaction based on the endpoints of the directional lights 21, 27, where the adjustment is based on the context logic received from the scene. For example, point interaction should be understood as the user pointing at something and can include the intersection or interaction between an element (such as an interactive element) and the directional lights 21, 27. In addition, the sensitivity of gesture recognition can be adjusted according to the context information, where the adjustment can include adjusting the threshold of the confidence value. The adjustment can include quickly moving the directional light to the interactive element.

[0127] Context information (e.g., context information 312) can include interactive elements present in the scene, such as user interface sliders, switches, and buttons, such as those in the scene information in extended reality. Information regarding selection interaction and point interaction can be utilized to improve the orientation, direction, and movement of the directional light 21. For example, when the directional light 21 directly points to or approaches an "interactive element that can be pinched", the pinch inference threshold can be adjusted. Similarly, the movement characteristics of the directional light 21 can also be changed relative to the interactive element, for example, according to the proximity to the interactive element. Such movement characteristics can be, for example, the speed of the directional light 21, or the inertia or momentum experienced by the user of the directional light 21, which can be characterized or experienced by the user as the "stickiness" or "gravity" of the directional light 21. In addition, the sensitivity of gesture recognition can be adjusted according to the trajectory and / or speed of the directional light, where the adjustment can include adjusting the threshold of the confidence value. For example, if the trajectory is atypical, indicating, for example, uncontrolled movement of the user, the threshold for detecting a pinch can be increased from, for example, 90% to 97%.

[0128] At least one user action output by the event interpreter 380 is received by the scene in the device 360. The controller can be configured to generate user interface instructions according to the user action and at least partially based on the determined user action. The controller 303 can be configured to transmit the user interface command to a device such as the device 360, and / or store the user interface command in the memory of the controller 303.

[0129] An operation example of system 300 including a head sensor 60, a wrist-worn inertial measurement unit 104, and a controller 103 is as follows. A user is using system 300, which includes a wrist-worn IMU 104 worn on the hand (where the hand can be, for example, the wrist, forearm, palm, or back of the hand, etc.) and a head sensor capable of measuring the head orientation. The user is looking at point H1, with their head facing towards point H1, and their arm (including the wrist-worn IMU 104) pointing downwards in the W1 direction. The user performs an action that includes: turning the head from point H1 to direction H2, raising the hand to a lateral position, extending the forearm, and pointing the index finger at point W2. The sensors of system 300 provide the following data during this action:

[0130] - The head sensor 60 provides a data stream 316 containing the orientation information of the user's head. Thus, the head sensor includes measuring the orientation change from point H1 to point H2 that occurs during the rotation of the user's head.

[0131] - The wrist-worn IMU 104 provides a gravity vector and a data stream 314, which reflects the orientation and movement of the user's wrist and / or hand, i.e., the hand trajectory of the user. Thus, the measurement of the wrist-worn IMU sensor includes measuring the change in the orientation of the wrist-worn IMU sensor from point W1 to point W2.

[0132] The data streams 316 and 314 are obtained by system 300, preferably included in what is obtained by the controller 303 in the system. The data stream 316 obtained by the system from the head sensor 60 is used to calculate the yaw component, which describes the position and orientation of the user's head relative to the wrist-worn IMU in the physical world coordinates, and thus can be used to correct the drift in the data stream of the wrist-worn IMU sensor. At the same time, the data stream 314 obtained by the device from the wrist-worn IMU 104 is used to calculate the normalized gravity vector, yaw component, and pitch component of the wrist-worn IMU 104 relative to the physical world coordinates. By combining the yaw component obtained from the wrist-worn IMU 104 with the yaw component of the head sensor 60, the yaw component of the directional ray 21 can be obtained from the data stream of the wrist-worn IMU 104 and drift correction can be performed. By combining the pitch component and the yaw component, the resulting directional ray can be used to interact with the interactive element, and the resulting directional ray 21 can be displayed to the user through a computer monitor or a computer display screen (for example, a head-mounted device worn by the user, which includes a display screen). By continuously and / or sequentially updating the directional ray by recalculating the pitch and yaw components, real-time or near-real-time input from the user and real-time or near-real-time visualization of the directional ray can be obtained. When the user moves their hand and thus moves the wrist-worn IMU 104, the yaw and pitch components of the directional ray 21 can be recalculated from the data streams of the head sensor 60 and the wrist-worn IMU 104.

[0133] Figure 6 shows system 700 that can support at least some embodiments of the present invention. System 700 may include a wrist-worn device (such as device 170), and / or a head-mounted device 160.

[0134] System 700 includes a controller 702. The controller includes at least one processor and at least one memory including computer program code and optional data. System 700 may further include a communication unit or interface. For example, such a unit may include a wireless and / or wired transceiver. System 700 may also include sensors, such as sensors 703, 704, 705, which are operatively connected to the controller. The sensors may include an IMU. System 700 may also include Figure 6 other elements not shown in the figure.

[0135] Although system 700 is described as including one processor, system 700 may include more processors. In one embodiment, the memory is capable of storing instructions, such as at least one of an operating system, various applications, models, neural networks, and / or preprocessing sequences. In addition, the memory may also include storage space for storing at least some of the information and data used in the disclosed embodiments.

[0136] Furthermore, the processor is capable of executing the stored instructions. In one embodiment, the processor may be embodied as a multi-core processor, a single-core processor, or a combination of one or more multi-core processors and one or more single-core processors. For example, the processor may be embodied as one or more of various processor devices, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing circuit with or without a DSP, or various other processor devices, including integrated circuits, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a microcontroller unit (MCU), a hardware accelerator, a dedicated computer chip, etc. In one embodiment, the processor may be configured to execute hard-coded functions. In one embodiment, the processor is embodied as an executor of software instructions, where the instructions may be specifically configured such that when executed, the processor can execute at least one of the models, sequences, algorithms, and / or operations described herein.

[0137] The memory may be embodied as one or more volatile storage devices, one or more non-volatile storage devices, and / or a combination of one or more volatile storage devices and non-volatile storage devices. For example, the memory may be embodied as a semiconductor memory (such as mask ROM, PROM, EPROM, flash ROM, RAM, etc.).

[0138] At least one memory and the computer program code may be configured to, together with at least one processor, cause system 700 to perform at least the following operations:

[0139] - Receive data from the wrist - worn IMU 104, the data including gravity information,

[0140] - Receive data from at least one head sensor 60, the sensor being configured to make measurements on the user 20, the data including, for example, the orientation of the user's head,

[0141] - Calculate the yaw component of the directional light rays 21, 27 based on the gravity information and at least partially based on the data received from the head sensor 60,

[0142] - Calculate the pitch component of the directional light rays 21, 27, wherein the calculation of the pitch is at least partially based on the gravity information, and

[0143] - Calculate the directional light rays 21, 27, wherein the directional light rays 21, 27 are based on a combination of the calculated yaw component and the calculated pitch component.

[0144] At least one memory and computer program code are configurable to, in conjunction with at least one processor, cause the system 700 to perform at least the following operations:

[0145] - Receive data from the wrist - worn IMU 104, the data including gravity information,

[0146] - Receive data from at least one head sensor 60, the sensor being configured to make measurements on the user 20, in particular the orientation of the user's head,

[0147] - Calculate a normalized gravity vector based at least partially on the gravity information,

[0148] - Calculate the yaw component of the directional light rays 21, 27 based on the normalized gravity vector and at least partially based on the data received from the head sensor 60,

[0149] - Calculate the pitch component of the directional light rays 21, 27, wherein the calculation of the pitch is at least partially based on the normalized gravity vector, and

[0150] - Calculate the directional light rays 21, 27, wherein the directional light rays 21, 27 are based on a combination of the calculated yaw component and the calculated pitch component.

[0151] At least one memory and computer program code are configurable to, in conjunction with at least one processor, cause the system 700 to perform at least the following operations:

[0152] - Receive data from the wrist - worn IMU 104, the data including gravity information,

[0153] - Receive data from at least one head sensor 60, which is configured to measure a user 20, in particular the orientation of the user's head.

[0154] - Calculate the yaw component of the directed light rays 21, 27 based on gravity information and at least partially based on the data received from the head sensor 60.

[0155] - Calculate the pitch component of the directed light rays 21, 27, where the pitch component is at least partially based on gravity information and the pitch component is calculated by a Madgwick filter that is configured to prioritize angular velocity correctness over orientation correctness.

[0156] - Calculate the directed light rays 21, 27, where the directed light rays 21, 27 are based on a combination of the calculated yaw component and the calculated pitch component.

[0157] A system, such as system 700, can be configured to wirelessly transmit data (such as a directed light ray data stream, user actions, and / or user interface frames) to another device, for example, using Bluetooth. The recipient of such data transmission can include at least one of the following: a connected wearable device, any wearable device, a smartphone.

[0158] The present disclosure is reflected in the following clauses.

[0159] Clause 1, a non-transitory computer-readable medium having a set of computer-readable instructions stored thereon, which when executed by at least one processor can cause a system to at least:

[0160] Use at least one wrist-worn inertial measurement unit and at least one head sensor to measure concurrent signals corresponding to at least the movement, position, or orientation of a user.

[0161] Receive the measured concurrent signals in a controller including at least one processor and at least one memory containing computer program code.

[0162] Combine the received concurrent signals by the controller.

[0163] Generate a directed light ray data stream by the controller based at least in part on the characteristics of the combination of the received signals.

[0164] Clause 2, a system including a wrist-worn device, the device including:

[0165] A strap

[0166] A wrist-worn inertial measurement unit 104

[0167] A controller, comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to, together with the at least one processor, cause control of at least:

[0168] - Receive concurrent signals from a head sensor and a wrist-worn inertial measurement unit 104,

[0169] - Combine the received concurrent signals,

[0170] - Generate an oriented light data stream based at least in part on characteristics of the combination of the received signals.

[0171] Clause 3, A method for calculating an oriented light, the method comprising:

[0172] Receiving data from at least one wrist-worn IMU 104, the data including gravity information,

[0173] Receiving data from at least one head sensor 60, the sensor being configured to measure the orientation of a user 20, in particular the user's head orientation,

[0174] Calculating the yaw component of the oriented lights 21, 27 based on the received gravity information and at least in part on the data received from the head sensor 60,

[0175] Calculating the pitch component of the oriented lights 21, 27, wherein the pitch is calculated based at least in part on the received gravity information, and

[0176] Calculating the oriented lights 21, 27 based on a combination of the calculated yaw component and the calculated pitch component.

[0177] Clause 4, A computer program configured to be able to execute the method according to Clause 3, wherein the program is stored on a non-transitory computer-readable medium.

[0178] Advantages of the present disclosure include the following. The responsiveness of the oriented light 21 is improved, particularly in embodiments where context logic is used to improve the light direction. By providing a more user-responsive light, embodiments of the present disclosure achieve the technical effect of providing a longer usage duration before battery or user fatigue due to power, accuracy, ergonomics, comfort, and / or cognitive factors. In other words, at least some embodiments provide a high number of task operations and / or user actions achievable per unit of time, thereby providing effective human-machine interaction.

[0179] The disclosed embodiments provide a technical solution to solve technical problems. One technical problem to be solved is the on-board calculation of improved directional light, which takes into account the application state and provides a high degree of availability, for example, through heterogeneity. In practice, this has been a problem because wrist-worn devices must be very lightweight and not overly bulky, which limits the power storage and on-board processing capabilities of such devices. However, off-board processing introduces latency to the system, which may result in poor response of the directional light.

[0180] Embodiments of the present invention overcome these limitations by calculating the yaw of the light using gravity information and head orientation information received from a second device. In addition, the pitch angle is calculated based on the gravity information and gyroscope data. In this way, the directional light can be calculated and provided more accurately and robustly. There are several benefits to doing so. First, a highly responsive directional light can be calculated and provided to the XR system in use. Second, the light can be provided at least partially by the wrist-worn device, so the user does not need additional equipment to interact with the XR system. Third, the wrist-worn device can be configured to further detect gestures made by the user. These embodiments can also bring other technical improvements and solve other technical problems.

[0181] It should be understood that the embodiments disclosed in the present invention are not limited to the specific structures, process steps, or materials described herein, but may extend to equivalents recognized by those of ordinary skill in the art. It should also be understood that the terms used herein are only for describing specific embodiments and are not intended to be limiting.

[0182] As used herein, "an embodiment" or "embodiments" means that the specific features, structures, or characteristics described in connection with the embodiment are included in at least one embodiment of the present invention. Thus, the phrases "in an embodiment" or "in embodiments" that appear in many places in this document do not necessarily all refer to the same embodiment. When a particular numerical value is mentioned using, for example, "about" or "substantially", the exact numerical value is also disclosed.

[0183] In this document, multiple objects, structural members, components, and / or materials may be presented in a common list for convenience. However, these lists should be considered such that each element in the list can be regarded independently as a separate and distinct element. Thus, the elements in such a list should not be considered as actual equivalents of any other element in the same list merely based on the fact that they appear in a common group without conflicting statements. Additionally, the various embodiments and examples of the present invention may also relate to alternatives of the various components. It should be understood that these embodiments, examples, and alternatives should not be considered as actual equivalents of each other, but rather as separate and autonomous manifestations of the present invention.

[0184] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this document, numerous specific details (such as examples of length, width, shape, etc.) are presented to facilitate an understanding of the embodiments of the present invention. However, those skilled in the art should understand that the present invention may be practiced without one or more of the specific details, or with other methods, components, materials, etc. In addition, known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the present invention.

[0185] Although the above embodiments illustrate the principles of the present invention in one or more specific applications, those of ordinary skill in the art should understand that many modifications may be made to the form, use, and details of the implementation without departing from the principles and concepts of the present invention and without the need for creative effort. Therefore, the present invention is intended to be limited only by the appended claims.

[0186] The verbs "comprise" and "include" are used herein as open-ended limitations, neither excluding nor necessarily requiring the presence of other unlisted features. Unless otherwise specified, the features recited in the dependent claims can be freely combined with each other. In addition, it should be understood that the use of "a" and "an" (i.e., the singular form) herein does not exclude the plural form.

[0187] Industrial Applicability

[0188] At least some embodiments of the present invention are industrially applicable to providing a user interface for a controllable device (such as a personal computer), for example, a user interface related to XR.

[0189] Glossary

[0190] ASIC Application-Specific Integrated Circuit AR Augmented Reality

[0191] CNN Convolutional Neural Network

[0192] DOF Degrees of Freedom

[0193] DSP Digital Signal Processor

[0194] EPROM Erasable PROM

[0195] EQ Equation

[0196] FPGA Field-Programmable Gate Array HID Human Interface Device

[0197] HMD Head-Mounted Device

[0198] HMI Human-Machine Interface

[0199] IIR Infinite Impulse Response

[0200] IMU Inertial Measurement Unit

[0201] LRA Linear Resonant Actuator

[0202] LSTM Long Short-Term Memory

[0203] MCU Microcontroller Unit

[0204] MEMS micro-electro-mechanical systems Mixed Reality

[0205] NN Neural Network

[0206] PCB Printed Circuit Board

[0207] PROMProgrammable ROM

[0208] RNN Recurrent Neural Network

[0209] RAM Random Access Memory ROM Read Only Memory

[0210] UI User Interface

[0211] VR Virtual Reality

[0212] XR Extended Reality Reference Numbers List

[0213]

[0214]

Claims

1. A system (100, 101), comprising: A wrist-worn device (170) comprising: a mounting component (105); a controller (103) comprising a processing core (701) and at least one memory (707) comprising computer program code; and a wrist-worn IMU (104) configured to measure a user (20), Wherein, the system configuration is: - receiving data from the wrist-worn IMU (104), the data including gravity information, - receiving data from at least one head sensor (60) configured to measure a user (20), the data including, for example, the user's head position, - calculating a yaw component of the directional light (21, 27) based on the gravity information and at least partly based on data received from the head sensor (60), - calculating a pitch component of the directional light (21, 27), wherein the pitch component is calculated at least in part based on the gravity information, and - calculating the directional ray (21, 27), wherein the directional ray (21, 27) is based on a combination of the calculated yaw component and the calculated pitch component.

2. The system (100) according to claim 1, wherein: The system configuration is: calculating a normalized gravity vector based at least in part on the gravity information, calculating a yaw component of the directional light (21, 27) based on the normalized gravity vector and at least in part based on data received from the head sensor (60), A pitch component of the directional light (21, 27) is calculated based at least in part on the calculated normalized gravity vector.

3. The system (100) of claim 1, wherein: The system configuration is: The pitch component of the directional light is calculated using a Madgwick filter configured to favor angular velocity correctness over orientation correctness.

4. The system (100) according to any one of the preceding claims, wherein: The system also includes a head unit (160) including a mounting component, a display unit, and a head unit controller (163), At least one of the head device controller (163) or the wrist-worn device controller (103) is configured to receive wrist-worn IMU data (114) and head sensor data (116), and calculate the yaw component, pitch component and directional light (21, 27).

5. The system (100) according to any one of the preceding claims, wherein: At least one of the head device controller (163) or the wrist mounted device controller (103) is configured to provide a data stream representing the directional light (210), for example to a head device (160) and / or a computing device.

6. The system according to claim 4 or 5, wherein: The head device (160) includes a head sensor (60, 61) and is configured to display the calculated directional light (21, 27) using the display unit.

7. A system according to any one of the preceding claims, wherein: The computation of the directional rays (21, 27) is further based on context information (312) received, for example, from a scene (360).

8. A system according to any one of the preceding claims, wherein: The system is further configured as follows: receiving optical sensor data from the wrist-worn device, and identifying a selection gesture based on at least one of the received optical data or the received IMU data, wherein the identifying is performed at least in part by a machine learning model, such as a neural network, wherein the system comprises the machine learning model, The selection gesture includes, for example, pinching.

9. The system according to claim 8, wherein: The system is further configured as follows: Gesture recognition is enabled or disabled based on the classification of the hand trajectory, where the classification of the hand trajectory is done, for example, by a ballistic / corrected phase classifier.

10. A system according to any one of the preceding claims, wherein: The system is further configured to calculate a series of confidence values ​​for each selection gesture and output the confidence values.

11. A system according to any one of the preceding claims, wherein: The system is further configured to adjust the sensitivity of the gesture recognition based on the received context information (312), the context information (312) for example including position information of the interactive element and / or proximity information relative to the endpoint of the directional light and the position of the interactive element, wherein adjusting the sensitivity for example includes adjusting a threshold of a confidence value used in the gesture recognition.

12. A system according to any one of the preceding claims, wherein: The system is further configured to adjust the sensitivity of the gesture recognition according to the trajectory of the directional light (21, 27), wherein adjusting the sensitivity includes, for example, adjusting a threshold of a confidence value used in the gesture recognition.

13. A system according to any one of the preceding claims, wherein: The system (399) is further configured to direct the directional light (21, 27) toward an origin, wherein the coordinates of the origin are determined by the orientation of the user's head or gaze (POV center), for example, the origin is equal to the center of the user's field of view.

14. A system according to any one of the preceding claims, wherein: The system is further configured to adjust a detection threshold of at least one point interaction based on an endpoint of the directional ray (21, 27), wherein the adjustment is based on contextual logic received from the scene.

15. A system according to any one of the preceding claims, wherein: The system is further configured to improve the accuracy of the directional light (21, 27) by implementing a filter to reduce interference caused by measurement errors of the user's pointing gesture.

16. A system according to any one of the preceding claims, wherein: The system is further configured to adjust the sensitivity of the machine learning model based on a user's previous selection history, such as a selection history.

17. A system according to any one of the preceding claims, wherein: The system also includes a feature extraction module configured to analyze the sensor data stream and identify additional user actions other than the selection gesture.

18. A system according to any one of the preceding claims, wherein: The directional light presents a curved curve at least between the interactive element and the wrist-worn IMU (104).

19. A system according to any one of the preceding claims, wherein: The elbow position of the arm is estimated and the directional light is aligned with the estimated elbow position and the position of the wrist-worn IMU (104).

20. A system according to any one of the preceding claims, wherein: The orientation of the directional light is corrected by eye tracking information.

21. A system according to any one of the preceding claims, wherein: The system configuration is: If the directional light ray (21, 27) is substantially parallel or anti-parallel relative to the normalized gravity vector, a yaw component of the directional light ray (21, 27) is recalculated based at least in part on the normalized gravity vector and at least in part on data obtained from the at least one wrist-worn IMU (104).

22. A method for calculating directional light, the method comprising: receiving data from at least one wrist-worn IMU (104), the data including gravity information, receiving data from at least one head sensor (60), the sensor being configured to measure a user (20), in particular a head position of the user, calculating a yaw component of the directional light (21, 27) based on the gravity information and at least in part on data received from the head sensor (60), calculating a pitch component of the directional light (21, 27), wherein the pitch is calculated at least in part based on the received gravity information, and A directional ray (21, 27) is calculated, wherein the directional ray (21, 27) is based on a combination of the calculated yaw component and the calculated pitch component.

23. The method according to claim 22, wherein: The method further comprises: calculating a normalized gravity vector based at least in part on the gravity information, calculating a yaw component of the directional light (21, 27) based on the normalized gravity vector and at least in part based on data received from the head sensor (60), and A pitch component of the directional light (21, 27) is calculated based at least in part on the calculated normalized gravity vector.

24. The method according to claim 22, wherein: The method further comprises: The pitch component of the directional light is calculated using a Madgwick filter configured to favor angular velocity correctness over orientation correctness.

25. A non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, enable a system to at least: receiving data from at least one wrist-worn IMU (104), the data including gravity information, and receiving data from at least one head sensor (60), the sensor being configured to measure a user (20), in particular a head position of the user, calculating a yaw component of the directional light (21, 27) based at least in part on the received gravity information and at least in part on data received from the head sensor (60), calculating a pitch component of the directional light (21, 27), wherein the pitch is calculated at least in part based on the received gravity information, and A directional ray (21, 27) is calculated, wherein the directional ray (21, 27) is based on a combination of the calculated yaw component and the calculated pitch component.

Images