Neural ray casting input for a display device

Abstract

A display device may receive first positional data about a plurality of keypoints of at least a portion of a body in a first image frame. A display device may receive second positional data about the plurality of keypoints of at least the portion of the body in a second image frame. A display device may select a keypoint of the plurality of keypoints as a ray point based on the first positional data and the second positional data as inputs to a model. A display device may generate a ray based on the ray point.

Description

NEURAL RAY CASTING INPUT FOR A DISPLAYDEVICEBACKGROUND

[0001] Ray casting is a technique used to detect and interact with objects using the movement of a tracked body part such as a user’s hand. For example, a device may track a position and movement of a user’s hand, and, from a point on the hand may generate a ray into a space. When the ray interacts with an object, the device may trigger an action with respect to the object. Some conventional ray casting techniques may have low sensitivity and / or instability (e.g., jitters).SUMMARY

[0002] In some aspects, the techniques described herein relate to a method including: receiving first positional data about a plurality of keypoints of at least a portion of a body in a first image frame; receiving second positional data about the plurality of key points of at least the portion of the body in a second image frame; selecting, by a model, a key point of the plurality of key points as a ray point based on the first positional data and the second positional data as inputs to the model; and generating a ray based on the ray point.

[0003] In some aspects, the techniques described herein relate to a display device including: at least one processor; and anon-transitory computer readable medium storing executable instructions that cause the at least one processor to execute: receive first positional data about a plurality of keypoints of at least a portion of a body in a first image frame; receive second positional data about the plurality of key points of at least the portion of the body in a second image frame; select, by a model, a key point of the plurality of key points as a ray point using the first positional data and the second positional data as inputs to the model; and generate a ray based on the ray point.

[0004] In some aspects, the techniques described herein relate to a computer program product including executable instructions that cause one or more processors to execute operations, the operations including: receiving first positional data about a plurality of keypoints of at least a portion of a body in a first image frame; receiving second positional data about the plurality' of key points of at least the portion of thebody in a second image frame; selecting, by a model, a keypoint of the plurality' of key points as a ray point based on the first positional data and the second positional data as inputs to the model; and generating a ray based on the ray point.

[0005] The details of one or more implementations are set forth in the accompanying drayvings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 A illustrates a head-mounted display device for displaying a ray based on hand movement according to an aspect.

[0007] FIG. IB illustrates a model (e.g., a neural network, a convolution neural network) for predicting one or more ray points of a ray according to an aspect.

[0008] FIG. 1C illustrates an example of an input to the model according to an aspect.

[0009] FIG. 2A illustrates a graph depicting results of a conventional raycasting approach.

[0010] FIG. 2B illustrates a graph depicting results of a ray generated by the head-mounted display device according to an aspect.

[0011] FIG. 3 illustrates a flowchart depicting example operations of generating a ray by a display device according to an aspect.DETAILED DESCRIPTION

[0012] This disclosure relates to a display device that includes a neural ray casting engine configured to generate and move a ray (e.g., a virtual line or beam) based on position and movement of a tracked body part (e.g., a user’s hand, foot, head, other body part, or, in some examples, the full body of the user) that overcomes one or more technical problems of low sensitivity and / or instability (e.g.. jitters). For example, the neural ray casting engine uses a model (e g., artificial intelligence (Al) model, a neural netyvork, etc.) to determine which key points (e.g., a palm point, a knuckle point, etc.) on the tracked body part among a collection of detected keypoints to use to cast a ray. which can provide one or more technical benefits of improved stability (e.g., reduction or elimination of ray jitters) and / or increased sensitivity (e.g., reducing the amount of user motion to move the ray).

[0013] In further detail, the neural ray casting engine detects a plurality ofkeypoints about the tracked body part from image data captured by a camera system on the display device. The neural ray casting engine includes a model (e.g., a neural network) configured to predict (e.g., determine, select, identify, etc.) one or more ray points of the ray using the plurality of keypoints as inputs to the model. The ray point(s) may be one or more keypoints on the tracked body part, which are determined (e.g., estimated, predicted, etc.) by the model to provide a relatively stable ray (e.g., increase a stability metric or reduce a jitter metric). The neural ray casting engine uses the ray point(s) to cast the ray on a display of the display device. For example, the ray may extend from a first ray point (e.g., keypoint #1 on the tracked body part) to and through a second ray point (e.g., keypoint #2 on the tracked body part) and into the display of the display device. The neural ray casting engine may determine at least one of the first ray point or the second ray point for a current image frame using the key points on the tracked body part over time (e.g., key points from the current image frame, and keypoints from one or more previous image frames) such that the ray is relatively stable.

[0014] Instead of attempting to detect certain predetermined keypoints (e.g., a palm point and a knuckle point) in image data and then casting a ray from those predetermined keypoints, the model determines which keypoint(s) to use as at least one of the first ray point or the second ray point so that a ray is relatively stable. Therefore, instead of using the same predetermined key points for the ray points across image frames, the keypoint(s) that are selected for the ray points may change over time. For example, a tracked body part may be defined by a collection of keypoints, e.g., keypoint #0 to keypoint #20. The neural ray casting engine attempts to detect keypoints #0 to keypoint #20 from the image data and uses the model to select which keypoint(s) provides a stable ray. For example, from a collection of keypoints in a first image frame, the model may select a keypoint (e.g., keypoint #13) as a ray point for casting a ray, and, from a collection of keypoints in a second image frame, the model may select a different keypoint (e.g., keypoint #15) for the same ray point.

[0015] The neural ray casting engine includes a key point detector configured to detect keypoints of a tracked body part and generate positional data about the keypoints from image data captured by a camera system of the display device. The positional data may include the 3D positions (e.g.. 3D coordinates) of the keypoints. The positional data generated by the key point detector includes the 3D positions in a coordinate space (e.g., a world space). The neural ray casting engine includes afeature processing engine configured to convert the positional data in the coordinate space to positional data in a localized space. The localized space may be the space defined by the tracked body part. For example, for hand ray casting, the localized space may be a localized space of the hand. The model (e.g., the neural network) receives the positional data in the localized space for a current image frame and the positional data in the localized space for one or more previous image frames as inputs, which causes the model (e.g., the neural network) to predict at least one of the first ray point or the second ray point by selecting at least one of the key points as the ray point(s) such that a ray is generated with reduced jitter. These and other features are further explained with reference to the figures.

[0016] FIGS. 1A to 1C illustrate a display device 100 with a neural ray casting engine 135 configured to generate and move a ray 126 based on position and movement of at least a portion of a body (e.g., a tracked body part) of a user. In some examples, the ray 126 is generated from a point on the tracked body part. The tracked body part may include a user’s hand, foot, head, other body part, or, in some examples, the full body of the user. The ray 126 is used to interact with an object (e.g., a virtual object or a physical object). For example, movement of the tracked body part may move the ray 126, and when the ray 126 interacts (e.g., intersects) with an object, the neural ray casting engine 135 may detect an interaction event, which can trigger an action with respect to the object. In some examples, the ray 126 is used to interact with an object in an augmented reality (AR) or virtual reality (VR) environment. The neural ray casting engine 135 uses a neural network 120 to determine which keypoints 110 (e.g., a palm point, a knuckle point, etc.) on the tracked body part among a collection of detected keypoints 110 to use to cast a ray 126, which can provide one or more technical benefits of improved stability (e.g., reduction or elimination of ray jitters) and / or increased sensitivity' (e.g., reducing the amount of user motion to move the ray 126). In some examples, the neural network 120 may be referred to as a model.

[0017] The ray 126 may be a virtual beam that originates from the tracked body part and points to a location on the display device 100 (e.g., extends into the AR or VR environment). In some examples, the neural ray casting engine 135 displays the ray 126 as a laser pointer or a colored line to give the user feedback about where they are aiming. In some examples, at least a portion of the ray 126 is visible to the user. In some examples, the ray 126 is displayed as a marker on the display 115. Insome examples, at least a portion of the ray 126 is invisible to the user. By manipulating a position and orientation of the tracked body part, a user can direct the ray 126 towards an object to trigger a specific action (e.g., highlighting, selecting, manipulating, or activating the object, etc.). In some examples, users can perform various gestures with a tracked body part to activate different functionalities. For example, a pinch gesture (e.g., using a hand) can be used to grab an object, while a pointing gesture can be used to trigger an action. Although some examples in this disclosure relate to hand ray casting, the ray casting techniques discussed herein may be applied to one or more other parts of a user body, such as a user’s head, feet, other body part, or full body.

[0018] The neural ray casting engine 135 detects a plurality of keypoints 110 on the tracked body part from image data 106 captured by a camera system 104. The neural ray casting engine 135 includes a neural network 120 configured to predict one or more ray points 122 of the ray 126 using the plurality of key points 110 as inputs to the neural network 120. The ray point(s) 122 may be one or more keypoints 110 on the tracked body part, which are predicted (e.g.. determined, selected, etc.) by the neural network 120 to provide a relatively stable ray (e.g., increase a stability metric or reduce a jitter metric). The neural ray casting engine 135 uses the ray point(s) 122 to cast the ray 126 on a display 115 of the display device 100. For example, the ray 126 may extend from a first ray point 122-1 on the tracked body part to and through a second ray point 122-2 on the tracked body part and into a display 115 of the display device 100. The neural ray casting engine 135 may determine (e.g., predict, estimate, etc.) at least one of the first ray point 122-1 or the second ray point 122-2 for a current image frame (e.g., image frame 106-1) using the keypoints on the tracked body part over time (e.g., key points from the current image frame (e.g., image frame 106-1), and keypoints from one or more previous image frames (e.g., image frames 106-2, 106-3)) such that the ray 126 is relatively stable.

[0019] Some conventional ray casting approaches track and detect the same predetermined points (e.g.. palm point and knuckle point) of a user’s hand across image frames, and then use those points to cast (e.g., generate) a ray. However, these conventional ray casting approaches may display a ray that is relatively unstable (e.g., jittery, oscillates, flickers, and / or distorts) because one or more of those predetermined key points may be occluded. Instead of computing a ray between at least two predetermined ray points and then attempting to detect those predeterminedray points in image data, the neural ray casting engine 135 uses a neural network 120 (e.g., a convolutional neural network (CNN)) to determine at least one of the first ray point 122-1 or the second ray point 122-2 from a collection of keypoints 110 detected in the image data 106 (e.g., from a current image frame or a current image frame and one or more previously captured images) by selecting a keypoint 110 as a ray point 122 from a plurality of potential candidates (e g., key points 110) to provide a stable ray 126. The neural ray casting engine 135 may overcome one or more technical problems associated with ray instability and / or low sensitivity as compared with some conventional approaches.

[0020] In some examples, the neural network 120 determines which keypoint(s) 110 to use for at least one of the first ray point 122-1 or the second ray point 122-2 so that a ray 126, when displayed, is relatively stable (e.g., jitter is reduced or eliminated). Therefore, instead of using the same predetermined keypoints for the ray points across image frames, the keypoint(s) 110 that are used for the raypoints 122 may change over time. For example, a tracked body part may be defined by a collection of keypoints, e.g., keypoint #0 to keypoint #20. The neural ray casting engine 135 attempts to detect keypoints #0 to keypoint #20 from the image data 106 and uses the neural network 120 to select which keypoint(s) 110 provides a stable ray 126. For example, from a collection of key points 110 detected in an image frame, the neural network 120 may select keypoint #13 as a ray point 122. and, from a collection of keypoints 110 in a subsequent image frame, the neural network may select a different keypoint 110 (e.g., keypoint #15) if that different keypoint 110 provides a ray 126 with less jitters.

[0021] A ray 126 may be defined as a path (e.g., a straight line, a curved line, or a bent line) between two or more ray points 122. The ray points 122 may be two or more key points 110 in the image data 106 of the tracked body part. In some examples, as shown in FIG. IB, the ray 126 is defined as a line between a ray point 122-1 and a ray point 122-2. In some examples, the ray point 122-1 is referred to as an anchor point. In some examples, the ray point 122-1 is a point in which the ray 126 originates. In some examples, the ray point 122-2 is referred to as an extension point. In some examples, the location of the ray point 122-2 determines the direction of the ray 126. The ray 126 may start at the ray point 122-1 and extend to (and, in some examples, through) the ray point 122-2 to a location on a display of the display device 100. Although two ray points 122 are used to define a ray 126, the ray 126may be defined by more than two ray points such as a first ray point, a second ray point, and a third ray point. Instead of using predetermined keypoint(s) as the ray point 122-1 (e.g., the anchor point) and / or the ray point 122-2, the neural network 120 is configured (e.g., trained) to select a keypoint 110 for at least one of the ray point 122-1 or the ray point 122-2 such that the ray 126 is relatively stable.

[0022] The display device 100 may be any type of user device. In some examples, the display device 100 is an extended reality (XR) device. In some examples, the display device 100 is an AR device. In some examples, the display device 100 is a VR device. In some examples, the display device 100 is a wearable device such as a head-mounted display device or other type of augmented reality (AR) or virtual reality (VR) device. The display device 100 may include an optical head-mounted display (OHMD) device, a transparent heads-up display (HUD) device, an augmented reality (AR) device, or other devices such as goggles or headsets having sensors, display, and computing capabilities. The display device 100 may also be a laptop, desktop, smartphone, television device, or other type of user device.

[0023] The display device 100 includes a camera system 104 configured to generate image data 106 about a user’s surroundings. The camera system 104 may include one or more camera devices that capture image data 106 in a field of view of the user. In some examples, the camera system 104 is a visual see through (VST) camera system that allows the user to see the real world through the camera's lens while also seeing digital information (e g., virtual content) overlaid on the real world. In some examples, the camera device(s) may be referred to as an AR camera, a mixed reality camera, a head-mounted display camera, a transparent display camera, or a combiner camera. The camera system’s field of view may be the angular extent of the scene that is captured by the camera system 104. The field of view may be measured in degrees and may be specified as a horizontal field of view and / or a vertical field of view. The field of view may be determined by the focal length of the lens and the size of the devices of the camera system 104.

[0024] The display device 100 includes a key point detector 108 that detects keypoints 110 of a tracked body part (e g., a user’s hand, foot, head, other body part, or, in some examples, the full body of the user) from the image data 106 captured by the camera system 104. The key points 110 may represent articulations, j oints, or other distinctive features on the tracked body part. In the case of hand tracking, a keypoint 110 may be a particular point on a user’s hand, and the keypoints 110 maybe separate points on the hand. As shown in FIG. IB, the keypoints 110 include key point #0 to keypoint #20. where each keypoint 110 relates to a different part of the user’s hand. However, the key point detector 108 may detect any number of key points 110 from the image data 106, including any number less than twenty or any number greater than twenty. The key point detector 108 generates positional data 112 about the key points 110 in the image data 106. The key point detector 108 may execute a key point detector algorithm to generate the positional data 112 about the keypoints 110. In some examples, the key point detector algorithm includes a body part pose estimation algorithm. In some examples, the key point detector algorithm includes a hand pose estimation algorithm. In some examples, the key point detector algorithm includes a body pose estimation.

[0025] The positional data 112 may include three-dimensional (3D) locations (e.g., x, y, z coordinates) of the keypoints 110. In some examples, the positional data 112 includes two-dimensional (2D) locations (e.g., x, y coordinates) of the keypoints 110, and the key point detector 108 includes a 3D estimator (e.g., a machine-learning (ML) model, a neural network, etc.) that computes the 3D positions of the keypoints 1 10 using the 2D locations. In some examples, the positional data 112 identifies the 3D locations of keypoints 110 in a coordinate space 114. A 3D location may be the coordinates in the x-direction, y-direction, and z-direction. In some examples, the coordinate space 114 is a world space of the virtual environment. The world space may be a global reference frame for the virtual environment, and the hand coordinates are represented in relation to this space, indicating their absolute position within the VR environment.

[0026] The key point detector 108 may detect the positional data 112 about the keypoints 110 over time. For example, the key point detector 108 may generate positional data 112 about the key points 110 in two or more temporally spaced image frames. For example, the key point detector 108 may detect the positional data 112 about the keypoints 110 in a sequence of image frames. For example, as shown in FIG. IB. the key point detector 108 detects positional data 112-1 in an image frame 106-1, positional data 112-2 in an image frame 106-2, and positional data 112-3 in an image frame 106-3. The positional data 112-1 may include the 3D locations of the key points 110 (or a portion thereof) detected in the image frame 106-1.

[0027] The positional data 112-2 may include the 3D locations of the keypoints 110 (or a portion thereof) detected in the image frame 106-2. Thepositional data 112-3 may include the 3D locations of the key points 110 (or a portion thereof) detected in the image frame 106-3. In some examples, the image frame 106- 1, the image frame 106-2, and the image frame 106-3 are temporally adjacent frames in a video frame sequence. In some examples, the image frame 106-1, the image frame 106-2, and the image frame 106-3 are consecutive frames in a video frame sequence. In some examples, in a video frame sequence, one or more image frames are disposed between the image frame 106-1 and the image frame 106-2. and / or one or more image frames are disposed between the image frame 106-2 and the image frame 106-3.

[0028] In some examples, the positional data 112 includes one or more position vectors, where each position vector includes a concatenation of the 3D locations of at least a portion of the keypoints 110 in a respective image frame (e.g., [x-position, y-position, z-position] for each detected keypoint 110). For example, the positional data 112 may include a first position vector with the 3D locations of the key points 110 in a first image frame, a second position vector with the 3D locations of the keypoints 110 in a second image frame, a third position vector with the 3D locations of the key points 110 in a third image frame and so forth.

[0029] In some examples, the display device 100 includes a feature processing engine 116 that receives the positional data 112 in the coordinate space 114 and generates positional data 112a in a localized space 118. For example, the feature processing engine 1 16 may convert the 3D locations of the keypoints 1 1 from the coordinate space 114 to the localized space 118. The localized space 118 is a space (e.g., 3D space) that is smaller than the coordinate space 114. In some examples, the localized space 118 is a localized coordinate space such as a palm coordinate space (e.g., in the case of hand tracking), a foot coordinate space (e.g., in the case of foot tracking), or a body coordinate space (e.g., in the case of body tracking).

[0030] In some examples, the feature processing engine 116 may generate the positional data 112a in the localized space 118 by offsetting the 3D locations by a 3D location of a reference keypoint (e.g., one of the keypoints 110). For example, the positional data 112-1 for the key points 110 in an image frame 106-1 may include [1, 1, 1] for keypoint #0, [2, 2, 2] for keypoint #1, [3, 3, 3] for keypoint #2, [4, 2, 6] for keypoint #3, and so forth. The feature processing engine 116 may convert the positional data 112 to the localized space 118 by offsetting a 3D location by a 3D location of a reference key point (e.g., keypoint #0), which results in the followingpositional data 112a: [0, 0, 0] for keypoint #0, [1, 1, 1] for keypoint #1, [2, 2, 2] for keypoint #3, [3, 1, 5] for keypoint #3, and so forth.

[0031] The neural ray casting engine 135 includes a neural network 120 that receives the positional data 112a in the localized space 118 and selects one or more of the key points 110 as at least one ray point 122 such that the ray 126 is stabilized and / or ray jitters are reduced, minimized, or eliminated. In some examples, the ray point 122-1 is a predetermined keypoint 110 (e.g.. keypoint #0), and the neural network 120 selects one of the key points 110 as the ray point 122-2 such that a stability metric is increased (e.g., maximized) or a jitter metric is decreased (e.g., minimized).

[0032] In some examples, the ray point 122-2 is a predetermined keypoint 1 10, and the neural network 120 selects one of the keypoints 110 as the ray point 122- 1 such that a stability' metric is increased (e.g., maximized) or ajitter metric is decreased (e.g., minimized). In some examples, the neural network 120 selects one of the key points 110 as the ray point 122-1 and selects one of the key points 110 as the ray point 122-2 such that a stability metric is increased (e.g., maximized) or ajitter metric is decreased (e.g., minimized).

[0033] A stability metric may be a value that indicates a level of ray stability' associated with the ray 126. Ajitter metric may be a value that indicates a level of jitteriness associated with the ray 126. The neural network 120 is configured (e.g.. trained) to determine (e g., select) ray points 122 that increases (e.g., maximizes) the stability metric or decreases (e.g., minimizes) the jitter metric. In some examples, the neural network 120 includes a convolutional neural network. The neural network 120 may include an input layer, one or more convolutional layers, one or more pooling layers, and one or more activation layers.

[0034] The neural ray casting engine 135 includes a ray generator 124 configured to generate the ray 126 using the ray point(s) 122. The ray generator 124 may generate a ray 126, where the ray 126 is defined by the ray points 122. In some examples, the ray 126 is a virtual line that extends between the ray point 122-1 and the ray point 122-2. The ray generator 124 may display at least a portion of the ray 126 on a display of the display device 100.

[0035] The ray 126 may be a virtual beam that originates from the tracked body part and points to a location on the display device 100 (e.g.. extends into the AR or VR environment). In some examples, the ray generator 124 displays the ray 126 asa laser pointer or a colored line to give the user feedback about where they are aiming. In some examples, at least a portion of the ray 126 is visible to the user. In some examples, at least a portion of the ray 126 is invisible to the user. By manipulating a position and orientation of the tracked body part, a user can direct the ray 126 towards an object to trigger a specific action (e.g., highlighting, selecting, manipulating, or activating the object, etc.). In some examples, users can perform various gestures with a tracked body part (e.g., hands) to activate different functionalities.

[0036] In some examples, the ray generator 124 may use keypoint #0 as the ray point 122-1 (e.g., the ray point 122-1 has a 3D location that corresponds to key point 0), and the neural network 120 is used to select one of the key points 110 for the ray point 122-2 such that the ray 126 is stabilized and / or ray jitters are reduced, minimized, or eliminated. For example, instead of always using a predetermined key point 110 (e.g., key point #11) as the ray point 122-2, the neural network 120 selects a particular key point 110 from the plurality of keypoints 110 as the ray point 122-2, which may be key point #12 (e.g., the ray point 122-2 would have a 3D location that corresponds to key point #12), key point #13 (e.g., the ray point 122-2 would have a 3D location that corresponds to keypoint #13), or keypoint #8 (e.g., the ray point 122-2 would have a 3D location that corresponds to keypoint #8), etc., whichever the neural network 120 determines as providing a stable ray 126. As such, the key point 110 that is used for the ray point 122-2 may change over time (e.g., across image frames).

[0037] A ray point 122 (e.g., ray point 122-1, ray point 122-2) may be a coordinate of the ray 126, and the coordinate may be the 3D location of a key point 110. In some examples, the ray point 122 is a 3D location in the localized space 118. In some examples, in the case of hand ray casting, the ray 126 is a palm-locked ray (e.g., controllable by the hand of the user). The ray 126 is displayed by the display device 100 according to the ray points 122. As indicated above, the neural network 120 may select at least one of the keypoints 110 as the ray point 122. A ray 126 is defined as a path (e.g.. a virtual line, a straight line, a curved line, a bent line, etc.) between at least two points, e g., ray point 122-1 and ray point 122-3. The ray 126 may extend from the ray point 122-1 and to (and, in some examples, through) the ray point 122-1. In some examples, a ray 126 is referred to as a ray vector. In some examples, the neural network 120 predicts the ray point 122-2. In some examples, the neural network 120 determines the ray point 122-1. In some examples, the neuralnetwork 120 determines the ray point 122-1 and the ray point 122-2.

[0038] In some examples, the ray generator 124 uses a reference key point (e.g., one of the keypoints 110) that was used to convert the positional data 112 to the localized space 118 as the ray point 122-1, and the neural network 120 determines the ray point 122-2 by selecting one of the keypoints 110. For example, as indicated above, the feature processing engine 116 may generate the positional data 112a in the localized space 118 by offsetting the 3D locations by a 3D location of a reference keypoint (e.g., one of the keypoints 110).

[0039] The positional data 112-1 for the keypoints 110 in an image frame 106- 1 may include [1, 1, 1] for keypoint #0, [2, 2, 2] for keypoint #1, [3, 3, 3] for keypoint #2, [4, 2, 6] for keypoint #3, and so forth. If keypoint #0 is used as the reference keypoint, the feature processing engine 116 converts the positional data 112 to the localized space 118 by offsetting a 3D location of each keypoint 110 by a 3D location of the keypoint #0, which results in the following positional data 112a: [0, 0, 0] for keypoint #0, [1, 1, 1] for keypoint #1, [2, 2, 2] for keypoint #3, [3, 1, 5] for keypoint #3, and so forth. In this case, the ray generator 124 uses the 3D location of keypoint #0 (e.g., the reference keypoint) as the ray point 122-1, and the neural network 120 selects a keypoint 110 as the ray point 122-2.

[0040] The neural network 120 may use the positional data 112a (e g., first positional data) for a current image frame (e.g., image frame 106-1) and positional data 112a (e.g., second positional data) for one or more previous image frames (e.g., image frame 106-2 and / or image frame 106-2) as inputs to determine the ray point 122. In other w ords, as shown in FIG. 1C, the neural network 120 may receive the collection of 3D locations of keypoints 110 over a plurality of image frames as an input to determine one or more ray points 122 for a current image frame. In some examples, the previous image frame(s) include one image frame. In some examples, the previous image frame(s) include two image frames. In some examples, the previous image frame(s) include three image frames. In some examples, the previous image frame(s) include more than three image frames.

[0041] The neural network 120 may be configured (e.g., trained) using angular augmentation. For example, during configuring (e.g., training), each ray 126 is rotated by a list of angles to generalize in orientation space. Also, the ground truth ray may be generated by summarizing each session with a single ray 126 by determining a medium ray of body-locked rays (e.g., palm-locked, foot-locked, etc.),which may reduce or remove the sudden jittery' rays from gesture events.

[0042] The display device 100 may include one or more processors 101, one or more memory devices 103, and an operating system 105. In some examples, the neural ray casting engine 135 is included as part of the operating system 105. In some examples, the neural ray casting engine 135 is included as part of an application executable by the operating system 105. In some examples, the neural ray casting engine 135 may operate in conjunction with one or more applications executable by the operating system 105. The processor(s) 101 may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processor(s) 101 can be semiconductor-based - that is. the processors can include semiconductor material that can perform digital logic. The memory device(s) 103 may include any type of storage device that stores information in a format that can be read and / or executed by the processor(s) 101. In some examples, the memory device(s) 103 is / are a non-transitory computer-readable medium. The memory device(s) 103 may store executable instructions that when executed by the processor(s) 101 may execute the operations discussed with reference to the display device 100.

[0043] FIG. 2A illustrates results of generating a ray according to a conventional ray cast approach. In FIG. 2A, a dot 152 indicates the ground truth ray vector coordinates and dots 154 indicate a series of rays, generated by a conventional ray cast approach. As shown in FIG. 2A, the conventional ray cast approach may provide ray vectors that deviate away from the ground truth ray vector, and, in some examples, the variance on the estimates may be relatively high. FIG. 2B illustrates results of generating a ray according to the display device 100 of FIGS. 1A to 1C. In FIG. 2B, a dot 156 indicates the ground truth ray vector coordinates and dots 158 indicate a series of rays, generated by the display device 100 of FIGS. 1A to 1C. As shown in FIG. 2B, the bias may be relatively low7(e g., the dot 156 coincides with the general centroid of dots 158) and the variance may be also low (e.g. the dots 158 have less spread). As shown in FIG. 2B, the display device 100 of FIGS. 1A to 1C provides a technical effect of generating rays with reduced jitters and accuracy even with sporadic finger and hand movements.

[0044] FIG. 3 is a flowchart 300 depicting example operations of a system for generating a ray using a head-mounted display device. The flowchart 300 may depict operations of a computer-implemented method. Although the flowchart 300 isexplained with respect to the display device 100 of FIGS. 1A through 1C, the flowchart 300 may be applicable to any of the implementations discussed herein. Although the flowchart 300 of FIG. 3 illustrates the operations in sequential order, it will be appreciated that this is merely an example, and that additional or alternative operations may be included. Further, operations of FIG. 3 and related operations may be executed in a different order than that shown, or in a parallel or overlapping fashion.

[0045] Operation 302 includes receiving first positional data about a plurality of keypoints of at least a portion of a body in a first image frame. Operation 304 includes receiving second positional data about the plurality of key points of at least the portion of the body in a second image frame. Operation 306 includes selecting, by a model (e.g., a neural network), a keypoint of the plurality of keypoints as a ray point based on the first positional data and the second positional data as inputs to the neural network. Operation 308 includes generating, by a display device, a ray based on the ray point.

[0046] Clause 1. A method comprising: receiving first positional data about a plurality of keypoints of at least a portion of a body in a first image frame; receiving second positional data about the plurality7of key points of at least the portion of the body in a second image frame; selecting, by a model, a key point of the plurality of key points as a ray point based on the first positional data and the second positional data as inputs to the model; and generating a ray based on the ray point.

[0047] Clause 2. The method of clause 1, wherein the ray point is a first ray point and the keypoint is a first keypoint, the ray being defined as a line between the first ray point and a second ray point, the method further comprising: selecting, by the model, the second ray point based on the first positional data and the second positional data as inputs to the model.

[0048] Clause 3. The method of clause 1 or 2, wherein the keypoint is a first keypoint, the method further comprising: receiving third positional data about the plurality of keypoints of at least the portion of the body in a third image frame; and selecting, by the model, a second key point of the plurality of key points as the ray point based on the second positional data and the third positional data as inputs to the model, the second keypoint being different from the first keypoint.

[0049] Clause 4. The method of any one of clauses 1 to 3. wherein the first positional data includes three-dimensional (3D) locations of the plurality of keypointsin a coordinate space, the method comprising: converting the 3D locations in the coordinate space to 3D locations in a localized space.

[0050] Clause 5. The method of clause 4, wherein the ray point is a first ray point, the ray being defined as a line between the first ray point and a second ray point, wherein the converting includes offsetting the 3D locations in the coordinate space by a 3D location of a reference keypoint, the second ray point including the reference keypoint.

[0051] Clause 6. The method of any one of clauses 1 to 5, wherein the first positional data includes a first vector of coordinates in a localized space, and the second positional data includes a second vector of coordinates in the localized space, the method further comprising: inputting the first vector and the second vector to the model to select the key point of the plurality of keypoints as the ray point.

[0052] Clause 7. The method of any one of clauses 1 to 6, further comprising: receiving third positional data about the plurality' of keypoints of at least the portion of the body in a third image frame; and selecting, by the model, the key point of the plurality of keypoints as the ray point using the first positional data, the second positional data, and the third positional data as inputs to the model.

[0053] Clause 8. A display device comprising: at least one processor; and a non-transitory computer readable medium storing executable instructions that cause the at least one processor to execute: receive first positional data about a plurality- of keypoints of at least a portion of a body in a first image frame; receive second positional data about the plurality of key points of at least the portion of the body in a second image frame; select, by a model, a key point of the plurality of key points as a ray point using the first positional data and the second positional data as inputs to the model; and generate a ray based on the ray point.

[0054] Clause 9. The display device of clause 8, yvherein the ray point is a first ray point and the keypoint is a first keypoint, the ray being defined as a line between the first ray point and a second ray point, yvherein the executable instructions include instructions that cause the at least one processor to: select, by the model, the second ray point based on the first positional data and the second positional data as inputs to the model.

[0055] Clause 10. The display device of clause 8 or 9. wherein the keypoint is a first keypoint, wherein the executable instructions include instructions that cause the at least one processor to: receiving third positional data about the plurality ofkey points of at least the portion of the body in a third image frame; and selecting, by the model, a second keypoint of the plurality’ of key points as the ray point based on the second positional data and the third positional data as inputs to the model, the second keypoint being different from the first keypoint.

[0056] Clause 11. The display device of any one of clauses 8 to 10, wherein the first positional data includes three-dimensional (3D) locations of the plurality of key points in a coordinate space, wherein the executable instructions include instructions that cause the at least one processor to: convert the 3D locations in the coordinate space to 3D locations in a localized space.

[0057] Clause 12. The display device of clause 11, wherein the ray point is a first ray point, the ray being defined as a line between the first ray point and a second ray point, wherein the executable instructions include instructions that cause the at least one processor to: offset the 3D locations in the coordinate space by a 3D location of a reference keypoint, the second ray point includes the reference key point.

[0058] Clause 13. The display device of any one of clauses 9 to 12, wherein the first positional data includes a first vector of coordinates in a localized space, and the second positional data includes a second vector of coordinates in the localized space, wherein the executable instructions include instructions that cause the at least one processor to: input the first vector and the second vector to the model to select the key point of the plurality of key points as the ray point.

[0059] Clause 14. The display device of any one of clauses 8 to 13, wherein the executable instructions include instructions that cause the at least one processor to: receive third positional data about the plurality of keypoints of at least the portion of the body in a third image frame; and select, by the model, the key point of the plurality of keypoints as the ray point using the first positional data, the second positional data, and the third positional data as inputs to the model.

[0060] Clause 15. A computer program product comprising executable instructions that cause one or more processors to execute operations, the operations comprising: receiving first positional data about a plurality of keypoints of at least a portion of a body in a first image frame; receiving second positional data about the plurality of key points of at least the portion of the body in a second image frame; selecting, by a model, a key point of the plurality of key points as a ray point based on the first positional data and the second positional data as inputs to the model; and generating a ray based on the ray point.

[0061] Clause 16. The computer program product of clause 15, wherein the ray point is a first ray point and the keypoint is a first keypoint, the ray being defined as a line between the first ray point and a second ray point, the operations further comprising: selecting, by the model, the second ray point based on the first positional data and the second positional data as inputs to the model.

[0062] Clause 17. The computer program product of clause 15 or 16, wherein the keypoint is a first keypoint, the operations further comprising: receiving third positional data about the plurality of key points of at least the portion of the body in a third image frame; and selecting, by the model, a second keypoint of the plurality of keypoints as the ray point based on the second positional data and the third positional data as inputs to the model, the second keypoint being different from the first keypoint.

[0063] Clause 18. The computer program product of any one of clauses 15 to 17, wherein the first positional data includes three-dimensional (3D) locations of the plurality of keypoints in a coordinate space, the operations further comprising: converting the 3D locations in the coordinate space to 3D locations in a localized space.

[0064] Clause 19. The computer program product of clause 18, wherein the ray point is a first ray point, the ray being defined as a line between the first ray point and a second ray point, the operations further comprising: offsetting the 3D locations in the coordinate space by a 3D location of a reference keypoint, the second ray point including the reference keypoint.

[0065] Clause 20. The computer program product of any one of clauses 15 to 19, wherein the first positional data includes a first vector of coordinates in a localized space, and the second positional data includes a second vector of coordinates in the localized space, the operations further comprising: inputting the first vector and the second vector to the model to select the key point of the plurality' of key points as the ray point.

[0066] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmableprocessor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to. a storage system, at least one input device, and at least one output device.

[0067] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine- readable medium” '‘computer-readable medium” refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory', Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine- readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0068] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0069] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end. middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

[0070] The computing system can include clients and servers. A client andserver are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0071] In this specification and the appended claims, the singular forms "a," "an" and "the" do not exclude the plural reference unless the context clearly dictates otherwise. Further, conjunctions such as "‘and / ’ “or,” and “and / or” are inclusive unless the context clearly dictates otherwise. For example, “A and / or B” includes A alone, B alone, and A with B. Further, connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and / or physical or logical couplings between the various elements. Many alternative or additional functional relationships, physical connections or logical connections may be present in a practical device. Moreover, no item or component is essential to the practice of the implementations disclosed herein unless the element is specifically described as “essential” or “critical”.

[0072] Terms such as, but not limited to, approximately, substantially, generally, etc. are used herein to indicate that a precise value or range thereof is not required and need not be specified. As used herein, the terms discussed above will have ready and instant meaning to one of ordinary skill in the art.

[0073] Moreover, use of terms such as up. down, top, bottom, side, end, front, back, etc. herein are used with reference to a currently considered or illustrated orientation. If they are considered with respect to another orientation, it should be understood that such terms must be correspondingly modified.

[0074] Further, in this specification and the appended claims, the singular forms "a," "an" and "the" do not exclude the plural reference unless the context clearly dictates otherwise. Moreover, conjunctions such as “and,” “or,” and “and / or” are inclusive unless the context clearly dictates otherw ise. For example, “A and / or B” includes A alone, B alone, and A with B.

[0075] Although certain example methods, apparatuses and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. It is to be understood that terminology employed herein is for the purpose of describing particular aspects and is not intended to be limiting. On the contrary, this patent covers all methods, apparatus and articles of manufacture fairly falling within the scope of the claims of this patent.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: receiving first positional data about a plurality of keypoints of at least a portion of a body in a first image frame; receiving second positional data about the plurality of key points of at least the portion of the body in a second image frame; selecting, by a model, a key point of the plurality of key points as a ray point based on the first positional data and the second positional data as inputs to the model; and generating a ray based on the ray point.

2. The method of claim 1, wherein the ray point is a first ray point and the keypoint is a first keypoint, the ray being defined as a line between the first ray point and a second ray point, the method further comprising: selecting, by the model, the second ray point based on the first positional data and the second positional data as inputs to the model.

3. The method of claim 1 or 2, wherein the key point is a first key point, the method further comprising: receiving third positional data about the plurality of keypoints of at least the portion of the body in a third image frame; and selecting, by the model, a second keypoint of the plurality of key points as the ray point based on the second positional data and the third positional data as inputs to the model, the second keypoint being different from the first keypoint.

4. The method of any one of claims 1 to 3, wherein the first positional data includes three-dimensional (3D) locations of the plurality of keypoints in a coordinate space, the method comprising: converting the 3D locations in the coordinate space to 3D locations in a localized space.

5. The method of claim 4, wherein the ray point is a first ray point, the ray being defined as a line between the first ray point and a second ray point, wherein theconverting includes offsetting the 3D locations in the coordinate space by a 3D location of a reference key point, the second ray point including the reference keypoint.

6. The method of any one of claims 1 to 5, wherein the first positional data includes a first vector of coordinates in a localized space, and the second positional data includes a second vector of coordinates in the localized space, the method further comprising: inputting the first vector and the second vector to the model to select the key point of the plurality of key points as the ray point.

7. The method of any one of claims 1 to 6, further comprising: receiving third positional data about the plurality of keypoints of at least the portion of the body in a third image frame; and selecting, by the model, the keypoint of the plurality of keypoints as the ray point using the first positional data, the second positional data, and the third positional data as inputs to the model.

8. A display device comprising: at least one processor; and a non-transitory computer readable medium storing executable instructions that cause the at least one processor to execute: receive first positional data about a plurality of keypoints of at least a portion of a body in a first image frame; receive second positional data about the plurality of keypoints of at least the portion of the body in a second image frame; select, by a model, a keypoint of the plurality of key points as a ray point using the first positional data and the second positional data as inputs to the model; and generate a ray based on the ray point.

9. The display device of claim 8, wherein the ray point is a first ray point and the keypoint is a first keypoint, the ray being defined as a line between the first ray pointand a second ray point, wherein the executable instructions include instructions that cause the at least one processor to: select, by the model, the second ray point based on the first positional data and the second positional data as inputs to the model.

10. The display device of claim 8 or 9. wherein the keypoint is a first keypoint, wherein the executable instructions include instructions that cause the at least one processor to: receiving third positional data about the plurality of keypoints of at least the portion of the body in a third image frame; and selecting, by the model, a second keypoint of the plurality of key points as the ray point based on the second positional data and the third positional data as inputs to the model, the second keypoint being different from the first keypoint.

11. The display device of any one of claims 8 to 10, wherein the first positional data includes three-dimensional (3D) locations of the plurality of key points in a coordinate space, wherein the executable instructions include instructions that cause the at least one processor to: convert the 3D locations in the coordinate space to 3D locations in a localized space.

12. The display device of claim 11, wherein the ray point is a first ray point, the ray being defined as a line between the first ray point and a second ray point, wherein the executable instructions include instructions that cause the at least one processor to: offset the 3D locations in the coordinate space by a 3D location of a reference keypoint, the second ray point includes the reference keypoint.

13. The display device of any one of claims 9 to 12, wherein the first positional data includes a first vector of coordinates in a localized space, and the second positional data includes a second vector of coordinates in the localized space, wherein the executable instructions include instructions that cause the at least one processor to: input the first vector and the second vector to the model to select the keypoint of the plurality of keypoints as the ray point.

14. The display device of any one of claims 8 to 13, wherein the executable instructions include instructions that cause the at least one processor to: receive third positional data about the plurality of keypoints of at least the portion of the body in a third image frame; and select, by the model, the keypoint of the plurality' of key points as the ray point using the first positional data, the second positional data, and the third positional data as inputs to the model.

15. A computer program product comprising executable instructions that cause one or more processors to execute operations, the operations comprising: receiving first positional data about a plurality’ of keypoints of at least a portion of a body in a first image frame; receiving second positional data about the plurality of keypoints of at least the portion of the body in a second image frame; selecting, by a model, a key point of the plurality of key points as a ray point based on the first positional data and the second positional data as inputs to the model; and generating a ray based on the ray point.

16. The computer program product of claim 15, wherein the ray point is a first ray point and the keypoint is a first keypoint, the ray being defined as a line between the first ray point and a second ray point, the operations further comprising: selecting, by the model, the second ray point based on the first positional data and the second positional data as inputs to the model.

17. The computer program product of claim 15 or 16, wherein the keypoint is a first keypoint, the operations further comprising: receiving third positional data about the plurality’ of key points of at least the portion of the body in a third image frame; and selecting, by the model, a second keypoint of the plurality of key points as the ray point based on the second positional data and the third positional data as inputs to the model, the second keypoint being different from the first keypoint.

18. The computer program product of any one of claims 15 to 17, wherein the first positional data includes three-dimensional (3D) locations of the plurality of keypoints in a coordinate space, the operations further comprising: converting the 3D locations in the coordinate space to 3D locations in a localized space.

19. The computer program product of claim 18, wherein the ray point is a first ray point, the ray being defined as a line between the first ray point and a second ray point, the operations further comprising: offsetting the 3D locations in the coordinate space by a 3D location of a reference key point, the second ray point including the reference keypoint.

20. The computer program product of any one of claims 15 to 19, wherein the first positional data includes a first vector of coordinates in a localized space, and the second positional data includes a second vector of coordinates in the localized space, the operations further comprising: inputting the first vector and the second vector to the model to select the key point of the plurality of key points as the ray point.

Images