Monocular hand tracking

Abstract

The invention relates to monocular hand tracking. Various implementations use one or more images from a single camera and registered hand data to predict a three-dimensional (3D) position and configuration of a hand. The registration data may provide information about the 3D hand shape and size (e.g., the fixation / actual distance between the joints). Such information may facilitate determining the depth and other 3D characteristics of the 3D position and configuration using images from a single camera (i.e., using 2D images from a single camera and not requiring triangulation using real-time images from multiple cameras or multiple viewpoints). The 3D position and configuration of the hand may be represented in a manner (e.g., format) suitable for Bayesian optimization. Some implementations may use (e.g., fit) a predetermined hand shape / size with captured image data to predict the 3D position and configuration of the hand.

Description

Technical Field

[0001] This disclosure relates in its entirety to detecting and interpreting user activities involving hand gestures and other movements during the use of electronic devices. Background Technology

[0002] Existing user interaction systems can be improved in terms of efficiently and effectively facilitating interaction based on user activities, including finger and other hand gestures and movements. Summary of the Invention

[0003] The various specific embodiments disclosed herein include devices, systems, and methods for predicting the three-dimensional (3D) position and configuration of a hand using images from a single camera and registered hand data. The registered data provides information about the shape and size of the 3D hand (e.g., fixed / actual distances between joints). The registered data facilitates the determination of depth and other 3D properties of the 3D position and configuration using only images from a single camera (e.g., using 2D images from a single camera and eliminating the need for triangulation using images from multiple cameras or multiple viewpoints).

[0004] Registration data can provide a model of the hand, for example, joints connected by bones, where the bones have a size determined based on the user's actual hand detected during registration, and the joints can be parameterized based on image data (e.g., with specified rotations, positions, etc.). The 3D position and configuration of the hand can be represented in a Bayesian-optimized manner (e.g., using a model format) (e.g., C2 continuous, linear (or spherically linear) for each component, for example, using pose encoding that uses 35 angles for hand joints, 4 floating-point numbers for wrist rotation (e.g., quaternions), and 3 floating-point numbers for wrist translation (e.g., xyz values)).

[0005] In some implementations, a hand model with a skeletal size determined from previous user registration is used in a two-step process: (a) first, using image data from a single camera (e.g., 2D images) to determine the current configuration and rotation of the hand model (i.e., a posed hand model), and (b) then fitting the posed hand model (in that current configuration / rotation) to the image data from the single camera (e.g., 2D images) to determine the 3D position of the hand. This process may involve predicting hand joint information (e.g., rotation angles) and / or information about the rotation of the entire hand, wrist joint, etc. This provides the current configuration and rotation of the hand model (i.e., the posed hand model), which is then used to determine the 3D position of the hand. This process can provide information about the distance of the hand from the camera (e.g., the 3D position of the wrist), which may be more uncertain in a single-camera scene (e.g., involving 2D images) compared to a scenario where real-time images from multiple cameras can be used for triangulation-based depth determination. Some specific implementations may utilize filtering (e.g., Kalman state) to smooth and / or process scenarios, especially when camera-based predictions deviate significantly from those expected from motion tracking (e.g., using past predictions to predict motion).

[0006] In some implementations, the processor executes the method by executing instructions stored on a computer-readable medium. The method involves acquiring real-time data comprising one or more images (e.g., 2D images) from a single camera of an electronic device, depicting a user's hand at one or more points in time. For example, this could involve an outward-facing camera on a head-mounted device (HMD) or extended reality (XR) glasses capturing one or more images. The process could involve a real-time video stream of images captured by the camera, for example, during an XR experience. One or more images may be cropped to focus on the portion depicting the hand.

[0007] This method involves obtaining registration data corresponding to hand size, wherein the registration data is based on sensor data about the user's hand portion acquired prior to the acquisition of real-time data. The registration data may include information about hand size / dimension, which can provide fixed / actual distances between joints. The registration data may be determined based on a registration process in which multiple cameras and / or depth sensors (e.g., more sensors than a single camera used to capture real-time data, such as 2D images) are used to generate a 3D model of the user's hand.

[0008] The method also involves determining parameters for hand representation (e.g., a posed hand model) based on real-time and registered data, wherein the parameters include rotational parameter values ​​corresponding to hand joints and hand pose parameter values ​​corresponding to hand pose / rotation. The rotational parameter values ​​corresponding to hand joints may, for example, provide 35 angles for the hand joints of the hand model. The hand pose parameters corresponding to hand pose / rotation can provide data representing the 3D position and orientation of the hand, such as four floating-point numbers (e.g., quaternions) for wrist rotation and three floating-point numbers (e.g., xyz values) for wrist translation.

[0009] This method may involve determining the 3D position and configuration of the hand based on parameters of the hand representation. This may involve fitting a posed hand model to image data from a single camera (e.g., matching keypoints) to determine the 3D position of the hand.

[0010] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and components for performing or causing to perform any of the methods described herein. Attached Figure Description

[0011] To enable those skilled in the art to understand this disclosure, more detailed descriptions can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.

[0012] Figure 1 Exemplary electronic devices operating in a physical environment according to some specific implementations are illustrated.

[0013] Figure 2 Examples are given of users performing direct interactions within certain specific implementations. Figure 1 A view of virtual elements within a 3D physical environment provided by the device.

[0014] Figure 3 Examples are provided via equipment according to some specific implementations. Figure 1 A view of virtual elements within a 3D physical environment, where the user performs indirect interactions.

[0015] Figure 4An exemplary hand model according to some specific implementations is illustrated.

[0016] Figure 5 An exemplary process is illustrated according to some specific implementations for determining hand / wrist position by fitting real-time image data from a single camera with registered hand data.

[0017] Figure 6 An exemplary process for determining the 3D position and configuration of a hand, according to some specific implementations, is illustrated.

[0018] Figures 7A-7C Examples are given of how motion prediction is used in some specific implementations to determine whether and how hand position and configuration should be adjusted.

[0019] Figures 8A-8D Examples are given of how motion prediction is used in some specific implementations to determine whether and when to adjust hand position and configuration.

[0020] Figure 9 This is a flowchart illustrating a method for determining the 3D position and configuration of a hand according to some specific implementations.

[0021] Figure 10 It is a block diagram based on some specific implementations of electronic devices.

[0022] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily scaled up or down. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0023] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0024] Figure 1 An exemplary electronic device 105 operating in physical environment 100 is illustrated. Figure 1In the example, physical environment 100 is a room including table 120. Electronic device 105 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about physical environment 100 and objects within it (e.g., images, sounds, lighting, characteristics, etc.) and information about user 102 of electronic device 105 and to evaluate the physical environment and objects within it. Information about physical environment 100 and / or user 102 may be used to provide visual and audio content and / or to identify the current location of physical environment 100 (e.g., including the location of objects such as table 120 in physical environment 100) and / or the location of the user within physical environment 100.

[0025] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., wearable devices such as HMDs) and / or 110 (e.g., handheld devices such as mobile devices, tablet computing devices, laptops, etc.). Such an XR environment may include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100, and a representation of user 102 generated based on camera images and / or depth camera images of user 102. Such an XR environment may include virtual content located at a 3D position relative to a 3D coordinate system (i.e., 3D space) associated with the XR environment, which may correspond to the 3D coordinate system of the physical environment 100.

[0026] In some embodiments, video (e.g., pass-through video depicting the physical environment) is received from the image sensor of a device (e.g., device 105 or device 110). In some embodiments, the 3D representation of the virtual environment is aligned with the 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment may be generated based on, for example, the scale of the physical environment or the positioning of open spaces, floors, walls, etc., such that the 3D representation is configured to align with corresponding features of the physical environment. In some embodiments, the viewpoint within the 3D coordinate system (e.g., that of user 102) may be determined based on the position of the electronic device within the physical environment. This viewpoint may be determined based on, for example, image data, depth sensor data, motion sensor data, etc., which may be retrieved via a virtual inertial odometry (VIO) system, a simultaneous localization and mapping (SLAM) system, etc.

[0027] In some implementations, a view of the XR environment is provided via an HMD (e.g., AR glasses) that includes a see-through display, in which virtual content is presented. For example, such a device may include one or more lenses and a presentation component (e.g., one or more light sources and / or one or more waveguides), through which a user 102 observes the physical environment 100, and the presentation component causes the virtual content to be viewed by the user within the view seen through the one or more lenses.

[0028] Figure 2 Examples of users performing direct interactions within it are shown. Figure 1 The XR environment is a view of virtual elements provided by the device within a 3D physical environment. In this example, user 102 makes hand gestures relative to content presented in views 210a to 210b of the XR environment provided by the device (e.g., device 105). Views 210a to 210b of the XR environment include an exemplary user interface 230 of an application (e.g., an example of virtual content) and a drawing 220 of a table 120 (i.e., an example of real content). As an example, in Figure 2 In this context, the user interface 230 is a two-dimensional virtual object (e.g., having a flat, front-facing surface). Providing such a view may involve determining the 3D properties of the physical environment 100 (e.g., the position of the table 120 in the physical environment 100, the size of the table 120, the size of the physical environment 100, etc.) and locating the virtual content (e.g., the user interface 230) in a 3D coordinate system corresponding to the physical environment 100.

[0029] exist Figure 2 In the example, user interface 230 includes various content items, including a background section 235 and icons 242, 244, 246, and 248. Icons 242, 244, 246, and 248 may be displayed on a flat user interface 230. User interface 230 may be the user interface of an application, as illustrated in this example. For illustrative purposes, user interface 230 is simplified, and user interfaces may in practice include any complexity, any number of content items, and / or combinations of 2D and / or 3D content. User interface 230 may be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.

[0030] In this example, the background portion 235 of the user interface 230 is flat. In this example, the background portion 235 includes all aspects (e.g., visual characteristics) of the user interface 230 being displayed, except for icons 242, 244, 246, and 248. Displaying the background portion of an operating system or application's user interface as a flat surface offers various advantages. For example, doing so provides easy understanding or additional ease of use within an XR environment to access the application's user interface. In some implementations, multiple user interfaces (e.g., corresponding to multiple different applications) are presented sequentially and / or simultaneously within an XR environment using one or more flat background portions, although other configurations are also possible.

[0031] In some implementations, the position and / or orientation of one or more user interfaces may be determined to facilitate visibility and / or usability. Within a 3D environment, one or more user interfaces may be in a fixed position and orientation. In such cases, user movement will not affect the position or orientation of the user interface within the 3D environment (e.g., causing the user interfaces to remain in their respective positions or orientations and not move relative to the user's viewpoint).

[0032] The position of the user interface within the 3D environment can be determined based on the distance between the user interface and the user (e.g., from the initial or current user position). The position and / or distance from the user can be determined based on various criteria, including but not limited to criteria that consider application type, application functionality, content type, content / text size, environment type, environment size, environment complexity, ambient lighting, the presence of other people in the environment, multiple users using the application or content, user preferences, user input, and other factors.

[0033] In some implementations, one or more user interfaces can be body-locked content, for example, having a certain distance and orientation offset relative to a part of the user's body (e.g., their torso). For example, the body-locked content of the user interface could be 0.5 meters to the left of the user's forward-facing vector and at a 45-degree angle to it. If the user's head rotates while the torso remains stationary, in a 3D environment, the body-locked user interface will appear to be stationary at 0.5 meters to the left of the torso's forward-facing vector and at a 45-degree angle to it. However, if the user does rotate their torso (e.g., by rotating in their chair), the body-locked user interface will follow the torso's rotation and reposition itself within the 3D environment such that it remains 0.5 meters to the left of the new forward-facing vector of their torso and at a 45-degree angle to it.

[0034] In other implementations, the user interface content is defined at a specific distance from the user, remaining stationary relative to the user's orientation (e.g., if initially displayed in a base orientation, it will remain in that base orientation regardless of any head or body movement). In this example, the orientation of the body-locked content is not referenced to any part of the user's body. In this different implementation, the body-locked user interface will not reposition itself based on torso rotation. For example, the body-locked user interface could be defined 2 meters away and initially displayed to the user's north based on the user's current facing direction. If the user rotates their torso 180 degrees to face south, the body-locked user interface will remain 2 meters north of the user, who is now directly behind them.

[0035] A body-locked user interface can also be configured to always remain aligned with the gravity line or the horizontal line, so that changes in head and / or body orientation during scrolling will not cause the body-locked user interface to move within the 3D environment. On the other hand, translational movement will reposition the body-locked content within the 3D environment to maintain distance offset.

[0036] exist Figure 2 In the example, user 102 moves their hand from an initial position, as illustrated by the position of hand drawing 222 in view 210a. The hand moves along path 250 to a later position, as illustrated by the position of drawing 222 in view 210b. As user 102 moves their hand along path 250, the fingers intersect with user interface 230. Specifically, as the fingers move along path 250, they virtually pierce icon 246 in user interface 230, and thus from the user's viewpoint, the fingertips (not shown) are optionally obscured by user interface 230 in view 210b.

[0037] The specific implementations disclosed herein track the 3D position and configuration of the hand over time for various reasons, including but not limited to interpreting user movement, such as... Figure 1 User 102 moves their hand / finger to provide direct interaction with virtual content, for example, moving their hand / finger along path 250 relative to user interface elements (such as icon 246 in user interface 230) to identify user input / interaction. The interpretation of user movement and other user activities can be based on identifying user intent using multiple potentially separate recognition processes corresponding to different input modalities. Using multiple potentially separate recognition processes for different modalities can improve functionality, accuracy, efficiency, and / or provide other device-based or interaction-based benefits.

[0038] exist Figure 2In the example, recognizing input might involve determining if the gesture is a direct interaction and then using a direct input recognition process to identify that gesture. For example, such a gesture could be interpreted as a tap input pointing to icon 246.

[0039] Note that the user's movement in the real world (e.g., physical environment 100) corresponds to movement within 3D space (e.g., an XR environment based on the real world and including virtual content such as a user interface positioned relative to a real-world object including the user). Thus, the user moves his hand in physical environment 100, for example, through empty space, but that hand (i.e., the drawing or representation of the hand) intersects with and / or pierces through the user interface 230 of the XR environment based on that physical environment. In this way, the user directly interacts with the virtual content virtually.

[0040] Figure 3 Examples provided by the device Figure 1 An exemplary view of virtual elements within a 3D physical environment, in which a user performs indirect interactions. In this example, user 102 makes hand gestures while viewing content presented in view 305 of the XR environment provided by a device (e.g., when user 102's gaze is directed toward the content). View 305 of the XR environment includes... Figure 2 An exemplary user interface 230. In Figure 3 In the example, user 102 makes a pinching gesture with their hand as illustrated in drawing 222 while gazing at user interface icon 246 along gaze direction path 310. In this example, the user activity (e.g., the pinching gesture along with the gaze pointing towards the UI element) corresponds to the user intent of interacting with user interface icon 246.

[0041] The specific implementations disclosed herein track the 3D position and configuration of the hand over time for various reasons, including but not limited to interpreting user activities such as user 102 forming a pinching hand gesture, pointing gesture, etc., to identify the user / interaction. For example, a pinching user activity can be interpreted as a tap input pointing to icon 246, such as selecting icon 246 in user interface 230, which causes the device to activate icon 246 and optionally display content corresponding to icon 246 (e.g., user interface).

[0042] Monocular hand pose detection

[0043] Some specific implementations disclosed herein involve monocular hand pose detection and / or tracking, for example, using 2D images from a single camera (i.e., using 2D images from a single camera and not requiring triangulation using images from multiple cameras or viewpoints) to determine and / or track the configuration, position, and / or orientation of the hand in 3D. Some implementations use limited information (e.g., RGB images from a single image sensor or RGB-D images from a single sensor) to determine and / or track hand configuration and / or pose. Hand configuration / pose detection can be accomplished by obtaining prior information about the user's hand (e.g., registration data) corresponding to the user's 3D hand shape and / or size, for example, connecting the size of the bones of the joints representing the 3D model of the hand. This can occur before the user experience (e.g., during the registration period).

[0044] As used herein, “registration” refers to any experience prior to the current experience during which current hand information is obtained. Registration provides information about the user’s hand that is not available during subsequent hand pose detection and / or tracking, such as information about hand size, shape, etc., and therefore provides useful information when hand pose detection and / or tracking can be performed with less information than would otherwise be required, for example, when such detection / tracking can be performed using 2D images from a single camera and / or when triangulation is not required from images from multiple cameras or multiple viewpoints. In one example, the user performs hand registration using a single device with multiple cameras. In another example, the user uses a single device (for both registration and the subsequent experience in which hand detection / tracking occurs), but obtains more information about the hand during registration than is available during the subsequent experience, for example, by using more cameras or other sensors, capturing data over a longer period of time, capturing data from more viewpoints, providing user input, etc.

[0045] The various specific implementations disclosed herein include devices, systems, and methods for predicting the 3D position and configuration of a hand using one or more images (e.g., 2D images) from a single camera and registered hand data. The registered data provides information about the shape and size of the 3D hand (e.g., fixed / actual distances between the joints of the hand model). The registered data facilitates the determination of depth and other 3D characteristics of the 3D position and configuration using images (e.g., 2D images) from only a single camera. During a hand tracking experience (e.g., after registration and during hand tracking), image data (e.g., from a single outward-facing camera on the HMD) is captured, and information identified in the images is used in conjunction with prior hand information to determine the 3D hand configuration and position.

[0046] Some implementations use captured image data to identify information about specific points on the hand model (e.g., the rotation angle of a specific hand point / joint) and / or about the rotation of the entire hand / wrist joint. This information can be fitted to previously registered data, and this fitting can help determine the distance between the hand and the camera, thereby better determining the hand's pose.

[0047] The 3D position and configuration of the hand can be represented in a Bayesian-optimized manner (e.g., format / model) (e.g., C2 continuous, linear (or spherically linear) for each component, for example, using pose coding that uses 35 angles for hand joints, 4 floating-point numbers for wrist rotation (e.g., quaternions), and 3 floating-point numbers for wrist translation (e.g., xyz values)).

[0048] Figure 4 An exemplary hand model 400 according to some specific implementation is illustrated. In this example, the hand is modeled using various joints 420a to 420w, 425, and 430. The joints may have different characteristics; for example, the first set of joints 420a to 420w may be encoded using angles representing hand joint angles, while wrist joints 425 and 430 may have 3D positions (e.g., xyz positions) associated with the overall position and / or orientation of the hand in the physical environment. Wrist joint 425 may represent wrist rotation (e.g., pointing downwards towards the arm). In this example, a root is present at the wrist, and the arm is capable of bending relative to the wrist as specified by the wrist joint.

[0049] Some specific implementations provide hand data in a format suitable for Bayesian optimization. For each component, this format can be C2 continuous, linear (or spherically linear). Such formats can be used to specify the following:

[0050]

[0051] In one exemplary implementation, the hand model provides hand pose encoding that includes 35 angles scaled between -1 and 1 for a minimum and maximum range, four floating-point values ​​(quaternions) for wrist rotation, and three floating-point values ​​(e.g., xyz) for wrist translation. This encoding allows for the calculation of velocity and acceleration for each item. The process can be configured such that uncertainties from the process (e.g., a neural network) are generated in the form of outputs that can be evaluated and / or combined. The process can be configured to provide a specific mathematical meaning of the "standard deviation," which facilitates the mathematically optimal combination of results.

[0052] In some implementations, the 3D position and configuration of the hand (e.g., finger positions, 3D hand shape, joint positions of a joint-based representation of the 3D shape of the hand, etc.) are determined first using image data (e.g., from a single real-time camera) to determine the hand joint rotations and / or hand rotation information of the hand model, such as the hand model size from registration data and the angles / rotations of a specific / current hand configuration from real-time image data. The determined hand joint / hand rotation information corresponding to the current configuration of the hand (i.e., a posed hand model) is then fitted to the image data, for example, by matching keypoints on the hand in that configuration with keypoints on the hand observed in the image data (e.g., from a single real-time camera). Such fitting provides information about depth and / or scale, such as how far the hand is from the camera position, and thus facilitates the determination of the 3D position of the hand and / or anchor points on the hand (e.g., wrist positioning).

[0053] Figure 5 An exemplary process for determining hand / wrist position based on fitting real-time image data from a single camera to registered hand data is illustrated. Process 500 involves combining pose angles (box 510) with wrist rotation and hand joint information (i.e., providing a posed hand model) and then fitting them to image data to determine hand / wrist position (box 520) (i.e., fitting the posed hand model to the current image data). Box 510 may involve determining wrist rotation values ​​(e.g., for joints corresponding to wrist rotation in the hand model) and / or pose angles (e.g., for joints corresponding to hand joint rotation in the hand model). For box 520, UVD points may be determined based on rays from the camera, for example, each UVD point having u, v, and depth uncertainties. In some specific implementations, a perspective n-point (PnP) process may be solved to provide rotation and position. If both rotation and position are solved, this may require iterative solving. However, it can be solved in a closed form, where the process is configured to solve for position instead of rotation. Process 500 generates an output that identifies the 3D position of the hand (e.g., the 3D position of the wrist).

[0054] Figure 6 An exemplary process 600 for determining the 3D position and configuration of a hand is illustrated. In this example, the image is captured by a camera (e.g., on an HMD), and at box 605, a portion of the image is identified as depicting the hand (e.g., cropped). The cropped portion of the image is provided to a single-frame inference block 610, and pose encoding and variance 615, wrist rotation and variance 620, and point and variance 625 are generated.

[0055] The pose encoding and variance 615 provide hand joint rotation values ​​for a hand model corresponding to a hand configuration (e.g., held flat, pointing one or more fingers, clenched, forming a fist, etc.). The pose encoding and variance 615 can be filtered by a pose Kalman filter 645. The pose encoding and variance 615 can be generated via an algorithm or machine learning model.

[0056] Wrist rotation and variance 620 can provide hand / wrist rotation values ​​for joints in a hand model corresponding to hand / wrist rotation (e.g., corresponding to the overall orientation of the hand). The wrist rotation and variance 620 information can be filtered by a rotated Kalman filter 645. Wrist rotation and variance 620 can be generated via an algorithm or machine learning model.

[0057] These processes may involve algorithms or other processes (e.g., machine learning processes) using a single cropped image as input (e.g., cropping around a hand). Outputs may include, but are not limited to: floating-point numbers (e.g., 35) of Euler angles (which represent different degrees of freedom in the hand pose), floating-point numbers of the standard deviation of each of these angles (e.g., 35), floating-point numbers of wrist rotation (quaternions) (e.g., 4), floating-point numbers of the standard deviation of the approximate direction to the ground truth rotation (e.g., 1), 3D floating-point numbers representing pixels and scaling depth (relative to the depth used to generate the crop) (e.g., 8); 3D floating-point numbers representing the standard deviation through those points (e.g., 8), bone length (e.g., 27); and / or the standard deviation of the bone length (e.g., 27). In some specific implementations, learning-based processes are used, such as one or more neural networks. Such processes may involve training using images and labels from a separate process that provides ground truth values ​​for the parameters (e.g., the parameters mentioned above). Training may involve a Gaussian loss that facilitates the prediction of each value and the uncertainty of each value. The loss can be expressed in units of standard deviation, so it can be simply added without additional hyperparameters. This process may involve a VIT transformer and / or an HrNet backbone.

[0058] The filtered pose encoding and variance 615 (potentially including covariance matrix 630) and the filtered wrist rotation and variance 620 (potentially including covariance matrix 635) can be combined with hand size and / or shape information represented in the hand model (e.g., from the registration process) to provide wrist rotation and pose angles for hand joint information, thereby providing a posed hand model. The posed hand model provides a 3D model with rotational and joint positions corresponding to both the user's hand shape / size and the user's current hand configuration and overall rotation. The posed hand model can then be used with the point and variance 625 (potentially including covariance matrix 640) at position Kalman filter block 665 and camera intrinsics 660 to determine 3D hand position and / or configuration, thereby providing a smooth 3D result. Kalman filters (e.g., pose Kalman filter 645, rotation Kalman filter 650, and position Kalman filter 665) can utilize time-varying information to ensure temporal smoothness in the 3D hand position and / or configuration results. Camera intrinsics can be used to project UVD points (pixels u, v in the clipping and depth relative to the estimated depth used to generate the clipping size) into Euclidean 3D space. Camera intrinsics can also be used to project the UVD standard deviation, which is rotated and projected along the ray from the camera to the point on the hand. When updating the Kalman state with new measurements, variance / covariance can be input and used. Typically, this is a constant based on what the input source is. Alternatively, the process can use predictions from each frame from the network, which allows it to weight the update amount based on confidence.

[0059] Over time, a series of 3D hand position and configuration information (e.g., position, velocity, acceleration, etc.) can be determined. This determination can be tracked and used to estimate user movements that occur over time, such as a user moving forward to point a finger and then retracting it. This determination can be used to predict subsequent user hand positions and configurations, and thus serves as a check on future 3D hand position and configuration determinations. This may involve implementing one or more adaptive Kalman filters to ensure smooth and confident tracking.

[0060] Figures 7A-7C An example is illustrated using motion prediction to determine whether and how hand position and configuration should be adjusted over time. Figure 7A In the diagram, hand position prediction 710a and hand prediction based on a motion model 720a are graphically illustrated. Hand position prediction 710a can be based on analyzing the current image of the hand, for example, via information about... Figure 5 The process described. Hand prediction based on a motion model 720a can predict future hand movements based on an analysis of previously determined hand position and / or configuration. This can be based on a motion model (e.g., a constant acceleration motion model).

[0061] Combined prediction 730a combines these results (e.g., via average, weighted average, etc.). Figure 7A The example illustrates a "controlled" result where the combined prediction 730a is within 2 standard deviations of the hand position prediction 710a. Since it is within the threshold (e.g., 2 standard deviations), the combined prediction 730a is used without correction.

[0062] exist Figure 7B and Figure 7C In the diagram, hand position prediction 710b and motion model-based hand prediction 720b are graphically illustrated. (Compared to...) Figure 7A compared to, Figure 7B An example of a "runaway" result is illustrated, where the combined prediction 730b is outside the threshold of the hand position prediction 710b (i.e., outside 2 standard deviations). This illustrates a result where the combined prediction 730b is inconsistent with either the hand position prediction 710a or the motion-based prediction 720b, and the variance indicates a relatively high confidence level in the result. To correct for this, the system can assume that the motion model (e.g., a constant acceleration motion model) does not represent the current state. It can increase the variance value (e.g., the value in the covariance matrix of the current state) to make the combined result more accurate (e.g., within 2 standard deviations of the prediction). Therefore, as... Figure 7C As illustrated, the corrected combined result 730c is adjusted to the second standard deviation of the hand position prediction 710b. This can function as a dead-zone filter, reducing / altering motion model uncertainty only when predictions are inconsistent. Such filtering can be employed using different degrees of freedom for wrist rotation / joint rotation and wrist position; for example, 1D for wrist rotation and joint rotation, and 3D for wrist position.

[0063] Figures 8A-8C A further example of using motion prediction to determine whether and when to adjust hand position and configuration is illustrated. This process can account for the fact that wrist covariance may be non-uniform. The process can account for cases where significant uncertainties exist, such as those concerning depth in the direction from the camera to the hand. Multiple observations can be combined to obtain a 3×3 covariance matrix of position (9×9 if velocity and acceleration are included). Direction can be additionally considered when determining uncertainties in the motion model. Figure 8A In the diagram, the uncertainty limit 810a of hand position prediction 820a and the uncertainty limit 830a of hand prediction based on the motion model are graphically illustrated. Since hand position prediction 820a is within the uncertainty limit 810a, the result is considered "controlled". Figure 8BIn the diagram, the uncertainty limit 810b of hand position prediction 820b and the uncertainty limit 830b of hand prediction based on motion models are graphically illustrated. Since hand position prediction 820b is not within the uncertainty limit 810b of hand position prediction 820b, the result is considered "out of control". Figure 8C An example of error correction using uniform scaling is provided, with an uncertainty limit of 840C. Figure 8D The example illustrates correction relative to the direction of the covariance matrix, providing an uncertainty limit of 840d. Uniform scaling is possible. The covariance matrix can be scaled individually along the direction of uncertainty.

[0064] Some embodiments disclosed herein use registered user data to provide information about a user's hand, which facilitates the subsequent determination of the 3D hand position and configuration using one or more images (e.g., 2D images) from a single camera. Given this additional user information provided by the registered user data, multiple cameras can be used, but may not be necessary. Some embodiments utilize registered bone lengths (for the hand model) to provide a hand shape with the correct size. The registered bone lengths can be used to predict rotations (e.g., predicting joint rotations in the hand model, with uncertainty). This avoids the need to predict bone lengths after registration; for example, bone lengths can be treated as predetermined constants. The registered bone lengths can be additionally or alternatively applied to the rotation output to determine local hand poses, such as the overall rotation / orientation of the user's hand / wrist. The registration information can additionally or alternatively provide useful modifications to rotation supervision during the supervision determination process, for example, enabling the use of simpler representations (e.g., 42-dimensional kinematic representations) and / or direct supervision of rotation via a target loss function in a machine learning process. Using hand registration data offers numerous benefits, including but not limited to enabling faster convergence, reducing uncertainty in rotational space, ensuring correct hand size, and / or ensuring constant bone length.

[0065] Some specific implementations disclosed herein fit, align, or otherwise use a posed hand model (e.g., determined based on registration data and current image data) to predict hand / joint positions. This may involve, for example, aligning components of the hand model (e.g., joints) with 3D keypoints to solve for wrist properties. For example, this may involve using total covariance to predict keypoints in 3D. Note that typical uncertainty will be greater in the "d" direction than in u, v. However, higher uncertainty ("d" direction error) may play a smaller role in hand center prediction. Alignment allows for the use of 3D predictions of keypoints to solve for wrist pose and / or transforming local hand poses into solved wrist poses. In some implementations, Bayesian loss is used to determine 3D hand position and configuration. This process may additionally utilize joint position loss. Highly uncertain joints (e.g., uncertainty above a threshold) can be ignored. The perspective n-point (PnP) process can be used to solve for the 3D position and rotation of an object given 3D keypoints. It may require iterative solving to obtain the optimal solution. An alternative is to use predictions (e.g., from an algorithm or machine learning model) to provide rotations and solve for position only, which can provide a closed-form solution that can be optimally solved for 3D position in a single step.

[0066] Figure 9 This is a flowchart illustrating an exemplary method for determining the 3D position and configuration of a hand. In some embodiments, a device such as electronic device 105 performs method 900. In some embodiments, method 900 is performed on a mobile device, desktop computer, laptop computer, HMD, or server device. Method 900 is performed by processing logic components, including hardware, firmware, software, or combinations thereof. In some embodiments, method 900 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., memory).

[0067] At box 902, method 900 relates to acquiring real-time data, including an image, from a camera of an electronic device depicting a user's hand at a given point in time. In some implementations, acquiring the real-time data includes capturing images or video from an outward-facing camera on an HMD or AR glasses. The real-time data can be a single image or a stream of images.

[0068] At box 904, method 900 involves obtaining registration data corresponding to the size of at least a portion of the hand, wherein the registration data is based on sensor data of the user's hand obtained prior to the acquisition of real-time data. The size of at least a portion of the hand can provide fixed distances between joints determined based on registration. The registration data includes fixed distances between hand joints representing the hand, which are determined based on sensor data about the user's hand portion obtained prior to the acquisition of real-time data. Such registration may involve a user experience in which multiple cameras and / or depth sensors (e.g., more sensors than during real-time capture) are used to generate a 3D model of the user's hand. For example, the registration data is determined based on registration where: multiple cameras simultaneously capture sensor data; or one or more cameras and depth sensors simultaneously capture sensor data.

[0069] The hand representation can have a format capable of Bayesian optimization using real-time and registered data. Rotation parameter values ​​can correspond to the angles of the hand joints. Hand pose parameter values ​​correspond to the 3D position and orientation of the hand. Hand pose parameter values ​​correspond to the wrist position and wrist rotation.

[0070] At box 906, method 900 involves determining parameters for hand representation based on real-time data and registered data, wherein the parameters include rotational parameter values ​​corresponding to hand joints and hand pose parameter values ​​corresponding to hand posture. The rotational parameter values ​​corresponding to hand joints may, for example, provide 35 angles for the hand joints of the hand model. The hand pose parameters corresponding to hand posture can provide data representing the 3D position and orientation of the hand, such as four floating-point numbers (e.g., quaternions) for wrist rotation and three floating-point numbers (e.g., xyz values) for wrist translation. Determining the parameters for hand representation may involve fitting real-time data and registered data. The 3D position of at least a portion of the hand may be determined based on such fitting.

[0071] At box 908, method 900 relates to determining the 3D position and configuration of the hand based on parameters of the hand representation. This may involve directly determining the 3D position of each joint of the joint-based representation of the hand, for example, the 3D position in a 3D coordinate system corresponding to the user's physical environment. Alternatively, this may involve determining the position of anchor points / reference points and the overall orientation associated with the hand (e.g., the position of a point on the wrist of the hand in a 3D coordinate system corresponding to the physical environment and data identifying the overall orientation of the hand relative to the anchor points / reference points in that 3D coordinate system), and determining the relative 3D position of the joints of the joint-based representation of the hand's 3D configuration relative to such anchor points / reference points and the overall hand orientation. In other words, the overall 3D position and orientation of the hand may be represented by a single hand pose (e.g., anchor point / reference point position and overall hand orientation). The configuration of the hand (e.g., flexion, finger positioning, etc.) can be represented using a hand representation (e.g., a joint-based representation) that specifies the relative positions of hand points (e.g., joints), and this hand representation can be localized and oriented using anchor points / reference points and the overall hand orientation in a 3D coordinate system of the user's environment. Once the positions of the anchor points and the overall orientation of the hand are known, and the current configuration of the hand, represented by the relative positioning of the joints in the hand representation, is known, the 3D positions of those joints in the 3D coordinate system of the user's physical environment can be determined. In some implementations, only the 3D position of the hand in the 3D coordinate system of the user's environment is determined. In some implementations, only the hand configuration, e.g., the 3D positions of the joints in the hand representation, is determined. Method 900 can determine the position and configuration of the hand using a single camera that captures an image of the user's hand (i.e., using a 2D image from a single camera, and without the need for triangulation using real-time images from multiple cameras or viewpoints). Method 900 may involve determining the 3D position and configuration of a hand over time based on image sequences from real-time data and registered data. Method 900 may involve smoothing the determined 3D position and configuration of the hand over time using one or more filters (e.g., Kalman state filters). Method 900 may involve adjusting the determined 3D position and configuration based on whether the determined 3D position and configuration of the hand over time deviates from a prediction based on motion tracking (e.g., exceeding a threshold). Method 900 may specifically handle scenarios where the prediction deviates significantly from those expected from motion tracking (e.g., as...). Figures 7A-7C and Figures 8A-8D The examples illustrate the different ways of handling "controlled" and "out of control".

[0072] While many of the examples described herein relate to wearable electronic devices, such as HMDs, other specific implementations are conceivable. The techniques disclosed herein may be implemented additionally or alternatively via other types of devices, including, but not limited to, mobile phones, tablets, laptops, and / or any other device having a camera capable of capturing one or more images of a hand.

[0073] Figure 10 This is a block diagram of electronic device 1000. Device 1000 illustrates an exemplary device configuration of electronic device 105 or 110. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure further relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 1000 includes one or more processing units 1002 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1006, one or more communication interfaces 1008 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, SPI, I2C and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1010, one or more output devices 1012 (e.g., including displays), one or more internal and / or external image sensor systems 1014, memory 1020, and one or more communication buses 1004 for interconnecting these components and various other components.

[0074] In some embodiments, one or more communication buses 1004 include circuitry for interconnecting system components and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 1006 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0075] In some embodiments, one or more output devices 1012 include one or more displays configured to present a view of a 3D environment to a user. In some embodiments, the one or more displays correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, the one or more displays correspond to waveguide displays such as diffraction, reflection, polarization, and holography. In one example, device 1000 includes a single display. In another example, device 1000 includes displays for each of the user's eyes.

[0076] In some embodiments, one or more output devices 1012 include one or more audio generating devices. In some embodiments, one or more output devices 1012 include one or more speakers, surround sound speakers, speaker arrays, or headphones for generating spatialized sound (e.g., 3D audio effects). Such devices can virtually place sound sources in a 3D environment, including behind, above, or below one or more listeners. Generating spatialized sound may involve transforming sound waves (e.g., using head-related transfer functions (HRTF), reverberation, or cancellation techniques) to simulate natural sound waves (including reflections from walls and floors) emanating from one or more points in the 3D environment. Spatialized sound can induce the listener's brain to interpret sound as occurring at one or more points in the 3D environment (e.g., from one or more specific sound sources), even if the actual sound may be generated by speakers in other locations. One or more output devices 1012 may additionally or alternatively be configured to generate haptic feedback.

[0077] In some embodiments, the one or more image sensor systems 1014 are configured to acquire image data corresponding to at least a portion of the physical environment. For example, the one or more image sensor systems 1014 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various embodiments, the one or more image sensor systems 1014 also include an illumination source emitting light, such as a flash. In various embodiments, the one or more image sensor systems 1014 also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0078] Memory 1020 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 1020 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1020 optionally includes one or more storage devices remotely located to one or more processing units 1002. Memory 1020 includes a non-transitory computer-readable storage medium.

[0079] In some embodiments, memory 1020 or a non-transitory computer-readable storage medium of memory 1020 stores an optional operating system 1030 and one or more instruction sets 1040. Operating system 1030 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 1040 includes executable software defined by binary information stored in charge. In some embodiments, instruction set 1040 is software executable by one or more processing units 1002 to implement one or more of the techniques described herein.

[0080] Instruction set 1040 includes user interaction instruction set 1042, which is configured to perform hand tracking as described herein (e.g., determining 3D hand position and configuration) upon execution. Instruction set 1040 may be embodied in a single software executable or multiple software executables.

[0081] Although instruction set 1040 is shown residing on a single device, it should be understood that in other embodiments, any combination of elements may reside in separate computing devices. Furthermore, the accompanying drawings serve more as a functional description of the various features present in a particular embodiment, and differ from the structural schematics of the embodiments described herein. As will be appreciated by those skilled in the art, individually shown items may be combined, and some items may be separate. The actual number of instruction sets and how features are allocated therein will vary depending on the specific embodiment and may depend in part on the particular combination of hardware, software, and / or firmware chosen for that particular embodiment.

[0082] It should be understood that the specific embodiments described above are cited by way of example, and the invention is not limited to what has been specifically shown and described above. Rather, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to those skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

[0083] As described above, one aspect of this technology involves collecting and using sensor data, which may include user data, to improve the user experience of electronic devices. This disclosure envisions that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or preferences of a particular person. Such personal information data may include motion data, physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0084] This disclosure recognizes that the use of such personal information data in the present invention can benefit users. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control over electronic devices. Furthermore, this disclosure also anticipates other uses of personal information data that benefit users.

[0085] This disclosure further envisions that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and maintain privacy policies and measures that are recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate purposes. Furthermore, such collection should only be conducted after receiving informed consent from users. Additionally, such entities should take any necessary steps to safeguard and protect access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices.

[0086] Regardless of the foregoing, this disclosure also contemplates specific implementations allowing users to selectively block the use or access to personal information data. That is, this disclosure contemplates providing hardware or software components to prevent or block access to such personal information data. For example, with regard to a content delivery service tailored to a user, the technology of this invention can be configured to allow a user to choose to "join" or "opt out" of the collection of personal information data during service registration. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow the transmission of anonymous information for improving device functionality.

[0087] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not become inoperable due to the absence of all or part of such personal information data. For example, preferences or settings can be inferred based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with a user, other non-personal information available to the content delivery service, or publicly available information, thereby selecting content and delivering it to the user.

[0088] In some implementations, data is stored using a public / private key system that allows only the data's owner to decrypt the stored data. In other implementations, data may be stored anonymously (e.g., without identification and / or without personal information about the user, such as legal name, username, time, and location data). This prevents other users, hackers, or third parties from identifying the user associated with the stored data. In some implementations, users can access their stored data from a different user device than the one used to upload the stored data. In these cases, users may need to provide login credentials to access their stored data.

[0089] This document sets forth numerous specific details to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods, apparatus, or systems known to a person of ordinary skill have not been described in detail so as not to obscure the claimed subject matter.

[0090] Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as “processing,” “calculating,” “calculating,” “determining,” and “identifying” refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data representing physical electronic or magnetic quantities within the memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0091] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein may be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0092] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the examples above can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.

[0093] The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Furthermore, the use of "based on" implies openness and inclusivity, as processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be restrictive.

[0094] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0095] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the term “comprising,” as used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0096] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrase "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" can be interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.

[0097] The foregoing description and summary of the present invention should be understood as illustrative and exemplary in every respect, and not restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the illustrative specific embodiments, but also by the full extent permitted by patent law. It should be understood that the specific embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method, the method comprising: In electronic devices with processors: Real-time data, including images, are obtained from the camera of the electronic device, the images depicting the user's hand at a given point in time; Obtain registration data corresponding to the size of at least a portion of the hand, wherein the registration data is based on sensor data of the user's hand obtained prior to the acquisition of the real-time data; Based on the real-time data and the registration data, parameters for hand representation are determined, wherein the parameters include rotational parameter values ​​corresponding to hand joints and hand posture parameter values ​​corresponding to hand postures; and The three-dimensional (3D) position and configuration of the hand are determined based on the parameters represented by the hand.

2. The method of claim 1, wherein the registration data includes a fixed distance between the hand joints represented by the hand, the fixed distance being determined based on sensor data about the user's hand portion obtained prior to the acquisition of the real-time data.

3. The method according to claim 1, wherein the registration data is determined based on registration, in which: Multiple cameras simultaneously capture the sensor data; or One or more cameras and depth sensors simultaneously capture the sensor data.

4. The method of claim 1, wherein the hand representation has a format capable of Bayesian optimization using the real-time data and the registration data.

5. The method according to claim 1, wherein the rotation parameter value corresponds to the angle of the hand joint.

6. The method according to claim 1, wherein the hand posture parameter value corresponds to the 3D position and orientation of the hand.

7. The method of claim 6, wherein the hand posture parameter value corresponds to wrist position and wrist rotation.

8. The method of claim 1, wherein determining the parameters for the hand representation includes fitting the real-time data and the registration data.

9. The method of claim 8, wherein the 3D position of at least a portion of the hand is determined based on the fitting.

10. The method of claim 1, further comprising determining the 3D position and configuration of the hand over time based on the image sequence in the real-time data and the registration data.

11. The method of claim 10, further comprising using a filter to smooth the 3D position and configuration of the hand as determined over time.

12. The method of claim 10, further comprising adjusting the determined 3D position and configuration based on whether the 3D position and configuration determined by the hand over time deviates from a prediction based on motion tracking.

13. The method of claim 1, wherein the real-time data is obtained via an outward-facing sensor on a head-mounted device (HMD) or augmented reality (AR) glasses.

14. A system comprising: Non-transitory computer-readable storage medium; as well as One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations, the operations including: Real-time data, including images, are obtained from the camera of the electronic device, the images depicting the user's hand at a given point in time; Obtain registration data corresponding to the size of at least a portion of the hand, wherein the registration data is based on sensor data of the user's hand obtained prior to the acquisition of the real-time data; Based on the real-time data and the registration data, parameters for hand representation are determined, wherein the parameters include rotational parameter values ​​corresponding to hand joints and hand posture parameter values ​​corresponding to hand postures; and The three-dimensional (3D) position and configuration of the hand are determined based on the parameters represented by the hand.

15. The system of claim 14, wherein the registration data includes a fixed distance between the hand joints represented by the hand, the fixed distance being determined based on sensor data about the user's hand portion obtained prior to the acquisition of the real-time data.

16. The system of claim 14, wherein the registration data is determined based on registration, in which: Multiple cameras simultaneously capture the sensor data; or One or more cameras and depth sensors simultaneously capture the sensor data.

17. The system of claim 14, wherein the hand representation has a format capable of Bayesian optimization using the real-time data and the registration data.

18. The system of claim 14, wherein the rotation parameter value corresponds to the angle of the hand joint.

19. The system of claim 14, wherein the hand posture parameter values ​​correspond to the 3D position and orientation of the hand.

20. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations including: Real-time data, including images, are obtained from the camera of the electronic device, the images depicting the user's hand at a given point in time; Obtain registration data corresponding to the size of at least a portion of the hand, wherein the registration data is based on sensor data of the user's hand obtained prior to the acquisition of the real-time data; Based on the real-time data and the registration data, parameters for hand representation are determined, wherein the parameters include rotational parameter values ​​corresponding to hand joints and hand posture parameter values ​​corresponding to hand postures; and The three-dimensional (3D) position and configuration of the hand are determined based on the parameters represented by the hand.

Images