Gesture detection method and system with hand prediction

Abstract

The invention relates to an augmented reality system and method. In an exemplary embodiment, a two-dimensional hand image is captured. Two-dimensional keypoints are identified using a two-dimensional hand image. And mapping the two-dimensional key points to the three-dimensional key points. And performing hand prediction by using the three-dimensional key points. Other embodiments are also provided.

Description

Background of the Invention

[0002] The present invention relates to an extended reality system and method.

[0003] Over the past decade, Extended Reality (XR) devices, including Augmented Reality (AR) devices and Virtual Reality (VR) devices, have become increasingly popular. Important design considerations and challenges for XR devices include performance, cost, and power consumption. Due to various limitations, existing XR devices have been unable to meet the requirements, as explained below.

[0004] New and improved XR systems and methods thereof are desired. Summary of the invention

[0005] The present invention relates to an extended reality system and method. In an exemplary embodiment, a two-dimensional hand image is captured. Two-dimensional key points are identified using the two-dimensional hand image. The two-dimensional key points are mapped to three-dimensional key points. Hand prediction is performed using the three-dimensional key points. There are other embodiments.

[0006] A system of one or more computers may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which software, firmware, hardware, or a combination thereof causes the system to perform actions when run. One or more computer programs may be configured to perform specific operations or actions by including instructions, which, when executed by a data processing device, cause the device to perform actions. A general aspect includes a hand prediction method, the method including capturing multiple images containing at least a first hand, the multiple images being in a two-dimensional (2D) space, the multiple images including a current image and a previous image. The method also includes identifying multiple previous 2D key points using the previous image. The method also includes identifying multiple current 2D key points using the current image. The method also includes mapping multiple previous 2D key points to multiple previous three-dimensional (3D) key points. The method also includes mapping multiple current 2D key points to multiple current 3D key points. The method also includes generating multiple 3D predicted key points in a 3D space using multiple previous 3D key points and multiple current 3D key points. The method also includes mapping multiple 3D predicted key points to multiple predicted 2D key points. The method also includes identifying erroneous hand detections using the plurality of predicted 2D keypoints.Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0007] Implementations may include one or more of the following features. The method may include projecting a plurality of previous 2D key points into a 3D space. The plurality of images further include at least a second hand, the plurality of 3D predicted key points being associated with the first hand and the second hand. The method may include defining a bounding box enclosing the first hand in the current image, and tracking the bounding box using a plurality of predicted 2D key points. The bounding box is defined using an upper left corner position and a lower right corner position, the bounding box including at least ten percent of an edge area surrounding the first hand. The plurality of 3D predicted key points are assigned confidence values, the method may include using the confidence values ​​to detect non-hand objects. The method may include tracking the first hand using a plurality of predicted 2D key points. The method may include initiating a hand tracking process when the first hand is detected. The method may include calculating a coordinate change between a plurality of previous 3D key points and a plurality of current 3D key points, each 3D key point may include three coordinates. The method may include using a plurality of 3D vectors of a plurality of previous 3D key points and a plurality of current 3D key points. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer accessible medium.

[0008] A general aspect relates to an extended reality device, which includes a housing having a front and a back. The device also includes a first camera configured on the front, the first camera configured to capture multiple two-dimensional (2D) images at a predefined frame rate, the multiple 2D images including a current image and a previous image. The device also includes a display configured on the back of the housing. The device also includes a memory coupled to the first camera and configured to store multiple 2D images. The device also includes a processor coupled to the memory. In the device, the processor is configured to: use at least the current image and the previous image to identify multiple 2D key points associated with a hand, map the multiple 2D key points to multiple three-dimensional (3D) key points, and use at least multiple 3D key points to provide hand prediction. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0009] Implementations may include one or more of the following features. The processor may include a neural processing unit configured to detect a hand using a first image captured by a first camera. The device may include a second camera, the first camera being located on a left side of the housing and the second camera being located on a right side of the housing. The processor is further configured to track the hand. The processor is further configured to generate a plurality of predicted 3D key points, and map the plurality of predicted 3D key points to a plurality of predicted 2D key points. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0010] One general aspect relates to a hand tracking method, the method comprising capturing a first image. The method also comprises detecting at least a first hand in the first image. The method also comprises capturing a plurality of images containing at least the first hand, the plurality of images being in a two-dimensional (2D) space, the plurality of images comprising a current image and a previous image. The method also comprises using the plurality of images to identify a plurality of 2D key points. The method also comprises mapping the plurality of 2D key points to a plurality of three-dimensional (3D) key points. The method also comprises using the plurality of 3D key points to generate a plurality of 3D predicted key points in a 3D space. The method also comprises mapping the plurality of 3D predicted key points to a plurality of predicted 2D key points. The method also comprises using the plurality of 3D predicted key points to track the first hand. Other embodiments of this aspect comprise corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0011] Implementations may include one or more of the following features. The method may include calculating confidence values ​​for a plurality of 3D prediction key points, and identifying false hand detections using at least the confidence values. The method may include identifying changes between a first image and a second image. The method may include: performing a deep learning process for hand detection using the first image. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer accessible medium.

[0012] It is to be appreciated that embodiments of the present invention provide many advantages over conventional techniques. In particular, hand shape prediction techniques can achieve more accurate and efficient hand tracking and bounding boxes. In addition, hand shape prediction techniques according to embodiments of the present invention can be performed in conjunction with gesture recognition techniques.

[0013] Embodiments of the present invention can be implemented in combination with existing systems and methods. For example, the hand calibration techniques according to the present invention can be used in various XR systems, including XR devices equipped with ranging components. In addition, various techniques according to the present invention can be applied to existing XR systems through software or firmware updates. There are other benefits.

[0014] The present invention achieves these and other benefits in the context of known technology.However, a further understanding of the nature and advantages of the present invention may be obtained by reference to the following portions of the specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A is a simplified diagram illustrating an extended reality (XR) device 115n according to an embodiment of the present invention.

[0016] Figure 1B is a simplified block diagram illustrating components of an extended reality device 115n according to an embodiment of the present invention.

[0017] Figure 2 is a simplified diagram illustrating the field of view of a camera on an extended reality device 210 according to an embodiment of the present invention.

[0018] Figure 3A is a simplified diagram showing key points defined on the right hand side according to an embodiment of the present invention.

[0019] Figure 3B is a simplified diagram illustrating exemplary gestures according to an embodiment of the present invention.

[0020] Figure 4 is a simplified block diagram showing the functional blocks of an extended reality device according to an embodiment of the present invention.

[0021] Figure 5 is a simplified block diagram illustrating functional modules in a gesture detection algorithm according to an embodiment of the present invention.

[0022] Figure 6 is a simplified flow chart illustrating the process of a hand prediction method according to an embodiment of the present invention.

[0023] Figure 7 is a simplified diagram illustrating hand prediction using 3D hand keypoints according to an embodiment of the present invention.

[0024] Figure 8 is a simplified diagram illustrating an exemplary state machine for hand tracking according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention relates to an extended reality system and method. In an exemplary embodiment, a two-dimensional hand image is captured. Two-dimensional key points are identified using the two-dimensional hand image. The two-dimensional key points are mapped to three-dimensional key points. Hand prediction is performed using the three-dimensional key points. There are other embodiments.

[0026] With the emergence of virtual reality and augmented reality applications, gesture-based control schemes are becoming increasingly popular. In recent years, commercial 3D hand tracking technology based on depth cameras on AR glasses has become popular, which directly measures the hands in 3D. Traditional research usually focuses on hand tracking algorithms based on RGB cameras. Compared with the algorithms, research work on actual hand tracking systems is very limited.

[0027] In particular, the ability to accurately and efficiently reconstruct human hand movements from images is expected to enable exciting new applications in immersive virtual and augmented reality, robotic control, and sign language recognition. Great progress has been made in recent years, especially with the advent of deep learning techniques. However, it remains a challenging task due to unconstrained global and local pose variations, frequent occlusions, local self-similarity, and high articulation. In various hand detection processes according to the present invention, the output includes a bounding box and a confidence value. During hand tracking, hand detection may be unstable, thereby reducing overall performance and accuracy. For example, existing hand detection methods generally adopt convolutional network-based methods, which use small models to interpret various hand data in the world, and the final results are often unsatisfactory. It should be noted that in addition to or instead of hand detection, hand prediction methods are also used to reduce the missing bounding box problem and the false positive bounding box problem. For example, the "missing bounding box" problem refers to the lack of detection of the true hand image, and the "false positive bounding box" problem refers to the incorrect detection of non-hand objects as hands. In the process of hand tracking, the restriction threshold used to define the bounding box may lead to different results. Too much restriction will lead to missing bounding boxes, while too little restriction will lead to wrong bounding boxes. In order to properly select the threshold, a machine learning algorithm can be configured into the system. After applying the present invention, the missing bounding boxes and false positive bounding boxes in the real-time 3D hand tracking results can be reduced to near zero.

[0028] According to various embodiments, the hand prediction mechanism involves using the 3D hand keypoints of the previous two frames (e.g., images captured by a camera at the previous two time stamps) and outputting a 2D bounding box surrounding the hand and a corresponding confidence value. For example, the hand prediction result is used as an input for bounding box tracking. In some implementations, a state machine is used to perform bounding box tracking and determine whether hand prediction should be used.

[0029] The following description is intended to enable one of ordinary skill in the art to make and use the invention and to incorporate it into the context of a particular application. Various modifications and various uses in different applications will be apparent to those skilled in the art, and the general principles defined herein may be applied to various embodiments. Therefore, the present invention is not limited to the embodiments described, but should have the widest scope consistent with the principles and novel features disclosed herein.

[0030] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without being limited to these specific details. In other cases, well-known structures and devices are shown in block diagram form, but are not shown in detail, to avoid obscuring the present invention.

[0031] Please note that all the documents and materials submitted at the same time as this specification and made available to the public at the same time as this specification are incorporated herein by reference. Unless otherwise expressly stated, all features disclosed in this specification (including any attached claims, abstracts and drawings) can be replaced by alternative features that serve the same, equivalent or similar purposes. Therefore, unless otherwise expressly stated, each feature disclosed is only an example of a series of equivalent or similar features.

[0032] In addition, any element in the claims that does not explicitly state a “means” for performing a specified function or a “step” for performing a specific function shall not be construed as a “means” or “step” provision as provided in 35 U.S.C. § 112, paragraph 6. In particular, the use of “step” or “action” in the claims herein is not intended to invoke the provisions of 35 U.S.C. § 112, paragraph 6.

[0033] Please note that labels such as left, right, front, back, top, bottom, forward, backward, clockwise, and counterclockwise, if used, are for convenience only and do not indicate any particular fixed direction. Instead, they are used to reflect the relative position and / or orientation of parts of an object.

[0034] Figure 1A1 is a simplified diagram (top view) showing an extended reality device 115n according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. It is to be understood that the term "extended reality" (ER) is broadly defined, including virtual reality (VR), augmented reality (AR), and / or other similar technologies. For example, the ER device 115 shown can be configured as VR, AR, or other. Depending on the specific implementation, the ER device 115 may include a small housing for AR applications, or a relatively large housing for VR applications. Cameras 180A and 180B are configured on the front of the device 115. For example, cameras 180A and 180B are mounted on the left and right sides of the ER device 115, respectively. In various applications, additional cameras can be configured below cameras 180A and 180B to provide additional field of view and distance estimation accuracy. For example, cameras 180A and 180B each include an ultra-wide angle or fisheye lens that provides a large field of view, and they share a common field of view. Due to the arrangement of the two cameras, the parallax of the two cameras (a known factor) can be used to estimate the object distance. Display 185 is configured on the back of ER device 115. For example, in AR applications, display 185 can be a semi-transparent display that overlays information on an optical lens. In VR implementations, display 185 can include a non-transparent display.

[0035] Figure 1B is a simplified block diagram showing the components of an extended reality device 115 according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. In some embodiments, an XR head-mounted device (e.g., the AR head-mounted device 115n shown, etc.) may include, but is not limited to, at least one of a processor 150, a data storage device 155, a speaker or earphone 160, an eye tracking sensor 165, a light source 170, an audio sensor or microphone 175, a front or front-facing camera 180, a display 185, and / or a communication interface 190 and / or the like.

[0036] In some cases, processor 150 may be communicatively coupled (e.g., via a bus, via a wired connector, or via electrical pathways (e.g., traces and / or pads, etc.) of a printed circuit board (PCB) or integrated circuit (IC), and / or the like) to each of one or more of data storage 155, speaker or earphone 160, eye tracking sensor 165, light source 170, audio sensor or microphone 175, front camera 180, display 185, and / or communication interface 190, and / or the like. In various embodiments, data storage 155 may include dynamic random access memory (DRAM) and / or non-volatile memory. For example, images captured by camera 180 may be temporarily stored in DRAM for processing, and executable instructions (e.g., hand shape calibration and gesture recognition algorithms) may be stored in non-volatile memory. In various embodiments, data storage 155 may be implemented as part of processor 150 in a system-on-chip (SoC) arrangement.

[0037] The eye tracking sensor 165, which may include but is not limited to at least one of one or more cameras, one or more motion sensors, or one or more tracking sensors, etc., tracks the gaze point of the user's eyes and is combined with the computational processing of the processor 150 to compare with the image or video taken in front of the ER device 115. The audio sensor 175 may include but is not limited to a microphone, a sound sensor, a noise sensor, etc., and may be used to receive or capture voice signals, sound signals, and / or noise signals, etc.

[0038] The front camera 180 includes respective lenses and sensors for capturing images or videos of the area in front of the ER device 115. For example, the front camera 180 includes Figure 1B The cameras 180A and 180B are shown as being respectively arranged on the left and right sides of the housing. In different implementations, the sensor of the front camera can be a low-resolution monochrome sensor, which is not only energy-saving (no color filter and its color processing is required), but also relatively cheap in terms of device size and cost.

[0039] Figure 2 is a simplified diagram showing the field of view of the cameras on the extended reality device 210 according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. The left camera 180A is mounted on the left side of the ER device housing 210 and the right camera 180B is mounted on the right side of the ER device housing 210. Each camera has an ultra-wide angle or fisheye lens capable of capturing a wide field of view. For example, camera 180A has an angle of θ on the left side. L The camera 180B has a field of view on the right side and an angle θ REither camera can detect hands or other objects.

[0040] Hand detection is a prerequisite for gesture recognition. When at least one camera in the XR device captures a hand image, the device detects the hand. For example, hand 221 can only be detected through the image captured by camera 180A, while hand 223 can only be detected through the image captured by camera 180B. When hand 222 is located in area 220, it is located in the common field of view of the two cameras 180A and 180B, and the image from either camera can be used for hand detection, and depth calculation and other calculations can also be performed. In use, the hand may move in and out of the field of view of the two cameras. Hand detection, hand tracking, and hand prediction processes can be performed. For example, when hand 221 moves to the left and moves out of the FOV of camera 180A, the hand prediction mechanism according to an embodiment of the present invention can reduce "false positive" hand recognition. In addition, when the hand is tracked, the bounding box surrounding the hand (i.e., the area within the image captured by camera 180A or camera 180B) will be updated using hand prediction techniques in particular.

[0041] Now return to reference Figure 1B In various embodiments, processor 150 is configured to perform hand detection and hand prediction processes. In various embodiments, processor 150 includes a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU). For example, the hand detection process may be performed by the NPU, while hand prediction may be performed by the CPU and / or the NPU.

[0042] In AR applications, the field of view of each front camera 180 overlaps the field of view of the user's 120 eyes, the captured image or video. Display screen and / or projector 185 can be used to display or project the generated image overlay (and / or display a composite image or video combined with the generated image overlay superimposed on the image or video of the actual area). Communication interface 190 provides wired or wireless communication with other devices and / or networks. For example, communication interface 190 can be connected to a computer for tethered operation, where the computer provides the processing power required for graphics-intensive applications.

[0043] Figure 3Ais a simplified diagram showing key points defined on the right hand according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, key points 0-19 are assigned to different areas of the user's right hand. Based on the positions of these key points, gestures can be determined. For example, by identifying the relative positions of these key points 0-19, different gestures can be determined. In various embodiments, the relative positions of the key points are calibrated (e.g., measured in pixel distances) during the initial hand shape calibration process, which makes the gesture recognition process more accurate.

[0044] Figure 3B is a simplified diagram showing exemplary gestures according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, a hand image captured as a two-dimensional (2D) image by a device camera may be mapped into a 3D space for processing. In various embodiments, a 2D image may be mapped into a 3D space using depth information and calibration parameters (e.g., hand shape). Since the hand moves in 3D space, the hand prediction process is performed using 3D vectors based on 3D coordinates. As Figure 3B As shown, for the left hand gesture, 21 (ie, the 0th to the 20th) key points are obtained. For example, the left hand gesture is converted into 3D key points, through which the extended reality device can recognize the gesture as an "OK" gesture. Other processes can also be performed.

[0045] Figure 4 is a simplified block diagram showing the functional blocks of an extended reality device according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. A person of ordinary skill in the art will recognize many variations, substitutions, and modifications. Figure 4 The system pipeline of the extended reality device 400 in the embodiment may include functional components, which may correspond to Figure 1B 115. At the front end, sensors, such as right fisheye camera 401, left fisheye camera 402, and inertial measurement unit (IMU) 403, capture images and other information and send the captured data to sensor processor 411 (e.g., a lightweight embedded CPU, such as Figure 1BThe sensor processor 411 performs various simple image processing (e.g., denoising, exposure control, etc.) and then packages the processed data to the XR server 421. For example, the XR server 421 is implemented as a data consumer and transmits data to various algorithms, such as 3D hand tracking 431, 6DoF 441, and others 451. The location of the 3D hand tracking algorithm 431 is configured after the XR server 421 as shown, followed by the APP module 432. In various embodiments, the 3D hand tracking algorithm 431 utilizes hand detection and hand prediction techniques.

[0046] The unified APP 432 receives the hand tracking results for different purposes, such as gaming, manipulation of virtual objects, etc. Additional functions may be configured as shown, such as object synthesizer 433, system rendering 434, asynchronous time warp (ATW) 435, and display 436. Depending on the specific implementation, there may also be other functional blocks.

[0047] Figure 5 is a simplified block diagram showing the functional modules in the gesture detection and prediction algorithm according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. A person of ordinary skill in the art will recognize many variations, substitutions, and modifications. According to various embodiments, it is possible to use Figure 1B The hand tracking system 500 implemented by the device 150 shown uses a two-hand tracking process for the left hand (l) and the right hand (r). For example, the system 500 provides real-time (i.e., 30 frames per second) hand tracking on an edge device, and it operates as a 3D hand tracking system. A stereo fisheye camera is used to acquire left and right images with known parallax calibration. The system includes a variety of algorithm sets, including hand acquisition 501, hand detection 502, hand prediction 503r and 503l, bounding box tracking 504r and 504l, 2D hand key point detection 505r (506r) and 505l (506l), 3D hand key point detection 507r and 507l, gesture recognition 508r and 508l, and hand shape calibration 570. In various embodiments, the hand prediction algorithm is part of the bounding box tracking process.

[0048] Figure 6 is a simplified flow chart showing the process of a hand prediction method according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, one or more steps may be added, deleted, repeated, modified, replaced, overlapped, and rearranged, which should not limit the scope of the claims.

[0049] At step 610, a two-dimensional (2D) hand image is captured, the captured image including at least a previous image and a current image. According to various embodiments, the image is captured by a 2D camera, and distance information may or may not be available. For example, the terms "previous image" and "current image" refer to two images captured at consecutive time intervals, where the current image is currently being processed and the previous image is the most recent. For example, the captured images are stored in a memory in chronological order and can therefore be easily retrieved for processing.

[0050] In step 620, a previous 2D keypoint and a current 2D keypoint are identified using the previous image and the current image, respectively. In various embodiments, the 2D keypoints are first used in the hand detection process, and the hand prediction process is performed only after one or more hands are detected. Depending on the specific implementation, various types of image recognition algorithms can be used. For example, a machine learning algorithm can be used for the image recognition process. A bounding box surrounding the first hand in the current image is defined by the upper left corner position and the lower right corner position. According to various embodiments, the bounding box includes at least ten percent of the edge area surrounding the first hand.

[0051] At step 630, the previous 2D keypoints are mapped to the previous 3D keypoints, and the current 2D keypoints are mapped to the current 3D keypoints. For example, the hand tracking process is initiated when a hand is detected, and since the hand moves in 3D space - even though only a 2D image of the hand is captured, the hand is tracked in 3D space. In various embodiments, 2D keypoints are projected into 3D space using information such as hand shape, hand distance, and hand size. For example, Figure 7 The use of 3D keypoints in hand prediction is shown.

[0052] Figure 7 is a simplified diagram illustrating hand prediction using 3D hand key points according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. As shown, by converting the 2D hand key points from the image captured by the camera, the hand prediction from the previous image P is obtained. t-1 and the current image P t 3D hand points. Figure 7 As shown, the previous 3D hand key points in frame P t-1 In the current 3D key point in frame P t The previous 3D hand key points and the current 3D key points are both generated at step 630 based on the 2D key points.

[0053] In step 640, the previous 3D key points and the current 3D key points generated in step 630 are used to generate a set of predicted 3D key points corresponding to the predicted hand position. In various embodiments, the vectors between the two corresponding previous and current 3D hand key points are calculated and used to generate the predicted 3D key points. For example, each vector includes changes in the coordinate values ​​of the x-axis, y-axis, and z-axis. In a specific embodiment, assuming that the direction of the hand movement is substantially linear and assuming that the speed of the hand movement is substantially constant, the predicted 3D key points can be easily inferred (e.g., applying the same difference of each key point coordinate). For example, where the previous 3D key point is (1, 1, 1) and the current 3D key point is (4, 5, 6), the predicted 3D key point will be (7, 9, 11). It should be recognized that linear extrapolation is a relatively simple calculation that can be performed by various types of processors. In various embodiments, other types of extrapolation mechanisms can be used, and more than two sets of 3D key points can be used for prediction. The calculation of 3D keypoint prediction can be performed in real time and meet predetermined performance requirements (e.g., 30 frames per second or faster). In various embodiments, a convolutional neural network is used to calculate the confidence value, which can be performed by one or more NPUs.

[0054] As an example, Figure 7 3D keypoint prediction is shown. As shown, the input to the prediction process is the 3D hand keypoints of two time stamps, including the previous frame (t-1) and the current frame (t). For example, image P t-1 is the image captured by the camera at the previous timestamp t-1, image P t is the image captured by the camera at the current timestamp t. Extrapolation is used to predict the 3D key points of the hand at the next timestamp t+1 (e.g., P t+1 =2*P t -P t-1 ). For example, the 3D hand keypoints are in the format of 21 keypoints in 3D rectilinear space (i.e., x, y, z), where each point (x, y, z) is the corresponding 3D position of the keypoint. As described above, when using the extrapolation formula P t+1 =2*P t-1 -P t The hand is assumed to move at a constant speed when . More complex formulas can be used to apply acceleration and direction changes to the predictions.

[0055] In addition to generating predicted 3D key points, a confidence value for each predicted key point may also be calculated. For example, each predicted 3D key point is assigned a confidence value between 0 and 1. The assigned confidence values ​​may be used in a variety of ways. For example, predicted key points with low confidence values ​​may be discarded. Depending on the implementation, the confidence values ​​of the predicted 3D key points may be calculated in a variety of ways. The total confidence value of the 21 key points is between 0 and 21, and if the total confidence value is below a predetermined threshold, the predicted frame may be discarded.

[0056] Now back to Figure 6 At step 650, the predicted 3D key points are mapped to predicted 2D key points. In addition, confidence values ​​are assigned to the predicted 3D key points. It is to be understood that, depending on the application and use, either 3D or 2D key points may be used. For example, for gesture recognition, 3D key points (e.g., see Figure 5 , blocks 507 and 508). Hand prediction can be used for bounding box tracking (see, for example, Figure 5 , blocks 503 and 504), for this application, 2D keypoints are more useful. For example, the predicted 3D keypoints are converted to 2D keypoints that can be used for bounding box tracking. For example, Figure 7 Frame P is shown t-1 The predicted 3D key points are mapped to 2D key points. In various embodiments, the predicted 3D key points are projected into 2D space. Figure 7 As shown, the predicted image P t+1 (e.g. 21 keypoints in (x, y, z) space) are projected as 2D hand keypoints (21*(u,v)) as output.

[0057] At step 660, the bounding box is tracked using the predicted 2D keypoints. In various embodiments, the overall confidence value of the predicted 2D or 3D keypoints may be used to identify false hand detections. For example, low confidence values ​​(e.g., less than 11 out of 21) may indicate that the predicted 3D keypoints (and corresponding 2D keypoints) may be incorrect (perhaps due to false hand detections) and that they should not be used for applications such as bounding box tracking and gesture detection.

[0058] Predicted 2D key points can be used to facilitate bounding box tracking in various ways. As described above, hand detection may be unreliable for various reasons, and the predicted key points - and their confidence values ​​- can be used to identify "false positive" hand detections. In various embodiments, the hand prediction mechanism according to embodiments of the present invention can be both accurate and efficient. For example, a hand prediction method performed in 3D space (in addition to or instead of hand detection) can reduce the problem of missing bounding boxes and the problem of false positive bounding boxes. In various embodiments, the limiting thresholds used to define the bounding box (e.g., using confidence values) may lead to different results and can be calibrated depending on the use case (e.g., dark environment or bright environment). The threshold for false hand detection can use a machine learning algorithm. It is to be understood that the hand prediction process combined with the bounding box can greatly improve performance, and the missing bounding boxes and false positive bounding boxes in the real-time 3D hand tracking results can be reduced to near zero.

[0059] In various embodiments, the predicted 2D keypoints may be used to define and update the size and position of a bounding box. For example, a bounding box is defined around the predicted 2D keypoints with a predetermined margin (e.g., 10% to 20% around the outermost keypoints). The predicted bounding box may also change size and shape (e.g., when a fist becomes a palm, or vice versa).

[0060] As an example, Figure 6 The steps shown in Figure 1B The camera module 180 may be used to capture images, as described in step 610. The hand detection and hand prediction processes may be performed by the processor 150. In a specific embodiment, Figure 8 An exemplary hand prediction process is shown. Figure 8 is a simplified diagram showing an exemplary state machine for hand tracking according to an embodiment of the present invention. Those skilled in the art will recognize many variations, substitutions, and modifications. For example, Figure 8 One or more of the steps shown may be added, deleted, repeated, modified, overlapped, rearranged, or substituted, which shall not limit the scope of the claims.

[0061] At block 810, hand tracking is in a "dead" state, where processes such as hand detection and hand tracking are not performed. For example, the XR device may be in the 810 state when idle (e.g., the image remains unchanged, with no movement or other types of changes). The XR device will remain in this "dead" state until activated (e.g., movement or change of the image is detected in the left or right image).

[0062] At block 820, the XR device boots up and prepares to perform various tasks, such as hand detection and hand tracking. As part of the initialization process at block 820, the camera is active and captures images, which are stored for processing.

[0063] At box 830, hand tracking is performed, which includes hand detection (box 831) and hand prediction (box 832). Hand detection 831 can be repeated until a hand is detected in one of the left image and the right image. As described above, the hand detection process 881 may incorrectly identify the hand. Once the hand is detected, the hand prediction process 832 is performed using at least the previous frame and the current frame. As part of the hand tracking process, hand prediction 832 can be repeated until the hand is lost or is not within the bounding box, and the hand prediction process can be performed when the hand is within the bounding box. For example, if the hand being tracked is no longer within the bounding box, hand detection 831 can be performed to define a new bounding box; hand detection 831 can also determine that the hand no longer exists and continue to box 840. In the present invention, the hand detection of the first two timestamps is mainly used for hand tracking, and then hand prediction is used extensively.

[0064] At block 840, the XR device is in a "lost" state in which a hand is no longer detected. For example, the hand prediction process 832 may identify a "false positive" hand detection and determine that a hand is not present. In the lost state, the XR components and processing may still be active to detect hand movement, and if movement is detected in the image (e.g., the difference between two consecutive timestamps), then return to block 830 to perform hand detection. For example, block 840 runs a loop (as shown) for a predetermined time before moving to the "dead" state in block 810. In some embodiments, blocks 810 and 840 may be implemented (or programmed) as the same state.

[0065] As an example, the pseudo code of the hand prediction processing mechanism according to the present invention is provided below:

[0066]

[0067]

[0068] As an example, the state machine 800 may be stored as instructions executed by a processor, which may include different computing cores (e.g., NPU and GPU). For example, the hand detection process 831 and the hand prediction process 832 may be executed by the NPU.

[0069] Return now Figure 5 The system 500 implements a set of outputs including 3D hand key points in blocks 507r and 507l. For example, the hand key points are as follows: Figure 3A and Figure 3B As shown. Note that although the captured images are two-dimensional, gesture detection is performed using 3D hand key points. For example, the results and / or intermediate calculations obtained in blocks 503l and 503r may be used in the gesture recognition process. For example, 2D to 3D mapping may be performed between blocks 505l and 507l, or obtained from blocks 5031 and 503r. Calibration parameters may be used in the mapping process.

[0070] As shown, the system 500 includes five components: a main thread 501, a hand detection thread 502, a right hand thread 502r, a left hand thread 5021, and a hand shape calibration thread 570. These components interact with each other.

[0071] As an example, the main thread 501 is used to copy images captured by the right fisheye camera 501r and the left fisheye camera 501l to the local memory of the system.

[0072] The hand detection thread 502 waits for the right fisheye image and the left fisheye image. Once the images are received, the hand detection thread 502 can use the hand detection convolutional network on the right fisheye image and the left fisheye image. For example, the hand detection thread 502 outputs confidence values ​​and bounding boxes for the right hand and the left hand.

[0073] The right hand thread 502r and the left hand thread 502l can be implemented symmetrically, receiving thread inputs from the right fisheye image and the left fisheye image, respectively. They also rely on their own bounding box tracking (i.e., boxes 504r and 504l). For example, confidence values ​​and bounding box tracking can be used to generate 3D hand key points that allow the recognition of gesture types.

[0074] Hand bounding box threads 504r and 504l provide tracking, and their input is the bounding box size (and shape), confidence value and bounding box prediction value from hand prediction box 503r and 503l. Hand bounding box threads 504r and 504l specifically output hand state (e.g., whether it exists) and bounding box data, etc.

[0075] like Figure 5 As shown, if a hand is present (as determined in blocks 504r and 504l), 2D hand keypoint detection (e.g., blocks 505r and / or 505l) crops out the hand using a bounding box from hand bounding box tracking of the captured image. For example, the captured image is cropped and resized to a predetermined size (e.g., 96 pixels by 96 pixels, which is an optimally small size for efficient processing). 2D hand keypoint detection (e.g., blocks 505r and 505l) uses a 2D keypoint detection convolutional network on the resized image and outputs 2D hand keypoints. As described above, if 2D keypoints are present, they are mapped to 3D keypoints for gesture detection.

[0076] Although specific embodiments are fully described above, various modifications, alternative constructions and equivalents may be used. Therefore, the above descriptions and illustrations should not be taken as limiting the scope of the invention, which is defined by the appended claims.

Claims

1. A hand prediction method, the method comprising: capturing a plurality of images including at least a first hand, the plurality of images being in a two-dimensional (2D) space, the plurality of images comprising a current image and a previous image; Using a previous image to identify multiple previous 2D keypoints; Use the current image to identify multiple current 2D key points; mapping the plurality of previous 2D keypoints to a plurality of previous three-dimensional (3D) keypoints; Mapping the plurality of current 2D key points to a plurality of current 3D key points; generating a plurality of 3D predicted keypoints in a 3D space using the plurality of previous 3D keypoints and the plurality of current 3D keypoints; Mapping the plurality of 3D predicted key points to a plurality of predicted 2D key points; as well as Potential false hand detections are determined using the plurality of predicted 2D keypoints.

2. The method according to claim 1, further comprising: The plurality of previous 2D keypoints are projected into 3D space.

3. The method according to claim 2, wherein: The plurality of images also include at least a second hand, and the plurality of 3D predicted keypoints are associated with the first hand and the second hand.

4. The method according to claim 1, further comprising: defining a bounding box enclosing the first hand in the current image; The bounding box is tracked using the plurality of predicted 2D keypoints.

5. The method according to claim 4, wherein: The bounding box is defined by an upper left corner position and a lower right corner position, and includes at least ten percent of a marginal area surrounding the first hand.

6. The method according to claim 1, wherein: The plurality of 3D predicted key points are assigned confidence values, and the method further comprises detecting non-hand objects using the confidence values.

7. The method according to claim 1, further comprising: The first hand is tracked using the plurality of predicted 2D keypoints.

8. The method according to claim 1, further comprising: A hand tracking process is initiated when the first hand is detected.

9. The method according to claim 1, further comprising: Coordinate changes between the plurality of previous 3D key points and the plurality of current 3D key points are calculated, each 3D key point including three coordinates.

10. The method according to claim 1, further comprising: A plurality of 3D vectors of the plurality of previous 3D key points and the plurality of current 3D key points are used.

11. An extended reality device, comprising: a housing having a front side and a back side; a first camera disposed at the front side and configured to capture a plurality of two-dimensional (2D) images at a predefined frame rate, the plurality of 2D images including a current image and a previous image; A display, arranged on the back side of the housing; a memory coupled to the first camera and configured to store the plurality of 2D images; as well as a processor coupled to the memory; Wherein, the processor is configured to: identifying a plurality of 2D key points associated with a hand using at least the current image and the previous image; mapping the plurality of 2D keypoints to a plurality of three-dimensional (3D) keypoints; and At least the plurality of 3D keypoints are used to provide hand prediction.

12. The device according to claim 11, wherein The processor includes a neural processing unit configured to detect a hand using a first image captured by the first camera. 13 . The device according to claim 11 , further comprising a second camera, wherein the first camera is located on a left side of the housing, and the second camera is located on a right side of the housing.

14. The device according to claim 11, wherein: The processor is also configured to track the hand.

15. The device according to claim 11, wherein The processor is further configured to: Generate multiple predicted 3D keypoints; The plurality of predicted 3D keypoints are mapped to a plurality of predicted 2D keypoints.

16. A hand tracking method, the method comprising: capturing a first image; detecting at least a first hand in the first image; capturing a plurality of images including at least the first hand, the plurality of images being in a two-dimensional (2D) space, the plurality of images comprising a current image and a previous image; identifying a plurality of 2D key points using the plurality of images; Mapping the plurality of 2D key points into a plurality of three-dimensional (3D) key points; generating a plurality of 3D predicted key points in a 3D space using the plurality of 3D key points; Mapping the plurality of 3D predicted key points to a plurality of predicted 2D key points; as well as The first hand is tracked using the plurality of 3D predicted key points.

17. The method according to claim 16, further comprising: Calculating confidence values ​​of the multiple 3D prediction key points; At least the confidence value is used to identify false hand detections.

18. The method according to claim 16, further comprising: Changes between the first image and the second image are identified.

19. The method according to claim 16, further comprising: A deep learning process is performed using the first image for hand detection.

Images