Gesture detection method and system with optimized hand detection

Abstract

The invention relates to an augmented reality system and method. In an exemplary embodiment, hand images captured by at least two cameras are used in a hand detection process performed on images alternately selected from among the images captured by the at least two cameras. Once a hand is detected, hand tracking is performed. Other embodiments are also provided.

Description

Background of the Invention

[0002] The present invention relates to an extended reality system and method.

[0003] Over the past decade, Extended Reality (XR) devices, including Augmented Reality (AR) devices and Virtual Reality (VR) devices, have become increasingly popular. Important design considerations and challenges for XR devices include performance, cost, and power consumption. Due to various limitations, existing XR devices have been unable to meet the requirements, as explained below.

[0004] New and improved XR systems and methods thereof are desired. Summary of the invention

[0005] The present invention relates to an extended reality system and method. In an exemplary embodiment, hand images captured by at least two cameras are used in a hand detection process, and the hand detection process is performed on images alternately selected from the images captured by the at least two cameras. Once the hand is detected, hand tracking is performed. There are other embodiments.

[0006] A system of one or more computers may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which, when run, causes the system to perform actions. One or more computer programs may be configured to perform specific operations or actions by including instructions, which, when executed by a data processing device, causes the device to perform actions. A general aspect includes a hand shape detection method. The method also includes capturing a first set of images through a left camera. The method also includes capturing a second set of images through a right camera. The method also includes storing the first set of images and the second set of images in a memory. The method also includes alternating between a first set of images and a second set of images, the first set of images including a first image associated with a first timestamp, the second set of images including a second image associated with a second timestamp, the first timestamp and the second timestamp being consecutive timestamps. The method also includes performing a first hand detection process using the first image. The method also includes: based on detecting the absence of a hand during the first hand detection process, performing a second hand detection process using the second image. The method also includes: if a hand is detected during the first hand detection process, performing a hand prediction process. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0007] Implementations may include one or more of the following features. A hand prediction process uses a first image and a second image. The method also includes generating a frame index, the frame index being updated based on at least a timestamp. A first hand detection process is performed by a neural processing unit. The method may include initiating a hand tracking process if a hand is detected, the hand tracking process using the first image and the second image. The method may include performing a third hand detection process if a hand is determined to be lost. The method may include performing a third hand detection process using a third image, and performing a fourth hand detection process using a fourth image, the first group of images including the third image, and the second group of images including the fourth image. The method may include tracking the hand in a bounding box if a hand is detected. The first hand detection process and the second hand detection process are performed within a predetermined time interval, the predetermined time interval being associated with a frame rate of at least twenty frames per second. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0008] One general aspect includes an extended reality device, the device including a housing, the housing including a front side and a back side. The device also includes a left camera, the left camera may include a first sensor and a first lens, the first lens having a first field of view, the left camera is configured in a left area of ​​the front side, the left camera is configured to capture a first set of images, the first set of images including a first image with a first timestamp. The device also includes a right camera, the right camera may include a second sensor and a second lens, the second lens having a second field of view, the right camera is configured in a right area of ​​the front side, the second field of view shares a common field of view with the first field of view, the right camera is configured to capture a second set of images, the second set of images including a second image with a second timestamp, the difference between the first timestamp and the second timestamp is within a predetermined interval. The device also includes a display configured on the back side of the housing. The device also includes a memory coupled to the first sensor and the second sensor, the memory configured to store the first set of images and the second set of images. The device also includes a first processor coupled to the memory. The device also includes a second processor coupled to the first processor and the memory, the second processor including a neural processing unit. The apparatus also includes: a first processor configured to alternately select between a first set of images and a second set of images; a second processor configured to perform a first hand detection process using the first image; the second processor is further configured to perform a second hand detection process using the second image in response to determining that a hand is not present during the first hand detection process; and the second processor is further configured to perform a hand prediction process when a hand is detected during the first hand detection process. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0009] Implementations may include one or more of the following features. The second processor in the device is configured to perform a hand detection process at a rate of at least 20 frames per second. The second processor is also configured to detect the absence of a hand from the first set of images or the second set of images. The first lens may include a fisheye lens. The first sensor may include a monochrome sensor with less than one million pixels. The capture rate of the left camera is at least 15 frames per second. The above predetermined interval is less than 30 milliseconds. Implementation of the described technology may include hardware, method or process, or computer software on a computer accessible medium.

[0010] A general aspect includes a hand shape detection method. The method includes capturing a first set of images through a left camera. The method also includes capturing a second set of images through a right camera. The method also includes storing the first set of images and the second set of images in a memory. The method also includes alternating between the first set of images and the second set of images. The method also includes performing a first hand detection process using the first selected image. The method also includes: based on detecting the absence of a hand using the first hand detection process, performing a second hand detection process using the second selected image. The method also includes: if a hand is detected in the first hand detection process or the second hand detection process, tracking the hand in a bounding box. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0011] Implementations may include one or more of the following features. The method may include performing a hand prediction process. Selecting the first set of images and the second set of images using a frame index number. The method may include switching from a hand detection process to a hand prediction process if a hand is not detected in a third selected image. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer accessible medium.

[0012] It is to be appreciated that embodiments of the present invention provide many advantages over conventional techniques. In particular, hand detection techniques are more efficient in terms of power and computation. Furthermore, hand shape detection techniques according to embodiments of the present invention can be performed at high frame rates and meet various performance requirements of XR devices.

[0013] Embodiments of the present invention can be implemented in combination with existing systems and methods. For example, the hand shape detection technology according to the present invention can be used in various XR systems. In addition, various technologies according to the present invention can be applied to existing XR systems through software or firmware updates. There are other benefits.

[0014] The present invention achieves these and other benefits in the context of known technology.However, a further understanding of the nature and advantages of the present invention may be obtained by reference to the following portions of the specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A is a simplified diagram illustrating an extended reality (XR) device 115n according to an embodiment of the present invention.

[0016] Figure 1B is a simplified block diagram illustrating components of an extended reality device 115n according to an embodiment of the present invention.

[0017] Figure 2 is a simplified diagram illustrating the field of view of a camera on an extended reality device 210 according to an embodiment of the present invention.

[0018] Figure 3 is a simplified diagram showing key points defined on the right hand side according to an embodiment of the present invention.

[0019] Figure 4 is a simplified block diagram showing the functional blocks of an extended reality device according to an embodiment of the present invention.

[0020] Figure 5 is a simplified block diagram illustrating functional modules in a gesture detection algorithm according to an embodiment of the present invention.

[0021] Figure 6 is a simplified flow chart showing the process of a hand detection method according to an embodiment of the present invention.

[0022] Figure 7 is a simplified flow chart illustrating the process of a hand tracking method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention relates to an extended reality system and method. In an exemplary embodiment, hand images captured by at least two cameras are used in a hand detection process, and the hand detection process is performed on images alternately selected from the images captured by the at least two cameras. Once the hand is detected, hand tracking is performed. There are other embodiments.

[0024] With the advent of virtual and augmented reality applications, gesture-based control schemes are becoming increasingly popular. The ability to accurately and efficiently reconstruct human hand movements from images promises to enable exciting new applications in immersive virtual and augmented reality, robotic control, and sign language recognition. Great progress has been made in recent years, especially with the advent of deep learning techniques. However, it remains a challenging task due to unconstrained global and local pose variations, frequent occlusions, local self-similarity, and high articulation. In recent years, commercial depth camera-based 3D hand tracking techniques on AR glasses have become popular, which directly measure the hands in 3D. Traditional research usually focuses on hand tracking algorithms based on RGB cameras, and compared to the algorithms, there is very limited research work on actual hand tracking systems.

[0025] Embodiments of the present invention provide a complete hand tracking system for AR glasses. The entire system can implement a variety of features, including real-time tracking, two-hand tracking, shape calibration, and simultaneous gesture recognition. In various embodiments, a stereo fisheye camera with a low-power sensor is used with a fisheye lens to capture a large field of view (FoV). It is to be understood that embodiments of the present invention use a stereo fisheye (or ultra-wide-angle) camera instead of a ToF module for distance measurement. Using a stereo camera, embodiments of the present invention provide a complete system-wide solution that can involve features such as real-time on edge devices (e.g., mobile phones, embedded devices) to simultaneously implement two-hand tracking and gesture recognition, and adjust the hand scale to match the real hand. The hand detection process according to an embodiment of the present invention utilizes a combined wide FoV of two (or more) cameras, and it alternately uses images from the left camera and the right camera. For example, the left image and the right image are selected in a cyclic manner for hand detection, and the computational complexity is reduced by half compared to the hand detection process using both the left image and the right image. In a specific implementation, the hand detection process runs at a speed of 18 milliseconds per timestamp, which greatly meets the desired frame rate of 30 frames per second. It is to be understood that the hand detection mechanism of the present invention may be implemented with other ER configurations such as a quad-camera setup.

[0026] The following description is intended to enable one of ordinary skill in the art to make and use the invention and to incorporate it into the context of a particular application. Various modifications and various uses in different applications will be apparent to those skilled in the art, and the general principles defined herein may be applied to various embodiments. Therefore, the present invention is not limited to the embodiments described, but should have the widest scope consistent with the principles and novel features disclosed herein.

[0027] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without being limited to these specific details. In other cases, well-known structures and devices are shown in block diagram form, but are not shown in detail, to avoid obscuring the present invention.

[0028] Please note that all the documents and materials submitted at the same time as this specification and made available to the public at the same time as this specification are incorporated herein by reference. Unless otherwise expressly stated, all features disclosed in this specification (including any attached claims, abstracts and drawings) can be replaced by alternative features that serve the same, equivalent or similar purposes. Therefore, unless otherwise expressly stated, each feature disclosed is only an example of a series of equivalent or similar features.

[0029] In addition, any element in the claims that does not explicitly state a “means” for performing a specified function or a “step” for performing a specific function shall not be construed as a “means” or “step” provision as provided in 35 U.S.C. § 112, paragraph 6. In particular, the use of “step” or “action” in the claims herein is not intended to invoke the provisions of 35 U.S.C. § 112, paragraph 6.

[0030] Please note that labels such as left, right, front, back, top, bottom, forward, backward, clockwise, and counterclockwise, if used, are for convenience only and do not indicate any particular fixed direction. Instead, they are used to reflect the relative position and / or orientation of parts of an object.

[0031] Figure 1A1 is a simplified diagram (top view) showing an extended reality device 115n according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. It is to be understood that the term "extended reality" (ER) is broadly defined, including virtual reality (VR), augmented reality (AR), and / or other similar technologies. For example, the ER device 115 shown can be configured as VR, AR, or other. Depending on the specific implementation, the ER device 115 may include a small housing for AR applications, or a relatively large housing for VR applications. Cameras 180A and 180B are configured on the front of the device 115. For example, cameras 180A and 180B are mounted on the left and right sides of the ER device 115, respectively. In various applications, additional cameras can be configured below cameras 180A and 180B to provide additional field of view and distance estimation accuracy. For example, cameras 180A and 180B each include an ultra-wide angle or fisheye lens that provides a large field of view, and they share a common field of view. Due to the arrangement of the two cameras, the parallax of the two cameras (a known factor) can be used to estimate the object distance. The combined field of view of the two wide angles can allow hand detection when one or more hands enter. Display 185 is configured on the back of ER device 115. For example, in AR applications, display 185 can be a semi-transparent display that overlays information on an optical lens. In VR implementations, display 185 can include a non-transparent display.

[0032] Figure 1B is a simplified block diagram showing the components of an extended reality device 115 according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. In some embodiments, an XR head-mounted device (e.g., the AR head-mounted device 115n shown, etc.) may include, but is not limited to, at least one of a processor 150, a data storage device 155, a speaker or earphone 160, an eye tracking sensor 165, a light source 170, an audio sensor or microphone 175, a front or front-facing camera 180, a display 185, and / or a communication interface 190 and / or the like.

[0033] In some cases, processor 150 may be communicatively coupled (e.g., via a bus, via a wired connector, or via electrical pathways (e.g., traces and / or pads, etc.) of a printed circuit board (PCB) or integrated circuit (IC), and / or the like) to each of one or more of data storage 155, speaker or earphone 160, eye tracking sensor 165, light source 170, audio sensor or microphone 175, front camera 180, display 185, and / or communication interface 190, and / or the like. In various embodiments, data storage 155 may include dynamic random access memory (DRAM) and / or non-volatile memory. For example, images captured by camera 180 may be temporarily stored in DRAM for processing, and executable instructions (e.g., hand shape calibration and gesture recognition algorithms) may be stored in non-volatile memory. In various embodiments, data storage 155 may be implemented as part of processor 150 in a system-on-chip (SoC) arrangement.

[0034] In various embodiments, the processor 150 includes different types of processing units, such as a central processing unit (CPU) 151 and a neural processing unit (NPU) 152. The processor 150 may also include a graphics processing unit. Different types of processing units are optimized for different types of calculations. For example, the CPU 151 handles various types of system functions, such as managing the camera 180 and moving the captured image to the data memory 155. The NPU 152 is optimized for convolutional neural networks and predictive models. In some embodiments, the NPU 152 is specifically used to perform ER-related calculations, such as hand tracking, gesture recognition, image recognition, and / or other calculations. The NPU 152 optimized for neural processing may consume relatively more power when running, and for some applications (such as hand detection), it may not be able to run at real-time speeds (e.g., 30 frames per second). In order to efficiently utilize the NPU 152 and reduce overall power consumption, an embodiment of the present invention adopts a loop scheme (described in further detail below), by which the hand detection process alternates between images captured by the left camera and images captured by the right camera, rather than processing these two images simultaneously. For example, by effectively allocating the NPU for hand detection, real-time performance of 30 frames per second can be achieved.

[0035] The eye tracking sensor 165, which may include but is not limited to at least one of one or more cameras, one or more motion sensors, or one or more tracking sensors, etc., tracks the gaze point of the user's eyes and is combined with the computational processing of the processor 150, the computing system 105a or 105b, and / or the AI ​​system to compare with the image or video taken in front of the ER device 115. The audio sensor 175 may include but is not limited to a microphone, a sound sensor, a noise sensor, etc., and may be used to receive or capture voice signals, sound signals, and / or noise signals, etc.

[0036] The front camera 180 includes respective lenses and sensors for capturing images or videos of the area in front of the ER device 115. For example, the front camera 180 includes Figure 1B The cameras 180A and 180B are shown as being respectively arranged on the left and right sides of the housing. In various implementations, the sensor of the front camera can be a low-resolution monochrome sensor, which is not only energy-saving (no color filter and its color processing is required), but also relatively cheap in terms of device size and cost.

[0037] Figure 2 is a simplified diagram showing the field of view of the cameras on the extended reality device 210 according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. The left camera 180A is mounted on the left side of the ER device housing 210 and the right camera 180B is mounted on the right side of the ER device housing 210. Each camera has an ultra-wide angle or fisheye lens capable of capturing a wide field of view. For example, camera 180A has an angle of θ on the left side. L The camera 180B has a field of view on the right side and an angle θ RField of view. Any camera can detect a hand or other object. For example, hand 221 is within the field of view of camera 180A, and hand 223 is within the field of view of camera 180B. In various embodiments, when the hand is only within the FOV of a single camera, the amount of processing is limited and calculations such as distance estimation and 3D mapping may not be performed. Region 220 is within the field of view of two cameras, and additional processing can be performed on objects within region 220. For example, region 220 can be referred to as a "common FOV", which is defined by two FOVs and a distance; that is, additional calculations and processing can be performed within a predetermined distance (e.g., the user's arm length) and FOV. In particular, since hand 222 is within the field of view of two cameras 180A and 180B, the difference between the images captured by the two cameras can be used to approximate the distance between hand 222 and housing 210. For example, when hand 222 is located in region 220 as shown, processing such as hand shape calibration, three-dimensional mapping, etc. can be performed. In some embodiments, a shape calibration process (described in further detail below) is implemented as part of an initial calibration process, and the user is prompted to position her hand to region 220 for shape calibration, and the calibration parameters generated during this process are later used for other calculations and processing.

[0038] Now return to reference Figure 1B . In various embodiments, the processor 150 is configured to calculate hand calibration parameters based on a left image captured by a left camera (e.g., camera 180A) and a right image captured by a right camera (e.g., camera 180B). The left image includes a hand (e.g., hand 222) located at a first position of the left image and within a common field of view (e.g., area 220). The right image includes a hand (e.g., hand 222) located at a second position of the right image and within the common field of view. During hand detection, images captured by cameras 180A and 180B are processed alternately; that is, only one of the two images captured simultaneously (e.g., sharing the same timestamp or within a predetermined time interval) is used during hand detection. For example, detection of hand 221 or hand 223 will cause the ER device to switch to a hand tracking and / or gesture recognition mode. In various embodiments, detection of hand 222 in the common field of view 220 using one of the images will trigger hand calibration, gesture recognition, and / or other processing using the simultaneously captured left and right images. For example, cameras 180A and 180B are configured to capture images at a predetermined frame rate (e.g., 24 frames per second or 30 frames per second). In order to perform hand detection using both the left image and the right image, the NPU may need to process the images at double the time (e.g., 48 frames per second or 60 frames per second, respectively). By using a looping scheme during the hand detection process, the workload of the NPU is reduced to a frame rate equal to the capture rate of a single camera (e.g., 24 frames per second or 30 frames per second).

[0039] In AR applications, the field of view of each front camera 180 overlaps the field of view of the user's 120 eyes, the captured image or video. Display screen and / or projector 185 can be used to display or project the generated image overlay (and / or display a composite image or video combined with the generated image overlay superimposed on the image or video of the actual area). Communication interface 190 provides wired or wireless communication with other devices and / or networks. For example, communication interface 190 can be connected to a computer for tethered operation, where the computer provides the processing power required for graphics-intensive applications.

[0040] Figure 3 is a simplified diagram showing key points defined on the right hand according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, key points 0-19 are assigned to different areas of the user's right hand. Based on the positions of these key points, gestures can be determined. For example, by identifying the relative positions of these key points 0-19, different gestures can be determined. For example, key points are used in hand detection, tracking, and gesture recognition processes (e.g., by the NPU in a CNN).

[0041] Figure 4 is a simplified block diagram showing the functional blocks of an extended reality device according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. A person of ordinary skill in the art will recognize many variations, substitutions, and modifications. Figure 4 The system pipeline of the extended reality device 400 in the embodiment may include functional components, which may correspond to Figure 1B 115. At the front end, sensors, such as right fisheye camera 401, left fisheye camera 402, and inertial measurement unit (IMU) 403, capture images and other information and send the captured data to sensor processor 411 (e.g., a lightweight embedded CPU, such as Figure 1B150 in the processor). The sensor processor 411 performs various simple image processing (e.g., denoising, exposure control, etc.), and then packages the processed data to the XR server 421. In various embodiments, the sensor processor 411 is implemented using a CPU or GPU, and its computational cost is generally not as expensive as an NPU. For example, the XR server 421 is implemented as a data consumer and transmits data to various algorithms 451, such as 3D hand tracking 431, 6DoF 441, etc. In one embodiment, the algorithm 451 includes a hand detection algorithm executed by one or more NPUs. The location of the 3D hand tracking algorithm 431 is configured after the XR server 421 as shown, followed by an APP module 432. For example, the unified APP 432 receives hand tracking results for different purposes, such as games, manipulation of virtual objects, etc. Additional functions, such as system rendering 434, asynchronous time warp (ATW) 435, and display 436, can be configured as shown. Depending on the specific implementation, there may also be other functional blocks.

[0042] Figure 5 is a simplified block diagram showing the functional modules in the gesture detection algorithm according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. A person of ordinary skill in the art will recognize many variations, substitutions, and modifications. According to various embodiments, it is possible to use Figure 1B The hand tracking system 500 implemented by the device 150 shown uses a two-hand tracking process for the left hand (l) and the right hand (r). For example, the system 500 provides real-time (i.e., 30 frames per second) hand tracking on an edge device, and it operates as a 3D hand tracking system. A stereo fisheye camera is used to acquire left and right images with known parallax calibration. The system includes a variety of algorithm sets, including hand acquisition 501, hand detection 502, hand prediction 503r and 503l, bounding box tracking 504r and 504l, 2D hand key point detection 505r and 505l, 3D hand key point detection 507r and 507l, gesture recognition 508r and 508l, and hand shape calibration 570.

[0043] The system 500 implements a set of outputs including 3D hand key points in blocks 507r and 507l. For example, the hand key points are as follows: Figure 3 As shown. Note that although the captured images are two-dimensional, gesture detection is performed using 3D hand key points. For example, 2D to 3D mapping is performed between blocks 505l and 507l, and calibration parameters can be used in the mapping process.

[0044] As shown, the system 500 includes five components: a main thread 501, a hand detection thread 502, a right hand thread 502r, a left hand thread 5021, and a hand shape calibration thread 570. These components interact with each other.

[0045] As an example, the main thread 501 is used to copy images captured by the right fisheye camera 501r and the left fisheye camera 501l to the local memory of the system.

[0046] The hand detection thread 502 waits for the right fisheye image and the left fisheye image. Once the images are received, the hand detection thread 502 can use the hand detection convolutional network on the right fisheye image and the left fisheye image. For example, the hand detection thread 502 outputs confidence values ​​and bounding boxes for the right hand and the left hand. In various embodiments, the images captured by the left camera and the right camera are alternately selected for the hand detection process.

[0047] Figure 6 is a simplified flow chart showing the process of the hand detection method according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. A person of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, Figure 6 One or more of the steps shown in the claims may be added, deleted, repeated, modified, overlapped, rearranged, and substituted, which shall not limit the scope of the claims.

[0048] In step 601, a hand image is captured by the left camera and the right camera, respectively. For example, the hand may be located in the image captured by the left camera or the right camera. However, it should be noted that if the hand is located in the common field of view, once the hand is detected, processes such as hand tracking and / or hand shape calibration can be performed. For example, when the hand 222 (see Figure 2 ) is located in the common FOV 220, allowing the two cameras 180A and 180B to record their respective images, where the hand 222 is located in slightly different positions in the left frame and the right frame, and the difference in position allows for distance calculation or approximation. According to various embodiments, the hand detection process (e.g., steps 601 to 605) is repeated until the hand is detected.

[0049] At step 602, the left and right images are stored in a memory. For example, a buffer memory may be used to store the left and right images. In various embodiments, the captured images are temporarily stored for processing and may be first stored in a volatile memory and subsequently transferred to a non-volatile memory. It is noted that the left and right images are stored with their respective metadata including timestamp information that allows for a selective hand detection process (e.g., using only one of the left and right images within a predetermined time interval for a hand detection process).

[0050] In step 603, a selection is alternately made between a first group of images and a second group of images, the first group of images including a first image associated with a first timestamp, the second group of images including a second image associated with a second timestamp, the first timestamp and the second timestamp being consecutive timestamps. According to a specific implementation, the left image and the right image may be alternately selected in a variety of ways. In a specific embodiment, frame numbers are assigned to images for alternate selection, wherein a frame index variable and its remainder are used to select between the left image and the right image. For example, a variable FrameIndex is used to record the current frame number of the image selection process, wherein the variable FrameIndex is associated with the image timestamp. For example, within a given time interval, the left image and the right image may share the same timestamp (or both timestamps are within a given time interval), the right image is selected when FrameIndex%2=0, and the left image is selected when FrameIndex%2=1. For the next time interval (or next timestamp), FrameIndex is increased by 1 (i.e., FrameIndex=FrameIndex+1), thereby switching from even to odd or from odd to even, and thus using the FrameIndex remainder calculation to select the alternating image (compared to the previous time interval). It is to be understood that according to embodiments of the present invention, the alternating selection between the left image and the right image can be implemented in a variety of ways.

[0051] Step 604, perform hand detection on the selected image. For example, based on the FrameIndex value updated with the timestamp, the left image or the right image can be selected as described. Moreover, hand detection is performed only on the selected images within a certain time interval (or according to the same timestamp). By performing hand detection using a single image within a certain time interval, the computational cost is greatly reduced (compared to the processing involving the left image and the right image in the same time interval). In addition, given that hand detection generally involves complex calculations, the XR device may not be able to process both the left image and the right image within a certain time interval (for example, 33 milliseconds for a frame rate of 30 frames per second), so performing hand detection using a single image at each time interval can help maintain the desired frame rate (for example, 30 frames per second).

[0052] In step 605, it is determined whether a hand is detected. In various embodiments, the bounding box used in hand detection is generated by locating a hand image and cropping the captured image. A first hand detection process is performed by a neural processing unit. If it is determined in step 605 that a hand is detected, step 606 and subsequent hand tracking steps are performed. On the other hand, if a hand is not detected, return to step 601 for additional hand image capture and processing. It should be noted that there may be variations in which steps are performed. For example, images may be captured and stored in batches, and the hand detection process may be repeated in step 602. In addition, the left and right images may be queued for processing using the frame indexes described above (e.g., to provide an alternating selection function in step 603), and the hand detection process 604 may be performed multiple times in iterations before repeating steps 601, 602, or 603.

[0053] In various cases, the computation and power savings of the hand detection process may no longer be required. Once a hand is detected, hand tracking, gesture recognition, and other processing can be performed, as described below. In this case, the variable FrameIndex is not updated, and both the left and right images are used, and hand tracking can be performed using a bounding box around the hand location.

[0054] At step 606, hand tracking is performed. If a hand is detected, a hand tracking process is initiated using the first image and the second image. The method also includes tracking the hand in the bounding box. For example, various techniques and processes may be performed, such as hand prediction, shape calibration, 2D-3D keypoint conversion, gesture recognition, etc. (e.g., such as Figure 5 ). Note that once the XR device is in hand tracking mode, it is assumed that hands are present, and therefore the hand detection process will not be performed until the hands are no longer captured by either the left or right camera and are therefore "lost".

[0055] In step 607, it is determined whether hand tracking is lost. If it is determined in step 607 that hand tracking is not lost, return to step 606 and maintain hand tracking. On the other hand, if hand tracking is lost, return to step 601 for additional hand image capture and processing. It is to be understood that hands may be lost in various ways and may require the XR device to take different actions. For example, a lost state may be related to a hand temporarily leaving the combined field of view of the two cameras; in some cases, the XR device may enter an inactive state (e.g., it is too dark to capture images that can be used for hand detection / tracking).

[0056] Figure 7 is a simplified flow chart showing the process of the hand tracking method according to an embodiment of the present invention. A person skilled in the art will recognize many variations, substitutions and modifications. For example, Figure 7 One or more of the steps shown in the claims may be added, deleted, repeated, modified, overlapped, rearranged, and substituted, which shall not limit the scope of the claims.

[0057] At block 710, hand tracking is in a "dead" state, where processes such as hand detection and hand tracking are not performed. For example, the XR device may be in the 710 state when idle (e.g., the image remains unchanged, with no movement or other types of changes). The XR device will remain in this "dead" state until activated (e.g., movement or change of the image is detected in the left image or the right image).

[0058] At block 720, the XR device starts and prepares to perform various tasks, such as hand detection and hand tracking. As part of the initialization process at block 720, the left and right cameras are active and capture images, which are stored for processing. In various implementations, variables such as FrameIndex are initialized to facilitate image selection for subsequent processing. Other processes may also be performed.

[0059] At box 730, hand tracking is performed, which includes hand detection (box 731) and hand prediction (box 732). Hand detection 731 is performed using an image selected between the left image and the right image per time interval (or timestamp). Hand detection 731 can be repeated until a hand is detected in one of the left image and the right image. Once a hand is detected, hand prediction 732 is performed. As part of the hand tracking process, hand prediction 732 can be repeated until the hand is lost or is not within the bounding box, in which case the hand prediction process can be performed. For example, if the hand being tracked is no longer within the bounding box, hand detection 731 can be performed to define a new bounding box; hand detection 731 can also determine that the hand no longer exists and continue to box 740.

[0060] At block 740, the XR device is in a "lost" state, in which a hand is no longer detected. In the lost state, the XR components and processing may still be active to detect hand motion, and if motion is detected in the image (e.g., the difference between two consecutive frames), it may return to block 730 to perform hand detection. For example, block 740 runs a loop for a predetermined time (as shown) before moving to the "dead" state in block 710. In some embodiments, blocks 710 and 740 may be implemented (or programmed) as the same state.

[0061] As an example, the following provides Figure 6 and Figure 7 Pseudo code of the implemented hand detection mechanism:

[0062]

[0063]

[0064] Return now Figure 5 The right hand thread 502r and the left hand thread 502l can be implemented symmetrically, receiving thread inputs from the right fisheye image and the left fisheye image, respectively. They also rely on their own bounding box tracking (i.e., boxes 504r and 504l). For example, confidence values ​​and bounding box tracking can be used to generate 3D hand key points that allow the recognition of gesture types.

[0065] Hand bounding box threads 504r and 504l provide tracking, and their input is bounding box size confidence value and bounding box prediction value from hand prediction box 503r and 503l. Hand bounding box threads 504r and 504l output hand state (e.g., whether it exists) and bounding box data, etc.

[0066] like Figure 5 As shown, if a hand is present (as determined in blocks 504r and 504l), 2D hand keypoint detection (e.g., blocks 505r and / or 505l) crops out the hand using a bounding box from hand bounding box tracking of the captured image. For example, the captured image is cropped and resized to a predetermined size (e.g., 96 pixels by 96 pixels, which is an optimally small size for efficient processing). 2D hand keypoint detection (e.g., blocks 505r and 505l) uses a 2D keypoint detection convolutional network on the resized image and outputs 2D hand keypoints. As described above, if 2D keypoints are present, they are mapped to 3D keypoints for gesture detection.

[0067] Although specific embodiments are fully described above, various modifications, alternative constructions and equivalents may be used. Therefore, the above descriptions and illustrations should not be taken as limiting the scope of the invention, which is defined by the appended claims.

Claims

1. A hand shape detection method, the method include: Capture the first set of images through the left camera; Capturing a second set of images via the right camera; storing the first set of images and the second set of images in a memory; alternately selecting between the first set of images and the second set of images, the first set of images comprising a first image associated with a first timestamp, the second set of images comprising a second image associated with a second timestamp, the first timestamp and the second timestamp being consecutive timestamps; performing a first hand detection process using the first image; Determining during the first hand detection process that a hand is not present; performing a second hand detection process using the second image; as well as In response to detecting a hand in the second hand detection process, a hand prediction process is performed.

2. The method according to claim 1, in, The hand prediction process uses the first image and the second image.

3. The method according to claim 1, further comprising: include: A frame index is generated, the frame index being updated based on at least the timestamp.

4. The method according to claim 1, in, The first hand detection process is performed by a neural processing unit.

5. The method according to claim 4, further comprising: include: When the hand is detected, a hand tracking process is initiated, the hand tracking process using the first image and the second image.

6. The method according to claim 1, further comprising: include: When it is determined that the hand is lost, a third hand detection process is performed.

7. The method according to claim 1, further comprising: include: A third hand detection process is performed using a third image, and a fourth hand detection process is performed using a fourth image, the first set of images including the third image, and the second set of images including the fourth image.

8. The method according to claim 1, further comprising: include: When a hand is detected, it is tracked within a bounding box.

9. The method according to claim 1, in, The first detection process and the second hand detection are performed within a predetermined time interval, the predetermined time interval being associated with a frame rate of at least twenty frames per second.

10. An extended reality device, include: A housing, the housing comprising a front side and a back side; a left camera, the left camera comprising a first sensor and a first lens, the first lens having a first field of view, the left camera being arranged at a left area of ​​the front side, the left camera being arranged to capture a first set of images, the first set of images comprising a first image having a first timestamp; a right camera, the right camera comprising a second sensor and a second lens, the second lens having a second field of view, the right camera being arranged in a right area of ​​the front face, the second field of view sharing a common field of view with the first field of view, the right camera being arranged to capture a second set of images, the second set of images comprising a second image having a second timestamp, the difference between the first timestamp and the second timestamp being within a predetermined interval; A display, arranged on the back side of the housing; a memory coupled to the first sensor and the second sensor and configured to store the first set of images and the second set of images; a first processor coupled to the memory; as well as a second processor coupled to the first processor and the memory, the second processor comprising a neural processing unit; in, The first processor is configured to alternately select between the first set of images and the second set of images; The second processor is configured to perform a first hand detection process using the first image; The second processor is further configured to: based on determining that there is no hand during the first hand detection process, perform a second hand detection using the second image; and The second processor is further configured to perform a hand prediction process when a hand is detected during the first hand detection process.

11. The device according to claim 10, in, The second processor is configured to perform the hand detection process at a rate of at least 20 frames per second.

12. The device according to claim 10, in, The second processor is further configured to detect an absence of a hand from the first set of images or the second set of images.

13. The device according to claim 10, in, The first lens includes a fisheye lens.

14. The device according to claim 10, in, The first sensor comprises a monochrome sensor having less than one million pixels.

15. The device according to claim 10, in, The capture rate of the left camera is at least 15 frames per second.

16. The device according to claim 10, in, The predetermined interval is less than 30 milliseconds.

17. A hand shape detection method, the method include: Capture the first set of images through the left camera; Capturing a second set of images via the right camera; storing the first set of images and the second set of images in a memory; alternately selecting between the first set of images and the second set of images; performing a first hand detection process using the first selected image; performing a second hand detection process using a second selected image based on detecting the absence of a hand in the first hand detection process; When a hand is detected in the first hand detection process or the second hand detection process, the hand is tracked in a bounding box.

18. The method according to claim 17, further comprising: include: Perform hand prediction process.

19. The method according to claim 17, in, The first set of images and the second set of images are selected using frame index numbers.

20. The method according to claim 17, further comprising: include: When no hand is detected in the third selected image, a switch is made from the hand detection process to the hand prediction process.

Images