Gesture detection method and system with hand shape calibration

Abstract

The invention relates to an augmented reality system and method. In an exemplary embodiment, hand images captured by at least two cameras are used in a hand shape calibration process, which associates the hand shape with a distance. The resulting calibration parameters are used for a subsequent gesture detection process. Other embodiments are also provided.

Description

Background of the Invention

[0002] The present invention relates to an extended reality system and method.

[0003] Over the past decade, Extended Reality (XR) devices, including Augmented Reality (AR) devices and Virtual Reality (VR) devices, have become increasingly popular. Important design considerations and challenges for XR devices include performance, cost, and power consumption. Due to various limitations, existing XR devices have been unable to meet the requirements, as explained below.

[0004] New and improved XR systems and methods thereof are desired. Summary of the invention

[0005] The present invention relates to an extended reality system and method. In an exemplary embodiment, hand images captured by at least two cameras are used in a hand shape calibration process, which associates hand shape with distance. The resulting calibration parameters are used in a subsequent gesture detection process. There are other embodiments.

[0006] A system of one or more devices may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which software, firmware, hardware, or a combination thereof causes the system to perform actions when run. One or more programs may be configured to perform specific operations or actions by including instructions, which, when executed by a data processing device, cause the device to perform actions. A general aspect includes a hand shape calibration method. The method also includes capturing a left image of a hand located at a first distance using a left lens, the hand having a first position and a first size in the left image, and the left lens having a first field of view. The method also includes storing the left image in a memory. The method also includes capturing a right image of the hand using a right lens, the hand having a second position and a second size in the right image, and the right lens having a second field of view. For example, the first size and the second size may be substantially the same. The method also includes storing the right image in a memory. The method also includes defining a common field of view based on an overlap between the first field of view and the second field of view. The method also includes calculating calibration parameters. The above-mentioned hand is located within the common field of view. The calibration parameters are based at least on the first size and the difference between the first position and the second position. The method also includes detecting gestures using at least the calibration parameters. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0007] Implementations may include one or more of the following features. The method may include determining a second distance using a difference between a first position and a second position. The calibration parameters may include a scaling factor based on the first size and a reference hand size. The method may include identifying left hand key points from a left image. The method may include identifying right hand key points and associating the left hand key points with the right hand key points. The method may include determining whether the hand is a left hand or a right hand. The left image is a monochrome image. The method may include performing image processing on the left image. The method may include tracking a hand. The method also includes processing gestures using a convolutional network. The left lens may include a fisheye lens or a wide-angle lens.

[0008] In various embodiments, the method may further include: capturing a gesture image of the gesture using the left lens or the right lens; cropping the gesture image; identifying a set of two-dimensional key points using the cropped gesture image, and generating a set of three-dimensional key points using the set of two-dimensional key points. Implementation of the described technology may include hardware, method or process, or computer software on a computer-accessible medium.

[0009] One general aspect relates to an extended reality device, which includes a housing having a front and a back. The device also includes a left camera, which includes a first sensor and a first lens, the first lens having a first field of view, and the left camera is configured in a left area of ​​the front. The device also includes a right camera, which includes a second sensor and a second lens, the second lens having a second field of view, and the right camera is configured in a right area of ​​the front, and the second field of view shares a common field of view with the first field of view. The device also includes a display configured on the back of the housing. The device also includes a memory coupled to the first sensor and the second sensor. The device also includes a processor coupled to the memory. The processor is configured to: calculate hand calibration parameters based on a left image captured by the left camera and a right image captured by the right camera, the left image includes a first hand located at a first position of the left image and within the common field of view, and the right image includes a first hand located at a second position of the right image and within the common field of view. The processor is also configured to determine the distance of the first hand based on the difference between the first position and the second position. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0010] Implementations may include one or more of the following features. The device stores the left image and the right image in a memory. The first lens may include a fisheye lens. The first sensor may include a monochrome sensor having less than one million pixels. The capture rate of the left camera is at least 15 frames per second. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer accessible medium.

[0011] One general aspect relates to a hand calibration method. The method includes capturing a left image of a hand at a first distance using a left lens, the hand having a first position and a first size in the left image, and the left lens having a first field of view. The method also includes capturing a right image of the hand using a right lens, the hand having a second position and a second size in the right image, and the right lens having a second field of view. For example, the first size and the second size can be substantially the same. The method also includes defining a common field of view based on an overlap between the first field of view and the second field of view. The method also includes calculating a hand scaling factor. The hand is located within the common field of view. The scaling factor is based at least on the first size and the difference between the first position and the second position. The method also includes capturing a gesture of the hand. The method also includes identifying two-dimensional key points associated with the gesture using at least the hand scaling factor. The method also includes identifying the gesture using at least the two-dimensional key points. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

[0012] Implementations may include mapping two-dimensional key points to three-dimensional key points.Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0013] It is to be appreciated that embodiments of the present invention provide many advantages over conventional techniques. In particular, the hand shape calibration technique can achieve effective gesture detection. In addition, the hand shape calibration technique according to embodiments of the present invention can be performed without using expensive range finding components such as a LiDAR module.

[0014] Embodiments of the present invention can be implemented in combination with existing systems and methods. For example, the hand calibration techniques according to the present invention can be used in various XR systems, including XR devices equipped with ranging components. In addition, various techniques according to the present invention can be applied to existing XR systems through software or firmware updates. There are other benefits.

[0015] The present invention achieves these and other benefits in the context of known technology.However, a further understanding of the nature and advantages of the present invention may be obtained by reference to the following portions of the specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A is a simplified diagram illustrating an extended reality (XR) device 115n according to an embodiment of the present invention.

[0017] Figure 1B is a simplified block diagram illustrating components of an extended reality device 115n according to an embodiment of the present invention.

[0018] Figure 2is a simplified diagram illustrating the field of view of a camera on an extended reality device 210 according to an embodiment of the present invention.

[0019] Figure 3A is a simplified diagram showing key points defined on the right hand side according to an embodiment of the present invention.

[0020] Figure 3B is a simplified diagram illustrating exemplary gestures according to an embodiment of the present invention.

[0021] Figure 4 is a simplified block diagram illustrating the functional blocks of an extended reality device according to an embodiment of the present invention.

[0022] Figure 5 is a simplified block diagram illustrating functional modules in a gesture detection algorithm according to an embodiment of the present invention.

[0023] Figure 6 is a simplified flow chart showing the process of a hand shape calibration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention relates to an extended reality system and method. In an exemplary embodiment, hand images captured by at least two cameras are used in a hand shape calibration process, which associates hand shape with distance. The resulting calibration parameters are used in a subsequent gesture detection process. There are other embodiments.

[0025] With the advent of virtual and augmented reality applications, gesture-based control schemes are becoming increasingly popular. The ability to accurately and efficiently reconstruct human hand movements from images promises to enable exciting new applications in immersive virtual and augmented reality, robotic control, and sign language recognition. Great progress has been made in recent years, especially with the advent of deep learning techniques. However, it remains a challenging task due to unconstrained global and local pose variations, frequent occlusions, local self-similarity, and high articulation. In recent years, commercial depth camera-based 3D hand tracking techniques on AR glasses have become popular, which directly measure the hands in 3D. Traditional research usually focuses on hand tracking algorithms based on RGB cameras, and compared to the algorithms, there is very limited research work on actual hand tracking systems.

[0026] Embodiments of the present invention provide a complete hand tracking system for AR glasses. The entire system can achieve a variety of features, including real-time tracking, two-hand tracking, shape calibration, and simultaneous gesture recognition. In various embodiments, a stereo fisheye camera with a low-power sensor is used with a fisheye lens to capture a large field of view (FoV). It should be noted that traditional three-dimensional (3D) hand tracking solutions for VR and AR applications use time-of-flight (ToF) devices to determine the depth map of the hand and output 3D hand key points. Although accurate, ToF devices also have disadvantages such as high cost, high complexity, and high energy consumption. It should be understood that embodiments of the present invention use stereo fisheye (or ultra-wide-angle) cameras instead of ToF modules for distance measurement. Using stereo cameras, embodiments of the present invention provide a complete system-wide solution that can involve features such as real-time on edge devices (e.g., mobile phones, embedded devices) to simultaneously achieve two-hand tracking and gesture recognition, and adjust the hand scale to match the real hand.

[0027] The following description is intended to enable one of ordinary skill in the art to make and use the invention and to incorporate it into the context of a particular application. Various modifications and various uses in different applications will be apparent to those skilled in the art, and the general principles defined herein may be applied to various embodiments. Therefore, the present invention is not limited to the embodiments described, but should have the widest scope consistent with the principles and novel features disclosed herein.

[0028] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without being limited to these specific details. In other cases, well-known structures and devices are shown in block diagram form, but are not shown in detail, to avoid obscuring the present invention.

[0029] Please note that all the documents and materials submitted at the same time as this specification and made available to the public at the same time as this specification are incorporated herein by reference. Unless otherwise expressly stated, all features disclosed in this specification (including any attached claims, abstracts and drawings) can be replaced by alternative features that serve the same, equivalent or similar purposes. Therefore, unless otherwise expressly stated, each feature disclosed is only an example of a series of equivalent or similar features.

[0030] In addition, any element in the claims that does not explicitly state a “means” for performing a specified function or a “step” for performing a specific function shall not be construed as a “means” or “step” provision as provided in 35 U.S.C. § 112, paragraph 6. In particular, the use of “step” or “action” in the claims herein is not intended to invoke the provisions of 35 U.S.C. § 112, paragraph 6.

[0031] Please note that labels such as left, right, front, back, top, bottom, forward, backward, clockwise, and counterclockwise, if used, are for convenience only and do not indicate any particular fixed direction. Instead, they are used to reflect the relative position and / or orientation of parts of an object.

[0032] Figure 1A 1 is a simplified diagram (top view) showing an extended reality device 115n according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. It is to be understood that the term "extended reality" (ER) is broadly defined, including virtual reality (VR), augmented reality (AR), and / or other similar technologies. For example, the ER device 115 shown can be configured as VR, AR, or other. Depending on the specific implementation, the ER device 115 may include a small housing for AR applications, or a relatively large housing for VR applications. Cameras 180A and 180B are configured on the front of the device 115. For example, cameras 180A and 180B are mounted on the left and right sides of the ER device 115, respectively. In various applications, additional cameras can be configured below cameras 180A and 180B to provide additional field of view and distance estimation accuracy. For example, cameras 180A and 180B each include an ultra-wide angle or fisheye lens that provides a large field of view, and they share a common field of view. Depending on the specific implementation, cameras 180A and 180B can be configured with different installation angles. For example, in VR applications where four or more cameras are installed on device 115, the cameras are configured with vertical and horizontal tilt angles to maximize the total field of view. AR applications typically use no more than two cameras, which can be slightly tilted to the side to increase the horizontal field of view, but they are substantially horizontal in the vertical direction when worn. In various embodiments, the vertical camera tilt is limited to less than 10 degrees in each direction, either upward or downward, to avoid potential occlusion (e.g., thick clothes that users may wear in winter, such as cotton clothes). Due to the arrangement of the two cameras, the distance between the two cameras (e.g., about 8-12 cm) and the resulting parallax (known factors) can be used to estimate the object distance. Display 185 is configured on the back of ER device 115. For example, in AR applications, display 185 can be a translucent display that superimposes information on an optical lens. In VR implementations, display 185 can include a non-transparent display.

[0033] Figure 1B is a simplified block diagram showing the components of an extended reality device 115 according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. In some embodiments, an XR head-mounted device (e.g., the AR head-mounted device 115n shown, etc.) may include, but is not limited to, at least one of a processor 150, a data storage device 155, a speaker or earphone 160, an eye tracking sensor 165, a light source 170, an audio sensor or microphone 175, a front or front-facing camera 180, a display 185, and / or a communication interface 190 and / or the like.

[0034] In some cases, processor 150 may be communicatively coupled (e.g., via a bus, via a wired connector, or via electrical pathways (e.g., traces and / or pads, etc.) of a printed circuit board (PCB) or integrated circuit (IC), and / or the like) to each of one or more of data storage 155, speaker or earphone 160, eye tracking sensor 165, light source 170, audio sensor or microphone 175, front camera 180, display 185, and / or communication interface 190, and / or the like. In various embodiments, data storage 155 may include dynamic random access memory (DRAM) and / or non-volatile memory. For example, images captured by camera 180 may be temporarily stored in DRAM for processing, and executable instructions (e.g., hand shape calibration and gesture recognition algorithms) may be stored in non-volatile memory. In various embodiments, data storage 155 may be implemented as part of processor 150 in a system-on-chip (SoC) arrangement.

[0035] The eye tracking sensor 165, which may include but is not limited to at least one of one or more cameras, one or more motion sensors, or one or more tracking sensors, etc., tracks the gaze point of the user's eyes and is combined with the computing processing of the processor 150, the computing system 105a or 105b, and / or the AI ​​system 140 to compare with the image or video taken in front of the ER device 115. The audio sensor 175 may include but is not limited to a microphone, a sound sensor, a noise sensor, etc., and may be used to receive or capture voice signals, sound signals, and / or noise signals, etc.

[0036] The front camera 180 includes respective lenses and sensors for capturing images or videos of the area in front of the ER device 115. For example, the front camera 180 includes Figure 1BThe cameras 180A and 180B are shown as being respectively arranged on the left and right sides of the housing. In different implementations, the sensor of the front camera can be a low-resolution monochrome sensor, which is not only energy-saving (no color filter and its color processing is required), but also relatively cheap in terms of device size and cost.

[0037] Figure 2 is a simplified diagram showing the field of view of the cameras on the extended reality device 210 according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. The left camera 180A is mounted on the left side of the ER device housing 210 and the right camera 180B is mounted on the right side of the ER device housing 210. Each camera has an ultra-wide angle or fisheye lens capable of capturing a wide field of view. For example, camera 180A has an angle of θ on the left side. L The camera 180B has a field of view on the right side and an angle θ R In various embodiments, cameras 180A and 180B are configured with an outward tilt angle θ R , to increase the combined field of view. In each embodiment, the tilt angle θ R The area 220 is within the field of view of both cameras, and additional processing can be performed on objects within the area 220. For example, the area 220 can be referred to as a "common FOV" that is defined by two FOVs and a distance; that is, additional calculations and processing can be performed within a predetermined distance (e.g., the user's arm length) and FOV. In various embodiments, the distance D between the two cameras 180A and 180B is approximately 8 cm to 12 cm, which is calibrated and optimized for parallax distance calculations. In a specific embodiment, the distance D is approximately 10 cm. In particular, since the hand 222 is within the field of view of both cameras 180A and 180B, the difference between the images captured by the two cameras, due to the distance D between the cameras 180A and 180B, can be used to approximate the distance between the hand 222 and the housing 210. For example, when the hand 222 is located in the area 220 as shown, processes such as hand shape calibration, three-dimensional mapping, etc. can be performed. In some embodiments, a shape calibration process (described in further detail below) is implemented as part of an initial calibration process, and the user is prompted to position her hand to the area 220 for shape calibration, and the calibration parameters generated in this process are later used for other calculations and processing.

[0038] Now return to reference Figure 1B . In various embodiments, the processor 150 is configured to calculate hand shape calibration parameters based on a left image captured by a left camera (e.g., camera 180A) and a right image captured by a right camera (e.g., camera 180B). The left image includes a hand (e.g., hand 222) located at a first position of the left image and within a common field of view (e.g., area 220). The right image includes a hand (e.g., hand 222) located at a second position of the right image and within the common field of view. The processor is configured to determine the distance of the hand based on the difference between the first position and the second position. Using information about the distance and size of the hand, the processor 150 can perform a hand shape calibration process that generates one or more calibration parameters.

[0039] In AR applications, the field of view of each front camera 180 overlaps the field of view of the user's 120 eyes, the captured image or video. Display screen and / or projector 185 can be used to display or project the generated image overlay (and / or display a composite image or video combined with the generated image overlay superimposed on the image or video of the actual area). Communication interface 190 provides wired or wireless communication with other devices and / or networks. For example, communication interface 190 can be connected to a computer for tethered operation, where the computer provides the processing power required for graphics-intensive applications.

[0040] Figure 3A is a simplified diagram showing key points defined on the right hand according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, key points 0-19 are assigned to different areas of the user's right hand. Based on the positions of these key points, gestures can be determined. For example, by identifying the relative positions of these key points 0-19, different gestures can be determined. In various embodiments, the relative positions of the key points are calibrated (e.g., measured in pixel distances) during the initial hand shape calibration process, which makes the gesture recognition process more accurate.

[0041] Figure 3B is a simplified diagram showing exemplary gestures according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, a hand image captured as a two-dimensional (2D) image by a device camera can be mapped into a 3D space for processing. As shown, a left hand gesture is converted into 3D key points, through which an extended reality device can recognize the gesture as an "OK" gesture. Additional processes may also be performed.

[0042] Figure 4is a simplified block diagram showing the functional blocks of an extended reality device according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. A person of ordinary skill in the art will recognize many variations, substitutions, and modifications. Figure 4 The system pipeline of the extended reality device 400 in the embodiment may include functional components, which may correspond to Figure 1B 115. At the front end, sensors, such as right fisheye camera 401, left fisheye camera 402, and inertial measurement unit (IMU) 403, capture images and other information and send the captured data to sensor processor 411 (e.g., a lightweight embedded CPU, such as Figure 1B 150 in the processor). The sensor processor 411 performs various simple image processing (e.g., denoising, exposure control, etc.), and then packages the processed data to the XR server 421. For example, the XR server 421 is implemented to act as a data consumer and transmits data to various algorithms 451, such as 3D hand tracking 431, 6DoF 441, etc. The location of the 3D hand tracking algorithm 431 is configured after the XR server 421 as shown, followed by the APP module 432. For example, the unified APP 432 receives the hand tracking results for different purposes, such as games, manipulation of virtual objects, etc. Additional functions, such as system rendering 434, asynchronous time warp (ATW) 435, and display 436, can be configured as shown. Depending on the specific implementation, there may also be other functional blocks.

[0043] Figure 5 is a simplified block diagram showing the functional modules in the gesture detection algorithm according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. A person of ordinary skill in the art will recognize many variations, substitutions, and modifications. According to various embodiments, it is possible to use Figure 1B The hand tracking system 500 implemented by the device 150 shown uses a two-hand tracking process for the left hand (l) and the right hand (r). For example, the system 500 provides real-time (i.e., 30 frames per second) hand tracking on an edge device, and it operates as a 3D hand tracking system. A stereo fisheye camera is used to acquire left and right images with known parallax calibration. The system includes a variety of algorithm sets, including hand acquisition 501, hand detection 502, hand prediction 503r and 503l, bounding box tracking 504r and 504l, 2D hand key point detection 505r and 505l, 3D hand key point detection 507r and 507l, gesture recognition 508r and 508l, and hand shape calibration 570.

[0044] In various embodiments, the hand shape calibration process is performed during an initial calibration process, which can be triggered by a new user. In particular, the hand shape calibration process relies on distance information calculated based on stereo images captured by left and right cameras, and the shape of the stereo hand image is related to the distance.

[0045] Figure 6 is a simplified flow chart showing the process of the hand shape calibration method according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. A person of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, one or more steps may be added, deleted, repeated, modified, replaced, overlapped, and rearranged, which should not limit the scope of the claims.

[0046] At step 610, hand images are captured by the left camera and the right camera, respectively, as long as the hand is within a common field of view that allows for camera-based distance calculations. For example, when the hand 222 (see Figure 2 ) is located in the common FOV 220, allowing the two cameras 180A and 180B to record their respective images, with the hand 222 being located in slightly different positions in the left frame and the right frame, and the difference in position allows distance calculation or approximation to be made.

[0047] At step 620, the left image and the right image are stored in a memory. For example, a buffer memory may be used to store the left image and the right image. In various embodiments, the captured images are temporarily stored for processing and may be first stored in a volatile memory and then transferred to a non-volatile memory.

[0048] In step 630, hand detection is performed on the left image and the right image stored in the memory. Depending on the specific implementation, various types of image recognition algorithms can be used. For example, a machine learning algorithm can be used to perform the hand detection process.

[0049] At step 640, it is determined whether the hand is within the common field of view. As described above, the common field of view is defined based on the overlap between the first field of view (e.g., the FOV of the left camera) and the second field of view (e.g., the FOV of the right camera). In various embodiments, the bounding box used in gesture detection is generated by locating the hand image and cropping the captured image. For example, Figure 2Area 220 in is defined as a common field of view. Since the XR device according to an embodiment of the present invention relies on images captured by at least two cameras and their known parallaxes, the hand distance can only be determined when the hand is within the common field of view and therefore appears in both images. If it is determined in step 640 that the hand is within the common field of view, step 650 and subsequent hand shape calibration steps are performed. On the other hand, if the hand is not within the common field of view (i.e., not present in one or both images), return to step 610 for additional hand image capture and processing. For example, the following pseudocode can be used to determine whether the hand is within the common field of view that allows distance calculation:

[0050] Hand3D=getHand3DPrediction()

[0051] leftCameraHand2D,rightCameraHand2D=projection(Hand3D)

[0052] insideStrereoView=leftCameraHand2D in Image&&rightCameraHand2D inImage

[0053] / / leftCameraHand2D and rightCameraHand2D are both (21,2) image coordinates, 21 is the total number of hand key points, 2 is the two-dimensional position (x, y),

[0054] / / Determine the key point p=(x,y) in the image

[0055] x>=0&&x<imageWidth&&y> =0&&y <imageHeight

[0056] / / Determine the leftCameraHand2D / rightCameraHand2D in the image, requiring all 21 hand key points to be within the image

[0057] At step 650, the hand shape is determined. For example, the term "hand shape" refers to various hand features, such as size, shape, hand key points, etc.

[0058] At step 660, the hand distance is determined using the captured images. For example, since the distance between the two cameras is known, the hand distance can be calculated using the difference between the hand positions of the images captured by different cameras. Other techniques may also be used depending on the specific implementation.

[0059] At step 670, one or more calibration parameters are created and / or updated. In a specific embodiment, the calibration parameters include a scaling factor, where the relative size of the hand (e.g., measured in pixels between two hand key points, such as Figure 3A The size scaling factor is calculated based on the distance between keypoint 8 and keypoint 19 in the image and its distance to the camera. For example, a "normal" reference hand size is assigned a default scaling factor of 10, while a relatively small hand may have a scaling factor of 8-9 and a relatively large hand may have a scaling factor of 10-11. In addition to the size scaling factor, the calibration parameters may also include the fingers (e.g., Figure 3A The distance between the key point 16 at the tip of the middle thumb and the key point 12 at the tip of the index finger, the size of each finger (e.g. Figure 3A The distance between key point 16 and key point 18), etc.

[0060] The calibration parameters are used in subsequent gesture detection processes. For example, if the shape calibration process has not been performed (e.g., the hand has never entered the common FOV area that allows distance calculation), the default calibration parameters (e.g., calibration for the reference hand shape) are used, and if the shape calibration process has been performed, the updated calibration parameters (e.g., updated in step 670) are used.

[0061] At step 680, one or more hand images are captured. The captured images may be stored in a memory and further processed (e.g., noise reduction, distance calculation, etc.). In various embodiments, the hand distance may be calculated using calibration parameters. For example, based on the size and scaling factor of the newly captured hand image, the hand distance may be estimated.

[0062] At step 690, gesture detection is performed using the calibration parameters. For example, calibration parameters such as hand size scaling factors can be used to improve accuracy during gesture detection. In various embodiments, gesture detection is performed using a conventional neural network while taking into account the calibration parameters. Depending on the specific implementation, the calibration parameters can be used in a variety of ways during gesture detection.

[0063] Return now Figure 5 The system 500 implements a set of outputs including 3D hand key points in blocks 507r and 507l. For example, the hand key points are as follows: Figure 3A and Figure 3B As shown. Note that although the captured images are two-dimensional, gesture detection is performed using 3D hand key points. For example, 2D to 3D mapping is performed between blocks 505l and 507l, and calibration parameters can be used in the mapping process.

[0064] As shown, the system 500 includes five components: a main thread 501, a hand detection thread 502, a right hand thread 502r, a left hand thread 5021, and a hand shape calibration thread 570. These components interact with each other.

[0065] As an example, the main thread 501 is used to copy images captured by the right fisheye camera 501r and the left fisheye camera 501l to the local memory of the system.

[0066] The hand detection thread 502 waits for the right fisheye image and the left fisheye image. Once the images are received, the hand detection thread 502 can use the hand detection convolutional network on the right fisheye image and the left fisheye image. For example, the hand detection thread 502 outputs confidence values ​​and bounding boxes for the right hand and the left hand.

[0067] The right hand thread 502r and the left hand thread 502l can be implemented symmetrically, receiving thread inputs from the right fisheye image and the left fisheye image, respectively. They also rely on their own bounding box tracking (i.e., boxes 504r and 504l). For example, confidence values ​​and bounding box tracking can be used to generate 3D hand key points that allow the recognition of gesture types.

[0068] Hand bounding box threads 504r and 504l provide tracking, and their input is the bounding box size confidence value and bounding box prediction value from hand prediction box 503r and 503l. Hand bounding box threads 504r and 504l specifically output hand status (e.g., whether it exists) and bounding box data, etc.

[0069] like Figure 5 As shown, if a hand is present (as determined in blocks 504r and 504l), 2D hand keypoint detection (e.g., blocks 505r and / or 505l) crops out the hand using a bounding box from hand bounding box tracking of the captured image. For example, the captured image is cropped and resized to a predetermined size (e.g., 96 pixels by 96 pixels, which is an optimally small size for efficient processing). 2D hand keypoint detection (e.g., blocks 505r and 505l) uses a 2D keypoint detection convolutional network on the resized image and outputs 2D hand keypoints. As described above, if 2D keypoints are present, they are mapped to 3D keypoints for gesture detection.

[0070] Although specific embodiments are fully described above, various modifications, alternative constructions and equivalents may be used. Therefore, the above descriptions and illustrations should not be taken as limiting the scope of the invention, which is defined by the appended claims.

Claims

1. A hand shape calibration method, the method comprising: capturing a left image of a hand at a first distance using a left lens, the hand having a first position and a first size in the left image, the left lens having a first field of view; storing the left image in a memory; capturing a right image of the hand using a right lens, the hand having a second position in the right image, the right lens having a second field of view; storing the right image in the memory; defining a common field of view based on an overlap between the first field of view and the second field of view; calculating calibration parameters, the hand being located within the common field of view, the calibration parameters being based on at least the first size and a difference between the first position and the second position; as well as A gesture is detected using at least the calibration parameters.

2. The method according to claim 1, further comprising: Using the difference between the first position and the second position, a second distance is determined.

3. The method according to claim 1, wherein: The calibration parameters include a scaling factor based on the first size and a reference hand size.

4. The method according to claim 1, further comprising: Left hand keypoints are identified from the left image.

5. The method according to claim 4, further comprising: Right hand key points are identified and the left hand key points are associated with the right hand key points.

6. The method according to claim 1, further comprising: Determine whether the hand is the left hand or the right hand.

7. The method according to claim 1, wherein: The left image is a monochrome image.

8. The method according to claim 1, further comprising: Image processing is performed on the left image.

9. The method according to claim 1, further comprising: The hand is tracked.

10. The method according to claim 1, further comprising: The gestures are processed using a convolutional network.

11. The method according to claim 1, wherein: The left lens includes a fisheye lens or a wide-angle lens.

12. The method according to claim 1, further comprising: capturing a gesture image of the gesture using the left camera or the right camera; cropping the gesture image; Using the cropped gesture image, a set of 2D key points are identified; as well as Generate a set of 3D keypoints using the set of 2D keypoints.

13. An extended reality device, comprising: A housing, the housing comprising a front side and a back side; A left camera, the left camera comprising a first sensor and a first lens, the first lens having a first field of view, and the left camera is arranged in a left area of ​​the front side; a right camera, the right camera comprising a second sensor and a second lens, the second lens having a second field of view, the right camera being arranged in a right area of ​​the front side, the second field of view sharing a common field of view with the first field of view; A display, arranged on the back side of the housing; a memory coupled to the first sensor and the second sensor; a processor coupled to the memory; Wherein, the processor is configured to: calculate hand shape calibration parameters based on a left image captured by the left camera and a right image captured by the right camera, the left image including a first hand located at a first position of the left image and within the common field of view, and the right image including the first hand located at a second position of the right image and within the common field of view; the processor is also configured to determine the distance of the first hand based on the difference between the first position and the second position.

14. The device according to claim 13, wherein: The left image and the right image are stored in the memory.

15. The device according to claim 13, wherein: The processor also.

16. The device according to claim 13, wherein: The first lens includes a fisheye lens.

17. The device according to claim 13, wherein: The first sensor comprises a monochrome sensor having less than one million pixels.

18. The device according to claim 13, wherein: The capture rate of the left camera is at least 15 frames per second.

19. A hand shape calibration method, the method comprising: capturing a left image of a hand at a first distance using a left lens, the hand having a first position and a first size in the left image, the left lens having a first field of view; capturing a right image of the hand using a right lens, the hand having a second position in the right image, the right lens having a second field of view; defining a common field of view based on an overlap between the first field of view and the second field of view; calculating a hand scaling factor, the hand being located within the common field of view, the scaling factor being based on at least the first size and a difference between the first position and the second position; capturing a gesture of the hand; identifying two-dimensional key points associated with the gesture using at least the hand scaling factor; as well as The gesture is recognized using at least the two-dimensional key points.

20. The method according to claim 19, further comprising: The two-dimensional key points are mapped to three-dimensional key points.

Images