Adaptive hand tracking and gesture recognition based on face-shoulder feature coordinate transformation

Abstract

A method for recognizing hand gestures when the hand is held in front of the user's upper body. The method detects primary features associated with the upper body, including features on the face (eyes and mouth, etc.) and the shoulder tips. The geometric relationships between these features are used to determine if the face / head has turned to the side relative to the shoulders. If not, the face-shoulder features are used to establish a face-shoulder coordinate system, and the hand image is transformed into this new coordinate system. This effectively defines the hand gestures relative to the user's face and shoulders, significantly reducing the size of the feature space for the hand gestures. This face-shoulder coordinate transformation can be combined with traditional appearance-based gesture recognition methods, or with traditional or hybrid deep neural networks for RGB or RGB-D images.

Description

Technical Field

[0001] This invention relates to human-machine interfaces, and more particularly to systems and methods for adaptive hand tracking and gesture recognition. Technical Background

[0002] Hand tracking and gesture recognition systems are used to create interactions between computers or machines and human users, where recognized gestures from the human user can be used to control the machine or convey information. This falls under the fields of human-computer interface (HCI) or human-machine interface (HMI).

[0003] One approach to meeting the advanced requirements of gesture-based HCI is glove-based sensing, which uses sensors (mechanical or optical) attached to the glove. The glove converts finger flexion into electrical signals to determine hand gestures. This approach is not natural because it requires the user to wear special gloves with wires and perform calibration and setup.

[0004] Another, more natural approach is vision-based sensing, which uses a camera to capture gestures. Gestures can be static (gestures) or dynamic (gesture sequences). Natural gesture acquisition vision systems use color cameras (called RGB cameras), depth cameras, or a combination of both (called RGB-D cameras, which detect color and depth information of the scene).

[0005] Methods for vision-based gesture recognition can be categorized into model-based methods and appearance-based methods. Model-based methods generate 3D model hypotheses of gestures and, upon acquisition...

[0006] The image is evaluated. This is essentially an optimization problem, where the cost function measures the distance between the expected image and the actual image due to model assumptions. Up to 28 degrees of freedom (DOF) have been used in various 3D gesture models, resulting in a significant computational burden on model-to-image transformations and optimization routines, although 3D input from a depth camera can significantly simplify the computation of model-based methods.

[0007] Appearance-based methods directly use information contained in images and assume there is no explicit 3D model of the hand. When only the appearance of the hand is known, distinguishing between gestures typically involves some statistical classifier using a set of image features of the hand, which can be extracted from the RGB or depth images of the hand, or a combination of both. It should be noted that in recent years, deep neural networks (DNNs) have been widely used as image classifiers, and they are equivalent to traditional statistical classifiers with combined image feature detectors, but they generally require more samples to train and can produce more robust classification results when done correctly.

[0008] Due to individual differences in hand appearance, gesture habits, and body orientation or movement, training a gesture classifier requires a large number of ground truth samples, which in turn requires significant effort and cost in sample collection and labeling.

[0009] Summarize

[0010] For many HCI applications, when a person performs a gesture, the hand is held in front of the upper body. There are several key features associated with the human upper body that are relatively easy and robust to detect, such as facial features (eyes and mouth, etc.), shoulder endplates (approximately the upper and outer endpoints of each shoulder), etc. Based on this insight, in embodiments of the invention, in the first step of hand tracking and gesture recognition, these features on the face and shoulders can be detected, and their geometric relationships can be used to determine whether the face / head has turned sideways relative to the shoulders. If not, these face-shoulder features can be used to establish a coordinate system, and the hand image can be transformed into this new coordinate system. This essentially defines the gesture relative to each user's face and shoulders, thereby significantly reducing the size of the gesture's feature space. This face-shoulder coordinate transformation can be combined with conventional appearance-based gesture recognition methods, or with a hybrid deep neural network for RGB-D images, which performs separate preprocessing of the depth image instead of the conventional convolutional layers for RGB images. Summary of the Invention

[0011] One object of the present invention is to provide an improved method for hand tracking and gesture recognition.

[0012] Additional features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practice of the invention. The objects and other advantages of the invention will be realized and obtained through the written description and its claims, as well as the structures particularly pointed out in the drawings.

[0013] To achieve the above objectives, the present invention provides a method for hand tracking and gesture recognition, the method comprising: receiving a current image from a camera system, the current image containing the face and shoulders of a human user;

[0014] Based on the current image, detect the user's facial and shoulder features; establish a current face-shoulder coordinate system based on the detected facial and shoulder features; transform a previous hand target region from the previous face-shoulder coordinate system to the current face-shoulder coordinate system to generate a transformed previous hand target region, which is a target region containing the user's hand and generated based on a previous image received from the camera system; extract a current hand target region from the current image, wherein the current hand target region is defined in the current face-shoulder coordinate system; and perform gesture recognition using the transformed previous hand target region and the current hand target region in the current face-shoulder coordinate system.

[0015] In some embodiments, the detection step includes detecting the user's eyes, mouth, and shoulders, and calculating the user's current left eye center position, right eye center position, current mouth center position, and current left shoulder bend end position and current right shoulder bend end position, and wherein the establishment step includes establishing the current face-shoulder coordinate system based on the current left eye center position, the current right eye center position, the current mouth center position, and the current left shoulder bend end position and current right shoulder bend end position.

[0016] In another aspect, the present invention provides a computer program product comprising a computer-usable non-transitory medium (e.g., a memory or storage device) having computer-readable program code embedded therein for controlling a data processing apparatus (e.g., a computer), the computer-readable program code being configured to cause the data processing apparatus to perform the methods described above.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory, and are intended to provide further explanation of the claimed invention. Attached Figure Description

[0018] Figure 1 (a) and Figure 1 (b) A vision-based gesture acquisition system in which embodiments of the present invention can be implemented is illustrated schematically. Figure 1 (a) illustrates an image acquisition and processing system; Figure 1 (b) shows the calibration output of the RGB-D camera.

[0019] Figure 2 This is a flowchart illustrating a hand tracking and gesture recognition process according to an embodiment of the present invention.

[0020] Figure 3 It is shown Figure 2The flowchart shown illustrates the adaptive hand tracking and gesture recognition process.

[0021] Figure 4 The face-shoulder coordinate system is illustrated schematically, showing the definition of key face-shoulder features and the right-hand face-shoulder coordinate system.

[0022] Figure 5 It is shown Figure 2 and Figure 3 The flowchart shown illustrates the process of initializing the face-shoulder coordinate system.

[0023] Figure 6 It is shown Figure 3 The flowchart shown illustrates the face-shoulder coordinate system update process.

[0024] Figure 7 A hybrid gesture recognition method according to an embodiment of the present invention is illustrated schematically. Detailed Implementation

[0025] The hand tracking and gesture recognition method according to embodiments of the present invention is based on the inventors' insights and recognition of the following facts and assumptions: For many HCI applications, most of the time when a person performs a gesture, the hand is held in front of the upper body. There are some key features associated with the human upper body that are relatively easy and robust to detect, such as facial features (eyes, eyebrows, mouth, and nose, etc.), shoulder endcaps (approximately the upper and outer ends of each shoulder), etc. Therefore, in embodiments of the present invention, in the first step of hand tracking and gesture recognition, these features on the face and shoulders (referred to as face-shoulder features in this disclosure) are detected, and their geometric relationships are used to determine whether the face / head has turned to the side relative to the shoulders. If not, a coordinate system (referred to as the face-shoulder coordinate system) is established using these face-shoulder features, and the hand image is transformed into this new coordinate system. This essentially defines the gesture relative to each user's face and shoulders, thereby significantly reducing the dimensionality of the gesture feature space. It is possible that one or both face-shoulder features are missing due to hand occlusion, since the hand is most likely between the face-shoulder plane and the camera; in this case, the face-shoulder coordinate system can also be updated at a cost by matching those still visible face-shoulder features with historical records (previous image frames during the same gesture recognition session).

[0026] The key advantages of embodiments of the present invention are that they significantly reduce the dimensionality of the feature space of gestures, thereby (1) reducing the dimensionality associated with gesture sample collection and annotation; (2) improving the robustness of hand tracking and the accuracy of gesture recognition; and (3) enabling the use of simpler algorithms for hand tracking and gesture recognition.

[0027] The preferred embodiment of the hand tracking and gesture recognition system includes an RGB-D camera connected to a computer or machine (hereinafter referred to as "machine" for simplicity). The machine includes one or more processors and non-volatile memory storing machine-executable program instructions, and / or logic circuitry configured to process signals and data to implement the hand tracking and gesture recognition methods described below. The machine may also include a display screen or speaker for displaying or otherwise conveying information to a user. The machine also includes a camera controller and a necessary processor that controls image acquisition and performs the calculations detailed below, both in real time (see [link to relevant documentation]). Figure 1 (a)). RGB and depth information are complementary and can simplify the hand tracking and gesture recognition process and improve robustness. However, it should be noted that the system can also be implemented using only an RGB camera or only a depth camera with appropriate modifications to the method described below. As an example, this disclosure uses an RGB-D camera and the resulting RGB and depth images in the following description.

[0028] Before use, calibrate the RGB-D camera to establish the intrinsic parameters and spatial relationships (extrinsic parameters) of each RGB and depth camera within the RGB-D camera system. The camera's intrinsic parameters include, but are not limited to, focal length, principal point, radial and tangential distortion, etc. The extrinsic parameters between the RGB and depth cameras include rotation matrices and translation vectors (see [link to documentation]). Figure 1 (b) Any suitable method can be used to perform camera calibration, which is generally known to those skilled in the art.

[0029] Figure 2 The entire hand tracking and gesture recognition process according to an embodiment of the present invention is illustrated. When the subject (user) is in front of the camera (step S201), an RGB-D image is captured, corrected, and aligned (step S202). The alignment (or fusion) of the RGB and depth images is completed by using the aforementioned rotation matrix and translation vector to generate an RGB-D image pair. Then, the face-shoulder coordinate system XYZ is initialized using features detected from the RGB-D image pair (step S203, further described in detail below). The machine then instructs the subject to begin a gesture to interact with the machine (step S204), and the system continuously captures RGB-D images (step S205) and performs adaptive hand tracking and gesture recognition (step S206, further shown in detail below) until a predefined "end" gesture is recognized or another stop command is received ("Yes" in step S207). When no "end" gesture is recognized and no other stop command is received ("N" in step S207), the system performs the task corresponding to the recognized gesture according to predefined rules (step S208).

[0030] Figure 3 This illustrates the process of adaptive hand tracking and gesture recognition using face-shoulder feature coordinate transformation. Figure 2 (S206 in the example). Perform the following on the Nth image. Figure 3 The steps shown are as follows. Initialization of the face-shoulder coordinate system (step S310 or...). Figure 2 After step S203, whenever a new (Nth or "current") corrected and aligned RGB-D image pair (data D301 and "Y" from step S301) is received, the system attempts to update the face-shoulder coordinate system with image features from the current image pair to generate an updated (current) face-shoulder coordinate system XYZ. N (Step S302). Before performing hand tracking and gesture recognition in the updated face-shoulder coordinate system (steps S304-S309), the previous (N-1)th hand image pair (the hand ROI in data D302) will be included. (N-1) The hand target area (ROI) in the previous face-shoulder coordinate system XYZ is converted from the target area. (N-1) Transform to the updated face-shoulder coordinate system XYZ N (Step S303). Transformation step S303 is performed by calculating the transformation from the previous face-shoulder coordinate system to the current face-shoulder coordinate system and then applying the same transformation to the previous hand ROI to generate the transformed previous hand ROI. This transforms the hand image pairs from different images (N-1 and Nth) to the same coordinate system (XYZN) before applying hand tracking and gesture recognition, and effectively serves as a normalization process to reduce gesture variations due to body rotation or other body movements. It should be noted that, in a preferred embodiment, the coordinate transformation is performed only when face-shoulder rotation or other movements are consistent. For example, if only the head rotates and the shoulders do not rotate, the geometric relationship check between features will detect it, and no coordinate system update will be performed. In other words, in a preferred embodiment, step S303 includes first determining whether face-shoulder rotation or other movements are consistent, and updating the face-shoulder coordinate system only when face-shoulder rotation or other movements are consistent.

[0031] After transforming the hand image to the updated face-shoulder coordinate system (step S303), hand tracking and gesture recognition can be implemented using any suitable algorithm, one example of which is... Figure 3 Steps S304-S309 are shown. In this example, the current face-shoulder coordinate system XYZ is first established. NIn step S304, hand tracking is performed to attempt to generate hand movements (translation and rotation) that move the hand from the previous (N-1)th hand ROI to the current (Nth)th image. This tracking is performed in the current face-shoulder coordinate system XYZ. N This process is performed because the previous hand ROI has been transformed into this coordinate system. If hand tracking is successful ("Yes" in step S305), the current hand ROI is extracted based on the tracking, i.e., the ROI in the current image containing the user's hand (step S308). The current hand ROI is defined in the current face-shoulder coordinate system. If hand tracking is unsuccessful ("No" in step S305), a hand search window is defined in the current image based on user statistics or a predefined window size (step S306), a suitable image recognition algorithm is used to detect the hand in the hand search window (step S307), and the current hand ROI is extracted from the current image based on the detected hand (step S308). Then, gestures are recognized based on the hand ROIs from the current image and one or more previous images, or the current hand ROI is... N Add it to the hand ROI queue for gesture recognition in subsequent iterations (step S309).

[0032] Note that in simple hand tracking methods, only a single previous ROI and current ROI are needed to perform hand tracking. In more complex tracking methods, multiple (two or more) previous ROIs are used for hand tracking. In the latter case, in step S303, the multiple previous ROIs already in the queue are transformed into the updated (current) face-shoulder coordinate system XYZ. N To perform tracing.

[0033] The face-shoulder coordinate system is defined on some key features in the face-shoulder region of the body, such as... Figure 4 As shown. Key features include two eyes (E). R and E L ) and mouth (M C ), and the top of the shoulder (S R and S LThe face-shoulder coordinate system can be defined, for example, as the point with the highest curvature in the outer contour of the shoulder. When one or both eyes are obscured by a hand, one or both eyebrows can be used instead, and when the mouth is obscured by a hand, the nose can be used instead. For simplicity, for all the processes described below, "eye" can refer to two eyes or two eyebrows or one eye and one eyebrow, and "mouth" can refer to the mouth or nose, although their detection may involve slightly different algorithms. Only the central locations of these features are used to establish the face-shoulder coordinate system. Replacing the eyes with eyebrows and the mouth with a nose introduces only very small errors at the origin of the face-shoulder coordinate system, and these effects can be ignored for hand tracking and gesture recognition purposes. Optionally, positional corrections can be applied before using eyebrows or nose as eye or mouth locations, for example, based on certain assumptions about the average distance between eyebrows and eyes, and between nose and mouth. The face-shoulder coordinate system essentially assumes that key features (or their centers) are in the same plane.

[0034] Figure 5 The text describes in detail how to initialize the face-shoulder coordinate system. Figure 2 Step S203 and Figure 3 Step S310). Based on the corrected RGB-D image pair (data D501), this process detects face-shoulder (FS) features in the RGB-D image (step S501) and extracts the corresponding face-shoulder target region (FSROI) from the RGB-D image (step S502). Then, this process detects the eyes and mouth in the extracted ROI (step S503) and calculates the center E of the left eye. L and right eye center E R and the center of the mouth M C (Step S504). This process also detects the shoulder in the extracted ROI (step S505) and calculates the apex S of the left shoulder bend. L and the S-shaped bend at the right shoulder R (Step S506). This process calculates the eye center EL and ER, and the mouth center M based on the inherent parameters of the depth camera (data D502). C and the S-shaped end of the shoulder bend L and S R The 3D coordinates are determined (step S507). Note that these 3D coordinates are defined in the camera's coordinate system, not in the face-shoulder coordinate system.

[0035] Then, the process calculates the eye center E. L and E R Mouth center M C and the top S of the shoulder bend L and S RThe topology is determined (step S508), and whether the topology is ideal is determined in the following manner (step S509).

[0036] During this initialization phase, since the subject is instructed to face the camera in a roughly forward and centered position, the subject's face-shoulder plane will generally not rotate too far from the plane perpendicular to the camera's optical axis. Examine the geometry of the five key feature centers to confirm this. If the subject's body is not rotated too far from the vertical plane (perfect or near-perfect orientation), the following ratio should be close to 1:

[0037]

[0038]

[0039]

[0040] The ratios within a specific range of 1 can be considered "ideal" (e.g., the range can be chosen in such a way that the subject's face-shoulder plane rotates no more than 30 degrees from a plane perpendicular to the camera's optical axis). This is to prevent the subject from facing the camera in a substantially rotated manner, which could lead to problems with hand tracking and gesture recognition. These initial topological ratios are recorded for future verification of whether the subject has rotated his face relative to his shoulders. The ratios in Equations 1-3 are merely examples of topologies that can define five key features; other parameters can be appropriately used to define the topologies of the five key features.

[0041] If the topology is ideal ("Yes" in step S509), then the eye center E L and E R Mouth center M C and / or the top S of the shoulder bend L and S R The position is used to establish an orthogonal face-shoulder coordinate system XYZ based on a set of predefined formulas (step S510). In one example (see...) Figure 4 (1) The origin O of the face-shoulder coordinate system is defined as the top of the left shoulder bend S. L and the S-shaped bend at the right shoulder R The midpoint between (2) and (M) is defined as the X direction (O, M). C (3) The Y direction is defined as (O, S) R The direction of (S) and the direction of (4) Z are defined as being related to the direction of (S). L S R M C The planes are defined as orthogonal. The unit of length can be based on distance E. L E R or S L SR This can be used to define the face-shoulder coordinate system. Other suitable conventions and formulas can also be used to define the face-shoulder coordinate system.

[0042] If the topology is not ideal ("No" in step S509), a command is displayed reminding the user to rotate toward the camera (step S511). For example, if E R S R E L S L If the command is long, the instruction will direct the user to turn right. Then, the RGB-D image is acquired again (captured, corrected, and blended) (step S512), and steps S501 to S509 are repeated using the newly acquired RGB-D image to establish the face-shoulder coordinate system. This coordinate initialization process generates the ROI of the FS region, eyes, mouth, and shoulders, as well as the face-shoulder coordinate system XYZ (data D503).

[0043] Figure 6 The details of how gesture interaction with the machine begins are described in detail. Figure 3 Step S302) describes how to update the face-shoulder coordinate system. There are some important differences between updating and initializing the face-shoulder coordinate system. First, in the update process, face-shoulder feature detection utilizes tracking (step S601) to quickly find the face-shoulder ROI whenever possible to reduce computation. Tracking is performed on the RGB-D image (data D601) using the face-shoulder ROI and related features from the previous image (data D602) to attempt to generate translations (translations and rotations) that move features from the previous image to features in the current image. Based on successful tracking ("Yes" in step S602), the FSROI is extracted (step S603), and FS features (eyes, mouth, shoulders) are detected in the FSROI (step S604). If tracking fails ("No" in step S602), the process only backtracks to the features (eyes, mouth, shoulders) to detect them across the entire image (step S605).

[0044] If the FS features (eyes, mouth, shoulders) are successfully detected ("Yes" in step S606), the process calculates the eye center E. L and E R Mouth center M C and the top S of the shoulder bend L and S R (Step S607) and calculate the 3D coordinates (in the camera coordinate system) of the five key features using the depth camera intrinsic parameters (data D603), and calculate the topology of the five key features (Step S608). In step S608, the topology is calculated from the 3D coordinates using the same equations 1-3 as in step S508 of the initialization process.

[0045] The second difference between updating and initialization is that, in updating, it is checked whether the key feature topology is an affine transformation of the initial topology calculated in step S508 (step S609). Since it is assumed that the key features are in a plane, this can be done by simply comparing the three ratios in Equations 1-3 separately to see if they are respectively equal to the three ratios of the initial topology within a predetermined threshold. If the key feature topology is an affine transformation of the initial topology ("Yes" in step S609), then the updated face-shoulder coordinate system XYZ is established in a manner similar to step S510. N (Step S610).

[0046] The third difference between updating and initialization is that, in updating, if the hand gesture occludes one or more key face-shoulder features (note that since eyes and eyebrows on the same side are considered the same feature, a feature is considered occluded if and only if both eyes and eyebrows on the same side are occluded) ("No" in step S606), other successfully detected face-shoulder features or general local image features (e.g., SURF (accelerated robust features), FAST (features from accelerated segmentation tests), BRIEF (binary robust independent basic features), and ORB (orientational FAST and rotational BRIEF), etc.) can be used to establish a mapping relationship (including rotation matrix R and translation vector T) that maps the previous face-shoulder ROI to the current face-shoulder ROI (steps S611 and S612), and then the same mapping (R and T) can be applied to the previous face-shoulder coordinate system XYZ. (N-1) This relationship is used to update the face-shoulder coordinate system, and the mapped face-shoulder coordinate system is used as the updated face-shoulder coordinate system XYZ. (N) (Step S613). In the special case where image feature mapping fails, mathematically it is equivalent to the rotation matrix R being an identity matrix and the translation vector being zero, so the update of the face-shoulder coordinate system remains the same as the previous one.

[0047] In step S610 or S613, an updated face-shoulder coordinate system XYZ is established. (N) Then, the FSROOI and FS feature ROI are updated to the updated face-shoulder coordinate system (step S614), and the updated data is stored (data D604).

[0048] Once the hand ROI is transformed into adaptive face-shoulder coordinates, gesture recognition can be performed using traditional appearance-based or model-based methods. Alternatively, methods such as... Figure 7The hybrid deep neural network (DNN) shown is used to perform gesture recognition, with the modules within the dashed boxes forming the hybrid DNN. The depth component of RGB-D is processed separately from the earlier convolutional layers of the RGB components because convolution is not an ideal way to extract low-level features from depth images. Face-shoulder coordinate transformation significantly reduces the complexity of the DNN (with far fewer layers and parameters) and alleviates the burden of sample collection and annotation required to train the originally large DNN, significantly reducing training and prediction time and making it easier to deploy on low-cost hardware platforms.

[0049] It will be apparent to those skilled in the art that various modifications and variations can be made to the hand tracking and gesture recognition method and related apparatus of the present invention without departing from the spirit or scope of the invention. Therefore, the present invention is intended to encompass modifications and variations falling within the scope of the appended claims and their equivalents.

Claims

1. A method for hand tracking and gesture recognition implemented in a computer system, comprising: Receive a current image from the camera system, the current image containing the face and shoulders of a human user; Based on the current image, detect the user's facial and shoulder features; A current face-shoulder coordinate system is established based on the detected facial and shoulder features; The previous hand target region is transformed from the previous face-shoulder coordinate system to the current face-shoulder coordinate system to generate a transformed previous hand target region, which is a target region containing the user's hand and is generated based on a previous image received from the camera system. Extract the current hand target region from the current image, wherein the current hand target region is defined in the current face-shoulder coordinate system; as well as Gesture recognition is performed using the transformed previous hand target region and the current hand target region in the current face-shoulder coordinate system.

2. The method according to claim 1, in, The detection steps include detecting the user's eyes, mouth, and shoulders, and calculating the user's current left eye center position, right eye center position, mouth center position, and the current left shoulder bend tip position and right shoulder bend tip position. The establishment step includes establishing the current face-shoulder coordinate system based on the current left eye center position, the current right eye center position, the current mouth center position, the current left shoulder bend end position, and the current right shoulder bend end position.

3. The method according to claim 2, wherein, The establishment steps include: The origin of the face-shoulder coordinate system is defined as the midpoint between the current left shoulder bend and the current right shoulder bend. The X-direction of the face-shoulder coordinate system is defined as the direction from the origin to the current center of the mouth; and The Y direction of the face-shoulder coordinate system is defined as the direction from the origin to the current left shoulder bend or the current right shoulder bend, and the Z direction of the face-shoulder coordinate system is defined as the direction orthogonal to the plane formed by the current left shoulder bend, the current right shoulder bend, and the current mouth center.

4. The method of claim 2, further comprising, after the detection step and before the establishment step: Calculate the current topology of the current left eye center position, the current right eye center position, the current mouth center position, the current left shoulder bend end position, and the current right shoulder bend end position; as well as The current topology is evaluated to confirm that the plane formed by the user's facial and shoulder features is within a predetermined angular range relative to the optical axis of the camera system.

5. The method according to claim 2, further comprising, after the detection step and before the establishment step: Calculate the current topology of the current left eye center position, the current right eye center position, the current mouth center position, the current left shoulder bend end position, and the current right shoulder bend end position, and The current topology is determined to be an affine transformation of the initial topology, which is calculated based on the initial left eye center position, the initial right eye center position, the initial mouth center position, and the initial left shoulder bend top position.

6. The method according to claim 5, wherein, Each of the current topology and the initial topology is defined by three ratios, the three ratios being: Among them, E L and E R It refers to the current or initial left and right eye centers, M. C It is the current or initial center of the mouth, S L and S R It is the current or initial shoulder bend end, and When the three ratios of the current topology are equal to the three ratios of the initial topology within a predefined threshold, the current topology is determined to be an affine transformation of the initial topology.

7. The method according to claim 1, wherein, The detection steps include: Obtain a previous face-shoulder target region, which is a target region in a previous image that includes the user's face and shoulders; The features in the previous face-shoulder target region are traced to the features in the current image; Based on the tracking, the current face-shoulder target region is extracted from the current image; and Detect the user's eyes, mouth, and shoulders in the current face-shoulder target area.

8. The method according to claim 1, wherein, The establishment steps include: Obtain a previous face-shoulder target region and a previous face-shoulder coordinate system, wherein the previous face-shoulder target region is the target region in the previous image that includes the user's face and shoulders, and the previous face-shoulder coordinate system has been established for the previous image; A mapping is established, which maps the previous face-shoulder target region to the current face-shoulder target region based on face-shoulder features or local image features in the corresponding face-shoulder target region, the mapping including rotation and translation; and The mapping is applied to the previous face-shoulder coordinate system to generate the current face-shoulder coordinate system.

9. The method according to claim 1, wherein, The steps for extracting the current hand target region include: Hand tracking is performed in the current face-shoulder coordinate system using the transformed previous hand target region; and The current hand target region is extracted from the current image based on the hand tracking.

10. The method according to claim 1, wherein, The steps for extracting the current hand target region include: A hand search window is defined in the current image based on user statistics or a predefined window size; Detect the hand in the hand search window; and The current hand target region is extracted from the current image based on the detected hand.

11. A computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code embedded therein for controlling a computer, the computer-readable program code being configured to cause the computer to perform a hand tracking and gesture recognition process, the process comprising: Receive a current image from the camera system, the current image containing the face and shoulders of a human user; Based on the current image, detect the user's facial and shoulder features; A current face-shoulder coordinate system is established based on the detected facial and shoulder features; The previous hand target region is transformed from the previous face-shoulder coordinate system to the current face-shoulder coordinate system to generate a transformed previous hand target region, which is a target region containing the user's hand and is generated based on a previous image received from the camera system. Extract the current hand target region from the current image, wherein the current hand target region is defined in the current face-shoulder coordinate system; as well as Gesture recognition is performed using the transformed previous hand target region and the current hand target region in the current face-shoulder coordinate system.

12. The computer program product according to claim 11, in, The detection steps include detecting the user's eyes, mouth, and shoulders, and calculating the user's current left eye center position, right eye center position, mouth center position, and the current left shoulder bend tip position and right shoulder bend tip position. The establishment step includes establishing the current face-shoulder coordinate system based on the current left eye center position, the current right eye center position, the current mouth center position, the current left shoulder bend end position, and the current right shoulder bend end position.

13. The computer program product according to claim 12, wherein, The establishment steps include: The origin of the face-shoulder coordinate system is defined as the midpoint between the current left shoulder bend and the current right shoulder bend. The X-direction of the face-shoulder coordinate system is defined as the direction from the origin to the current center of the mouth; and The Y direction of the face-shoulder coordinate system is defined as the direction from the origin to the current left shoulder bend or the current right shoulder bend, and the Z direction of the face-shoulder coordinate system is defined as the direction orthogonal to the plane formed by the current left shoulder bend, the current right shoulder bend, and the current mouth center.

14. The computer program product of claim 12, wherein after the detection step and before the establishment step, the process further comprises: Calculate the current topology of the current left eye center position, the current right eye center position, the current mouth center position, the current left shoulder bend end position, and the current right shoulder bend end position; as well as The current topology is evaluated to confirm that the plane formed by the user's facial and shoulder features is within a predetermined angular range relative to the optical axis of the camera system.

15. The computer program product of claim 12, wherein after the detection step and before the establishment step, the process further comprises: Calculate the current topology of the current left eye center position, the current right eye center position, the current mouth center position, the current left shoulder bend end position, and the current right shoulder bend end position, and The current topology is determined to be an affine transformation of the initial topology, which is calculated based on the initial left eye center position, the initial right eye center position, the initial mouth center position, and the initial left shoulder bend top position.

16. The computer program product according to claim 15, wherein, Each of the current topology and the initial topology is defined by three ratios, the three ratios being: Among them, E L and E R It refers to the current or initial left and right eye centers, M. C It is the current or initial center of the mouth, S L and S R It is the current or initial shoulder bend end, and When the three ratios of the current topology are equal to the three ratios of the initial topology within a predefined threshold, the current topology is determined to be an affine transformation of the initial topology.

17. The computer program product according to claim 11, wherein, The detection steps include: Obtain a previous face-shoulder target region, which is a target region in a previous image that includes the user's face and shoulders; The features in the previous face-shoulder target region are traced to the features in the current image; Based on the tracking, the current face-shoulder target region is extracted from the current image; and Detect the user's eyes, mouth, and shoulders in the current face-shoulder target area.

18. The computer program product according to claim 11, wherein, The establishment steps include: Obtain a previous face-shoulder target region and a previous face-shoulder coordinate system, wherein the previous face-shoulder target region is the target region in the previous image that includes the user's face and shoulders, and the previous face-shoulder coordinate system has been established for the previous image; A mapping is established, which maps the previous face-shoulder target region to the current face-shoulder target region based on face-shoulder features or local image features in the corresponding face-shoulder target region, the mapping including rotation and translation; and The mapping is applied to the previous face-shoulder coordinate system to generate the current face-shoulder coordinate system.

19. The computer program product according to claim 11, wherein, The steps for extracting the current hand target region include: Hand tracking is performed in the current face-shoulder coordinate system using the transformed previous hand target region; and The current hand target region is extracted from the current image based on the hand tracking.

20. The computer program product according to claim 11, wherein, The steps for extracting the current hand target region include: A hand search window is defined in the current image based on user statistics or a predefined window size; Detect the hand in the hand search window; and The current hand target region is extracted from the current image based on the detected hand.

Images