Gesture recognition method and system based on hand key points and palm center spatial relationship

Abstract

The application provides a gesture recognition method and system based on hand key points and palm center spatial relationship, and belongs to the technical field of computer vision and human-computer interaction; comprising the following steps: S1: configuring an image acquisition device, simultaneously acquiring an RGB image and a depth image including a user's hand, and fusing to form an RGB-D image with three-dimensional information; S2: constructing a mixed data set and a deep learning model and pre-training to obtain a trained deep learning model; S3: inputting the RGB-D image acquired in step S1 into the trained deep learning model, and outputting three-dimensional coordinates of 22 hand key points, including 1 key point at the wrist, 4 key points on each finger and 1 key point at the palm center; S4: based on the three-dimensional coordinates of the acquired hand key points, evaluating hand spatial posture features and spatial relationship, classifying gestures, and outputting recognized gesture labels.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and human-computer interaction technology, and in particular to a gesture recognition method and system based on key points of the hand and the spatial relationship of the palm. Background Technology

[0002] Currently, gesture recognition technology is a key human-computer interaction method in fields such as VR / AR, autonomous driving, and smart homes. Mainstream solutions, such as Google's MediaPipe Hands, use a 21-point hand keypoint model to describe hand posture. While these technologies are widely used, existing technologies suffer from the following inherent defects and pressing technical problems: 1. Incomplete model information: The standard 21-point model lacks an independent palm keypoint that accurately represents the geometric center of the hand. This results in limited accuracy when judging gestures that depend on the overall orientation and position of the palm, such as pushing the palm forward or making a fist, due to a lack of reference information. 2. Weak 3D spatial perception: Existing solutions often rely on 2D projection relationships or simple 3D distances, making it difficult to effectively determine the crucial 3D spatial state of whether the palm is obscured by fingers. This prevents the system from accurately distinguishing gestures that are similar in 2D images but have different 3D structures. 3. Limited judgment dimensions: Gesture judgment logic focuses primarily on the distance and angle between keypoints, lacking the utilization of higher-level geometric relationships such as whether multiple keypoints are on the same spatial plane, thus limiting the definition and recognition capabilities of complex gestures.

[0003] Therefore, there is an urgent need to provide a gesture recognition method and system based on hand key points and palm spatial relationships. By additionally configuring palm key points to establish a hand key point model with 22 three-dimensional coordinates, it has a deep three-dimensional spatial understanding capability, accurately distinguishes complex gestures, and improves the accuracy, robustness and interpretability of gesture recognition. This is very necessary. Summary of the Invention

[0004] In view of this, the present invention proposes a gesture recognition method and system based on key points of the hand and the spatial relationship of the palm, which improves the ability to recognize palm occlusion by adding key points at the palm and facilitates the recognition of complex gestures.

[0005] On one hand, the present invention provides a gesture recognition method based on key points of the hand and the spatial relationship of the palm, comprising the following steps: S1: Use an image acquisition device to acquire RGB and depth images, including the user's hand, and fuse them to form an RGB-D image with three-dimensional information; S2: Construct a hybrid dataset and a deep learning model and pre-train it to obtain the trained deep learning model; S3: Input the RGB-D image obtained in step S1 into the trained deep learning model and output the three-dimensional coordinates of 22 hand key points, including 1 key point at the wrist, 4 key points on each finger and 1 key point at the palm. S4: Based on the obtained 3D coordinates of hand key points, evaluate the hand spatial posture features and spatial relationships, perform gesture classification, and output the recognized gesture labels.

[0006] Based on the above technical solutions, preferably, step S1 involves: pre-configuring an RGB camera and a depth camera; aligning the resolution and viewing angle of the depth image with that of the RGB image, ensuring that the same pixel position corresponds to the same point in physical space; converting the RGB image to YUV space; obtaining the pixels of the hand region based on the value range of the luminance channel and color difference component; setting a binary mask for the pixels of the hand region, setting the mask of the pixels of the hand region to 1 and the mask of the pixels of the background region to 0, removing the pixels of the background region; directly reading the depth value from the corresponding position in the depth image aligned with the pixels of the hand region; and combining this with the intrinsic parameters of the RGB camera to obtain the three-dimensional coordinates in the RGB camera coordinate system. X , Y , Z ).

[0007] Based on the above technical solutions, preferably, step S2 includes the following: The hybrid dataset includes real-world hand images and synthetic images. The synthetic images are generated by rendering a 3D hand model and displaying the hand from different viewpoints, under different lighting conditions, or under occlusion conditions. The 3D hand images include hand images with the palm facing the image acquisition device and hand images with the palm facing away from the image acquisition device. The synthetic images contain annotations of key points. The synthetic images and real-world hand images are sampled at a 4:1 ratio. The hand images are uniformly scaled to a fixed size. The depth information is normalized and then stitched with the RGB values ​​of the hand images. The 3D ground truth values ​​of all key points are transferred to the same RGB camera coordinate system to obtain a hand RGB image with the 3D coordinates of each pixel. The deep learning model structure includes an input layer, a backbone layer, and a detection head. The input layer is used to batch process several RGB images of the hand. The backbone layer uses HRNet-W32 to output a 2D heatmap. After global average pooling of the 2D heatmap, a multilayer perceptron (MLP) is used to regress the 22×3 dimensional root-relative coordinates. The root-relative coordinates are obtained by subtracting the baseline coordinates of the wrist keypoints from the coordinates of the hand keypoints. The training strategy is to first train the deep learning model using synthetic images to learn the 3D structure of the hand and the geometric relationship of the 22 keypoints. Then, real-world hand images are used to adjust the model parameters. The optimizer is AdamW, the batch size does not exceed 32, and cosine annealing is used for learning rate scheduling. The learning rate in the first 5% of the total iterations of the annealing heating phase does not exceed 3×10. -4 Then the learning rate is smoothly decreased along a cosine curve until the maximum number of iterations is reached.

[0008] Preferably, the backbone layer uses HRNet-W32, which includes two consecutively configured convolutional modules, a layer 1 module, several transition structure modules (Transition and Stage), and a 2D convolutional module. The two consecutively configured convolutional modules are used to achieve continuous dimensionality reduction, with a 3×3 kernel and a stride of 2. The layer 1 module is used to adjust the number of channels in the continuously dimensionality-reduced image without changing the image size. The transition structure module (Transition) is used to expand the image input to the layer 1 module with a downsampling scale branch that runs parallel to the main scale branch. The stage module (Stage) is configured with the same number of residual units as the input scale branches, and the output of each residual unit is processed by either an upsampling unit or a downsampling unit and then scaled accordingly. The number of branches is matched one-to-one with the output fusion; several transition structure modules (Transition) and stage main modules (Stage) are set in alternating intervals; if the fusion output of the stage main module (Stage) is sent to the next stage transition structure module (Transition), then an additional parallel downsampled scale branch is added based on the number of branches at the current scale; if the fusion output of the stage main module (Stage) is sent to the 2D convolution module, then after the stage main module (Stage) performs one-to-one fusion output according to the number of scale branches, the images of each downsampled scale branch parallel to the main scale branch are processed by the upsampling unit to uniformly restore the image size of the main scale branch, and then sent to the 2D convolution module after fusion processing. The 2D convolution module is used to output the 2D heatmap.

[0009] Preferably, the multilayer perceptron (MLP) includes three sequentially arranged fully connected layers. The first fully connected layer performs global average pooling on the global features of the 2D heatmap. f The first fully connected layer maps the output of the first fully connected layer to a higher dimension and increases its non-linear expressive power through the ReLU activation function; the second fully connected layer further maps the output of the first fully connected layer to a higher dimension and also increases its non-linear expressive power through the ReLU activation function; the third fully connected layer projects directly to the 22×3 dimension of the root-relative coordinates and does not use an activation function.

[0010] More preferably, the loss function of the deep learning model is obtained by weighted summation of the smoothed L1 loss of keypoint coordinates, the bone length constraint loss, and the visibility loss of palm keypoints. The smoothed L1 loss of keypoint coordinates is obtained by averaging the differences between the root-relative coordinates of each keypoint predicted by the deep learning model and the actual root-relative coordinates of each keypoint.

[0011] Preferably, step S4 involves feeding the RGB-D image obtained in step S1 into the trained deep learning model, outputting the root-relative coordinates of 22 hand key points, then calculating the core spatial pose based on the root-relative coordinates of the key points; configuring a lightweight classification model, and using the trained lightweight classification model to output the label of the recognized gesture based on the root-relative coordinates of the key points and the core spatial pose.

[0012] More preferably, the calculation of the core space pose based on the root-relative coordinates of the key points is a reference plane constructed from the wrist key points, the index finger root key points, and the little finger root key points. S normal vector N Calculate the normal vector N The observation vector of the optical center of the image acquisition device pointing to the key point on the palm. V The angle between i :if i If the angle is less than 75° and the depth of the key point on the palm is less than the average depth of the three key points constituting the reference plane, then the palm is considered visible; if i If the angle is greater than 105° and the depth of the key point on the palm is greater than the average depth of the three key points constituting the reference plane, then the palm is considered to be occluded; if 75° ≤ i If the angle is ≤105°, the palm is considered to be partially occluded. Then, the coplanarity of the keypoints is calculated. All fingertip and wrist keypoints are selected to form a point set P. The least squares method is used to fit the optimal plane of point set P. The distance from each keypoint in point set P to the optimal plane is calculated, and the root mean square error of the distance from each keypoint to the optimal plane is used as a measure of coplanarity.C 0.

[0013] Furthermore, the trained lightweight classification model has a structure comprising three linear layers. The outputs of the first and second linear layers are equipped with ReLU activation and Dropout regularization. A Softmax function is applied after the third linear layer to measure the root-relative coordinates of 22 keypoints, palm occlusion state, and coplanarity. C After being vectorized into zero, it serves as the 88-dimensional input feature of the lightweight classification model, outputting the probability of the gesture category. The loss function of the lightweight classification model is obtained by summing the classification loss term with the weighted L2 regularization term.

[0014] On the other hand, the present invention also provides a gesture recognition system based on key points of the hand and the spatial relationship of the palm, for implementing the above method, including: The image acquisition unit is used to acquire RGB images and depth images, including those of the user's hand, and fuse them to form an RGB-D image with three-dimensional information. The deep learning model building and training unit is used to build a hybrid dataset and a deep learning model, and to pre-train the deep learning model. The output unit uses the trained deep learning model to obtain the 3D coordinates of key hand points, evaluate the hand spatial pose features and spatial relationships, perform gesture classification, and output the recognized gesture labels.

[0015] The present invention provides a gesture recognition method and system based on key points of the hand and the spatial relationship of the palm, which has the following advantages compared with the prior art: 1. By aligning the images from the RGB camera and the depth camera, the RGB pixels are given real Z-axis information, thus providing a data foundation for accurately identifying coplanar relationships, bone length, and distances to key points on the wrist in the same camera coordinate system.

[0016] 2. A hybrid dataset training strategy is adopted, utilizing a large amount of precisely labeled synthetic data to allow the model to first learn the general 3D geometry of the hand, and then fine-tuning it using real image data. This reduces the dependence on real 3D annotations and improves the model's generalization ability and localization accuracy in real complex scenes. The improved HRNet-W32 backbone outputs a 2D heatmap that retains high-resolution spatial details, and then uses global average pooling and MLP structure to regress root-relative 3D coordinates, balancing 2D localization accuracy and 3D structure inference; the use of root-relative coordinates eliminates the ambiguity of absolute depth scale.

[0017] 3. Using the coordinates of 22 3D key points, based on the angle between the reference plane S normal vector N and the observation vector V. i It compares depth and calculates palm occlusion state and finger coplanarity measurement. By constructing feature vectors and inputting them into a minimal MLP structure, it automatically learns the combination relationship of these complex 3D features, accurately recognizes gestures, and uses early stopping to avoid invalid iterations. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an overall flowchart of a gesture recognition method and system based on key points of the hand and the spatial relationship of the palm, according to the present invention. Figure 2 This is a schematic diagram of 22 key hand points in a gesture recognition method and system based on key hand points and the spatial relationship of the palm, according to the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] Existing gesture recognition technologies suffer from the following problems: 1. Incomplete model information: The standard 21-point model lacks an independent palm keypoint that accurately represents the geometric center of the hand. This results in limited accuracy when the system judges gestures that depend on the overall orientation and position of the palm, such as pushing the palm forward or making a fist, due to the lack of reference information. 2. Weak 3D spatial perception: Existing solutions mostly rely on 2D projection relationships or simple 3D distances, making it difficult to effectively determine the crucial 3D spatial state of whether the palm is obscured by fingers. This prevents the system from accurately distinguishing gestures that are similar in 2D images but have different 3D structures. 3. Limited judgment dimension: The gesture judgment logic is mostly focused on the distance and angle between key points, lacking the utilization of higher-level geometric relationships such as whether multiple key points are on the same spatial plane, thus limiting the definition and recognition capabilities of complex gestures.

[0022] In view of this, such as Figure 1 As shown, in one aspect, the present invention provides a gesture recognition method based on key points of the hand and the spatial relationship of the palm, including the following steps: S1: Use an image acquisition device to acquire RGB images and depth images, including those of the user's hand, and fuse them to form an RGB-D image with three-dimensional information.

[0023] Specifically, the process involves: pre-configuring an RGB camera and a depth camera; aligning the resolution and viewpoint of the depth image with those of the RGB image, ensuring that the same pixel location corresponds to the same point in physical space; and converting the RGB image to YUV space, where Y = 0.299R + 0.587G + 0.114B, U = -0.1687R - 0.3313G + 0.5B + ​​128, and V = 0.5R - 0.4187. G-0.0813B+128, the pixels of the hand region are obtained based on the value ranges of the luminance channel and color difference components: the luminance channel Y value is 80-230, the blue difference component U value is 90-125, and the red difference component V value is 135-170; a binary mask is set for the pixels of the hand region, setting the mask of the pixels of the hand region to 1 and the mask of the pixels of the background region to 0, removing the pixels of the background region, and directly reading the depth value of the corresponding position in the depth image aligned with the pixels of the hand region. Combined with the intrinsic parameters of the RGB camera, the three-dimensional coordinates in the RGB camera coordinate system are obtained. X , Y , Z ), , , , These are the pixel coordinates in the RGB image. pixel coordinates in an RGB image The corresponding depth value in the depth image. The optical center coordinates of the RGB camera. It's the focal length.

[0024] S2: Construct a hybrid dataset and a deep learning model and pre-train it to obtain the trained deep learning model.

[0025] Step S2 includes: The hybrid dataset includes real-world hand images and synthetic images. The synthetic images are generated by rendering a 3D hand model and displaying the hand from different viewpoints, under different lighting conditions, or under occlusion conditions. The 3D hand images include hand images with the palm facing the image acquisition device and hand images with the palm facing away from the image acquisition device. The synthetic images contain annotations of key points. The synthetic images and real-world hand images are sampled at a 4:1 ratio. The hand images are uniformly scaled to a fixed size. The depth information is normalized and then stitched with the RGB values ​​of the hand images. The 3D ground truth values ​​of all key points are transferred to the same RGB camera coordinate system to obtain a hand RGB image with the 3D coordinates of each pixel.

[0026] The deep learning model structure includes an input layer, a backbone layer, and a detection head. The input layer is used to batch process several RGB images of the hand. The backbone layer uses HRNet-W32 to output a 2D heatmap. After global average pooling of the 2D heatmap, a multilayer perceptron (MLP) is used to regress the 22×3 dimensional root-relative coordinates. The root-relative coordinates are obtained by subtracting the baseline coordinates of the wrist keypoints from the coordinates of the hand keypoints. The training strategy is to first train the deep learning model using synthetic images to learn the 3D structure of the hand and the geometric relationship of the 22 keypoints. Then, real-world hand images are used to adjust the model parameters. The optimizer is AdamW, the batch size does not exceed 32, and cosine annealing is used for learning rate scheduling. The learning rate in the first 5% of the total iterations of the annealing heating phase does not exceed 3×10. -4 Then, the learning rate is smoothly decreased along a cosine curve until the maximum number of rounds is reached. ,in To smooth out the learning rate during the descent phase, This represents the minimum learning rate. This represents the maximum learning rate at the end of the annealing phase. t For the current iteration round, t >5, T This represents the maximum number of iteration rounds.

[0027] In the above scheme, the backbone layer uses HRNet-W32, which includes two consecutively set convolutional modules, a layer1 module, three transition structure modules, three stage backbone modules, and a 2D convolutional module. The two consecutively set convolutional modules are used to achieve continuous dimensionality reduction. The convolutional kernel of the convolutional module is 3×3, and the stride is 2. Let the size of the input image be 256×192×3. The image size obtained after processing by the first convolutional module is 128×96×64, and the image size obtained after processing by the second convolutional module is 64×48×64, that is, the length and width of the image are halved. The structure of both convolutional modules includes sequentially set conv2d layers, batch normalization (BN) and ReLU activation functions.

[0028] The layer1 module is used to adjust the number of channels in continuously dimensionality-reduced images without changing the image size. For example, it can increase the number of channels in an input 64×48×64 image, such as by adjusting the number of channels to 256, which is the same as the function of the layer1 module in ResNet.

[0029] The Transition module and the Stage module have the same number of modules and are set alternately. The output of each residual unit is processed by either an upsampling unit or a downsampling unit and then fused according to the number of scale branches. If the fused output of the Stage module is sent to the next Transition module, a parallel downsampling scale branch is added on top of the current number of scale branches. If the fused output of the Stage module is sent to the 2D convolution module, after the Stage module performs fusion according to the number of scale branches, it processes the images of each downsampling scale branch parallel to the main scale branch through an upsampling unit to restore them to the image size of the main scale branch. After fusion processing, the images are sent to the 2D convolution module, which is used to output a 2D heatmap.

[0030] This embodiment includes a first transition structure module Transition1, a second transition structure module Transition2, and a third transition structure module Transition3, as well as a first stage backbone module Stage2, a second stage backbone module Stage3, and a third stage backbone module Stage4; each transition structure module Transition is used to expand the image input to the layer1 module with a downsampling scale branch that runs parallel to the main scale branch. The first transition structure module, Transition1, has an image size of 64×48×32 for its main scale branch and 32×24×64 for its second scale branch. The Stage backbone module is configured with the same number of residual units as the input-side scale branches. The first stage backbone module, Stage2, is configured with two residual units, Basic Blocks×4, the same number as the number of branches in the previous stage's first transition structure module, Transition1. The Basic Blocks×4 are obtained by stacking Basic Blocks four times. The structure of a single Basic Block includes a backbone, on which five layers are sequentially set: a 3×3 convolutional layer, a batch normalization (BN) stage, a ReLU activation function, another 3×3 convolutional layer, and a batch normalization (BN) stage. The processing result of the five layers of the backbone on the original input is then fed into an adder and fused with the original input. After that, it is fed into a ReLU activation function for further processing, keeping the size and number of channels of the original input feature map unchanged, and only performing nonlinear feature transformation processing. Then, in the subsequent second transition structure module Transition2, the image dimensions of the main scale branch remain unchanged. A parallel downsampling third scale branch is further added to the second scale branch, with an image size of 16×12×128. Similarly, in the third transition structure module Transition3, the image dimensions of the main scale branch and the second scale branch remain unchanged. A parallel downsampling fourth scale branch is further added to the third scale branch, with an image size of 8×6×256.

[0031] For Stage 2, the results of the Basic Block × 4 residual units corresponding to the main scale branch and the Basic Block × 4 residual units corresponding to the second scale branch are fused together after being processed by an Up × 2 sampling unit. The results of the Basic Block × 4 residual units corresponding to the second scale branch are also fused together with the results of the Basic Block × 4 residual units corresponding to the main scale branch after being processed by a Down × 2 sampling unit. Similarly, for Stage 3, the results of the Basic Block × 4 residual units corresponding to the main scale branch, the results of the Basic Block × 4 residual units corresponding to the second scale branch (after being processed by an Up × 2 sampling unit), and the results of the Basic Block × 4 residual units corresponding to the third scale branch (after being processed by an Up × 4 sampling unit) are fused together. The results of the Basic Block × 4 residual units corresponding to the second scale branch are then fused together with the results of the Basic Block × 4 residual units corresponding to the main scale branch (after being processed by a Down × 2 sampling unit) and the results of the Basic Block × 4 residual units corresponding to the third scale branch. The results of Block×4 are processed by the upsampling unit Up×2 and then fused for output. The results of the residual unit Basic Block×4 corresponding to the third scale branch are fused with the results of the residual unit Basic Block×4 corresponding to the main scale branch processed by the downsampling unit Down×4, and the results of the residual unit Basic Block×4 corresponding to the second scale branch processed by the downsampling unit Down×2, and then fused for output. The processing of the main module Stage4 in the third stage is basically similar, but there is an additional step where all the fused outputs are upsampled to the image size of the main scale branch (64×48×32) and then fed into the 2D convolution module.

[0032] The upsampling unit Up×n, n=2,4 used here has a network structure that includes sequentially set Conv2d convolutional layers, batch normalization (BN) stages, and upsampling operations. The upsampling operation expands the length and width of the image in the current scale branch by a factor of n, while reducing the number of channels to 1 / n of the current number.

[0033] The downsampling unit Down×2 includes sequentially set Conv2d convolutional layers and batch processing (BN) stages, which are used to reduce the length and width of the image in the current scale branch to half of the current size, and expand the number of channels to twice the current size. The downsampling unit Down×4 includes a Conv convolutional layer, a Conv2d convolutional layer, and a batch processing unit BN arranged sequentially. The Conv convolutional layer is used to reduce the length and width of the image in the current scale branch by 1 / 2, while keeping the number of channels unchanged; the Conv2d convolutional layer is used to further reduce the length and width of the image to 1 / 4, while expanding the number of channels to 4 times the current number.

[0034] In one embodiment, a multilayer perceptron (MLP) includes three sequentially arranged fully connected layers. The first fully connected layer performs global average pooling on the global features of the 2D heatmap. f The first fully connected layer maps the output of the first fully connected layer to a higher dimension and enhances its non-linear expressiveness through the ReLU activation function. The second fully connected layer further maps the output of the first fully connected layer to a higher dimension, also enhancing its non-linear expressiveness through the ReLU activation function. The third fully connected layer directly projects the output to the 22×3 dimension of the root-relative coordinates without using an activation function. This process preserves high-resolution spatial details through a 2D heatmap and regresses the 3D root-relative coordinates using global average pooling and a multilayer perceptron (MLP), balancing 2D localization accuracy with 3D structural inference.

[0035] The loss function of the deep learning model is obtained by weighted summing of the smoothed L1 loss of keypoint coordinates, the bone length constraint loss, and the visibility loss of palm keypoints. The smoothed L1 loss of keypoint coordinates is obtained by averaging the differences between the root-relative coordinates predicted by the deep learning model and the actual root-relative coordinates of each keypoint. The loss function of the deep learning model is... ,in Smoothing L1 loss for keypoint coordinates, , Let be the sequence number of the key point, and the smoothed L1 loss be . , The first prediction for the deep learning model j The root-relative coordinates of the key points For the first j The actual root-relative coordinates of each key point; For bone length constraint loss, , This refers to the bone sequence number. For the predicted first k Based on the length of the hand bones, The first prediction obtained by the deep learning model k The root-relative coordinates at both ends of the root hand bones. For the statistics from the mixed dataset kThe average length of the bones in the root hand; For the loss of visibility of key points on the palm, , Visibility labels for key points on the palm. s Reference planes constructed from the palm key point to the wrist key point, the key point at the base of the index finger, and the key point at the base of the little finger. S distance, It is a distance threshold. This is a binary classification prediction head that outputs the occlusion probability of key points on the palm. It is the Sigmoid activation function. f The feature representation input to the prediction head is derived from the global features obtained after global average pooling of the 2D heatmap. For the weight vector, b For bias terms; These are weight terms, with values ​​of 1.0, 0.1, and 0.02 respectively.

[0036] S3: Input the RGB-D image obtained in step S1 into the trained deep learning model and output the three-dimensional coordinates of 22 hand key points, including 1 key point at the wrist, 4 key points on each finger and 1 key point at the palm.

[0037] refer to Figure 2 The difference between this and the standard 21-point hand model is that the wrist key point is the 0 point, and the thumb, index finger, middle finger, ring finger and little finger are numbered in order of distance from the wrist key point. The key points on the thumb are numbered 1-4, the key points on the index finger are numbered 5-8, and so on, with the palm key point numbered 21.

[0038] S4: Based on the obtained 3D coordinates of hand key points, evaluate the hand spatial posture features and spatial relationships, perform gesture classification, and output the recognized gesture labels.

[0039] Specifically, the RGB-D image obtained in step S1 is fed into the trained deep learning model, which outputs the root-relative coordinates of 22 hand key points. Then, based on the root-relative coordinates of the key points, the core spatial pose is calculated. A lightweight classification model is configured, and the trained lightweight classification model is used to output the label of the recognized gesture based on the root-relative coordinates of the key points and the core spatial pose.

[0040] Among them, the core space pose is calculated based on the root-relative coordinates of the keypoints, which serves as the reference plane constructed from the wrist keypoint, the index finger base keypoint, and the little finger base keypoint. S normal vector N Calculate the normal vector NThe observation vector of the optical center of the image acquisition device pointing to the key point on the palm. V The angle between i :if i If the angle is less than 75° and the depth of the key point on the palm is less than the average depth of the three key points constituting the reference plane, then the palm is considered visible; if i If the angle is greater than 105° and the depth of the key point on the palm is greater than the average depth of the three key points constituting the reference plane, then the palm is considered to be occluded; if 75° ≤ i If the angle is ≤105°, the palm is considered to be partially occluded. Then, the coplanarity of the keypoints is calculated. All fingertip and wrist keypoints are selected to form a point set P. The least squares method is used to fit the optimal plane of point set P. The distance from each keypoint in point set P to the optimal plane is calculated, and the root mean square error of the distance from each keypoint to the optimal plane is used as a measure of coplanarity. C 0.

[0041] The lightweight classification model mentioned here has a structure consisting of three linear layers. The outputs of the first and second linear layers are equipped with ReLU activation and Dropout regularization. A Softmax function is applied after the third linear layer to store the root-relative coordinates of 22 keypoints, palm occlusion state, and coplanarity measure. C After zero-vectorization, the 88-dimensional input features are used as the lightweight classification model. The output dimension of the first linear layer is 64, and the output dimension of the second linear layer is 32. The third linear layer outputs the gesture category and transforms it into a probability output of the gesture category through the Softmax function. The training policy is Adam, and the learning rate is 10. -3 The number of iterations does not exceed 200 rounds; the convergence strategy uses a loss function combined with early stopping. The loss function of the lightweight classification model is obtained by summing the classification loss term and the weighted L2 regularization term. , These represent the label corresponding to the maximum probability of the gesture category output by the lightweight classification model, the true label of the gesture category, and the classification loss term. , W The sum of squares of the weights of the three linear layers, weights l The initial value is 10 -4 The training process uses independent training and validation sets; and when the macro F1 score no longer improves after N consecutive iterations, it reverts to the model weight parameters with the highest macro F1 score on the validation set, where N is between 10 and 15.

[0042] As one application of this embodiment, the recognized gestures can be further used to interact with the AI ​​assistant. For example, in the scenario of AI painting, when the user's gesture is recognized as a fist using the present invention, the AI ​​assistant automatically responds by grabbing the paintbrush. When the gesture of palm facing forward is recognized, the AI ​​assistant automatically executes the instruction to bring up the color palette.

[0043] In a smart cockpit scenario, when the system recognizes that the user's gesture is only thumbs up, it will play music. When the system recognizes that the user's palm is facing forward and swiping to the left of the user's body, it will switch to the previous song.

[0044] On the other hand, the present invention also provides a gesture recognition system based on key points of the hand and the spatial relationship of the palm, for implementing the above method, including: The image acquisition unit is used to acquire RGB images and depth images, including those of the user's hand, and fuse them to form an RGB-D image with three-dimensional information. The deep learning model building and training unit is used to build a hybrid dataset and a deep learning model, and to pre-train the deep learning model. The output unit uses the trained deep learning model to obtain the 3D coordinates of key hand points, evaluate the hand spatial pose features and spatial relationships, perform gesture classification, and output the recognized gesture labels.

[0045] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A gesture recognition method based on key hand points and the spatial relationship of the palm, characterized in that, Includes the following steps: S1: Use an image acquisition device to acquire RGB and depth images, including the user's hand, and fuse them to form an RGB-D image with three-dimensional information; S2: Construct a hybrid dataset and a deep learning model and pre-train it to obtain the trained deep learning model; S3: Input the RGB-D image obtained in step S1 into the trained deep learning model and output the three-dimensional coordinates of 22 hand key points, including 1 key point at the wrist, 4 key points on each finger and 1 key point at the palm. S4: Based on the obtained 3D coordinates of hand key points, evaluate the hand spatial posture features and spatial relationships, perform gesture classification, and output the recognized gesture labels.

2. The gesture recognition method based on key hand points and palm spatial relationship according to claim 1, characterized in that, Step S1 involves pre-configuring an RGB camera and a depth camera, aligning the resolution and viewpoint of the depth image with the RGB image, ensuring that the same pixel position corresponds to the same point in physical space; converting the RGB image to YUV space, obtaining the pixels of the hand region based on the value range of the luminance channel and color difference components; setting a binary mask for the pixels of the hand region, setting the mask of the pixels of the hand region to 1, and setting the mask of the pixels of the background region to 0, removing the pixels of the background region, directly reading the depth value of the corresponding position in the depth image aligned with the pixels of the hand region, and combining it with the intrinsic parameters of the RGB camera to obtain the three-dimensional coordinates in the RGB camera coordinate system. X , Y , Z ).

3. The gesture recognition method based on key hand points and palm spatial relationship according to claim 1, characterized in that, Step S2 includes the following: The hybrid dataset includes real-world hand images and synthetic images. The synthetic images are generated by rendering a 3D hand model and displaying the hand from different viewpoints, under different lighting conditions, or under occlusion conditions. The 3D hand images include hand images with the palm facing the image acquisition device and hand images with the palm facing away from the image acquisition device. The synthetic images contain annotations of key points. The synthetic images and real-world hand images are sampled at a 4:1 ratio. The hand images are uniformly scaled to a fixed size. The depth information is normalized and then stitched with the RGB values ​​of the hand images. The 3D ground truth values ​​of all key points are transferred to the same RGB camera coordinate system to obtain a hand RGB image with the 3D coordinates of each pixel. The deep learning model consists of an input layer, a backbone layer, and a detection head. The input layer is used to batch process several RGB images of the hand. The backbone layer uses HRNet-W32 to output a 2D heatmap. After global average pooling of the 2D heatmap, a multilayer perceptron (MLP) is used to regress the 22×3-dimensional root-relative coordinates. The root-relative coordinates are obtained by subtracting the baseline coordinates of the wrist keypoints from the coordinates of the hand keypoints. The training strategy involves first training the deep learning model using synthetic images to learn the 3D structure of the hand and the geometric relationships of 22 key points. Then, real-world hand images are used to adjust the model parameters. The optimizer is AdamW, with a batch size not exceeding 32, and cosine annealing is used for learning rate scheduling. The learning rate during the first 5% of the total iterations in the annealing warm-up phase does not exceed 3 × 10⁻⁶. -4 Then the learning rate is smoothly decreased along a cosine curve until the maximum number of iterations is reached.

4. The gesture recognition method based on key hand points and palm spatial relationship according to claim 3, characterized in that, The backbone layer uses HRNet-W32, which includes two consecutively configured convolutional modules, a layer 1 module, several transition structure modules (Transition), a stage backbone module (Stage), and a 2D convolutional module. The two consecutively configured convolutional modules are used for continuous dimensionality reduction, with 3×3 kernels and a stride of 2. The layer 1 module adjusts the number of channels in the continuously dimensionality-reduced image without changing its size. The transition structure module (Transition) expands the input image of the layer 1 module with a downsampling scale branch running parallel to the main scale branch. The stage backbone module (Stage) is configured with the same number of residual units as the input scale branches. The outputs of each residual unit are processed by either an upsampling unit or a downsampling unit and then fused according to the number of scale branches. The number of transition structure modules (Transition) is the same as the stage backbone module (Stage), and they are alternately set. If the fused output of the stage backbone module (Stage) is fed into the next stage transition structure module (Transition), a parallel downsampling scale branch is added on top of the current number of scale branches. If the Stage main module's fusion output is fed into the 2D convolution module, then after the Stage performs one-to-one fusion output according to the number of scale branches, the images of each downsampled scale branch that runs parallel to the main scale branch are processed by the upsampling unit to restore them to the image size of the main scale branch. After fusion processing, the images are then fed into the 2D convolution module, which is used to output a 2D heatmap.

5. The gesture recognition method based on key hand points and palm spatial relationship according to claim 4, characterized in that, The Multilayer Perceptron (MLP) consists of three sequentially connected layers. The first fully connected layer uses the global features obtained by global average pooling of the 2D heatmap. f The first fully connected layer maps the output of the first fully connected layer to a higher dimension and increases its non-linear expressive power through the ReLU activation function; the second fully connected layer further maps the output of the first fully connected layer to a higher dimension and also increases its non-linear expressive power through the ReLU activation function; the third fully connected layer projects directly to the 22×3 dimension of the root-relative coordinates and does not use an activation function.

6. The gesture recognition method based on key hand points and palm spatial relationship according to claim 5, characterized in that, The loss function of the deep learning model is obtained by weighted summation of the smoothed L1 loss of keypoint coordinates, the bone length constraint loss, and the visibility loss of palm keypoints. The smoothed L1 loss of keypoint coordinates is obtained by averaging the differences between the root-relative coordinates of each keypoint predicted by the deep learning model and the actual root-relative coordinates of each keypoint.

7. The gesture recognition method based on key hand points and palm spatial relationship according to claim 3, characterized in that, Step S4 involves feeding the RGB-D image obtained in step S1 into the trained deep learning model, outputting the root-relative coordinates of 22 hand key points, and then calculating the core spatial pose based on the root-relative coordinates of the key points. A lightweight classification model is configured, and the trained lightweight classification model outputs the label of the recognized gesture based on the root-relative coordinates of the key points and the core spatial pose.

8. The gesture recognition method based on key hand points and palm spatial relationship according to claim 7, characterized in that, The core spatial pose calculation based on the root-relative coordinates of the keypoints is a reference plane constructed from the wrist keypoint, the index finger base keypoint, and the little finger base keypoint. S normal vector N Calculate the normal vector N The observation vector of the optical center of the image acquisition device pointing to the key point on the palm. V The angle between θ :if θ If the angle is less than 75° and the depth of the key point on the palm is less than the average depth of the three key points constituting the reference plane, then the palm is considered visible; if θ If the angle is greater than 105° and the depth of the key point on the palm is greater than the average depth of the three key points constituting the reference plane, then the palm is considered to be occluded; if 75° ≤ θ If the angle is ≤105°, then the palm is considered to be partially obscured; Then, the coplanarity of the keypoints is calculated. All fingertip and wrist keypoints are selected to form a point set P. The least squares method is used to fit the optimal plane of point set P. The distance from each keypoint in point set P to the optimal plane is calculated, and the root mean square error of the distance from each keypoint to the optimal plane is obtained as a measure of coplanarity. C 0.

9. A gesture recognition method based on key hand points and palm spatial relationship according to claim 8, characterized in that, The trained lightweight classification model consists of three linear layers. The outputs of the first and second linear layers are configured with ReLU activation and Dropout regularization. A Softmax function is applied after the third linear layer to store the root-relative coordinates, palm occlusion state, and coplanarity measure of 22 keypoints. C After being vectorized into zero, it serves as the 88-dimensional input feature of the lightweight classification model, outputting the probability of the gesture category. The loss function of the lightweight classification model is obtained by summing the classification loss term with the weighted L2 regularization term.

10. A gesture recognition system based on key hand points and palm spatial relationships, used to implement the method as described in any one of claims 1-9, characterized in that, include: The image acquisition unit is used to acquire RGB images and depth images, including those of the user's hand, and fuse them to form an RGB-D image with three-dimensional information. The deep learning model building and training unit is used to build a hybrid dataset and a deep learning model, and to pre-train the deep learning model. The output unit uses the trained deep learning model to obtain the 3D coordinates of key hand points, evaluate the hand spatial pose features and spatial relationships, perform gesture classification, and output the recognized gesture labels.

Images