Double-branch hand interaction posture estimation model combining skeleton features

Through the dual-branch model of joint bone features, combined with U-shaped network and Swin Transformer module, the two-dimensional Gaussian heat map is used to supervise the corrected feature information, and the problems of low inference efficiency and neglected topological relationships of the existing hand pose estimation model are solved, and the problem of low inference efficiency and neglected topological relationships is realized. More efficient hand posture estimation and more accurate prediction.

CN118918632APending Publication Date: 2024-11-08LANZHOU JIAOTONG UNIV

Patent Information

Application Number
CN202410579993.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-10
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the process of improving the prediction accuracy of a single joint node, the existing hand posture estimation model increases network complexity and ignores the intrinsic topological relationship of the hand joint, resulting in poor long-distance feature acquisition effect.

Method used

Using a dual-branch hand interactive pose estimation model with joint bone features, rough and fine features are obtained through the hand feature extraction network, combined with the U-shaped network and the Swin Transformer module, global and local features are captured, and corrections are supervised by two-dimensional Gaussian heat maps Extract feature information and perform pose prediction.

Benefits of technology

It improves the accuracy and performance of hand posture estimation, significantly reduces the calculation amount, can better understand the hand structure and dynamics, and provides more accurate prediction results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918632A_ABST
    Figure CN118918632A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of interactive attitude model construction, and particularly discloses a skeleton feature-combined double-branch hand interactive attitude estimation model, which comprises the following steps: S1, a data source input unit: firstly, sending a hand region depth image into a rough gesture feature acquisition unit of a hand feature extraction network, firstly, rough hand features are obtained through a 7 * 7 convolutional layer, and then relatively fine hand features under initial resolution are obtained through a fine gesture feature obtaining unit. According to the method, the global features and the local joint point features are captured through the Swin Transformer and the CNN respectively, after fusion, the two-dimensional Gaussian heat map is added for supervision and correction for feature extraction, through the method, the effect of the hand posture estimation method can be improved, and better performance can be obtained in practical application.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of interactive posture model construction, and in particular to a dual-branch hand interactive posture estimation model combining skeleton features. Background Art

[0002] With the rapid development of computer vision and deep learning technology, hand interaction posture estimation is increasingly widely used in human-computer interaction, virtual reality, robotics and other fields. However, due to the complexity of hand structure and the variability of posture, accurately estimating the interactive posture of the hand is still a challenging task. Hand interaction posture estimation refers to extracting the interactive posture information of the hand through computer vision and deep learning technology based on the input hand image or video data. This technology can accurately locate the joints of the hand and reconstruct the posture structure of the hand, including the position of the joints, skeleton constraints, etc. In virtual reality and human-computer interaction, it can help users interact with computers and virtual objects through natural gestures, providing a more intuitive and immersive experience. In the field of medical rehabilitation, hand interaction posture estimation can be used to analyze the precise quantitative data of hand movements, which is of great significance for the early diagnosis of diseases such as Parkinson's disease. In addition, hand interaction posture estimation also plays an important role in the fields of film and television animation, remote control, virtual meetings, etc.

[0003] Most current hand posture estimation methods use CNN models to construct feature extraction networks. However, this type of network, which increases the prediction accuracy of a single joint point by increasing network complexity, is usually large in scale, has low inference efficiency, and ignores the inherent topological relationship of the hand joints, making it ineffective when acquiring long-distance features.

[0004] To this end, we propose a dual-branch hand interaction posture estimation model that combines skeleton features to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a dual-branch hand interaction posture estimation model combining skeleton features to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides a dual-branch hand interaction posture estimation model combining skeleton features, comprising the following steps:

[0007] S1, data source input unit: first, the hand area depth image is sent to a coarse gesture feature acquisition unit of a hand feature extraction network, firstly a 7×7 convolution layer is used to obtain a coarse hand feature, and then a fine gesture feature acquisition unit is used to obtain a relatively fine hand feature at the initial resolution;

[0008] S2, feature fusion unit: Then, the hand features are sent to the dual-branch acquisition unit in parallel. One branch is a local feature unit of the hand joints under the multi-resolution aggregation output of the U-shaped network to obtain the local feature information of each hand joint, and the other branch is a global feature unit of the hand joints to obtain the global feature information containing the topological connection of the hand joints under the attention calculation. After merging, they are summarized and fused by the dual-branch fusion unit.

[0009] S3, heat map supervision unit: supervises and corrects the extracted feature information by adding a Gaussian heat map with joint point distribution information;

[0010] S4, posture estimation unit: After the fused joint point features are corrected and processed, the interactive posture is predicted. This layer adopts a regression network structure and maps the fused features to the three-dimensional coordinate space of the joint points through a fully connected layer;

[0011] S5, data source output unit: Finally, the features containing rich information are sent to the back-end network to obtain the coordinate information of the hand joint points.

[0012] In a further embodiment, in S1, when the rough gesture acquisition unit processes and then enters the fine gesture feature acquisition unit for processing, a relatively fine hand feature at the initial resolution is obtained through a pooling unit and two bottleneck residual units.

[0013] In a further embodiment, when the fine gesture feature acquisition unit in S1 is processed, the network integrates feature information at different levels through top-down and bottom-up feature propagation. In the top-down process, the bottleneck residual unit and the maximum pooling unit appear in groups, and the network starts to reduce the resolution at twice the speed from the high-resolution feature map. Each step will enhance the detail information of the feature, and the number of feature channels will also increase exponentially. In the bottom-up process, the bottleneck residual structure and the upsampling structure appear in groups. This part requires that the output result before each layer be upsampled twice in order to restore the feature map resolution. This step is also to be added to the low-level hand features processed by the bottleneck residual unit at the same resolution through jump connections. At the same time, the network starts to increase exponentially from the low-resolution feature map, and the number of feature channels also decreases at twice the speed until it is finally restored to the original state. The feature obtained at this time is the aggregation of hand feature information at four different resolutions.

[0014] In a further embodiment, in the U-shaped network in S2, the number of feature channels also shows a U-shaped trend, first increasing and then decreasing, and the features obtained are also different. The introduction of the bottleneck residual unit helps the network to better transmit residual information, which not only retains and transmits global and local information, but also captures and aggregates hand feature information at different scales.

[0015] In a further embodiment, the global feature unit of the hand joints in S2 includes two key parts, the first part is a window self-attention module and the second part is a shifted window self-attention module. Each part is composed of a normalization layer module, an attention module, and a multi-layer perceptron module. The overall structure consists of an input part, a Patch Partition block structure, and four Stages to extract hand features.

[0016] In a further embodiment, during the hand feature extraction process in S2, the 48×48 feature map is first sent to the Swin Transformer Block, and the window size is set to 4×4. At this time, the image is divided into 12×12 windows, and Self-Attention calculation is performed in each window, and the regional pixels are converted into a one-dimensional vector as the effective input sequence length. Then, the residual connection is introduced through the MLP multi-layer perceptron to perform Linear Embedding linear encoding.

[0017] In a further embodiment, when a Gaussian heat map with joint point distribution information is used to supervise and correct the extracted feature information in S3, it is necessary to cycle through such steps twice to obtain more abundant and accurate key hand information.

[0018] In a further embodiment, the variance of the Gaussian distribution in S3 is generally controlled by a pre-set parameter, the highest intensity value of each pixel is the position of the key point, and the intensity values ​​of the surrounding pixels gradually decrease. The Gaussian distributions generated for all key points are superimposed to obtain the heat map value of the pixel. Therefore, by finding the position with the highest confidence value in the heat map, the key points in the hand can be accurately located. At the same time, the topological connection of the joint points is also retained according to the peak distribution. Therefore, the model can process multiple key points at the same time, and the model can better understand the structure and dynamics of the hand and make more accurate predictions.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] First, in the present invention, global features and local joint features are captured respectively by Swin Transformer and CNN, and a two-dimensional Gaussian heat map supervision correction is added after fusion to perform feature extraction. Through this method, the effect of the hand posture estimation method can be improved, and better performance can be obtained in practical applications.

[0021] Secondly, in the present invention, in the global feature unit of the hand joints, the hand features are extracted by the input part, the Patch Partition block structure and the four Stages. This structure significantly reduces the amount of calculation compared to the traditional Transformer module. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a structural schematic diagram of the present invention;

[0023] Figure 2 It is a schematic diagram of the components of the dual-branch acquisition unit in the present invention;

[0024] Figure 3 This is a schematic diagram of the components of the global feature unit of the hand joints in the present invention;

[0025] Figure 4 Schematic diagram of a window self-attention module and a shifted window self-attention module in the present invention; Figure 5 Flow chart of the heat map supervision module in the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] See also Figure 1-Figure 5 In an embodiment of the present invention, a dual-branch hand interaction posture estimation model combining skeleton features includes the following steps:

[0028] S1, data source input unit: first, the hand area depth image is sent to a coarse gesture feature acquisition unit of a hand feature extraction network, firstly a 7×7 convolution layer is used to obtain a coarse hand feature, and then a fine gesture feature acquisition unit is used to obtain a relatively fine hand feature at the initial resolution;

[0029] When the fine gesture feature acquisition unit in S1 is processed, the network integrates feature information at different levels through top-down and bottom-up feature propagation. In the top-down process, the bottleneck residual unit and the maximum pooling unit appear in groups, and the network starts to reduce the resolution at twice the speed from the high-resolution feature map. Each step will enhance the detail information of the feature, and the number of feature channels will also increase exponentially. In the bottom-up process, the bottleneck residual structure and the upsampling structure appear in groups. This part requires the output result of each layer to be upsampled twice in order to restore the feature map resolution. This step is also added to the low-level hand features processed by the bottleneck residual unit at the same resolution through jump connections. At the same time, the network starts to increase exponentially from the low-resolution feature map, and the number of feature channels also decreases at twice the speed until it is finally restored to the original state. The feature obtained at this time is the aggregation of hand feature information at four different resolutions. The feature obtained at this time is the aggregation of hand feature information at four different resolutions. The upsampling method used in this paper is bilinear interpolation, which estimates the value of the target pixel by considering the weights of the four adjacent pixels, and interpolates each pixel of the input feature map to generate a feature map of a larger size, thereby achieving smooth upsampling of the image and restoring the original information of the feature map to a large extent;

[0030] S2, feature fusion unit: Then, the hand features are sent to the dual-branch acquisition unit in parallel. One branch is a local feature unit of the hand joints under the multi-resolution aggregation output of the U-shaped network to obtain the local feature information of each hand joint, and the other branch is a global feature unit of the hand joints to obtain the global feature information containing the topological connection of the hand joints under the attention calculation. After merging, they are summarized and fused by the dual-branch fusion unit.

[0031] Swin Transformer is used to extract the topological information of the hand joints. Compared with the traditional Vision Transformer mechanism, Swin Transformer introduces a windowed multi-head attention mechanism to divide the input features into small blocks, but through multi-level attention mechanisms and cross-layer transfer fusion, the model can more efficiently process long-range dependency information and fuse the local hand feature information captured at different levels into a global representation;

[0032] The global feature unit of the hand joints in S2 consists of two key parts. The first part is the window self-attention module and the second part is the shift window self-attention module. Each part is composed of a normalization layer module, an attention module, and a multi-layer perceptron module. The overall structure consists of an input part, a Patch Partition block structure, and four stages to extract hand features.

[0033] In the U-shaped network in S2, the number of feature channels also shows a U-shaped trend, first increasing and then decreasing, and the features obtained are also different. The introduction of the bottleneck residual unit helps the network to better transmit residual information, not only retaining and transmitting global and local information, but also capturing hand feature information at different scales and aggregating them. The bottleneck residual unit is a module used to build deep neural networks in deep learning. It is an improvement on the traditional residual network ResNet structure. By adding a bottleneck layer, more depth and nonlinear transformations are added to the network, making the network more adaptable to the extraction of multi-scale features and having better flexibility. For multi-resolution feature maps, the bottleneck residual unit can play a more effective role in hand feature extraction than ordinary residuals. The bottleneck residual unit is compared with the traditional structure. The structure, where Conv represents the convolution operation, is constructed through 1x1, 3x3 and 1x1 convolution kernels, which effectively reduces the number of parameters in the network, making the model lighter and easier to handle multi-scale features. The bottleneck residual structure reduces the number of input feature channels in the first 1x1 convolution layer, then performs deeper feature extraction in the second 3x3 convolution layer, and finally restores the depth of the feature in the third 1x1 convolution layer, so it can extract richer and more complex feature information while maintaining the size of the feature map. In addition, the effect of gradient propagation is particularly important under multi-resolution feature maps, because when extracting features at different scales, the stable propagation of gradients can help the network better learn feature representations at different scales. The bottleneck residual structure passes the input directly to the output through the residual connection, which helps alleviate the gradient vanishing problem and makes the network easier to train;

[0034] In the process of hand feature extraction in S2, the 48×48 feature map is first sent to the Swin Transformer Block, and the window size is set to 4×4. At this time, the image will be divided into 12×12 windows, and the Self-Attention calculation is performed in each window, and the regional pixels are converted into a one-dimensional vector as the effective input sequence length. Then, the residual connection is introduced through the MLP multi-layer perceptron for Linear Embedding linear encoding. In this process, the hand feature map obtained after W-MSA will pass through the SW-MSA structure again, and the window position will be moved by 2 pixels in the x and y directions at the same time, so as to transfer information between different windows. The remaining three stages are composed of Patch Merging operations to adjust the size and number of channels of the feature map, and the Swin Transformer Block module performs a series of linear transformations and attention mechanism operations. Each stage will change the dimension of the tensor, so as to capture hand features at different levels and assist the model in understanding feature information globally. Finally, the Layer Norm normalization layer is used to maintain the consistency of the distribution and obtain the corresponding output results.

[0035] S3, heat map supervision unit: supervise and correct the extracted feature information by adding a Gaussian heat map with joint point distribution information; when S3 uses a Gaussian heat map with joint point distribution information to supervise and correct the extracted feature information, it is necessary to cycle through these steps twice to obtain more abundant and accurate key information of the hand. The variance of the Gaussian distribution in S3 is usually controlled by a pre-set parameter. The highest intensity value of each pixel is the position of the key point, while the intensity values ​​of the surrounding pixels gradually decrease. The Gaussian distributions generated by all key points are superimposed to obtain the heat map value of the pixel. Therefore, by finding the position with the highest confidence value in the heat map, the key points in the hand can be accurately located. At the same time, the topological connection of the joint points is also retained according to the peak distribution. Therefore, the model can process multiple key points at the same time, and the model can better understand the structure and dynamics of the hand and make more accurate predictions.

[0036] After ablation self-contrast experiments, it was found that adding heat map supervision after fusion has the best effect. The structural design of the heat map supervision module is as follows: Figure 5 As shown in the figure: the input dimension is a two-dimensional matrix, which contains the distribution information of key points, where the intensity value of each pixel represents the confidence of the key point; the role of the convolution kernel is to slide on the key point heat map, and the number of input feature channels is increased to 256 through a 1×1 convolution, and the learned weights are used to identify and capture the patterns and features of the hand key points. The features processed by the 1×1 convolution are then processed by a 3×3 convolution operation to output a 24×24 Gaussian heat map with the number of channels corresponding to the number of hand key points, and the patterns and features of the hand key points are extracted from the input features.

[0037] S4, posture estimation unit: After the fused joint point features are corrected and processed, the interactive posture is predicted. This layer adopts a regression network structure and maps the fused features to the three-dimensional coordinate space of the joint points through a fully connected layer;

[0038] S5, data source output unit: Finally, the features containing rich information are sent to the back-end network to obtain the coordinate information of the hand joint points.

[0039] In a further embodiment, in S1, when the rough gesture acquisition unit processes and then enters the fine gesture feature acquisition unit for processing, a relatively fine hand feature at the initial resolution is obtained through a pooling unit and two bottleneck residual units.

[0040] The experimental operations and analyses related to this scheme are as follows:

[0041] Experimental environment and configuration: The experimental operating system uses Ubuntu 16.04, the hardware development environment is Intel (R) Xeon (R) Platinum 8255C CPU @ 2.50GHz, the graphics card memory is 40GB, and the GPU is RTX 3080. The deep learning framework is PyTorch 1.7.1, and the CUDA version is 10.2. The model presentation learning rate is set to 0.001 and the momentum is set to 0.9. The total Epochs of training is 300, and the batch size is set to 64. All experiments use the adaptive optimizer Adam with default hyperparameters, and training, verification and testing are performed under the same hyperparameters;

[0042] Experimental data set: It mainly includes two parts according to the research background and application scenarios. The first part is a self-collected data set, which collects hand data in complex backgrounds and uses the 21-joint point labeling method to label the hand joints. The second part is the MSRA data set, which contains the corresponding depth map and joint point labeling. In order to make the subsequent experimental data labels clear, this article simplifies the 21-joint point labeling information and represents the five fingers of the hand: little finger, ring finger, middle finger, index finger and thumb with P, R, M, I, T respectively. Each finger is represented by a fingertip TIP and three other joints DIP, PIP, MCP, and the center of the wrist is represented by Wrist. In the experimental charts, for the convenience of explanation, TIP, DIP, PIP, and MCP are further simplified to be replaced by numbers 1-4, and Wrist is replaced by C;

[0043] 3. Comparative method experimental results: In order to evaluate the performance of the proposed algorithm in the hand detection task compared with other mainstream detection algorithms, a comparative experiment was conducted. DeepModel, REN, DeepPrior++, PoseREN and DenseReg were selected as the comparison network models. The two key indicators of the average error of the hand joint points and the average prediction frame rate were compared respectively. The experimental results are shown in Table 1;

[0044] Table 1 Comparative experimental results

[0045]

[0046] It can be seen that compared with several mainstream hand posture estimation algorithms, the MPJPE value of the proposed method is lower than other methods, the estimation error of hand joints is smaller, and the detection effect is better. Although the average prediction frame rate is lower than that of DeeplModel and REN model, it is within a reasonable range. This proves the advantages of the proposed method in hand posture estimation methods;

[0047] 4. Ablation experiment results: In order to verify the effectiveness of the proposed global feature extraction module (A), multi-resolution local feature aggregation module (B), and heat map supervision module (C) for the prediction network, this section designs an ablation experiment. Table 2 shows the experimental results of superimposing different modules on the basis of the baseline model, which verifies the impact of different improvements on the hand joint point estimation error;

[0048] Table 2 Ablation experiment results

[0049]

[0050] It can be seen that adding the global feature extraction module and the multi-resolution local feature module to the baseline model respectively reduces the error estimation of the hand joint points to a certain extent. At the same time, the error value produced by the global feature extraction module and the multi-resolution local feature module is the smallest, which is reduced by 4.541, proving that the proposed model is effective for hand posture estimation. In addition, under the correction of the heat map supervision module, the predicted joint points are clearer than the background in the image, guiding the network to learn better autonomously, further reducing the joint point prediction error, proving the effectiveness of this step;

[0051] In the above scheme, B-RES is the bottleneck residual unit, POOL is the pooling unit, Swin Transformer is the global feature unit of the hand joints, U-shaped network is the local feature unit of the hand joints, Window MSA abbreviation W-MSA is the window self-attention module, Window MSA, namely SW-MSA is the shift window self-attention module, and UP is the upsampling structure;

[0052] In summary, this scheme uses Swin Transformer and CNN to capture global and local features respectively, and optimizes the feature extraction process through feature fusion and two-dimensional Gaussian heat map supervision correction. The effectiveness of the proposed method is verified by comparison and ablation experiments.

[0053] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0054] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A dual-branch hand interaction posture estimation model combining skeleton features, characterized in that: The following steps are involved: S1, data source input unit: first, the hand area depth image is sent to a coarse gesture feature acquisition unit of a hand feature extraction network, firstly a 7×7 convolution layer is used to obtain a coarse hand feature, and then a fine gesture feature acquisition unit is used to obtain a relatively fine hand feature at the initial resolution; S2, feature fusion unit: Then, the hand features are sent to the dual-branch acquisition unit in parallel. One branch is a local feature unit of the hand joints under the multi-resolution aggregation output of the U-shaped network to obtain the local feature information of each hand joint, and the other branch is a global feature unit of the hand joints to obtain the global feature information containing the topological connection of the hand joints under the attention calculation. After merging, they are summarized and fused by the dual-branch fusion unit. S3, heat map supervision unit: supervises and corrects the extracted feature information by adding a Gaussian heat map with joint point distribution information; S4, posture estimation unit: After the fused joint point features are corrected and processed, the interactive posture is predicted. This layer adopts a regression network structure and maps the fused features to the three-dimensional coordinate space of the joint points through a fully connected layer; S5, data source output unit: Finally, the features containing rich information are sent to the back-end network to obtain the coordinate information of the hand joint points.

2. A dual-branch hand interaction posture estimation model combining skeleton features according to claim 1, characterized in that: In S1, when the rough gesture acquisition unit processes and then enters the fine gesture feature acquisition unit for processing, a relatively fine hand feature at the initial resolution is obtained through a pooling unit and two bottleneck residual units.

3. The dual-branch hand interaction posture estimation model combining skeleton features according to claim 2, characterized in that: When the fine gesture feature acquisition unit in S1 is processed, the network integrates feature information at different levels through top-down and bottom-up feature propagation. In the top-down process, the bottleneck residual unit and the maximum pooling unit appear in groups, and the network starts to reduce the resolution at twice the speed from the high-resolution feature map. Each step will enhance the detail information of the feature, and the number of feature channels will also increase exponentially. In the bottom-up process, the bottleneck residual structure and the upsampling structure appear in groups. This part requires that the output result of each layer before is upsampled twice in order to restore the feature map resolution. This step is also added to the low-level hand features processed by the bottleneck residual unit at the same resolution through jump connections. At the same time, the network starts to increase exponentially from the low-resolution feature map, and the number of feature channels also decreases at twice the speed until it is finally restored to the original state. The feature obtained at this time is the aggregation of hand feature information at four different resolutions.

4. The dual-branch hand interaction posture estimation model combining skeleton features according to claim 1, characterized in that: In the U-shaped network in S2, the number of feature channels also shows a U-shaped trend, first increasing and then decreasing, and the features obtained are also different. The introduction of the bottleneck residual unit helps the network to better transmit residual information, which not only retains and transmits global and local information, but also captures and aggregates hand feature information at different scales.

5. The dual-branch hand interaction posture estimation model combining skeleton features according to claim 1, characterized in that: The global feature unit of the hand joints in S2 includes two key parts, the first part is the window self-attention module and the second part is the shifted window self-attention module. Each part is composed of a normalization layer module, an attention module, and a multi-layer perceptron module. The overall structure is composed of an input part, a Patch Partition block structure, and four Stages to extract hand features.

6. The dual-branch hand interaction posture estimation model combining skeleton features according to claim 1, characterized in that: In the process of hand feature extraction in S2, the 48×48 feature map is first sent to the SwinTransformer Block, and the window size is set to 4×4. At this time, the image is divided into 12×12 windows, and Self-Attention calculation is performed in each window, and the regional pixels are converted into a one-dimensional vector as the effective input sequence length. Then, the residual connection is introduced through the MLP multi-layer perceptron for Linear Embedding linear encoding.

7. The dual-branch hand interaction posture estimation model combining skeleton features according to claim 1, characterized in that: When the extracted feature information is supervised and corrected by a Gaussian heat map with joint point distribution information in S3, it is necessary to cycle through such steps twice to obtain more abundant and accurate key hand information.

8. The dual-branch hand interaction posture estimation model combining skeleton features according to claim 1, characterized in that: The variance of the Gaussian distribution in S3 is usually controlled by a pre-set parameter. The highest intensity value of each pixel is the position of the key point, and the intensity values ​​of the surrounding pixels gradually decrease. The Gaussian distributions generated by all key points are superimposed to obtain the heat map value of the pixel. Therefore, by finding the position with the highest confidence value in the heat map, the key points in the hand can be accurately located. At the same time, the topological connection of the joint points is also retained according to the peak distribution. Therefore, the model can process multiple key points at the same time, and the model can better understand the structure and dynamics of the hand and make more accurate predictions.

Citation Information

Patent Citations

  • Hand posture estimation system and method based on RGBD fusion network

    CN110175566A

  • Two-dimensional hand posture estimation method and device, equipment and storage medium

    CN115511968A

  • Light-weight three-dimensional hand posture estimation method based on RGB image

    CN115880724A

  • Hand model reconstruction method based on feature interaction

    CN116563452A

Cited By

  • Hand posture estimation method based on cross-scale cross-space feature interaction

    CN119964236A