XR gesture tracking and recognition method based on multi-view depth fusion
By employing a multi-view deep fusion method, combining multi-view image information with hand kinematic constraints, the cumulative error and insufficient semantic relevance of existing multi-view visual gesture tracking methods are resolved. This achieves high-precision and highly robust gesture tracking and recognition, enhancing the interactive experience in XR environments.
Patent Information
- Application Number
- CN202511123317.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-12
- Publication Date
- 2025-12-05
AI Technical Summary
Existing multi-view vision gesture tracking methods rely heavily on high-precision camera calibration, which easily leads to cumulative errors. Furthermore, they fail to fully explore the semantic relationships between multiple viewpoints, resulting in deviations between gesture posture output and actual motion patterns, and lack modeling of the kinematic characteristics of human hand joints.
By employing a multi-view deep fusion method, combining multi-view image information with hand kinematic constraints, and using perspective cropping to reduce distortion, high-precision 3D hand pose is generated through feature fusion using a multi-head attention mechanism and camera intrinsic and extrinsic parameters. Furthermore, efficient classification is performed using joint rotation angles and spatial coordinates, and the hand pose is calculated by combining a forward kinematics model with a linear hybrid skinning algorithm.
It significantly improves the occlusion robustness and pose reconstruction accuracy of gesture interaction, enhances generalization ability, achieves high-precision and high-robust gesture tracking and recognition, and improves the interactive experience in XR environment.
Smart Images

Figure CN121074976A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of gesture tracking for XR (Extended Reality, XR) natural interaction, and particularly relates to a multi-view depth fusion XR gesture tracking and recognition method. BACKGROUND
[0002] In recent years, the global has listed Extended Reality (XR) and the metaverse as strategic priorities, China is striving to promote the research and development and industrialization of VR, AR and other technologies, and especially emphasizes breaking through core interaction capabilities such as gesture recognition and eye tracking. Gesture interaction, as the most natural and intuitive human-computer interaction method in the XR environment, has become one of the core functions of the extended reality system. Gesture recognition technology enables users to directly control virtual objects without holding a controller, such as grabbing and throwing objects in VR games, or simulating real body language in virtual meetings. Gesture interaction does not require additional hardware (such as a joystick or glove), which can significantly reduce user learning costs while improving immersion and operational efficiency.
[0003] Current mainstream gesture tracking and recognition technologies mainly fall into two categories: vision-based methods and sensor-based methods. Vision-based non-contact gesture interaction has become the mainstream technology in the XR field to improve interaction convenience, as it does not require additional devices and conforms to natural interaction intuition. Vision-based methods are mainly divided into monocular vision methods and multi-view vision methods, but monocular cameras lack depth perception capabilities, making it difficult to accurately track three-dimensional gesture poses, so multi-view vision methods are often used in practical applications. Multi-view vision methods fuse multi-view image information to calculate the three-dimensional position and motion trajectory of the hand, effectively overcoming the limitations of monocular methods and significantly improving recognition accuracy and system robustness.
[0004] However, current multi-view-based gesture tracking methods mostly use geometric constraints (such as triangulation) to directly estimate 3D key point coordinates, which has the following limitations: first, this method highly depends on high-precision camera calibration, and calibration errors can accumulate during calculation, and it fails to fully exploit the semantic correlation between multiple views, limiting the fusion effect of multi-view information; second, it lacks modeling of human hand joint kinematics, resulting in deviations between output poses and real hand motion patterns. SUMMARY
[0005] In view of the technical bottleneck, the application provides a multi-view depth fusion XR gesture tracking and recognition method. The application realizes high-precision and high-robustness gesture tracking and recognition by combining multi-view image information and hand kinematics constraints. Compared with the traditional method relying on single-view two-dimensional features, the application uses multi-view feature fusion and three-dimensional joint degree of freedom regression, breaks through the occlusion, blind area and background interference, directly generates high-precision three-dimensional hand posture, and efficiently classifies through joint rotation angle and spatial coordinate dual information, not only improves the occlusion robustness and posture reconstruction accuracy, but also has stronger generalization ability, significantly improves the gesture interaction experience in the XR environment.
[0006] The technical scheme of the application is specifically introduced as follows.
[0007] A multi-view depth fusion XR gesture tracking and recognition method, comprising the following steps:
[0008] (1) A perspective cropping method is used to construct a virtual camera to position and crop the hand region of the multi-view image, reducing the perspective distortion (i.e. geometric distortion mainly caused by shooting angle and distance) of the hand region image;
[0009] (2) The two-dimensional gesture features extracted from the cropped image are mapped to the three-dimensional space by combining the internal and external parameter information of the virtual camera, and the position coding information is supplemented to realize more accurate image view space representation; a multi-head attention mechanism is used to fuse the three-dimensional features from multiple views of multiple virtual cameras to generate a unified hand feature representation;
[0010] (3) The pose regression module based on the multi-layer residual structure is used to predict the pose parameters of the hand from the fused multi-view features, i.e. the 6 degree of freedom (6DoF) pose information of the wrist joint and the rotation angle of each joint; based on the predicted pose parameters of the hand, the forward kinematics model and the linear blend skinning algorithm are used to calculate the complete hand pose and skeletal key point coordinates, realizing gesture tracking; based on the skeletal key point coordinates and the joint rotation angle, gesture recognition is realized through template matching.
[0011] In the application, in step (1), in the perspective cropping process, the first frame uses a lightweight gesture detection model to extract the hand region, and the subsequent frames use the skeletal key points output by the model of the previous frame for perspective cropping to obtain the hand region; at the same time, the perspective cropping also constructs virtual camera parameters for subsequent algorithms to use for multi-view feature fusion. In a specific embodiment, the lightweight gesture detection model uses the lightweight gesture detection model (palm_detection_full.tflite) of MediaPipe.
[0012] In the application, in step (2), a ResNet backbone network is used as a feature extraction module to extract two-dimensional gesture features S from the image C×d , wherein C represents the number of perspectives, and d represents the feature dimension of each perspective; the gesture features of the cropped image are extracted by using the ResNet backbone network, so that efficient and robust feature expression can be realized.
[0013] In the application, in step (2), the two-dimensional features are back-projected to a three-dimensional space by using an extrinsic parameter matrix [R|t] of a virtual camera SE(3) and combining an intrinsic parameter matrix K as follows: , wherein is a 3D feature point formed by channel splicing of the two-dimensional features; the intrinsic parameter matrix and the extrinsic parameter matrix are both constructed in the perspective cropping process.
[0014] In the application, in step (2), the position encoding information includes camera extrinsic parameter encoding PE ext , cropping frame encoding PE crop and sine-cosine position encoding PE sin ; finally, the features of each perspective joint are formed by splicing the image features S and the first two types of position encodings, and the sine-cosine encoding is added: F = concat (S, PE crop , PE ext ) + PE sin , F R C×d , C represents the number of perspectives, and d represents the feature dimension.
[0015] In the application, in step (2), the input features of the multi-head attention mechanism include gesture feature vectors and position encodings containing camera intrinsic and extrinsic parameters, the feature correlation between different perspectives is calculated through self-attention weight, and the fused global hand features are output.
[0016] In the application, in step (3), the pose regression module based on the multi-layer residual structure includes a regressor, the regressor includes a plurality of series-connected residual blocks, an adaptive average pooling layer and a linear regression head, each residual block includes two convolutional layers, batch normalization and a nonlinear activation function; the regressor outputs joint rotation angles in a world coordinate system and wrist joint sampling points, and the wrist joint sampling points can obtain wrist joint 6DoF position information through Procrustes alignment. In a specific embodiment, the number of joint rotation angles is 20, and each finger corresponds to 4 degrees of freedom of joints.
[0017] In the application, in step (3), the forward kinematics and linear hybrid skinning algorithm are based on a skeletal chain model, the skeletal rotation is driven by joint angles, the 3D coordinates of skeletal key points and global gesture poses are calculated by combining the weight relationship between the hand model grid vertex and the skeleton. In a specific embodiment, the skeletal key points are 21.
[0018] In the present application, in step (3), KDTree based on a pre-constructed template library is used to accelerate the KNN algorithm to realize gesture recognition. The KNN algorithm calculates the matching degree of the candidate template and the current gesture feature based on the Euclidean distance, and then determines the gesture type according to the matching degree. The specific steps include:
[0019] The joint rotation angle predicted by the posture regression module and the skeletal key point coordinates calculated by the linear blend skinning algorithm together constitute the feature vector of the current gesture.
[0020] The feature vector of the current gesture is used to perform nearest neighbor search in the pre-constructed template library, and KDTree is used to accelerate the search to quickly locate a number of template samples most similar to the input feature.
[0021] The gesture categories of the K neighboring samples searched are then counted, and the majority voting method is used to determine the final category of the input gesture.
[0022] In the present application, the pre-constructed template library includes three types of 20 gestures, and the three types of gestures are natural interactive action gestures, digital counting gestures, and cultural and symbolic gestures.
[0023] Compared with the prior art, the present application has the following advantages:
[0024] (1) Multi-view cooperation improves gesture tracking accuracy and robustness: perspective cropping method and multi-head attention mechanism are used to model the geometric relationship between views, and the semantic correlation of multi-view 3D features is deeply mined to realize more accurate and robust multi-view semantic fusion. On the one hand, it solves the problem that traditional multi-view methods highly depend on high-precision camera calibration and are prone to cumulative errors; on the other hand, it solves the problem that traditional multi-view methods fail to fully mine the semantic correlation between multi-views, limiting the feature fusion effect.
[0025] (2) Kinematic constraint enhances accuracy: forward kinematics and linear skinning (LBS) algorithm are introduced to convert joint rotation angles into 3D key point coordinates consistent with human kinematics, avoiding the physical inconsistency problem caused by traditional triangulation methods to estimate gesture posture through geometric constraints, significantly improving the posture restoration accuracy. BRIEF DESCRIPTION OF DRAWINGS
[0026] Figure 1 : XR gesture tracking and recognition method flowchart based on multi-view fusion.
[0027] Figure 2 : Perspective cropping principle and effect diagram.
[0028] Figure 3: Feature extraction backbone network structure diagram.
[0029] Figure 4 : Multi-view fusion module structure diagram.
[0030] Figure 5 : Hand 21 key point schematic diagram.
[0031] Figure 6 : Gesture category diagram.
[0032] Figure 7 : Gesture category table.
[0033] Figure 8 : Gesture category editor interface diagram. DETAILED DESCRIPTION
[0034] The technical solutions of the present application will be further described in detail below in combination with the drawings and examples.
[0035] The system of the present application comprises the following three core modules:
[0036] (1) Multi-view geometric correction module: To solve the hand region distortion problem caused by direct cropping, first perform hand gesture detection and perspective cropping on multi-view images. By constructing a virtual camera model, combined with the real-time hand gesture detection results of each frame of image, dynamically extract the geometrically corrected region of interest (ROI). This module uses a three-dimensional projection matrix to realize the conversion of image space to ergonomic view, ensuring that the subsequent processing obtains input data conforming to the true hand shape.
[0037] (2) Multi-view feature fusion module: To establish a global unified hand representation, a ResNet network is used to extract 2D gesture features from the corrected images, then through a feature conversion layer, the 2D gesture features are mapped to a 3D space, and position encoding information based on camera internal and external parameters is supplemented to preserve spatial relationships. Further introduce a multi-head attention mechanism, dynamically allocate weights to multi-view 3D features and model cross-view correlation, and finally generate a fusion feature vector with view invariance. Compared with traditional multi-view fusion methods, the present application realizes geometric consistency optimization in the feature fusion process by explicitly introducing camera space parameters, significantly improving fusion accuracy
[0038] (3) Kinematic constraint identification module: To make up for the deficiency of traditional methods in modeling kinematic characteristics, a kinematic constraint framework containing regressor is adopted. After the regressor outputs the joint rotation angle and 6DoF information of the wrist joint (i.e., the rotation and translation information of the wrist joint), the complete hand posture and 3D coordinates of the key points of the skeleton are calculated by combining the forward kinematic model and the linear hybrid skinning algorithm. Finally, based on the pre-defined gesture template library (containing standard joint rotation angle and key point coordinates of the skeleton), the KNN algorithm with KDTree acceleration search is used to realize high-robustness gesture classification.
[0039] Further, the present application proposes a multi-view depth fusion XR gesture tracking and recognition method (as shown in Figure 1 ), which mainly includes: gesture multi-view perspective cropping (to realize multi-view geometric correction), multi-view feature fusion, and gesture tracking, recognition and posture three-dimensional space reconstruction (kinematic constraint identification). Specifically, first, a virtual camera is constructed by using the perspective cropping method, the hand region is positioned and cropped in the multi-view image, and the perspective distortion of the hand region image is reduced. Then, a 3D space multi-view feature fusion method based on camera internal and external parameters is proposed, which uses camera internal and external parameters to map the two-dimensional representation extracted by the ResNet backbone network to the three-dimensional space, supplements the spatial position information under each view by external parameter coding, cropping frame coding and sine position coding, and uses the multi-head attention mechanism to fuse the three-dimensional features from multiple virtual cameras to generate a unified hand feature representation in the reference camera space. Finally, gesture tracking is completed through the regressor, forward kinematics and linear skinning algorithm, and high-precision and high-robustness gesture recognition is realized by using the KDTree accelerated KNN algorithm.
[0040] 1. Gesture detection and perspective cropping
[0041] 1.1 Implementation process of perspective cropping
[0042] In order to solve the inconsistency problem of feature extraction caused by perspective distortion, scale change and center offset of hand images under different views, the present application adopts a perspective cropping method (as shown in Figure 2 ), and the specific process is as follows:
[0043] 1) In the multi-view gesture tracking system, let the model of the i-th original camera be C i =(K i ,R i ,t i ), where is the internal parameter matrix, and (R i ,t i ) is the external parameter (rotation matrix and translation vector).
[0044] 2) First, based on the hand three-dimensional key point set estimated at the last moment Perform perspective projection on each point: u ij =Π(K i ,R i X j +t i ), where Π represents the standard projection operation from the camera coordinate system to the pixel plane.
[0045] 3) Then all u ij The bounding box is obtained by taking the minimum and maximum values of the horizontal and vertical coordinates on the pixel plane.
[0046] 4) To allow for extra margins, B i Enlarge the frame by a scaling factor s > 1 to obtain the cropping frame: Where scale indicates scaling the cropping box by a factor of s.
[0047] 5) Based on Construct a "virtual camera" with the output resolution (W′, H′): C i ′=(K i ′,R i ,t i ):
[0048] a. Main point:
[0049] b. Focal length:
[0050] c. Extrinsic parameter retention: R′ i =R i ,t′ i =t i .
[0051] Therefore, any world point X projected by the virtual camera is: u′=Π(K′) i ,R i X+t i It can directly crop and perspective correct hand sub-images, significantly reducing perspective distortion and scale differences.
[0052] 1.2 Handling of the first frame and the hand re-entering the field of view after it has moved out of view.
[0053] 1) In the first frame after system startup, or when the hand re-enters the field of view after being occluded or out of view, the previous 3D pose X is missing. In this case, a lightweight palm detector, palm_detection_full.tflite, is used to output a palm detection bounding box from the original image: B palm =[x min ,y min ,x max ,y max ];
[0054] 2) B palm By a fixed magnification factor s d >1 expansion: And using the same mapping relationship as 1.1, the is mapped to the output size (W', H'), and the virtual camera C' is constructed palm :
[0055] a. Principal point: b. Focal length: Where f init is the preset initial focal length.
[0056] 2. Extract gesture features using ResNet network
[0057] As Figure 3 shown, the ResNet network in this embodiment is mainly composed of the following parts in turn, to extract robust two-dimensional gesture features from the single-channel hand image after perspective correction and cropping:
[0058] 2.1 Initial convolution and down-sampling module
[0059] The input is a single-channel grayscale image after perspective cropping, with a size of HxW. First, 3x3 convolution (output channel number C0), BatchNorm and ReLU activation are performed, and then 2x2 max pooling (stride = 2) is performed to down-sample the feature map, reducing the spatial resolution to 1 / 2 of the original.
[0060] 2.2 Multi-layer residual block stacking
[0061] Modularly stack 4 residual layers according to the hierarchy shown in Figure 3 , each consisting of several BasicBlocks. Each BasicBlock contains two 3x3 convolutions + BatchNorm + ReLU, and a 1x1 convolution branch (Downsample) is used to align the residual branch in channels or space. The channel numbers of the four residual layers are: C0→C0→2C0→4C0, and the spatial resolutions are down-sampled as: H / 2→H / 4→H / 8→H / 8, W / 2→W / 4→W / 8→W / 8. Through this deep residual structure, the network can not only maintain gradient stability, but also effectively extract multi-scale hand geometry and texture features.
[0062] 2.3 Feature projection layer
[0063] After the output of the last layer of the stacked residual block, a 1x1 convolution is added for channel compression and feature mapping, and then the N-channel feature map output by the residual network is projected to the preset K-channel (i.e. the two-dimensional feature dimension required by the present application). The projection layer is used to remove redundant features and provide a unified feature representation for subsequent three-dimensional mapping and cross-view attention fusion.
[0064] 3. 2D feature mapping and spatial coding based on camera intrinsic and extrinsic parameters
[0065] The present application maps the two-dimensional gesture features extracted from the image to the three-dimensional space by combining the intrinsic and extrinsic parameters of the camera, and supplements various types of spatial position coding to enhance the sensitivity of the features to the view angle, scale and geometric structure. Specifically, it includes the following three aspects:
[0066] 3.1 Feature mapping from image space to three-dimensional space
[0067] After hand detection and perspective correction, two-dimensional gesture features are extracted from each input view image: S ∈ R C×d , where C represents the number of views, and d represents the feature dimension of each view. To map these features from the image plane to the three-dimensional space, the system uses the extrinsic matrix of each camera: [R | t] ∈ SE(3), and combines the intrinsic matrix K to project the two-dimensional features back to the three-dimensional space by the following calculation: , where is the 3D feature point formed by concatenating the two-dimensional features by channel. This operation allows all joint features under different views to be uniformly mapped to the reference three-dimensional space.
[0068] To address the problem of inaccurate representation caused by the lack of spatial information of image features in three-dimensional space, the present application proposes a spatial information modeling method based on position coding, which uses camera intrinsic and extrinsic parameters to compensate for the lack of spatial information, thereby achieving more accurate image view spatial representation:
[0069] (1) Camera extrinsic parameter coding (PE ext )
[0070] Use quaternions [q w , q x , q y , q z ] to represent rotation, where q w represents the rotation angle, q x , q y , q z represents the rotation axis, and vectors [t x , t y , t zrespectively, which constitute a 7-dimensional extrinsic parameter vector.
[0071] (2) Crop box encoding (PE crop )
[0072] Based on the detected hand crop box in each image, the pixel positions of its four corner points and the center point are extracted, and they are converted into the offset in the camera angle space by the following formula:
[0073]
[0074] where (p x , p y ) is the principal point coordinate, (f x , f y ) is the focal length, and (x, y) is the sampling point coordinate. A 10-dimensional encoding is generated to represent the geometric position of the crop box relative to the camera projection center.
[0075] (3) Sine-cosine position encoding (PE sin )
[0076] To give the model position perception ability, the classic position encoding method based on sine and cosine is introduced. For each view position p (p represents different views, p = 0 represents the first view), the following encoding is generated:
[0077]
[0078] where d is the final feature dimension, and k is the encoding dimension index. This encoding provides a structural and sequential hint for the attention mechanism.
[0079] (4) Feature representation based on multi-position encoding splicing
[0080] Finally, the feature of each view-joint is composed of image features and two types of position encoding splicing, and the sine-cosine encoding is added: F = concat (S, PE crop , PE ext ) + PE sin , F ∈ R C×d .
[0081] 4. Multi-view three-dimensional feature fusion and reference space mapping
[0082] The present application adopts a multi-head attention mechanism to fuse three-dimensional features under multiple views (as shown in Figure 4 ), so as to realize global modeling of the hand structure. Taking the first view as the reference view, its feature representation is F1 ∈ R d , and the remaining views are spliced into source features: In the attention layer, F1 is used as Query, F 2:C As Key and Value, multi-view attention is calculated:
[0083] F * = A·F 2:C
[0084] To enhance the expression ability, a multi-head parallel structure is adopted:
[0085] F at = MultiHead(F1,F 2:C ,F 2:C )∈R d
[0086] F at is the fused spatial feature.
[0087] 5. Pose regression model based on multi-layer residual structure
[0088] The present application designs a pose regressor based on a multi-layer residual structure, which is used to directly predict the pose parameters of a hand from the fused multi-view features. The output of the module includes the rotation angles of 20 joints of the hand (4 degrees of freedom for each finger) and the 6DoF pose information of the wrist joint.
[0089] The overall structure of the regressor is composed of several concatenated convolution residual blocks, each of which contains two convolution layers, batch normalization (BatchNorm) and a nonlinear activation function (ReLU). This structure has good deep feature modeling capability and is suitable for nonlinear modeling of high-degree-of-freedom targets such as gestures. After inputting the fused feature map, it is first passed through the residual feature extraction module, and the output size is [B, C, H, W], where B is the batch size, C is the number of channels, and H and W are the spatial dimensions. Then, through the global average pooling (Global Average Pooling) operation:
[0090]
[0091] where f gap ∈R C represents a one-dimensional feature vector after pooling, which is used as the input of the subsequent linear regression.
[0092] This vector is then mapped to the output vector f by a linear regression head.
[0093] · represents the rotation angles of 20 joints, represented by axis-angle;
[0094] · represents the predicted wrist sampling point set.
[0095] To ensure the geometric consistency and rigid body legality of the wrist pose parameters, the system applies an SVD-based alignment algorithm to rigidly fit the predicted point set in the training and inference stages. The specific steps are as follows:
[0096] Given the source point set and the predicted point set Rigid alignment is achieved by minimizing the following objective function:
[0097]
[0098] where R ∈ SO(3) is the rotation matrix, is the translation vector. It is calculated by singular value decomposition (SVD):
[0099] UΣV T = SVD(P T Q), R = VU T
[0100] The final R represents the wrist joint rotation matrix, and t represents the wrist joint translation vector.
[0101] The regression module outputs the joint rotation angle and wrist joint 6DoF pose in the world coordinate system, laying the foundation for subsequent forward kinematics calculation and gesture recognition. Compared with the traditional two-stage method based on intermediate pose estimation, the present invention realizes direct prediction from image features to motion parameters through end-to-end regression, with higher timeliness and accuracy, especially suitable for high-performance pose modeling in real-time gesture interaction scenarios.
[0102] 6. Gesture tracking and pose reconstruction based on kinematic constraints
[0103] In the present invention, to achieve accurate modeling and three-dimensional key point reconstruction of hand poses, forward kinematics (FK) and linear blend skinning (LBS) are used for pose estimation. The details are as follows:
[0104] 6.1 Forward kinematics modeling
[0105] Forward kinematics is used to recursively calculate the global transformation matrix of each joint in the world coordinate system based on the given wrist pose and joint rotation angle. For each finger, it is composed of several joints, and the local transformation of each joint is determined by its rotation axis, initial pose, and current angle. The local transformation matrix of each joint is The calculation formula is as follows:
[0106]
[0107] where ω i is the unit vector of the rotation axis of the ith joint; θ i is the rotation angle of the joint; t i is the position vector of the joint in the rest state; R(·) represents the 3D rotation matrix generated by the rotation vector. On this basis, the global transformation of each joint in the wrist coordinate system is recursively derived from bottom to top by chain matrix multiplication:
[0108]
[0109] where the transformation of the root node (wrist) is directly predicted by the regression module.
[0110] 6.2 Linear blend skinning modeling
[0111] In the linear blend skinning model, the final position of each skeletal key point (such as shown in Figure 5 ) after pose change is defined as the weighted sum of the influence of multiple skeletal transformations. For any key point v j , its position after linear blend skinning is calculated as follows:
[0112]
[0113] where: is the homogeneous coordinates of the point, T k is the 4 × 4 pose transformation matrix of the kth skeleton (i.e., joint); w jk is the skin weight of the key point to the kth skeleton, satisfying
[0114] 7. Gesture classification and recognition
[0115] The present application realizes efficient and accurate gesture classification by using the KNN algorithm through a predefined gesture template library. Specifically, the following contents are included:
[0116] 7.1 Defining 20 gestures
[0117] Considering the VR scene multi-user interaction environment and various dynamic virtual environments, in order to enhance the natural operation experience of users, support intuitive expression and collaboration efficiency in cross-cultural context, the following three types of gestures are designed (see Figure 6 , Figure 7 for specific gesture definitions):
[0118] 1) Natural interaction gestures: single finger pinch / grab simulates real grip, used for point and click or grab in VR / AR; fist (number "0") and palm open (number "5") are common base gestures. Meanwhile these gestures are widely supported in various VR devices, such as: Meta Quest 3, Pico support pinch / grab / point; Apple Vision Pro supports open / pinch gestures.
[0119] 2) Digital counting gestures: single index finger = "1", V shape = "2", three fingers = "3", four fingers = "4", five fingers = "5"
[0120] = "0", etc., while considering cultural differences, such as the difference in "3" gestures between north and south China, and defining the corresponding gestures.
[0121] 3) Cultural and symbolic gestures: including general symbols and popular culture, where general symbols include thumbs up (approval), OK gesture (affirmation / normal), rock gesture (passion), and two-finger pointing (indication). Popular culture includes single-hand heart (love / friendship, derived from Korean wave), and call (supporting idols, inspired by fans waving).
[0122] Meanwhile, the present application designs a special gesture definition interface (see Figure 8 ), through which the user can intuitively view and select the pre-defined 20 single-hand static gestures. The gesture definition interface clearly defines the joint rotation angle range and spatial characteristics of each gesture through the aid of joint rotation angle and bone key point space coordinates standardized data, making it convenient for the construction and matching process of the gesture template library.
[0123] 7.2 Gesture classification algorithm
[0124] The present application uses the KNN (K-Nearest Neighbor) algorithm based on KDTree to realize gesture classification, and the specific process is as follows:
[0125] 1) Input features: extract the hand rotation angle and bone key point coordinates output from the regressor to form the feature vector of the current gesture.
[0126] 2) KDTree query: use the feature vector to perform nearest neighbor search in the pre-constructed template library KDTree, quickly locate a number of template samples most similar to the input features.
[0127] 3) KNN classification decision: count the gesture categories to which the K neighboring samples belong, and determine the final category of the current input gesture using the majority voting method.
[0128] 4) Confidence calculation: according to the distance distribution of adjacent samples and input features, the confidence index of gesture recognition is calculated to provide reliable recognition feedback for upper-layer applications.
[0129] In summary, the present application firstly adopts perspective cropping method to construct virtual cameras, and locates and crops the hand region from multi-view images. Through gesture detection and perspective cropping of each frame of image, the hand region after geometric correction is dynamically extracted, effectively solving the problem of hand region distortion caused by direct cropping. Secondly, the present application proposes a 3D space multi-view feature fusion method based on camera internal and external parameters. The spatial position information under each view is supplemented by external parameter coding, cropping frame coding and sine position coding, and the multi-head attention mechanism is used to model the geometric relationship between views. The three-dimensional features from multiple virtual cameras are fused, the semantic correlation of multi-view 3D features is deeply mined, the global consistent hand representation is constructed, and the hand feature representation in the unified reference camera space is generated. Compared with the traditional multi-view fusion method, the present application realizes the geometric consistency optimization in the feature fusion process by explicitly introducing the camera space parameters, significantly improving the fusion accuracy. At the same time, aiming at the deficiency of the conventional geometric constraint method in the modeling of hand kinematics characteristics, the pose regression module is designed to output the 6 degrees of freedom (6DoF) information of the wrist joint and the rotation angle of each joint, and combined with the forward kinematics and linear skinning algorithm, the complete hand pose and 21 key point coordinates are generated. Finally, based on the key point position and pose parameters, the KDTree accelerated KNN algorithm is used to realize gesture recognition. This method realizes the collaborative optimization of geometric correction, semantic correlation modeling and kinematics constraint, significantly improves the accuracy and robustness of gesture recognition while ensuring real-time, and is suitable for various XR scenes.
Claims
1. A multi-view depth fusion based XR gesture tracking and recognition method, characterized in that, The method comprises the following steps: (1) A virtual camera is constructed by using a perspective cropping method to position and crop the hand region of the multi-view image, thereby reducing the perspective distortion of the hand region image; (2) The two-dimensional gesture features extracted from the cropped image are mapped to a three-dimensional space in combination with the internal and external parameter information of the virtual camera, and position coding information is supplemented to realize more accurate image perspective space representation; a multi-head attention mechanism is used to fuse the three-dimensional features from multiple virtual cameras under multiple perspectives to generate a unified hand feature representation; (3) A pose regression module based on a multi-layer residual structure is used to predict the pose parameters of the hand, i.e., the 6-DoF pose information of the wrist joint and the rotation angles of each joint, from the fused multi-view features; based on the predicted pose parameters of the hand, a forward kinematics model and a linear blend skinning algorithm are used to calculate the complete hand pose and the coordinates of the skeletal key points, thereby realizing gesture tracking; based on the coordinates of the skeletal key points and the rotation angles of the joints, gesture recognition is realized through template matching.
2. The multi-view depth-fused XR gesture tracking and recognition method of claim 1, wherein, In step (1), in the perspective cropping process, a lightweight gesture detection model is used to extract the hand region in the first frame, and in subsequent frames, the skeletal key points output by the model in the previous frame are used for perspective cropping to obtain the hand region.
3. The multi-view depth fused XR gesture tracking and recognition method of claim 1, wherein, In step (2), a ResNet backbone network is used as a feature extraction module to extract two-dimensional gesture features S∈R C×d where C represents the number of views, and d represents the feature dimension of each view.
4. The multi-view depth-fused XR gesture tracking and recognition method of claim 1, wherein, In step (2), the extrinsic matrix of the virtual camera is used: [R | t] e SE(3), and combined with the intrinsic matrix K, the two-dimensional features are back-projected to the three-dimensional space by the following calculation: wherein, is the 3D feature point formed by splicing the two-dimensional features according to the channel.
5. The multi-view depth-fused XR gesture tracking and recognition method of claim 1, wherein, In step (2), the position encoding information includes camera extrinsic parameter encoding PE ext , crop frame encoding PE crop , and sine-cosine position encoding PE sin ; finally, the feature of each view joint node is composed of image features S and the splicing of the first two types of position encodings, and the sine-cosine encoding is added: F = concat(S, PE crop , PE ext ) + PE sin , F ∈ R C×d , C represents the number of views, and d represents the feature dimension.
6. The multi-view depth-fused XR gesture tracking and recognition method of claim 1, wherein, In step (3), the pose regression module based on the multi-layer residual structure comprises a regressor, which includes a plurality of serially connected residual blocks, an adaptive average pooling layer and a linear regression head, each residual block includes two convolution layers, batch normalization and a nonlinear activation function; the regressor outputs the joint rotation angles and the wrist joint sampling points in the world coordinate system, and the 6DoF position information of the wrist joint can be obtained through Procrustes alignment.
7. The multi-view depth-fused XR gesture tracking and recognition method of claim 1, wherein, In step (3), the forward kinematics and linear blend skinning algorithm are based on a skeletal chain model, which drives the rotation of the skeleton through the joint angles, and calculates the 3D coordinates of the skeletal key points and the global gesture pose in combination with the weight relationship between the hand model grid vertices and the skeleton.
8. The multi-view depth-fused XR gesture tracking and recognition method of claim 1, wherein, In step (3), a KDTree-based KNN algorithm is used to realize gesture recognition based on a pre-constructed template library.
9. The multi-view depth-fused XR gesture tracking and recognition method of claim 8, wherein, The pre-constructed template library includes three types of 20 gestures, and the three types of gestures are natural interactive action gestures, digital counting gestures and cultural and symbolic gestures.
