Monocular 3D human gesture pose estimation method

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a monocular 3D human gesture posture estimation method.The application comprises a construction algorithm module, 3D gesture posture estimation is performed according to the 3D position and posture of a palm and the rotation angle of a finger joint of a human hand shape feature; the algorithm module comprises four sub-networks; the specific steps comprise obtaining picture data in a real environment through a camera, and inputting the picture into the algorithm module; the algorithm module processes the picture data; firstly, human hand detection is performed, and it is judged whether the hand is detected, then the region of interest is intercepted according to the detection result, the projection parameters are intercepted from the region of interest, and the 3D gesture posture of the human hand is calculated; the four sub-networks are sequentially analyzed and processed, and the 3D coordinates of the rotated finger joint and the gesture posture 3D coordinates of the translated palm are obtained; the application solves the problem that the existing reconstruction method depends on accurate 2D joint positioning, thereby greatly improving the accuracy of 3D posture estimation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a monocular 3D human gesture pose estimation method. Background Technology

[0002] Human hand morphological characteristics can be divided into two parts based on changes in hand shape: rigid body rotation (rotation of the palm) and rigid body deformation (flexion of the fingers). Rigid body rotation includes the rotational changes of the joints in the palm, and the relationships between these joints remain relatively fixed during hand gesture changes. 2D joint prediction for this part is less affected by occlusion than for the finger joints, resulting in relatively accurate and stable 2D joint prediction. Rigid body deformation includes the bending changes of each finger joint, which are constrained by the biological characteristics of the human hand. The joints at the base of the fingers exhibit two dimensions of rotational change: left-right swinging and bending. Typically, the left-right swinging range is relatively small, generally ±15 degrees; the bending range is relatively large, generally 0–110 degrees.

[0003] Currently, the general process for monocular 3D gesture pose estimation is to first obtain 2D joint information from the image, and then use the IK method to calculate the 3D gesture pose. The drawback of this approach is that self-occlusion of the gesture pose leads to inaccurate 2D joint prediction, which in turn results in inaccurate overall 3D pose estimation. Summary of the Invention

[0004] The purpose of this invention is to provide a monocular 3D human gesture pose estimation method with high estimation accuracy and convenient calculation.

[0005] The monocular 3D human gesture pose estimation method proposed in this invention includes constructing a monocular 3D human gesture pose estimation algorithm module. This module estimates the 3D gesture (human) pose based on the morphological features of the human hand, namely the 3D position and pose of the palm and the rotation angle of the finger joints. The monocular 3D human gesture pose estimation algorithm module includes: a sub-network. Subnetwork Subnetwork Subnetwork in:

[0006] Subnetwork I obtained from the camera mage Images and historical 2D finger joints 2D Hand Joints Parsing yields segmentation semantic features F seg 2D semantic features of fingers and 2D semantic features of the palm

[0007] Subnetwork Parameter K for projection from camera cam Hand template joint length B len The high-dimensional semantic features F are obtained by mapping. pcb ;

[0008] Subnetwork Used from 2D semantic features of the palm Hand segmentation semantic features F seg High-dimensional spatial semantic features F pcb The analysis yields the hand rotation R. palm and the 3D position of the palm T palm ;

[0009] Subnetwork Used to segment semantic features F from the hand seg 2D semantic features of finger joints Analysis yields finger joint rotation In order to further obtain the 3D coordinates of the finger joints after rotation and the 3D coordinates of the hand gesture after translation.

[0010] The sub-network Subnetwork Subnetwork Subnetwork Typically, they can be constructed using convolutional neural networks; they can also be constructed using fully connected networks, and in other cases, they can be constructed using other types of neural networks.

[0011] The monocular 3D human gesture pose estimation method proposed in this invention has the following specific steps:

[0012] (1) Obtain image data of the real environment through the camera and input the image into the algorithm module;

[0013] (2) The algorithm module processes the obtained image data; first, it detects a hand and determines whether a hand has been detected, then it extracts the region of interest based on the detection results, extracts the projection parameters from the region of interest, and calculates the 3D hand gesture posture; specifically:

[0014] (2.1) The I obtained by the camera mage Image, historical 2D finger joints 2D Hand Joints Input to obtain subnetwork Obtain the segmentation semantic features F seg 2D semantic features of fingers and 2D semantic features of the palm Among them, the segmentation semantic feature F seg Generate hand segmentation S eg Segmentation by hand egIt only participates during training to improve the segmentation of semantic features F. seg Expressive ability; does not participate in the reasoning process;

[0015] For historical finger 2D joints 2D Hand Joints Its 2D prior information L 2d It is obtained in the following way:

[0016] During the training process, L 2d =L 2d +N (N being noise) and the same set of data are repeatedly trained n times, with the current output as the input for the next iteration. This current output is called the 2D prior information L of the key points. 2d ;

[0017] 2D Prior Information L of Joints 2d During input, position encoding is performed first:

[0018] γ(L 2d )=(sin(2 0 πL 2d ), cos(2 0 πL 2d ), …, sin(2 n-1 πL 2d ), cos(2 n-1 πL 2d )), (1)

[0019] To improve the network's generalization ability; n is generally 7, but can also be adjusted to a suitable empirical value based on the actual training situation.

[0020] (2.2) Through 2D semantic features of fingers 2D semantic features of the palm Obtain the corresponding 2D finger joints 2D Hand Joints It can simultaneously improve the 2D semantic features of fingers 2D semantic features of the palm It has expressive power; it can also serve as prior information to provide prior information for the next reasoning;

[0021] (2.3) Project the camera projection parameters K cam Hand template joint length B len Input subnetwork Mapping yields high-dimensional semantic features F pcb ;

[0022] (2.4) The 2D semantic features of the palm obtained in step (2.1) Hand segmentation semantic features Fseg The high-dimensional spatial semantic features F obtained in step (2.2) pcb Input subnetwork Solve together the hand rotation R palm and the 3D position of the palm T palm ;

[0023] Specifically, subnetwork In the middle, based on the 2D semantic features of the palm joints Hand segmentation semantic features F seg Camera parameters, hand template joint length, high-dimensional semantic features F pcb This forms a qkv mutual attention mechanism; it allows for 3D hand position and pose search in space, yielding:

[0024]

[0025] (2.5) The hand segmentation semantic features F obtained in step (2.1) seg 2D semantic features of finger joints Input subnetwork Solving for finger joint rotation

[0026] (2.6) Finger joint rotation 3D coordinates of finger joints in hand template By f 1 Solving for the 3D coordinates of only the finger joints after rotation (Equation (3));

[0027] Hand template, hand joint 3D coordinates 3D coordinates of finger joints Rotate the palm R palm By f 2 Solve the problem by adding the hand's translation T. palm Obtain the 3D coordinates K of the gesture. 3d (Equation (4)):

[0028]

[0029]

[0030] In this invention, the sub-network Subnetwork Subnetwork Subnetwork They can be constructed from either convolutional neural networks or fully connected networks.

[0031] In this invention, the camera is any one of an RGB camera, a grayscale camera, or a depth camera.

[0032] The beneficial effects of this invention are as follows:

[0033] Based on the biological characteristics of the human hand, this invention transforms the 3D reconstruction of the human hand into the rotation and translation of the palm, plus the rotation of the finger joints. This greatly solves the problem of existing reconstruction methods relying on precise 2D joint localization (2D joint localization is limited by self-occlusion and cannot improve accuracy), thereby greatly improving the accuracy of 3D pose estimation. Attached Figure Description

[0034] Figure 1 This is a flowchart of the monocular 3D human gesture pose estimation method of the present invention. In the figure: I mage : Enter image, K cam : for projection parameters (f x f y (c) represents the camera's focal length; x c y B is the center point of the image; len : is the length of the hand template joint; F seg Segmenting semantic features by hand can yield S eg (Manual division); 2D semantic features of finger joints; 2D semantic features of palm joints; F pcb Camera projection parameters and hand template joint lengths are mapped to high-dimensional semantic features; S eg Manual segmentation, auxiliary results; 2D semantic features of finger joints The 2D position of the finger joint is obtained, and the auxiliary result is used as the 2D prior input for the next frame. 2D semantic features of palm joints The 2D position of the hand joint is obtained, and the auxiliary result is used as the 2D prior input for the next frame. Finger joint rotation (a biological characteristic of the finger, which can be represented as rotation around an axis from 0 to 90 degrees; only the rotation angle needs to be solved); R palm : Palm rotation; T palm 3D position of the palm; 3D coordinates of finger joints in a hand template; 3D coordinates of finger joints after solving; Hand template, 3D coordinates of hand joints; K 3d 3D coordinates of the hand joints after solving; K 3d This refers to the complete solution of 3D gesture poses.

[0035] Figure 2 This is a structural diagram of the virtual reality gesture interaction system of the present invention.

[0036] Figure 3This is a structural diagram of the hardware layer in this invention.

[0037] Figure 4 This is a structural diagram of the operating system layer in this invention.

[0038] Figure 5 This is a structural diagram of the application layer in this invention.

[0039] Figure 6 This is a flowchart illustrating the use of the present invention in the system.

[0040] The diagram is labeled as follows: 1. Hardware layer, 2. Operating system layer, 3. Application layer; 5. Camera device, 6. Processor, 7. Display; 8. Camera reading module, 9. Algorithm processing module, 10. Virtual interaction module; 20. Clock management module, 21. Interrupt handling module, 22. Device driver module, 23. Process management module, 24. Memory management module, 25. Device management module, 26. Non-kernel functional modules. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] The monocular 3D human gesture pose estimation method proposed in this invention is implemented through a virtual reality gesture interaction system, which consists of a hardware layer 1, an operating system layer 2, and an application layer 3, as follows: Figure 2 As shown.

[0043] Hardware layer 1 is located at the bottom layer, and its structure is as follows: Figure 3 As shown, the system includes a camera device 5, a processor 6, and a display 7. The camera 5 can generally be any one of an RGB camera, a grayscale camera, or a depth camera; the camera device 5 transmits data to the processor 6 for processing via an I / O bus. The processor 6 performs calculations and can be any one of a general-purpose CPU (such as an ARM or x86 processor) or a high-performance computing processor (such as a TPU, GPU, etc.). The processor 6 is connected to the display 7 via a video bus.

[0044] Operating system layer 2 runs on processor 6. Operating system layer 2 consists of a clock management module 20, an interrupt handling module 21, a device driver module 22, a process management module 23, a memory management module 24, a device management module 25, and a non-kernel function module 26. Its structure is as follows: Figure 4 As shown, the hardware layer 1 is controlled through the operating system layer 2, and the device driver module 22 is used to control the camera 5, the processor 6 and the display 7 respectively.

[0045] Application layer 3 includes camera reading module 8, algorithm processing module 9, and virtual interaction module 10, such as... Figure 4As shown, the output of the camera reading module 8 is connected to the input of the algorithm processing module 9, and the output of the algorithm processing module 9 is connected to the input of the virtual interaction module 10.

[0046] The camera reading module 8 is the data input module for the virtual reality gesture interaction system. It calls the lower-level driver layer to obtain image data from the real environment and then passes the data to the algorithm processing module 9.

[0047] Algorithm processing module 9 processes the acquired image data. First, it detects and determines whether a hand is detected. Then, based on the detection results, it extracts the region of interest (ROI) and its projection parameters. Using prior information, it calculates the 3D hand gesture posture. Virtual interaction module 10 models a 3D hand in virtual space based on the 3D hand gesture posture and interacts with objects in the virtual space, completing the virtual interaction. The interactive screen is then projected onto a display screen.

[0048] The usage process of the gesture interaction system of the present invention is as follows: Figure 5 As shown, the system first captures an image of the environment using a camera; then, the captured image is processed by an algorithm. The algorithm first detects whether there is a hand in the image; if so, it captures the portion of the image containing the hand and reconstructs a 3D hand from this portion. The resulting 3D hand is then used for interaction in a virtual environment, and the interactive image is projected onto a monitor.

Claims

1. A method for estimating human gesture pose in monocular 3D, characterized in that, This includes constructing a monocular 3D human hand gesture pose estimation algorithm module. This algorithm module estimates the 3D hand gesture pose based on the morphological features of the human hand, namely the 3D position and pose of the palm and the rotation angle of the finger joints. The monocular 3D human hand gesture pose estimation algorithm module includes: a sub-network. Subnetwork Subnetwork Subnetwork in: Subnetwork I obtained from the camera mage Images and historical 2D finger joints 2D Hand Joints Parsing yields segmentation semantic features F seg 2D semantic features of fingers and 2D semantic features of the palm Subnetwork Parameter K for projection from camera cam Hand template joint length B len The high-dimensional semantic features F are obtained by mapping. pcb ; Subnetwork Used from 2D semantic features of the palm Hand segmentation semantic features F seg High-dimensional spatial semantic features F pcb The analysis yields the hand rotation R. palm and the 3D position of the palm T palm ; Subnetwork Used to segment semantic features F from the hand seg 2D semantic features of finger joints Analysis yields finger joint rotation In order to further obtain the 3D coordinates of the finger joints after rotation and the 3D coordinates of the hand gesture after translation; The specific steps for monocular 3D human gesture pose estimation are as follows: (1) Obtain image data of the real environment through the camera and input the image into the algorithm module; (2) The algorithm module processes the obtained image data; first, it detects a hand and determines whether a hand has been detected, then it extracts the region of interest based on the detection results, extracts the projection parameters from the region of interest, and calculates the 3D hand gesture posture; specifically: (2.1) The I obtained by the camera mage Images and historical 2D finger joints 2D Hand Joints Input to obtain subnetwork Obtain the segmentation semantic features F seg 2D semantic features of fingers and 2D semantic features of the palm Among them, the segmentation semantic feature F seg Generate hand segmentation S eg Segmentation by hand eg It only participates during training to improve the segmentation of semantic features F. seg Expressive ability; does not participate in the reasoning process; For historical finger 2D joints 2D Hand Joints Its 2D prior information L 2d It is obtained in the following way: During the training process, L 2d =L 2d +N and the same set of data are used to perform repeated n iterations of training, where N is noise; the current output is used as the input for the next iteration of training, and this current output is called the 2D prior information L of the joint. 2d ; 2D Prior Information L of Joints 2d During input, position encoding is performed first: γ(L 2d )=(sin(2 0 πL 2d ),cos(2 0 πL 2d ),…,sin(2 n-1 πL 2d ),cos(2 n-1 πL 2d )),(1) To enhance network generalization capabilities; (2.2) Through 2D semantic features of fingers 2D semantic features of the palm Obtain the corresponding 2D finger joints 2D Hand Joints It can simultaneously improve the 2D semantic features of fingers 2D semantic features of the palm Expressive ability; and it also serves as prior information to provide prior information for the next reasoning; (2.3) Project the camera projection parameters K cam Hand template joint length B len Input subnetwork Mapping yields high-dimensional semantic features F pcb ; (2.4) The 2D semantic features of the palm obtained in step (2.1) Hand segmentation semantic features F seg The high-dimensional spatial semantic features F obtained in step (2.2) pcb Input subnetwork Solve together the hand rotation R palm and the 3D position of the palm T palm ; Specifically, subnetwork In the middle, based on the 2D semantic features of the palm joints Hand segmentation semantic features F seg Camera parameters, hand template joint length, high-dimensional semantic features F pcb This establishes a qkv mutual attention mechanism; by performing a 3D position and pose search for the hand in space, the following results are obtained: (2.5) The hand segmentation semantic features F obtained in step (2.1) seg 2D semantic features of finger joints Input subnetwork Solving for finger joint rotation (2.6) Finger joint rotation 3D coordinates of finger joints in hand template By f 1 Solving for the 3D coordinates of only the finger joints after rotation Hand template, palm joint 3D coordinates 3D coordinates of finger joints Rotate your palm R palm By f 2 Solve the problem by adding the hand's translation T. palm Obtain the 3D coordinates K of the gesture. 3d : .

2. The monocular 3D human gesture pose estimation method according to claim 1, characterized in that, The sub-network Subnetwork Subnetwork Subnetwork They can be constructed from either convolutional neural networks or fully connected networks.

3. The monocular 3D human gesture pose estimation method according to claim 1, characterized in that, The camera is any one of an RGB camera, a grayscale camera, or a depth camera.

Images