A posture estimation method and device based on multi-view rendering

Abstract

The present invention provides a posture estimation method and device based on multi-view rendering, the method comprising the following steps: detecting an object area in a depth image; extracting a three-dimensional point cloud of the object from the object area of ​​the depth image; evenly setting multiple virtual cameras on a sphere centered on the three-dimensional point cloud of the object; selecting a small number of virtual cameras suitable for object posture estimation from the set virtual cameras through a neural network; projecting the three-dimensional point cloud of the object onto the selected virtual camera, rendering object depth images of multiple perspectives; using a three-dimensional posture estimation network to perform three-dimensional object posture estimation on depth images of multiple perspectives; fusing the three-dimensional object postures of multiple perspectives to obtain the final three-dimensional object posture. After actual use verification, the present invention has the advantages of high automation, high precision and real-time performance, and can meet professional or popular application needs.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, gesture recognition, and human-computer interaction, and specifically relates to a posture estimation method and device based on multi-view rendering, which is applicable to the entire or partial human body (such as the human body, human hand, human face, etc.), animals, robots, and general rigid or non-rigid objects. Background Art

[0002] Pose estimation, especially hand pose estimation, is a hot topic in computer vision and human-computer interaction, with widespread applications in virtual reality, intelligent control, and terminal devices. The task of hand pose estimation primarily involves detecting the hand's position from an image and estimating the 3D coordinates of all associated joints. Hand pose estimation methods can be categorized into two types: RGB image-based and depth image-based. Compared to RGB image-based methods, depth image-based methods incorporate more depth information into their input, resulting in more accurate 3D hand joint coordinates.

[0003] Current depth image-based pose estimation methods primarily extract features directly from depth images, and the accuracy of pose estimation depends on the size of the extracted features. While extracting too many features can improve accuracy, it also leads to a significant increase in parameters and a decrease in speed. Because depth images contain 2.5D information rather than 3D information, the two-dimensional convolution commonly used for feature extraction cannot fully extract the depth information in the image. Therefore, balancing speed and accuracy in hand pose estimation tasks, and improving the accuracy of hand pose estimation while ensuring real-time performance, is a key challenge in the problem. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a posture estimation method and device based on multi-view rendering, which is applicable to the whole or part of the human body (such as the human body, human hand, human face, etc.), animals, robots, and general rigid or non-rigid objects.

[0005] The present invention provides a posture estimation method based on multi-view rendering, the method comprising the following steps:

[0006] Detect object regions in the depth image;

[0007] Extracting a three-dimensional point cloud of the object from the object region of the depth image;

[0008] Multiple virtual cameras are evenly arranged on a spherical surface centered on the three-dimensional point cloud of the object;

[0009] Selecting a small number of virtual cameras suitable for object pose estimation from a plurality of virtual cameras set by a neural network;

[0010] Project the 3D point cloud of the object onto the selected virtual camera to render depth images of the object from multiple perspectives;

[0011] Through the 3D pose estimation network, the depth images of objects from multiple perspectives are used to estimate the 3D poses and obtain the 3D object poses from multiple perspectives.

[0012] The 3D object poses from multiple views are fused to obtain the final 3D object pose.

[0013] Here, "object" refers to the object for posture estimation, such as the whole or part of the human body (such as the human body, human hand, human face, etc.), animals, robots, and other general rigid or non-rigid objects.

[0014] Furthermore, detecting the object area in the depth image includes: performing object detection or segmentation on the depth image to obtain the object image area with the background eliminated.

[0015] Furthermore, the three-dimensional point cloud of the object is extracted from the object area of ​​the depth image, and the formula used to extract the three-dimensional point cloud of the object is as follows:

[0016]

[0017] Among them, u, v, d are the image coordinates and depth of the image point p in the foreground area of ​​the object in the depth image; K is the intrinsic parameter matrix of the camera; M is the extrinsic parameter matrix of the camera, which represents the rotation and translation of the camera coordinate system and the world coordinate system, and the expression is Where R represents the rotation matrix of the camera, t represents the translation vector of the camera; p w =(x w ,y w ,z w ) is the three-dimensional coordinate of the image point p after being converted to the world coordinate system; all image points in the foreground area of ​​the object are transformed to the point cloud P in the world coordinate system w .

[0018] Furthermore, the evenly arranging multiple virtual cameras on a spherical surface with the three-dimensional point cloud of the object as the center is to arrange the virtual cameras on a spherical surface with the centroid of the three-dimensional point cloud of the object as the center of the sphere.

[0019] Furthermore, the method of selecting a virtual camera suitable for object pose estimation from a plurality of virtual cameras set up through a neural network is to select the virtual camera using a view confidence network; the view confidence network extracts image features from the depth image of the object, and maps the image features into confidences of a plurality of preset virtual cameras, and selects K virtual cameras with the highest view confidence for subsequent operations.

[0020] Furthermore, the multi-view rendering formula used for rendering the depth images of the object from multiple perspectives is as follows:

[0021]

[0022] Among them, I k represents the depth image rendered by the kth virtual camera, I k (u, v) represents the depth value at the position (u, v) on the rendered depth image, and the point cloud P of the object in the world coordinate system is w Projected onto the kth virtual camera image, the image point set P is obtained k , for any point p∈P k , and its depth value is recorded as D k (p); S in the formula k,u,v ∈P k Represents the image point set P k The set of points p with pixel coordinates (u, v) in .

[0023] Furthermore, the three-dimensional pose estimation network is used to perform three-dimensional pose estimation on depth images of objects from multiple perspectives to obtain three-dimensional object poses from multiple perspectives. The rendered depth map is used as input, and a neural network is used to estimate the three-dimensional object poses from K perspectives respectively.

[0024] Furthermore, the fusion of the three-dimensional object posture from multiple perspectives is to fuse the coordinates of the key points of K perspectives (for the whole human body or parts such as human hands and faces, the key points can be joint points; for non-human objects, surface feature points or joint rotation centers can be used as key points). The formula used is as follows:

[0025]

[0026] in, and They represent the key point coordinates of the three-dimensional object estimated from the k-th perspective and the key point coordinates of the fused three-dimensional object, N represents the number of key points of the object, and R k and t k are the rotation matrix and translation vector of the kth virtual camera selected, c k is the confidence weight. After performing the SoftMax operation on the confidence values ​​of K virtual cameras, the confidence weight of the k-th perspective is c k .

[0027] The present invention also provides a posture estimation device based on multi-view rendering, the device comprising:

[0028] An object detection module is used to detect object areas in the depth image;

[0029] A view selection module is used to select a virtual camera suitable for object pose estimation from multiple virtual cameras;

[0030] The multi-view rendering module is used to render the depth image of the object captured by the virtual camera, that is, to project the 3D point cloud of the object onto the selected virtual camera and render the depth image of the object from multiple perspectives;

[0031] A 3D object pose estimation module is used to estimate the 3D object pose from the object depth images at multiple perspectives through a 3D pose estimation network to obtain the 3D object poses at multiple perspectives;

[0032] The perspective fusion module is used to fuse the 3D object postures from multiple perspectives to obtain the final 3D object posture.

[0033] The advantages and beneficial effects of the present invention are:

[0034] The main problem addressed by the present invention is to improve the accuracy of three-dimensional pose estimation tasks while ensuring real-time performance. The present invention provides a method and apparatus for extracting three-dimensional pose estimates from a single depth image. Performing three-dimensional pose estimation from multiple perspectives can effectively improve estimation accuracy. By selecting a small number of virtual cameras suitable for pose estimation from a uniformly arranged set of virtual cameras, the speed of pose estimation can be significantly increased without sacrificing excessive accuracy. Verified in actual use, the present invention has the advantages of high automation, high accuracy, and real-time performance, and can meet the needs of both professional and popular applications.

[0035] The pose estimation method of this invention can be applied to the entire human body, parts of the human body, animals or robots, and general rigid or non-rigid bodies. Parts of the human body include hands, upper limbs, lower limbs, or faces. This pose estimation method is applicable to input data from various 3D vision sensors, such as depth cameras, binocular cameras, and laser scanners. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the overall architecture diagram of the present invention.

[0037] Figure 2 It is a schematic diagram of using multiple virtual cameras according to the present invention.

[0038] Figure 3 Schematic diagram of the view confidence network of the present invention.

[0039] Figure 4 It is a schematic diagram of three-dimensional human hand posture estimation and fusion of the present invention. DETAILED DESCRIPTION

[0040] The method of the present invention is described below in detail by taking human hand posture estimation as an example.

[0041] Figure 1 The method of the present invention is summarized. The present invention uses a single depth image as input. First, human hands are detected on the input depth image, and after obtaining the hand bounding box, the depth map of the hand area is cropped from the depth image. The depth image of the hand area is input into the perspective confidence network, and the confidence of multiple preset virtual cameras is calculated, and the K virtual cameras with the highest confidence are selected. The human hand point cloud is extracted from the depth map of the human hand area, projected onto the selected K virtual cameras, and the human hand depth images of K perspectives are rendered. The three-dimensional human hand joint point coordinates of K perspectives are estimated through the three-dimensional posture estimation network, and finally these three-dimensional human hand joint points are fused to obtain the final three-dimensional human hand joint point coordinates.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to specific embodiments and accompanying drawings.

[0043] Step 1: Extract hand point cloud from depth image

[0044] The image points on the foreground area of ​​the hand in the depth image (depth value is not 0) can be mapped into a three-dimensional world coordinate system. Let the coordinates of the image point p on the foreground area of ​​the hand in the depth image be (u, v) and the depth value be d, where (u, v) represents the coordinates of the image point in the image coordinate system. Let the three-dimensional point in the world coordinate system corresponding to the image point p be p w =(x w ,y w ,z w ), then p w The calculation formula is:

[0045]

[0046] in:

[0047]

[0048]

[0049] K is the camera's internal parameter matrix, including f x ,f y , u0, v0 and other camera intrinsic parameters; M is the camera's extrinsic parameter matrix, representing the rotation and translation between the camera coordinate system and the world coordinate system, and R and t are the rotation matrix and translation vector, respectively. The position of the hand in the depth image is detected or segmented using hand detection or hand segmentation methods. The hand area is cropped from the depth image, and each image point with a non-zero depth in the hand area is converted to the world coordinate system to obtain the 3D point cloud Pw of the hand in the world coordinate system.

[0050] Step 2: Setting up the virtual camera

[0051] A sphere is constructed with the center of mass of the hand point cloud as the center of the sphere, and the radius is the depth value of the point on the pixel coordinate system corresponding to the center of mass. 25 virtual cameras are preset on the sphere, and the camera's azimuth angle α and altitude angle β on the sphere are used to represent the camera's position. Excessive azimuth and altitude angles will cause serious occlusion and holes in the rendered depth map, so the limit 25 preset virtual cameras are evenly arranged in a 5×5 grid in a restricted area. The rotation matrix R and translation vector t of each preset virtual camera are calculated to obtain the external parameters M of each virtual camera. The virtual camera effect is as follows Figure 2 shown.

[0052] Step 3: Select a virtual camera

[0053] The cropped depth image of the hand area is input into the view confidence network, and finally the confidence of 25 preset virtual cameras can be obtained. Figure 3 As shown. The network input is image I∈R H×W , H and W are the height and width of the cropped image. Image features are extracted using an image feature extraction network (such as a residual neural network (ResNet18)). These features are then mapped into confidence scores for 25 preset virtual cameras using a fully connected layer. The K virtual camera perspectives with the highest confidence scores are selected for subsequent operations, where 1 ≤ K ≤ 25 and K ∈ Z. K is a hyperparameter that can be selected based on computational speed requirements. A larger value of K results in slower computation.

[0054] Step 4: Multi-view rendering

[0055] Projecting the hand point cloud in the world coordinate system onto the selected K virtual cameras can render the hand depth images from K perspectives. Let M k Represents the external parameters of the kth selected virtual camera. According to formula (1), the hand point cloud P in the world coordinate system can be w Projection obtains the point set P in the kth virtual camera image coordinate system k , for any point p∈P k , whose depth value is D k (p). Let S k,u,v ∈P k Indicates P k The set of points p with pixel coordinates (u, v) in the image is set to the depth image rendered by the k-th virtual camera. k , I k (u,v) represents the depth value of the image point at (u,v), and its calculation formula is:

[0056]

[0057] Step 5: 3D hand pose estimation and fusion

[0058] Use a 3D hand pose estimation network to estimate the 3D hand joint coordinates of K perspective depth images respectively. Then use the confidence calculated in step 4 to fuse the K hand joint coordinates to obtain the final 3D hand joint coordinates. The present invention can use A2J network as the 3D hand pose estimation network. After performing SoftMax operation on the K confidence values ​​calculated in step 4, the confidence weight of the k-th perspective is c k .set up and They represent the three-dimensional hand joint coordinates estimated from the k-th view and the fused three-dimensional hand joint coordinates, and N represents the number of hand joints. The formula for fusing the joint coordinates of K views is:

[0059]

[0060] where R k and t k The rotation matrix and translation vector of the camera coordinate system and world coordinate system of the selected k-th virtual camera are respectively. The process of step 5 is as follows Figure 4 shown.

[0061] The solution of the present invention can be implemented by software or hardware, for example:

[0062] In one embodiment, a multi-view rendering human hand posture estimation device is provided, comprising:

[0063] The hand detection module is used to detect the hand area in the depth image;

[0064] A perspective selection module is used to select a virtual camera suitable for hand posture estimation from multiple virtual cameras;

[0065] The multi-view rendering module is used to render the depth image of the human hand captured by the virtual camera, that is, to project the 3D point cloud of the human hand onto the selected virtual camera and render the depth image of the human hand from multiple perspectives;

[0066] A 3D hand pose estimation module is used to estimate the 3D hand pose from the hand depth images at multiple perspectives through a 3D hand pose estimation network to obtain the 3D hand poses at multiple perspectives.

[0067] The perspective fusion module is used to fuse the 3D hand postures from multiple perspectives to obtain the final 3D hand posture.

[0068] In addition, the device may also include:

[0069] 1) Data preprocessing module, which is used to preprocess the input depth image. By cropping, scaling and data enhancement of the input image, the view selection and hand pose estimation are made more accurate.

[0070] 2) Network construction and training module, responsible for building and training the view selection network and hand pose estimation network.

[0071] The solution of the present invention can be applied to the entire or partial human body (such as the human body, human hands, human faces, etc.), animals, robots, general rigid or non-rigid objects, and is also applicable to input data from various three-dimensional vision sensors such as depth cameras, binocular cameras and laser scanners.

[0072] In another embodiment, an electronic device (computer, server, mobile phone, etc.) is provided, which includes a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0073] In another embodiment, a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0074] In the present invention, the basic network for 3D hand posture estimation in step 5 is not limited to the A2J network, and can be replaced by other similar basic networks.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the principles and scope of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A posture estimation method based on multi-view rendering, characterized in that: The following steps are involved: Detect object regions in a single depth image; Extracting a three-dimensional point cloud of the object from the object region of the depth image; A plurality of virtual cameras are evenly arranged on a spherical surface centered on the three-dimensional point cloud of the object; Selecting a virtual camera suitable for object pose estimation from a plurality of virtual cameras set by a neural network; Project the 3D point cloud of the object onto the selected virtual camera and render the depth image of the object from K perspectives; Through the 3D pose estimation network, the 3D pose of the object at K perspectives is estimated on the depth image, and the 3D pose of the object at K perspectives is obtained; Fuse the 3D object poses from K perspectives to obtain the final 3D object pose; The method of selecting a virtual camera suitable for object pose estimation from multiple virtual cameras through a neural network is to use a view confidence network to select the virtual camera; the view confidence network extracts image features from the depth image of the object, and maps the image features into the confidence of multiple virtual cameras preset, and selects K virtual cameras with the highest view confidence for subsequent operations; the view confidence network includes an image feature extraction network and a fully connected layer.

2. The method according to claim 1, characterized in that The detecting of the object area in the single depth image comprises: Perform object detection or segmentation on the depth image to obtain the object image area with the background eliminated.

3. The method according to claim 1, characterized in that The three-dimensional point cloud of the object is extracted from the object area of ​​the depth image, and the formula used to extract the three-dimensional point cloud of the object is as follows: Among them, u, v, d are the image coordinates and depth of the image point p on the foreground area of ​​the object in the depth image; K is the internal parameter matrix of the camera; M is the external parameter matrix of the camera, which represents the rotation and translation of the camera coordinate system and the world coordinate system, and the expression is Where R represents the rotation matrix of the camera, t represents the translation vector of the camera; p w =(x w ,y w ,z w ) is the three-dimensional coordinate of the image point p after being converted to the world coordinate system; transform all image points in the foreground area of ​​the object to the point cloud P in the world coordinate system w .

4. The method according to claim 1, characterized in that: The method of evenly arranging a plurality of virtual cameras on a spherical surface with the three-dimensional point cloud of the object as the center is to arrange the virtual cameras on a spherical surface with the centroid of the three-dimensional point cloud of the object as the center of the sphere.

5. The method according to claim 1, characterized in that The formula for rendering the depth image of the object from multiple perspectives is as follows: Among them, I k represents the depth image rendered by the kth virtual camera, I k (u,v) represents the depth value at the position (u,v) on the rendered depth image, and the point cloud P of the object in the world coordinate system w Projected onto the kth virtual camera image, we get the image point set P k , for any point p∈P k , and its depth value is recorded as D k (p); S in the formula k,u,v ∈P k Represents the image point set P k The set of points p with pixel coordinates (u, v) in .

6. The method according to claim 1, characterized in that The three-dimensional pose estimation network is used to perform three-dimensional pose estimation on depth images of objects at multiple perspectives to obtain the three-dimensional object poses at multiple perspectives. The rendered depth map is used as input, and a neural network is used to estimate the three-dimensional object poses at K perspectives respectively.

7. The method according to claim 1, characterized in that The fusion of the 3D object posture from multiple perspectives is to fuse the coordinates of the key points of K perspectives, and the formula used is as follows: in, and They represent the key point coordinates of the three-dimensional object estimated from the kth perspective and the key point coordinates of the fused three-dimensional object, N represents the number of key points of the object, and R k and t k are the rotation matrix and translation vector of the kth virtual camera selected, c k is the confidence weight. After performing the SoftMax operation on the confidence values ​​of K virtual cameras, the confidence weight of the kth perspective is c k .

8. A posture estimation device based on multi-view rendering, characterized in that: include: An object detection module is used to detect object areas in a single depth image; A view selection module, used to select a virtual camera suitable for object pose estimation from multiple virtual cameras; The multi-view rendering module is used to render the depth image of the object captured by the virtual camera, that is, to project the 3D point cloud of the object onto the selected virtual camera and render the depth image of the object from K perspectives; A 3D pose estimation module is used to estimate the pose of a 3D object from the depth images of the object at K perspectives through a 3D pose estimation network to obtain the pose of the 3D object at K perspectives; A view fusion module is used to fuse the 3D object postures of K viewpoints to obtain the final 3D object posture; The method of selecting a virtual camera suitable for object pose estimation from multiple virtual cameras is to select the virtual camera using a view confidence network; the view confidence network extracts image features from a depth image of the object, maps the image features into confidences of multiple preset virtual cameras, and selects K virtual cameras with the highest view confidence for subsequent operations; the view confidence network includes an image feature extraction network and a fully connected layer.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

Images