Vision-based hand joint occlusion discrimination method and device

By obtaining the adjacent relationship between the joint nodes and joint areas of the hand, combined with the convex hull algorithm and cylinder construction, the problem of not being able to obtain the joint occlusion information in the prior art is solved, and the accuracy in any hand interaction state is achieved Improved accuracy of occlusion judgment and hand posture estimation.

CN118942117APending Publication Date: 2024-11-12GUANGXI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411006032.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art cannot effectively obtain information that all hand joints are blocked in any hand interaction state, especially when facing one-hand self-blocking, two-hand mutual blocking or object blocking.

Method used

By obtaining the adjacent relationship between the joint nodes and joint areas of the hand, using the camera position as the coordinate origin, the hand joint model is photographed, and combining the convex hull algorithm to fit the outer contour and build a small radius cylinder to determine whether the hand joint is blocked. And comprehensively consider the situation of self-blocking with one hand, mutual blocking with two hands and object blocking.

Benefits of technology

Accurately obtaining information about the obstructed joint of each hand in any hand interaction state, improving the accuracy of hand posture estimation and filling the gap in the occlusion labels in the existing dataset.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942117A_ABST
    Figure CN118942117A_ABST
Patent Text Reader

Abstract

The invention discloses a vision-based hand joint shielding judgment method and device, and relates to the field of hand posture estimation.The method comprises the steps that a camera position serves as a coordinate origin to shoot a hand joint model, the hand joint point serves as a target point and is connected with an adjacent point and the origin respectively, and when the angle formed by the two connecting lines is smaller than 15 degrees, the hand joint shielding judgment result is obtained; if yes, single-hand self-shielding or double-hand mutual shielding is achieved; projecting the target point and a non-adjacent hand joint area on an image plane of a camera, and fitting an outer contour of the projection of the non-adjacent hand joint area; when the projection of the target point is located in the outer contour, and the vertical distance between the target point and the image plane is greater than the vertical distance between the non-adjacent hand joint area and the image plane, single-hand self-shielding or double-hand mutual shielding is determined; a cylinder is drawn with the connecting line of the target point and the original point as the axis, and when the cylinder contains the object point cloud data, the object is shielded. According to the invention, the information that all hand joints are shielded in any hand interaction state can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of hand posture estimation, and in particular to a method and device for distinguishing hand joint occlusion based on vision. Background Art

[0002] Vision-based hand interaction can intuitively express human intentions in augmented reality operations, sign language conversations, grasping objects, etc., and has become an important means of human activity scenes.

[0003] In the prior art, the datasets for hand posture estimation and hand model reconstruction do not provide information about hand occlusion, and the datasets obtained based on the hand do not provide occlusion labels. Existing algorithms use time-series images or multi-view images to alleviate the information loss caused by joint occlusion. During the calculation process of the algorithm, the probability of fingers being occluded is roughly predicted. Different modeling strategies are adopted in the algorithm for different probabilities, such as using a single Gaussian distribution modeling for visible joints and a multi-component Gaussian mixture model modeling for occluded joints, thereby guiding subsequent hand posture estimation results.

[0004] However, even if the above technical solution adopts a modeling strategy to alleviate the information loss caused by joint occlusion, it cannot take into account the situation of self-occlusion of one hand, mutual occlusion of both hands or occlusion by objects, and obtain the information of each hand joint being occluded. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for determining hand joint occlusion based on vision, which can solve the problem in the prior art that it is impossible to obtain information on occlusion of all hand joints in any hand interaction state.

[0006] The embodiment of the present invention provides a method for distinguishing hand joint occlusion based on vision, comprising the following steps: obtaining a plurality of hand joint points and a plurality of hand joint areas on two hand joint models, and distinguishing the adjacent relationship between each hand joint point and the remaining hand joint points and all hand joint areas, wherein the adjacent relationship is whether there is a hand joint point or a hand joint area between each hand joint point and the remaining hand joint points and all hand joint areas; taking the camera position as the coordinate origin to shoot the hand joint model, taking a hand joint point as the target point, and connecting the target point with the adjacent points of the target point and the origin respectively, and when the angle formed by the two connecting lines is less than fifteen degrees, it is judged that the target point is blocked by the adjacent points of the target point and cannot be photographed; projecting the target point and the non-adjacent hand joint areas of the target point The shadow is projected on the image plane of the camera, and the outer contour of the projection of the non-adjacent hand joint area is fitted by the convex hull algorithm; when the projection of the target point is within the outer contour, and the first perpendicular distance between the target point and the image plane is greater than the second perpendicular distance between the non-adjacent hand joint area and the image plane, it is judged that the target point is blocked by the non-adjacent hand joint area of ​​the target point and cannot be photographed; when the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point belong to the same hand as the target point, it is judged as single-hand self-occlusion; when the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point do not belong to the same hand as the target point, it is judged as mutual occlusion of both hands; the target point is connected to the origin, and a small-radius cylinder is constructed with the connecting line as the axis. When any point cloud data of the object is inside the cylinder, it is judged as object occlusion.

[0007] Furthermore, the acquisition of multiple hand joint points and multiple hand joint areas on the hand joint model specifically includes: using the hand posture estimation parameterized model MANO to extract joint features of the hand joint model; performing plane cutting on the hand model to obtain 21 hand joint points and 21 hand joint areas.

[0008] Furthermore, the target point and the non-adjacent hand joint area of ​​the target point are projected onto the image plane of the camera, and the specific steps include: obtaining point cloud data of the non-adjacent hand joint area of ​​the target point; and simultaneously projecting the point cloud data of the non-adjacent hand joint area and the target joint point onto the image plane of the camera.

[0009] Furthermore, the determination of mutual occlusion of both hands also includes the division of non-adjacent hand joint regions, specifically including: defining all hand joint regions that are not in the same hand as the target point as non-adjacent hand joint regions.

[0010] Furthermore, the determination as object occlusion specifically includes the following steps: when one-hand self-occlusion and object occlusion exist simultaneously, determining that the object is held by one hand; when only object occlusion exists, determining that the object is held by both hands or by one hand.

[0011] Furthermore, the construction of a small radius cylinder specifically includes: connecting the target point and the origin, and constructing a cylinder with a radius of 5 mm with the connecting line as the axis.

[0012] The embodiment of the present invention provides a hand joint occlusion discrimination device based on vision, comprising: an information acquisition module, used to obtain multiple hand joint points and multiple hand joint areas on two hand joint models, and to distinguish the adjacent relationship between each hand joint point and the remaining hand joint points and all hand joint areas, wherein the adjacent relationship is whether there is a hand joint point or a hand joint area between each hand joint point and the remaining hand joint points and all hand joint areas; an occlusion judgment module, used to shoot the hand joint model with a camera position as a coordinate origin, regard a hand joint point as a target point, and connect the target point with the adjacent points of the target point and the origin respectively, and when the angle formed by the two connecting lines is less than fifteen degrees, it is judged that the target point is blocked by the adjacent points of the target point and cannot be photographed; the target point and the non-adjacent hand joint areas of the target point are connected. The projected image is projected onto the image plane of the camera, and the outer contour of the projection of the non-adjacent hand joint area is fitted using the convex hull algorithm; when the projection of the target point is within the outer contour, and the first perpendicular distance between the target point and the image plane is greater than the second perpendicular distance between the non-adjacent hand joint area and the image plane, it is judged that the target point is blocked by the non-adjacent hand joint area of ​​the target point and cannot be photographed; a hand interaction state judgment module is used to judge that it is a single-hand self-occlusion when the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point belong to the same hand as the target point; when the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point do not belong to the same hand as the target point, it is judged that the two hands are mutually occluded; the target point is connected to the origin, and a small-radius cylinder is constructed with the connecting line as the axis. When any point cloud data of the object is inside the cylinder, it is judged as object occlusion.

[0013] The embodiment of the present invention provides a method and device for judging hand joint occlusion based on vision. Compared with the prior art, the beneficial effects thereof are as follows:

[0014] The camera position is used as the coordinate origin to shoot the hand joint model, a hand joint point is regarded as the target point, and the target point is connected with the adjacent point of the target point and the origin respectively. When the angle formed by the two connecting lines is less than 15 degrees, it is judged that the target point is blocked by the adjacent point of the target point and cannot be photographed; the target point and the non-adjacent hand joint area of ​​the target point are projected on the image plane of the camera, and the outer contour of the projection of the non-adjacent hand joint area is fitted using the convex hull algorithm; when the projection of the target point is within the outer contour, and the first perpendicular distance between the target point and the image plane is greater than the angle between the non-adjacent hand joint area and the image plane, the target point is blocked by the adjacent point of the target point and the origin. The second perpendicular distance to the image plane determines that the target point is blocked by the non-adjacent hand joint area of ​​the target point and cannot be photographed; when the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point belong to the same hand as the target point, it is determined to be self-occluded by one hand; when the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point do not belong to the same hand as the target point, it is determined to be mutual occlusion of both hands; connect the target point and the origin, and construct a small radius cylinder with the connecting line as the axis. When any point cloud data of the object is inside the cylinder, it is determined to be object occlusion.

[0015] Among them, when judging whether each hand joint point is occluded, the occlusion situation of each hand joint point in all hand interaction states such as self-occlusion of one hand, mutual occlusion of both hands and occlusion of objects is comprehensively considered. Finally, the occlusion information of each hand joint can be obtained in any hand interaction state. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A MANO model segmentation diagram of a hand joint occlusion discrimination method based on vision provided by an embodiment of the present invention;

[0017] Figure 2 An adjacent point and non-adjacent point judgment diagram of a hand joint occlusion judgment method based on vision provided by an embodiment of the present invention;

[0018] Figure 3 An occlusion relationship diagram of a target point and its adjacent points in a hand joint occlusion discrimination method based on vision provided by an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of determining the occlusion relationship between a target point and its non-adjacent hand joint regions in the case of one-hand self-occlusion in a vision-based hand joint occlusion determination method provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of determining the occlusion relationship between a target point and its non-adjacent hand joint regions in a method for determining hand joint occlusion based on vision provided by an embodiment of the present invention when both hands are occluded with each other;

[0021] Figure 6A schematic diagram of determining the occlusion relationship between a target point and a grasped object in a hand joint occlusion determination method based on vision provided by an embodiment of the present invention under the condition of object occlusion;

[0022] Figure 7 A physical analysis diagram of a vision-based hand joint occlusion discrimination method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0024] See also Figure 1 to Figure 7 The embodiment of the present invention provides a method for determining hand joint occlusion based on vision, comprising the following steps:

[0025] Step 1: Obtain multiple hand joint points and multiple hand joint areas on the two hand joint models, as well as the adjacent relationship between each hand joint point and the remaining hand joint points, and each hand joint point and all hand joint areas. Specifically: Use the hand posture estimation parameterized model MANO to extract the joint features of the hand joint model, perform plane cutting on the hand model, and obtain 21 hand joint points and 21 hand joint areas.

[0026] Step 2: Use the camera position as the coordinate origin to shoot the hand joint model. Consider a hand joint point as the target point, and connect lines with its adjacent points and the origin respectively. When the angle formed by the two connecting lines is less than 15 degrees, it is judged that the target point is blocked by its adjacent points and cannot be photographed.

[0027] Step 3: Project the target point and the non-adjacent hand joint area of ​​the target point onto the image plane of the camera, and use the convex hull algorithm to fit the outer contour of the projection of the non-adjacent hand joint area. The image plane is a plane perpendicular to the camera shooting direction. When the projection of the target point is within the outer contour, and the first perpendicular distance between the target point and the image plane is greater than the second perpendicular distance between the non-adjacent hand joint area and the image plane, it is determined that the target point is blocked by the non-adjacent hand joint area of ​​the target point and cannot be photographed. Among them, the point cloud data of the non-adjacent hand joint area and the target joint point need to be projected onto the image plane of the camera at the same time.

[0028] Step 4: When the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point belong to the same hand as the target point, it is judged as single-hand self-occlusion; when the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point do not belong to the same hand as the target point, it is judged as mutual occlusion of both hands. Connect the target point and the origin, and construct a cylinder with a radius of 5 mm with the connecting line as the axis. When any point cloud data of the object is inside the cylinder, it is judged as object occlusion.

[0029] Specifically, object occlusion judgment includes the following steps: first, perform single-hand self-occlusion judgment on each joint of the hand model; regard a hand joint point as the target point, connect it with the camera coordinate origin, and construct a cylinder with a radius of 5mm with the line segment as the main axis; if any point in the 3D point cloud data of the object appears inside the cylinder, it means that the hand joint point is occluded by the object; if the hand joint point as the target point meets the condition of single-hand self-occlusion or is occluded by the object, it is judged that the hand joint point is occluded.

[0030] Existing hand posture estimation-related datasets cannot provide occlusion labels for each hand joint, which makes it impossible for the algorithm to improve the estimation accuracy of hand posture based on occlusion information. In addition, none of the existing datasets provide occlusion labels, so most of the existing algorithms use time-series images or multi-view images to alleviate the information loss caused by joint occlusion. However, time-series images will have cumulative errors, and multi-view images require that more than two cameras must be installed in the hand interaction scene, both of which have certain limitations.

[0031] The problem to be solved by the present invention is to propose an occlusion joint identification method based on a hand model, and at the same time, based on this method, a hand gesture posture estimation data set containing occlusion information is produced, and the data set is learned through artificial intelligence, so that the hand posture can be estimated more accurately in hand posture estimation.

[0032] The present invention proposes a hand joint occlusion discrimination method applicable to any occlusion situation, which includes: single-hand self-occlusion discrimination, double-hand mutual occlusion discrimination, and object occlusion discrimination. Based on this method, a complete single-hand self-occlusion hand dataset can be constructed, which includes: original RGB image and depth information, hand joint 2D label, hand joint 3D label, hand joint self-occlusion label, hand mask image, and MANO label.

[0033] The present invention also has the following advantages:

[0034] 1. This method fills the gap of unoccluded labels in the field of hand pose estimation, so that subsequent algorithms can discuss hand joints separately based on occluded labels, further improving the accuracy of hand pose estimation.

[0035] 2. Based on this method, the present invention also provides occlusion labels for other hand posture estimation datasets, supplementing the lack of occlusion labels in hand posture estimation datasets.

[0036] 3. The present invention provides a complete hand posture estimation data set including self-occlusion information, which provides a guarantee for artificial intelligence training and hand posture estimation.

[0037] The embodiment of the present invention provides a hand joint occlusion determination device based on vision, comprising:

[0038] The information acquisition module is used to acquire multiple hand joint points and multiple hand joint areas on the two hand joint models, and to identify the adjacent relationship between each hand joint point and the remaining hand joint points and all hand joint areas, wherein the adjacent relationship is whether there is a hand joint point or a hand joint area between each hand joint point and the remaining hand joint points and all hand joint areas. The occlusion judgment module is used to shoot the hand joint model with the camera position as the coordinate origin, regard a hand joint point as a target point, and connect the target point with the adjacent point and the origin of the target point respectively. When the angle formed by the two connecting lines is less than 15 degrees, it is judged that the target point is blocked by the adjacent point of the target point and cannot be photographed; the target point and the non-adjacent hand joint area of ​​the target point are projected on the image plane of the camera, and the outer contour of the projection of the non-adjacent hand joint area is fitted by the convex hull algorithm; when the projection of the target point is within the outer contour, and the first perpendicular distance between the target point and the image plane is greater than the second perpendicular distance between the non-adjacent hand joint area and the image plane, it is judged that the target point is blocked by the non-adjacent hand joint area of ​​the target point and cannot be photographed. The hand interaction state judgment module is used to judge that when the adjacent points of the target point or the non-adjacent hand joint areas of the target point belong to the same hand as the target point, it is single-hand self-occlusion; when the adjacent points of the target point or the non-adjacent hand joint areas of the target point do not belong to the same hand as the target point, it is judged that the two hands are mutually occluded; connect the target point with the origin, and construct a small radius cylinder with the connecting line as the axis. When any point cloud data of the object is inside the cylinder, it is judged as object occlusion.

[0039] A specific embodiment is as follows:

[0040] (1) First, the hand model feature points are plane-cut based on the MANO model to obtain 21 different hand joint regions, such as Figure 1 shown.

[0041] (2) All 21 joints of the hand are taken as the target points for judgment and judged one by one. Each joint point has two kinds of relationships with the other 20 joint points: adjacent and non-adjacent. For example, Figure 2The reason for distinguishing between adjacent points and non-adjacent points is that, for each joint point, the occlusion of adjacent points and the occlusion of non-adjacent points are judged in different ways.

[0042] (3) Based on the projection principle of the camera, we define the angle formed by the line connecting the target point and the adjacent point and the line connecting the target point and the origin of the camera coordinate system as a. When the angle a is less than 15 degrees, the target point is blocked by the adjacent point.

[0043] (4) As described in (1), the hand model is divided into 21 areas based on the plane where the joints are located, and each joint has a non-adjacent area. To determine whether the joint is blocked by the non-adjacent area, it is necessary to use projection to determine whether the non-adjacent area point cloud and the target joint are simultaneously projected onto the image plane of the camera, and the convex hull algorithm is used to fit the projection outer contour of the non-adjacent area, and then determine whether the projection point of the target joint is within the contour. If the target joint is within the contour, it means that there is a possibility of being blocked. At this time, compare the depth value of the joint with the depth value of the non-adjacent area. If the depth value of the joint is greater than the depth value of the non-adjacent area, it means that the joint is blocked by the non-adjacent area.

[0044] (5) Based on (3) and (4), it can be determined whether the target joint is blocked and cannot be photographed.

[0045] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A vision-based hand joint occlusion discrimination method, characterized in that: The following steps are involved: Acquire multiple hand joint points and multiple hand joint areas on two hand joint models, and identify the adjacent relationship between each hand joint point and the remaining hand joint points and all hand joint areas, wherein the adjacent relationship is whether there is a hand joint point or a hand joint area between each hand joint point and the remaining hand joint points and all hand joint areas; The camera position is used as the coordinate origin to shoot the hand joint model, a hand joint point is regarded as the target point, and the target point is connected with the adjacent point of the target point and the origin respectively. When the angle formed by the two connecting lines is less than 15 degrees, it is determined that the target point is blocked by the adjacent point of the target point and cannot be photographed; The target point and the non-adjacent hand joint area of ​​the target point are projected onto the image plane of the camera, and the outer contour of the projection of the non-adjacent hand joint area is fitted using the convex hull algorithm; when the projection of the target point is within the outer contour, and the first perpendicular distance between the target point and the image plane is greater than the second perpendicular distance between the non-adjacent hand joint area and the image plane, it is determined that the target point is blocked by the non-adjacent hand joint area of ​​the target point and cannot be photographed; When the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point belong to the same hand as the target point, it is judged as single-hand self-occlusion; When the adjacent points of the target point or the non-adjacent hand joint area of ​​the target point do not belong to the same hand as the target point, it is judged that the two hands are occluding each other; Connect the target point and the origin, and construct a small-radius cylinder with the line as the axis. When any point cloud data of the object is inside the cylinder, it is judged as object occlusion.

2. A method for determining hand joint occlusion based on vision as claimed in claim 1, characterized in that: The acquisition of multiple hand joint points and multiple hand joint areas on the hand joint model specifically comprises the following steps: Use the hand posture estimation parameterized model MANO to extract the joint features of the hand joint model; The hand model is plane-cut to obtain 21 hand joint points and 21 hand joint areas.

3. A method for determining hand joint occlusion based on vision as claimed in claim 1, characterized in that: The step of projecting the target point and the non-adjacent hand joint regions of the target point onto the image plane of the camera specifically includes: Obtaining point cloud data of non-adjacent hand joint areas of the target point; The point cloud data of the non-adjacent hand joint area and the target joint point are simultaneously projected onto the image plane of the camera, wherein the image plane is a plane perpendicular to the shooting direction of the camera.

4. A method for determining hand joint occlusion based on vision as claimed in claim 1, characterized in that: The determination of mutual occlusion of both hands also includes the division of non-adjacent hand joint areas, specifically including: All hand joint regions that are not in the same hand as the target point are defined as non-adjacent hand joint regions.

5. The method for determining hand joint occlusion based on vision as claimed in claim 1, characterized in that: The specific steps of judging whether the object is blocked include: When one-hand self-occlusion and object occlusion exist at the same time, it is judged as one-hand holding the object; When only an object is occluded, it is judged as the object being held by both hands or one hand.

6. A method for determining hand joint occlusion based on vision as claimed in claim 1, characterized in that: The construction of a small radius cylinder specifically includes: Connect the target point and the origin, and construct a cylinder with a radius of 5 mm with the connecting line as the axis.

7. A hand joint occlusion judgment device based on vision, characterized in that: include: An information acquisition module, used to acquire multiple hand joint points and multiple hand joint areas on two hand joint models, and identify the adjacent relationship between each hand joint point and the remaining hand joint points and all hand joint areas, wherein the adjacent relationship is whether there is a hand joint point or a hand joint area between each hand joint point and the remaining hand joint points and all hand joint areas; The occlusion judgment module is used to shoot the hand joint model with the camera position as the coordinate origin, regard a hand joint point as the target point, and connect the target point with the adjacent point of the target point and the origin respectively. When the angle formed by the two connecting lines is less than fifteen degrees, it is judged that the target point is blocked by the adjacent point of the target point and cannot be photographed; the target point and the non-adjacent hand joint area of ​​the target point are both projected on the image plane of the camera, and the outer contour of the projection of the non-adjacent hand joint area is fitted by the convex hull algorithm; when the projection of the target point is within the outer contour, and the first perpendicular distance between the target point and the image plane is greater than the second perpendicular distance between the non-adjacent hand joint area and the image plane, it is judged that the target point is blocked by the non-adjacent hand joint area of ​​the target point and cannot be photographed; The hand interaction state judgment module is used to judge that when the adjacent points of the target point or the non-adjacent hand joint areas of the target point belong to the same hand as the target point, it is single-hand self-occlusion; when the adjacent points of the target point or the non-adjacent hand joint areas of the target point do not belong to the same hand as the target point, it is judged that the two hands are mutually occluded; Connect the target point and the origin, and construct a small-radius cylinder with the line as the axis. When any point cloud data of the object is inside the cylinder, it is judged as object occlusion.