A marker-enhanced intelligent capture method for precise hand movements

Abstract

The present invention discloses a marker-enhanced precise hand motion intelligent capture method. The method comprises: firstly arranging marker points at each joint of the hand, using multiple RGB cameras to synchronously collect the hand motion arranged with the marker points, then determining the two-dimensional position of each marker point in the original hand image, and then using a deep learning method to determine the name of each joint and the two-dimensional position of the joints after removing the marker points in the original hand image; based on the two-dimensional position of each marker point in each original image and the name and two-dimensional position of each joint, respectively reconstructing the three-dimensional position of each marker point and the three-dimensional position of each joint, and matching the marker point with the joint name, and obtaining the compensated three-dimensional position of the joint after calculation; in the subsequent hand motion capture, the real-time calculation of the compensated three-dimensional position of the joint is realized through the previous and next frames and the joint inheritance relationship. The present invention realizes the precise capture of hand motions, and has high real-time performance and strong robustness.

Description

Technical Field

[0001] The present invention relates to a hand motion capture method in the fields of virtual reality, deep learning and image recognition, and in particular to a marker-enhanced precise hand motion intelligent capture method. Background Art

[0002] In large and complex product virtual simulation environments, hand movement is the most important human-computer interaction activity, so high-precision hand motion capture is crucial. However, due to the high dimensionality of hand shape changes and unpredictable occlusion of fingers, high-precision finger-level motion capture is still a challenging problem.

[0003] Conventional human hand motion capture methods require the placement of markers or sensors, which restricts the user's hand movements and affects the user's human-computer interaction experience; and the motion capture equipment is expensive, which is not conducive to promotion. Hand motion capture algorithms based on deep learning usually use RGB or RGBD cameras as motion capture equipment, which are low-cost, easy to deploy, and have broad application prospects. However, its accuracy cannot be guaranteed because there is no unified quantitative standard to define the position of the hand joints on the image when creating the training set. Only a rough area can be qualitatively determined, and it is believed that defining the joints within this area meets the requirements. Therefore, the deep learning algorithm trained using this training set is also difficult to quantitatively determine the joint position, and its hand motion capture accuracy cannot meet the requirements of virtual simulation of large and complex products. Summary of the invention

[0004] In order to improve the accuracy of hand motion capture, the present invention provides a marker-enhanced precise hand motion intelligent capture method, which improves the hand motion capture accuracy by arranging marker points at important joints of the hand and capturing their movements.

[0005] The technical solution adopted by the present invention is:

[0006] S1: Marking points are arranged at each joint of the hand, and multiple RGB cameras are used to synchronously capture the hand movement with the marking points arranged, and obtain multiple original hand images of the initial frame;

[0007] S2: Select an original hand image of the current frame, then perform YUV color domain conversion on the current original hand image to obtain a corresponding V color domain hand image, binarize the current V color domain hand image according to the marker threshold, obtain a corresponding hand marker image, then use the contour tracking algorithm to extract the marker contour of the hand marker image to obtain a corresponding marker contour map; according to the marker contour map, use the minimum enclosing circle method to extract the two-dimensional position of each marker on the current original hand image;

[0008] S3: determining each joint of the hand in the V color domain hand image in S2, and obtaining each joint name and two-dimensional position of the joint in the current hand original image;

[0009] S4: using the Hungarian algorithm, the two-dimensional position of each marker point on the current hand marker point image is matched with the two-dimensional position of each joint in the restored bare hand image, so as to obtain the joint name and two-dimensional position of each marker point in the current hand original image;

[0010] S5: repeat S2-S4, perform marker point and joint processing on the remaining original hand images of the current frame, and obtain the names and two-dimensional positions of each joint in all original hand images in the current frame, as well as the names and two-dimensional positions of the joints to which each marker point belongs;

[0011] S6: Reconstruct the three-dimensional position of each marker point in the current frame using a multi-eye stereo vision method according to the joint name to which each marker point belongs and the two-dimensional position of the marker point in all original hand images of the current frame;

[0012] S7: Reconstruct the three-dimensional position of each joint in the current frame using a multi-eye stereo vision method according to the names and two-dimensional positions of each joint in all original hand images of the current frame;

[0013] S8: Based on the distance d between each marker point and the corresponding joint, and the three-dimensional position p of each joint in the current frame joint and the three-dimensional position p of each marker point marker Compensate the joint 3D position and calculate the compensated 3D position p' of each joint in the current frame. marker ;

[0014] S9: Multiple RGB cameras continue to synchronously capture hand movements in each frame to obtain multiple original hand images in each frame. If the data of the previous frame of each frame is valid, S10 is executed to obtain the compensated three-dimensional positions of each joint in the current frame; if the data of the previous frame of each frame is not valid, S2-S8 are repeated to compensate the three-dimensional positions of the joints according to the multiple original hand images of the current frame to obtain the compensated three-dimensional positions of each joint in the current frame; when there are occluded marker points in the current frame that cannot be reconstructed in three dimensions, the compensated three-dimensional positions of the joints corresponding to the occluded marker points that cannot be reconstructed in three dimensions are predicted using the missing marker point position prediction method;

[0015] S10: Repeat S2, extract the positions of the marker points of the multiple original hand images of the current frame, and obtain the two-dimensional positions of each marker point of the multiple original hand images of the current frame; then use the Hungarian algorithm to match the two-dimensional positions of each marker point of the multiple original hand images of the current frame with the joint names and two-dimensional positions of the joints to which the marker points of the original hand images corresponding to the previous frame belong, and obtain the joint names and two-dimensional positions of the marker points of the multiple original hand images of the current frame, and then use the multi-eye stereo vision method to reconstruct the three-dimensional positions of each marker point in the current frame; according to the three-dimensional positions of each joint in the previous frame, use the tracking algorithm to predict the three-dimensional positions of each joint in the current frame, and finally repeat S8 to calculate and obtain the three-dimensional positions of each compensated joint in the current frame.

[0016] In S1, the joints of the hand include the first interphalangeal joints, the second interphalangeal joints, the third interphalangeal joints and the wrist joints arranged in sequence from the base to the fingertip of each finger, wherein the marking points of adjacent joints between the index finger, middle finger, ring finger and little finger of the hand have different colors.

[0017] The S3 is specifically:

[0018] S3.1: Binarize the V color domain hand image in S2 according to the hand threshold value to obtain the hand image excluding the marked points, extract the hand contour of the hand image excluding the marked points using the contour tracking algorithm, and obtain the initial hand contour map;

[0019] S3.2: According to the marker point contour map in S and the initial hand contour map, obtain the hand contour map without the marker points, and then use the curve fitting method to perform contour fitting on the hand contour map without the marker points to obtain the hand contour restoration map; use the contour internal filling algorithm to perform regional processing on the hand contour restoration map to obtain the hand region map, multiply the pixel value of the hand region map with the corresponding hand marker point image to obtain the marker point area to be filled;

[0020] S3.3: Based on the current original hand image, fill the hand texture color of the marker point area to be filled using a diffusion-based image restoration algorithm to obtain a restored bare hand image;

[0021] S3.4: Use a deep learning algorithm to extract the names and two-dimensional positions of each joint in the restored bare hand image, that is, the names and two-dimensional positions of each joint in the current original hand image.

[0022] In S2, the contour map of the marker points extracted by the contour tracking algorithm is also denoised, specifically:

[0023] First, the Douglas-Peuker algorithm is used to screen each marker point area in the marker point contour map extracted by the contour tracking algorithm to obtain the screened marker point area; then, the pixel area covered by the screened marker point area is calculated, and the marker point area whose pixel area is larger than the area threshold is selected and recorded as the valid marker point area, thereby obtaining the denoised marker point contour map, and the denoised marker point contour map is used as the final marker point contour map.

[0024] In S2, the minimum enclosing circle method is used to calculate the center and radius of the enclosing circle corresponding to each marked point area in the marked point contour map, and the minimum enclosing circle area A corresponding to the marked point area is calculated according to the center and radius of each enclosing circle. circle , according to the minimum enclosing circle area A of the marked point area circle , determine whether each marking point area is an independent marking point or an overlapping marking point,

[0025] If it is an independent marking point, the two-dimensional position of the center of the enclosing circle corresponding to the current independent marking point is taken as the two-dimensional position of the current marking point;

[0026] When there are overlapping marker points, they are separated according to their colors to obtain two marker points. Then, the minimum enclosing circle method is used to determine the center of the marker point with the larger pixel area. At the same time, the minimum enclosing rectangle method is used to determine the enclosing rectangle of the marker point with the smaller pixel area. The center of the marker point with the smaller pixel area is reconstructed according to the position of the tangent pixels between the enclosing rectangle and the marker point with the smaller pixel area, and the two-dimensional positions of the centers of the two marker points are used as the two-dimensional positions of the two marker points.

[0027] In S3.3, the contour points of the marked points in the marked point area to be filled in the hand are taken as inner contour points, and the inner contour points are taken as points to be filled. For each point q to be filled, it is determined whether each pixel point p in the neighborhood of the current point q to be filled belongs to the hand area in the hand contour map after removing the marked points. If it belongs to the hand area, the pixel point p is valid, otherwise, the pixel point p is invalid. Based on the current original hand image, the pixel value of the current point q to be filled is obtained by pixel weighted averaging of the valid pixels in the neighborhood of the current point q to be filled and filled, and then the inside of the marked point area to be filled is gradually searched and filled until the filling is completed.

[0028] The compensated three-dimensional position p' of the joint in S8 marker The calculation formula is as follows:

[0029]

[0030] Here, || represents the modulus of the orientation quantity.

[0031] In S9, the data of the previous frame includes the joint names and two-dimensional positions of the joints to which the marking points of each original hand image in the previous frame belong, the three-dimensional positions of each joint, and the three-dimensional positions of the joints obtained after compensation.

[0032] In S9, the missing marker position prediction method is used to predict the compensated three-dimensional position of the joint corresponding to the marker point that is blocked and cannot be three-dimensionally reconstructed, specifically:

[0033] An objective function of the compensated three-dimensional joint position corresponding to the fully occluded marker points is established, and the damped least squares algorithm is used to solve the objective function to obtain the compensated three-dimensional joint position corresponding to the marker points that are occluded and cannot be reconstructed in three dimensions.

[0034] The calculation formula of the objective function of the compensated joint three-dimensional position corresponding to the blocked and incapable three-dimensional reconstruction marker point is as follows:

[0035] E=min(ω temp E temp +ω dl E dl +ω bone E bone )

[0036]

[0037] E bone =(‖p j -p i ‖2-l ij ) 2

[0038]

[0039] Where E represents the total three-dimensional position error, min() represents the minimum value operation, ω temp represents the first constraint weight, E temp represents the joint 3D position error obtained after compensation in the previous frame, represents the compensated 3D position of the joint corresponding to the marker point i that is blocked and cannot be reconstructed in 3D in the previous frame, p i represents the compensated 3D position of the joint corresponding to the occluded and incapable 3D reconstruction marker point i, ω bone represents the second constraint weight, E bone represents the bone length error, p j represents the compensated joint 3D position corresponding to the previous level marker point j of the marker point i that is blocked and cannot be reconstructed in 3D, l ij represents the bone length between marker point i and marker point j, ‖‖2 represents the two-norm operation, ω dlrepresents the third constraint weight, E dl represents the 3D position error of the joint in the previous frame, It indicates the three-dimensional position of the joint corresponding to the marker point i in the current frame predicted by the tracking algorithm or the three-dimensional position of the joint corresponding to the marker point i in the current frame which is blocked and cannot be three-dimensionally reconstructed obtained by reconstruction in S7.

[0040] The beneficial effects of the present invention are:

[0041] 1. The present invention uses multiple RGB cameras to improve the hand motion capture accuracy of the deep learning algorithm by arranging marker points at each joint of the hand.

[0042] 2. The diffusion-based image restoration method proposed in the present invention fills in the hand texture color in the marker point area and restores the real image of the hand, thereby avoiding the creation of a hand motion capture training set with marker points arranged at the hand joints.

[0043] 3. The present invention proposes to introduce front and back frame constraints and bone constraints, and use a damped least squares algorithm to solve the position of missing marker points to improve the stability of hand tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a marker-enhanced precise hand motion intelligent capture method.

[0045] Figure 2 It is a flow chart of marker point extraction and hand texture color filling.

[0046] Figure 3 It is a diagram of the hand marker arrangement.

[0047] Figure 4 This is a processing flow chart when the marking points overlap.

[0048] Figure 5 It is a position relationship diagram between the marker area and the hand area. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0050] like Figure 1 As shown, the present invention comprises the following steps:

[0051] S1: Place markers at each joint of the hand, such as Figure 3 As shown, multiple RGB cameras are used to synchronously capture the hand movements with markers arranged thereon, and multiple original hand images of the initial frame are obtained;

[0052] In S1, each joint of the hand includes the first interphalangeal joint, the second interphalangeal joint, the third interphalangeal joint and the wrist joint, which are arranged in sequence from the base of the finger to the fingertip of each finger. The inheritance relationship of the finger joints is set along the base of the finger to the fingertip. The upper joint of the third interphalangeal joint is the second interphalangeal joint, the upper joint of the second interphalangeal joint is the first interphalangeal joint, and the upper joint of the first interphalangeal joint is the wrist joint. The marking points of the adjacent joints between the index finger, middle finger, ring finger and little finger of the hand have different colors. In a specific implementation, the marking point colors are red and green. Adjacent joints refer to the upper, lower, left and right joints of each joint. For example, the adjacent joints of the first interphalangeal joint of the index finger are the second interphalangeal joint of the index finger and the first interphalangeal joint of the middle finger, and the first interphalangeal joint of the index finger and the second interphalangeal joint of the middle finger are not adjacent joints.

[0053] S2: Figure 2 As shown, an original hand image of the current frame is selected. Due to the influence of background color, ambient light, etc., the RGB color domain cannot effectively separate the marker point area. Then, the current original hand image is converted to YUV color domain to obtain the corresponding V color domain hand image. The current V color domain hand image is binarized according to the marker point threshold to obtain the corresponding hand marker point image.

[0054] The marker area is divided into three categories according to its relationship with the hand area: the marker area completely overlaps with the hand area, the marker area partially overlaps with the hand area, and the marker area does not overlap with the hand area at all, as shown in Figure 2. Figure 5 As shown in (a), (b) and (c). Then, the Suzuki contour tracking algorithm is used to extract the marker point contour of the hand marker point image to obtain the corresponding marker point contour map; according to the marker point contour map, the minimum enclosing circle method is used to extract the two-dimensional position of each marker point in the current hand original image on the current hand original image; in the specific implementation, the position of the center of the marker point in the marker point contour map is used as the two-dimensional position of the marker point on the current hand original image.

[0055] In S2, the V color domain hand image is binarized, and noise points are introduced when obtaining the hand marker point image. In order to reduce the influence of uncertain factors such as noise points on the extraction of marker point positions, the marker point contour image obtained by the contour tracking algorithm can also be denoised, specifically:

[0056] First, the Douglas-Peuker algorithm is used to screen each marker point area in the marker point contour map extracted by the contour tracking algorithm to obtain a valid marker point area; in the specific implementation, the thinning threshold is set according to the thinning accuracy in the algorithm, and the Douglas-Peuker algorithm is used to measure each possible marker point area from a morphological perspective. If the number of contour points in the marker point area simplified by the Douglas-Peuker algorithm is greater than the preset circular threshold (empirical value), the circular threshold is 4, then it is considered to be a valid marker point area. Then, the pixel area A covered by the screened marker point area is calculated. marker (openCV contourArea), generally the area of ​​noise points is small, the pixel area is larger than the area threshold to filter out the marker area and record it as the valid marker area. In the specific implementation, the area threshold is set according to the actual marker size and the position of the camera, so as to obtain the denoised marker contour map, and use the denoised marker contour map as the final marker contour map.

[0057] In S2, the minimum enclosing circle method is used to calculate the center and radius of the enclosing circle corresponding to each marked point area in the marked point contour map, and the minimum enclosing circle area A corresponding to the marked point area is calculated according to the center and radius of each enclosing circle. circle , according to the minimum enclosing circle area A of the marked point area circle , determine whether each marking point area is an independent marking point or an overlapping marking point. Specifically, when the minimum enclosing circle area A of the marking point area circle At least larger than the pixel area A covered by the independent marker area marker When it is one-third, the marker area is an overlapping marker outline, otherwise it is an independent marker area.

[0058] like Figure 4 As shown, when it is an independent marking point, the two-dimensional position of the center of the enclosing circle corresponding to the current independent marking point is taken as the two-dimensional position of the current marking point;

[0059] When there are overlapping marker points, they are separated according to their colors to obtain two marker points. The marker point with a larger pixel area is the unobstructed marker point area, and the marker point with a smaller pixel area is the obstructed marker point area. The minimum enclosing circle method is then used to determine the center of the marker point with a larger pixel area. At the same time, the minimum enclosing rectangle method is used to determine the enclosing rectangle of the marker point with a smaller pixel area. The center of the marker point with a smaller pixel area is reconstructed according to the position of the tangent pixels between the enclosing rectangle and the marker point with a smaller pixel area, and the two-dimensional positions of the centers of the two marker points are used as the two-dimensional positions of the two marker points.

[0060] S3: determining each joint of the hand in the V color domain hand image in S2, and obtaining each joint name and two-dimensional position of the joint in the current hand original image;

[0061] S3 is specifically:

[0062] S3.1: Binarize the V color domain hand image in S2 according to the hand threshold value to obtain the hand image excluding the marked points, and use the Suzuki contour tracking algorithm to extract the hand contour (including the marked point contour) of the hand image excluding the marked points to obtain the initial hand contour map;

[0063] S3.2: According to the marker point contour map in S and the initial hand contour map, obtain the hand contour map without the marker points, and then use the cubic curve fitting method to perform contour fitting on the hand contour map without the marker points to obtain the hand contour restoration map; use the contour internal filling algorithm (OpenCV-DrawContour) to perform regional processing on the hand contour restoration map to obtain the hand area map, multiply the pixel value of the hand area map with the corresponding hand marker point image to obtain the marker point area to be filled;

[0064] S3.3: Since creating a large number of accurate training sets of hand markers is a time-consuming and laborious process, and the existing hand motion training sets do not contain a large number of samples with markers on hand joints, a diffusion-based image filling algorithm is used to restore the marker area to the original texture color of the hand. Based on the current original hand image, the diffusion-based image restoration algorithm is used to fill the hand texture color of the marker area to be filled, and the restored bare hand image is obtained;

[0065] In S3.3, the contour points of the marked points in the marked point area to be filled in the hand are taken as inner contour points, and the contour points where the marked points and the hand intersect in the marked point area to be filled are taken as outer contour points. The inner contour points are taken as the points to be filled, and for each point q to be filled, it is determined whether each pixel point p in the neighborhood of the current point q to be filled belongs to the hand area in the hand contour map without the marked points. If it belongs to the hand area, the pixel point p is valid, otherwise, the pixel point p is invalid; based on the current original hand image, the pixel value of the current point q to be filled is obtained by pixel weighted averaging of the valid pixels in the neighborhood of the current point q to be filled and filled, and then the inside of the marked point area to be filled is searched and filled step by step until the filling is completed. In the specific implementation, during the first round of filling, the points to be filled are obtained by multiplying the marked point area to be filled with the initial hand contour map, and the pixel values ​​of the points to be filled after the pixel values ​​in the hand area binary map are filled are set to 1. After each round of filling, the hand area binary map is updated, and then the contour tracking algorithm is used to extract the contour of the updated hand area binary map to obtain an updated hand contour map, and the updated hand contour map is multiplied with the marked point area to be filled to obtain the inner contour point with a pixel value of 1 and use it as the point to be filled in the next round of filling, until there is no point with a pixel value of 1 after the updated hand image except the marked point is multiplied with the marked point area to be filled.

[0066] S3.4: Use a deep learning algorithm to extract the names and two-dimensional positions of each joint in the restored bare hand image, that is, the names and two-dimensional positions of each joint in the current original hand image; in the specific implementation, the deep learning algorithm is MidiaPipe-Hand Solution.

[0067] S4: using the Hungarian algorithm, the two-dimensional position of each marker point on the current hand marker point image is matched with the two-dimensional position of each joint in the restored bare hand image, so as to obtain the joint name and two-dimensional position of each marker point in the current hand original image;

[0068] The process of using the Hungarian algorithm in S4 to establish a matching relationship for the marker points of the current frame can be expressed as: The two-dimensional position of the marker points extracted from the current frame on the image is denoted as F t The two-dimensional position of each joint predicted by deep learning, the position of the marker predicted by the tracking algorithm, or the joint position reprojected by the hand posture of the previous frame is used as prior knowledge and is uniformly recorded as F label , then the matching process can be expressed as:

[0069]

[0070] in, represents the optimal match between the detected marker position and the prior joint position in frame t, Indicates the i-th marker point detected in the current frame, represents the position of the i-th joint of the hand, x ij is a decision variable, indicating whether the i-th marker matches the j-th joint. If so, it is 1, otherwise it is 0. At the same time, the accuracy of marker point matching is guaranteed by adding the following constraints:

[0071]

[0072] Different δ can be used for different sports conditions. max Value, δ max Represents the distance threshold. If the distance between the marker position and the prior joint position exceeds this distance threshold, the two must not match.

[0073] S5: repeat S2-S4, perform marker point and joint processing on the remaining original hand images of the current frame, and obtain the names and two-dimensional positions of each joint in all original hand images in the current frame, as well as the names and two-dimensional positions of the joints to which each marker point belongs;

[0074] S6: Reconstruct the three-dimensional position of each marker point in the current frame using a multi-eye stereo vision method according to the joint name to which each marker point belongs and the two-dimensional position of the marker point in all original hand images of the current frame;

[0075] S7: Reconstruct the three-dimensional position of each joint in the current frame using a multi-eye stereo vision method according to the names and two-dimensional positions of each joint in all original hand images of the current frame;

[0076] S8: Based on the distance d between each marker point and the corresponding joint, and the three-dimensional position p of each joint in the current frame joint and the three-dimensional position p of each marker point marker Compensate the joint's 3D position. The distance d is the distance between the center of the marked point and the corresponding joint center. It is obtained through experimental calibration. The compensated joint 3D position p' in the current frame is calculated. marker ;

[0077] The compensated joint 3D position p' marker The calculation formula is as follows:

[0078]

[0079] Among them, || represents the operation of taking the vector modulus.

[0080] S9: Multiple RGB cameras continue to synchronously capture hand movements in each frame to obtain multiple original hand images in each frame. If the data of the previous frame of each frame is valid (i.e., exists), execute S10 to obtain the compensated three-dimensional positions of each joint in the current frame; if the data of the previous frame of each frame is not valid, repeat S2-S8 to compensate the three-dimensional positions of the joints according to the multiple original hand images of the current frame to obtain the compensated three-dimensional positions of each joint in the current frame; when there are occluded and three-dimensionally unreconstructed marker points in the current frame, the occluded and three-dimensionally unreconstructed marker points refer to marker points whose pixel area of ​​the occluded area is greater than two-thirds of the pixel area covered by the independent marker point area. The missing marker point position prediction method is used to predict the compensated three-dimensional joint positions corresponding to the occluded and three-dimensionally unreconstructed marker points;

[0081] In S9, the data of the previous frame includes the joint names and two-dimensional positions of the joints to which the marking points of each original hand image in the previous frame belong, the three-dimensional positions of each joint, and the three-dimensional positions of the joints obtained after compensation.

[0082] Since the deep learning hand motion capture algorithm has a certain robustness against self-occlusion between fingers, the hand posture predicted and reconstructed by deep learning can be used to make up for the missing joints. However, the joint point positions obtained based on deep learning are not accurate enough, so parameterized skeletons are introduced to determine the positions of the missing joints through an optimization algorithm. It is generally believed that the marker points arranged on the wrist and the first knuckle of each finger will not be missing, while the marker points arranged on the second knuckle and the third knuckle are prone to missing. And it is believed that the degree of freedom of the second knuckle and the third knuckle of the finger is 1, so the movement of the three joints of each finger is coplanar. In S9, the missing marker point position prediction method is used to predict the compensated three-dimensional position of the joint corresponding to the marker points that are occluded and cannot be reconstructed in three dimensions, specifically:

[0083] The compensated three-dimensional joint position constraints, bone length constraints and joint three-dimensional position constraints obtained from the previous frame corresponding to the marker points of the previous frame are introduced, and the objective function of the compensated three-dimensional joint positions corresponding to the marker points that are occluded and cannot be reconstructed in three dimensions is established. The damped least squares algorithm is used to solve the objective function to obtain the compensated three-dimensional joint positions corresponding to the marker points that are occluded and cannot be reconstructed in three dimensions.

[0084] The calculation formula of the objective function of the compensated joint 3D position corresponding to the occluded and unreconstructed marker point is as follows:

[0085] E=min(ω temp E temp +ω dl E dl +ω bone Ebone )

[0086]

[0087] E bone =(‖p j -p i ‖2-l ij ) 2

[0088]

[0089] Where E represents the total three-dimensional position error, min() represents the minimum value operation, ω temp represents the first constraint weight, E temp represents the joint 3D position error obtained after compensation in the previous frame, represents the compensated 3D position of the joint corresponding to the marker point i that is blocked and cannot be reconstructed in 3D in the previous frame, p i represents the compensated 3D position of the joint corresponding to the occluded and unreconstructed marker point i, ω bone represents the second constraint weight, E bone represents the bone length error, p j The compensated 3D position of the interphalangeal joint corresponding to the marker point j of the interphalangeal joint of the previous level corresponding to the marker point i that is blocked and cannot be reconstructed in 3D, l ij represents the bone length between marker point i and marker point j, ‖‖2 represents the two-norm operation, ω dl represents the third constraint weight, E dl represents the 3D position error of the joint in the previous frame, It indicates the three-dimensional position of the joint corresponding to the marker point i in the current frame predicted by the tracking algorithm or the three-dimensional position of the joint corresponding to the marker point i in the current frame which is occluded and cannot be three-dimensionally reconstructed and is reconstructed in S7.

[0090] If the current frame is the first frame of the motion sequence or the hand motion capture of the previous frame fails (that is, the data of the previous frame is not valid), then ω temp Set to 0 and As the initial position of the missing markers. The 3D position of the markers in the previous frame is available. As the initial position of the missing markers.

[0091] S10: Repeat S2, extract the positions of the marker points of the multiple original hand images of the current frame, and obtain the two-dimensional positions of the marker points of the multiple original hand images of the current frame; then use the Hungarian algorithm to match the two-dimensional positions of the marker points of the multiple original hand images of the current frame with the joint names and two-dimensional positions of the joints to which the marker points of the original hand images corresponding to the previous frame belong, and obtain the joint names and two-dimensional positions of the marker points of the multiple original hand images of the current frame, and then use the multi-eye stereo vision method to reconstruct the three-dimensional positions of each marker point in the current frame; according to the three-dimensional positions of each joint in the previous frame, use the tracking algorithm to predict the three-dimensional positions of each joint in the current frame, specifically, predict its three-dimensional position in the current frame by constructing a Kalman filter position velocity constant rate model; finally, repeat S8 to calculate and obtain the three-dimensional positions of each compensated joint in the current frame.

Claims

1. A marker-enhanced precise hand motion intelligent capture method, characterized in that: The following steps are involved: S1: Marking points are arranged at each joint of the hand, and multiple RGB cameras are used to synchronously capture the hand movement with the marking points arranged, and obtain multiple original hand images of the initial frame; S2: Select an original hand image of the current frame, then perform YUV color domain conversion on the current original hand image to obtain a corresponding V color domain hand image, binarize the current V color domain hand image according to the marker threshold, obtain a corresponding hand marker image, then use the contour tracking algorithm to extract the marker contour of the hand marker image to obtain a corresponding marker contour map; according to the marker contour map, use the minimum enclosing circle method to extract the two-dimensional position of each marker on the current original hand image; S3: determining each joint of the hand for the V color domain hand image in S2, and obtaining each joint name and two-dimensional position of the joint of the current original hand image; S4: using the Hungarian algorithm, the two-dimensional position of each marker point on the current hand marker point image is matched with the two-dimensional position of each joint in the restored bare hand image, so as to obtain the joint name and two-dimensional position of each marker point in the current hand original image; S5: repeat S2-S4, perform marker point and joint processing on the remaining original hand images of the current frame, and obtain the names and two-dimensional positions of each joint in all original hand images in the current frame, as well as the names and two-dimensional positions of the joints to which each marker point belongs; S6: Reconstruct the three-dimensional position of each marker point in the current frame using a multi-eye stereo vision method according to the joint name to which each marker point belongs and the two-dimensional position of the marker point in all original hand images of the current frame; S7: Reconstruct the three-dimensional position of each joint in the current frame using a multi-eye stereo vision method according to the names and two-dimensional positions of each joint in all original hand images of the current frame; S8: Based on the distance d between each marker point and the corresponding joint, and the three-dimensional position p of each joint in the current frame joint and the three-dimensional position p of each marker point marker Compensate the joint 3D position and calculate the compensated 3D position p' of each joint in the current frame. marker ; S9: Multiple RGB cameras continue to synchronously capture hand movements in each frame to obtain multiple original hand images in each frame. If the data of the previous frame of each frame is valid, S10 is executed to obtain the compensated three-dimensional positions of each joint in the current frame; if the data of the previous frame of each frame is not valid, S2-S8 are repeated to compensate the three-dimensional positions of the joints according to the multiple original hand images of the current frame to obtain the compensated three-dimensional positions of each joint in the current frame; when there are occluded marker points in the current frame that cannot be reconstructed in three dimensions, the compensated three-dimensional positions of the joints corresponding to the occluded marker points that cannot be reconstructed in three dimensions are predicted using the missing marker point position prediction method; S10: Repeat S2, extract the positions of the marker points of the multiple original hand images of the current frame, and obtain the two-dimensional positions of each marker point of the multiple original hand images of the current frame; then use the Hungarian algorithm to match the two-dimensional positions of each marker point of the multiple original hand images of the current frame with the joint names and two-dimensional positions of the joints to which the marker points of the original hand images corresponding to the previous frame belong, and obtain the joint names and two-dimensional positions of the marker points of the multiple original hand images of the current frame, and then use the multi-eye stereo vision method to reconstruct the three-dimensional positions of each marker point in the current frame; according to the three-dimensional positions of each joint in the previous frame, use the tracking algorithm to predict the three-dimensional positions of each joint in the current frame, and finally repeat S8 to calculate and obtain the three-dimensional positions of each compensated joint in the current frame.

2. The method for intelligently capturing precise hand movements enhanced by markers according to claim 1, characterized in that: In S1, the joints of the hand include the first interphalangeal joints, the second interphalangeal joints, the third interphalangeal joints and the wrist joints arranged in sequence from the base to the fingertip of each finger, wherein the marking points of adjacent joints between the index finger, middle finger, ring finger and little finger of the hand have different colors.

3. The method for intelligently capturing precise hand movements enhanced by markers according to claim 1, characterized in that: The S3 is specifically: S3.1: Binarize the V color domain hand image in S2 according to the hand threshold value to obtain the hand image excluding the marked points, extract the hand contour of the hand image excluding the marked points using the contour tracking algorithm, and obtain the initial hand contour map; S3.2: According to the marker point contour map in S and the initial hand contour map, obtain the hand contour map without the marker points, and then use the curve fitting method to perform contour fitting on the hand contour map without the marker points to obtain the hand contour restoration map; use the contour internal filling algorithm to perform regional processing on the hand contour restoration map to obtain the hand region map, multiply the pixel value of the hand region map with the corresponding hand marker point image to obtain the marker point area to be filled; S3.3: Based on the current original hand image, fill the hand texture color of the marker point area to be filled using a diffusion-based image restoration algorithm to obtain a restored bare hand image; S3.4: Use a deep learning algorithm to extract the names and two-dimensional positions of each joint in the restored bare hand image, that is, the names and two-dimensional positions of each joint in the current original hand image.

4. The method for intelligently capturing precise hand movements enhanced by markers according to claim 1, characterized in that: In S2, the contour map of the marker points extracted by the contour tracking algorithm is also denoised, specifically: First, the Douglas-Peuker algorithm is used to screen each marker point area in the marker point contour map extracted by the contour tracking algorithm to obtain the screened marker point area; then, the pixel area covered by the screened marker point area is calculated, and the marker point area whose pixel area is larger than the area threshold is selected and recorded as the valid marker point area, thereby obtaining the denoised marker point contour map, and the denoised marker point contour map is used as the final marker point contour map.

5. The method for intelligently capturing precise hand movements enhanced by markers according to claim 1, characterized in that: In S2, the minimum enclosing circle method is used to calculate the center and radius of the enclosing circle corresponding to each marked point area in the marked point contour map, and the minimum enclosing circle area A corresponding to the marked point area is calculated according to the center and radius of each enclosing circle. circle , according to the minimum enclosing circle area A of the marked point area circle , determine whether each marking point area is an independent marking point or an overlapping marking point, If it is an independent marking point, the two-dimensional position of the center of the enclosing circle corresponding to the current independent marking point is taken as the two-dimensional position of the current marking point; When there are overlapping marker points, they are separated according to their colors to obtain two marker points. Then, the minimum enclosing circle method is used to determine the center of the marker point with the larger pixel area. At the same time, the minimum enclosing rectangle method is used to determine the enclosing rectangle of the marker point with the smaller pixel area. The center of the marker point with the smaller pixel area is reconstructed according to the position of the tangent pixels between the enclosing rectangle and the marker point with the smaller pixel area, and the two-dimensional positions of the centers of the two marker points are used as the two-dimensional positions of the two marker points.

6. The method for intelligently capturing precise hand movements enhanced by markers according to claim 3, characterized in that: In S3.3, the contour points of the marked points in the marked point area to be filled in the hand are taken as inner contour points, and the inner contour points are taken as points to be filled. For each point q to be filled, it is determined whether each pixel point p in the neighborhood of the current point q to be filled belongs to the hand area in the hand contour map after removing the marked points. If it belongs to the hand area, the pixel point p is valid, otherwise, the pixel point p is invalid. Based on the current original hand image, the pixel value of the current point q to be filled is obtained by pixel weighted averaging of the valid pixels in the neighborhood of the current point q to be filled and filled, and then the inside of the marked point area to be filled is gradually searched and filled until the filling is completed.

7. The method for intelligently capturing precise hand movements enhanced by markers according to claim 1, characterized in that: The compensated three-dimensional position p' of the joint in S8 marker The calculation formula is as follows: Here, || represents the modulus of the orientation quantity.

8. The method for intelligently capturing precise hand movements enhanced by markers according to claim 1, characterized in that: In S9, the data of the previous frame includes the joint names and two-dimensional positions of the joints to which the marking points of each original hand image in the previous frame belong, the three-dimensional positions of each joint, and the three-dimensional positions of the joints obtained after compensation.

9. The method for intelligently capturing precise hand movements enhanced by markers according to claim 1, characterized in that: In S9, the missing marker position prediction method is used to predict the compensated three-dimensional position of the joint corresponding to the marker point that is blocked and cannot be three-dimensionally reconstructed, specifically: An objective function of the compensated three-dimensional joint position corresponding to the fully occluded marker points is established, and the damped least squares algorithm is used to solve the objective function to obtain the compensated three-dimensional joint position corresponding to the marker points that are occluded and cannot be reconstructed in three dimensions.

10. The method for intelligently capturing precise hand movements enhanced by markers according to claim 9, characterized in that: The calculation formula of the objective function of the compensated joint three-dimensional position corresponding to the blocked and incapable three-dimensional reconstruction marker point is as follows: E=min(ω temp E temp +oh dl E dl +oh bone E bone ) AND bone =(‖p j -p i ‖2-l ij ) 2 Where E represents the total three-dimensional position error, min() represents the minimum value operation, ω temp represents the first constraint weight, E temp represents the joint 3D position error obtained after compensation in the previous frame, represents the compensated 3D position of the joint corresponding to the marker point i that is blocked and cannot be reconstructed in 3D in the previous frame, p i represents the compensated 3D position of the joint corresponding to the occluded and incapable 3D reconstruction marker point i, ω bone represents the second constraint weight, E bone represents the bone length error, p j represents the compensated joint 3D position corresponding to the previous level marker point j of the marker point i that is blocked and cannot be reconstructed in 3D, l ij represents the bone length between marker point i and marker point j, ‖‖2 represents the two-norm operation, ω dl represents the third constraint weight, E dl represents the 3D position error of the joint in the previous frame, It indicates the three-dimensional position of the joint corresponding to the marker point i in the current frame predicted by the tracking algorithm or the three-dimensional position of the joint corresponding to the marker point i in the current frame which is blocked and cannot be three-dimensionally reconstructed obtained by reconstruction in S7.

Images