Method for Producing 2D Human Joint Data Set Based on Kinect Camera

Abstract

The present invention relates to the field of dataset production, and particularly to a method for producing a two-dimensional human joint dataset based on a Kinect camera. First, a joint position model required is defined, physical markers are pasted at human joints, a human demonstration video is recorded by the Kinect camera, and then an image set is obtained through frame splitting. The image set is converted into the HSV space and morphological processing is performed to obtain the positions of the marked joints. Then, the vast majority of ordered joint coordinates are obtained through a relative position matching algorithm, and then the positions of occluded joints are inferred through mean deviation fitting. Finally, the production of the two-dimensional human joint dataset is completed through manual recheck and fine-tuning. The present invention uses a low-cost Kinect camera to obtain human demonstration images and an accurate two-dimensional joint position dataset. The joint positions can be set by oneself, with strong flexibility, effectively improving the phenomenon of relatively serious drift errors in the built-in joint skeleton function of the Kinect.

Description

Technical Field

[0001] The present invention relates to the field of dataset production, and particularly to a method for producing a two-dimensional human joint dataset based on a Kinect camera. Background Art

[0002] The human joint dataset is the core prerequisite for obtaining human joint postures using deep learning methods. Currently, the existing human posture datasets are mainly collected by large foreign teams through recording the markings of joints attached to demonstrators using dozens of cameras, which makes the production of the dataset have great difficulty and a high threshold. In addition, only using foreign datasets for deep learning training restricts the flexibility of the algorithm and cannot perform targeted learning for different scenarios and specific actions required. Summary of the Invention

[0003] In order to solve the above technical problems existing in the prior art, the present invention proposes a method for producing a two-dimensional human joint dataset based on a Kinect camera, and its specific technical solution is as follows:

[0004] A method for producing a two-dimensional human joint dataset based on a Kinect camera includes the following steps:

[0005] Step 1: Define the required human joint position model and obtain the human joint action picture set;

[0006] Step 2: Process the images in the human joint action picture set to obtain the two-dimensional human joint coordinate information of the marked points in the images;

[0007] Step 3: Fit and infer the position coordinates of the occluded joints;

[0008] Step 4: View and calibrate the annotation information through a visualization platform to complete the production of the two-dimensional human joint dataset.

[0009] Further, the specific content of Step 1 includes:

[0010] Step 1.1: Paste physical color markers at the joint positions set for the required dataset; <,

[0011] Step 1.2: Use a Kinect camera to record a video of human actions in the mkv format;

[0012] Step 1.3: Split the recorded video frame by frame and store it as an RGB color image to obtain the human joint action picture set.

[0013] Further, the specific content of Step 2 includes:

[0014] Step 2.1: Convert the images in the human joint action picture set into HSV color space representation composed of three parts: hue, saturation, and value.

[0015] Step 2.2: Extract the defined color range area as a mask, optimize the mask area through image processing morphological operations, judge the contour of the relevant area by calculating the image gradient, calculate the centroid of all contours to express the position of the joint points, and obtain an unordered set of joint point coordinates. U ;

[0016] Step 2.3: Use the joint point tracking function built into the Kinect camera, and utilize the internal and external parameter matrices of the camera and the camera distortion matrix to convert the coordinates in the depth camera coordinate system to the color camera coordinate system to obtain a reference coordinate set. K ;

[0017] Step 2.4: Through the relative position matching algorithm, traverse the unordered set of joint point coordinates in Step 2.2. U , find the joint index with the smallest Euclidean metric from the current joint point. u , if the joint index value has not been matched or the Euclidean metric is less than the previously matched key point, then store the coordinates and its index of the joint point. u .

[0018] Further, the specific steps of Step 3 include:

[0019] Step 3.1: Judge whether the index of the joint point coordinates obtained in Step 2 meets the setting of covering the human joint data set.

[0020] Step 3.2: If Step 3.1 is true and meets the setting, enter Step 3.4, otherwise enter Step 3.3.

[0021] Step 3.3: Use the occlusion joint position inference method based on the mean deviation fitting method. If it is found that the coordinates of one joint point in the current frame are missing, check whether the joint exists in the next three frames of this frame to obtain the coordinates. If it exists, regard this as a short-term detection missing, and calculate the corresponding joint coordinates using local mean shift. If the joint coordinates in adjacent frames are all in the missing state, it is considered that the joint is occluded. Therefore, use the offset of the global position mean of the corresponding joint coordinates in this joint and the reference coordinate set. K " Calculate the current coordinate position of this joint.

[0022] Step 3.4: Obtain the complete two-dimensional joint point position data annotation information and store it as a JSON file. The file mainly includes image information: image index, image name, width, height, and annotation information: the number of annotated joints, joint positions, corresponding image index. The joint positions include the X and Y direction coordinates of the joint positions and joint visibility.

[0023] Further, step 4 specifically includes:

[0024] Step 4.1: Synchronously display the two-dimensional joint position data annotation information obtained in step 3 and the RGB color image in step 1 through an interactive visualization method;

[0025] Step 4.2: Judge whether there are problems of misdetection and deviation through manual observation;

[0026] Step 4.3: If there is a misdetection problem in step 4.2, correct it through mouse interaction. Left click means the current joint is visible, and right click means the joint is occluded;

[0027] Step 4.4: If there is a deviation problem in step 4.2, adjust the X and Y directions of the annotation coordinates by moving the keyboard up, down, left, and right.

[0028] Further, step 4.1 includes the following sub-steps:

[0029] Step 4.1.1: Read the annotation information stored as a JSON file and display the images of the corresponding image set;

[0030] Step 4.1.2: At the joint positions on the image, draw pixel points using the OpenCV library and set colors through visibility;

[0031] Step 4.1.3: When switching pictures, automatically store the identification status of the current image into the annotation data file.

[0032] Beneficial effects: The present invention only uses a low-cost Kinect camera to obtain human demonstration images and an accurate two-dimensional joint position data set. The joint positions can be set by oneself, with strong flexibility, effectively improving the phenomenon of serious drift error in the built-in joint skeleton function of Kinect, and the data set can be quickly browsed and calibrated through a visualization platform. Brief Description of the Drawings

[0033] Figure 1 It is a process schematic diagram of the method for making a two-dimensional human joint data set based on a Kinect camera according to the present invention;

[0034] Figure 2 It is a schematic diagram of the definition of general human joints in the prior art;

[0035] Figure 3 It is a schematic diagram of the definition of an arm joint in an embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of a joint point image in the HSV color space in an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of a visualization platform display interface according to an embodiment of the present invention;

[0038] Figure 6 This is a revised diagram of the right elbow joint markings according to an embodiment of the present invention;

[0039] Figure 7 This is a left-hand occlusion mark correction diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and technical effect of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0041] like Figure 1 As shown, the present invention provides a method for producing a two-dimensional human joint dataset based on a Kinect camera. First, the required joint position model is defined, physical markers are attached to the human joints, and a human demonstration video is recorded using the Kinect camera. Then, an image set is obtained by frame splitting. The image set is converted into HSV space and morphological processing is performed to obtain the marked joint positions. Then, the majority of ordered joint coordinates are obtained by a relative position matching algorithm. Then, the occluded joint positions are inferred by mean deviation fitting. Finally, the dataset is completed through manual review and fine-tuning. The method specifically includes the following steps:

[0042] Step 1: Define the required human joint position model and obtain a set of human joint action images, including:

[0043] Step 1.1: Stick physical colored markers at the joint locations specified in the desired dataset, e.g. Figure 2 and Figure 3 The following are the definition methods of body limb joints and arm joints respectively. Figure 3 The arm joint definition method is described in detail;

[0044] Step 1.2: Use the Kinect camera to record the human body movement in MKV format.

[0045] Step 1.3: Split the recorded video into frames and store them as RGB color images to obtain a set of human joint action images.

[0046] Step 2: Process the images in the human joint action image set to obtain the two-dimensional joint coordinate information of the marked points in the image, specifically including:

[0047] Step 2.1: Convert the image in the image set obtained in step 1 into three parts: hue, saturation, and brightness. Figure 4The HSV color space representation shown is closer to human color perception experience and is more conducive to the segmentation and tracking of a certain color;

[0048] Step 2.2: Extract the defined color range area as a mask, optimize the mask area through a series of image processing morphological operations such as erosion and dilation, judge the contour of the relevant area by calculating the image gradient, and calculate the centroid of all contours to represent the position of the joint point, obtaining an unordered set of joint point coordinates U ;

[0049] Step 2.3: Use the built-in joint tracking function of the Kinect camera, and utilize the internal and external parameter matrices and camera distortion matrix of the camera to convert the coordinates in the depth camera coordinate system to the color camera coordinate system, obtaining a reference coordinate set K , and there are often phenomena such as unstable joint positions and large offsets in the tracking of some frames;

[0050] Step 2.4: Through the relative position matching algorithm, traverse the unordered set of joint point coordinates in Step 2.2 U , and find the joint index with the smallest Euclidean metric from the current joint point u . If the joint index value has not been matched or the Euclidean metric is less than the previously matched key point, store the coordinates and index of the joint point u .

[0051] Step 3: Fit and infer the position coordinates of the occluded joint, specifically including:

[0052] Step 3.1: Judge whether the index of the joint point coordinates obtained in Step 2 meets the setting of covering the data set, that is, judge whether all 9 arm joint positions as shown in Figure 3 are obtained in this embodiment;

[0053] Step 3.2: If Step 3.1 is true and meets the setting, enter Step 3.4, otherwise enter Step 3.3;

[0054] Step 3.3: Use the occlusion joint position inference method based on the mean deviation fitting method. If it is found that the coordinates of the right elbow joint point in the current frame are missing, check whether the coordinates of this joint exist in the three frames backward from this frame. If they exist, regard this as a short-term detection missing, and calculate the corresponding joint coordinates using the local mean shift. If the joints in adjacent frames are all in the missing state, it is considered that the joint is occluded within a certain period of time. Therefore, use the offset of the global position mean of the right elbow joint coordinates in the reference coordinate set K to calculate the current coordinate position of the right elbow joint;

[0055] Step 3.4: Obtain the complete two-dimensional joint position data annotation information and store it as a JSON file, which mainly includes image information: image index, image name, width, height, and annotation information: the number of annotated joints, joint positions, and corresponding image index. The joint positions include the X and Y coordinates of the joint positions and joint visibility. Visible is set to 0, and invisible is set to 1.

[0056] Step 4: View and calibrate the annotation information through the visualization platform to complete the production of the two-dimensional human joint dataset, specifically including:

[0057] Step 4.1: Synchronously display the two-dimensional joint position data annotation information obtained in Step 3 and the RGB color image in Step 1 through the Figure 5 interactive visualization method, specifically including:

[0058] Step 4.1.1: Read the annotation information stored as a JSON file and display the images of the corresponding image set;

[0059] Step 4.1.2: At the joint positions on the image, draw pixel points using the OpenCV library and set the color according to visibility. Visible is set to red, and invisible is set to green;

[0060] Step 4.1.3: When switching pictures, automatically store the identification status of the current image to the annotation data file;

[0061] Step 4.2: Quickly browse through manual observation to determine whether there are problems of misdetection and deviation;

[0062] Step 4.3: If there is a misdetection problem in Step 4.2, correct it through mouse interaction. Left-click represents that the current joint is visible, and right-click represents that the joint is occluded. Figure 6 In the case where the left hand joint is occluded and there is a deviation in the fitting result, right-click on the correct position with the mouse to correct the annotation information;

[0063] Step 4.4: If there are occasional deviation problems in Step 4.2, adjust the X and Y directions of the annotation coordinates by moving the keyboard up, down, left, or right. Figure 7 In the case where the right elbow joint marker has a position deviation due to environmental reflection, use the keyboard to move it to the correct position in the lower left.

[0064] As described above, it is only the preferred implementation case of the present invention and does not impose any form of limitation on the present invention. Although the implementation process of the present invention has been described in detail above, for those familiar with the field, they can still modify the technical solutions recorded in the foregoing examples or make equivalent replacements for some of the technical features. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for producing a two-dimensional human joint dataset based on a Kinect camera, characterized in that: The following steps are involved: Step 1: Define the required human joint position model and obtain a set of human joint action pictures; Step 2: Process the images in the human joint action picture set to obtain the two-dimensional human joint coordinate information of the marked points in the image; Step 3: Fit and infer the position coordinates of the occluded joints; Step 4: View and calibrate the annotation information through the visualization platform to complete the production of the 2D human joint dataset; The step 2 specifically includes: Step 2.1: Convert the images in the human joint action image set into the HSV color space representation consisting of hue, saturation, and brightness; Step 2.2: Extract the limited color range area and set it as the mask. Optimize the mask area through image processing morphological operations. Determine the contour of the relevant area by calculating the image gradient. Calculate the centroid of all contours to express the location of the joint points and obtain the unordered joint point coordinate set. U ; Step 2.3: Use the built-in joint tracking function of the Kinect camera, the camera intrinsic and extrinsic parameter matrices and the camera distortion matrix to convert the coordinates in the depth camera coordinate system to the color camera coordinate system to obtain the reference coordinate set K ; Step 2.4: Traverse the unordered joint point coordinate set in step 2.2 through the relative position matching algorithm U , find the current joint point u The joint index with the smallest Euclidean metric. If the joint index value has not been matched or the Euclidean metric is less than the key point of the previous match, the joint point is stored. u The coordinates and index of The step 3 specifically includes: Step 3.1: Determine whether the index of the joint point coordinates obtained in step 2 meets the setting of covering the human joint dataset; Step 3.2: If step 3.1 is true and the setting is met, go to step 3.4, otherwise go to step 3.3; Step 3.3: Use the occluded joint position inference method based on the mean deviation fitting method. If the coordinates of a joint point in the current frame are missing, check whether the coordinates of the joint exist in the three frames after the current frame. If so, it is considered as a short-term detection loss. The corresponding joint coordinates are calculated using the local mean offset. If the joint coordinates in the adjacent frames are all missing, the joint is considered to be occluded, and the joint is used to compare with the reference coordinate set. K The offset of the global position mean of the corresponding joint coordinate in is used to calculate the current coordinate position of the joint; Step 3.4: Obtain complete two-dimensional joint point position data annotation information and store it as a JSON file. The file mainly includes image information: image index, image name, width, height, and annotation information: number of annotated joints, joint positions, and corresponding image index. The joint positions include the X and Y coordinates of the joint position and joint visibility.

2. The method for producing a two-dimensional human joint dataset based on a Kinect camera as claimed in claim 1, wherein: The step 1 specifically includes: Step 1.1: Stick the physical colored markers at the joint locations specified in the desired dataset. Step 1.2: Use the Kinect camera to record the human body movement in MKV format. Step 1.3: Split the recorded video into frames and store them as RGB color images to obtain a set of human joint action images.

3. The method for producing a two-dimensional human joint dataset based on a Kinect camera as claimed in claim 1, wherein: The step 4 specifically includes: Step 4.1: Synchronously display the two-dimensional joint point position data annotation information obtained in step 3 and the RGB color image obtained in step 1 through interactive visualization; Step 4.2: Through manual observation, determine whether there are any false detection and deviation problems; Step 4.3: If there is a misdetection problem in step 4.2, correct it through mouse interaction. Left click means the current joint is visible, and right click means the joint is blocked. Step 4.4: If there is a deviation problem in step 4.2, adjust the X and Y directions of the marked coordinates by moving the keyboard up, down, left, and right.

4. The method for producing a two-dimensional human joint dataset based on a Kinect camera according to claim 3, wherein: The step 4.1 includes the following sub-steps: Step 4.1.1: Read the annotation information stored as a JSON file and display the images of the corresponding image set; Step 4.1.2: Use the OpenCV library to draw pixels at the joint locations on the image and set the color by visibility; Step 4.1.3: When switching between images, the identification status of the current image is automatically stored in the annotation data file.

Images