Monocular RGB-D camera-based track and field athlete body posture estimation method

Through monocular RGB-D camera and multi-layer network processing, combined with object detection and occlusion robust algorithm, the problems of complex environment and multi-person detection difficulty in monocular human posture estimation are solved, and the accurate estimation of track and field athletes' pose estimation and Information extraction.

CN119068542APending Publication Date: 2024-12-03CHINA INST OF SPORT SCI
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410946541.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The monocular human posture is estimated to face complex intrinsic correlations and high degree of freedom of body movements, diverse dress, and the difficulty of detection caused by complex environments and camera angles, especially when multiple people detect each other, it is difficult to solve the problem of mutual occlusion.

Method used

Using a monocular RGB-D camera, human body parts are extracted through rectangular boxes, HRNet and posture correction network are used to combine multi-layer expansion convolution to process heat map information and motion context, local and global hidden state models are designed, combining target detection and occlusion robustness Algorithm, conducts pose prediction and information exchange, and uses kinematic tree model and encoder-decoder network for pose estimation.

Benefits of technology

It realizes accurate estimation of human postures of track and field athletes in complex environments, simplifies the calculation process, can effectively deal with the mutual occlusion problem in multi-person detection, and extract the most posture information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068542A_ABST
    Figure CN119068542A_ABST
Patent Text Reader

Abstract

The invention discloses a monocular RGB-D camera-based track and field athlete body posture estimation method, and the key points of the technical scheme are that the method comprises the following steps: S1, photographing to obtain an image: carrying out the real-time photographing of an athlete through a monocular RGB-D camera, and storing the photographed image; s2, posture time sequence merging: firstly, framing a human body needing posture estimation from a current frame, a previous frame and a next frame by adopting a rectangular frame, and estimating heat maps of all parts of the human body such as the head, the left and right feet and the like through HRNet; during attitude time sequence combination, grouping is performed according to parts, and then grouping convolution is performed; and S3, attitude residual fusion: when attitude residual fusion is carried out, motion residual is obtained by subtracting the current frame from the next frame and removing the previous frame before the current frame, so that motion context information is obtained. According to the method, the track and field athlete body posture estimation based on the monocular RGB-D camera can be realized, and the calculation is simpler.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of sports, and in particular to a method for estimating the human body posture of track and field athletes based on a monocular RGB-D camera. Background Art

[0002] Vision-based monocular human pose estimation is one of the most basic and challenging tasks in computer vision, which aims to obtain human poses using input images and video sequences. With the continuous development of deep learning in recent years, HPE based on deep learning technology has also made significant progress.

[0003] Reference may be made to the existing Chinese patent with publication number CN112183316B, which discloses: a method for measuring the human posture of an athlete, comprising the following steps: obtaining multi-directional views of the athlete to be measured in time order during the movement as a sequence of image groups to be measured; obtaining a two-dimensional image coordinate sequence of each joint point of the athlete to be measured based on the sequence of image groups to be measured; obtaining a preliminary three-dimensional space coordinate sequence of each joint point of the athlete to be measured based on the two-dimensional image coordinate sequence of each joint point of the athlete to be measured; constructing a three-dimensional skeleton template of the human body of the athlete to be measured based on the preliminary three-dimensional space coordinate sequence of each joint point of the athlete to be measured; driving the three-dimensional skeleton template of the human body based on the kinematic chain relationship between each joint point of the skeleton, and matching and optimizing it with the preliminary three-dimensional space coordinate sequence of each joint point of the athlete to be measured, to obtain an actual three-dimensional space coordinate sequence of each joint point of the athlete to be measured.

[0004] Another example is the Chinese patent with the existing publication number CN112364815A, which discloses: a high jump posture detection method for a high jumper based on a three-dimensional model, the method first uses a key point detection network to detect the human body key points of the high jumper in the high jump image and generates human body key point label data; Gaussian kernel convolution is performed on the human body key points, and the obtained two-dimensional key point sequence of the high jumper is input into the TCN network, and the three-dimensional posture sequence of the high jumper is obtained after processing by the TCN network; secondly, a parameter-adjustable three-dimensional model of the high jumper is established, and the variable posture of the three-dimensional model is achieved by adjusting the posture parameters of the three-dimensional model, and the high jump posture of the high jumper is matched with the high jump posture of the high jumper in the high jump image; finally, the high jump posture of the high jumper presented by the three-dimensional model is compared with the standard three-dimensional action posture using the trained twin network to obtain the detection result.

[0005] The above-mentioned posture test has many advantages, but it has some unique characteristics and challenges for the current monocular human posture estimation. The challenges faced by human posture estimation are mainly in three aspects: 1. The flexible human body means that there are more complex internal connections between key points and higher degrees of freedom of body movements, which poses a higher challenge to model training; 2. The human body's clothing means a variety of body shapes; 3. Complex environments may make it difficult to extract foreground information (people hidden in the background), or when performing multi-person detection, mutual occlusion between different individuals will cause a surge in detection difficulty; similarly, the camera's shooting position and angle will increase the difficulty of monocular estimation. Summary of the invention

[0006] In view of the problems mentioned in the background technology, the purpose of the present invention is to provide a method for estimating the human posture of track and field athletes based on a monocular RGB-D camera to solve the problems mentioned in the background technology.

[0007] The above technical objectives of the present invention are achieved through the following technical solutions:

[0008] A method for estimating the human posture of track and field athletes based on a monocular RGB-D camera comprises the following steps:

[0009] S1. Take photos to obtain images: The monocular RGB-D camera takes photos of the athletes in real time and stores the photos;

[0010] S2, posture time series merging: First, a rectangular frame is used to frame the human body that needs to be estimated from the current frame, the previous frame, and the next frame, and the heat map of each part of the human body, such as the head, left and right feet, is estimated through HRNet; when merging the posture time series, the parts are grouped and then grouped and convolved;

[0011] S3, posture residual fusion: When posture residual fusion is performed, the motion residual is obtained by subtracting the current frame from the next frame and the current frame from the previous frame, so as to obtain motion context information;

[0012] S4. Posture prediction: In the posture correction network, multi-layer dilated convolution is designed to process the heat map information and motion context information together to give a posture prediction.

[0013] Preferably, in S2, the relationship between joints is modeled to explicitly embed the joint degrees of freedom and bone lengths, and a three-dimensional coordinate system is established for each joint, so that each joint can be represented by the coordinate system of the previous joint through operations such as rotation and translation.

[0014] Preferably, in S2, a local hidden state is set for each frame in the video to save the motion information of each frame, and then a global hidden state is set to model the motion information of the entire time series, and each local hidden state exchanges information with its two adjacent frames before and after, transmits and exchanges information, and adaptively adjusts the number of steps required for recursion.

[0015] Preferably, in S3, a single human body is detected using a target detection algorithm, and the posture is estimated based on the single human body, and the posture estimation is performed in a regression-based and heat map-based manner. The regression-based method includes a combination of a deep network and a linear regression method, and the heat map-based method includes a key point-heat map conversion, and a high-resolution image + spatial information + multi-scale fusion method.

[0016] Preferably, in S2, all key points of the human body are detected using key points, and the key points are combined into different people based on positional relationships or other auxiliary information.

[0017] Preferably, in S4, the positions of different people are first detected and a bounding box is generated for each person, and then the image of the bounding box is cropped out and the single person posture estimation is performed separately, each body block existing in each input image is first predicted to perform a target detection, and then they are assembled separately using the human model algorithm to form complete posture estimation data, wherein the body block can be a pixel block divided by bone key points, limbs or other forms.

[0018] Preferably, the S1 performs information processing on the occluded image, uses an occlusion robust posture mapping algorithm to process the occlusion information, and uses a local classification regression network to perform processing in three stages, including: using R-CNN to detect the position of the person, assigning each posture candidate a nearest anchor posture scored by a classifier, and refining the final posture using a regressor respectively.

[0019] Preferably, in S4, relative joint positions are predicted from a 2D heatmap according to a kinematic tree model, and a kinematic object model is embedded into a network according to the kinematic tree model using global 2D joint positions and local body part images for providing general articulated object pose estimation with orientation and rotation constraints, and smoothing filtering and skeleton fitting are performed respectively using temporal information and a human motion model, and an encoder-decoder network is used to learn a latent deformable body model without 2D or 3D annotations under self-supervision, and then a pre-trained encoder is used to predict a 3D pose.

[0020] In summary, the present invention mainly has the following beneficial effects:

[0021] The present invention can realize the human posture estimation of track and field athletes based on a monocular RGB-D camera. The calculation is simpler and complex environments can be processed. The most posture information can be extracted as much as possible. When performing multi-person detection, the problem of a surge in detection difficulty caused by mutual occlusion between different individuals can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] refer to Figure 1 , a method for estimating the human posture of track and field athletes based on a monocular RGB-D camera, comprising the following steps:

[0025] S1. Take photos to obtain images: The monocular RGB-D camera takes photos of the athletes in real time and stores the photos;

[0026] S2, posture time series merging: First, a rectangular frame is used to frame the human body that needs to be estimated from the current frame, the previous frame, and the next frame, and the heat map of each part of the human body, such as the head, left and right feet, is estimated through HRNet; when merging the posture time series, the parts are grouped and then grouped and convolved;

[0027] S3, posture residual fusion: When posture residual fusion is performed, the motion residual is obtained by subtracting the current frame from the next frame and the current frame from the previous frame, so as to obtain motion context information;

[0028] S4. Posture prediction: In the posture correction network, multi-layer dilated convolution is designed to process the heat map information and motion context information together to give a posture prediction.

[0029] Among them, in S2, the relationship between joints is modeled to explicitly embed the joint degrees of freedom and bone lengths, and a three-dimensional coordinate system is established for each joint, so that each joint can be represented by the coordinate system of the previous joint through operations such as rotation and translation.

[0030] Among them, in S2, a local hidden state is set for each frame in the video to save the motion information of each frame, and then a global hidden state is set to model the motion information of the entire time series, and each local hidden state exchanges information with its two adjacent frames before and after, transmits and exchanges information, and adaptively adjusts the number of steps required for recursion.

[0031] Among them, in S3, a single human body is detected using a target detection algorithm, and the posture is estimated based on the single human body. The posture estimation is performed in the form of regression-based and heat map-based. The regression-based method includes a combination of deep network and linear regression, and the heat map-based method includes key point-heat map conversion, high resolution of the image + spatial information + multi-scale fusion.

[0032] Among them, in S2, all key points of the human body are detected using key points, and the key points are combined into different people based on positional relationships or other auxiliary information.

[0033] Among them, in S4, the positions of different people are first detected and a bounding box is generated for each person, and then the image of the bounding box is cropped out and the single person posture estimation is performed separately. Each body block existing in each input image is first predicted to perform a target detection, and then they are assembled separately using the human model algorithm to form complete posture estimation data, wherein the body block can be a skeletal key point, a limb or a pixel block divided in other forms.

[0034] Among them, S1 processes the information of the occluded picture, uses the occlusion robust posture mapping algorithm to process the occlusion information, and uses a local classification regression network to process it through three stages, including: using R-CNN to detect the position of the person, assigning each posture candidate a nearest anchor posture scored by a classifier, and refining the final posture using a regressor respectively.

[0035] Among them, in S4, relative joint positions are predicted from 2D heatmap according to the kinematic tree model, and the kinematic object model is embedded into the network according to the kinematic tree model using the global 2D joint positions and local body part images for general articulated object pose estimation providing direction and rotation constraints, and smoothing filtering and skeleton fitting are performed respectively using temporal information and human motion model, and an encoder-decoder network is used to learn a latent deformable body model without 2D or 3D annotations under self-supervision, and then the pre-trained encoder is used to predict the 3D pose.

[0036] Among them, the present invention can realize the human posture estimation of track and field athletes based on a monocular RGB-D camera. The calculation is simpler, and complex environments can be processed. It can extract as much posture information as possible. When performing multi-person detection, it can solve the problem that mutual occlusion between different individuals will cause a surge in detection difficulty.

[0037] Among them, flexible body construction represents complex inter-articular joints and high-degree-of-freedom limbs, which may lead to self-occlusion or rare / complex poses. Body appearance includes different clothes and parts that are similar to each other. Complex environments may cause foreground occlusions, nearby people occlusions or similar parts, various perspectives, and truncation in camera views. Human body modeling is a key component of human pose estimation. The human body is a flexible and complex non-rigid object with many specific features such as kinematic structure, body shape, surface texture, and the location of body parts or body joints. A mature human body model does not have to contain all human attributes, but should meet the requirements of specific tasks of constructing and describing human poses.

[0038] Based on different levels of representation and application scenarios, there are three commonly used human models in human pose estimation: skeleton-based models, contour-based models, and volume-based models. Skeleton-based models represent a set of joint positions (usually between 10 and 30) and the corresponding limb directions following the human skeleton structure. Skeleton-based models can also be described as a graph in which vertices indicate the constraints of joints in the skeleton structure and edges encode constraints or prior connections of joints. This human topology is very simple and flexible and has been widely used in 2D and 3D human pose estimation and human pose datasets. Although it has the obvious advantages of simple and flexible representation, it still has many shortcomings, such as the lack of texture information, which leads to the lack of human width and contour information. Contour-based models: Contour-based models have been widely used in early human pose estimation methods. They contain rough width and contour information of limbs and torso. Human body parts are roughly represented by rectangles or the boundaries of the human outline. Widely used contour-based models include cardboard models and active shape models. Volume-based models: 3D human shapes and poses are usually represented by volume-based geometric shapes or mesh models. Older geometric shapes used to model body parts include cylinders, cones, etc. Modern volume-based models are represented in the form of meshes, usually captured through 3D scanning. Widely used volume-based models include human shape completion and animation, skinned multi-person linear models, and unified deformation models.

[0039] Among them, the rgb image in the RGB-D image provides the x and y coordinates in the pixel coordinate system, while the depth map directly provides the camera coordinate system. According to the information of the RGB-D image and the camera's internal parameters, the coordinates of any pixel in the camera coordinate system can be calculated. According to the information of the RGB-D image and the camera's internal and external parameters, the coordinates of any pixel in the world coordinate system can be calculated. Within the camera's field of view, the coordinates of the obstacle points in the camera coordinate system are the point cloud sensor data, that is, the point cloud data in the camera coordinate system. The point cloud sensor data can be calculated based on the coordinates provided by the RGB-D image and the camera's internal parameters. The coordinates of all obstacle points in the world coordinate system are the point cloud map data, that is, the point cloud data in the world coordinate system. The point cloud map data can be calculated based on the coordinates provided by the RGB-D image and the camera's internal and external parameters. The point cloud data in the camera coordinate system is the x, yx, yx, y coordinates in the pixel coordinate system provided by the rgb image (that is, u, vu, vu, v in the formula) and the camera's internal parameters, which can calculate the X, YX, YX, YY coordinate values ​​in the camera coordinate system. At the same time, the depth map directly provides the Z coordinate value in the camera coordinate system. Then the coordinates in the camera coordinate system are obtained. The coordinates of the obstacle points in the camera coordinate system are the point cloud sensor data, that is, the point cloud data in the camera coordinate system. Generally speaking, if the color information is added to the corresponding point position in the formula, a color point cloud can be formed.

[0040] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for estimating the human posture of track and field athletes based on a monocular RGB-D camera, characterized by: The following steps are involved: S1. Take photos to obtain images: The monocular RGB-D camera takes photos of the athletes in real time and stores the photos; S2, posture time series merging: First, a rectangular frame is used to frame the human body that needs to be estimated from the current frame, the previous frame, and the next frame, and the heat map of each part of the human body, such as the head, left and right feet, is estimated through HRNet; when merging the posture time series, the parts are grouped and then grouped and convolved; S3, posture residual fusion: When posture residual fusion is performed, the motion residual is obtained by subtracting the current frame from the next frame and the current frame from the previous frame, so as to obtain motion context information; S4. Posture prediction: In the posture correction network, multi-layer dilated convolution is designed to process the heat map information and motion context information together to give a posture prediction.

2. The method for estimating the human posture of track and field athletes based on a monocular RGB-D camera according to claim 1, characterized in that: In S2, the relationship between joints is modeled to explicitly embed the joint degrees of freedom and bone lengths, and a three-dimensional coordinate system is established for each joint, so that each joint can be represented by the coordinate system of the previous joint through operations such as rotation and translation.

3. The method for estimating the human posture of track and field athletes based on a monocular RGB-D camera according to claim 1, characterized in that: In S2, a local hidden state is set for each frame in the video to save the motion information of each frame, and then a global hidden state is set to model the motion information of the entire time series. Each local hidden state exchanges information with its two adjacent frames before and after, transmits and exchanges information, and adaptively adjusts the number of steps required for recursion.

4. The method for estimating the human posture of track and field athletes based on a monocular RGB-D camera according to claim 3, characterized in that: In S3, a single human body is detected using a target detection algorithm, and the posture is estimated based on the single human body. The posture estimation is performed in the form of regression-based and heat map-based. The regression-based method includes a combination of deep network and linear regression, and the heat map-based method includes key point-heat map conversion, high resolution of the image + spatial information + multi-scale fusion.

5. The method for estimating the human posture of track and field athletes based on a monocular RGB-D camera according to claim 1, characterized in that: In S2, all key points of the human body are detected using key points, and the key points are combined into different people based on positional relationships or other auxiliary information.

6. The method for estimating the human posture of track and field athletes based on a monocular RGB-D camera according to claim 1, characterized in that: In S4, the positions of different people are first detected and a bounding box is generated for each person. Then, the image of the bounding box is cropped and the single person posture estimation is performed separately. Each body block existing in each input image is first predicted to perform a target detection. Then, the human model algorithm is used to assemble them separately to form complete posture estimation data, wherein the body block can be a pixel block divided by bone key points, limbs or other forms.

7. The method for estimating the human posture of track and field athletes based on a monocular RGB-D camera according to claim 1, characterized in that: In the S1, information processing is performed on the occluded image, an occlusion robust posture mapping algorithm is used to process the occlusion information, and a local classification regression network is used to perform processing in three stages, including: using R-CNN to detect the position of the person, assigning each posture candidate a nearest anchor posture scored by a classifier, and refining the final posture using a regressor respectively.

8. The method for estimating the human posture of track and field athletes based on a monocular RGB-D camera according to claim 1, characterized in that: In S4, relative joint positions are predicted from 2D heatmaps according to a kinematic tree model. According to the kinematic tree model, a kinematic object model is embedded into a network using global 2D joint positions and local body part images for general articulated object pose estimation providing orientation and rotation constraints. Temporal information and a human motion model are used for smoothing filtering and skeleton fitting, respectively. An encoder-decoder network is used to learn a latent deformable body model without 2D or 3D annotations under self-supervision, and then a pre-trained encoder is used to predict 3D poses.

Citation Information

Patent Citations

  • A method for measuring the human posture of athletes

    CN112183316B

  • High jump posture detection method for high jump athlete based on three-dimensional model

    CN112364815A