A vision-based real-time human pose estimation and localization method

Abstract

This invention discloses a real-time human pose estimation and localization method based on pure vision. The method includes: acquiring image sequences in real-time using a binocular camera and obtaining target region images from these sequences; normalizing the target region images and extracting multi-layer feature maps from the normalized images; fusing the multi-layer feature maps to obtain multi-scale feature maps; using a dual-head mechanism to predict the multi-scale feature maps to obtain multi-channel heatmaps and 3D image coordinates for each feature point; converting the 3D image coordinates of the feature points into world coordinates; creating a physical corrector to correct the world coordinates of the feature points using the multi-channel heatmaps as a reference; and using post-processing to ensure the corrected feature points form a stable and continuous pose. This invention can reduce the computational load by more than 8 times, enabling real-time human pose estimation on mobile devices and solving the problems of previous models being too large and slow, making it suitable for real-time interactive scenarios.

Description

Technical Field

[0001] This invention relates to the field of human pose estimation and localization technology, specifically to a method for real-time human pose estimation and localization based on pure vision. Background Technology

[0002] Pure vision-based human pose estimation and localization is a technology that uses only camera images, without the assistance of external sensors, to simultaneously acquire the spatial pose of key human joints (such as limb rotation angles) and the position of the human body as a whole or in parts of the scene (such as the human body's standing coordinates and hand positions). Its core is to solve the visual perception problems of "how a person moves" and "where a person is".

[0003] Human pose estimation and localization often work together, but each has its own focus, together forming a complete description of the human motion state:

[0004] (1) Pure visual human pose estimation: Focusing on "pose", that is, by recognizing key human joints (such as shoulders, elbows, and knees) through images, calculating information such as relative angles between joints and limb orientation, the output form is mostly three-dimensional coordinates of joint points (relative to the camera or human body's own coordinate system) or Euler angles / quaternions (describing joint rotation).

[0005] (2) Pure visual human body positioning: Focusing on "position", that is, determining the absolute coordinates of the whole or part of the human body (such as the head and hands) in the real scene coordinate system (such as the room and street). It is necessary to associate the image with the physical space scale to solve the problem of "where exactly the human body is in the scene".

[0006] The rise of this technology is essentially to meet the demand for "contactless, high-precision human body sensing" in multiple fields, and it is mainly applied in three major scenarios:

[0007] (1) Consumer-level interactive scenario requirements: AR / VR and motion-sensing games (such as Xbox Kinect) need to capture human posture and position in real time to achieve "motion mapping" (such as waving to control virtual objects and body movement corresponding to virtual character displacement); smart homes need to determine user behavior through posture positioning (such as getting up to turn on the lights and waving to adjust the air conditioner).

[0008] (2) Industrial and medical scenarios: In the industrial field, robotic arm collaboration needs to locate the position of workers' hands to avoid collisions, and at the same time, it needs to judge the standardization of operation through posture estimation (such as hand posture when assembling parts); In the medical field, rehabilitation training needs to accurately track the patient's limb posture and movement trajectory, and evaluate the rehabilitation effect (such as knee flexion and extension angle, gait position changes).

[0009] (3) Security and autonomous driving scenario requirements: Security monitoring needs to identify abnormal behavior (such as wandering in restricted areas or falling down) through posture positioning; autonomous driving needs to locate pedestrian positions, estimate limb postures (such as pedestrians reaching out to stop cars or turning to cross the road), predict movement intentions, and improve driving safety.

[0010] Despite the widespread application of the technology, some shortcomings remain unresolved:

[0011] (1) Robustness to occlusion and complex poses: Self-occlusion of the human body (such as hugging arms or raising legs) and mutual occlusion (such as crowded scenes with many people) can lead to missing joints. Existing algorithms are prone to pose estimation errors and positioning deviations.

[0012] (2) Scale and scene adaptability: The monocular solution cannot determine the absolute scale without scene prior; the multi-view solution is affected by the camera calibration accuracy and is prone to positioning drift in dynamic scenes (such as camera movement).

[0013] (3) Real-time performance and low computing power adaptation: High-precision 3D pose and positioning algorithms (such as end-to-end models based on Transformer) require a lot of computing power and are difficult to run in real time on mobile phones and embedded devices (such as small robots), which limits their widespread application in consumer scenarios. Summary of the Invention

[0014] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for real-time human pose estimation and localization based on pure vision.

[0015] To achieve the above objectives, the present invention provides a method for real-time human pose estimation and localization based on pure vision, comprising:

[0016] Image sequences are acquired in real time using a binocular camera, and images of the target region are obtained from the image sequences.

[0017] The target region image is normalized, and multi-layer feature maps are extracted from the normalized target region image.

[0018] The multi-layer feature maps are fused to obtain multi-scale feature maps;

[0019] A dual-head mechanism is used to predict multi-scale feature maps to obtain multi-channel heatmaps and three-dimensional image coordinates for each feature point;

[0020] Convert the three-dimensional image coordinates of the feature points into world coordinates;

[0021] A physical corrector is created to correct the world coordinates of the feature points using the multi-channel heatmap as a reference, and post-processing is used to make the posture of the corrected feature points stable and continuous.

[0022] Furthermore, the target region image is obtained in the following way:

[0023] The initial frame or interval frame image in the image sequence is converted into a Mat matrix. Then, each input channel of the Mat matrix is ​​convolved using only one convolution kernel without cross-channel computation. Then, 1×1 convolution is used to combine and linearly map the channels into output channels. Finally, matrix cropping is used to obtain the target region image.

[0024] Furthermore, the target region image is normalized and then scaled to 128 pixels × 128 pixels.

[0025] Furthermore, the multi-layer feature map includes low-layer feature maps and high-layer feature maps, and its acquisition method is as follows:

[0026] First, 1×1 convolution is used to expand the number of channels in the normalized target region image data. Then, channel-wise convolution is used to extract spatial features. Finally, 1×1 convolution is used to compress the number of channels, and residual connections are added to obtain the feature map.

[0027] The feature map is pooled using spatial pyramid pooling, which has two pooling scales: 2×2 and 4×4. Each pooling scale uses a pooling layer to divide the feature map into a number of grid regions of that scale to obtain the low-level feature map and the high-level feature map.

[0028] Furthermore, the multi-scale feature map is formed by unfolding and stitching together all the low-level and high-level feature maps.

[0029] Furthermore, the multi-channel heatmap and three-dimensional image coordinates of the feature points are obtained as follows:

[0030] For the low-level feature map, a depthwise separable convolution is performed, followed by upsampling, to output a multi-channel heatmap of size [H / 4, W / 4, k], where k is the number of keypoints. The value of each pixel represents the two-dimensional image coordinates of the feature point at that location. );

[0031] For high-level feature maps, after global pooling and filtering, the output is the 3D image coordinates of the feature points. ).

[0032] Furthermore, the method for converting the three-dimensional image coordinates of the feature points into world coordinates is as follows:

[0033] First, calculate the camera intrinsic parameter matrix K as follows:

[0034]

[0035] in, , These are the camera's focal length x and y values, respectively. , These are the x and y values ​​of the image center point, respectively.

[0036] Calculate the camera depth value Z based on the physical parameters of the stereo camera:

[0037]

[0038] Where B is the distance between camera centers, f is the camera focal length, and d is the parallax value;

[0039] Transform the feature point image coordinate system to the camera coordinate system, and calculate the three-dimensional coordinates of the feature points in the camera coordinate system. )for:

[0040]

[0041] Finally, the feature point camera coordinate system is transformed into the world coordinate system, and the three-dimensional coordinate values ​​of the feature points in the world coordinate system are calculated. , , )for:

[0042]

[0043] Where R is a 3×3 rotation matrix and t is a 3×1 translation vector.

[0044] Furthermore, the physical corrector is represented as:

[0045]

[0046] in, This indicates that the parameter P of the body posture is being optimized. To predict the error between the world coordinates and projected coordinates of feature points, For human body topological constraints As a symmetry constraint, For attitude physical constraints. , , All are weighting coefficients.

[0047] Furthermore, the post-processing method is as follows:

[0048] Temporal filtering is used to fuse feature point information from multiple frames of a video sequence to smooth inter-frame jitter; depth is normalized to the proportion of human body size; the torso orientation is adjusted so that the center of the human body is in the center of the image; if the bone length change is greater than 10%, it is corrected according to the proportion of the previous frame.

[0049] Beneficial effects: This invention employs a two-step "detection + tracking" optimization method, focusing only on the target region image, significantly reducing computational load and increasing frame rate by 3-5 times while maintaining accuracy; it develops a lightweight and efficient backbone network using techniques such as "depthiable separable convolution + feature extraction + spatial pyramid pooling," featuring fewer parameters (<10MB), low latency (≤10ms), and high accuracy (error ≤10px), reducing computational load by more than 8 times, enabling real-time human pose estimation on mobile devices, and solving the problems of previous models being too large and too slow; it develops a physical corrector using geometric constraint methods such as "human topological structure constraints + symmetry constraints + pose physical constraints," combined with post-processing methods such as spatiotemporal filtering and depth normalization, making the output human pose smooth, continuous, and jitter-free, reducing positional drift in motion, and enabling pose estimation in occluded states, suitable for real-time interactive scenarios. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating a method for real-time human pose estimation and localization based on pure vision, according to an embodiment of the present invention. Detailed Implementation

[0051] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0052] like Figure 1 As shown, this embodiment of the invention provides a method for real-time human pose estimation and localization based on pure vision, including:

[0053] Image sequences are acquired in real time using a binocular camera, and the target region (ROI) image is obtained from these sequences. Specifically:

[0054] The initial or interval frame images in the image sequence are converted into Mat matrices. Then, each input channel of the Mat matrix is ​​convolved using only one kernel, without cross-channel computation. A 1×1 convolution is then used to combine and linearly map the channels into output channels. Finally, matrix cropping is used to obtain the target region image. Alternatively, the initial or interval frame images can be converted into OpenCV Mat matrices using WebCamTextureToMatHelper. By reducing the number of channels and lowering the resolution, computational load is reduced, and full image traversal is avoided.

[0055] The target region image is normalized, and multi-layer feature maps are extracted from the normalized target region image. Specifically, after normalization, the image is scaled to 128 pixels × 128 pixels to reduce the impact of scale changes on the model. The aforementioned multi-layer feature maps include low-level feature maps and high-level feature maps, and their acquisition methods are as follows:

[0056] First, 1×1 convolutions are used to expand the number of channels in the normalized target region image data. Then, channel-wise convolutions are used to extract spatial features. Finally, 1×1 convolutions are used to compress the number of channels, and residual connections are added to obtain a feature map. Then, Spatial Pyramid Pooling (SPP) is used to pool the feature map. Spatial Pyramid Pooling has two pooling scales: 2×2 and 4×4. Each pooling scale uses a pooling layer to divide the feature map into a number of grid regions of that scale, obtaining low-level and high-level feature maps. The low-level feature maps provide local edge information, while the high-level feature maps provide human structural relationships. Depthwise separable convolutions are used, decomposing the convolution into two steps: "spatial convolution" and "channel convolution." Compared to ordinary convolutions, the computational cost is reduced by more than 8 times, ensuring real-time operation on mobile devices.

[0057] Multi-level feature maps are fused to obtain multi-scale feature maps. Specifically, low-level and high-level feature maps are fully expanded and then concatenated to form multi-scale feature maps.

[0058] A dual-head mechanism is employed to predict multi-scale feature maps, thereby obtaining multi-channel heatmaps and 3D image coordinates for each feature point, as detailed below:

[0059] For the low-level feature map, a depthwise separable convolution is performed, followed by upsampling, to output a multi-channel heatmap of size [H / 4, W / 4, k], where k is the number of keypoints. The value of each pixel represents the two-dimensional image coordinates of the feature point at that location. For high-level feature maps, after global pooling and filtering, the 3D image coordinates of the feature points are output. ).

[0060] The method for converting the 3D image coordinates of feature points to world coordinates is as follows:

[0061] First, calculate the camera intrinsic parameter matrix K as follows:

[0062]

[0063] in, , These are the camera's focal length x and y values, respectively. , These are the x and y values ​​of the image center point, respectively. , , , All data is in pixels.

[0064] Calculate the camera depth value Z based on the physical parameters of the stereo camera:

[0065]

[0066] Where B is the camera center-to-center distance in meters. f is the camera focal length in pixels. d is the parallax value, which is the difference in the horizontal coordinates of the pixels between the two cameras, in pixels.

[0067] Transform the feature point image coordinate system to the camera coordinate system, and calculate the three-dimensional coordinates of the feature points in the camera coordinate system. )for:

[0068]

[0069] Finally, the feature point camera coordinate system is transformed into the world coordinate system, and the three-dimensional coordinate values ​​of the feature points in the world coordinate system are calculated. , , )for:

[0070]

[0071] Where R is a 3×3 rotation matrix and t is a 3×1 translation vector.

[0072] A physical corrector is created to correct the world coordinates of feature points, and post-processing is used to ensure the pose of the corrected feature points is stable and continuous. Specifically, the physical corrector is represented as follows:

[0073]

[0074] in, This indicates that the parameter P of the body posture is being optimized. To predict the error between the world coordinates and projected coordinates of feature points, For human body topological constraints As a symmetry constraint, For attitude physical constraints. , , All are weighting coefficients. Pure vision models are prone to joint shifts due to occlusion and lighting changes. Post-processing using human kinematic geometry constraints improves stability and consistency. Based on human topological constraints, the human body is a "kinematic chain system" with fixed connections and relative length constraints between feature points. Preset relative lengths between feature point world coordinates (e.g., thigh length ≈ calf length, upper arm length ≈ forearm length). If the predicted spacing of feature point world coordinates exceeds a reasonable range, the coordinates are automatically corrected. Based on symmetry constraints, the human body is bilaterally symmetrical, correcting unilateral drift, stabilizing the posture center, and estimating occluded feature points. Based on posture physics constraints, ensuring the posture conforms to human physiological limits. Each joint's rotation angle has a physical upper limit (e.g., elbow flexion angle 0-140°, knee flexion angle 0-180°), and abnormal angles are corrected using an inverse kinematics (IK) model.

[0075] The post-processing method described above is as follows:

[0076] Temporal filtering is used to fuse feature point information from multiple frames of a video sequence to smooth inter-frame jitter; depth is normalized to the proportion of human body size; the torso orientation is adjusted to center the human body in the image; if the bone length change is greater than 10%, it is corrected according to the proportion of the previous frame. This reduces pose jitter in fast-moving and occluded scenes, maintaining stable and consistent pose.

[0077] The above description is merely a preferred embodiment of the present invention. It should be noted that for those skilled in the art, other parts not specifically described are existing technology or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for real-time human pose estimation and localization based on pure vision, characterized in that, include: Image sequences are acquired in real time using a binocular camera, and images of the target region are obtained from the image sequences. The target region image is normalized, and multi-layer feature maps are extracted from the normalized target region image. The multi-layer feature maps are fused to obtain multi-scale feature maps; A dual-head mechanism is used to predict multi-scale feature maps to obtain multi-channel heatmaps and three-dimensional image coordinates for each feature point; Convert the three-dimensional image coordinates of the feature points into world coordinates; A physical corrector is created to correct the world coordinates of the feature points using the multi-channel heatmap as a reference, and post-processing is used to make the posture of the corrected feature points stable and continuous. The target region image is obtained in the following way: The initial frame or interval frame image in the image sequence is converted into a Mat matrix. Then, each input channel of the Mat matrix is ​​convolved using only one convolution kernel without cross-channel calculation. Then, 1×1 convolution is used to combine and linearly map the channels into output channels. Finally, matrix cropping is used to obtain the target region image. The multi-layer feature map includes low-layer feature maps and high-layer feature maps, and its acquisition method is as follows: First, 1×1 convolution is used to expand the number of channels in the normalized target region image data. Then, channel-wise convolution is used to extract spatial features. Finally, 1×1 convolution is used to compress the number of channels, and residual connections are added to obtain the feature map. Spatial pyramid pooling is used to pool the feature map. Spatial pyramid pooling is set with two pooling scales: 2×2 and 4×4. Each pooling scale uses a pooling layer to divide the feature map into a number of grid regions of that scale to obtain the low-level feature map and the high-level feature map. The multi-channel heatmap and three-dimensional image coordinates of the feature points are obtained as follows: For the low-level feature map, a depthwise separable convolution is performed, followed by upsampling, to output a multi-channel heatmap of size [H / 4, W / 4, k], where k is the number of keypoints. The value of each pixel represents the two-dimensional image coordinates of the feature point. ); For high-level feature maps, after global pooling and filtering, the output is the 3D image coordinates of the feature points. ); The method for converting the 3D image coordinates of the feature points to world coordinates is as follows: First, calculate the camera intrinsic parameter matrix K as follows: in, , These are the camera's focal length x and y values, respectively. , These are the x and y values ​​of the image center point, respectively. Calculate the camera depth value Z based on the physical parameters of the stereo camera: Where B is the distance between camera centers, f is the camera focal length, and d is the parallax value; Transform the feature point image coordinate system to the camera coordinate system, and calculate the three-dimensional coordinates of the feature points in the camera coordinate system. )for: Finally, the feature point camera coordinate system is transformed into the world coordinate system, and the three-dimensional coordinate values ​​of the feature points in the world coordinate system are calculated. , , )for: Where R is a 3×3 rotation matrix and t is a 3×1 translation vector.

2. The method for real-time human pose estimation and localization based on pure vision according to claim 1, characterized in that, The target region image is normalized and then scaled to 128 pixels × 128 pixels.

3. The method for real-time human pose estimation and localization based on pure vision according to claim 1, characterized in that, The multi-scale feature map is formed by unfolding and stitching together the low-level feature map and the high-level feature map.

4. The method for real-time human pose estimation and localization based on pure vision according to claim 1, characterized in that, The physical corrector is represented as: in, This indicates that the parameter P of the body posture is being optimized. To predict the error between the world coordinates and projected coordinates of feature points, For human body topological constraints As a symmetry constraint, For attitude physical constraints. , , All are weighting coefficients.

5. The method for real-time human pose estimation and localization based on pure vision according to claim 1, characterized in that, The post-processing method is as follows: Temporal filtering is used to fuse feature point information from multiple frames of a video sequence to smooth inter-frame jitter; depth is normalized to the proportion of human body size; the torso orientation is adjusted so that the center of the human body is in the center of the image; if the bone length change is greater than 10%, it is corrected according to the proportion of the previous frame.

Images