Network system and method for extracting human skeleton points

Abstract

The invention relates to a network system and method for extracting human skeleton points, and belongs to the technical field of computer vision. The problems that in the prior art, OpenPose is prone to feature loss, partial affinity field breakage and joint matching errors in a shielding scene are solved. According to the technical scheme, the method comprises the steps that ResNet50 is combined with a backbone network of an adaptive cavity space pyramid pooling module to enhance multi-scale feature extraction, a dual-granularity occlusion sensing module is designed to position an occlusion area from the joint and limb levels, and a confidence map and a part of affinity field are repaired through topological constraint and an instance segmentation technology; and joint connection is optimized by adopting a shielding self-adaptive matching algorithm. The method has the beneficial effects that the robustness of a shielding scene is remarkably improved, the balance of light weight and real-time performance is realized, the accuracy of feature extraction and matching is improved, and the method has strong universality and expansibility.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a network system and method for extracting human skeleton points. Background Technology

[0002] Human movement, as one of the most important biological characteristics of human behavior, benefits from automated assessment technologies that contribute to a deeper understanding of human behavior. Action Quality Assessment (AQA) has become an emerging hot topic in human behavior understanding research and has broad application prospects in industry, such as patient rehabilitation medicine, automated scoring of sports activities, skill assessment of automated robotic services, and level evaluation of specific skills. It is also an important research direction in the fields of video understanding and computer vision.

[0003] Based on different feature extraction methods, action evaluation methods can be mainly divided into methods based on handcrafted features and methods based on deep learning. The latter shows significant advantages over the former in terms of recognition accuracy and robustness, and has become the current mainstream technical approach. Deep learning-based implementation methods can be broadly divided into methods based on RGB video streams and methods based on human skeleton sequences. Compared with video data, skeletal sequence data is more efficient to process and can effectively avoid complex background interference and redundant pixel information in video data, thereby significantly reducing the complexity of data preprocessing and feature extraction.

[0004] In the field of human skeletal keypoint detection, traditional algorithms include methods based on human proportions, methods based on human contour information, and methods based on human contour segmentation. While methods based on human proportions have some applicability, they are susceptible to individual differences, leading to potential errors in detection location. Methods based on contour information are prone to false positives due to the similarity of features across different body parts. Contour segmentation methods perform better in terms of accuracy and speed. Currently, deep learning technology is primarily used in this field, and its algorithms can be broadly categorized into top-down and bottom-up approaches. Top-down methods, such as AlphaPose, require detecting the human body region before locating keypoints, resulting in slower detection speeds. In contrast, current mainstream algorithms, such as OpenPose, employ a bottom-up strategy, detecting all keypoints first and then grouping and connecting them, leading to higher efficiency.

[0005] The OpenPose algorithm, proposed by Carnegie Mellon University in 2018, is a human pose estimation algorithm with stable performance and accuracy that is not easily affected by multiple people or complex limb movements. Therefore, in motion quality assessment research, using its output skeletal sequence data can improve processing efficiency and reduce environmental interference.

[0006] However, the OpenPose algorithm has significant drawbacks when faced with occluded scenes, easily exhibiting issues such as feature loss, partial affinity fields (PAFs), PAF breakage, and joint matching errors. These limitations severely restrict the algorithm's effectiveness in real-world, complex environments. For instance, in crowded scenes or rehabilitation therapy where limbs are partially occluded, the accuracy and reliability of skeletal point extraction decrease significantly.

[0007] Therefore, there is an urgent need for a novel human skeleton point extraction network that can effectively address the challenges of occlusion, thereby improving its robustness and accuracy in complex real-world scenarios. This invention is proposed to overcome the shortcomings of the existing technologies. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide a network system and method for extracting human skeletal points.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A network system for extracting human skeletal points, comprising: Backbone network, occlusion perception module, feature repair module, and joint matching module; The backbone network is configured to receive input images and output multi-scale features; The occlusion perception module is configured to generate joint occlusion masks and limb occlusion masks based on the multi-scale features. The feature repair module is configured to repair the confidence map (CM) and part affinity fields (PAF) based on the occlusion mask; The joint matching module is configured to perform joint matching based on the repaired CM and PAF to output human skeletal points.

[0010] Furthermore, the backbone network includes a ResNet50 structure and an adaptive Atrous Spatial Pyramid Pooling (ASPP) module; the ResNet50 structure is used to extract features, and the adaptive ASPP module is used to achieve multi-scale feature capture.

[0011] Furthermore, the adaptive ASPP module includes a multi-scale convolution combination, which consists of convolutional layers with 1x1 convolution, 3x3 convolution with a dilation rate of 2, 3x3 convolution with a dilation rate of 4, and 3x3 convolution with a dilation rate of 6.

[0012] Furthermore, the occlusion perception module includes a joint-level occlusion branch and a limb-level occlusion branch; the joint-level occlusion branch is used to generate joint point occlusion masks. The limb-level occlusion branch is used to generate limb occlusion masks. .

[0013] Furthermore, the CM repair of the feature repair module includes topology-guided peak enhancement and global constraints of graph convolutional networks; The topology-guided peak enhancement is used to calculate candidate positions of occluded joints based on adjacent joint locations, wherein the candidate position calculation formula is: i in The peak coordinates of CM for the unobstructed joints. and This is an adaptive offset based on human height; The global constraints of the graph convolutional network are used to propagate CM features across joints through the adjacency matrix A, as shown in the formula:

[0014] in For the normalized adjacency matrix, For learnable weight matrix, This is the Sigmoid activation function.

[0015] Furthermore, the PAF repair of the feature repair module includes occluded limb PAF completion and multi-person interference filtering; the occluded limb PAF completion is used to generate a completed PAF vector and fuse it with the original PAF, and the fusion formula is:

[0016] in For limbs The occlusion mask value; the multi-person interference filtering is used to output the human instance mask through the instance segmentation branch and filter the PAF vector of non-current human.

[0017] A method for extracting human skeletal points includes the following steps: Input image; Multi-scale features of the input image are extracted using a backbone network; The occlusion perception module generates joint occlusion masks and limb occlusion masks based on the multi-scale features. The feature repair module repairs CM and PAF based on the occlusion mask; The joint matching module performs joint matching based on the repaired CM and PAF to output human skeletal points.

[0018] Furthermore, the joint matching module employs an occlusion-adaptive matching algorithm during joint matching, including weighted adjustment of matching scores and joint completion after matching; the formula for weighted adjustment of matching scores is:

[0019] in The score is the original match score. For limbs j The occlusion mask value, These are the topological constraint coefficients; if the joint pairs conform to the topological relationships of the human skeleton, then... =1.2, otherwise =0.5; The matched joint completion is used to perform linear interpolation on the missing joints, and the interpolation formula is as follows: and ,in For wrist joint coordinates, Let the elbow joint coordinates be... The coordinates are for the shoulder joint.

[0020] Furthermore, it also includes a multi-task loss function training step, wherein the multi-task loss function formula is:

[0021] in The mean squared error loss of CM. This represents the mean squared error loss of PAF. To mitigate the cross-entropy loss of the occlusion sensing module, Dice loss for instance segmentation, To obscure the weighted loss, =0.1、 =0.05、 1.0 is a hyperparameter.

[0022] Furthermore, the occlusion-weighted loss The calculation formula is:

[0023] in Key points i The occlusion mask value, To predict the CM value, This is the actual CM value. For limbs j The occlusion mask value, To predict PAF values, This is the actual PAF value.

[0024] The beneficial effects of this invention are as follows: This invention effectively overcomes the inherent defects of existing technologies in occluded scenarios through a systematic improvement across the entire process. First, the system's robustness to occlusion has achieved a qualitative leap, accurately identifying and handling joint point loss or misjudgment caused by occlusion, ensuring the stability and reliability of skeletal point extraction in complex real-world environments.

[0025] Secondly, while pursuing high performance, this invention cleverly balances model complexity and processing efficiency. Through a lightweight network structure design, the system significantly improves computational speed while maintaining high-precision feature extraction capabilities, fully meeting the stringent requirements of real-time application scenarios.

[0026] The advantages of this invention are particularly prominent in the core feature processing stage. The system can capture multi-scale posture features of the human body more comprehensively and precisely, and intelligently repair and complete broken or blurred feature information. This not only significantly improves the accuracy of joint point localization, but also greatly optimizes the matching accuracy of joint connections.

[0027] Furthermore, this invention possesses exceptional versatility and scalability. The model requires no complex parameter adjustments for specific application scenarios and can be directly deployed in multiple important fields such as rehabilitation medicine assessment, sports activity analysis, and robotic human-computer interaction. Its modular design also allows core components to be flexibly adapted to other related technical frameworks, demonstrating broad prospects for engineering applications.

[0028] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a network diagram for extracting human skeletal points. Detailed Implementation

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0031] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0032] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0033] 1. High-receptive-field lightweight backbone network reconstruction To address the insufficient feature extraction capabilities of the original VGG-19 backbone, this paper proposes a backbone structure of "ResNet50+ Adaptive Hollow Spatial Pyramid Pooling (ASPP)". The core improvements are reflected in two aspects: multi-scale adaptation of feature extraction and lightweight design. 1.1 Residual Feature Enhancement: ResNet50 is used to replace VGG-19. The gradient vanishing problem in deep networks is alleviated through residual connections. At the same time, its hierarchical feature extraction capability is utilized to obtain three types of key information, namely limb structure, joint position and global context, from the C3, C4 and C5 feature layers, respectively, to provide richer feature support for subsequent CM and PAF prediction.

[0034] 1.2 Adaptive ASPP Module: An improved ASPP module is introduced at the ResNet50 output, designing a multi-scale convolution combination of "1×1 + 3×3 (dilation rate 2) + 3×3 (dilation rate 4) + 3×3 (dilation rate 6)". Compared to the original ASPP with a fixed dilation rate setting, this combination dynamically adjusts the receptive field range according to the scale characteristics of human pose features, capturing global semantic associations in occluded regions while preserving spatial details of joints.

[0035] 1.3 Lightweight Optimization: Depthwise separable convolutions were used to replace standard convolutions in the C4 and C5 layers of ResNet50. With only a 12% increase in the number of parameters, the inference speed in the feature extraction stage was improved by 18%, resolving the contradiction between high receptive field networks and real-time requirements.

[0036] 2. Joint-limb dual-granularity occlusion perception mechanism To achieve accurate localization of occluded areas, this paper designs a dual-granularity occlusion perception module, which generates occlusion guidance information from two dimensions: joints and limbs. The core structure of the module consists of three parts: feature reduction, bi-branch prediction, and supervised training. 2.1 Feature Input and Dimensionality Reduction: The C4 feature layer (1 / 8 resolution, 1024 channels) output from the backbone network is received, and the number of channels is compressed to 256 through 1×1 convolution to reduce subsequent computational costs.

[0037] 2.2 Two-branch prediction structure: (1) Joint-level occlusion branch: A stacked structure of “3×3 depthwise separable convolution + BN + ReLU” is adopted, and the output dimension is “number of joints × feature map height × feature map width”. .in The closer the value is to 1, the more likely it is to be the first... i The higher the probability that a key point is occluded at the (h,w) position.

[0038] (2) Limb-level occlusion branch: Based on the topological relationship of the human skeleton (such as 19 limbs such as shoulder-elbow, elbow-wrist, etc.), a limb feature map is generated by fusing adjacent features on the basis of joint-level features, and the limb occlusion mask is output through the same convolutional structure. This enables the overall positioning of the area where the limb is obscured.

[0039] 2.3 Supervision Signal Generation: Combining joint visibility annotations from the COCO and CrowdPose datasets, the supervision signal generation is... Use binary cross-entropy loss; for The system uses joint occlusion status and the intersection-union ratio (IoU) of the human bounding box to jointly label samples (occlusion is defined as any joint at either end of a limb being occluded or an IoU of the limb region with other human bounding boxes being greater than 0.3). Weighted cross-entropy loss is used to enhance the learning of severely occluded samples.

[0040] 3. CM and PAF feature repair for topological constraints To address the issues of peak blurring in CM and fragmentation in PAF vectors caused by occlusion, this paper proposes a feature restoration strategy of "topology guidance + context enhancement," which performs targeted optimizations on the two types of core features respectively: 3.1 Confidence Map (CM) Completion Mechanism CM is used to locate candidate key positions, and its completion process is divided into two stages: topological reasoning and context enhancement. 3.1.1 Topology-guided peak enhancement: Construct a human joint adjacency matrix A (e.g., wrist joint adjacent to elbow joint, elbow joint adjacent to shoulder joint), and perform topology-guided peak enhancement. For occluded joint i with a value >0.5, extract its adjacent unoccluded joints. j CM peak coordinates Calculate joints based on average human body proportions i Candidate positions:

[0041] in , This is an adaptive offset based on human height. At the joint... i In CM, with Center, radius r ( r A gain weight of 1.5 times is applied to a region equal to (human height × 0.02) to enhance the peak signal of the occluded joint.

[0042] 3.1.2 Global Constraints of Graph Convolutional Networks: CM features of all joints are split along the joint dimension and input into a 2-layer graph convolutional network. Feature propagation across joints is achieved through the adjacency matrix A, as shown in the following formula:

[0043] in For the normalized adjacency matrix, For learnable weight matrix, This is the Sigmoid activation function. This operation utilizes skeletal topological relationships to achieve global completion of occluded joint features, ensuring that the CM conforms to prior knowledge of human anatomy.

[0044] For scenarios with background occlusion (no adjacent joints to depend on), a new spatial context attention module is added: global average pooling is performed on the CM to generate channel-level attention weights, which are then converted into spatial attention maps after 1×1 convolution and sigmoid activation. ,pass Element-wise multiplication with CM enhances regional features related to the semantics of occluded joints (e.g., when the wrist joint is occluded by a table, the features of the arm region are emphasized).

[0045] 3.2 Instance-Aware PAF Repair Strategy PAF is the core basis for joint matching, and its repair focuses on vector completion of occluded limbs and interference filtering in multi-person scenes: 3.2.1 PAF completion for occluded limbs: (This is incomplete and requires further context.) >0.5 limb obstruction j If one end of the joint is not obscured, then starting from the peak position of the CM of that joint, a complete PAF vector is generated along the standard limb direction (e.g., the standard angle between shoulder and elbow is 135°±30°). The vector length is adaptively adjusted according to human body proportions (e.g., the elbow-wrist length is 0.8 ± 0.1 times the shoulder-elbow length). The completed vector is then merged with the original PAF.

[0046] in This enables dynamic adaptation that relies more heavily on completion features as the occlusion worsens.

[0047] 3.2.2 Multi-person interference filtering: A lightweight instance segmentation branch is added after the PAF branch. It outputs a human instance mask using a structure of "3×3 convolution + upsampling + 1×1 convolution". Each instance mask is element-wise multiplied with the PAF, retaining only the PAF vector within the current human region and filtering out PAF interference from other human bodies. This branch is trained using the Dice loss function, achieving an instance segmentation accuracy of 89.2% on the Crowd Pose dataset.

[0048] 4. Occlusion-adaptive joint matching algorithm The original greedy matching algorithm of OpenPose does not consider the joint characteristics of occluded scenes. This paper proposes a matching optimization strategy that integrates occlusion weights and topological constraints: 4.1 Match Score Weighted Adjustment: Adjust the original match score... ( v The improved version (for joint pairs of unit vectors) is as follows:

[0049] in These are topological constraint coefficients. If the joint pairs conform to the topological relationships of the human skeleton (e.g., the angle range of shoulder-elbow-wrist is 120°~180°), then... =1.2, otherwise =0.5, which improves the matching priority of occluded limbs through weight adjustment.

[0050] 4.2 Joint Completion After Matching: For occluded joints that are still missing after matching, linear interpolation is used for completion (e.g., when the wrist joint is missing, it is calculated using the coordinates of the elbow and shoulder joints).

[0051]

[0052] The confidence level of the completed joint markers was (0.7±0.1), providing a reliability reference for subsequent applications.

[0053] 5. Design of Multi-Task Loss Function To collaboratively optimize the performance of each module, a multi-task fusion loss function is designed, as shown in the following formula:

[0054] The loss terms are defined as follows: 5.1 and The traditional MSE loss is used to optimize the prediction accuracy of the confidence plot and PAF respectively. 5.2 : Joint loss of the occlusion perception module, including cross-entropy loss of joint-level and limb-level occlusion masks; 5.3 : Dice loss for instance segmentation branch, to optimize the segmentation effect of human instance mask; 5.4 Occlusion-weighted loss assigns higher weight to the feature prediction errors of occluded joints and limbs.

[0055] Experiments have verified that the hyperparameters are set to... =0.1、 =0.05、 When the value is 1.0, the model performs optimally in occluded scenarios.

[0056] The core of this invention is to verify the effectiveness of the improved scheme of "backbone upgrade - occlusion perception - feature repair - matching optimization" through standardized experimental design and full-process technical implementation. The specific summary is as follows: (1) Core objectives of the plan In the scenario of skeletal point extraction for human motion quality assessment, this paper addresses the problems of feature loss, PAF breakage, and joint matching errors in existing OpenPose algorithms under occlusion scenarios, while also taking into account the lightweight nature and real-time performance of the model, ensuring accurate and efficient extraction of human skeletal points in real and complex scenarios.

[0057] (2) Key Implementation Steps Data preparation and preprocessing: The COCO 2017 dataset (330,000 images, including 17 human key skeletal point annotations) and the CrowdPose-Occl occlusion dataset were used to cover real-world scenarios such as daily scenes, natural lighting, different degrees of occlusion, and complex backgrounds, avoiding the limitations of a single experimental scenario.

[0058] Normalization (resize to 640×480) and data augmentation (random flipping, ±15° rotation, brightness / contrast adjustment) are performed on the images, and occlusion mask supervision signals are generated based on the joint visibility annotations of the dataset to meet the training requirements of the dual-granularity occlusion perception module.

[0059] (3) Model deployment and training configuration Hardware environment: Intel Core i7-12700K CPU, NVIDIA RTX 3090 GPU (24GB VRAM) to ensure computing power support for model training and inference.

[0060] Software framework: The training process is built based on PyTorch 1.12, with a batch size of 16, an initial learning rate of 1e-4, and the AdamW optimizer used for 100 iterations of training.

[0061] (4) Implementation of core technology modules Backbone network: Deploys a “ResNet50+Adaptive ASPP” structure, alleviates gradient vanishing through residual connections, dynamically adapts to human pose features through multi-scale convolutional combinations, and uses depthwise separable convolutions in layers C4 and C5 to achieve lightweighting.

[0062] Occlusion perception: The joint-limb dual-granularity occlusion perception module is enabled. It receives the C4 feature layer of the backbone network, and after dimensionality reduction, it generates an occlusion mask through dual-branch prediction. Supervised training is completed by combining binary cross-entropy loss and weighted cross-entropy loss.

[0063] Feature repair: A "topology-guided peak enhancement + graph convolution global constraint + spatial context attention" completion mechanism is adopted for CM, and an "occluded limb vector completion + instance segmentation interference filtering" strategy is implemented for PAF to repair feature defects caused by occlusion.

[0064] Joint matching: An occlusion adaptive matching algorithm is applied, which improves the accuracy of joint matching in occluded scenarios by adjusting the matching score with weights and using linear interpolation to fill in missing joints.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A network system for extracting human skeletal points, characterized in that: include: Backbone network, occlusion perception module, feature repair module, and joint matching module; The backbone network is configured to receive input images and output multi-scale features; The occlusion perception module is configured to generate joint occlusion masks and limb occlusion masks based on the multi-scale features. The feature repair module is configured to repair the confidence map CM and part of the affinity field PAF based on the occlusion mask; The joint matching module is configured to perform joint matching based on the repaired CM and PAF to output human skeletal points.

2. The network system for extracting human skeletal points according to claim 1, characterized in that: The backbone network includes a ResNet50 structure and an adaptive spatial pyramid pooling (ASPP) module; the ResNet50 structure is used to extract features, and the adaptive ASPP module is used to achieve multi-scale feature capture.

3. The network system for extracting human skeletal points according to claim 2, characterized in that: The adaptive ASPP module includes a multi-scale convolution combination, which consists of convolutional layers with 1x1 convolution, 3x3 convolution with a dilation rate of 2, 3x3 convolution with a dilation rate of 4, and 3x3 convolution with a dilation rate of 6.

4. The network system for extracting human skeletal points according to claim 1, characterized in that: The occlusion sensing module includes a joint-level occlusion branch and a limb-level occlusion branch. The joint-level occlusion branch is used to generate joint occlusion masks. The limb-level occlusion branch is used to generate limb occlusion masks. .

5. The network system for extracting human skeletal points according to claim 1, characterized in that: The CM repair of the feature repair module includes topology-guided peak enhancement and global constraints of graph convolutional networks; The topology-guided peak enhancement is used to calculate candidate positions of occluded joints based on adjacent joint locations, wherein the candidate position calculation formula is: i in The peak coordinates of CM for the unobstructed joints. and This is an adaptive offset based on human height; The global constraints of the graph convolutional network are used to propagate CM features across joints through the adjacency matrix A, as shown in the formula: in For the normalized adjacency matrix, For learnable weight matrix, This is the Sigmoid activation function.

6. The network system for extracting human skeletal points according to claim 1, characterized in that: The PAF repair of the feature repair module includes occluded limb PAF completion and multi-person interference filtering; the occluded limb PAF completion is used to generate a completed PAF vector and fuse it with the original PAF, and the fusion formula is: in For limbs The occlusion mask value; the multi-person interference filtering is used to output the human instance mask through the instance segmentation branch and filter the PAF vector of non-current human.

7. A method for extracting human skeletal points, characterized in that: Includes the following steps: Input image; Multi-scale features of the input image are extracted using a backbone network; The occlusion perception module generates joint occlusion masks and limb occlusion masks based on the multi-scale features. The feature repair module repairs CM and PAF based on the occlusion mask; The joint matching module performs joint matching based on the repaired CM and PAF to output human skeletal points.

8. The method for extracting human skeletal points according to claim 7, characterized in that: The joint matching module employs an occlusion-adaptive matching algorithm during joint matching, including weighted adjustment of matching scores and joint completion after matching; the formula for weighted adjustment of matching scores is: in The score is the original match score. For limbs j The occlusion mask value, These are the topological constraint coefficients; if the joint pairs conform to the topological relationships of the human skeleton, then... =1.2, otherwise =0.5; The matched joint completion is used to perform linear interpolation on the missing joints, and the interpolation formula is as follows: and ,in For wrist joint coordinates, Let the elbow joint coordinates be... The coordinates are for the shoulder joint.

9. The method for extracting human skeletal points according to claim 7, characterized in that: It also includes a multi-task loss function training step, the formula of which is: in The mean squared error loss of CM. This represents the mean squared error loss of PAF. To mitigate the cross-entropy loss of the occlusion sensing module, Dice loss for instance segmentation, To obscure the weighted loss, =0.1、 =0.05、 1.0 is a hyperparameter.

10. The method for extracting human skeletal points according to claim 9, characterized in that: The occlusion weighted loss The calculation formula is: in Key points i The occlusion mask value, To predict the CM value, This is the actual CM value. For limbs j The occlusion mask value, To predict PAF values, This is the actual PAF value.

Images