Human body joint angle identification method and system based on improved OpenPose network

Abstract

The invention discloses a human body joint angle identification method and system based on an improved OpenPose network, and relates to the technical field of computer vision and medical treatment, and the method comprises the steps: obtaining a target human body image through an RGB image collection device; human skeleton key points are extracted based on a PAF vector field algorithm of an improved OpenPose framework, and high-precision skeleton information is generated; human body joint information is constructed according to the skeleton key points, and then a joint buckling angle, an abduction angle and a rotation angle are calculated through a geometric algorithm; and comparing and analyzing the joint angle data with parameters in a medical standard database, identifying an abnormal activity range or pathological characteristics, and generating a personalized medical suggestion report which comprises rehabilitation training guidance, a posture correction scheme or potential disease early warning. According to the method, the problem of skeleton misconnection in a shielding and multi-person scene can be solved, and an efficient and low-cost joint function evaluation scheme is provided for clinical diagnosis, postoperative rehabilitation and chronic disease monitoring in combination with a knowledge base in the medical field.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and medical technology, and in particular to a method for recognizing human joint angles based on an improved OpenPose network. Background Art

[0002] Changes in human joint angles reflect the movement function of the joints and can provide strong support for early diagnosis of diseases, rehabilitation treatment and chronic disease monitoring.

[0003] In existing technologies, OpenPose, an open-source library for real-time multi-person pose estimation, can cause joint misconnection in complex poses or motion patterns. This occurs when one joint is incorrectly connected to an unrelated joint. This misconnection can severely impact the calculation and analysis of joint angles, leading to incorrect motion pattern recognition and pathological diagnoses. Human pose estimation involves the collection and processing of image and video data, which can pose a risk of privacy breaches. Preventing occlusion and misconnection of human joints is a critical issue that needs to be addressed. Summary of the Invention

[0004] The embodiments of the present invention provide a method and system for recognizing human joint angles based on an improved OpenPose network, which can solve the problem in the prior art that human joint points cannot be blocked or misconnected.

[0005] The embodiment of the present invention provides a method for recognizing human joint angles based on an improved OpenPose network, comprising the following steps: Obtain RGB image data of the target human body; The RGB image data of the target human body is input into the improved OpenPose network. The PAF vector field algorithm with an occlusion-aware layer added in the generation phase is used to obtain the PAF vector field connecting the human skeletal joints. A bidirectional matching strategy is used to eliminate the misconnected joints caused by occlusion in the PAF vector field and generate the human skeleton information. According to the human skeleton information, the inverse kinematics (IK) algorithm is used to identify the human joint angles, and the dynamic time warping (DTW) algorithm is used to obtain the trend of human joint angle changes over time, and the human joint angles in the active state are obtained.

[0006] Furthermore, the PAF vector field algorithm that adds an occlusion perception layer in the generation stage is used to obtain a PAF vector field connecting the joints of the human skeleton, and a two-way matching strategy is used to eliminate the misconnection of joints caused by occlusion in the PAF vector field to generate human skeleton information. The specific steps include: Based on the RGB image data of the target human body, a convolutional neural network (CNN) is used to generate a joint point heat map. The PAF vector field algorithm that adds an occlusion perception layer in the generation phase is used to obtain the PAF vector field connecting the human skeleton joints. The formula is: ; when( x , y ) in the connection area, = ;otherwise =0; p A and p B are the coordinates of the two joint points; The bidirectional matching strategy is used to eliminate the misconnection of joint points caused by occlusion in the PAF vector field. The formula is: Define the joint cost: ; Among them, the PAF vector field is L c , c is the limb type; p( u ) is the joint point v i arrive v j Path, joint points v i The coordinates are , joint points v j The coordinates are , d ij = p j - p i ; Get the minimum cost matching from all costs using the Hungarian algorithm and add the following constraints: ; in, is the preset minimum length, is the preset maximum length; The joint points are filtered according to the confidence threshold in the PAF vector field to obtain the human skeleton information.

[0007] Furthermore, after identifying the human body joint angles using the inverse kinematics IK algorithm, the method further includes: correcting the angle deviation caused by the image perspective distortion through an error compensation mechanism.

[0008] Furthermore, the method for identifying human joint angles based on the improved OpenPose network also includes: comparing the trend of changes in human joint angles over time with preset standard parameters, identifying abnormal joint movement characteristics, and generating a recommendation report based on the identification results.

[0009] Furthermore, before generating the recommendation report based on the recognition result, the method further includes: performing fuzzy processing on the face and identity identification information in the RGB image data of the target human body, and retaining the skeleton data.

[0010] The embodiment of the present invention provides a human joint angle recognition system based on an improved OpenPose network, comprising: Image acquisition module, used to obtain RGB image data of the target human body; The network improvement module is used to input the RGB image data of the target human body into the improved OpenPose network. The PAF vector field algorithm with an occlusion-aware layer added to the generation phase is used to obtain the PAF vector field connecting the human skeletal joints. The bidirectional matching strategy is used to eliminate the misconnection of joints caused by occlusion in the PAF vector field and generate human skeleton information. The human joint angle recognition module is used to identify the human joint angles based on the human skeleton information using the inverse kinematics IK algorithm, and obtain the trend of human joint angle changes over time through the dynamic time warping DTW algorithm to obtain the human joint angles in the active state.

[0011] The embodiments of the present invention provide a method and system for recognizing human joint angles based on an improved OpenPose network. Compared with the prior art, the method and system have the following advantages: An improved OpenPose network is used to process the RGB image data of the target human body. The improved OpenPose network includes: obtaining the PAF vector field connecting the human skeletal joints through the PAF vector field algorithm with an occlusion-aware layer in the generation phase, and eliminating joint misconnections caused by occlusion in the PAF vector field through a two-way matching strategy to generate human skeleton information. When encountering occlusion in multi-person scenes, the occlusion-aware layer can prevent human joints from being occluded and misconnected. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a method for recognizing human joint angles based on an improved OpenPose network provided in an embodiment of the present invention; Figure 2 A UI interface display diagram of a human joint angle recognition system based on an improved OpenPose network provided by an embodiment of the present invention; Figure 3 A schematic diagram of medical measurement provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0014] See also Figure 1 , an embodiment of the present invention provides a method for recognizing human joint angles based on an improved OpenPose network, comprising the following steps: Step 1: Obtain RGB image data of the target human body.

[0015] Step 2: Input the target person's RGB image data into the improved OpenPose network. Using the PAF vector field algorithm with an occlusion-aware layer added to the generation phase, the PAF vector field connecting the human skeletal joints is obtained. A bidirectional matching strategy is used to eliminate misconnected joints in the PAF vector field due to occlusion, generating the human skeleton information. The specific steps include: generating a joint heat map using a convolutional neural network (CNN) based on the target person's RGB image data; using the PAF vector field with an occlusion-aware layer added to the generation phase, mapping the joints in the joint heat map to the person, and using a bidirectional matching strategy to eliminate misconnected joints due to occlusion; and filtering joints based on the confidence threshold in the PAF vector field to obtain the human skeleton information.

[0016] Step 3: Based on the human skeleton information, the inverse kinematics (IK) algorithm is used to identify the human joint angles. The dynamic time warping (DTW) algorithm is then used to determine the temporal trend of the human joint angles, thereby obtaining the joint angles in the active state. An error compensation mechanism is used to correct for angle deviations caused by image perspective distortion.

[0017] Step 4: Compare the trend of human joint angle changes over time with preset standard parameters to identify abnormal joint movement characteristics; blur the face and identity information in the RGB image data of the target human body, retaining the skeleton data; generate a recommendation report based on the recognition results.

[0018] The specific contents are as follows: Step 1: Image acquisition and preprocessing: Use RGB image acquisition equipment to obtain static or dynamic image data of the target human body. To ensure data quality, the following preprocessing is performed, including:

[0019] Step 1.1: Image illumination correction, using histogram equalization technology to reduce the impact of illumination changes on image quality.

[0020] Step 1.2: Noise suppression: Remove random noise from the image through Gaussian filtering.

[0021] Step 1.3: Human body region segmentation: Use the deep learning model to accurately segment the human body region in the image and remove background interference.

[0022] Step 2: Using the improved OpenPose framework, the PAF vector field algorithm is used to extract the key points of the human skeleton and generate skeleton information, such as Figure 2 The skeleton information displayed by the system includes: Step 2.1. Use convolutional neural network (CNN) to generate joint point heatmap and PAF vector field. The PAF vector field is defined as the direction vector field connecting two joint points, and its formula is:

[0023] .

[0024] Among them, if (x,y) is in the connected area, then = ,otherwise =0.

[0025] and, p A and p B are the coordinates of the two joint points respectively.

[0026] Step 2.2: Use a bidirectional matching mechanism to ensure the uniqueness and accuracy of the connection when connecting joints, and eliminate misconnections caused by occlusion. Specifically, let V = { v i | i =1,…,N}, V represents the set of all joint points (such as hands, elbows, shoulders, etc.) detected in the image, It is i The coordinates or features of the joint points. The PAF vector field is L c ( c is the limb type), indicating the limb type c (e.g. "left forearm"). It is a vector field where the direction of each point points to the direction of the limb and the intensity indicates that the point belongs to the limb. c Confidence of the joint point. Define the joint point connection cost function:

[0027] .

[0028] Among them, p( u )=(1-u ) p i + up j For joint points v i (Its coordinates are )arrive v j (Its coordinates are ), d ij = p j - p i ; u is a normalized path parameter with a value range of [0,1], indicating that ( )arrive ( ) on the path; the joint connection cost function Used to measure joints and Are they part of the same limb? c The lower the cost, the higher the connection possibility. The calculation method is along the path Integral PAF vector field With unit direction vector The dot product of .

[0029] Use the Hungarian algorithm to solve the minimum cost matching and add constraints: .

[0030] Among them, the constraint is to limit the limb length to a reasonable range ( and are preset minimum and maximum lengths) to avoid accidentally connecting distant joints.

[0031] Step 2.3: Based on the confidence threshold of the PAF vector field, filter the valid joint points and generate a complete skeleton topology structure.

[0032] Step 3: Based on the coordinates of the skeleton key points, construct vectors and reference systems, and use geometric algorithms to calculate joint angles, including: Step 3.1. Calculate the joint angle formula based on the two vectors formed by the three coordinate points: .

[0033] Step 3.2: Dynamic Time Warping (DTW) algorithm is used to analyze the trend of joint angle changes over time and identify abnormal movement patterns. The goal of the DTW algorithm is to minimize the distance between time series. Its formula is:

[0034] .

[0035] Among them, Q and C are two time series, and d is the Euclidean distance.

[0036] Step 4: Medical analysis and recommendation generation.

[0037] Compare joint angle data with standard parameters in the medical knowledge base to identify abnormal range of motion or pathological characteristics and generate a medical advice report.

[0038] Specific functions include: anomaly detection, which compares joint angles with the standard range to identify anomalies in joint flexion, abduction, or rotation angles. For example, in step 3.2, if the DTW dynamic time warping analyzes that the difference between the joint angle sequence Q and the standard action template C is greater than the clinical risk threshold, an alarm is triggered; personalized recommendations are generated based on the analysis results to generate rehabilitation training guidance, posture correction plans, or disease warning reports.

[0039] Step 5. Privacy protection.

[0040] The privacy protection module is used to blur the sensitive information in the image, and only the skeleton data is retained for analysis.

[0041] Step 5.1: Blur the face. Use the ROI blur technique to blur the face area in the image to meet medical data privacy standards such as HIPAA.

[0042] Step 5.2: Background replacement: Use image segmentation technology to replace the original background with a non-sensitive image.

[0043] Step 6: System optimization, including: Lightweight network structure: In the PAF vector field generation stage, a lightweight model (such as MobileNet) is used to achieve real-time computing on embedded devices.

[0044] Error compensation mechanism: Through the inverse kinematics (IK) algorithm, the joint angle deviation caused by image perspective distortion is reduced.

[0045] The embodiment of the present invention provides a human joint angle recognition system based on an improved OpenPose network, comprising: The image acquisition module is used to obtain RGB image data of the target human body.

[0046] The network improvement module is used to input the RGB image data of the target human body into the improved OpenPose network. The PAF vector field algorithm with an occlusion-aware layer added in the generation phase is used to obtain the PAF vector field connecting the human skeletal joints. The bidirectional matching strategy is used to eliminate the misconnection of joints caused by occlusion in the PAF vector field and generate human skeleton information.

[0047] The human joint angle recognition module is used to identify the human joint angles based on the human skeleton information using the inverse kinematics IK algorithm, and obtain the trend of human joint angle changes over time through the dynamic time warping DTW algorithm to obtain the human joint angles in the active state.

[0048] The present invention has the following beneficial effects: (1) High-precision skeleton extraction: Through the improved PAF vector field algorithm and bidirectional matching strategy, the misconnection rate in occlusion and multi-person scenes is reduced by more than 30%, significantly improving the accuracy of joint angle calculation.

[0049] (2) Dynamic anomaly detection capability: Combined with the DTW algorithm, it can effectively capture the timing deviation of joint movements, and the recognition accuracy of gait abnormalities and sports injuries is over 90%.

[0050] (3) Adaptability to medical scenarios: Through the deep integration of the medical knowledge base, accurate matching of joint function assessment and disease characteristics can be achieved, providing data support for clinical diagnosis.

[0051] (4) Privacy and security protection: Desensitization technology is used to retain only necessary skeleton information to meet medical data privacy compliance requirements.

[0052] A specific embodiment is as follows: This embodiment discloses a method for recognizing human joint angles based on an improved OpenPose network. The specific steps are as follows: 1. Knee joint rehabilitation training evaluation.

[0053] S1. Data acquisition and preprocessing: The patient performs a standard knee flexion movement. The RGB camera captures the video stream at 25fps and uses histogram equalization to eliminate the influence of uneven lighting.

[0054] S2. Skeleton extraction and angle calculation: The improved OpenPose network generates joint point heatmaps (such as Figure 2 As shown in Figure 3), the PAF vector field connects the hip, knee, and ankle joints through a bidirectional matching strategy.

[0055] The knee flexion angle is calculated according to the joint point coordinates. The formula is as follows: .

[0056] S3. Dynamic analysis and report generation: The DTW algorithm compares the patient's angle curve with the standard rehabilitation curve, detects insufficient or asymmetric movement range, and generates personalized rehabilitation recommendations.

[0057] 2. Scoliosis screening.

[0058] S1. Privacy protection processing: Blur the faces of people in classroom surveillance videos and only retain key points of the spine (such as the cervical, thoracic, and lumbar vertebrae).

[0059] S2. Spine angle calculation: Calculate the scoliosis angle based on the coordinates of the spine key points: .

[0060] If the angle exceeds the clinical threshold (>10°), the system marks the high-risk individual and generates an early warning report.

[0061] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. The method for recognizing human joint angles based on the improved OpenPose network is characterized by: The following steps are involved: Obtain RGB image data of the target human body; The RGB image data of the target human body is input into the improved OpenPose network. The PAF vector field algorithm with an occlusion-aware layer added in the generation phase is used to obtain the PAF vector field connecting the human skeletal joints. A bidirectional matching strategy is used to eliminate the misconnected joints caused by occlusion in the PAF vector field and generate the human skeleton information. According to the human skeleton information, the inverse kinematics (IK) algorithm is used to identify the human joint angles, and the dynamic time warping (DTW) algorithm is used to obtain the trend of human joint angle changes over time, and the human joint angles in the active state are obtained.

2. The method for recognizing human joint angles based on the improved OpenPose network according to claim 1, wherein: The method uses a PAF vector field algorithm with an occlusion perception layer added to the generation phase to obtain a PAF vector field connecting the joints of the human skeleton, and eliminates the misconnection of joints caused by occlusion in the PAF vector field through a two-way matching strategy to generate human skeleton information. The specific steps include: Based on the RGB image data of the target human body, a convolutional neural network (CNN) is used to generate a joint point heat map. The PAF vector field algorithm that adds an occlusion perception layer in the generation phase is used to obtain the PAF vector field connecting the human skeleton joints. The formula is: ； when( x , y ) in the connection area, = ;otherwise =0; p A and p B are the coordinates of the two joint points; The bidirectional matching strategy is used to eliminate the misconnection of joint points caused by occlusion in the PAF vector field. The formula is: Define the joint cost: ； Among them, the PAF vector field is L c , c is the limb type; p( u ) is the joint point v i arrive v j Path, joint points v i The coordinates are , joint points v j The coordinates are , d ij = p j - p i ; Get the minimum cost matching from all costs using the Hungarian algorithm and add the following constraints: ； in, is the preset minimum length, is the preset maximum length; The joint points are filtered according to the confidence threshold in the PAF vector field to obtain the human skeleton information.

3. The method for recognizing human joint angles based on the improved OpenPose network according to claim 1, wherein: After identifying the human body joint angles using the inverse kinematics IK algorithm, the method further includes: The angular deviation caused by image perspective distortion is corrected through the error compensation mechanism.

4. The method for recognizing human joint angles based on the improved OpenPose network according to claim 1, wherein: The method for recognizing human joint angles based on the improved OpenPose network further includes: Compare the trend of human joint angle changes over time with preset standard parameters, identify abnormal joint movement characteristics, and generate a recommendation report based on the identification results.

5. The method for recognizing human joint angles based on the improved OpenPose network according to claim 4, characterized in that: Before generating the recommendation report according to the identification result, the following steps are also included: The face and identity information in the RGB image data of the target human body are blurred, and the skeleton data is retained.

6. A human joint angle recognition system based on an improved OpenPose network, characterized in that: include: Image acquisition module, used to obtain RGB image data of the target human body; The network improvement module is used to input the RGB image data of the target human body into the improved OpenPose network. The PAF vector field algorithm with an occlusion-aware layer added to the generation phase is used to obtain the PAF vector field connecting the human skeletal joints. The bidirectional matching strategy is used to eliminate the misconnection of joints caused by occlusion in the PAF vector field and generate human skeleton information. The human joint angle recognition module is used to identify the human joint angles based on the human skeleton information using the inverse kinematics IK algorithm, and obtain the trend of human joint angle changes over time through the dynamic time warping DTW algorithm to obtain the human joint angles in the active state.

Images