A Deep Learning-Based Kinematic Analysis System and Method for Human Sitting-Standing Process

Abstract

This invention discloses a deep learning-based kinematic analysis system and method for the human sitting-to-standing process, belonging to the field of motion analysis technology. It solves the technical problems of high testing costs, low accuracy, and low stability in existing STS (Seated-to-Stand) motion testing technologies, which hinder the effective promotion of STS motion testing. Based on deep learning technology, this invention automatically tracks human joint nodes and achieves fully automated analysis of multi-dimensional, refined motion characteristics for various human movements from sitting to standing. It can comprehensively evaluate the range of motion, efficiency, stability, and coordination of human movement. Furthermore, this invention only requires video acquisition equipment to achieve comprehensive kinematic analysis of STS motion, solving the problems of expensive, space-consuming, and time-consuming motion capture systems using motion sensors and multi-camera motion capture systems with reflective markers. This significantly reduces the operational difficulty and testing cost of STS kinematic analysis.

Description

Technical Field

[0001] This invention belongs to the field of motion analysis and relates to kinematic analysis technology for the human sitting-standing process, specifically a deep learning-based kinematic analysis system and method for the human sitting-standing process. Background Technology

[0002] Seated-to-stand transition (STS) is a fundamental task frequently used in daily life, with non-disabled adults performing approximately 60 STS repetitions daily. From a biomechanical perspective, STS can be defined as the transitional movement from a seated to an upright standing posture, requiring the horizontal and vertical shift of the body's center of gravity from a stable position to a less stable one using the extended lower limbs. Therefore, to better perform STS, the body needs coordinated and symmetrical cooperation between the left and right sides of the body, as well as stable and fluid collaboration between the upper and lower limbs.

[0003] Existing methods for evaluating STS (Sports Therapy for Movement) mainly include: subjective evaluation, sensor methods, and methods using multiple cameras and multiple reflective markers. Subjective evaluation methods primarily rely on scales and scoring, based on visual observation of the STS process, providing a subjective score to the test subject. This method suffers from low accuracy, poor stability, and is highly susceptible to subjective bias. Sensor methods capture kinematic and dynamic data of the test subject's STS process by having them wear sensors. However, these devices are cumbersome to wear and place the test subject in a weight-bearing posture, restricting and interfering with normal STS movement, leading to unavoidable errors in the test results compared to normal physiological conditions. Methods using multiple cameras and multiple reflective markers involve attaching reflective markers to various joint points and analyzing the test subject's motor function using a photoelectric motion capture system. However, this method requires a large experimental area, involves complex joint point positioning, suffers from subjective errors, and may be expensive or inaccurate due to marker obstruction during movement. Therefore, there is an urgent need to find a low-cost, convenient, and highly accurate analytical method to address the difficulty in effectively promoting STS movement testing. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art; to this end, this invention proposes a deep learning-based human sitting-standing kinematic analysis system and method to solve the technical problem that the existing technology has high testing costs, low testing accuracy and testing stability in the STS motion testing process, which makes it difficult to effectively promote STS motion testing.

[0005] To achieve the above objectives, a first aspect of the present invention provides a deep learning-based human sitting-standing kinematic analysis system, including a central analysis module and a data acquisition module connected thereto;

[0006] STS motion videos are acquired using a video acquisition device connected to the data acquisition module, and preliminary analysis of the STS motion videos is performed. At the same time, the set joint nodes and torso segments are acquired through the data acquisition module. The preliminary analysis includes format analysis or content analysis.

[0007] The central analysis module combines the set joint nodes and trunk segments to analyze the received STS motion video, obtain the stage duration of each motion phase of a single STS motion, and the motion characteristics of key segments; among which, motion characteristics include angle change, curvature, distance, displacement, velocity or instantaneous velocity variance;

[0008] The central analysis module combines the motion characteristics of key segments to obtain STS motion analysis results.

[0009] Preferably, the central analysis module is communicatively and / or electrically connected to the data acquisition module and the smart terminal, respectively; wherein the smart terminal includes a mobile phone or a computer, used to display the analysis process and analysis results;

[0010] The data acquisition module is communicatively and / or electrically connected to several video acquisition devices; wherein, the video acquisition devices acquire STS motion videos from multiple angles.

[0011] Preferably, the joint nodes include the head, eyes, neck, shoulders, elbows, hands, spine, hips, knees, toes, and heels; the trunk segments include head-neck, neck-shoulder, or shoulder-elbow.

[0012] Key segments include the neck, shoulder, hip, knee, ankle, head and neck, and trunk.

[0013] Preferably, the motion phase includes:

[0014] Phase 1: From the start of the movement to the moment when the hip joint angle reaches its minimum;

[0015] Second stage: from the moment when the hip joint angle reaches its minimum to the moment when the knee joint lateral movement reaches its maximum;

[0016] The third stage: from the moment when the lateral movement of the knee joint reaches its maximum to the moment when the hip joint angle reaches its maximum;

[0017] Phase 4: From the moment the hip joint angle reaches its maximum until the test subject reaches a stable state.

[0018] Preferably, calculating the angular change of the torso segment includes:

[0019] During any phase of movement or a single STS movement, a reference vector is constructed pointing from the hip center recognition point to the neck recognition point.

[0020] The difference between the angle between this reference vector and the horizontal direction during the test is taken as the tilt angle of the torso segment.

[0021] A second aspect of the present invention provides a deep learning-based kinematic analysis method for the human sitting-standing process, comprising:

[0022] Select a test subject and match a chair to the test subject; the chair height should be the same as the test subject's knee height.

[0023] The test subject performed STS motion using a matched chair, and the STS motion video was captured by several video capture devices. The STS motion video was then preliminarily analyzed. The video capture devices were specifically cameras.

[0024] By combining the set joint nodes and trunk segments to analyze STS motion videos, the duration of each motion phase of a single STS motion and the motion characteristics of key segments are obtained, and the STS motion analysis results are obtained.

[0025] A third aspect of the present invention provides a deep learning-based human sitting-standing kinematics analysis device, comprising a processor and a storage medium; the storage medium is used to store operation instructions, and the processor executes the operation instructions to control the operation of the deep learning-based human sitting-standing kinematics analysis system.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] 1. This invention automatically tracks human joint nodes based on deep learning technology, and realizes fully automatic analysis of multi-dimensional and refined motion characteristics of the human body in various movements from sitting to standing. It can comprehensively evaluate the range of human motion function, motion efficiency, motion stability and motion coordination.

[0028] 2. This invention can achieve comprehensive kinematic analysis of STS motion using only video acquisition equipment, solving the problems of expensive, space-consuming, time-consuming, and labor-intensive motion capture systems using motion sensing devices and multi-camera motion capture systems with reflective markers, and greatly reducing the operational difficulty and testing cost of STS kinematic analysis. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the system principle of the present invention;

[0031] Figure 2 This is a schematic diagram illustrating the working steps of the present invention. Detailed Implementation

[0032] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Please see Figure 1 The first aspect of this invention provides a deep learning-based human sit-to-stand kinematic analysis system, including a central analysis module and a data acquisition module connected thereto. The system acquires STS motion videos via a video acquisition device connected to the data acquisition module and performs preliminary analysis on the STS motion videos. Simultaneously, the data acquisition module acquires set joint nodes and trunk segments. The central analysis module analyzes the received STS motion videos in conjunction with the set joint nodes and trunk segments to obtain the duration of each motion phase of a single STS motion and the motion characteristics of key segments. The central analysis module, combined with the motion characteristics of key segments, obtains the STS motion analysis results.

[0034] The human sitting-standing kinematics analysis system, method, and device based on deep learning of the present invention are not intended for the diagnosis or treatment of diseases.

[0035] The main equipment needed for STS motion video capture is a video capture device and a chair. The video capture device can be a mobile phone or a home camera, and the chair height needs to be adjusted to match the test subject's knee height. Ideally, the video capture device should be placed in front of and to the side of the test subject.

[0036] In addition to traditional STS motion, the content of STS motion videos (requiring a video frame rate of 30 frames per second) can also include videos of test subjects performing various STS variations, which can be used to evaluate the test subjects' motor function.

[0037] 1) Standard STS Exercise: At the start, the test subject places their hands crossed on their shoulders, keeping their knees at a 90° angle and their feet naturally separated shoulder-width apart and on the same horizontal line. Upon hearing the "start" command, the test subject stands up from the chair at their own pace (keeping both upper limbs still throughout the exercise). Once fully standing and stable, the test subject holds this position for five seconds. To avoid STS exercise abnormalities due to the test subject's unfamiliarity with the testing procedure, the test subject is allowed to perform three preparatory tests before the formal test, with at least a 2-minute interval between the preparatory tests and the formal test.

[0038] 2) Five STS exercises: The test subject is required to perform five STS exercises according to the traditional STS exercise requirements to assess whether the test subject has abnormal posture transitions and exercise endurance.

[0039] 3) 30-second STS exercise: The test subject is required to perform as many STS exercises as possible within 30 seconds. The test subject's motor function is evaluated by the number of times the test subject completes the exercise and the movement trajectory.

[0040] In this invention, STS motion videos captured by various video acquisition devices such as mobile phones, home cameras, and industrial cameras are automatically imported into the central nervous system analysis module. The central nervous system analysis module can automatically identify and track 19 joint nodes of the human torso (head, eyes, neck, shoulders, elbows, hands, spine, hips, knees, toes, and heels), and can also analyze human torso segments, such as head-neck, neck-shoulder, and shoulder-elbow.

[0041] The software analyzes each frame of the video in real time. For example, the size of each frame is determined by the camera resolution (1920×1080). During the analysis, a two-dimensional coordinate system is established with the 1920 value as the Y-axis and the 1080 value as the X-axis.

[0042] This invention divides a single STS movement into four stages—the first, second, third, and fourth—by analyzing the variation patterns of the joint nodes and the length and direction of each bone, obtaining the duration of each stage, and simultaneously identifying the movement characteristics of key segments (neck, shoulder, hip, knee, ankle, head and neck, and trunk).

[0043] Definitions of each phase of STS exercise: Phase 1: From the start of the movement to the moment when the hip joint angle reaches its minimum; Phase 2: From the moment when the hip joint angle reaches its minimum to the moment when the lateral movement of the knee joint reaches its maximum; Phase 3: From the moment when the lateral movement of the knee joint reaches its maximum to the moment when the hip joint angle reaches its maximum; Phase 4: From the moment when the hip joint angle reaches its maximum to the moment when the test subject reaches a stable state.

[0044] Definition of angle change (taking the hip joint as an example):

[0045] Definition of hip joint angle: In the recognition points, the angle formed by the lines connecting the hip recognition point as the vertex to the knee joint recognition point and the neck recognition point respectively;

[0046] Hip joint angle range: The maximum hip joint angle minus the minimum hip joint angle during a certain phase or the entire process of a single STS.

[0047] The relevant indicators for angle changes include the range of knee joint angle change, the range of ankle joint angle change, the maximum dorsiflexion range of the ankle joint, the knee joint angular velocity, and the hip joint angular velocity.

[0048] Displacement in this invention (taking the neck as an example):

[0049] In a certain stage or during a single STS process, let the coordinates of the neck recognition point at the start be (X1, Y1) and the coordinates of the neck recognition point at the end be (X2, Y2). The displacement is calculated using the following formula: √[(x1-x2)2+(y1-y2)2]. The related indicators include the neck X-axis displacement, neck Y-axis displacement, total neck displacement, shoulder joint X-axis displacement, shoulder joint Y-axis displacement, total shoulder joint displacement, hip joint X-axis displacement, hip joint Y-axis displacement, total hip joint displacement, knee joint X-axis displacement, knee joint Y-axis displacement, and total knee joint displacement.

[0050] The curvature in this invention (in the order of shoulder joint):

[0051] In a certain stage or during a single STS process, the maximum vertical distance between the actual path of the shoulder joint and the starting and ending points of the path is divided by the total shoulder displacement. Related indicators include shoulder curvature, neck curvature, hip and knee curvature in the first, second, and third stages and throughout the entire process.

[0052] The instantaneous velocity variance in this invention (taking the knee joint as an example):

[0053] Calculate the instantaneous velocity of the knee joint in each frame: the bit position of the knee joint between every two frames is reduced to the time represented by one frame; calculate the instantaneous velocity variance: based on the instantaneous velocities obtained in the steps, the variance is calculated using the following formula: S2 = [(m1-M) + (m2-M) + ... + (mn-M)] / n; where M is the average of m1, m2...mn, n represents the velocity sample size, and m represents the instantaneous velocity. Related indicators include the instantaneous velocity variances of the neck, shoulder, knee, and hip joints in the first, second, and third stages and the overall process.

[0054] This invention can also analyze and evaluate test subjects by combining trunk segments. Specifically, it calculates the center of gravity using parameters of the trunk segments and performs analysis and evaluation based on the center of gravity displacement, instantaneous velocity, etc., as shown in the following examples:

[0055] The center of gravity was calculated using trunk segment parameters, as shown in the table below.

[0056]

[0057] Center of mass of trunk segments:

[0058] The coordinates of the head, hip, knee, ankle, and toe joints are used to determine the proximal and distal coordinates of the head-neck, thigh, lower leg, and foot segments. Then, the centroid of each trunk segment is defined using the following formula:

[0059] Where xCM and y CM These are the centroid coordinates of each segment, x p and y p It is the near-end coordinate, x d and y d It is the remote coordinate, l p and l d This represents the percentage of proximal and distal segment lengths. The center of gravity is the weighted average of the centers of mass of the seven trunk segments, calculated using the following formula, where x... COG and y COG These are the centroid (COG) coordinates, x i and y i The coordinates of segment i are m. i M is the mass of the i-th segment, and M is the total mass.

[0060] In one embodiment, the central analysis module compares the motion characteristics of the aforementioned key segments with the corresponding normal range to determine where the test subject has an abnormality and provide reasonable suggestions to the test subject.

[0061] In another embodiment, evaluating test subjects based on a deep learning model includes:

[0062] Collect STS motion videos of normal and abnormal testers, and extract motion features of key segments; summarize the corresponding motion features according to the motion segments (which need to be numerically processed), and concatenate the motion features of the four summarized motion segments to generate model input data, and use the abnormal type of the tester as the corresponding model output data.

[0063] Deep learning models are trained using input and output data; these models are built upon either backpropagation (BP) neural networks or backpropagation (RBF) neural networks.

[0064] Then, following the method for constructing model input data, the relevant motion features of the test subject are integrated and input into the deep learning model to obtain the corresponding motion analysis results; then, the results are sent to the test subject's smart terminal, and warnings are issued in conjunction with the set program.

[0065] Please see Figure 2 The second aspect of the present invention provides a deep learning-based method for kinematic analysis of the human sitting-standing process, comprising: selecting a test subject and matching a chair to the test subject; the test subject performing STS movements using the matched chair, acquiring STS movement videos through several video acquisition devices, and performing preliminary analysis of the STS movement videos; combining the set joint nodes and trunk segments to analyze the STS movement videos, obtaining the stage duration of each movement stage and the movement characteristics of key segments of a single STS movement, and obtaining the STS movement analysis results.

[0066] A third aspect of the present invention provides a deep learning-based human sitting-standing kinematics analysis device, comprising a processor and a storage medium; the storage medium is used to store operation instructions, and the processor executes the operation instructions to control the operation of the deep learning-based human sitting-standing kinematics analysis system.

[0067] The processor and storage medium work together to control the central analysis module. Specifically, the processor executes the operation instructions of the storage medium to control the central analysis module. The central analysis module then acquires STS motion videos through the data acquisition module and analyzes the STS motion videos to obtain the STS motion test results of the test subject.

[0068] The data in the above formula are all calculated by removing the dimensions and taking the numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0069] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A deep learning-based kinematic analysis system for the human sitting-standing process, comprising a central analysis module and a data acquisition module connected thereto; characterized in that: The video acquisition device connected to the data acquisition module acquires seated-to-standing STS motion video and performs preliminary analysis on the STS motion video; at the same time, the data acquisition module acquires the set joint nodes and torso segments; the preliminary analysis includes format analysis or content analysis. The central analysis module combines the set joint nodes and trunk segments to analyze the received STS motion video, obtain the stage duration of each motion phase of a single STS motion, and the motion characteristics of key segments; among which, motion characteristics include angle change, curvature, distance, displacement, velocity or instantaneous velocity variance; The instantaneous velocity is the bit removal between every two frames of the key segment, expressed as the time represented by one frame; the variance of the instantaneous velocity is obtained according to the following formula: S2=[(m1-M)+(m2-M)+…+(mn-M)] / n; Where the average of m1, m2...mn is M, n represents the velocity sample size, and m represents the instantaneous velocity; The central nervous system analysis module combines the motion characteristics of key segments to obtain STS motion analysis results; The motion phase includes: Phase 1: From the start of the movement to the moment when the hip joint angle reaches its minimum; Second stage: from the moment when the hip joint angle reaches its minimum to the moment when the knee joint lateral movement reaches its maximum; The third stage: from the moment when the lateral movement of the knee joint reaches its maximum to the moment when the hip joint angle reaches its maximum; Phase 4: From the moment the hip joint angle reaches its maximum until the test subject reaches a stable state; Evaluation of test takers based on deep learning models includes: Collect STS motion videos of normal and abnormal testers, extract motion features of key segments; summarize the corresponding motion features according to the motion stage, and concatenate the motion features of the four motion stages to generate model input data, and use the abnormal type of the tester as the corresponding model output data. Deep learning models are trained using input and output data; these models are built upon either backpropagation (BP) neural networks or backpropagation (RBF) neural networks. The motion characteristics of the test subject are integrated according to the model input data construction method and input into the deep learning model to obtain the corresponding motion analysis results. The motion analysis results are sent to the test subject's smart terminal and warnings are issued in conjunction with the set program.

2. The deep learning-based human sitting-standing kinematic analysis system according to claim 1, characterized in that, The central analysis module is communicatively and / or electrically connected to the data acquisition module and the smart terminal, respectively; wherein the smart terminal includes a mobile phone or a computer, used to display the analysis process and analysis results; The data acquisition module is communicatively and / or electrically connected to several video acquisition devices; wherein, the video acquisition devices acquire STS motion videos from multiple angles.

3. The deep learning-based human sitting-standing kinematic analysis system according to claim 1, characterized in that, The joint nodes include the head, eyes, neck, shoulders, elbows, hands, spine, hips, knees, toes, and heels; the trunk segments include head-neck, neck-shoulder, or shoulder-elbow. Key segments include the neck, shoulder, hip, knee, ankle, head and neck, and trunk.

4. The deep learning-based human sitting-standing kinematic analysis system according to claim 3, characterized in that, Calculating the angular change of the torso segment includes: During any phase of movement or a single STS movement, a reference vector is constructed pointing from the hip center recognition point to the neck recognition point. The difference between the angle between this reference vector and the horizontal direction during the test is taken as the tilt angle of the torso segment.

5. A deep learning-based kinematic analysis method for the human sitting-standing process, operating the deep learning-based kinematic analysis system for the human sitting-standing process according to any one of claims 1 to 4, characterized in that, include: Select a test subject and match a chair to the test subject; the chair height should be the same as the test subject's knee height. The test subject performed STS motion using a matched chair, and the STS motion video was captured by several video capture devices. The STS motion video was then preliminarily analyzed. The video capture devices were specifically cameras. By combining the set joint nodes and trunk segments to analyze STS motion videos, the duration of each motion phase of a single STS motion and the motion characteristics of key segments are obtained, and the STS motion analysis results are obtained.

6. A deep learning-based kinematic analysis device for the human sitting-standing process, characterized in that, It includes a processor and a storage medium; the storage medium is used to store operation instructions, and the processor executes the operation instructions to control the operation of the deep learning-based human sitting-standing kinematic analysis system according to any one of claims 1 to 4.

Images