An automated method for measuring joint range of motion based on Kinect

Abstract

The present invention belongs to the field of intelligent processing of medical information, and specifically relates to an automated joint range of motion measurement method based on Kinect. The Kinect sensor is used to obtain the time series data of human joint points, which is smoothed using Kalman filtering. The stable state search algorithm is used to obtain the data of the effective joint range of motion measurement time period, and the time series of the human joint range of motion is segmented to automatically select the key point information during the user's measurement process, thereby completing the effective measurement of the joint range of motion and excluding the interference of non-target actions. The present invention realizes for the first time the segmentation of the joint range of motion measurement time period based on Kinect data and the selection of effective joint range of motion measurement data, and has the advantages of being real-time, automatic, high-precision and easy to operate.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent processing of medical information, and particularly relates to an automated joint range of motion measurement method based on Kinect. Background Art

[0002] The measurement of the range of joint motion generally refers to examining and quantifying the angle between the initial position and the end position passed by a joint during rotation, and its result is one of the important indicators for identifying the disability level of a limb and evaluating the joint rehabilitation condition. Classified according to the instruments used in the measurement, it is mainly divided into traditional mechanical measurement, computer vision measurement, and electromagnetic measurement:

[0003] The method of traditional mechanical measurement is as follows: Measuring tools such as protractors and rulers are used to obtain quantitative results by measuring the motion arcs of the proximal and distal bones of the joint. This method requires a high level of professionalism from physicians, and the degree of automation and measurement efficiency are relatively low. The accuracy of the measurement results will be affected by factors such as the deformation of limb soft tissues.

[0004] The method of computer vision is as follows: The joint point trajectory information during movement is obtained by means of RGB data stream, depth image data stream, etc. By comparing the joint point coordinates and rotation angles at the start and end of the movement, the joint range of motion is obtained. For example, Antonio used the human joint point coordinates obtained by a Kinect sensor to measure the human joint angles and preliminarily verified its effectiveness in clinical practice. The advantages of this method are convenient operation and high degree of automation, and the results can be obtained quickly; the disadvantage is that the accuracy may be reduced or even misjudged when the joint points or marker points are self-occluded.

[0005] The method of electromagnetic measurement is as follows: During the movement of the joint point, the magnetic fields in the electromagnetic transmitter and the receiving sensor interact with each other to obtain six-dimensional data for positioning, and then an accurate joint angle measurement result is obtained. For example, K. Yamaura et al. used a three-dimensional electromagnetic sensor system (EMS) to verify the analysis of elbow joint range of motion. The advantage of this method is relatively high measurement accuracy, and the disadvantage is that the subject needs to wear a transmitter, the operation is more cumbersome, and there cannot be metal materials in the measurement environment.

[0006] At present, many studies have focused on the measurement of joint range of motion, but most of them are quantitative studies on the accuracy of the results. However, in the actual use process, when users perform the required operations, there are often some interfering actions before and after the movements, such as walking, shaking, side - stepping, waving hands, etc. If these time periods are not removed and the data analysis is directly carried out, it will lead to inaccurate measurement results. Qu Chang et al. from Nantong University disclosed a method for measuring the range of motion of upper limb joints based on Kinect sensors. By using Kinect sensors to capture the positions of each joint of the upper limb, record, calculate and feedback the information of joint range of motion, the measurement of joint range of motion is completed. This method studies and analyzes the measurement accuracy and clinical effectiveness of joint range of motion, but does not explore how to intercept the effective motion time period, select key time points, improve the operation portability and automation. Summary of the Invention

[0007] In view of the deficiencies of the above - mentioned prior art, the present invention proposes an automated method for measuring joint range of motion based on Kinect. By using a steady - state search algorithm to obtain data of the effective joint range of motion measurement time period, and through a key - point extraction algorithm to segment the time series of human joint range of motion, the key angle information during the user's measurement process is automatically selected, and the interference of non - target actions is excluded, making the measurement of joint range of motion more accurate and convenient.

[0008] The present invention is realized through the following technical solutions:

[0009] An automated method for measuring joint range of motion based on Kinect includes the following steps:

[0010] Step 1: A computer device acquires the RGBD data and the time - series data of joint - point coordinates collected by the Kinect sensor;

[0011] Step 2: Use Kalman filtering to smooth the time series of joint - point coordinates;

[0012] Step 3: Use a steady - state search algorithm to intercept data of the effective joint range of motion measurement time period from the time series after Kalman filtering smoothing in Step 2, and combine the user's body position to judge and detect the effective joint range of motion measurement time period, obtaining the joint range of motion sequence within the corresponding time period of the effective joint range of motion;

[0013] Step 4: Use a key - point extraction algorithm to extract key joint motion data points from the joint range of motion sequence obtained in Step 3;

[0014] Step 5: Calculate the target joint range of motion according to the key joint motion data points extracted in Step 4.

[0015] Furthermore, Step 1 includes the following steps:

[0016] Step 1.1: Configure the Azure Kinect for Windows SDK on a computer with a Windows operating system, including the sensor driver and interface program; the hardware needs to connect and fix the Azure Kinect. The user should be 0.5 - 3 meters away from the camera, and it is required to stably and accurately identify the joint points needed for measurement.

[0017] Step 1.2: The user selects the measurement items through the interface on the computer. The measurement items include the measurement of the range of motion of the shoulder joint, elbow joint, and hip joint, specifically involving: flexion / extension of the left and right shoulder joints, internal rotation / external abduction of the left and right shoulder joints, flexion / extension of the left and right elbow joints, internal rotation / external rotation of the left and right hip joints in the sitting position.

[0018] Step 1.3: Under the selected item, the user takes different postures and conducts tests according to the interface prompts. The software will record the user data and archive it in the csv format.

[0019] Furthermore, Step 3 includes the following steps:

[0020] Step 3.1: According to the part to be tested, select the stable joints of the tested person among the joint coordinate points after Kalman filtering processing; the stable joints are the joint points that remain stationary during the measurement process.

[0021] Step 3.2: Use the joint point coordinates corresponding to the stable joints to calculate the curve of the angle change of the stable joints.

[0022] Step 3.3: Perform mean filtering on the curve of the angle change of the stable joints, calculate the difference of the filtered curve of the angle change of the stable joints to obtain a difference sequence; select the longest continuous sequence less than the preset threshold in the difference sequence.

[0023] Step 3.4: Calculate the mean and variance of the curve of the angle of the stable joints corresponding to the longest continuous sequence. Use 2 times the standard deviation as the threshold to screen out the longest continuous time point sequence, which is the data of the effective joint range of motion measurement time period, and the joint range of motion sequence within the corresponding time period is the effective joint range of motion.

[0024] Furthermore, Step 4 includes the following steps:

[0025] Step 4.1: Traverse the importance of each point in the joint range of motion sequence, screen out the points with higher importance, and arrange them in descending order of importance to obtain the joint range of motion sequence with importance; the importance is defined by the influence of the data points on the shape of the time series; the data points that have a greater impact on the overall shape of the time series are considered more important.

[0026] Step 4.2: Calculate the trend between adjacent points in the joint range of motion sequence with importance. If the trends are close, they are merged.

[0027] Step 4.3: Find the turning points among the unmerged points, and use this point as a candidate key point in the joint range of motion angle sequence, and add it to the key joint motion sequence;

[0028] Furthermore, step 4 further includes calculating the points with the top 25% importance according to the calculation platform example before calculating the trend between adjacent points.

[0029] Further, step 5 includes the following steps:

[0030] Divide the key points selected in step 4.3 into the end point of joint motion and the starting point of joint motion, and the difference between the average values of the end point of joint motion and the starting point of joint motion is the target joint range of motion.

[0031] An automated joint range of motion measurement device based on Kinect includes a Kinect sensor for collecting RGBD data and human joint point coordinate data; a computer for receiving RGBD data and human joint point coordinate data, and the computer is programmed to execute the above-mentioned automated joint range of motion measurement steps based on Kinect.

[0032] The present invention realizes the automated measurement of joint range of motion based on the Kinect sensor. First, it obtains the RGBD data and joint point data collected by the Kinect sensor, suppresses the noise in the Kinect human joint point tracking algorithm by using linear Kalman filtering, obtains the data of the effective joint range of motion measurement time period by using the steady state search algorithm, extracts the key joint motion data points by using the time series segmentation algorithm based on important point perception and trend change, and automatically completes the effective measurement of the joint range of motion.

[0033] Compared with the existing joint range of motion measurement methods, the present invention realizes for the first time the segmentation of the joint range of motion measurement time period based on Kinect data and the selection of effective joint range of motion measurement data. The operation process is simpler, the requirement for professional skills is lower, the influence of soft tissue on the goniometer is avoided, and at the same time, the influence of non-measurement actions on the joint range of motion evaluation can be automatically eliminated; compared with electromagnetic tracking and other methods, the present invention uses simple equipment, has low requirements for the site, and the subject does not need to wear additional items. Description of the Drawings

[0034] Figure 1 It is the system construction diagram of the present invention;

[0035] Figure 2 The algorithm flow chart of the present invention Detailed Embodiment

[0036] The present invention will be described in detail below with reference to the drawings and embodiments.

[0037] An automated joint range of motion measurement method based on Kinect provided in this embodiment includes the following steps:

[0038] Step 1: The computer device obtains the RGBD data and human joint point coordinate data collected by the Kinect sensor.

[0039] Step 1.1: Configure the Azure Kinect for windows SDK on a Windows 10 system computer, including the driver and interface program of the sensor; the hardware needs to connect and fix the Azure Kinect, and the user is required to be 0.5 - 3 meters away from the camera, and it is required to stably and accurately identify the joint points required for measurement.

[0040] Step 1.2: Use the cross-section on the computer to select the measurement items. The measurement items include the measurement of the range of motion of the shoulder joint, elbow joint, and hip joint, specifically involving: flexion / extension of the left and right shoulder joints, internal rotation / external abduction of the left and right shoulder joints, flexion / extension of the left and right elbow joints, internal rotation / external rotation of the left and right hip joints in the sitting position.

[0041] Step 1.3: After the item is selected, the user takes different postures and conducts tests according to the interface prompts, and the software will record the user data and archive it in the csv format.

[0042] During the joint range of motion test, to accurately measure the range of motion of upper or lower limb movements, it is divided into two situations: standing position and sitting position according to the measurement items.

[0043] Obtain the left and right shoulder joint point coordinates P0 / P1, left and right hand joint point coordinates P2 / P3, left and right knee joint point coordinates P4 / P5, and pelvis joint point coordinate P6 through the Kinect. When the user is ready to test the shoulder joint extension / flexion, adduction / abduction, it is required to first meet the standing position; when the user is ready to test the hip joint internal rotation / external rotation and elbow joint range of motion, it is required to meet the sitting position. The coordinate of each point is (x, y, z), where x is the horizontal coordinate, y is the vertical coordinate, and z is the depth coordinate.

[0044] During the standing position preparation, the relationship between the user's joint point coordinates needs to satisfy:

[0045] Max(P 0z ,P 1z ,P 2z ,P 3z ,P 4z ,P 5z ,P 6z ) - Min(P 0z ,P 1z ,P 2z ,P 3z ,P 4z,P 5z ,P 6z ) ≤ L1

[0046] Among them, P kz is the coordinate value of joint point k on the z-axis, and L1 is the threshold for judging whether the user is standing facing the device. According to experience, L1 is taken as 0.2.

[0047] If the coordinate relationship of the user's joint points meets the above requirements after 3s, the joint range of motion measurement can be entered; otherwise, the user is prompted to stand facing the camera with both hands hanging naturally.

[0048] During the sitting preparation period, the coordinate relationship of the user's joint points needs to simultaneously meet:

[0049] Max(P 0z , P 1z , P 2z , P 3z , P 6z ) - Min(P 0z , P 1z , P 2z , P 3z , P 6z ) ≤ L2

[0050] and

[0051] Max(P 4y , P 5y , P 6y ) - Min(P 4y , P 5y , P 6y ) ≤ L3

[0052] Among them, P ky is the coordinate value of joint point k on the y-axis. L2 and L3 are the thresholds for judging whether the user is sitting facing the device. According to experience, L2 is taken as 0.2 and L3 is taken as 0.3.

[0053] If the coordinate relationship of the user's joint points meets the above requirements after 3s, the joint range of motion measurement can be entered; otherwise, the user is prompted to stand facing the camera with both hands hanging naturally.

[0054] After obtaining the RGBD data and human joint point coordinate data according to Step 1, in this embodiment, the target joint range of motion is calculated according to Steps 2 to 5, as Figure 2 shown.

[0055] Step 2: Use Kalman filtering to smooth the time series of joint point coordinates, reduce the noise jitter in the human joint point tracking algorithm of Kinect, and improve the accuracy of the joint point coordinates required for measurement. Specifically:

[0056] Step 2.1: For the joint points to be processed, define the system state vector X and the observation vector Z of the Kalman filter at time t as

[0057] X t =(x(t), y(t), z(t), v x (t), v y (t), v z (t))

[0058] Z t =(x(t), y(t), z(t))

[0059] where x(t), y(t), and z(t) are the three-dimensional coordinates of the joint points to be processed, and v x (t), v y (t), and v z (t) are the central velocities of the joint points. The system equation and the observation equation for establishing the Kalman filter are as follows:

[0060] X t+1 = AX t + CW t

[0061] Z t+1 = HX t + V t

[0062] where A is the state matrix, C is the drive matrix, and H represents the transformation matrix from the state to the measurement. W t is the process noise, and V t is the observation noise. It is assumed that both are independent and white noise with a normal distribution. In this environment, the control matrix B in the Kalman filter system equation is approximated as 0, so the term BU t is omitted.

[0063] Step 2.2: Substitute the state value of the system at time t defined in Step 2.1 into the equation model to predict the state value of the system at time t + 1.

[0064] Step 3: Obtain the joint range of motion sequence within the corresponding time period of the effective joint range of motion:

[0065] Step 3.1: According to the part to be tested, select the stable joints of the tested person from the joint coordinate points after Kalman filter processing; the stable joints are the joint points that remain stationary during the measurement.

[0066] Step 3.2: Use the joint point coordinates after Kalman filter processing to calculate the stable joint angle change curve If the stable joint point during measurement is the right knee, the i-th right knee joint angle value is Assume that at this time, the coordinates of the right hip P1 are (x1, y1, z1), the coordinates of the right ankle P2 are (x2, y2, z2), and the coordinates of the right knee P3 are (x3, y3, z3). The vector can be obtained as Vector Then the included angle θ i is equal to:

[0067]

[0068] Step 3.3: Perform mean filtering on the joint angle curve. M is the length of the mean filtering convolution kernel. According to experience, M is taken as 5.

[0069] Then the joint angle after mean filtering is

[0070]

[0071] To ensure that the sequence length of the joint angle before and after convolution remains unchanged, the original time series needs to be filled. The reverse filling algorithm is adopted, that is, the filled sequence is the reverse of the real sequence. θ ps is the joint point filled at the front end of the joint angle change curve, and θ pe is the joint point filled at the back end of the joint angle change curve.

[0072]

[0073]

[0074] Differentiate the filtered joint angle change curve to obtain the difference sequence [d1,…,d n-1 . Set the threshold ε, and select the longest continuous sequence L max [d s ,d e in the difference sequence, where the value is less than ε. The data of this sequence is the data of the effective joint range of motion measurement time period; in this embodiment, the threshold ε is selected as 3 here.

[0075] Step 3.4: Calculate the mean and variance of the original stable joint change curve corresponding to L max The calculation method is as follows:

[0076]

[0077]

[0078] Taking [μ - 2σ, μ + 2σ] as the threshold, select the longest continuous time point sequence [Strat,…,End] of θ n ∈[μ - 2σ, μ + 2σ] in [θ1…θ i . If the target joint point angle curve is [a1…a n, then [a Start …a End is the target stable joint range of motion curve.

[0079] Step 4: Use the key point extraction algorithm to extract key joint motion data points from the joint range of motion sequence obtained in Step 3. Specifically, it includes the following steps:

[0080] Step 4.1: Calculate the importance of the stable joint range of motion curve. The importance of a data point is defined by its influence on the shape of the time series. Data points that have a greater impact on the overall shape of the time series are considered more important. The specific calculation pseudocode is as follows:

[0081]

[0082] The distance in the above algorithm is the vertical distance. Suppose there are three points, P1=(x1, y1), P2=(x2, y2), P3=(x3, y3). The calculation method of the vertical distance d between point P3 and points P1, P2 is as follows:

[0083]

[0084]

[0085] y c = sx c - sx2 + y2

[0086]

[0087] where s is the slope of the straight line P1P2, and P c =(x c , y c ) is the projection of P3 on the straight line P1P2.

[0088] Step 4.2: Select key joint motion points according to the calculation platform example. In this algorithm, the points L with the top 25% importance are selected according to experience 0.25 , and then through the trend change, the points that can accurately represent the maximum joint range of motion are selected. The specific calculation method is as follows:

[0089] Calculate the mean change of each point in L 0.25 . For point x:

[0090]

[0091]

[0092] ms = |m x1 - m x2 |

[0093] where m x1 is the forward mean of point x, and m x2 is the backward mean of point x. If the difference ms between the forward mean and the backward mean is greater than 0.5, it is considered that point x is in an upward or downward trend, and then point x is merged with its adjacent points. If ms is less than 0.5, no merging is performed.

[0094] Step 4.3. Select the turning points in L 0.25 . If the angular difference ΔD (i-1,i) between point i - 1 and point i and the angular difference ΔD (i,i+1) between point i + 1 and point i have different signs, it indicates that point i is a turning point:

[0095] ΔD (i-1,i) = L 0.25 [i] - L 0.25 [i - 1]

[0096] ΔD (i,i+1) = L 0.25 [i + 1] - L 0.25 [i]

[0097]

[0098] If a certain point is a turning point and has not been merged, then this point is an important point in the joint range of motion angle sequence, and it is used as a candidate point for joint motion measurement and added to the candidate point sequence L ca . Step 5. Target joint range of motion calculation

[0099] Classify the key joint motion sequences L ca selected in the above steps. Points with values greater than the mean of L ca are used as the end points of the joint range of motion and placed in the list L max . Calculate the list length l1; points with values less than the mean of L ca are used as the starting points of the joint range of motion and placed in the list L min . Subtract the mean of the starting points of the joint range of motion from the mean of the end points of the joint range of motion, and then the accurate measurement value of the target joint range of motion can be obtained:

[0100]

[0101] This embodiment also provides an automated joint range of motion measurement device based on Kinect, as Figure 1 shown, including a Kinect sensor for collecting RGBD data and human joint point coordinate data; a computer for receiving RGBD data and human joint point coordinate data, and the computer is programmed to execute the above-mentioned automated joint range of motion measurement steps based on Kinect.

Claims

1. An automated joint range of motion measurement method based on Kinect, characterized in that, It includes the following steps: Step 1: The computer device acquires the RGBD data and joint coordinates collected by the Kinect sensor; Step 2: Use Kalman filtering to smooth the time series of joint coordinates; Step 3: Use the steady state search algorithm to intercept the data of the effective joint range of motion measurement time period from the time series after Kalman filtering smoothing in Step 2, and combine the user's body position judgment to detect the effective joint range of motion measurement time period, and obtain the joint range of motion sequence corresponding to the effective joint range of motion measurement time period; The specific process includes: Step 3.1: Select the stable joints of the test subject according to the test part; the stable joints are the joint points that remain stationary during the measurement; Step 3.2: Use the joint coordinates corresponding to the stable joints to calculate the stable joint angle change curve; Step 3.3: Perform mean filtering on the stable joint angle change curve, calculate the difference of the filtered stable joint angle change curve to obtain a difference sequence; select the longest continuous sequence less than the preset threshold in the difference sequence; Step 3.4: Calculate the mean and variance of the stable joint angle change curve corresponding to the longest continuous sequence, and use 2 times the standard deviation as the threshold to screen out the longest continuous time point sequence, which is the data of the effective joint range of motion measurement time period, and the joint range of motion sequence corresponding to the measurement time period is the effective joint range of motion; Step 4: Use the key point extraction algorithm to extract the key joint motion data points from the joint range of motion sequence obtained in Step 3; the specific process includes: Step 4.1: Traverse the importance of each point in the joint range of motion sequence, screen out the points with higher importance, and arrange them from high to low according to importance to obtain the joint range of motion sequence with importance; the importance is defined by the influence of the data point on the shape of the time series; the data point that has a greater impact on the overall shape of the time series is considered more important; Step 4.2: Calculate the trend between adjacent points in the joint range of motion sequence with importance, and merge them if the trends are close; Step 4.3: Find the turning points among the unmerged points, and use the turning points as the key joint motion data points in the joint range of motion sequence and add them to the key joint motion sequence; Step 5: Calculate the target joint range of motion according to the key joint motion data points extracted in Step 4; the specific process includes: Divide the key joint motion data points selected in Step 4.3 into the joint motion end point and the joint motion start point, and the difference between the average values of the joint motion end point and the joint motion start point is the target joint range of motion; among them, the joint motion end point is the point where the data is greater than the average value of the key joint motion sequence, and the joint motion start point is the point where the data is less than the average value of the key joint motion sequence.

2. The automated joint range of motion measurement method based on Kinect according to claim 1, wherein, Step 1 includes the following steps: Step 1.1: Configure the Azure Kinect for Windows SDK on a computer with a Windows operating system, including the driver and interface program of the sensor; the hardware needs to connect and fix the Azure Kinect, and the user is 0.5 - 3 meters away from the camera, and it is required to stably and accurately identify the joint points required for measurement; Step 1.2: The user selects measurement items through the interface on the computer. The measurement items include the measurement of the range of motion of the shoulder joint, elbow joint, and hip joint, specifically involving: flexion / extension of the left and right shoulder joints, internal rotation / external abduction of the left and right shoulder joints, flexion / extension of the left and right elbow joints, internal rotation / external rotation of the left and right hip joints in the sitting position; Step 1.3: Under the selected item, the user takes different postures and conducts tests according to the interface prompts. The software will record the user data and archive it in the csv format.

3. The automated joint range of motion measurement method based on Kinect according to claim 1, characterized in that: Step 4 also includes calculating the points with the top 25% importance according to the calculation platform example before calculating the trend between adjacent points.

Images