Method and system for 3D modeling of spine

Abstract

The present invention proposes a method and system for 3D modeling of the spine. First, the body tracking function of the Azure Kinect sensor is initialized in Unity software to capture joint coordinate data, and the bone length constraint method and unscented Kalman filter method are used to process the data to generate control points. Catmull-Rom spline interpolation is then used to obtain a fitted spinal curve. Secondly, a universal 3D spinal model is established in the scene and aligned to the fitted spinal curve to obtain a personalized 3D spinal model that is updated in real time. Finally, the Cobb angle in the coronal plane and the thoracic kyphosis angle and lumbar lordosis angle in the sagittal plane are automatically calculated based on the curvature analysis algorithm of the fitted spinal line. The advantages of the present invention are that the 3D spinal model can be visualized in real time and the spinal angle parameters can be accurately evaluated. There is no ionizing radiation, low cost, and it is convenient and easy to deploy. It is suitable for scoliosis screening and spinal rehabilitation training evaluation.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical auxiliary equipment, and particularly relates to a method and system for 3D modeling of the spine. Background Art

[0002] Scoliosis is a common spinal condition that primarily develops during adolescence and is known as adolescent idiopathic scoliosis. Clinically, scoliosis is assessed using X-rays. A professional physician manually locates the upper and lower vertebrae, calculates the angle between the upper and lower edge planes to obtain the Cobb angle, and determines the degree of scoliosis based on the size of the angle. The same calculation is used for the sagittal thoracic kyphosis and lumbar lordosis angles. This method relies entirely on the physician's subjective judgment and is susceptible to X-ray clarity and individual vertebral rotation. Furthermore, X-rays pose a risk of radiation damage. Scoliosis patients undergoing non-surgical treatments such as brace correction require X-rays every three to six months. Excessive radiation exposure can cause cancerous lesions in radiation-sensitive organs.

[0003] Therefore, many studies are currently using inclinometers, three-dimensional ultrasound probes, and three-dimensional scanners to detect and reconstruct scoliosis in patients, providing continuous and objective non-radioactive examinations. The Adam flexion test involves measuring trunk rotation angles using a handheld inclinometer. However, this method has high sensitivity and specificity, and the correlation coefficient between the measured value and the Cobb angle is not ideal. Three-dimensional ultrasound imaging involves manually positioning the ultrasound probe to complete three-dimensional spinal reconstruction and obtain the Cobb angle, but its measurements have only moderate correlation with those obtained by X-ray imaging. Both methods rely on manual operation, placing a significant burden on the examiner and prone to misjudgment due to fatigue. The DIERS spine analysis system uses a three-dimensional scanner to scan the three-dimensional contour of the back during movement, reconstruct a spinal model, and calculate the Cobb angle during movement, enabling harmless and rapid measurement. However, 3D scanners are expensive and require specialized site conditions and professional operation, making them unsuitable for everyday measurements.

[0004] Human pose estimation is a technology that captures human pose sequences and outputs joint coordinates to reflect pose characteristics. The Vicon system is capable of high-precision human pose estimation, but this method requires the deployment of multiple capture cameras and the wearing of capture clothing, which is expensive. In recent years, computer vision-based pose estimation methods have emerged one after another. VideoPose3D is a classic 3D pose estimation method. This method can capture human joints in real time through RGB cameras, but its accuracy is not high enough and it is not well open source, making it difficult to integrate and interact with other software. Azure Kinect is the third-generation depth sensor launched by Microsoft. It uses deep learning methods combined with RGBD images to achieve skeleton tracking. The accuracy has been greatly improved compared to the previous two generations. It has been identified as a portable and cost-effective device that can be used for gait assessment. All functions are encapsulated in the Azure Kinect SDK and Azure Kinect Body Tracking SDK, which can be easily developed.

[0005] Although Azure Kinect's skeletal tracking has achieved high accuracy, its depth camera is based on the time-of-flight (TOF) principle of light, which inevitably affects the environment's dust, light, and obstructions, causing joint data to exhibit white noise or spike jitter. Therefore, the resulting data needs to be processed using a smoothing filter algorithm. Common methods include median filtering, exponential smoothing, and Kalman filtering. The median filter eliminates data spikes, but is less effective at removing white noise. The exponential smoothing algorithm can achieve a smoothing effect, but has a significant delay. While the Kalman filter can provide relatively accurate estimates, it is limited to linear systems. Furthermore, these algorithms are limited to processing data for each joint point, ignoring the relationship between bone lengths at each joint point.

[0006] Because the 3D coordinates of joint points output by Azure Kinect exhibit nonlinear transformations, traditional Kalman filters cannot effectively model these nonlinear transformations. However, the Unscented Kalman Filter (UKF) can handle nonlinear state transitions and transformations, providing high-precision filtering of white noise. Using bone length as a constraint, the UKF process further filters the 3D coordinates of joint points that match their true positions, removing erroneous coordinates that cause spikes to achieve accurate estimation of joint points. Summary of the Invention

[0007] Therefore, in response to the actual improvement needs of the existing technology, the present invention proposes a method and system for 3D modeling of the spine, which uses Azure Kinect to capture the coordinates of some joints of the human body, and uses the UKF method and bone length constraint method to smooth the data of the joints. The Catmull-Rom spline is obtained by generating control points from the joints to represent the fitted human spine line. The 3D spine model is aligned on the spine line and the coronal Cobb angle and the sagittal thoracic kyphosis angle and lumbar lordosis angle are automatically calculated.

[0008] First, the body tracking function of the Azure Kinect sensor is initialized in Unity software to capture joint coordinate data. The data is then processed using the bone length constraint method and the unscented Kalman filter method to generate control points. Catmull-Rom spline interpolation is then used to obtain the fitted spinal curve. Secondly, a universal 3D spinal model is established in the scene and aligned to the fitted spinal curve to obtain a personalized 3D spinal model that is updated in real time. Finally, the curvature analysis algorithm of the fitted spinal line is used to automatically calculate the Cobb angle in the coronal plane and the thoracic kyphosis angle and lumbar lordosis angle in the sagittal plane. The advantages of the present invention are that it can visualize the 3D spinal model in real time and accurately evaluate spinal angle parameters. It does not emit any ionizing radiation, is low-cost, and is convenient and easy to deploy. It is suitable for scoliosis screening and spinal rehabilitation training assessment.

[0009] The technical solution adopted by the present invention to solve the technical problem is:

[0010] A method for 3D spine modeling is characterized by: first, initializing the body tracking function of the Azure Kinect sensor in Unity software to capture joint coordinate data, then processing the data using the bone length constraint method and unscented Kalman filter method to generate control points, and then using Catmull-Rom spline interpolation to obtain a fitted spinal curve; second, establishing a universal 3D spine model in the scene and registering it to the fitted spinal curve to obtain a personalized 3D spine model that is updated in real time; finally, performing curvature analysis on the spline projections in the coronal and sagittal planes to calculate the Cobb angle, thoracic kyphosis angle, and lumbar lordosis angle.

[0011] Furthermore, the Azure Kinect environment was configured in Unity and the body tracking function was initialized. The raw joint point data was queued and input according to the Azure Kinect data stream. Control points were generated based on the original joint points and the additional calculated joint points to accurately model and locate the vertebrae. The obtained joint point data was processed using the UKF and bone length constraint method to improve accuracy.

[0012] Furthermore, the specific method for generating control points is as follows: for the joint points automatically tracked by Azure Kinect body tracking, four points related to the spine are selected: Neck, Spine Chest, Spine Naval, and Pelvis as basic control points; Left Shoulder and Left Hip, Right Shoulder and Right Hip are connected respectively to represent the shoulder-hip connection vectors on the left and right sides of the body; then the midpoints of the connection lines at 25% and 65% of the left and right shoulder-hip connection vectors, as well as the midpoints of Left Shoulder and Right Shoulder are added as additional control points.

[0013] Furthermore, the specific method of generating control points is as follows: The bone length constraint method is specifically:

[0014] The joint point data is represented as Where k represents the number of frames captured by the sensor, i represents the joint number; the bone length between joints is expressed as the Euclidean distance between two points: in:

[0015]

[0016] Take the data of the joint points with smaller fluctuations to find the average value, and finally get L i,i+1 The calculated value is:

[0017]

[0018] L i,i+1 ∈L i,i+1 ±δ i,i+1

[0019] Since the sensor has different accuracy performance for different joints, different bone lengths correspond to different thresholds; different error thresholds δ are set for the lengths of the left and right shoulder joints and the lengths of the left and right hip joints. i,i+1 ,When the error is greater than the set threshold, data spike jitter is considered to occur, and the collected joint point position information is corrected through the filter.

[0020] Furthermore, the specific method for generating control points is as follows: the specific application method of the unscented Kalman filter method is as follows: the coordinate data of the joint points are filtered by the unscented Kalman filter method; the unscented Kalman filter method converts the state transfer matrix F and the measurement transfer matrix H in the Kalman filter into nonlinear functions f(x) and h(x), and calculates the mean and covariance of the random variables after the nonlinear function transformation by sampling method; the joint point data previously acquired by Azure Kinect and the data being captured are used to establish a state transition function, and data is obtained from the unscented Kalman filter algorithm module through iteration, and then the coordinates of each joint point are calculated using the observation function.

[0021] Furthermore, the specific process of using Catmull-Rom spline interpolation to obtain the fitted spinal curve is as follows: since each control point of the Catmull-Rom spline is located on the spline, when the spinal model is aligned to the Catmull-Rom spline, each vertebra can be moved and rotated more accurately according to the transformation of the control point; for the control points obtained after data processing, the Catmull-Rom interpolation algorithm is used to generate the Catmull-Rom spline function as the fitted spinal curve.

[0022] Furthermore, the specific process of establishing and registering the universal 3D spine model is as follows: using Catmull-Rom splines obtained by interpolation of control points to represent the center contour curve of the spine to perform three-dimensional reconstruction of the spinal vertebrae model;

[0023] After importing the generic vertebra model into Unity, a local coordinate system is established for each vertebra model, with the axis of the coordinate system positioned at the center of the vertebra;

[0024] After that, the first step is to align the vertebral model position:

[0025] Based on the vertebral model parameters, the proportional relationship of each vertebra relative to the entire spine length is calculated to obtain the percentage position of each vertebra on the fitted spine curve; the axis of each vertebral model is fixed to the point corresponding to the percentage position of the fitted spine line, and the Z-axis direction is set to the tangent direction of the point on the fitted spine line, and the Y-axis direction is set to the normal direction of the point on the fitted spine line and toward the camera plane, thereby aligning each vertebra to the fitted spine curve;

[0026] Next is the automatic scaling of the vertebrae model:

[0027] The length of the spine line corresponding to the established universal spine model is expressed as L normal , the length of the fitted spinal curve calculated for each test on different subjects is expressed as L, and the scaling factor of the vertebral model is expressed as:

[0028] ScaleFactor = L normal ÷L Formula 10

[0029] By applying the obtained scaling factor to the scaling component of each vertebral model, the vertebral model can be automatically scaled according to the fitted spinal curve characteristics of different people, and finally a personalized 3D spinal model is obtained.

[0030] Furthermore, the specific process of calculating the Cobb angle is:

[0031] Catmull-Rom spline is used to fit the spinal curve. Since the Catmull-Rom spline function has C2 continuity, that is, the second-order derivative function is continuous, the first-order derivative of the Catmull-Rom spline function is calculated to calculate the slope:

[0032]

[0033] Find the second-order derivative to calculate the extreme point:

[0034] P″(α)=(2P i-2 -5P i-1 +4P- i P i+1 )+(-3P i-2 +9P i-1 -9P+3P i+1 )α Formula 12

[0035] Obtain the extreme points (x1, z1), (x2, z2), (x3, z3) of the coronal plane slope, as well as the slopes k1, k2, and k3 corresponding to these points; then calculate the Cobb angle according to Formula 13:

[0036]

[0037] The greater of the two angles is defined as the Cobb angle of scoliosis, which is used to assess the degree of spinal curvature in the coronal plane.

[0038] Furthermore, the specific process of calculating the thoracic kyphosis angle and lumbar lordosis angle is as follows:

[0039] The thoracic kyphosis angle was defined as the angle between the tangent lines of the upper end of the fourth thoracic vertebra (T4) and the lower end of the twelfth thoracic vertebra (T12), and the lumbar lordosis angle was defined as the angle between the tangent lines of the upper end of the first lumbar vertebra (L1) and the upper end of the coccyx (S1).

[0040] By interpolating the fitted spinal curve on the sagittal plane, the slope k4 at the center of T4, the slope k5 at the center of T12, the slope k6 at the center of L1, and the slope k7 at the center of L5 in the established 3D spinal model are calculated to obtain TK and LL:

[0041]

[0042] And, a system for 3D modeling of the spine, characterized by: according to the method for 3D modeling of the spine as described above, comprising:

[0043] Data capture module, data processing and optimization module, and Unity scene modeling module;

[0044] The data capture module is used to configure the Azure Kinect usage environment in Unity, call built-in functions in the Azure Kinect SDK and Azure Kinect Body Tracking SDK, and input the joint point data captured by the sensor into Unity in the form of a queue;

[0045] The data optimization processing module is used to generate control points for interpolation fitting through joint points and additional points, and process the data of the control points using bone length constraint method and unscented Kalman filter algorithm in turn to reduce errors;

[0046] The Unity scene modeling module is used to perform Catmull-Rom spline interpolation based on the control points obtained from data processing to obtain a fitted spinal curve, and align the imported 3D universal spinal model with it to obtain a personalized 3D spinal model. The fitted spinal curve parameters are evaluated using an angle analysis algorithm to calculate the coronal Cobb angle and the sagittal TK angle and LL angle. Finally, the Unity scene is refreshed and continuously updated based on the data transmission from Azure Kinect.

[0047] The design of the present invention and its preferred embodiment can be applied to the fields of scoliosis screening and spinal rehabilitation training. It has the advantages of no ionizing radiation, high precision, automatic calculation of angle parameters and 3D spine visualization, low cost and easy use. It can replace the existing methods using inclinometers, three-dimensional ultrasound probes, three-dimensional scanners and the like.

[0048] Compared with existing technologies, it has the following advantages and uses:

[0049] 1. No ionizing radiation. Azure Kinect's depth camera measures depth information based on the time-of-flight (TOF) principle and does not emit any ionizing radiation to the body.

[0050] 2. High algorithm accuracy. Azure Kinect, a high-precision depth sensor, is used to capture joint points. Bone length constraints and unscented Kalman filtering are used to reduce data errors, ensuring accurate and reliable results.

[0051] 3. Automatically calculate angle parameters and achieve 3D spine visualization. Based on the data captured and processed, the coronal Cobb angle, sagittal TK angle, and LL angle are automatically calculated to assess the degree of scoliosis. The visualized 3D spine helps physicians explain spinal rehabilitation exercises to patients.

[0052] Low cost and easy to use. This invention requires only an Azure Kinect depth sensor, a camera tripod, and a standard laptop computer, making it less expensive than systems using 3D ultrasound probes, 3D scanners, and other devices. Furthermore, all that is required is installing Unity software on a laptop, importing the project package, setting up the tripod, and connecting the sensor to the computer operating system to automatically calculate angle parameters and achieve 3D spine visualization. This eliminates the need for tedious manual operations, making the product highly deployable. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0054] Figure 1 3D spine modeling and system flow chart for an embodiment of the present invention;

[0055] Figure 2 This is a flow chart of joint point detection and processing according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of Azure Kinect joint points according to an embodiment of the present invention;

[0057] Figure 4 Schematic diagram of the local coordinate system of the vertebral model according to an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of Cobb angle measurement according to an embodiment of the present invention; ((a) is a schematic diagram of the principle of the coronal Cobb angle; (b) is a schematic diagram of the Cobb angle measurement);

[0059] Figure 6 This is a schematic diagram of sagittal plane parameter calculation according to an embodiment of the present invention;

[0060] Figure 7 This is a block diagram of 3D spine modeling and system design according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] To make the features and advantages of this patent more clearly understood, the following embodiments are specifically described in detail as follows:

[0062] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art to which this application belongs.

[0063] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0064] The embodiment of the present invention integrates the body tracking function of Azure Kinect in Unity, uses the unscented Kalman filter method and the bone length constraint method to smooth the data of the joint points, uses Catmull-Rom spline interpolation to fit the spinal curve for the control points obtained from the joint points, establishes a universal 3D spinal model and aligns it to the fitted spinal line to form an automatically calibrated personalized 3D spinal model, and finally performs curvature analysis on the projection of the spline in the coronal and sagittal planes to calculate the Cobb angle, thoracic kyphosis angle, and lumbar lordosis angle. The flow chart is as follows Figure 1 shown.

[0065] 1. Joint point detection and processing

[0066] This example configures the Azure Kinect environment in Unity and initializes the body tracking function. The original joint point data is queued and input in the form of the Azure Kinect data stream. Control points are generated based on the original joint points and additional calculated joint points to accurately model and locate the vertebrae. The UKF and bone length constraint methods are used to process the obtained joint point data to improve accuracy. The flowchart is shown below. Figure 2 shown.

[0067] 1.1 Control point generation

[0068] For the joints automatically tracked by Azure Kinect body tracking, select four points related to the spine: Neck, Spine Chest, Spine Naval, and Pelvis as basic control points. Connect Left Shoulder and Left Hip, Right Shoulder and Right Hip respectively to represent the shoulder-hip connection vectors on the left and right sides of the body. Then use the midpoints of the connection lines at 25% and 65% of the left and right shoulder-hip connection vectors, as well as the midpoints of Left Shoulder and Right Shoulder as additional control points. Figure 3 As shown, the gray dotted marks in the middle are basic control points, and the gray non-dotted marks in the middle are additional control points, which work together to generate the spinal curve.

[0069] 1.2 Bone length constraint

[0070] When estimating human posture, rapid human motion or occlusion can cause spikes in joint data. In reality, human bones are constant in length, and this error can be reduced by limiting the distance between joints. Therefore, the present invention measures the lengths of the left and right shoulder joints and the left and right hip joints, continuously collecting multiple frames of data with minimal fluctuations and averaging them to build a bone length model.

[0071] Joint point data is represented as Where k represents the number of frames captured by the sensor, and i represents the joint number. The bone length between joints is expressed as the Euclidean distance between two points: in:

[0072]

[0073] Take the joint point data with smaller fluctuations to find the average value, and finally get L i,i+1 The calculated value is:

[0074]

[0075] L i,i+1 ∈L i,i+1 ±δ i,i+1

[0076] Since the sensor has different accuracy for different joints, different bone lengths correspond to different thresholds. Different error thresholds δ are set for the lengths of the left and right shoulder joints and the lengths of the left and right hip joints. i,i+1 ,When the error is greater than the set threshold, considering the data spike jitter, the filter will correct the collected joint point position information.

[0077] 1.3 Unscented Kalman Filter (UKF)

[0078] Since the three-dimensional coordinates of the joint points output by Azure Kinect undergo nonlinear transformations, traditional Kalman filtering cannot effectively model such nonlinear transformations. Therefore, an embodiment of the present invention uses the UKF method to filter the coordinate data of the joint points. UKF transforms the state transfer matrix F and the measurement transfer matrix H in the Kalman filter into nonlinear functions f(x) and h(x), and calculates the mean and covariance of the random variables after the nonlinear function transformation by sampling. UKF state update has two stages: prediction stage and update stage.

[0079] In the prediction phase, based on the state vector x at time t t and the covariance matrix P t , use the sampling algorithm to obtain 2n+1 sampling points {X i,i=0,1,...,2n} and their corresponding weights Where n represents the dimension of the state vector x. i}Substitute into f(x) to solve the corresponding function value {Y i}, so the predicted state vector and covariance matrix at time t+1 are expressed as:

[0080]

[0081] In the update phase, we need to first calculate the sample point {Y i}After the nonlinear function h(x) is passed, {Z i The sample mean and variance of} are:

[0082]

[0083] Furthermore, the residual vector and Kalman gain are calculated:

[0084]

[0085] Finally, the updated state vector x at time t+1 is t+1 and the covariance matrix P t+1 :

[0086]

[0087] The state transition function is established using the joint point data previously acquired by Azure Kinect and the data being captured. Data is obtained from the UKF system through iteration, and the coordinates of each joint point are calculated using the observation function.

[0088] 2. Interpolation fitting of the spine curve

[0089] The biggest difference between Catmull-Rom splines and Bezier splines is that each of their control points lies on the spline. This allows each vertebra to be more accurately moved and rotated based on the transformation of the control points when the spine model is aligned to the Catmull-Rom spline. Therefore, the Catmull-Rom interpolation algorithm is used to generate the Catmull-Rom spline function for the spinal curve fitting based on the control points obtained after data processing.

[0090] Catmull-Rom spline interpolation is essentially a piecewise polynomial interpolation. The four points of each segment are named P i-2 、P i-1 、P i 、P i+1 , then according to the coordinates of the four points in P i-1 、P iA cubic polynomial curve is constructed between them, and a complete continuous curve can be constructed by splicing each segment of the spline curve. The cubic polynomial of each segment is:

[0091] P(t)=C0+C1α+C2α 2 +C3α 3 Formula 7

[0092] Where: floating point coordinate α∈[0,1], spline function coefficients C0~C3 are obtained by the constraints of P(t) at the endpoints α=0 and α=1:

[0093]

[0094] Where T is called the shape factor. When T is set to 0.5, the interpolation fitting effect is the best. Then, the coefficients C0 to C3 are obtained from formula 8 and then inserted into formula 7 to obtain the interpolation point P. i-1 、P i The Catmull-Rom curve equation between:

[0095]

[0096] 3. Generic 3D Spine Model Construction and Registration

[0097] 3.1 General 3D spine model establishment

[0098] To more intuitively assess scoliosis, this embodiment uses Catmull-Rom splines, derived from interpolated control points, to represent the central contour curve of the spine for 3D reconstruction of the vertebral model. The spine is a complex structure, consisting of seven cervical vertebrae, 12 thoracic vertebrae, five lumbar vertebrae, the sacrum, and the coccyx. Research indicates that scoliosis primarily occurs in the thoracic (T1-T12) and lumbar (L1-L5) vertebrae.

[0099] Therefore, we pre-built 3D models of 17 vertebrae, including the thoracic and lumbar vertebrae, based on the geometric proportions of the vertebrae from the literature. These models were then imported into Unity software. The relative percentage height of each vertebra's center point relative to the spinal column after subtracting the intervertebral disc height is shown in Table 1.

[0100] Table 1 Relative percentage of vertebral height

[0101]

[0102] In reality, there are intervertebral disc spaces between the vertebrae of the human spine. Studies have shown that the height between all intervertebral discs accounts for approximately 25%-30% of the total spinal length. Therefore, intervertebral disc height should be considered when building a universal 3D spine model. According to literature statistics, the percentage of intervertebral disc height between the thoracic and lumbar vertebrae is shown in Table 2.

[0103] Table 2 Intervertebral disc height percentage

[0104]

[0105]

[0106] Taking into account the vertebral height and intervertebral disc height, the actual percentage height of each vertebra was calculated as shown in Table 3.

[0107] Table 3 Actual vertebral height percentage

[0108]

[0109] 3.2 General 3D Spine Model Registration

[0110] (1) Vertebral model position registration

[0111] After importing the general vertebra model into Unity, create the following for each vertebra model: Figure 4 The local coordinate system shown in FIG2 locates the axis of the coordinate system at the center of the vertebra (the junction of the vertebral body and the pedicle). The left is the sagittal view, and the right is the coronal view. The Z axis and Y axis are perpendicular and parallel to the upper plane of the vertebra, respectively.

[0112] In three-dimensional space, determining the coordinate axis of an object and the directions of its two coordinate axes can fix the object's position. Based on the vertebral model parameters in Tables 1 and 2, the proportional relationship of each vertebra relative to the entire spine length in Table 3 is calculated to obtain the percentage position of each vertebra on the fitted spine curve. The axis of each vertebral model is fixed to the point corresponding to the percentage position of the fitted spine line, and the Z-axis direction is set to the tangent direction of the point on the fitted spine line, and the Y-axis direction is set to the normal direction of the point on the fitted spine line and toward the camera plane, thereby aligning each vertebra to the fitted spine curve.

[0113] (2) Automatic scaling of vertebral models

[0114] The length of the spine line corresponding to the established universal spine model is expressed as L normal , the length of the fitted spinal curve calculated for each test on different subjects is expressed as L, and the scaling factor of the vertebral model is expressed as:

[0115] ScaleFactor = L normal ÷L formula 10

[0116] By applying the obtained scaling factor to the scaling component of each vertebral model, the vertebral model can be automatically scaled according to the fitted spinal curve characteristics of different people, and finally a personalized 3D spinal model is obtained.

[0117] 4. Automatically calculate coronal and sagittal plane angle parameters

[0118] 4.1 Coronal Cobb angle

[0119] The Cobb angle is the most commonly used indicator for determining the degree of scoliosis. It is defined as the maximum angle between the upper edge plane of the upper vertebra and the lower edge plane of the lower vertebra in the coronal plane of the human spine. Figure 5 The method of this embodiment calculates the Cobb angle based on the fitted spinal curve, which can automatically locate the position where the spinal line has the greatest curvature, while avoiding the influence of the traditional method caused by the abnormal rotation of a single vertebra.

[0120] Figure 5 (b) is a schematic diagram of the Cobb angle measurement of the present invention. Since the positions of the upper and lower vertebrae are the points with the largest inclination in the spinal curve, and their slopes are the extreme points of the slope of the curve, the extreme points of the slope can be obtained by calculating the points where the second-order derivative of the fitted spinal curve is zero, thereby locating the positions of the upper and lower vertebrae.

[0121] The present invention uses Catmull-Rom spline to fit the spinal curve. Since the Catmull-Rom spline function has C2 continuity, that is, the second-order derivative function is continuous, the slope can be calculated by taking the first-order derivative of the Catmull-Rom spline function according to formula 9:

[0122]

[0123] Find the second-order derivative to calculate the extreme point:

[0124] P″(α)=(2P i-2 -5P i-1 +4P- i P i+1 )+(-3P i-2 +9P i-1 -9P+3P i+1 )α Formula 12

[0125] Obtain the extreme points (x1, z1), (x2, z2), (x3, z3) of the coronal plane slope, as well as the slopes k1, k2, and k3 corresponding to these points. Calculate the Cobb angle according to formula 13:

[0126]

[0127] The greater of the two angles is defined as the Cobb angle of scoliosis, which is used to assess the degree of spinal curvature in the coronal plane.

[0128] 4.2 Sagittal plane parameters

[0129] The sagittal plane evaluation parameters of the present invention are thoracic kyphosis (TK) and lumbar lordosis (LL). Existing studies have shown that the two have a high correlation with the degree of scoliosis.

[0130] like Figure 6 As shown, the thoracic kyphosis angle is defined as the angle between the upper end of the fourth thoracic vertebra (T4) and the lower end of the twelfth thoracic vertebra (T12), and the lumbar lordosis angle is defined as the angle between the upper end of the first lumbar vertebra (L1) and the upper end of the coccyx (S1). The present invention interpolates and fits the spinal curve in the sagittal plane and calculates the slope k4 at the center point of T4, the slope k5 at the center point of T12, the slope k6 at the center point of L1, and the slope k7 at the center point of L5 in the established 3D spinal model to obtain TK and LL:

[0131]

[0132] 5. System Design

[0133] Based on the above method design, the system design provided by the embodiment of the present invention mainly includes three parts, namely data capture module, data processing optimization module and Unity scene modeling module. Figure 7 shown.

[0134] The data capture module configures the Azure Kinect usage environment in Unity, calls built-in functions in the Azure Kinect SDK and Azure Kinect Body Tracking SDK, and inputs the joint point data captured by the sensor into Unity in the form of a queue.

[0135] The data optimization processing module generates control points for interpolation fitting through joint points and additional points, and processes the data of the control points using the bone length constraint method and the unscented Kalman filter algorithm in turn to reduce errors.

[0136] The Unity scene modeling module performs Catmull-Rom spline interpolation based on the control points obtained from data processing to obtain a fitted spinal curve, and aligns the imported 3D universal spinal model with it to obtain a personalized 3D spinal model. The fitted spinal curve parameters are evaluated through an angle analysis algorithm, and the coronal Cobb angle and sagittal TK angle and LL angle are automatically calculated. Finally, the Unity scene is refreshed and continuously updated based on the data transmission of Azure Kinect.

[0137] 6. The specific usage process of this embodiment is as follows:

[0138] 1. Place the camera tripod horizontally at a height of 1.4 to 1.6 meters and install the Azure Kinect sensor.

[0139] 2. The subject stands upright and faces the sensor at a distance of 0.25 to 2.0 meters;

[0140] 3. Launch Unity software, open the 3D spine modeling and system project, and click the play button to run the system to monitor spine morphology and evaluate angle parameters in real time.

[0141] 4. Click the Pause button to export the data to the database and collect the subject's static data;

[0142] Click the play button again to end the system and complete the acquisition.

[0143] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0144] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0148] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of spinal 3D modeling methods and systems based on the inspiration of this patent. All equivalent changes and modifications made within the scope of the patent application of this invention should be covered by this patent.

Claims

1. A method for 3D modeling of a spine, characterized by: First, the Azure Kinect sensor's body tracking function was initialized in Unity software to capture joint coordinate data. Bone length constraints and unscented Kalman filtering were then used to process the data to generate control points. Catmull-Rom spline interpolation was then used to obtain the fitted spinal curve. Secondly, a universal 3D spine model is created in the scene and registered to the fitted spine curve, resulting in a personalized 3D spine model that is updated in real time. Finally, curvature analysis is performed on the projections of the spline in the coronal and sagittal planes to calculate the Cobb angle, thoracic kyphosis, and lumbar lordosis. Configure the Azure Kinect environment in Unity and initialize body tracking. Queue the raw joint data according to the Azure Kinect data stream. Generate control points based on the raw joints and additional calculated joints to accurately model and locate the vertebrae. Process the resulting joint data using the UKKF and bone length constraint methods to improve accuracy. The specific method for generating control points is as follows: For the joints automatically tracked by Azure Kinect body tracking, select four points related to the spine: Neck, Spine Chest, Spine Naval, and Pelvis as basic control points; connect the Left Shoulder and Left Hip, and the Right Shoulder and Right Hip respectively to represent the shoulder-hip connection vectors on the left and right sides of the body; then use the midpoints of the connecting lines at 25% and 65% of the left and right shoulder-hip connection vectors, as well as the midpoints of the Left Shoulder and Right Shoulder as additional control points; The specific method of generating control points is as follows: The bone length constraint method is specifically: Represent the joint point data as , where k represents the number of frames captured by the sensor, i represents the joint number; the bone length between joints is expressed as the Euclidean distance between two points: ,in: Formula 1 Take the data of the joint points with smaller fluctuations to find the average value, and finally get The calculated value is: Formula 2 Since the sensor has different accuracy performance for different joints, different bone lengths correspond to different thresholds; different error thresholds are set for the lengths of the left and right shoulder joints and the lengths of the left and right hip joints. ,When the error is greater than the set threshold, data spike jitter is considered to occur, and the collected joint point position information is corrected through the filter.

2. The method for 3D modeling of the spine according to claim 1, wherein: The specific method for generating control points is as follows: the specific application method of the unscented Kalman filter method is as follows: the coordinate data of the joint points are filtered by the unscented Kalman filter method; the unscented Kalman filter method transforms the state transfer matrix F and the measurement transfer matrix H in the Kalman filter into nonlinear functions f(x) and h(x), and calculates the mean and covariance of the random variables after the nonlinear function transformation by sampling method; the state transition function is established using the joint point data previously acquired by Azure Kinect and the data being captured, and data is obtained from the unscented Kalman filter algorithm module through iteration, and then the coordinates of each joint point are calculated using the observation function.

3. The method for 3D modeling of the spine according to claim 1, wherein: The specific process of using Catmull-Rom spline interpolation to obtain the fitted spinal curve is as follows: since each control point of the Catmull-Rom spline is located on the spline, when the spinal model is aligned to the Catmull-Rom spline, each vertebra can be moved and rotated more accurately according to the transformation of the control point; for the control points obtained after data processing, the Catmull-Rom interpolation algorithm is used to generate the Catmull-Rom spline function as the fitted spinal curve.

4. The method for 3D modeling of the spine according to claim 1, wherein: The specific process of establishing and registering a universal 3D spine model is as follows: Catmull-Rom splines obtained by interpolation of control points are used to represent the central contour curve of the spine to perform three-dimensional reconstruction of the spinal vertebrae model; After importing the generic vertebra model into Unity, a local coordinate system is established for each vertebra model, with the axis of the coordinate system positioned at the center of the vertebra; After that, the first step is to align the vertebral model position: Based on the vertebral model parameters, the proportional relationship of each vertebra relative to the entire spine length is calculated to obtain the percentage position of each vertebra on the fitted spine curve; the axis of each vertebral model is fixed to the point corresponding to the percentage position of the fitted spine line, and the Z-axis direction is set to the tangent direction of the point on the fitted spine line, and the Y-axis direction is set to the normal direction of the point on the fitted spine line and toward the camera plane, thereby aligning each vertebra to the fitted spine curve; Next is the automatic scaling of the vertebrae model: The length of the spine line corresponding to the established universal spine model is expressed as , the length of the fitted spinal curve calculated for each test of different subjects is expressed as , then the scaling factor of the vertebral model is expressed as: Formula 10 By applying the obtained scaling factor to the scaling component of each vertebral model, the vertebral model can be automatically scaled according to the fitted spinal curve characteristics of different people, and finally a personalized 3D spinal model is obtained.

5. The method for 3D modeling of the spine according to claim 1, wherein: The specific process of calculating the Cobb angle is: Catmull-Rom spline is used to fit the spinal curve. Since the Catmull-Rom spline function has C2 continuity, that is, the second-order derivative function is continuous, the first-order derivative of the Catmull-Rom spline function is calculated to calculate the slope: Formula 11 Find the second-order derivative to calculate the extreme point: Formula 12 Get the extreme point of the coronal slope 、 、 , and the slopes corresponding to these points 、 、 ; Then calculate the Cobb angle according to formula 13: Formula 13 The greater of the two angles is defined as the Cobb angle of scoliosis, which is used to assess the degree of spinal curvature in the coronal plane.

6. The method for 3D modeling of the spine according to claim 1, wherein: The specific process of calculating the thoracic kyphosis angle and lumbar lordosis angle is as follows: The thoracic kyphosis angle was defined as the angle between the tangent lines of the upper end of the fourth thoracic vertebra (T4) and the lower end of the twelfth thoracic vertebra (T12), and the lumbar lordosis angle was defined as the angle between the tangent lines of the upper end of the first lumbar vertebra (L1) and the upper end of the coccyx (S1); The slope of the T4 center point in the established 3D spine model was calculated by interpolating the fitted spinal curve in the sagittal plane. , the slope of the center point of T12 , the slope at the center of L1 , the slope at the center of L5 , we get TK and LL: Formula 14.

7. A system for 3D modeling of the spine, characterized by: The method for 3D modeling of a spine according to any one of claims 1 to 6, comprising: Data capture module, data processing and optimization module, and Unity scene modeling module; The data capture module is used to configure the Azure Kinect usage environment in Unity, call built-in functions in the Azure KinectSDK and Azure Kinect Body Tracking SDK, and input the joint point data captured by the sensor into Unity in the form of a queue; The data optimization processing module is used to generate control points for interpolation fitting through joint points and additional points, and process the data of the control points using bone length constraint method and unscented Kalman filter algorithm in turn to reduce errors; The Unity scene modeling module is used to perform Catmull-Rom spline interpolation based on the control points obtained from data processing to obtain a fitted spinal curve, and align the imported 3D universal spinal model with it to obtain a personalized 3D spinal model. The fitted spinal curve parameters are evaluated using an angle analysis algorithm to calculate the coronal Cobb angle and the sagittal TK angle and LL angle. Finally, the Unity scene is refreshed and continuously updated based on the data transmission from Azure Kinect.

Images