Rapid scoliosis screening method and device

Abstract

The invention discloses a scoliosis rapid screening method and device, and the method comprises the steps: S1, obtaining frame-by-frame images of a human spine through a visual sensor, and forming a human standing video sequence; s2, acquiring three-dimensional coordinate information of main skeleton key points of a human body; s3, analyzing the three-dimensional coordinate information of the main skeleton key points of the human body; s4, collecting human body front and back double-view-angle stoop motion videos through a color vision sensor; s5, carrying out frame-by-frame processing segmentation on the motion video sequence of human body stooping; s6, splicing the two-dimensional back contours extracted frame by frame to generate a complete back three-dimensional curved surface model; and S7, judging whether the scoliosis risk exists in the testee or not by combining the dynamic change indexes of the back contour features in the motion state, combining the back contour features of each frame extracted frame by frame and a plurality of indexes calculated according to the skeleton key points and inputting the indexes into the joint learning model.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a method and device for rapid screening of scoliosis. Background Art

[0002] Traditional scoliosis detection methods mainly include X-ray image analysis, etc. Although there are fast screening methods, such as using ATR protractors, they still rely on a lot of manual labor from doctors, making it difficult to carry out screening work on a large scale. Existing artificial intelligence screening methods mostly use 3D scanning methods or depth cameras to directly use depth information to build a three-dimensional human back model, but the cost of 3D scanning equipment is too high, and depth cameras mostly use depth information to directly generate a three-dimensional model of the human back, but the depth camera has serious hole problems and unstable depth values, which greatly affect the accuracy of direct three-dimensional reconstruction using depth cameras, and most of the direct three-dimensional reconstructions of depth cameras are based only on static point cloud data. The present invention is based on the back features of the human body's dynamic bending movements from the front and back perspectives and the dynamic back change features, and through the use of time series alignment, spline interpolation, optimization based on skeletal key points, dual-perspective matching and other methods after two-dimensional segmentation, it greatly avoids the hole fluctuation problem of direct three-dimensional reconstruction, and combines six indicators of front standing posture measurement, data-driven joint learning model, and a large number of physician-annotated data sets, which has significant advantages in many aspects. Summary of the Invention

[0003] The embodiments of the present invention provide a method and device for rapid screening of scoliosis for adolescents.

[0004] In order to achieve the above object, the present invention provides the following method:

[0005] A rapid scoliosis screening method, comprising:

[0006] S1: Use visual sensors to obtain frame-by-frame images of the human spine to form a human standing video sequence;

[0007] S2: Obtain the 3D coordinate information of the main skeletal key points of the human body. The 3D coordinate information of the skeletal key points is obtained by inputting the human standing video sequence into DF-POSE;

[0008] S3: Analyze the three-dimensional coordinate information of the key points of the main bones of the human body, and calculate indicators such as the coronal plane deflection angle of the left and right shoulders, the sagittal plane deflection angle of the left and right shoulders, the coronal plane deflection angle of the left and right hips, the sagittal plane deflection angle of the left and right hips, the center deflection angle, and the cervical lordosis angle based on the three-dimensional coordinate information of the key points of the bones;

[0009] S4: Using a color vision sensor to collect dual-view videos of a person bending over from both the front and back perspectives, a video sequence of a person bending over from both the front and back perspectives facing the color vision sensor is formed;

[0010] S5: Process and segment the motion video sequence of a person bending over frame by frame to obtain the two-dimensional back contour curve of the person in each frame, obtain the characteristics of the two-dimensional back contour curve of each frame, and the dynamic change characteristics of the back contour characteristics in the motion state;

[0011] S6: Use methods such as dual-view matching, temporal alignment, skeleton key point optimization, and spline interpolation to stitch the two-dimensional back contours extracted frame by frame to generate a complete three-dimensional surface model of the back;

[0012] S7: By combining the dynamic change indicators of the back contour features in the motion state, and combining the back contour features of each frame extracted frame by frame and multiple indicators calculated based on the key points of the skeleton, the joint learning model is input to determine whether the subject has the risk of scoliosis.

[0013] Furthermore, the DF-POSE uses an end-to-end model training method, including the following steps:

[0014] S11: The human standing video sequence is first input into the 2D posture detector, and the posture feature vector is generated by the 2D human posture detection method;

[0015] S12: Input the posture feature vector into the 3D posture detector and combine it with the SMPL model to generate the 3D coordinate information of the main skeleton key points of the human body.

[0016] Furthermore, the two-dimensional human posture detection method in step S11 includes the following steps:

[0017] S111: first adding an improved CBAM module to the improved ResNet-50 network structure of the two-dimensional posture detector to increase the degree of attention of the convolutional neural network to different feature channels and spatial positions;

[0018] S112: Adjust multiple basic blocks B1 and B2 in the ResNet-50 network to accelerate the model;

[0019] S113: The ResNet-50 network is divided into four parts. The first part contains one B1 and two B2s; the second part includes one B1 and three B2s; the third part includes one B1 and five B2s; the fourth part includes one B1 and two B2s. Ultimately, a two-dimensional posture vector for the human body is obtained, and finally the final three-dimensional human posture key point coordinates are obtained through a three-dimensional human posture detection model.

[0020] Furthermore, in step S3, the three-dimensional information of the key points of the main skeleton is analyzed to measure indicators such as the coronal plane deflection angle of the left and right shoulders, the sagittal plane deflection angle of the left and right shoulders, the coronal plane deflection angle of the left and right hips, the sagittal plane deflection angle of the left and right hips, the center deflection angle, and the cervical lordosis angle, including the following steps:

[0021] S31: After the model is detected, the skeleton key point information P is obtained. i =(x i ,y i ,z i ), nose P1=(x1,y1,z1), left eye P2=(x2,y2,z2), right eye P3=(x3,y3,z3), left ear P4=(x4,y4,z4), right ear P5=(x5,y5,z5), left shoulder P6=(x6,y6,z5), right shoulder P7=(x7,y7,z7), left elbow P8=(x8,y8,z8), right elbow P9=(x9,y9,z9), left wrist P 10 =(x 10 ,y 10 ,z 10 ), right wrist P 11 =(x 11 ,y 11 ,,z 11 ), left hip P 12 =(x 12 ,y 12 ,z 12 ), right hip P 13 =(x 13 ,y 13 ,z 13 ), left knee P 14 =(x 14 ,y 14 ,z 14 ), right knee P 15 =(x 15 ,y 15 ,z 15 ), left ankle P 16 =(x 16 ,y 16 ,z 16 ), right ankle P 17 =(x 17 ,y 17 ,z 17 );

[0022] S32: Select the left and right shoulder joints P6 and P7, then the left and right shoulder coronal plane deflection angles θ1 and sagittal plane deflection angles θ 11 It can be obtained by the following formula;

[0023]

[0024] S33: Select the left and right hip joint points P 12 , P 13 , the coronal plane deflection angle of the left and right hips can be obtained The sagittal plane deflection angle of the left and right hip is

[0025] S34: Select left shoulder P6, right shoulder P7, left hip P 12 , right hip P 13 The center rotation angle θ3 can be obtained by the following formula:

[0026]

[0027] S35: Select the nose key point P1, left shoulder P6, and right shoulder P7 key point information, and the cervical lordosis deflection angle θ4 can be obtained by the following formula:

[0028]

[0029] Furthermore, it also includes the angles θ1, θ 11 ,θ2,θ 22 , θ3, θ4 optimization, the specific steps include:

[0030] T1: fitting angle lines by polynomial regression method;

[0031] θ(t)=a0+a1t+a2t 2 +...+a n t n ;

[0032] And take the derivative to obtain the first-order angle curve, then the angular velocity k at a certain moment is i It can be expressed by the following formula:

[0033]

[0034] T2: The burrs in the angular velocity are removed by using the improved Kalman filter algorithm, using the improved Jacobian matrix calculation. When the state is updated, the modified extended Kalman filter is modified by iteratively correcting the gain and state estimate.

[0035] Step T2 specifically includes:

[0036] T21: Create state vector:

[0037]

[0038] f is the nonlinear state transfer function, u k is the control input, based on the physical relationship between angle and angular velocity:

[0039] X K|K-1 =f(X k-1 ,u k );

[0040] T22: Calculate the Jacobian matrix F of the state transfer function to the state variables k-1 :

[0041]

[0042] T23: Calculate the predicted state covariance, where Q k is the process noise covariance:

[0043]

[0044] T24: Calculate Kalman gain:

[0045]

[0046] Among them, P kk-1 is the predicted state covariance, H k is the observation matrix, R k is the observation noise covariance;

[0047] T25: Update initial status:

[0048] x k|k =x k|k-1 +K k (z k -h(x k|k-1 ));

[0049] Among them, z k is the observed value, h(x) is the nonlinear model;

[0050] T26: Perform iterative correction, first calculate the state increment:

[0051] Δx k =K k (z k -h(x k|k-1 ));

[0052] Correct the state estimate, each iteration according to the increment Δx k Update the state estimate:

[0053]

[0054] Where i represents the number of iterations;

[0055] After each iteration, the Kalman gain is recalculated, the state estimate is further adjusted, and the Kalman gain is updated:

[0056]

[0057] Until the change in state estimation is less than the preset threshold β = 10 -5 , or the maximum number of iterations, 1000, is reached, and the iteration is terminated;

[0058] After each iteration, update the state covariance P kk-1 :

[0059]

[0060] After multiple iterations, the final state estimate This is the angle after the optimized speed.

[0061] Furthermore, the acquisition of the front and back dual-view bending motion video of the human body includes:

[0062] S41: The human body performs a specified bending action facing the color sensor, thereby capturing a motion video of the human body bending forward.

[0063] S42: The back of the human body faces the color sensor to perform a specified bending action, thereby acquiring a motion video of the back of the human body bending action, and finally obtaining a complete front and back dual-viewpoint bending motion video sequence of the human body.

[0064] Furthermore, step S5 of "processing and segmenting the motion video sequence of a human body bending down frame by frame to obtain a two-dimensional back contour curve of the human body in each frame" includes the following steps:

[0065] S51: Segment each frame of the human body bending motion video sequence from the front and back of the camera frame by frame to obtain the two-dimensional back contour curve of each frame. The back contour of each frame is composed of a discrete point set. Where t is the time frame number and i is the profile index.

[0066] Furthermore, the method further comprises the steps of:

[0067] S52: performing Gaussian filtering on the binary segmentation image of the human body obtained by segmentation, and enhancing the contrast by normalization;

[0068] S53: Binarize the image using Otsu threshold segmentation;

[0069] S54: Optimize the binary image using morphological closing operation;

[0070] S55: Use contour detection to extract the target area of ​​the human mask and retain the outer contour with the largest area. Calculate the highest and lowest points of the contour. Based on multiple equidistant medians, only retain the contour points above the median three times. Finally, perform local weighted regression smoothing on the contour.

[0071] Furthermore, the Gaussian filter in step S52 is smoothed using a Gaussian filter with a 5×5 kernel, the morphological closing operation in step S54 is processed using a 3×3 elliptical structure element, and the contour detection in step S55 is performed using a maximum area external contour selection method.

[0072] Furthermore, the two-dimensional back contour curve features of each frame in step S5 include:

[0073] The percentage of the average first derivative of the left and right sides of the back contour to measure the change in the curvature of the back;

[0074] The area percentage used to measure the difference between the left and right areas of the back;

[0075] The ratio of the slope of the centroid point to the slope of the geometric center point is used to analyze the relative situation of the centroid and the geometric center of the back;

[0076] The ratio of the fitted ellipse to the outline used to measure the degree of deformation of the back morphology;

[0077] The centroid shift percentage used to measure the centroid shift situation;

[0078] The dynamic changes of back contour features in motion include:

[0079] The average curvature change rate and maximum curvature change of the back contour used to measure the changes in the back shape during bending movement. The curvature of each contour point is The curvature change between each frame is The average curvature change rate can be obtained Where T is the total number of frames, the maximum curvature change rate max(C t );

[0080] The standard deviation of the time series of contour area used to detect the degree of change in back morphology at different time points is: A t is the area enclosed by the back contour of each frame, is the mean of the area series;

[0081] The mean of the cosine similarity of the contour points is calculated as the set P of contour points between adjacent frames. t With P t+1 The mean of the cosine similarity

[0082] Furthermore, the method for generating a three-dimensional back surface in step S6 includes:

[0083] S61: Using a dual-view matching method, the back contour data of each frame of the human body bending towards the sensor is obtained in sequence from both the front and back, thus obtaining the most complete back information of the human body through dual-viewing.

[0084] S62: Using the timing alignment method, time synchronization interpolation is used to make the intervals between frames uniform, and multi-feature fusion is introduced into the traditional dynamic time warping (DTW) to match and achieve timing alignment:

[0085] The multiple features introduced include:

[0086] The percentage of the average value of the first derivative of the back contour on the left and right, the percentage of the area of ​​the contour on the left and right, the ratio of the slope of the centroid to the slope of the geometric center, the ratio of the fitted ellipse to the contour, and the percentage of the centroid offset; the curvature change between each frame is C t ; Changes in angles of key bone points: Introducing P6, P 12 、P 16 、P7、P 13 、P 17 Six key bone points are used to calculate the deflection angle of the left shoulder, left hip and left ankle during the sagittal plane bending movement. , and the angle of the three-point structure of the right shoulder, right hip and right ankle

[0087] S63: Skeletal key point optimization is used to introduce key point information of the left shoulder, left hip, right shoulder, and right hip during the bending process;

[0088] Finally, spline interpolation is used to generate the final three-dimensional back surface.

[0089] The present invention also provides a scoliosis rapid screening device, comprising a metal cabinet (14), a display screen (1) mounted on the metal cabinet (14), a color camera (2) mounted on the display screen, and a bending assist device;

[0090] The metal cabinet (14) is provided with an openable metal door (4), and an operating keyboard (3) and a mouse (6) are provided on the metal cabinet (14); a host computer (5) of a computer system is installed in the metal cabinet (14), the host computer (5) is connected to an external power supply via a first power line (7), the display screen (1) is connected to an external power supply via a second power line (15), the display screen (1) is connected to the host computer (5) via a display screen adapter line (16), and the operating keyboard (3) and the mouse (6) are both connected to the host computer (5) via wireless Bluetooth;

[0091] The bending assist device comprises a standard foot placement indicator pad (8), a head placement platform (9) placed on the standard foot placement indicator pad (8), a hydraulic rod (10) installed at the bottom of the head placement platform (9), a universal wheel (11) installed at the bottom of the hydraulic rod, and a voice prompter (12) installed on the hydraulic rod (10).

[0092] The above-mentioned joint learning model collected standing three-dimensional posture index data and back index data of more than 2,000 subjects, and had medical experts annotate the presence of scoliosis. A joint learning framework was used to determine whether the subjects were at risk of scoliosis based on multi-dimensional features. The base classifiers included support vector machine (SVM), random forest (RF), K-nearest neighbor classifier (KNN), and XGBoost (XGB). The meta-classifier included support vector machine (SVM) with radial basis function (RBF) kernel. The trained model was used to classify and detect the back contour features segmented from each frame of the newly input bending motion video sequence and the features measured by standing posture, and the classification results were stored.

[0093] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 16 when executed.

[0094] Compared with the prior art, the present invention has the following beneficial effects:

[0095] 1. The present invention introduces six indicators of the three-dimensional posture of the human body in the standing state: the coronal plane deflection angle of the left and right shoulders, the sagittal plane deflection angle of the left and right shoulders, the coronal plane deflection angle of the left and right hips, the sagittal plane deflection angle of the left and right hips, the center deflection angle, and the cervical lordosis angle;

[0096] 2. This invention collects a large number of accurate data sets annotated by doctors for training, which is more accurate.

[0097] 3. The present invention proposes a variety of joint learning methods, which can more accurately predict and determine whether the subject has the risk of scoliosis.

[0098] 4. The present invention proposes a process of having the subject face the color vision sensor from the front and back respectively, and collecting the subject's dual-perspective bending dynamic motion video sequence, thereby improving the integrity of the back data collection.

[0099] 5. The present invention introduces a method for detecting human bending behavior based on time-series dynamic motion trajectory analysis. This method extracts the changes in dynamic trajectory characteristics during the human bending process, including changes in back contour characteristics, time-series stability and other indicators, which are further used for scoliosis analysis.

[0100] 6. The present invention proposes a method for generating a three-dimensional back surface model from a two-dimensional back contour curve sequence using improved dynamic time adjustment (MF-EDTW), spline interpolation, optimization based on skeleton key points, dual-view matching and other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below.

[0102] Figure 1 This is a flow chart of the rapid scoliosis screening system provided by the present invention.

[0103] Figure 2 This is the two-dimensional posture detector provided by the present invention.

[0104] Figure 3 This is the network diagram of the three-dimensional human posture detection model, three-dimensional back surface generation model, and joint learning model combined with the SMPL model provided by the present invention.

[0105] Figure 4 It is a schematic diagram of the device of the present invention;

[0106] Figure 5 This is the three-dimensional image of the back proposed by the present invention.

[0107] In the figure, 1-display screen; 2-color camera; 3-operation keyboard; 4-metal door; 5-host; 6-mouse; 7-first power cord; 8-standard foot placement indicator pad; 9-head placement platform; 10-hydraulic rod; 11-universal wheel; 12-voice prompter; 13-bending aid power cord; 14-metal cabinet; 15-display screen power cord; 16-display screen adapter cable. Specific implementation methods

[0108] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific examples described herein are only used to explain the present invention, but are not intended to limit the present invention.

[0109] Reference Figure 1 , a rapid scoliosis screening method, comprising:

[0110] S1: Use visual sensors to obtain frame-by-frame images of the human spine to form a human standing video sequence;

[0111] S2: Obtain the 3D coordinate information of the main skeletal key points of the human body. The 3D coordinate information of the skeletal key points is obtained by inputting the human standing video sequence into DF-POSE;

[0112] S3: Analyze the three-dimensional coordinate information of the key points of the main bones of the human body, and calculate indicators such as the coronal plane deflection angle of the left and right shoulders, the sagittal plane deflection angle of the left and right shoulders, the coronal plane deflection angle of the left and right hips, the sagittal plane deflection angle of the left and right hips, the center deflection angle, and the cervical lordosis angle based on the three-dimensional coordinate information of the key points of the bones;

[0113] S4: Using a color vision sensor to collect dual-view videos of a person bending over from both the front and back perspectives, a video sequence of a person bending over from both the front and back perspectives facing the color vision sensor is formed;

[0114] S5: Process and segment the motion video sequence of a person bending over frame by frame to obtain the two-dimensional back contour curve of the person in each frame, obtain the characteristics of the two-dimensional back contour curve of each frame, and the dynamic change characteristics of the back contour characteristics in the motion state;

[0115] S6: Use methods such as dual-view matching, temporal alignment, skeleton key point optimization, and spline interpolation to stitch the two-dimensional back contours extracted frame by frame to generate a complete three-dimensional surface model of the back;

[0116] S7: By combining the dynamic change indicators of the back contour features in the motion state, and combining the back contour features of each frame extracted frame by frame and multiple indicators calculated based on the key points of the skeleton, the joint learning model is input to determine whether the subject has the risk of scoliosis.

[0117] The DF-POSE uses an end-to-end model training method, including the following steps:

[0118] S11: The human standing video sequence is first input into the 2D posture detector, and the posture feature vector is generated by the 2D human posture detection method;

[0119] S12: Input the posture feature vector into the 3D posture detector and combine it with the SMPL model to generate the 3D coordinate information of the main skeleton key points of the human body.

[0120] The two-dimensional human posture detection method in step S11 comprises the following steps:

[0121] S111: first adding an improved CBAM module to the improved ResNet-50 network structure of the two-dimensional posture detector to increase the degree of attention of the convolutional neural network to different feature channels and spatial positions;

[0122] S112: Adjust multiple basic blocks B1 and B2 in the ResNet-50 network to accelerate the model;

[0123] S113: The ResNet-50 network is divided into four parts. The first part contains one B1 and two B2s; the second part includes one B1 and three B2s; the third part includes one B1 and five B2s; the fourth part includes one B1 and two B2s. Ultimately, a two-dimensional posture vector for the human body is obtained, and finally the final three-dimensional human posture key point coordinates are obtained through a three-dimensional human posture detection model.

[0124] In step S3, the three-dimensional information of the main skeletal key points is analyzed to measure indicators such as the coronal plane deflection angle of the left and right shoulders, the sagittal plane deflection angle of the left and right shoulders, the coronal plane deflection angle of the left and right hips, the sagittal plane deflection angle of the left and right hips, the center deflection angle, and the cervical lordosis angle, including the following steps:

[0125] S31: After the model is detected, the skeleton key point information P is obtained. i =(x i ,y i ,z i ), nose P1=(x1,y1,z1), left eye P2=(x2,y2,z2), right eye P3=(x3,y3,z3), left ear P4=(x4,y4,z4), right ear P5=(x5,y5,z5), left shoulder P6=(x6,y6,z5), right shoulder P7=(x7,y7,z7), left elbow P8=(x8,y8,z8), right elbow P9=(x9,y9,z9), left wrist P 10 =(x 10 ,y 10 ,z 10 ), right wrist P 11 =(x 11 ,y 11 ,,z 11 ), left hip P 12 =(x 12 ,y 12 ,z 12 ), right hip P 13 =(x 13 ,y 13 ,z 13 ), left knee P 14 =(x 14 ,y 14 ,z 14 ), right knee P 15 =(x 15 ,y 15 ,z 15 ), left ankle P 16 =(x 16 ,y 16 ,z 16 ), right ankle P 17 =(x 17 ,y17 ,z 17 );

[0126] S32: Select the left and right shoulder joints P6 and P7, then the left and right shoulder coronal plane deflection angles θ1 and sagittal plane deflection angles θ 11 It can be obtained by the following formula;

[0127]

[0128] S33: Select the left and right hip joint points P 12 , P 13 , the coronal plane deflection angle of the left and right hips can be obtained The sagittal plane deflection angle of the left and right hip is

[0129] S34: Select left shoulder P6, right shoulder P7, left hip P 12 , right hip P 13 The center rotation angle θ3 can be obtained by the following formula:

[0130]

[0131] S35: Select the nose key point P1, left shoulder P6, and right shoulder P7 key point information, and the cervical lordosis deflection angle θ4 can be obtained by the following formula:

[0132]

[0133] Also includes the angles θ1, θ 11 ,θ2,θ 22 , θ3, θ4 optimization, the specific steps include:

[0134] T1: fitting angle lines by polynomial regression method;

[0135] θ(t)=a0+a1t+a2t 2 +...+a n t n ;

[0136] And take the derivative to obtain the first-order angle curve, then the angular velocity k at a certain moment is i It can be expressed by the following formula:

[0137]

[0138] T2: The burrs in the angular velocity are removed by using the improved Kalman filter algorithm, using the improved Jacobian matrix calculation. When the state is updated, the modified extended Kalman filter is modified by iteratively correcting the gain and state estimate.

[0139] Step T2 specifically includes:

[0140] T21: Create state vector:

[0141]

[0142] f is the nonlinear state transfer function, u k is the control input, based on the physical relationship between angle and angular velocity:

[0143] X K|K-1 =f(X k-1 ,u k );

[0144] T22: Calculate the Jacobian matrix F of the state transfer function to the state variables k-1 :

[0145]

[0146] T23: Calculate the predicted state covariance, where Q k is the process noise covariance:

[0147]

[0148] T24: Calculate Kalman gain:

[0149]

[0150] Among them, P kk-1 is the predicted state covariance, H k is the observation matrix, R k is the observation noise covariance.

[0151] T25: Update initial status:

[0152] x kk =x kk-1 +K k (z k -h(x kk-1 ))

[0153] Among them, z k are observations, and h(x) is the nonlinear model.

[0154] T26: Perform iterative correction, first calculate the state increment:

[0155] Δx k =K k (z k -h(x kk-1 ))

[0156] Correct the state estimate, each iteration according to the increment Δx k Update the state estimate:

[0157]

[0158] Where i represents the number of iterations.

[0159] After each iteration, the Kalman gain is recalculated, the state estimate is further adjusted, and the Kalman gain is updated:

[0160]

[0161] Until the change in state estimation is less than the preset threshold β = 10 -5 , or the maximum number of iterations, 1000, is reached, and the iteration is terminated.

[0162] After each iteration, update the state covariance P kk-1 :

[0163]

[0164] After multiple iterations, the final state estimate This is the angle after the optimized speed.

[0165] The acquisition of the front and back dual-viewpoint bending motion video of the human body includes:

[0166] S41: The human body performs a specified bending action facing the color sensor, thereby capturing a motion video of the human body bending forward.

[0167] S42: The back of the human body faces the color sensor to perform a specified bending action, thereby acquiring a motion video of the back of the human body bending action, and finally obtaining a complete front and back dual-viewpoint bending motion video sequence of the human body.

[0168] Step S5 of "processing and segmenting the motion video sequence of a human body bending over frame by frame to obtain a two-dimensional back contour curve of the human body in each frame" includes the following steps:

[0169] S51: Segment each frame of the human body bending motion video sequence from the front and back of the camera frame by frame to obtain the two-dimensional back contour curve of each frame. The back contour of each frame is composed of a discrete point set. Where t is the time frame number and i is the profile index.

[0170] Also includes the steps:

[0171] S52: performing Gaussian filtering on the binary segmentation image of the human body obtained by segmentation, and enhancing the contrast by normalization;

[0172] S53: Binarize the image using Otsu threshold segmentation;

[0173] S54: Optimize the binary image using morphological closing operation;

[0174] S55: Use contour detection to extract the target area of ​​the human mask and retain the outer contour with the largest area. Calculate the highest and lowest points of the contour. Based on multiple equidistant medians, only retain the contour points above the median three times. Finally, perform local weighted regression smoothing on the contour.

[0175] The Gaussian filter in step S52 is smoothed by using a Gaussian filter with a 5×5 kernel. The morphological closing operation in step S54 is processed by using a 3×3 elliptical structure element. The contour detection in step S55 is performed by using the maximum area external contour selection method.

[0176] The two-dimensional back contour curve features of each frame in step S5 include:

[0177] The percentage of the average first derivative of the left and right sides of the back contour to measure the change in the curvature of the back;

[0178] The area percentage used to measure the difference between the left and right areas of the back;

[0179] The ratio of the slope of the centroid point to the slope of the geometric center point is used to analyze the relative situation of the centroid and the geometric center of the back;

[0180] The ratio of the fitted ellipse to the outline used to measure the degree of deformation of the back morphology;

[0181] The centroid shift percentage used to measure the centroid shift situation;

[0182] The dynamic changes of back contour features in motion include:

[0183] The average curvature change rate and maximum curvature change of the back contour used to measure the changes in the back shape during bending movement. The curvature of each contour point is The curvature change between each frame is The average curvature change rate can be obtained Where T is the total number of frames, the maximum curvature change rate max(C t );

[0184] The standard deviation of the time series of contour area used to detect the degree of change in back morphology at different time points is: A t is the area enclosed by the back contour of each frame, is the mean of the area series;

[0185] The mean of the cosine similarity of the contour points is calculated as the set P of contour points between adjacent frames. t With P t+1 The mean of the cosine similarity

[0186] The method for generating a three-dimensional back surface in step S6 includes:

[0187] S61: Using a dual-view matching method, the back contour data of each frame of the human body bending towards the sensor is obtained in sequence from both the front and back, thus obtaining the most complete back information of the human body through dual-viewing.

[0188] S62: Using the timing alignment method, time synchronization interpolation is used to make the intervals between frames uniform. In addition, multi-feature fusion is introduced into the traditional dynamic time warping (DTW) to match and achieve timing alignment:

[0189] The multiple features introduced include:

[0190] The percentage of the average value of the first derivative of the back contour on the left and right, the percentage of the area of ​​the contour on the left and right, the ratio of the slope of the centroid to the slope of the geometric center, the ratio of the fitted ellipse to the contour, and the percentage of the centroid offset; the curvature change between each frame is C t ; Changes in angles of key bone points: Introducing P6, P 12 、P 16 、P7、P 13 、P 17 Six key bone points are used to calculate the deflection angle of the left shoulder, left hip and left ankle during the sagittal plane bending movement. , and the angle of the three-point structure of the right shoulder, right hip and right ankle

[0191] S63: Skeletal key point optimization is used to introduce key point information of the left shoulder, left hip, right shoulder, and right hip during the bending process;

[0192] Finally, spline interpolation is used to generate the final three-dimensional back surface.

[0193] The present invention also provides a rapid scoliosis screening device, comprising a metal cabinet 14, a display screen 1 mounted on the metal cabinet 14, a color camera 2 mounted on the display screen, and a bending assist device;

[0194] The metal cabinet 14 has an openable metal door 4, and an operating keyboard 3 and a mouse 6 are provided on the metal cabinet 14; a host computer 5 of a computer system is installed in the metal cabinet 14, and the host computer 5 is connected to an external power supply via a first power cord 7, and the display screen 1 is connected to an external power supply via a second power cord 15. The display screen 1 is connected to the host computer 5 via a display screen adapter cable 16, and the operating keyboard 3 and the mouse 6 are both connected to the host computer 5 via wireless Bluetooth;

[0195] The bending aid includes a standard foot placement indicator pad 8, a head placement platform 9 placed on the standard foot placement indicator pad 8, a hydraulic rod 10 installed at the bottom of the head placement platform 9, a universal wheel 11 installed at the bottom of the hydraulic rod, and a voice prompter 12 installed on the hydraulic rod 10.

[0196] Specifically, the voice prompter 12 is connected to the external power supply through the bending assist device power line 13 , and of course it also needs to be connected to the host 5 and the display screen 1 .

[0197] It is worth noting that the above-mentioned joint learning model collected standing three-dimensional posture index data and back index data of more than 2,000 subjects, and the presence of scoliosis was marked by medical experts. The joint learning framework was used to determine whether the subject had the risk of scoliosis based on multi-dimensional features. The base classifiers included support vector machine SVM, random forest RF, K nearest neighbor classifier KNN, and XGBoostXGB. The meta-classifiers included support vector machine SVM with radial basis function RBF kernel. The trained model was used to classify and detect the back contour features segmented from each frame of the newly input bending motion video sequence and the features measured in the standing posture, and the classification results were stored.

[0198] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 16 when executed.

[0199] Compared with the prior art, the present invention has the following beneficial effects:

[0200] 1. The present invention introduces six indicators of the three-dimensional posture of the human body in the standing state: the coronal plane deflection angle of the left and right shoulders, the sagittal plane deflection angle of the left and right shoulders, the coronal plane deflection angle of the left and right hips, the sagittal plane deflection angle of the left and right hips, the center deflection angle, and the cervical lordosis angle;

[0201] 2. This invention collects a large number of accurate data sets annotated by doctors for training, which is more accurate.

[0202] 3. The present invention proposes a variety of joint learning methods, which can more accurately predict and determine whether the subject has the risk of scoliosis.

[0203] 4. The present invention proposes a process of having the subject face the color vision sensor from the front and back respectively, and collecting the subject's dual-perspective bending dynamic motion video sequence, thereby improving the integrity of the back data collection.

[0204] 5. The present invention introduces a method for detecting human bending behavior based on time-series dynamic motion trajectory analysis. This method extracts the changes in dynamic trajectory characteristics during the human bending process, including changes in back contour characteristics, time-series stability and other indicators, which are further used for scoliosis analysis.

[0205] 6. The present invention proposes a method for generating a three-dimensional back surface model from a two-dimensional back contour curve sequence using an improved dynamic time adjustment MF-EDTW, spline interpolation, optimization based on skeleton key points, and dual-view matching.

Claims

1. A rapid scoliosis screening method, characterized in that: The method comprises: S1: Use visual sensors to obtain frame-by-frame images of the human spine to form a human standing video sequence; S2: Obtain the 3D coordinate information of the main skeletal key points of the human body. The 3D coordinate information of the skeletal key points is obtained by inputting the human standing video sequence into DF-POSE; S3: Analyze the three-dimensional coordinate information of the key points of the main bones of the human body, and calculate indicators such as the coronal plane deflection angle of the left and right shoulders, the sagittal plane deflection angle of the left and right shoulders, the coronal plane deflection angle of the left and right hips, the sagittal plane deflection angle of the left and right hips, the center deflection angle, and the cervical lordosis angle based on the three-dimensional coordinate information of the key points of the bones; S4: Using a color vision sensor to collect dual-view videos of a person bending over from both the front and back perspectives, a video sequence of a person bending over from both the front and back perspectives facing the color vision sensor is formed; S5: Process and segment the motion video sequence of a person bending over frame by frame to obtain the two-dimensional back contour curve of the person in each frame, obtain the characteristics of the two-dimensional back contour curve of each frame, and the dynamic change characteristics of the back contour characteristics in the motion state; S6: Use methods such as dual-view matching, temporal alignment, skeleton key point optimization, and spline interpolation to stitch the two-dimensional back contours extracted frame by frame to generate a complete three-dimensional surface model of the back; S7: By combining the dynamic change indicators of the back contour features in the motion state, and combining the back contour features of each frame extracted frame by frame and multiple indicators calculated based on the key points of the skeleton, the joint learning model is input to determine whether the subject has the risk of scoliosis.

2. The rapid scoliosis screening method according to claim 1, characterized in that: The DF-POSE uses an end-to-end model training method, including the following steps: S11: The human standing video sequence is first input into the 2D posture detector, and the posture feature vector is generated by the 2D human posture detection method; S12: Input the posture feature vector into the 3D posture detector and combine it with the SMPL model to generate the 3D coordinate information of the main skeleton key points of the human body.

3. The two-dimensional human posture detection method according to claim 2 generates a posture feature vector, characterized in that: The two-dimensional human posture detection method in step S11 comprises the following steps: S111: first adding an improved CBAM module to the improved ResNet-50 network structure of the two-dimensional posture detector to increase the degree of attention of the convolutional neural network to different feature channels and spatial positions; S112: Adjust multiple basic blocks B1 and B2 in the ResNet-50 network to accelerate the model; S113: The ResNet-50 network is divided into four parts. The first part contains one B1 and two B2s; the second part includes one B1 and three B2s; the third part includes one B1 and five B2s; the fourth part includes one B1 and two B2s. Ultimately, a two-dimensional posture vector for the human body is obtained, and finally the final three-dimensional human posture key point coordinates are obtained through a three-dimensional human posture detection model.

4. The rapid scoliosis screening method according to claim 1, characterized in that: In step S3, the three-dimensional information of the main skeletal key points is analyzed to measure indicators such as the coronal plane deflection angle of the left and right shoulders, the sagittal plane deflection angle of the left and right shoulders, the coronal plane deflection angle of the left and right hips, the sagittal plane deflection angle of the left and right hips, the center deflection angle, and the cervical lordosis angle, including the following steps: S31: After the model is detected, the skeleton key point information P is obtained. i =(x i ,y i ,z i ), nose P1=(x1,y1,z1), left eye P2=(x2,y2,z2), right eye P3=(x3,y3,z3), left ear P4=(x4,y4,z4), right ear P5=(x5,y5,z5), left shoulder P6=(x6,y6,z5), right shoulder P7=(x7,y7,z7), left elbow P8=(x8,y8,z8), right elbow P9=(x9,y9,z9), left wrist P 10 =(x 10 ,y 10 ,z 10 ), right wrist P 11 =(x 11 ,y 11 ,,z 11 ), left hip P 12 =(x 12 ,y 12 ,z 12 ), right hip P 13 =(x 13 ,y 13 ,z 13 ), left knee P 14 =(x 14 ,y 14 ,z 14 ), right knee P 15 =(x 15 ,y 15 ,z 15 ), left ankle P 16 =(x 16 ,y 16 ,z 16 ), right ankle P 17 =(x 17 ,y 17 ,z 17 ); S32: Select the left and right shoulder joints P6 and P7, then the left and right shoulder coronal plane deflection angles θ1 and sagittal plane deflection angles θ 11 It can be obtained by the following formula; S33: Select the left and right hip joint points P 12 , P 13 , the coronal plane deflection angle of the left and right hips can be obtained The sagittal plane deflection angle of the left and right hip is S34: Select left shoulder P6, right shoulder P7, left hip P 12 , right hip P 13 The center rotation angle θ3 can be obtained by the following formula: S35: Select the nose key point P1, left shoulder P6, and right shoulder P7 key point information, and the cervical lordosis deflection angle θ4 can be obtained by the following formula:

5. The rapid scoliosis screening method according to claim 4, characterized in that: Also includes the angles θ1, θ 11 ,θ2,θ 22 , θ3, θ4 optimization, the specific steps include: T1: fitting angle lines by polynomial regression method; θ(t)=a0+a1t+a2t 2 +...+a n t n ; And take the derivative to obtain the first-order angle curve, then the angular velocity k at a certain moment is i It can be expressed by the following formula: T2: The burrs in the angular velocity are removed by using the improved Kalman filter algorithm, using the improved Jacobian matrix calculation. When the state is updated, the modified extended Kalman filter is modified by iteratively correcting the gain and state estimate. Step T2 specifically includes: T21: Create state vector: f is the nonlinear state transfer function, u k is the control input, based on the physical relationship between angle and angular velocity: X K|K-1 =f(X k-1 ,u k ); T22: Calculate the Jacobian matrix F of the state transfer function to the state variables k-1 : T23: Calculate the predicted state covariance, where Q k is the process noise covariance: T24: Calculate Kalman gain: Among them, P k|k-1 is the predicted state covariance, H k is the observation matrix, R k is the observation noise covariance; T25: Update initial status: x k|k =x k|k-1 +K k (z k -h(x k|k-1 )); Among them, z k is the observed value, h(x) is the nonlinear model; T26: Perform iterative correction, first calculate the state increment: Δx k =K k (z k -h(x k|k-1 )); Correct the state estimate, each iteration according to the increment Δx k Update the state estimate: Where i represents the number of iterations; After each iteration, the Kalman gain is recalculated, the state estimate is further adjusted, and the Kalman gain is updated: Until the change in state estimation is less than the preset threshold β = 10 -5 , or the maximum number of iterations, 1000, is reached, and the iteration is terminated; After each iteration, update the state covariance P k|k-1 : After multiple iterations, the final state estimate This is the angle after the optimized speed.

6. The rapid scoliosis screening method according to claim 1, characterized in that: The acquisition of the front and back dual-viewpoint bending motion video of the human body includes: S41: The human body performs a specified bending action facing the color sensor, thereby capturing a motion video of the human body bending forward. S42: The back of the human body faces the color sensor to perform a specified bending action, thereby acquiring a motion video of the back of the human body bending action, and finally obtaining a complete front and back dual-viewpoint bending motion video sequence of the human body.

7. The rapid scoliosis screening method according to claim 1, characterized in that: In step S5, "processing and segmenting the motion video sequence of a human body bending over frame by frame to obtain a two-dimensional back contour curve of the human body in each frame" includes the following steps: S51: Segment each frame of the human body bending motion video sequence from the front and back of the camera frame by frame to obtain the two-dimensional back contour curve of each frame. The back contour of each frame is composed of a discrete point set. Where t is the time frame number and i is the profile index.

8. The rapid scoliosis screening method according to claim 7, characterized in that: Also includes the steps: S52: performing Gaussian filtering on the binary segmentation image of the human body obtained by segmentation, and enhancing the contrast by normalization; S53: Binarize the image using Otsu threshold segmentation; S54: Optimize the binary image using morphological closing operation; S55: Use contour detection to extract the target area of ​​the human mask and retain the outer contour with the largest area. Calculate the highest and lowest points of the contour. Based on multiple equidistant medians, only retain the contour points above the median three times. Finally, perform local weighted regression smoothing on the contour.

9. The rapid scoliosis screening method according to claim 8, characterized in that: The Gaussian filter in step S52 is smoothed by using a Gaussian filter with a 5×5 kernel. The morphological closing operation in step S54 is processed by using a 3×3 elliptical structure element. The contour detection in step S55 is performed by using the maximum area external contour selection method.

10. The rapid scoliosis screening method according to claim 1, characterized in that: The two-dimensional back contour curve features of each frame in step S5 include: The percentage of the average first derivative of the left and right sides of the back contour to measure the change in the curvature of the back; The area percentage used to measure the difference between the left and right areas of the back; The ratio of the slope of the centroid point to the slope of the geometric center point is used to analyze the relative situation of the centroid and the geometric center of the back; The ratio of the fitted ellipse to the outline used to measure the degree of deformation of the back morphology; The centroid shift percentage used to measure the centroid shift situation; The dynamic changes of back contour features in motion include: The average curvature change rate and maximum curvature change of the back contour used to measure the changes in the back shape during bending movement. The curvature of each contour point is The curvature change between each frame is The average curvature change rate can be obtained Where T is the total number of frames, the maximum curvature change rate max(C t ); The standard deviation of the time series of contour area used to detect the degree of change in back morphology at different time points is: A t is the area enclosed by the back contour of each frame, is the mean of the area series; The mean of the cosine similarity of the contour points is calculated as the set P of contour points between adjacent frames. t With P t+1 The mean of the cosine similarity 11. The rapid scoliosis screening method according to claim 1, characterized in that: The method for generating a three-dimensional back surface in step S6 includes: S61: Using a dual-view matching method, the back contour data of each frame of the human body bending towards the sensor is obtained in sequence from both the front and back, thus obtaining the most complete back information of the human body through dual-viewing. S62: Using the timing alignment method, time synchronization interpolation is used to make the intervals between frames uniform, and multi-feature fusion is introduced into the traditional dynamic time warping (DTW) to match and achieve timing alignment: The multiple features introduced include: The percentage of the average value of the first derivative of the back contour on the left and right, the percentage of the area of ​​the contour on the left and right, the ratio of the slope of the centroid to the slope of the geometric center, the ratio of the fitted ellipse to the contour, and the percentage of the centroid offset; the curvature change between each frame is C t ; Changes in angles of key bone points: Introducing P6, P 12 、P 16 、P7、P 13 、P 17 Six key bone points are used to calculate the deflection angle of the left shoulder, left hip and left ankle during the sagittal plane bending movement. , and the angle of the three-point structure of the right shoulder, right hip and right ankle S63: Skeletal key point optimization is used to introduce key point information of the left shoulder, left hip, right shoulder, and right hip during the bending process; Finally, spline interpolation is used to generate the final three-dimensional back surface.

12. A rapid scoliosis screening device, characterized in that: It comprises a metal cabinet (14), a display screen (1) mounted on the metal cabinet (14), a color camera (2) mounted on the display screen, and a bending assist device; The metal cabinet (14) is provided with an openable metal door (4), and an operating keyboard (3) and a mouse (6) are provided on the metal cabinet (14); a host computer (5) of a computer system is installed in the metal cabinet (14), the host computer (5) is connected to an external power supply via a first power line (7), the display screen (1) is connected to an external power supply via a second power line (15), the display screen (1) is connected to the host computer (5) via a display screen adapter line (16), and the operating keyboard (3) and the mouse (6) are both connected to the host computer (5) via wireless Bluetooth; The bending assist device comprises a standard foot placement indicator pad (8), a head placement platform (9) placed on the standard foot placement indicator pad (8), a hydraulic rod (10) installed at the bottom of the head placement platform (9), a universal wheel (11) installed at the bottom of the hydraulic rod, and a voice prompter (12) installed on the hydraulic rod (10).

Images