Dynamic three-dimensional ultrasound spinal column detection method and imaging system

Abstract

The application discloses a dynamic three-dimensional ultrasonic spine detection method and an imaging system, and belongs to the medical field. The method adopts a mature three-dimensional spine ultrasonic imaging system, cooperates with a wearable wireless ultrasonic probe to collect real-time ultrasonic images, and uses a depth camera to record the spatial coordinate change of a spine feature point of a patient when the spine is bent. The three-dimensional ultrasonic image, the real-time ultrasonic image and a dynamic three-dimensional image of a back shape are fused to obtain a dynamic three-dimensional ultrasonic spine image. According to the depth learning algorithm, the Lenke classification parameter is calculated, and the spine is dynamically and comprehensively evaluated. The classification of the spine scoliosis is upgraded from the traditional X-ray plane evaluation method based on radiation hazards to the dynamic three-dimensional ultrasonic analysis without radiation. The problems of high error rate of data collected in a static condition and insufficient collection parameters in the prior art are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medicine, and in particular to a dynamic three-dimensional ultrasound spine detection method and imaging system for obtaining Lenke classification parameters of scoliosis. BACKGROUND

[0002] Scoliosis refers to a spinal deformity in which one or several segments of the spine curve sideways or are accompanied by vertebral rotation. Adolescent idiopathic scoliosis (AIS) is the most common form of scoliosis. Diagnosis of scoliosis is currently mainly performed by X-ray, using the Cobb angle measurement method to measure the angle of scoliosis.

[0003] Three-dimensional spine ultrasound imaging has also been clinically applied in many aspects internationally. Commercial three-dimensional spine ultrasound systems represented by Scolioscan are effective and reliable in measuring coronal deformities of scoliosis patients, and are as good as traditional X-ray Cobb angle gold standards in screening a large number of patients, monitoring spinal changes, and evaluating treatment results, which helps to reduce the radiation dose during continuous monitoring.

[0004] However, three-dimensional spine ultrasound imaging systems are all for detecting static indicators, i.e., they require patients to be still during scanning. During scanning, the body posture of the subject can adversely affect the results, and the state of human motion corresponding to the detection results cannot be confirmed when judging the detection results. Therefore, the data collected in such a static condition has a high error rate, and has limited guiding effect on surgical and non-surgical treatment decisions for scoliosis.

[0005] Machine vision technology has also been applied in scoliosis in recent years. Color images and depth maps are collected by a depth camera, and the surface information of the human body is reconstructed in three dimensions. The Cobb angle can be calculated according to the three-dimensional profile of the back. This method relying only on machine vision has strict requirements for the shooting environment and poor applicability for patients with unclear waist concave-convex and unclear back midline.

[0006] Surgical and non-surgical treatment decisions for scoliosis require static and dynamic comprehensive evaluation of the spine, mainly relying on ① the overall balance of the spine, the real-time activity of lateral flexion of the spine, the flexibility and stability of the spine; ② the opening degree of the intervertebral space near the top vertebra and the upper and lower end vertebrae during lateral flexion; ③ the improvement of the vertebral rotation degree in the top vertebra area. However, current technologies such as X-ray, three-dimensional spine ultrasound imaging, and machine vision cannot achieve the evaluation of all the above parameters. SUMMARY

[0007] Therefore, it is necessary to provide a dynamic three-dimensional ultrasound spine detection method and imaging system to solve the problems of high error rate of data collected in static conditions and insufficient collection parameters in the prior art, and better provide guidance for surgical and non-surgical treatment decisions for scoliosis.

[0008] To achieve the above-mentioned purpose, the present application provides a dynamic three-dimensional ultrasound spine detection method, which uses a computer terminal 4 in communication connection with a three-dimensional spine ultrasound imaging device 1, a wearable wireless ultrasound scanning unit 2, and a depth camera 3 with a built-in optical sensor, and the method comprises the following steps:

[0009] S1) using the three-dimensional spine ultrasound imaging device 1 to perform three-dimensional ultrasound scanning on the spine of a patient to obtain a three-dimensional spine image 100, including coronal and sagittal spine projection images;

[0010] S2) obtaining the positions of one or more main curved apex vertebrae and upper and lower end vertebrae of scoliosis from the coronal spine projection image of the three-dimensional spine image 100, and fixing the ultrasound probe of the wearable wireless ultrasound scanning unit 2 at the obtained curved apex vertebrae, upper end vertebrae and lower end vertebrae positions respectively to collect real-time ultrasound images 200 at the corresponding positions, and the outer end surface of the ultrasound probe is provided with an optical marker;

[0011] S3) directing the depth camera 3 with a built-in optical sensor towards the back of the patient to obtain real-time video, including multiple frames of dynamic back images 300 of back depth maps, optical markers and body surface maps;

[0012] S4) the patient bends the torso according to the instructions until the desired curvature is reached, and the computer terminal collects dynamic back images 300 including back depth maps and body surface maps and real-time ultrasound images 200 during the scoliosis bending action; the body surface map is a color map or a gray scale map; the bending of the torso according to the instructions includes lateral flexion, forward flexion or any desired test posture, or bending to any suitable curvature for examination;

[0013] S5) using the optical marker in the body surface map to obtain the real-time three-dimensional spatial position and direction of each ultrasound probe during the scoliosis bending action;

[0014] S6) the computer terminal 4 synchronizes and fuses the real-time ultrasound images 200 with the back depth maps in the dynamic back images 300 and the three-dimensional spine image 100 respectively to obtain dynamic three-dimensional ultrasound images during the scoliosis bending action; wherein, during the image synchronization and fusion process, the real-time three-dimensional spatial position and direction of the ultrasound probe obtained in step S5) are used as reference information;

[0015] S7) On the computer terminal, according to the dynamic three-dimensional ultrasound image fused in step S6), the curvature of the spine is analyzed, including the measurement of the Lenke classification parameters, the real-time activity of the lateral curvature of the spine, the flexibility and stability of the spine are analyzed, and the opening degree of the intervertebral space near the top vertebra and the upper and lower end vertebrae during lateral bending activity, and the rotation of the vertebral body are evaluated.

[0016] The application also provides a dynamic three-dimensional ultrasound spine imaging system, which comprises an existing three-dimensional spine ultrasound imaging device 1, and further comprises:

[0017] A wearable wireless ultrasound scanning unit 2 is used to generate local two-dimensional ultrasound image frames of the spine, which contains three or more ultrasound probes;

[0018] A depth camera unit 3 is used to record the spatial position changes of the ultrasound probes and the dynamic three-dimensional shape of the back, and the depth camera unit contains a depth camera with an internal optical sensor;

[0019] A computer terminal 4 is connected to the three-dimensional spine ultrasound imaging device 1, the wearable wireless ultrasound scanning unit 2, and the depth camera unit 3, and is used to receive the three-dimensional spine image 100 of the three-dimensional spine ultrasound imaging device 1, the real-time ultrasound image 200 of the wearable wireless ultrasound scanning unit 2, and the dynamic back image 300 of the depth camera unit 3, and by fusing the three-dimensional spine image 100, the real-time ultrasound image 200, and the dynamic back image 300, the curvature of the spine is analyzed, including the measurement of the Lenke classification parameters of the lateral curvature of the spine, and the dynamic evaluation of the spine during lateral bending activity.

[0020] Due to the method and system of the application, a mature three-dimensional spine ultrasound imaging system is used, a wearable wireless ultrasound probe is used to collect real-time ultrasound images, and a depth camera with an internal optical sensor is used to record the spatial coordinate changes of the characteristic points of the spine during lateral bending of the patient in real time. The three-dimensional ultrasound images collected by the three-dimensional spine ultrasound imaging system in the upright and curved states, the real-time ultrasound images collected by the wearable wireless ultrasound system, and the dynamic three-dimensional images of the back shape collected by the depth camera are fused to obtain a dynamic three-dimensional ultrasound spine image. According to the dynamic three-dimensional ultrasound spine image, the Lenke classification parameters of the lateral curvature of the spine are measured quickly and accurately in real time by combining the computer vision feedback of deep learning, and the overall balance of the spine, the real-time activity of the lateral curvature of the spine, the flexibility and stability of the spine, the opening degree of the intervertebral space near the top vertebra and the end vertebra during lateral bending activity, and the improvement of the rotation degree of the vertebral body in the top vertebra area are dynamically and comprehensively evaluated, so that the classification of the lateral curvature of the spine is upgraded from the traditional X-ray plane evaluation method based on radiation hazards to dynamic three-dimensional ultrasound analysis without radiation, solving the problems of high error rate of data collected in a static state and insufficient collection parameters in the prior art. The application can also be applied to the testing of the forward bending of the spine and other spine bending tests, and the specific process is similar to that of lateral bending. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a schematic diagram showing the numbering of the human thoracic and lumbar vertebrae involved in an embodiment of the present invention.

[0023] Figure 2 The logic block diagram of the dynamic three-dimensional ultrasound spinal imaging system provided in the embodiments of the present invention is shown.

[0024] Figure 3 A schematic diagram illustrating the key technologies of the dynamic three-dimensional ultrasound spinal imaging system provided in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of the steps of the dynamic three-dimensional ultrasound spinal detection method provided in an embodiment of the present invention.

[0026] Figure 5 Two-dimensional ultrasound images of the spine cross-section provided in an embodiment of the present invention.

[0027] Figure 6 The diagram shows the connection between the transverse processes on both sides of the end vertebra in a three-dimensional spinal volume projection imaging image and an X-ray image, which are provided for embodiments of the present invention.

[0028] Figure 7 This is an illustration of the effect of identifying and marking the position of an ultrasound probe in a fused image of a human back depth image from a depth camera and a three-dimensional ultrasound volume projection imaging, provided by an embodiment of the present invention. Detailed Implementation

[0029] This application provides a dynamic three-dimensional ultrasound spinal detection method and imaging system, which solves the problems of high error rate and insufficient acquisition parameters in the static data acquisition of the prior art, and makes the classification of scoliosis rise from the traditional X-ray planar assessment method based on radiation hazards to the radiation-free dynamic three-dimensional ultrasound analysis.

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] The embodiments of the present application will be further described below with reference to the accompanying drawings. It should be understood that the embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0032] In actual clinical diagnosis, doctors have corresponding names for each vertebra of the spine. The human spine has a total of 33 vertebrae, of which 7 are cervical vertebrae, 12 are thoracic vertebrae, 5 are lumbar vertebrae, and 9 are sacrum and coccyx. In the scoliosis Lenke classification system, the Cobb angle measurement does not involve the cervical vertebrae and the sacrum and coccyx. For example Figure 1 As shown in the numbering diagram of the thoracic and lumbar vertebrae of the human body, L5-L1 represents the lumbar vertebrae from bottom to top, and T12-T1 represents the thoracic vertebrae.

[0033] One embodiment of the present application provides a dynamic three-dimensional ultrasonic spine imaging system, as shown in Figure 2 The logic block diagram of the system includes a three-dimensional spine ultrasonic imaging device 1, and further includes:

[0034] A wearable wireless ultrasonic scanning unit 2 is used to generate a local two-dimensional ultrasonic image frame of the spine, which contains three or more ultrasonic probes.

[0035] A depth camera unit 3 is used to record the spatial position change of the ultrasonic probe and the dynamic three-dimensional shape of the back. The depth camera unit contains a depth camera with a built-in optical sensor.

[0036] A computer terminal 4 is connected to the three-dimensional spine ultrasonic imaging device 1, the wearable wireless ultrasonic scanning unit 2, and the depth camera unit 3, and is used to receive the three-dimensional spine image 100 of the three-dimensional spine ultrasonic imaging device 1, the real-time ultrasonic image 200 of the wearable wireless ultrasonic scanning unit 2, and the dynamic back image 300 of the depth camera unit 3. By fusing the three-dimensional spine image, the real-time ultrasonic image, and the dynamic back image, a dynamic three-dimensional ultrasonic spine image is obtained, and a spine curvature shape analysis is performed, including measuring the Lenke classification parameters of scoliosis, and dynamically evaluating the spine in lateral flexion activity.

[0037] The three-dimensional spine ultrasonic imaging device 1 is a mature three-dimensional spine ultrasonic imaging device. By acquiring ultrasonic pictures of the spine of a patient, a three-dimensional image 100 of the spine is obtained after volume reconstruction. By volume projection imaging method, the projection images of the spine in the coronal plane and the sagittal plane can be further obtained.

[0038] The wearable wireless ultrasonic scanning unit 2 contains ultrasonic probes and wireless controllers. In the embodiment of the present application, the wearable wireless ultrasonic scanning unit 2 contains 3 or more probes. The ultrasonic probes acquire real-time ultrasonic images 200 of the spine.

[0039] The depth camera unit 3 includes a depth camera with built-in optical sensor and a bracket, the depth camera is located on the bracket, is aligned with the back of the patient, and collects a dynamic back image 300 of the patient.

[0040] The computer terminal includes a display terminal for displaying the three-dimensional spine image 100, the real-time ultrasound image 200, the dynamic back image 300, and the fused image.

[0041] As shown in Figure 3 Fig. 1 shows a schematic diagram of a key technology of a dynamic three-dimensional ultrasound spine imaging system according to an embodiment of the present application. The three-dimensional spine ultrasound imaging system uses a volume projection imaging method to provide a projection image of the patient's spine in the coronal plane, from which the curvature of each vertebral body of the spine can be seen. For each major curvature, the positions of the top vertebra, the upper end vertebra, and the lower end vertebra are located, and three ultrasound probes are installed at the corresponding positions of the patient's spine, corresponding to the positions of the top vertebra, the upper end vertebra, and the lower end vertebra, respectively. The top vertebra, the upper end vertebra, and the lower end vertebra are located at the center positions of the scanning windows of the three probes, respectively. The positions of the probes are fine-tuned in combination with the real-time ultrasound images of the three probes displayed by the computer terminal, so that the complete bilateral lamina reflection surfaces are seen in the ultrasound image field of view. The outer cross sections of the three probes are provided with optical markers, and wired connections are used between each probe and the wireless controller.

[0042] An embodiment of the present application provides a dynamic three-dimensional ultrasound spine detection method, as shown in Figure 4 Fig. 2 shows a schematic diagram of the steps of a dynamic three-dimensional ultrasound spine detection method according to an embodiment of the present application, including the following steps:

[0043] S1) Use the three-dimensional spine ultrasound imaging device 1 to perform three-dimensional ultrasound scanning on the patient's spine to obtain a three-dimensional spine image 100, including coronal and sagittal spine projection images.

[0044] S2) Obtain the positions of the top vertebra, the upper end vertebra, and the lower end vertebra of one or more major curvatures of the spine from the coronal spine projection image of the three-dimensional spine image 100, and fine-tune the positions of the ultrasound probes of the wearable wireless ultrasound scanning unit 2 according to the complete bilateral lamina reflection surfaces in the ultrasound image field of view. Real-time ultrasound images 200 of the corresponding positions are collected, and optical markers are provided on the outer end surfaces of the ultrasound probes.

[0045] S3) Place the depth camera 3 with a built-in optical sensor towards the back of the patient to obtain real-time video, including a dynamic back image 300 of multiple frames of back depth images, optical markers, and body surface images.

[0046] S4) The patient bends the trunk according to the instruction until the required curvature is reached, and the computer terminal collects the dynamic back depth map and body surface map dynamic back image 300 during the spinal curvature movement in real time, as well as real-time ultrasound images 200; the body surface map is a color map or a gray-scale map; the bending of the trunk according to the instruction includes lateral flexion, forward bending or any required test posture, or bending to any suitable curvature for examination.

[0047] S5) The real-time three-dimensional spatial position and direction of each ultrasound probe during the spinal curvature movement are obtained by using the optical markers in the body surface map.

[0048] S6) The computer terminal 4 synchronizes and fuses the real-time ultrasound images 200 with the back depth map in the dynamic back image 300 and the three-dimensional spinal image 100, respectively, to obtain dynamic three-dimensional ultrasound images during the spinal curvature movement; during the image synchronization and fusion process, the real-time three-dimensional spatial position and direction of the ultrasound probe obtained in step S5) are used as reference information.

[0049] S7) Based on the fused dynamic three-dimensional ultrasound images in step S6), spinal curvature shape analysis is performed on the computer terminal, including Lenke classification parameter calculation, analysis of the real-time activity, flexibility and stability of spinal lateral flexion, and evaluation of the opening degree of the intervertebral space near the top vertebra and the upper and lower end vertebrae and the rotation of the vertebral body during lateral flexion.

[0050] The ultrasound probe contains a patch, a cable, a sticking and fixing device and a wireless controller, and the wireless controller transmits the patch ultrasound signal conducted by the cable to the nearby computer terminal 4 through WiFi.

[0051] The ultrasound probe can be a patch ultrasound probe, a flexible ultrasound probe, a one-dimensional array or a two-dimensional array ultrasound probe.

[0052] In the embodiment of the present application, the flexible ultrasound probe can be an array probe and a unit probe, which can generate focused and clear sound beams in a local detection surface, thereby obtaining higher resolution.

[0053] In the embodiment of the present application, the depth camera 3 with an optical sensor built-in can be installed on a support, and the support can control the depth camera to move in six directions, i.e., forward, backward, upward, downward, left and right, so that the human back is always within the field of view of the camera when the spine is bent.

[0054] In an embodiment of the method of the present application, step S1 comprises performing an upright standing position spinal ultrasound scan of the patient using the three-dimensional spinal ultrasound imaging device 1 to obtain a three-dimensional spinal image 100 of the patient in an upright standing position, including coronal and sagittal spinal projection images.

[0055] In another embodiment of the method of the present application, step S1 further comprises performing an ultrasound scan of the patient in one or more different curved positions using the three-dimensional spinal ultrasound imaging device 1 to obtain a three-dimensional spinal image 100 of the patient in one or more different curved positions, including coronal and sagittal spinal projection images. The curved positions are dynamic three-dimensional spinal configurations of the patient in a lateral bending, forward bending, or other bending action for spinal evaluation.

[0056] In an embodiment of the method of the present application, if more than one main curve is obtained from the coronal spinal projection image in step S2, then for each of the other curves, steps S2-S6 are repeated to collect data.

[0057] In an embodiment of the method of the present application, the step S2 of obtaining the positions of the apex vertebra and the upper and lower end vertebrae of the main curve of scoliosis from the coronal spinal projection image in the three-dimensional spinal image 100 comprises the following steps:

[0058] S21) numbering each vertebra;

[0059] S22) determining the number of main curves;

[0060] S23) for each curve, identifying and recording the numbers of the apex vertebra and the upper and lower end vertebrae.

[0061] A common color camera can capture and record all objects within the camera's field of view, but the data recorded does not include the distance of these objects from the camera. Only through semantic analysis of the image can we determine which objects are farther away and which are closer, but there is no exact data. The depth camera used in the present application solves this problem. The depth camera can obtain both a color image (or a gray-scale image) and a depth image at the same time. Through the data of the depth image, we can accurately know the distance of each point in the image from the camera. Thus, by adding the (x, y) coordinates of the point in the 2D image, we can obtain the three-dimensional spatial coordinates of each point in the image. Through the three-dimensional coordinates, we can restore the real scene and achieve scene modeling and other applications. Therefore, the real-time three-dimensional dynamic image of the patient captured by the depth camera can be effectively fused with the real-time ultrasound image and the three-dimensional ultrasound image obtained by the three-dimensional ultrasound system to calculate and classify the spinal parameters at the dynamic three-dimensional level.

[0062] Image fusion refers to the fusion of two or more images together to generate a fused image. In the process of image fusion, the key information of each original image can be retained, while the detailed information in different images is used for correction and enhancement, so as to generate a clearer and more distinct image.

[0063] Image fusion is usually performed on multiple images obtained from different sources or different sensors. Through fusion, the noise problem existing in images obtained by different sensors can be eliminated, the quality and resolution of the images can be improved, and the recognition and analysis ability of the images can be improved, thereby providing a better basis for further processing and analysis of the images.

[0064] Image fusion can use various methods, such as pixel-level fusion, feature-level fusion, high-level neural network-based fusion, etc. These methods depend on specific features of the image, such as spatial resolution, color, gray scale, and texture, etc.

[0065] In the embodiment of the present application, the specific steps of fusing and synchronizing the real-time ultrasound image 200 with the back depth map in the dynamic back image 300 in step S6 are as follows:

[0066] S61) Identify the optical marker on each ultrasound probe by image recognition algorithm. The optical marker in the embodiment of the present application is in the form of two-dimensional code, and other planar patterns or three-dimensional forms with obvious texture features can also be used, which is not limited in the present application.

[0067] S62) Obtain the real-time three-dimensional space position and direction of each ultrasound probe through image analysis;

[0068] S63) Fuse the real-time ultrasound image collected by each probe with the back depth map according to the position and direction information. The specific method of fusion is as follows: synchronize the real-time ultrasound image with the back depth image in time, unify the real-time ultrasound image and the back depth map of the patient at the same time or at a similar time to the same coordinate system, obtain the corresponding depth information for the pixel points in the real-time ultrasound image range, and realize the fusion of the real-time ultrasound image and the back depth map.

[0069] The fusion of the real-time ultrasound image 200 and the three-dimensional spine image 100 is based on the position of each ultrasound probe relative to the spine, and the real-time ultrasound image collected by the ultrasound probe is fused with the three-dimensional spine image, so that the real-time spine image features of the local area corresponding to the ultrasound probe and the overall spine information of the three-dimensional spine image are fused to obtain a dynamic three-dimensional spine ultrasound image. The specific method of fusion is: based on the position of each probe relative to the spine, the real-time ultrasound image collected by the probe is extracted for feature points such as spines, laminae and transverse processes, and through feature matching and fusion method, the extracted feature points are one-to-one corresponding to the three-dimensional spine image, realizing the fusion and matching of two-dimensional ultrasound image and three-dimensional spine image.

[0070] Lenke classification is mainly aimed at idiopathic scoliosis, and plays an important role in the diagnosis of scoliosis and treatment design. Lenke classification consists of three parts: coronal curve type (1-6), lumbar correction type (A, B, C), and thoracic sagittal sequence correction type (-, N, +).

[0071] When performing Lenke classification, first determine the type of bend according to the top vertebra interval. Table 1 shows the correspondence between thoracic and lumbar bend types and top vertebra intervals.

[0072] Table 1 Correspondence table of thoracic and lumbar bend types and top vertebra intervals

[0073] Chest, lumbar curve type Top vertebrae interval Proximal thoracic (PT) T2-T5 Main thoracic (MT) T6-T11 / 12 Thoracolumbar (TL) T12, L1 Lumbar (L) L1 / 2-L4

[0074] After determining the type of bend, judge the primary and secondary bends, structural bends and non-structural bends, and then refer to Table 2 to determine the coronal curve type (1-6).

[0075] Table 2 Lenke coronal curve type (1-6)

[0076] Type Proximal thoracic Main thoracic Thoracolumbar / Lumbar Curve type 1 Non-structural Structural Non-structural Main thoracic 2 Structural Structural Non-structural Double thoracic 3 Non-structural Structural Structural Double main 4 Structural Structural Structural Triple main 5 Non-structural Non-structural Structural Thoracolumbar / Lumbar 6 Non-structural Structural Structural Thoracolumbar / Lumbar - Main thoracic

[0077] Note: * indicates the position of the primary bend

[0078] The specific steps of calculating the Lenke classification parameters according to the fused image in step S6 on the computer software in step S7 are as follows:

[0079] S71) Obtain the Cobb angle of the coronal plane and the Cobb angle of the sagittal plane in the upright position according to the fused image in the upright position, and determine the position of the top vertebra;

[0080] S72) Obtain the change of Cobb angle in lateral flexion according to the fused image in lateral flexion;

[0081] S73) According to the top vertebra position of step S71) and the Cobb angle of step S72), the type of the curve, the primary curve, the secondary curve, the structural curve and the non-structural curve are determined, and the lateral curve type of the Lenke classification is determined in combination with the Lenke classification principle;

[0082] S74) According to the relative relationship between the position of the top vertebra of the lumbar curve and the center vertical line of the sacrum, the lumbar correction type of the Lenke classification is obtained; S75) According to the Cobb angle of the sagittal plane when standing upright, the sagittal sequence correction type of the thoracic vertebra of the Lenke classification is determined.

[0083] The surgical and non-surgical treatment decision for scoliosis needs a comprehensive evaluation of the spine in static and dynamic state, mainly depending on ① the overall balance of the spine, the real-time activity of lateral flexion of the spine, the flexibility and stability of the spine; ② the opening degree of the intervertebral space near the top vertebra and the end vertebra during lateral flexion; ③ the improvement of the rotation degree of the vertebrae in the top vertebra area.

[0084] Dynamic three-dimensional ultrasound is based on three-dimensional ultrasound and adds time tracking. Through the dynamic three-dimensional ultrasound image, the continuous changes of each vertebra of the patient during lateral flexion can be observed, and the evaluation parameters of scoliosis can be calculated in real time. In step S8, the real-time activity, flexibility and stability of the spine during lateral flexion are analyzed, and the opening degree of the intervertebral space near the top vertebra and the upper and lower end vertebrae during lateral flexion, and the rotation of the vertebrae are evaluated, which are all based on the dynamic comprehensive evaluation of the spine by the dynamic three-dimensional ultrasound image constructed in step S6. The results of dynamic comprehensive evaluation can form a written report to provide a diagnostic basis for doctors.

[0085] Machine learning is classified according to the learning form, which can be divided into supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. The difference lies in that supervised learning needs to provide a labeled sample set, unsupervised learning does not need to provide a labeled sample set, semi-supervised learning needs to provide a small amount of labeled samples, and reinforcement learning needs a feedback mechanism.

[0086] Through deep learning of a large number of collected dynamic three-dimensional ultrasound images after step S6 fusion and their corresponding scoliosis evaluation parameters, the dynamic three-dimensional ultrasound images after step S6 fusion are automatically recognized and the scoliosis evaluation parameters are automatically calculated.

[0087] In an embodiment of the present application, the deep learning includes marker recognition in two-dimensional cross-sectional spine ultrasound images. Specifically, after a large number of cross-sectional spine ultrasound images are collected, the spinous process and the two side laminae of each frame are manually marked. The marked frames are used as the training set of the supervised deep learning algorithm, and through the training of a large amount of data, the goal of simulating manual marker recognition and marking is achieved, so as to automatically calculate the rotation degree of the vertebral body and realize real-time activity tracking of the vertebral body in the lateral bending action. The shape of the spinous process and the inclination of the two side laminae in the two-dimensional cross-sectional spine ultrasound image play an important role in judging the spatial rotation angle of the vertebral body and in judging the real-time activity degree, flexibility and stability of the vertebral body in lateral bending. As shown in FIG. 1, Figure 5 As shown in FIG. 1, the shape of the spinous process and the two side laminae in the two-dimensional cross-sectional spine ultrasound image is clear and distinguishable, so the embodiment of the present application uses a deep learning algorithm to automatically recognize the spinous process and the laminae in the two-dimensional cross-sectional spine ultrasound image. After a large number of cross-sectional spine ultrasound images are collected, the spinous process and the two side laminae of each frame are manually marked. The marked frames are used as the training set of the supervised deep learning algorithm, and through the training of a large amount of data, the goal of simulating manual marker recognition and marking is achieved, so as to automatically calculate the rotation degree of the vertebral body and realize real-time activity tracking of the vertebral body in lateral bending.

[0088] In an embodiment of the present application, the deep learning also includes angle measurement of three-dimensional spine volume projection imaging ultrasound images. Specifically, after a large number of three-dimensional spine volume projection ultrasound images are collected, the two side transverse processes of the end vertebrae of each curve are manually marked as the training set of the supervised deep learning algorithm, and through the training of a large amount of data, the goal of simulating manual marker recognition and marking is achieved, so as to automatically calculate the angle of each spinal scoliosis curve of the patient, and then realize real-time activity tracking of the angle change of the spinal scoliosis curve in lateral bending. Clinically, three-dimensional spine volume projection imaging ultrasound images are used to measure the angle of spinal scoliosis (analogous to X-ray Cobb angle). When a spinal scoliosis patient has one or more spinal deformity curves, the angle formed by the end vertebrae at both ends of each curve is used to mark the severity of the curve. In the three-dimensional spine volume projection imaging ultrasound image, the current manual marking method can use the angle of the connecting line between the two side transverse processes of the end vertebrae or the laminae of the upper segment lumbar vertebrae and the upper articular process projection of the lower segment lumbar vertebrae. As shown in FIG. 2, Figure 6 (a) shows a connecting line diagram of the two side transverse processes of the end vertebrae in the three-dimensional spine volume projection imaging image, Figure 6(b) X-ray film end of the two sides of the transverse process of the line diagram. After collecting a large number of three-dimensional spine volume projection ultrasound images, manually mark the transverse process of each curve end of the two sides as a training set for supervised deep learning algorithm. Through the training of a large amount of data, the goal of simulating manual identification and marking of markers is achieved, so as to automatically calculate the angle of each spinal scoliosis curve of the patient, and then realize the real-time activity tracking of the angle change of the spinal scoliosis curve in lateral flexion.

[0089] In an embodiment of the present application, the deep learning further includes real-time measurement of the spatial position change of the ultrasound probe. Specifically, it refers to the algorithm of unsupervised learning, which uses optical markers to automatically identify, track and record the position of the ultrasound probe. This embodiment captures and records the spatial position change of the ultrasound probe in real time through a depth camera, thereby establishing the spatial coordinate system of the detected internal markers of the spine. As shown in Figure 7 As shown in the figure, the effect of identifying and labeling the position of the ultrasound probe in the depth camera human back depth image and three-dimensional ultrasound volume projection imaging fusion image. Because the relative position change of the ultrasound probe in the body surface spatial coordinate system is different from the observed anatomical markers in the body, and its shape and material are different from the human body structure and easy to identify, the algorithm of unsupervised learning can be used to complete the automatic identification, tracking and recording of the position of the ultrasound probe.

[0090] Because medical images have the characteristics of blur, unevenness, individual differences, complexity and diversity, the use of deep learning algorithm to process medical images has more advantages than traditional algorithms. However, because medical images involve patient privacy and are difficult to obtain, the limited sample data set obtained needs to be expanded.

[0091] The method and system of the present application, on the basis of using a mature three-dimensional spine ultrasound imaging system, cooperate with a wearable wireless ultrasound probe to collect real-time ultrasound images, and a depth camera with a built-in optical sensor to record the spatial coordinate changes of the spine feature points when the patient bends, fuse the three-dimensional ultrasound images collected by the three-dimensional spine ultrasound imaging system in the upright and curved state, the real-time ultrasound images collected by the wearable wireless ultrasound system, and the dynamic three-dimensional images of the back shape collected by the depth camera to obtain dynamic three-dimensional ultrasound spine images. According to the dynamic three-dimensional ultrasound spine image, combined with the real-time feedback of computer vision of deep learning, the Lenke classification parameters of scoliosis are measured quickly and accurately, and the overall balance of the spine, the real-time activity of lateral flexion of the spine, the flexibility and stability of the spine, the opening degree of the intervertebral space near the top and end vertebrae during lateral flexion, and the improvement of the rotation degree of the vertebrae in the top vertebrae area are dynamically and comprehensively evaluated, so that the classification of scoliosis is upgraded from the traditional X-ray plane evaluation method based on radiation hazards to dynamic three-dimensional ultrasound analysis without radiation, solving the problems of high error rate of data collected in static conditions and insufficient collected parameters in the prior art. The present application can also be applied to the test of spinal flexion and other needed spinal curvature, and the specific process and lateral flexion are similar.

[0092] The above is only a specific embodiment of the present application, which cannot limit the scope of the present application. Equivalent changes made by those skilled in the art according to the present application, and changes well known in the art, should still fall within the scope of the present application.

Claims

1. A dynamic three-dimensional ultrasonic spinal column detection method, characterized by: The method uses a computer terminal (4) in communication connection with a three-dimensional spine ultrasound imaging device (1), a wearable wireless ultrasound scanning unit (2), and a depth camera (3) with built-in optical sensors. The method comprises the following steps: S1) using the three-dimensional spine ultrasound imaging device (1) to perform three-dimensional ultrasound scanning on the spine of a patient to obtain three-dimensional spine images (100), including coronal and sagittal spine projection images; S2) obtaining the positions of one or more primary curve apex vertebrae and upper and lower end vertebrae of the scoliosis from the coronal spine projection image of the three-dimensional spine images (100), and fixing the ultrasound probes of the wearable wireless ultrasound scanning unit (2) at the obtained curve apex vertebrae, upper end vertebrae, and lower end vertebrae positions, respectively, to collect real-time ultrasound images (200) of the corresponding positions, with optical markers provided on the outer end surfaces of the ultrasound probes; S3) directing the depth camera (3) with built-in optical sensors towards the back of the patient to obtain real-time video, including multiple frames of dynamic back images (300) of back depth maps, optical markers, and body surface maps; S4) the patient bends the torso according to instructions until the desired curvature is reached, and the computer terminal collects dynamic back images (300) including back depth maps and body surface maps, and real-time ultrasound images (200) during the scoliosis bending action; the body surface map is a color map or a grayscale map; the bending of the torso includes lateral flexion, forward bending, or any desired test posture, or bending to any suitable curvature for examination; S5) using the optical markers in the body surface map to obtain the real-time three-dimensional spatial positions and directions of each ultrasound probe during the scoliosis bending action; S6) the computer terminal (4) synchronizes and fuses the real-time ultrasound images (200) with the back depth maps in the dynamic back images (300) and the three-dimensional spine images (100), respectively, to obtain dynamic three-dimensional ultrasound images during the scoliosis bending action; during the image synchronization and fusion process, the real-time three-dimensional spatial positions and directions of the ultrasound probes obtained in step S5) are used as reference information; S7) based on the fused dynamic three-dimensional ultrasound images in step S6), the computer terminal performs scoliosis bending shape analysis, including Lenke classification parameter calculation, real-time activity analysis of scoliosis lateral flexion, flexibility, and stability, and evaluation of the opening degree of the intervertebral space near the apex vertebrae and the upper and lower end vertebrae and the rotation of the vertebral body during lateral flexion.

2. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, The step S1) includes using the three-dimensional spine ultrasound imaging device (1) to perform spine ultrasound scanning on the patient in an upright standing position to obtain three-dimensional spine images (100) of the patient in an upright standing position, including coronal and sagittal spine projection images.

3. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, The step S1) comprises using a three-dimensional spine ultrasound imaging device (1) to perform ultrasound scanning of the spine of a patient in one or more different bending postures, and obtaining three-dimensional spine images (100) of the patient in one or more different bending postures, including coronal and sagittal spine projection images.

4. The dynamic three-dimensional ultrasound spinal column detection method of claim 3, wherein, The bending posture of the spine is the dynamic three-dimensional shape of the spine of the patient when the patient performs lateral bending, forward bending, or other bending actions for spine evaluation.

5. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, Between steps S6) and S7), there is further included a step S6a): if the number of main bending positions obtained from the coronal spine projection image in step S2) is greater than one, then for each of the other bending positions, steps S2)-S6) are repeated to collect data.

6. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, In the step S2), the positions of the main bending apex vertebra and the upper and lower end vertebrae of the scoliosis are obtained from the coronal spine projection image in the three-dimensional spine image (100), comprising the following steps: S21) numbering each vertebra; S22) determining the number of main bends; S23) for each bend, identifying and recording the numbers of the apex vertebra and the upper and lower end vertebrae.

7. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, The specific steps of fusing and synchronizing the real-time ultrasound images (200) with the back depth map in the dynamic back image (300) in the step S6) are as follows: S61) identifying the optical markers on each of the ultrasound probes by image recognition algorithm; S62) obtaining the real-time three-dimensional spatial position and direction of each of the ultrasound probes by image analysis; S63) fusing the real-time ultrasound images collected by each probe with the back depth map according to the position and direction information; The fusion of the real-time ultrasound images (200) with the three-dimensional spine image (100) is based on the position of each ultrasound probe relative to the spine, and the real-time ultrasound images collected by the ultrasound probe are fused with the three-dimensional spine image, so that the real-time spine image features of the local area corresponding to the ultrasound probe and the overall spine information of the three-dimensional spine image are fused to obtain a dynamic three-dimensional spine ultrasound image.

8. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, The specific steps of calculating the Lenke classification parameters according to the fused image in step S6) on the computer terminal in step S7) are as follows: S71) obtaining the coronal Cobb angle and the sagittal Cobb angle in the upright position according to the fused image in the upright position, and determining the position of the apex vertebra; S72) obtaining the change of the Cobb angle in the lateral bending position according to the fused image in the lateral bending position; S73) judging the main bend, the secondary bend, the structural bend and the non-structural bend according to the position of the apex vertebra in step 71) and the Cobb angle in step 72), and determining the scoliosis type of the Lenke classification according to the Lenke classification principle; S74) obtaining the lumbar revision type of the Lenke classification according to the relative relationship between the position of the lumbar apex vertebra and the center vertical line of the sacrum; S75) determining the thoracic sagittal sequence revision type of the Lenke classification according to the sagittal Cobb angle in the upright position.

9. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, Further included is a step S8) of automatically identifying and automatically calculating the scoliosis evaluation parameters of the dynamic three-dimensional ultrasound image fused in step S6) by deep learning on a large number of collected dynamic three-dimensional ultrasound images and their corresponding scoliosis evaluation parameters.

10. The dynamic three-dimensional ultrasound spinal column detection method of claim 9, wherein, The deep learning in the step S8) includes two-dimensional spine cross-section ultrasound image marker recognition in a detection range frame, including: after a large number of spine cross-section ultrasound images are collected, a spinous process and two side laminae of a corresponding frame are manually marked; the marked frame is used as a training set of a supervised deep learning algorithm, a large amount of data is trained, a goal of simulating manual marker recognition and marking is achieved, and thus the rotation degree of a vertebral body is automatically calculated, and real-time activity tracking of the vertebral body in a spine bending action is realized.

11. The dynamic three-dimensional ultrasound spinal column detection method of claim 9, wherein, The deep learning in the step S8) includes angle measurement of a three-dimensional spine volume projection imaging ultrasound image, including collecting a large number of three-dimensional spine volume projection ultrasound images, marking a projection of a transverse process on two sides of an end vertebra or a lamina of an upper segment lumbar vertebra and an upper articular process of a lower segment lumbar vertebra, training the marked image, achieving a goal of simulating manual marker recognition and marking, and thus automatically calculating an angle of a spine scoliosis curve of each segment of a patient, and further realizing real-time activity tracking of the angle of the spine scoliosis curve in lateral flexion.

12. The dynamic three-dimensional ultrasound spinal column detection method of claim 9, wherein, The deep learning in the step S8) includes real-time measurement of spatial position change of an ultrasound probe, including an unsupervised learning algorithm, and automatic recognition, tracking and recording of the position of the ultrasound probe are completed by using an optical marker.

13. The dynamic three-dimensional ultrasonic spinal column survey method of any of claims 1-12, wherein, The ultrasound probe includes a patch, a cable, a sticking fixing device and a wireless controller, and the wireless controller transmits patch ultrasound signals conducted by the cable to a nearby computer terminal (4) through WiFi.

14. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, The ultrasound probe is a patch ultrasound probe.

15. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, The ultrasound probe is a flexible ultrasound probe.

16. The dynamic three-dimensional ultrasound spinal column detection method of claim 1, wherein, The ultrasound probe is a one-dimensional array or two-dimensional array ultrasound probe.

17. The dynamic three-dimensional ultrasound spinal column detection method of claim 6, wherein, The depth camera in the step S3) is located on a support, and the support can control the depth camera to move in six directions, namely forward, backward, upward, downward, left and right, so that the back of the human body is always in the field of view of the camera when the spine bends.

18. A dynamic three-dimensional ultrasonic spinal imaging system for implementing the dynamic three-dimensional ultrasonic spinal detection method as claimed in claim 1, comprising a three-dimensional spinal ultrasonic imaging device (1), characterized in that, Further comprising: A wearable wireless ultrasound scanning unit (2) for generating local two-dimensional ultrasound image frames of the spine, including three or more ultrasound probes; A depth camera unit for recording spatial position changes of the ultrasound probes and dynamic three-dimensional shapes of the back, the depth camera unit including a depth camera with an optical sensor built-in; A computer terminal (4) connected with the three-dimensional spine ultrasound imaging device (1), the wearable wireless ultrasound scanning unit (2) and the depth camera unit, for receiving three-dimensional spine images (100) of the three-dimensional spine ultrasound imaging device (1), real-time ultrasound images (200) of the wearable wireless ultrasound scanning unit (2) and dynamic back images (300) of the depth camera unit, and performing spine bending shape analysis by fusing the three-dimensional spine images (100), the real-time ultrasound images (200) and the dynamic back images (300), including measuring Lenke classification parameters of a scoliosis and dynamically evaluating the spine in lateral flexion.

Images