Dynamic three-dimensional ultrasonic spine detection method and imaging system

Abstract

The invention discloses a dynamic three-dimensional ultrasonic spine detection method and imaging system, and belongs to the field of medicine.The method adopts a mature three-dimensional spine ultrasonic imaging system to be matched with a wearable wireless ultrasonic probe to collect real-time ultrasonic images, a depth camera records space coordinate changes of spine feature points in real time when the spine of a patient is bent, and the dynamic three-dimensional spine ultrasonic imaging system is obtained. A three-dimensional ultrasonic image, a real-time ultrasonic image and a dynamic three-dimensional image of a back form are fused to obtain a dynamic three-dimensional ultrasonic spine image, and a deep learning algorithm is combined according to the dynamic three-dimensional ultrasonic spine image to carry out Lenke typing parameter measurement and calculation and carry out dynamic comprehensive evaluation on the spine. Scoliosis typing is improved from a traditional X-ray plane evaluation mode based on radiation hazards to non-radiation dynamic three-dimensional ultrasonic analysis, and the problems that in the prior art, the error rate of data collected under the static condition is high, and collection parameters are insufficient are solved.

Description

Technical Field

[0001] The present invention relates to the medical field, and in particular to a dynamic three-dimensional ultrasonic spine detection method and imaging system for obtaining Lenke classification parameters of scoliosis. Background Art

[0002] Scoliosis refers to a spinal deformity in which one or more segments of the spine bend sideways or are accompanied by vertebral rotation. Adolescent idiopathic scoliosis (AIS) is the most common form of scoliosis. The diagnosis of scoliosis is currently mainly carried out through X-rays, using the Cobb angle measurement method to measure the scoliosis angle.

[0003] Three-dimensional spinal ultrasound imaging has also been widely used in clinical applications internationally. Commercial three-dimensional spinal ultrasound systems represented by Scolioscan are effective and reliable in measuring coronal plane deformity in patients with scoliosis. They are comparable to the traditional X-ray Cobb angle gold standard in screening large numbers of patients, monitoring spinal changes, and evaluating treatment outcomes, helping to reduce radiation dose during continuous monitoring.

[0004] However, three-dimensional spinal ultrasound imaging systems detect static indicators, that is, the patient is required to be scanned while lying still. During the scanning process, the subject's posture will have an adverse effect on the result, and the corresponding state of human body movement cannot be confirmed when judging the test results. Therefore, the data collected in this static situation has a high error rate and has limited guiding role in decision-making on surgical and non-surgical treatments for scoliosis.

[0005] Machine vision technology has also been used in scoliosis in recent years. Through the acquisition of color and depth images by depth cameras, the surface information of the human back is reconstructed in three dimensions, and the Cobb angle can be calculated based on the three-dimensional contour of the back. This method that relies solely on machine vision has strict requirements on the shooting environment and is not suitable for patients with unclear waist concavity and unclear back midline.

[0006] The decision of surgical and non-surgical treatment for scoliosis requires a comprehensive static and dynamic evaluation of the spine, which mainly depends on ① the overall balance of the spine, the real-time range of motion of the spine in lateral flexion, the flexibility and stability of the spine; ② the degree of opening of the intervertebral space near the top vertebra and the upper and lower end vertebrae during lateral flexion; ③ the improvement of the degree of vertebral rotation in the top vertebra area. However, it is still impossible to evaluate all the above parameters by relying solely on X-ray, three-dimensional spinal ultrasound imaging, machine vision and other technologies. Summary of the invention

[0007] In view of this, it is necessary to provide a dynamic three-dimensional ultrasonic spine detection method and imaging system to solve the problems of high error rate and insufficient acquisition parameters of data collected under static conditions in the existing technology, so as to better provide guidance for surgical and non-surgical treatment decisions for scoliosis.

[0008] To achieve the above object, the present invention provides a dynamic three-dimensional ultrasonic spine detection method, using a computer terminal 4 that is communicatively connected to a three-dimensional spine ultrasonic imaging device 1, a wearable wireless ultrasonic scanning unit 2, and a depth camera 3 with a built-in optical sensor, and the method includes the following steps:

[0009] S1) performing a three-dimensional ultrasonic scan on the patient's spine using the three-dimensional spine ultrasonic imaging device 1 to obtain a three-dimensional spine image 100, including spine projection images in the coronal plane and sagittal plane;

[0010] S2) obtaining the positions of one or more main curvature apex vertebrae and upper and lower end vertebrae of scoliosis from the coronal spine projection image in the three-dimensional spine image 100, and fixing the ultrasonic probe of the wearable wireless ultrasonic scanning unit 2 at the obtained curvature apex vertebrae, upper end vertebrae and lower end vertebrae positions respectively based on the complete vertebral plate reflection surfaces on both sides seen in the field of view of the ultrasonic image, so as to collect the real-time ultrasonic image 200 of the corresponding positions, and the outer end surface of the ultrasonic probe is provided with an optical marker;

[0011] S3) directing a depth camera 3 with a built-in optical sensor toward the patient's back to obtain a real-time video including a dynamic back image 300 of multiple frames of back depth maps, optical markers, and body surface maps;

[0012] S4) the patient bends the trunk according to the instructions until the required curvature is reached, and the computer terminal collects in real time a dynamic back image 300 including a back depth map and a body surface map and a real-time ultrasound image 200 during the spinal bending action; the body surface map is a color map or a grayscale map; the bending of the trunk according to the instructions includes lateral flexion, forward flexion or any required examination posture, or bending to any curvature suitable for examination;

[0013] S5) using the optical markers in the body surface image to obtain the real-time three-dimensional spatial position and direction of each of the ultrasound probes during the spinal flexion movement;

[0014] S6) the computer terminal 4 synchronizes and fuses the real-time ultrasound image 200 with the back depth map in the dynamic back image 300 and the three-dimensional spine image 100, respectively, to obtain a dynamic three-dimensional ultrasound image during the spine bending action; wherein, during the image synchronization and fusion processing, the real-time three-dimensional spatial position and direction of the ultrasound probe obtained in step S5) is used as reference information;

[0015] S7) On a computer terminal, a spinal curvature morphology analysis is performed based on the dynamic three-dimensional ultrasound image fused in step S6), including calculation of Lenke classification parameters, analysis of the real-time range of motion, flexibility and stability of the scoliosis, and evaluation of the degree of opening of the intervertebral spaces near the top vertebra and the upper and lower end vertebrae and the rotation of the vertebral bodies during lateral flexion.

[0016] The present invention also provides a dynamic three-dimensional ultrasonic spine imaging system, comprising an existing three-dimensional spine ultrasonic imaging device 1, and further comprising:

[0017] A wearable wireless ultrasound scanning unit 2, used for generating a local two-dimensional ultrasound image frame of the spine, comprising three or more ultrasound probes;

[0018] A depth camera unit 3, used to record the spatial position change of the ultrasound probe and the dynamic three-dimensional shape of the back, and the depth camera unit includes a depth camera with a built-in optical sensor;

[0019] The computer terminal 4 is connected to the three-dimensional spine ultrasound imaging device 1, the wearable wireless ultrasound scanning unit 2, and the depth camera unit 3, and is used to receive the three-dimensional spine image 100 of the three-dimensional spine ultrasound imaging device 1, the real-time ultrasound image 200 of the wearable wireless ultrasound scanning unit 2, and the dynamic back image 300 of the depth camera unit 3. By fusing the three-dimensional spine image 100, the real-time ultrasound image 200, and the dynamic back image 300, a spinal curvature morphology analysis is performed, including measuring the Lenke classification parameters of scoliosis, and a dynamic evaluation of the spine in lateral flexion activity is performed.

[0020] Due to the method and system of the present invention, a mature three-dimensional spinal ultrasound imaging system is used, and a wearable wireless ultrasound probe is used to collect real-time ultrasound images, and a depth camera with a built-in optical sensor is used to record the spatial coordinate changes of the characteristic points of the spine when the patient is lateralized, and the three-dimensional ultrasound images in the upright and flexed states collected by the three-dimensional spinal ultrasound imaging system, the real-time ultrasound images collected by the wearable wireless ultrasound system, and the dynamic three-dimensional images of the back morphology collected by the depth camera are fused to obtain a dynamic three-dimensional ultrasound spinal image. According to the dynamic three-dimensional ultrasound spinal image, the real-time feedback of computer vision combined with deep learning is used to quickly and accurately measure the Lenke classification parameters of scoliosis, and the overall balance of the spine, the real-time range of motion of the spinal lateral flexion, the flexibility and stability of the spine, the degree of opening of the intervertebral space near the top vertebra and the end vertebra during lateral flexion, and the improvement of the degree of rotation of the vertebral body in the top vertebra area are dynamically and comprehensively evaluated, so that the classification of scoliosis is upgraded from the traditional X-ray plane assessment method based on radiation hazards to the dynamic three-dimensional ultrasound analysis without radiation, which solves the problems of high error rate and insufficient acquisition parameters of data collected in static conditions in the prior art. The present invention can also be applied to spinal flexion and other spinal curvature tests as required, and the specific process is similar to that of lateral flexion. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 This is a schematic diagram of the numbering of the thoracic vertebrae and lumbar vertebrae of the human body involved in the embodiment of the present invention.

[0023] Figure 2 A logic block diagram of a dynamic three-dimensional ultrasonic spine imaging system provided in an embodiment of the present invention.

[0024] Figure 3 A schematic diagram of key technologies of a dynamic three-dimensional ultrasonic spine imaging system provided in an embodiment of the present invention.

[0025] Figure 4 A schematic diagram of the steps of a dynamic three-dimensional ultrasonic spine detection method provided in an embodiment of the present invention.

[0026] Figure 5 A two-dimensional spinal cross-sectional ultrasound image provided by an embodiment of the present invention.

[0027] Figure 6 A schematic diagram of the connection of the transverse processes on both sides of the end vertebrae in a three-dimensional spine volume projection imaging image and a schematic diagram of the connection of the transverse processes on both sides of the end vertebrae in an X-ray film provided by an embodiment of the present invention.

[0028] Figure 7 This is a rendering of the effect of identifying and marking the position of the ultrasound probe in the fusion image of the depth image of the human back of the depth camera and the three-dimensional ultrasound volume projection imaging provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present application provide a dynamic three-dimensional ultrasonic spine detection method and imaging system, which solves the problems of high error rate and insufficient acquisition parameters of data collected under static conditions in the prior art, thereby improving the classification of scoliosis from the traditional X-ray plane assessment method with radiation hazards to dynamic three-dimensional ultrasonic analysis without radiation.

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings of the specification. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0032] In actual clinical diagnosis, doctors have corresponding names for each vertebra of the spine. The human spine has 33 vertebrae in total, including 7 cervical vertebrae, 12 thoracic vertebrae, 5 lumbar vertebrae, and 9 sacrum and coccyx. In the Lenke classification system for scoliosis, Cobb angle measurement does not involve the cervical vertebrae, sacrum, coccyx, etc. Figure 1 Shown is a schematic diagram of the numbering of the thoracic and lumbar vertebrae of the human body. From bottom to top, L5-L1 represents the lumbar vertebrae, and T12-T1 represents the thoracic vertebrae.

[0033] One embodiment of the present invention provides a dynamic three-dimensional ultrasound spine imaging system. Figure 2 The system is shown as a logic block diagram, including a three-dimensional spine ultrasound imaging device 1, and further comprising:

[0034] A wearable wireless ultrasound scanning unit 2, used for generating a local two-dimensional ultrasound image frame of the spine, comprising three or more ultrasound probes;

[0035] A depth camera unit 3, used to record the spatial position change of the ultrasound probe and the dynamic three-dimensional shape of the back, and the depth camera unit includes a depth camera with a built-in optical sensor;

[0036] The computer terminal 4 is connected to the three-dimensional spine ultrasound imaging device 1, the wearable wireless ultrasound scanning unit 2, and the depth camera unit 3, and is used to receive the three-dimensional spine image 100 of the three-dimensional spine ultrasound imaging device 1, the real-time ultrasound image 200 of the wearable wireless ultrasound scanning unit 2, and the dynamic back image 300 of the depth camera unit 3, and obtain a dynamic three-dimensional ultrasound spine image by fusing the three-dimensional spine image, the real-time ultrasound image, and the dynamic back image, and perform spinal curvature morphology analysis, including measuring the Lenke classification parameters of scoliosis, and dynamically evaluate the spine in lateral flexion activities.

[0037] Among them, the three-dimensional spine ultrasound imaging device 1 is a mature three-dimensional spine ultrasound imaging device. By collecting ultrasound images of the patient's spine and performing volume reconstruction, a three-dimensional image 100 of the spine is obtained. Through volume projection imaging, projection images of the spine in the coronal plane and sagittal plane can be further obtained.

[0038] The wearable wireless ultrasound scanning unit 2 includes an ultrasound probe and a wireless controller. In the embodiment of the present invention, the wearable wireless ultrasound scanning unit 2 includes three or more probes, and the ultrasound probes collect real-time ultrasound images 200 of the spine.

[0039] The depth camera unit 3 includes a depth camera with a built-in optical sensor and a bracket. The depth camera is located on the bracket and is aimed at the patient's back to collect a dynamic back image 300 of the patient.

[0040] The computer terminal includes a display terminal for displaying a three-dimensional spine image 100 , a real-time ultrasound image 200 , a dynamic back image 300 and a fused image.

[0041] like Figure 3 The figure shows a schematic diagram of the key technologies of the dynamic three-dimensional ultrasonic spine imaging system provided by an embodiment of the present invention. The three-dimensional spine ultrasonic imaging system uses volume projection imaging to provide a projection of the patient's spine in the coronal plane, from which the curvature of each vertebra of the spine can be seen. For each major curvature, the position of the top vertebra, the upper vertebra, and the lower vertebra are located, and three ultrasonic probes are installed at the corresponding positions of the patient's spine, corresponding to the positions of the top vertebra, the upper vertebra, and the lower vertebra, respectively. The top vertebra, the upper vertebra, and the lower vertebra are located at the center of the scanning window of the three probes, respectively. The position of the probe is fine-tuned in combination with the real-time ultrasonic images of the three probes displayed on the computer terminal, so that the complete reflection surfaces of the vertebral plates on both sides can be seen in the field of view of the ultrasonic image. Optical markers are set on the outward sections of the three probes, and wired connections are used between each probe and the wireless controller.

[0042] An embodiment of the present invention provides a dynamic three-dimensional ultrasonic spine detection method, such as Figure 4 The figure is a schematic diagram of the steps of the dynamic three-dimensional ultrasonic spine detection method provided by an embodiment of the present invention, which includes the following steps:

[0043] S1) Using a three-dimensional spine ultrasound imaging device 1, a three-dimensional ultrasound scan is performed on the patient's spine to obtain a three-dimensional spine image 100, including spine projection images in the coronal and sagittal planes.

[0044] S2) The positions of one or more main apical vertebrae and upper and lower end vertebrae of scoliosis are obtained from the coronal spine projection image in the three-dimensional spine image 100, and the complete vertebral plate reflection surfaces on both sides are seen in the field of view of the ultrasound image. The ultrasound probe of the wearable wireless ultrasound scanning unit 2 is respectively fixed at the obtained positions of the apical vertebrae, upper end vertebrae and lower end vertebrae to collect real-time ultrasound images 200 at corresponding positions, and an optical marker is provided on the outer end surface of the ultrasound probe.

[0045] S3) Directing the depth camera 3 with a built-in optical sensor toward the patient's back to obtain real-time video, including multiple frames of back depth maps, optical markers, and dynamic back images 300 of body surface maps.

[0046] S4) The patient bends the trunk according to the instructions until the required curvature is reached, and the computer terminal collects in real time the dynamic back depth map and the dynamic back image 300 of the body surface map during the spinal bending movement, as well as the real-time ultrasound image 200; the body surface map is a color map or a grayscale map; the bending of the trunk according to the instructions includes lateral flexion, forward flexion or any required examination posture, or bending to any curvature suitable for examination.

[0047] S5) Using the optical markers in the body surface image, the real-time three-dimensional spatial position and direction of each of the ultrasound probes during the spinal flexion movement are obtained.

[0048] S6) The computer terminal 4 synchronizes and fuses the real-time ultrasound image 200 with the back depth map in the dynamic back image 300 and the three-dimensional spine image 100 respectively to obtain a dynamic three-dimensional ultrasound image during the spinal bending movement; wherein, during the image synchronization and fusion processing, the real-time three-dimensional spatial position and direction of the ultrasound probe obtained in step S5) is used as reference information.

[0049] S7) On a computer terminal, a spinal curvature morphology analysis is performed based on the dynamic three-dimensional ultrasound image fused in step S6), including calculation of Lenke classification parameters, analysis of the real-time range of motion, flexibility and stability of the scoliosis, and evaluation of the degree of opening of the intervertebral spaces near the top vertebra and the upper and lower end vertebrae and the rotation of the vertebral bodies during lateral flexion.

[0050] The ultrasonic probe comprises a patch, a cable, an adhesive fixing device and a wireless controller. The wireless controller transmits the patch ultrasonic signal transmitted by the cable to a nearby computer terminal 4 via WiFi.

[0051] The ultrasound probe may be a patch ultrasound probe, a flexible ultrasound probe, a one-dimensional array ultrasound probe, or a two-dimensional array ultrasound probe.

[0052] Among them, the flexible ultrasonic probe can be an array probe and a unit probe. In the embodiment of the present invention, it can be an array probe of a one-dimensional array or a two-dimensional array, which can generate a focused and clear sound beam within a local detection surface, thereby obtaining a higher resolution.

[0053] In the embodiment of the present invention, the depth camera 3 with a built-in optical sensor can be installed on a bracket, which can control the depth camera to move in six directions: forward, backward, up, down, left, and right, so that the back of the human body is always within the field of view of the camera when the spine bends.

[0054] In an embodiment of the method of the present invention, step S1 includes using a three-dimensional spine ultrasound imaging device 1 to perform a spine ultrasound scan on a patient in an upright standing position, and obtaining a three-dimensional spine image 100 of the patient in an upright standing position, including spine projection images in the coronal plane and the sagittal plane.

[0055] In another embodiment of the present invention, step S1 further comprises using a three-dimensional spine ultrasound imaging device 1 to perform ultrasound scanning of the patient's spine in one or more different bending states, and obtaining a three-dimensional spine image 100 of the patient in one or more different bending postures, including coronal and sagittal spine projection images. The spinal bending posture is the dynamic three-dimensional morphology of the patient's spine when performing lateral flexion, forward flexion, or other bending movements for spinal assessment.

[0056] In the embodiment of the present invention, if there is more than one main curvature portion obtained from the coronal spine projection image in step S2, then steps S2 to S6 are repeated for each other curvature portion to collect data.

[0057] In the embodiment of the present invention, the step S2 of obtaining the positions of the top vertebra and the upper and lower end vertebrae of the main curvature of scoliosis from the coronal spine projection image in the three-dimensional spine image 100 includes the following steps:

[0058] S21) numbering each vertebra;

[0059] S22) determining the number of main bends;

[0060] S23) For each bend, identify and record the numbers of the top vertebra, upper and lower vertebrae.

[0061] The pictures taken by ordinary color cameras can see and record all objects within the camera's field of view, but the data recorded does not include the distance of these objects from the camera. Only semantic analysis of the image can be used to determine which objects are farther away from us and which are closer, but there is no exact data. The depth camera used in the present invention solves this problem. The depth camera can obtain color images (or grayscale images) and depth images at the same time. Through the data of the depth image, the distance of each point in the image from the camera can be accurately known. In this way, the (x, y) coordinates of the point in the 2D image are added to obtain the three-dimensional spatial coordinates of each point in the image. The real scene can be restored through three-dimensional coordinates, and applications such as scene modeling can be realized. Therefore, the real-time three-dimensional dynamic image of the patient captured by the depth camera can be effectively fused with the real-time ultrasound image and the three-dimensional ultrasound image obtained by the three-dimensional ultrasound system to perform dynamic three-dimensional spinal parameter measurement and classification.

[0062] Image fusion refers to fusing two or more images together to generate a fused image. In the process of image fusion, the key information of each original image can be retained, while the detailed information in different images can be used to correct and enhance them, thus generating a clearer and more vivid image.

[0063] Image fusion is usually performed on multiple images acquired from different sources or different sensors. Through fusion, the noise problem existing in images acquired by different sensors can be eliminated, the image quality and resolution can be improved, and the image recognition and analysis capabilities can be improved, providing a better basis for further processing and analysis of the image.

[0064] Image fusion can be performed in a variety of ways, such as pixel-level fusion, feature-level fusion, advanced neural network-based fusion, etc. These methods rely on specific features of the image, such as spatial resolution, color, grayscale, and texture.

[0065] In the embodiment of the present invention, the specific steps of fusing and synchronizing the real-time ultrasound image 200 with the back depth map in the dynamic back image 300 in step S6 are:

[0066] S61) The optical markers on each of the ultrasound probes are identified by an image recognition algorithm. The optical markers in the embodiment of the present invention are in the form of a two-dimensional code, and may also be in the form of other planar patterns or three-dimensional forms with obvious texture features, which is not limited by the present invention.

[0067] S62) obtaining the real-time three-dimensional spatial position and direction of each of the ultrasound probes through image analysis;

[0068] S63) fusing the real-time ultrasound image collected by each probe with the back depth map according to the position and direction information, wherein the specific fusing method is: synchronizing the time of the real-time ultrasound image and the back depth map, unifying the real-time ultrasound image and the back depth map of the patient at the same time or at a similar time into the same coordinate system, acquiring the corresponding depth information for the pixel points within the range of the real-time ultrasound image, and realizing the fusing of the real-time ultrasound image and the back depth map;

[0069] The fusion of the real-time ultrasound image 200 and the three-dimensional spine image 100 is based on the position of each ultrasound probe relative to the spine, and the real-time ultrasound image collected by the ultrasound probe is fused with the three-dimensional spine image, so that the real-time spine image features of the local area corresponding to the ultrasound probe and the overall spine information of the three-dimensional spine image are fused to obtain a dynamic three-dimensional spine ultrasound image. The specific method of fusion is: based on the position of each probe relative to the spine, the real-time ultrasound image collected by the probe is extracted for feature points such as spinous process, vertebral plate, and transverse process, and the extracted feature points are matched one by one with the three-dimensional spine image through feature matching and fusion methods, so as to realize the fusion matching of the two-dimensional ultrasound image and the three-dimensional spine image.

[0070] Lenke classification is mainly for idiopathic scoliosis and plays an important role in diagnosing scoliosis and designing treatment. Lenke classification consists of three parts: coronal scoliosis type (1-6), lumbar modified type (A, B, C), and thoracic sagittal sequence modified type (-, N, +).

[0071] When performing Lenke classification, the type of curve must first be determined based on the apical vertebral interval. Table 1 shows the correspondence between the thoracic and lumbar curve types and the apical vertebral interval.

[0072] Table 1 Correspondence between thoracic and lumbar curve types and apical vertebral intervals

[0073] Chest and waist curve type Apical vertebral area Proximal thoracic (PT) T2-T5 Main thoracic (MT) T6-T11 / 12 Thoracolumbar (TL) T12, L1 Lumbar (L) L1 / 2-L4

[0074] After determining the type of curve, judge the major and minor curves, structural curves and non-structural curves, and then refer to Table 2 to determine the type of coronal scoliosis (1-6).

[0075] Table 2 Lenke coronal scoliosis type (1-6)

[0076] type Upper chest curve Main thoracic curve Thoracolumbar bend / lumbar bend Bending Type 1 Non-structural Structural* Non-structural Main thoracic curve 2 Structural Structural* Non-structural Double chest bend 3 Non-structural Structural* Structural Double main bend 4 Structural Structural* Structural* Three main bends 5 Non-structural Non-structural Structural* Thoracolumbar bend / lumbar bend 6 Non-structural Structural Structural* Thoracolumbar curve / lumbar curve-main thoracic curve

[0077] Note: * indicates the main bend position

[0078] The specific steps of calculating the Lenke classification parameters on the computer software in step S7 according to the fused image in S6 are as follows:

[0079] S71) obtaining the Cobb angle of the coronal plane and the Cobb angle of the sagittal plane when standing upright according to the fused image of the upright standing posture, and determining the position of the apical vertebra;

[0080] S72) obtaining a change of the Cobb angle during lateral flexion according to the fused image during lateral flexion;

[0081] S73) judging the type of curve, primary curve, secondary curve, structural curve and non-structural curve according to the apical vertebra position of step S71) and the Cobb angle of step S72), and determining the type of scoliosis of Lenke classification in combination with Lenke classification principle;

[0082] S74) The modified lumbar spine type of Lenke classification is obtained based on the relative relationship between the position of the top vertebra of the lumbar curve and the vertical line of the center of the sacrum; S75) The modified sagittal sequence type of the thoracic spine of Lenke classification is determined based on the Cobb angle of the sagittal plane when standing upright.

[0083] Decisions on surgical and non-surgical treatment of scoliosis require a comprehensive static and dynamic assessment of the spine, which mainly depends on ① the overall balance of the spine, real-time lateral flexion range of motion, spinal flexibility and stability; ② the degree of openness of the intervertebral space near the apical vertebra and distal vertebra during lateral flexion; ③ the improvement of the degree of vertebral rotation in the apical vertebrae.

[0084] Dynamic three-dimensional ultrasound adds time tracking to three-dimensional ultrasound. Through dynamic three-dimensional ultrasound images, the continuous changes of each vertebra during the lateral flexion of the patient can be observed, and the scoliosis assessment parameters can be measured in real time. In step S8, the real-time range of motion, flexibility and stability of lateral flexion are analyzed, and the degree of openness of the intervertebral space near the top vertebra and the upper and lower end vertebrae and the rotation of the vertebral body are evaluated during lateral flexion. These are all dynamic comprehensive assessments of the spine based on the dynamic three-dimensional ultrasound images constructed in step S6. The results of the dynamic comprehensive assessment can be formed into a written report to provide a basis for diagnosis for doctors.

[0085] Machine learning can be classified into supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. according to the learning form. The difference is that supervised learning requires a labeled sample set, unsupervised learning does not require a labeled sample set, semi-supervised learning requires a small number of labeled samples, and reinforcement learning requires a feedback mechanism.

[0086] Through deep learning of a large number of dynamic three-dimensional ultrasound images acquired after fusion in step S6 and their corresponding scoliosis assessment parameters, the dynamic three-dimensional ultrasound images acquired after fusion in step S6 are automatically identified and the scoliosis assessment parameters are automatically calculated.

[0087] In one embodiment of the present invention, deep learning includes the recognition of markers in two-dimensional spinal cross-sectional ultrasound images within the detection range frame. Specifically, after collecting a large number of spinal cross-sectional ultrasound images, the spinous processes and bilateral lamina of the corresponding frames are manually marked; the marked frames are used as the training set of the supervised deep learning algorithm, and through training with a large amount of data, the goal of simulating manual identification of markers and marking is achieved, so as to automatically calculate the rotation degree of the vertebral body and realize the real-time activity tracking of the vertebral body during spinal bending movements. The morphology of the spinous processes and the inclination degree of the lamina on both sides in the two-dimensional spinal cross-sectional ultrasound image play an important role in judging the spatial rotation angle of the vertebral body itself, as well as the real-time activity, flexibility and stability of the vertebral body during lateral flexion. Figure 5 The figure shows a two-dimensional cross-sectional ultrasound image of the spine, in which the spinous process and the lamina on both sides are clearly identifiable. Therefore, the embodiment of the present invention uses a deep learning algorithm to automatically identify the spinous process and lamina in the two-dimensional cross-sectional ultrasound image of the spine. After collecting a large number of cross-sectional ultrasound images of the spine, the spinous process and the lamina on both sides of the corresponding frames are manually marked; the marked frames are used as training sets for the supervised deep learning algorithm. Through training with a large amount of data, the goal of simulating manual identification of markers and marking is achieved, thereby automatically calculating the rotation of the vertebral body and realizing real-time activity tracking of the vertebral body during lateral flexion.

[0088] In one embodiment of the present invention, deep learning also includes angle measurement of three-dimensional spinal volume projection imaging ultrasound images. Specifically, it means that after collecting a large number of three-dimensional spinal volume projection imaging ultrasound images, the transverse processes on both sides of the end vertebrae of each curve are manually marked as the training set of the supervised deep learning algorithm. Through the training of a large amount of data, the goal of simulating the manual identification of markers and marking is achieved, so as to automatically calculate the angle of each segment of the patient's scoliosis curve, and then realize the real-time activity tracking of the change of the scoliosis curve angle during lateral flexion. Clinically, three-dimensional spinal volume projection imaging ultrasound images are used to measure the scoliosis angle (analogous to the X-ray Cobb angle). When a scoliosis patient has one or more spinal deformity scoliosis curves, the angle formed by the end vertebrae at both ends of each curve is used to mark the severity of the curve. In three-dimensional spinal volume projection imaging ultrasound images, the current manual marking method can be quantified by the transverse processes on both sides of the end vertebrae or the angle of the line connecting the projection of the lamina of the upper lumbar vertebra and the upper articular process of the lower lumbar vertebra. Such as Figure 6 (a) shows the schematic diagram of the connection of the transverse processes on both sides of the end vertebra in the three-dimensional spine volume projection imaging image. Figure 6(b) shows the connection diagram of the transverse processes on both sides of the end vertebra of the X-ray film. After collecting a large number of three-dimensional spinal volume projection ultrasound images, the transverse processes on both sides of the end vertebra of each curve are manually marked as the training set of the supervised deep learning algorithm. Through the training of a large amount of data, the goal of simulating manual identification of markers and marking is achieved, so as to automatically calculate the angle of each segment of the patient's scoliosis curve, and then realize the real-time activity tracking of the change of the scoliosis curve angle during lateral flexion.

[0089] In one embodiment of the present invention, deep learning also includes real-time measurement of the spatial position change of the ultrasound probe. Specifically, it refers to the use of an unsupervised learning algorithm to automatically identify, track and record the position of the ultrasound probe using optical markers. This embodiment uses a depth camera to capture in real time and record the spatial position change of the ultrasound probe, thereby establishing a spatial coordinate system for the detected intraspinal markers. Figure 7 The figure shows the effect of identifying and marking the position of the ultrasound probe in the fusion image of the depth camera's back depth image and the three-dimensional ultrasound volume projection imaging. Since the relative position change of the ultrasound probe in the body surface space coordinate system is different from the anatomical landmarks observed in the body, and its shape and material are also different from the human body structure and easy to identify, an unsupervised learning algorithm can be used to use optical markers to automatically identify, track and record the position of the ultrasound probe.

[0090] Since medical images have the characteristics of blur, unevenness, individual differences, complexity and diversity, the present invention uses deep learning algorithms to process medical images, which is more advantageous than traditional algorithms. However, because medical images involve patient privacy and are difficult to obtain, the limited sample data set obtained needs to be expanded.

[0091] The method and system of the present invention, based on the use of a mature three-dimensional spinal ultrasound imaging system, cooperates with a wearable wireless ultrasound probe to collect real-time ultrasound images, and a depth camera with a built-in optical sensor to record the spatial coordinate changes of the characteristic points of the spine when the patient is lateralized, and fuses the three-dimensional ultrasound images of the upright and bent states collected by the three-dimensional spinal ultrasound imaging system, the real-time ultrasound images collected by the wearable wireless ultrasound system, and the dynamic three-dimensional images of the back morphology collected by the depth camera to obtain a dynamic three-dimensional ultrasound spinal image. According to the dynamic three-dimensional ultrasound spinal image, the computer vision of deep learning is combined with real-time feedback and rapid and accurate measurement of the Lenke classification parameters of scoliosis, and the overall balance of the spine, the real-time range of motion of the spinal lateral flexion, the flexibility and stability of the spine, the degree of opening of the intervertebral space near the top vertebra and the end vertebra during lateral flexion, and the improvement of the degree of rotation of the vertebral body in the top vertebra area are dynamically and comprehensively evaluated, so that the classification of scoliosis is upgraded from the traditional X-ray plane assessment method based on radiation hazards to the dynamic three-dimensional ultrasound analysis without radiation, solving the problems of high error rate and insufficient acquisition parameters of the data collected in static conditions in the prior art. The present invention can also be applied to spinal flexion and other spinal curvature tests as required, and the specific process is similar to that of lateral flexion.

[0092] The above are only specific implementations of the present invention, which cannot be used to limit the scope of the present invention. Equivalent changes made by ordinary technicians in this technical field based on this creation, as well as changes known to technicians in this field, should still fall within the scope of the present invention.

Claims

1. A dynamic three-dimensional ultrasonic spine detection method, characterized in that: Using a computer terminal (4) that is communicatively connected to a three-dimensional spinal ultrasound imaging device (1), a wearable wireless ultrasound scanning unit (2), and a depth camera (3) with a built-in optical sensor, the method comprises the following steps: S1) using the three-dimensional spine ultrasound imaging device (1) to perform a three-dimensional ultrasound scan on the patient's spine to obtain a three-dimensional spine image (100), including spine projection images in the coronal plane and sagittal plane; S2) obtaining the positions of one or more main apical vertebrae and upper and lower end vertebrae of scoliosis from the coronal spine projection image in the three-dimensional spine image (100), and fixing the ultrasonic probe of the wearable wireless ultrasonic scanning unit (2) at the obtained positions of the apical vertebrae, upper end vertebrae and lower end vertebrae respectively based on the complete reflection surfaces of the vertebral plates on both sides seen in the field of view of the ultrasonic image, so as to collect real-time ultrasonic images (200) at corresponding positions, and the outer end surface of the ultrasonic probe is provided with an optical marker; S3) pointing a depth camera (3) with a built-in optical sensor toward the patient's back to obtain a real-time video including a dynamic back image (300) of multiple frames of back depth maps, optical markers, and body surface maps; S4) the patient bends the trunk according to the instructions until the required curvature is reached, and the computer terminal collects in real time a dynamic back image (300) including a back depth map and a body surface map and a real-time ultrasound image (200) during the spinal bending action; the body surface map is a color map or a grayscale map; the bending of the trunk according to the instructions includes lateral flexion, forward flexion or any required examination posture, or bending to any curvature suitable for examination; S5) using the optical markers in the body surface image to obtain the real-time three-dimensional spatial position and direction of each of the ultrasound probes during the spinal flexion movement; S6) the computer terminal (4) synchronizes and fuses the real-time ultrasound image (200) with the back depth map in the dynamic back image (300) and the three-dimensional spine image (100) to obtain a dynamic three-dimensional ultrasound image during the spine bending movement; wherein, during the image synchronization and fusion processing, the real-time three-dimensional spatial position and direction of the ultrasound probe obtained in step S5) is used as reference information; S7) On a computer terminal, a spinal curvature morphology analysis is performed based on the dynamic three-dimensional ultrasound image fused in step S6), including calculation of Lenke classification parameters, analysis of the real-time range of motion, flexibility and stability of the scoliosis, and evaluation of the degree of opening of the intervertebral spaces near the top vertebra and the upper and lower end vertebrae and the rotation of the vertebral bodies during lateral flexion.

2. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: The step S1 includes using a three-dimensional spine ultrasound imaging device (1) to perform a spine ultrasound scan on a patient in an upright standing position, and obtaining a three-dimensional spine image (100) of the patient in an upright standing position, including spine projection images in the coronal plane and sagittal plane.

3. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: The step S1 comprises using a three-dimensional spinal ultrasound imaging device (1) to perform an ultrasound scan of the patient's spine in one or more different curvature postures, and obtaining a three-dimensional spinal image (100) of the patient in one or more different curvature postures, including spinal projection images in the coronal plane and the sagittal plane.

4. The dynamic three-dimensional ultrasonic spine detection method according to claim 3, characterized in that: The spinal curvature posture is the dynamic three-dimensional shape of the spine when the patient performs lateral flexion, forward flexion, or other bending movements for spinal assessment.

5. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: Between step S6) and step S7), there is also step S6a), if there is more than one curved part obtained from the coronal spine projection image in step S2, then for each other curved part, steps S2-S6 are repeated to collect data.

6. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: The step S2 of obtaining the positions of the top vertebra and the upper and lower end vertebrae of the main curvature of the scoliosis from the coronal spine projection image in the three-dimensional spine image (100) comprises the following steps: S21) numbering each vertebra; S22) determining the number of main bends; S23) For each bend, identify and record the numbers of the top vertebra, upper and lower vertebrae.

7. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: The specific steps of fusing and synchronizing the real-time ultrasound image (200) with the back depth map in the dynamic back image (300) in step S6 are: S61) identifying the optical marker on each of the ultrasound probes by an image recognition algorithm; S62) obtaining the real-time three-dimensional spatial position and direction of each of the ultrasound probes through image analysis; S63) fusing the real-time ultrasound image collected by each probe with the back depth map according to the position and direction information; The fusion of the real-time ultrasound image (200) and the three-dimensional spine image (100) is based on the position of each ultrasound probe relative to the spine. The real-time ultrasound image collected by the ultrasound probe is fused with the three-dimensional spine image, so that the real-time spine image characteristics of the local area corresponding to the ultrasound probe and the overall spine information of the three-dimensional spine image are fused to obtain a dynamic three-dimensional spine ultrasound image.

8. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: The specific steps of calculating the Lenke classification parameters on the computer terminal according to the fused image in S6 in step S7 are as follows: S71) obtaining the Cobb angle of the coronal plane and the Cobb angle of the sagittal plane when standing upright according to the fused image of the upright standing posture, and determining the position of the apical vertebra; S72) obtaining a change of the Cobb angle during lateral flexion according to the fused image during lateral flexion; S73) judging the primary and secondary curves, structural curves and non-structural curves according to the apical vertebra position in step 71) and the Cobb angle in step 72), and determining the type of scoliosis according to Lenke classification in combination with Lenke classification principles; S74) The modified lumbar spine type of Lenke classification is obtained based on the relative relationship between the position of the top vertebra of the lumbar curve and the vertical line of the sacrum center; S75) Determine the modified sagittal alignment of the thoracic spine according to the Lenke classification based on the Cobb angle in the sagittal plane when standing upright.

9. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: It also includes step S8) automatically identifying the dynamic three-dimensional ultrasound images fused in step S6 and automatically calculating the scoliosis assessment parameters through deep learning of a large number of dynamic three-dimensional ultrasound images fused in step S6 and their corresponding scoliosis assessment parameters.

10. The dynamic three-dimensional ultrasonic spine detection method according to claim 9, characterized in that: The deep learning described in step S8 includes the recognition of markers in two-dimensional spinal cross-sectional ultrasound images within the detection range frame, including: after collecting a large number of spinal cross-sectional ultrasound images, manually marking the spinous processes and bilateral lamina of the corresponding frames; using the marked frames as the training set of the supervised deep learning algorithm, and through training with a large amount of data, achieving the goal of simulating manual identification of markers and marking, thereby automatically calculating the rotation degree of the vertebral body, and realizing real-time activity tracking of the vertebral body during spinal bending movements.

11. The dynamic three-dimensional ultrasonic spine detection method according to claim 9, characterized in that: The deep learning in step S8 includes angle measurement of three-dimensional spinal volume projection imaging ultrasound images, including collecting a large number of three-dimensional spinal volume projection ultrasound images, marking the transverse processes on both sides of each curved end vertebra, or the vertebral plate of the upper lumbar vertebra and the projection of the upper articular process of the lower lumbar vertebra, and training the marked images to achieve the goal of simulating manual identification of markers and marking them, thereby automatically calculating the angle of each segment of the patient's scoliosis curve, and then realizing real-time activity tracking of the change in the angle of the scoliosis curve during lateral flexion.

12. The dynamic three-dimensional ultrasonic spine detection method according to claim 9, characterized in that: The deep learning in step S8 includes real-time measurement of the spatial position change of the ultrasound probe, including adopting an unsupervised learning algorithm and using optical markers to automatically identify, track and record the position of the ultrasound probe.

13. The dynamic three-dimensional ultrasonic spine detection method according to any one of claims 1 to 12, characterized in that: The ultrasonic probe (2) comprises a patch, a cable, an adhesive fixing device and a wireless controller. The wireless controller transmits the patch ultrasonic signal transmitted by the cable to a nearby computer terminal (4) via WiFi.

14. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: The ultrasonic probe is a patch ultrasonic probe.

15. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: The ultrasonic probe is a flexible ultrasonic probe.

16. The dynamic three-dimensional ultrasonic spine detection method according to claim 1, characterized in that: The ultrasonic probe is a one-dimensional array or a two-dimensional array ultrasonic probe.

17. The dynamic three-dimensional ultrasonic spine detection method according to claim 6, characterized in that: The depth camera in step S3 is located on a bracket, which can control the depth camera to move in six directions: forward, backward, up, down, left, and right, so that the back of the human body is always within the field of view of the camera when the spine is bent.

18. A dynamic three-dimensional ultrasound spine imaging system, comprising a three-dimensional spine ultrasound imaging device (1), characterized in that: Also includes: A wearable wireless ultrasound scanning unit (2), used for generating a local two-dimensional ultrasound image frame of the spine, comprising three or more ultrasound probes; A depth camera unit (3), used to record the spatial position change of the ultrasound probe and the dynamic three-dimensional shape of the back, the depth camera unit comprising a depth camera with a built-in optical sensor; The computer terminal (4) is connected to the three-dimensional spinal ultrasound imaging device (1), the wearable wireless ultrasound scanning unit (2), and the depth camera unit (3), and is used to receive the three-dimensional spinal image (100) of the three-dimensional spinal ultrasound imaging device (1), the real-time ultrasound image (200) of the wearable wireless ultrasound scanning unit (2), and the dynamic back image (300) of the depth camera unit (3), and to perform spinal curvature morphology analysis by fusing the three-dimensional spinal image (100), the real-time ultrasound image (200), and the dynamic back image (300), including measuring the Lenke classification parameters of scoliosis, and performing dynamic evaluation of the spine in lateral flexion activities.

Images