A fully automatic spine ultrasound scanning method based on multi-source vision

Abstract

The present invention discloses a fully automatic spinal ultrasound scanning method based on multi-source vision, which comprises the following steps: using an RGB-D sensor to obtain an RGB image and a depth image of the human back, obtaining a three-dimensional point cloud of the human back area, and calculating the total number of scanning points J and the initial scanning point position; analyzing the ultrasound image through a multi-task deep neural network to obtain view classification information and vertebral key point information of the ultrasound image; calculating the j+1th scanning point position and the robotic arm posture based on the ultrasound image obtained by placing the ultrasound probe at the jth scanning point position based on a multi-vision fusion algorithm; the robotic arm scans the j+1th scanning point according to the calculation result until the scanning of J scanning points is completed; and displaying the spinal information obtained by the automatic scanning after the scanning is completed. The method of the present invention realizes the fusion of multi-source visual information of RGB-D images and ultrasound images, realizes the full automation of the spinal ultrasound scanning process, and can provide clear and rich spinal information.

Description

Technical Field

[0001] The present invention relates to the field of fully automatic ultrasonic scanning, in particular to fully automatic ultrasonic scanning of the human spine, and more specifically to a fully automatic spine ultrasonic scanning method based on multi-source vision. Background Art

[0002] The spine is a vital structure that supports human movement. Currently, commonly used imaging methods for observing the spine include X-ray imaging, CT imaging, and ultrasound imaging.

[0003] While X-ray and CT imaging provide clear images, they inevitably expose the human body to ionizing radiation. Ultrasound imaging, due to its radiation-free and convenient nature, has also been used for spinal scanning and status observation. Free-arm ultrasound scanning of the spine requires a highly trained and qualified ultrasound physician to scan the spine using a handheld probe. This repetitive operation increases the risk of occupational illness for physicians. Therefore, the use of a spinal ultrasound robotic system to assist physicians in automatically completing spinal scans is crucial.

[0004] Currently, robotic ultrasound scanning of the spine is primarily achieved through single-source vision, with two main approaches: 1) Using an RGB-D sensor to capture images of the human back, the scanning path is automatically pre-planned based on the indentations in the spinal region of the back. The depth image is then used to determine the ultrasound probe's posture during scanning, completing the spinal scan. This method relies on the subject's body shape and the visibility of the spinal indentations, potentially resulting in incomplete or erroneous scanning paths. Furthermore, the ultrasound imaging quality of the spine cannot be assessed during the scan, potentially leading to loss of view of the spine. 2) Using ultrasound images to automatically locate the vertebrae during scanning, combined with force sensors to determine the ultrasound probe's posture, completing the spinal scan. However, this method requires manual determination of the initial scanning point, preventing fully automated scanning. Furthermore, it ignores the rich three-dimensional information of the human back and complicates the probe posture calculation process.

[0005] In summary, the current robotic ultrasound scanning method for the spine has certain limitations and cannot fully utilize multi-source visual information to achieve fully automatic ultrasound scanning of the spine. Summary of the Invention

[0006] The purpose of the present invention is to solve the above-mentioned defects in the prior art and to provide a fully automatic spinal ultrasound scanning method based on multi-source vision, which integrates multi-source visual information such as RGB images, depth images and ultrasound images to complete the navigation of automatic spinal ultrasound scanning.

[0007] The purpose of the present invention can be achieved by taking the following technical solutions:

[0008] A fully automatic spinal ultrasound scanning method based on multi-source vision is applied to a system consisting of an RGB-D sensor, a six-degree-of-freedom robotic arm, an ultrasound platform, and a personal computer. The ultrasound scanning method includes the following steps:

[0009] S1. Use the TCP / IP protocol to establish communication connections between the robotic arm and the personal computer, and between the ultrasound platform and the personal computer. Establish a coordinate system for the RGB-D sensor and define it as coordinate system K. Establish a coordinate system for the RGB image captured by the RGB-D sensor and define it as coordinate system I. Establish a coordinate system for the ultrasound image and define it as coordinate system B. Establish a coordinate system for the base of the robotic arm and define it as coordinate system R. Establish a coordinate system for the end of the robotic arm and define it as coordinate system G. Use the robotic arm to clamp the probe in the ultrasound platform, and complete the calibration between the probe and the end of the robotic arm, and between the RGB-D sensor and the base of the robotic arm.

[0010] S2. Use the RGB-D sensor to obtain the RGB image and depth image of the human back, obtain the three-dimensional point cloud of the human back area based on the RGB image and depth image of the human back, and calculate the total number of scanning points J and the initial scanning point position That is, the scan needs to be performed at J scanning points in total Scan at

[0011] S3. After the scan begins, the ultrasound image is analyzed using a multi-task deep neural network (VGG11-FPN) to perform view classification and vertebral key point detection on the ultrasound image, obtaining view category information of the ultrasound image and the coordinates of the vertebral key points in the ultrasound image.

[0012] S4. Based on the ultrasound image obtained by placing the ultrasound probe at the j-th scanning point position, the j+1-th scanning point position and the robotic arm posture are calculated based on the multi-source vision fusion algorithm. The robotic arm scans the j+1-th scanning point according to the calculation results until the scanning of the J scanning points is completed;

[0013] S5. After the scanning is completed, the position of each scanning point and the posture of the robotic arm at each scanning point are comprehensively analyzed. The view classification information and vertebral key point detection information of the ultrasound image at each scanning point show the spinal column information obtained by automatic scanning.

[0014] Furthermore, in step S1, according to the cross-line experiment, the transformation matrix from coordinate system B to coordinate system G is calculated. G T B Complete the calibration between the probe and the end of the robotic arm; according to the chessboard calibration and Levenberg-Marquardt algorithm, calculate the transformation matrix from coordinate system R to coordinate system K K T RAnd the transformation matrix from coordinate system K to coordinate system R R T K Complete the calibration between the RGB-D sensor and the base of the robotic arm; use the focal length of the infrared camera in the RGB-D sensor to calculate the transformation matrix from coordinate system I to coordinate system K K T I And the transformation matrix from coordinate system K to coordinate system I I T K ; Read the transformation matrix from coordinate system G to coordinate system R from the robot controller R T G ;

[0015] The conversion from the ultrasound image coordinate system B to the RGB image coordinate system I is achieved by the following formula:

[0016] P I =(x I ,y I )= I T K K T R R T G G T B P B

[0017] Among them, P B Represents the coordinates of a point in coordinate system B, P I Indicates P B The coordinate after conversion to coordinate system I, x I P I The coordinate on the x-axis in coordinate system I, y I P I The coordinate on the y-axis in coordinate system I;

[0018] The conversion between the RGB image coordinate system I and the robotic arm base coordinate system R is achieved through the following formula:

[0019] P R = R T K K T I P I

[0020] Among them, P I represents the coordinates in coordinate system I, P R Indicates P I The coordinates after conversion to coordinate system R.

[0021] After the communication connection and calibration are completed, the components in the system can smoothly work together by converting coordinates in each coordinate system.

[0022] Furthermore, in step S2, a pixel set with color components R>G>B in the RGB image of the back of the human body is extracted, the largest connected area in the pixel set is calculated, and the largest connected area is used as the back area of ​​the human body, and the transformation matrix is ​​used to calculate the maximum connected area. K T I The back area of ​​the human body is converted into a three-dimensional point cloud under the coordinate system K. 4. The fully automatic spine ultrasound scanning method based on multi-source vision according to claim 3 is characterized in that x min 、x max 、y min and y max Respectively represent the minimum and maximum coordinates of the back area of ​​the human body on the x-axis and y-axis in the coordinate system I, through Calculate the initial scanning point in coordinate system I Then, the initial scanning point in coordinate system I is converted to the initial scanning point in coordinate system R through the conversion relationship between coordinate system I and coordinate system R. Thus, the initial scanning point position is obtained, and finally J=x max -x min Calculate the total number of scan points. This step fully utilizes the color characteristics of the human back to locate the human back area in the RGB image, allowing the robotic arm to accurately and automatically move to the human back area to begin scanning.

[0023] Furthermore, the multi-task VGG11-FPN deep neural network is used to automatically identify the location of vertebrae in ultrasound images and determine the view category of the spinal ultrasound image as one of three view categories: "left-biased," "center-biased," or "right-biased." Simultaneously, three vertebral key points are detected: the vertex of the vertebral center, the left transverse process of the thoracic vertebra or the left articular process of the lumbar vertebra, and the right transverse process of the thoracic vertebra or the right articular process of the lumbar vertebra, to obtain the coordinates of these three vertebral key points in coordinate system B. The vertebral key point detection results obtained by the multi-task deep neural network can be used in multi-source visual fusion and, combined with the view classification results, for automatic display of spinal scan results.

[0024] Furthermore, the multi-task deep neural network Multi-task VGG11-FPN includes a shared part, a view classification part and a vertebral key point detection part, wherein the shared part includes a first feature extraction module, a first pooling layer, a second feature extraction module, a second pooling layer, a third feature extraction module and a third pooling layer connected in sequence, and the third pooling layer is also connected to the fourth feature extraction module of the view classification part and the sixth feature extraction module of the vertebral key point detection part respectively; the view classification part includes a fourth feature extraction module, a fourth pooling layer, a fifth feature extraction module, a fifth pooling layer, a sixth pooling layer, a fully connected layer and a Softmax layer connected in sequence; the vertebral key point detection part includes a sixth feature extraction module, a seventh pooling layer ... a seventh feature extraction module, an eighth pooling layer, an eighth feature extraction module, a first upsampling layer, a first fuser, a second upsampling layer, a second fuser, a third upsampling layer, a third fuser, a ninth feature extraction module, and a fourth upsampling layer; the vertebral key point detection part further includes a first feature fusion layer, a second feature fusion layer, and a third feature fusion layer, wherein the input and output of the first feature fusion layer are respectively connected to the second pooling layer and the third fuser; the input and output of the second feature fusion layer are respectively connected to the third pooling layer and the second fuser; the input and output of the third feature fusion layer are respectively connected to the seventh pooling layer and the first fuser; at the same time, the eighth feature extraction module, the first and second fusers are also respectively connected to the ninth feature extraction module;

[0025] The first and second feature extraction modules of the multi-task deep neural network Multi-task VGG11-FPN have the same structure, both of which are convolution layers with a convolution kernel size of 3*3, and both use the ReLU nonlinear activation function; the third, fourth, fifth, sixth, seventh and ninth feature extraction modules have the same structure, both include two convolution layers with a convolution kernel size of 3*3 connected in series, and both use the ReLU nonlinear activation function; the first, second, third, fourth, fifth, seventh and eighth pooling layers are all maximum pooling layers with a size of 2*2; the sixth pooling layer is an average pooling layer with a size of 7*7; the first, second, third and fourth upsampling modules are all bilinear interpolation upsampling layers; the first, second and third feature fusion layers and the eighth feature extraction module are all convolution layers with a size of 1*1; the first, second and third fusion devices respectively realize the fusion of the two input feature maps, that is, the pixel values ​​of the corresponding points of the two feature maps are added. The above-mentioned multi-task deep neural network integrates the features of different tasks and simplifies the network structure by sharing the common parts in the network, thereby reducing the time cost of the network operation in the scanning process without affecting the performance of the two tasks and improving the efficiency of the entire scanning.

[0026] Furthermore, after the ultrasound image is input into the network, it will first pass through the shared part, then the view classification part and the vertebral key point detection part at the same time. The softmax layer outputs the predicted probabilities of the three view categories, and the view category with the highest probability is used as the prediction result. The fourth upsampling layer outputs three predicted images. The pixel value of each pixel in the predicted image is between 0 and 1. The pixel coordinates of the maximum pixel value in the three images are the coordinates of the three vertebral key points respectively.

[0027] Furthermore, in step S4, after the robotic arm scans the j-th scanning point, the ultrasonic image of the j-th scanning point is input into the multi-task deep neural network Multi-task VGG11-FPN to obtain the coordinates of the vertex of the vertebral center in the ultrasonic image of the j-th scanning point in coordinate system B. The robot arm position of the j+1th scanning point is calculated using a multi-source vision fusion algorithm. The specific process is as follows:

[0028]

[0029] Among them, P Ij for Coordinates in coordinate system I, x Ij P Ij The coordinate on the x-axis in coordinate system I, y Ij P Ij The coordinate on the y-axis in coordinate system I;

[0030] P I(j+1) =(x I(j+1) ,y I(j+1) )=(x Ij -1,y Ij )

[0031] Among them, P I(j+1) is the coordinate of the j+1th scanning point in coordinate system I, x I(j+1) P I(j+1) The coordinate on the x-axis in coordinate system I, y I(j+1) P I(j+1) The coordinate on the y-axis in coordinate system I;

[0032]

[0033] in, is the position of the robotic arm at the j+1th scanning point in the coordinate system R.

[0034] Through a multi-source visual fusion algorithm, the vertebral key point information provided in the RGB image and ultrasound image is fused to calculate the position and posture of the robotic arm in real time, thereby tracking the position of the spinal vertebrae in real time during the scanning process to ensure that the probe does not deviate from the spinal area.

[0035] Furthermore, in step S4, the back three-dimensional point cloud provided by step S2 is used to Select three points around to form a plane, and calculate the normal vector of the plane. Make the probe parallel to the normal vector and the end of the probe perpendicular to the scanning path, so as to obtain the posture of the robotic arm at the j+1th scanning point The robotic arm begins scanning the j+1th scanning point until it has completed scanning points J. The robotic arm posture calculated based on the depth information of the human back can enable the probe to be tightly coupled to the human skin, improving the imaging quality of the ultrasound image.

[0036] Furthermore, in step S5, the vertebral key points of the ultrasound image classified as "center" by the view are extracted, and the corresponding scanning point positions and robotic arm postures are converted from coordinate system B to coordinate system I and coordinate system R to complete the two-dimensional and three-dimensional visualization of the key structures of the spine.

[0037] The present invention has the following advantages and effects compared to the prior art:

[0038] 1. This invention integrates multi-source visual information from RGB images, depth images, and ultrasound images to accurately and fully automatically track the human spine, thereby enabling ultrasound scanning of the spine. Compared to using ultrasound image information alone, this invention improves the degree of scanning automation. Compared to using RGB and depth images alone, this invention can automatically analyze ultrasound image information in real time, ensuring that the scan does not deviate from the spine, thereby improving the quality of ultrasound imaging during automatic scanning.

[0039] 2. The present invention can provide two-dimensional and three-dimensional information of the spine based on the information collected during the scan. There is no need to complete three-dimensional reconstruction of the spinal ultrasound image, which reduces the amount of calculation during reconstruction. The extraction and display of key points of the spinal vertebrae also makes the provided spinal information clearer and richer. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0041] Figure 1 1 is a system structure diagram of a fully automatic spinal ultrasound scanning method based on multi-source vision in an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of the coordinate system of each component of the system constructed in an embodiment of the present invention;

[0043] Figure 3is a schematic diagram of determining the initial scanning point position and the total number of scanning points in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the multi-task deep neural network Multi-task VGG11-FPN;

[0045] Figure 5 is a schematic diagram of view categories of spinal ultrasound images, where Figure 5 (a) is a diagram of the view category of the vertebra "left-leaning". Figure 5 (b) is a diagram of the view category of the vertebrae "median", Figure 5 (c) is a diagram of the view categories of the vertebrae "biased to the right";

[0046] Figure 6 is a schematic diagram of the key points of the vertebrae in the ultrasound images of the thoracic and lumbar spine, where Figure 6 (a) is a schematic diagram of the key points of the vertebrae in the thoracic ultrasound image. Figure 6 (b) is a schematic diagram of vertebral key points in the lumbar ultrasound image;

[0047] Figure 7 It is a two-dimensional and three-dimensional visualization of the key structures of the spine when the scanned object is lying normally, among which, Figure 7 (a) is a two-dimensional visualization of the key structures of the spine. Figure 7 (b) is a three-dimensional visualization of the key structures of the spine;

[0048] Figure 8 It is a 2D and 3D visualization of the key structures of the spine when the scanned subject's body is tilted, Figure 8 (a) is a two-dimensional visualization of the key structures of the spine. Figure 8 (b) is a three-dimensional visualization of the key structures of the spine. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] Example

[0051] This embodiment discloses a fully automatic spinal ultrasound scanning method based on multi-source vision, which is applied to Figure 1The system shown in FIG. 1 is composed of an RGB-D sensor, a six-degree-of-freedom robotic arm, an ultrasound platform, and a personal computer. Based on the system, a fully automated spinal ultrasound scanning method includes the following steps:

[0052] S1. Use TCP / IP protocol to realize the communication connection between the robot arm and the personal computer, and between the ultrasound platform and the personal computer. Figure 2 As shown, a coordinate system is established for the RGB-D sensor and defined as coordinate system K; a coordinate system is established for the RGB image captured by the RGB-D sensor and defined as coordinate system I; a coordinate system is established for the ultrasound image and defined as coordinate system B; a coordinate system is established for the base of the robotic arm and defined as coordinate system R, and a coordinate system is established for the end of the robotic arm and defined as coordinate system G. The robotic arm is used to clamp the probe in the ultrasound platform, and calibration is completed between the probe and the end of the robotic arm, and between the RGB-D sensor and the base of the robotic arm.

[0053] S2. Use the RGB-D sensor to obtain the RGB image and depth image of the human back, obtain the three-dimensional point cloud of the human back area based on the RGB image and depth image of the human back, and calculate the total number of scanning points J and the initial scanning point position That is, the scan needs to be performed at J scanning points in total Scan at

[0054] S3. After the scan begins, the ultrasound image is analyzed using a multi-task deep neural network (VGG11-FPN) to perform view classification and vertebral key point detection on the ultrasound image, obtaining view classification information of the ultrasound image and the coordinates of the vertebral key points in the ultrasound image.

[0055] S4. Based on the ultrasound image obtained by placing the ultrasound probe at the j-th scanning point position, the j+1-th scanning point position and the robotic arm posture are calculated based on the multi-source vision fusion algorithm. The robotic arm scans the j+1-th scanning point according to the calculation results until the scanning of the J scanning points is completed;

[0056] S5. After the scanning is completed, the position of each scanning point and the posture of the robotic arm at each scanning point are comprehensively analyzed. The view classification information and vertebral key point detection information of the ultrasound image at each scanning point show the spinal column information obtained by automatic scanning.

[0057] In this embodiment, in step S1, according to the cross line experiment, the transformation matrix from coordinate system B to coordinate system G is calculated. G T BComplete the calibration between the probe and the end of the robotic arm; establish a nonlinear equation system for calculating the transformation matrix based on the chessboard calibration, and then use the Levenberg-Marquardt algorithm used in JJ More's paper "The Levenberg-Marquardt algorithm: Implementation and theory" to solve the nonlinear equation system to calculate the transformation matrix from coordinate system R to coordinate system K K T R And the transformation matrix from coordinate system K to coordinate system R R T K , complete the calibration between the RGB-D sensor and the base of the robotic arm; use the focal length of the infrared camera in the RGB-D sensor to calculate the transformation matrix from coordinate system I to coordinate system K K T I And the transformation matrix from coordinate system K to coordinate system I I T K ; Read the transformation matrix from coordinate system G to coordinate system R from the robot controller R T G .

[0058] The conversion from the ultrasound image coordinate system B to the RGB image coordinate system I is achieved by the following formula:

[0059] P I =(x I ,y I )= I T K K T R R T G G T B P B

[0060] Among them, P B Represents the coordinates of a point in coordinate system B, P I Indicates P B The coordinate after conversion to coordinate system I, x I P I The coordinate on the x-axis in coordinate system I, y I P I The coordinate on the y-axis in coordinate system I;

[0061] The conversion between the RGB image coordinate system I and the robotic arm base coordinate system R is achieved through the following formula:

[0062] P R = R T K K TI P I

[0063] Among them, P I represents the coordinates in coordinate system I, P R Indicates P I The coordinates after conversion to coordinate system R.

[0064] In this embodiment, in step S2, before scanning, the scanned object is placed flat on its back directly below the RGB-D sensor. The RGB-D sensor is used to capture the back of the scanned object. The resolution of the captured RGB image and depth image is 640*480. The pixel set with color components R>G>B in the RGB image of the human back is extracted, and the largest connected area in the above pixel set is calculated. The largest connected area is used as the human back area. The focal length of the infrared camera in the RGB-D sensor is used to convert this area into a three-dimensional point cloud in the K coordinate system. The conversion formula is as follows:

[0065] x K =(x I -320)×z I ÷525

[0066] y K =(y I -240)×z K ÷525

[0067] z K =z I

[0068] Among them, (320,240) is the midpoint coordinate of the RGB image, (x I ,y I ) is the pixel coordinate in the RGB image, z I Is the depth value corresponding to the coordinate in the depth image. (x K ,y K ,z K ) are the three-dimensional coordinates of the point in the K coordinate system. 525 mm is the focal length of the infrared camera used to collect depth information in this RGB-D sensor.

[0069] In this embodiment, Figure 3 Shown with x min 、x max 、y min and y max Respectively represent the minimum and maximum coordinates of the back area of ​​the human body on the x-axis and y-axis in the coordinate system I, through Calculate the initial scanning point in coordinate system I Then, the initial scanning point in coordinate system I is converted to the initial scanning point in coordinate system R through the conversion relationship between coordinate system I and coordinate system R. Thus, the initial scanning point position is obtained, and finally J=x max -x min Calculate the total number of scan points.

[0070] In this embodiment, in step S3, the multi-task deep neural network Multi-task VGG11-FPN is as follows: Figure 4 As shown in the figure, the network is used to automatically identify the location of the vertebra in the ultrasound image and determine the view category of the spinal ultrasound image as follows: Figure 5 One of the three view categories shown in the figure: "Left", "Center", and "Right"; Figure 6 The three vertebral key points above the vertex of the vertebra center, the left transverse process of the thoracic vertebra or the left articular process of the lumbar vertebra, and the right transverse process of the thoracic vertebra or the right articular process of the lumbar vertebra are detected to obtain the coordinates of the three vertebral key points in the coordinate system B.

[0071] In this embodiment, the multi-task deep neural network Multi-task VGG11-FPN includes a common part, a view classification part and a vertebral key point detection part, wherein the common part includes a first feature extraction module, a first pooling layer, a second feature extraction module, a second pooling layer, a third feature extraction module and a third pooling layer connected in sequence, and the third pooling layer is also connected to the fourth feature extraction module of the view classification part and the sixth feature extraction module of the vertebral key point detection part respectively; the view classification part includes a fourth feature extraction module, a fourth pooling layer, a fifth feature extraction module, a fifth pooling layer, a sixth pooling layer, a fully connected layer and a Softmax layer connected in sequence; the vertebral key point detection part includes a sixth feature extraction module, a seventh pooling layer ... a seventh feature extraction module, an eighth pooling layer, an eighth feature extraction module, a first upsampling layer, a first fuser, a second upsampling layer, a second fuser, a third upsampling layer, a third fuser, a ninth feature extraction module, and a fourth upsampling layer; the vertebral key point detection part further includes a first feature fusion layer, a second feature fusion layer, and a third feature fusion layer, wherein the input and output of the first feature fusion layer are respectively connected to the second pooling layer and the third fuser; the input and output of the second feature fusion layer are respectively connected to the third pooling layer and the second fuser; the input and output of the third feature fusion layer are respectively connected to the seventh pooling layer and the first fuser; at the same time, the eighth feature extraction module, the first and second fusers are also respectively connected to the ninth feature extraction module;

[0072] In this embodiment, the first and second feature extraction modules of the multi-task deep neural network Multi-task VGG11-FPN have the same structure, both of which are convolution layers with a convolution kernel size of 3*3, and both use the ReLU nonlinear activation function; the third, fourth, fifth, sixth, seventh, and ninth feature extraction modules have the same structure, both include two convolution layers with a convolution kernel size of 3*3 connected in series, and both use the ReLU nonlinear activation function; the first, second, third, fourth, fifth, seventh, and eighth pooling layers are all maximum pooling layers with a size of 2*2; the sixth pooling layer is an average pooling layer with a size of 7*7; the first, second, third, and fourth upsampling modules are all bilinear interpolation upsampling layers; the first, second, third feature fusion layers and the eighth feature extraction module are all convolution layers with a size of 1*1; the first, second, and third fusion devices respectively realize the fusion of the two input feature maps, that is, the pixel values ​​of the corresponding points of the two feature maps are added.

[0073] In this embodiment, after an ultrasound image is input into the network, it first passes through the shared component, then the view classification component, and the vertebral keypoint detection component. The softmax layer outputs the predicted probabilities of the three view categories, with the view category with the highest probability being used as the prediction result. The fourth upsampling layer outputs three predicted images, where each pixel value in the predicted images is between 0 and 1. The pixel coordinates of the maximum pixel values ​​in the three images are the coordinates of the three vertebral keypoints.

[0074] In this embodiment, 4818 frames of ultrasound images of the spinal region were collected using a curved probe on an ultrasound platform and the categories and vertebral key points were annotated for training the multi-task deep neural network Multi-task VGG11-FPN. When training the classification task part of the network, the cross entropy loss function is used and the learning rate is set to 2E-4. When training the vertebral key point detection task part of the network, the mean square error loss function is used and the learning rate is set to 3E-3. The network will first train the common part and the view classification part. During training, the network layer parameters of the vertebral key point detection part will be locked. After the training is completed, the network layer parameters of the common part and the view classification part will be locked, and then the vertebral key point detection part will be trained. The original resolution of the collected ultrasound image is 322*356. The resolution of the ultrasound image will be reset to 224*224 before entering the network.

[0075] In this embodiment, in step S4, after the robotic arm scans the j-th scanning point, the obtained ultrasound image of the j-th scanning point is input into the multi-task deep neural network Multi-task VGG11-FPN to obtain the coordinates of the vertex of the vertebral center in the ultrasound image of the j-th scanning point in coordinate system B. The robot arm position of the j+1th scanning point is calculated using a multi-source vision fusion algorithm. The specific process is as follows:

[0076]

[0077] Among them, P Ij for Coordinates in coordinate system I, x Ij P Ij The coordinate on the x-axis in coordinate system I, y Ij P Ij The coordinate on the y-axis in coordinate system I;

[0078] P I(j+1) =(x I(j+1) ,y I(j+1) )=(x Ij -1,y Ij )

[0079] Among them, P I(j+1) is the coordinate of the j+1th scanning point in coordinate system I, x I(j+1) P I(j+1) The coordinate on the x-axis in coordinate system I, y I(j+1) P I(j+1) The coordinate on the y-axis in coordinate system I;

[0080]

[0081] in, is the position of the robotic arm at the j+1th scanning point in the coordinate system R.

[0082] In this embodiment, the back 3D point cloud provided in step S1 is used. Select three points around to form a plane, and calculate the normal vector of the plane. Make the probe parallel to the normal vector and the end of the probe perpendicular to the scanning path, so as to obtain the posture of the robotic arm at the j+1th scanning point The robot arm starts scanning the j+1th scanning point until it completes scanning J scanning points.

[0083] In this embodiment, in step S5, the key points of the vertebrae in the ultrasound image classified as "median" by the view are extracted, and the corresponding scanning point positions and robot arm postures are converted from the coordinate system B to the coordinate system I and the coordinate system R to complete the two-dimensional and three-dimensional visualization of the key structures of the spine, as shown in FIG. Figure 7 This is the scan result when the scanned object lies flat normally. Figure 8 This is the scan result when the subject's body is tilted.

[0084] In summary, the present invention integrates multi-source visual information of RGB images, depth images and ultrasound images, and can automatically track the human spine, thereby realizing fully automatic and accurate ultrasound scanning of the spine, overcoming the shortcomings of single-source visual ultrasound scanning, such as low automation and easy loss of spinal vertebrae field of view; at the same time, it can provide three-dimensional information of the spine based on the information collected in the scan, without the need to complete three-dimensional reconstruction of the ultrasound image of the spine, reducing the amount of calculation during reconstruction, and making the spinal information in the ultrasound scan clearer and richer.

[0085] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A fully automatic spinal ultrasound scanning method based on multi-source vision, applied to a system consisting of an RGB-D sensor, a six-degree-of-freedom robotic arm, an ultrasound platform, and a personal computer, characterized in that: The ultrasonic scanning method comprises the following steps: S1. Use the TCP / IP protocol to establish communication connections between the robotic arm and the personal computer, and between the ultrasound platform and the personal computer. Establish a coordinate system for the RGB-D sensor and define it as coordinate system K. Establish a coordinate system for the RGB image captured by the RGB-D sensor and define it as coordinate system I. Establish a coordinate system for the ultrasound image and define it as coordinate system B. Establish a coordinate system for the base of the robotic arm and define it as coordinate system R. Establish a coordinate system for the end of the robotic arm and define it as coordinate system G. Use the robotic arm to clamp the probe in the ultrasound platform, and complete the calibration between the probe and the end of the robotic arm, and between the RGB-D sensor and the base of the robotic arm. S2. Use the RGB-D sensor to obtain the RGB image and depth image of the human back, obtain the three-dimensional point cloud of the human back area based on the RGB image and depth image of the human back, and calculate the total number of scanning points J and the initial scanning point position That is, the scan needs to be performed at J scanning points in total Scan at S3. After the scan begins, the ultrasound image is analyzed using a multi-task deep neural network Multi-task VGG11-FPN, and view classification and vertebral key point detection are performed on the ultrasound image to obtain view category information of the ultrasound image and the coordinates of the vertebral key points in the ultrasound image. The multi-task deep neural network Multi-task VGG11-FPN is used to automatically identify the position of the vertebra in the ultrasound image and determine the view category of the spinal ultrasound image as one of the three view categories of "left", "center", and "right". At the same time, the three vertebral key points above the vertex center, the left transverse process of the thoracic vertebra or the left articular process of the lumbar vertebra, and the right transverse process of the thoracic vertebra or the right articular process of the lumbar vertebra are detected to obtain the coordinates of the three vertebral key points in coordinate system B. S4. Based on the ultrasound image obtained by placing the ultrasound probe at the j-th scanning point position, the j+1-th scanning point position and the robotic arm posture are calculated based on the multi-source vision fusion algorithm. The robotic arm scans the j+1-th scanning point according to the calculation results until the scanning of the J scanning points is completed; S5. After the scanning is completed, the position of each scanning point and the posture of the robotic arm at each scanning point are comprehensively analyzed. The view classification information and vertebral key point detection information of the ultrasound image at each scanning point display the spinal information obtained by automatic scanning. Among them, the vertebral key points of the ultrasound image classified as "center" are extracted, and the corresponding scanning point positions and robotic arm postures are converted from coordinate system B to coordinate systems I and R to complete the two-dimensional and three-dimensional visualization of the key structures of the spine.

2. The method for fully automatic spinal ultrasound scanning based on multi-source vision according to claim 1, characterized in that: In step S1, according to the cross line experiment, the transformation matrix from coordinate system B to coordinate system G is calculated. G T B Complete the calibration between the probe and the end of the robotic arm; according to the chessboard calibration and Levenberg-Marquardt algorithm, calculate the transformation matrix from coordinate system R to coordinate system K K T R And the transformation matrix from coordinate system K to coordinate system R R T K Complete the calibration between the RGB-D sensor and the base of the robotic arm; use the focal length of the infrared camera in the RGB-D sensor to calculate the transformation matrix from coordinate system I to coordinate system K K T I And the transformation matrix from coordinate system K to coordinate system I I T K ; Read the transformation matrix from coordinate system G to coordinate system R from the robot controller R T G ; The conversion from the ultrasound image coordinate system B to the RGB image coordinate system I is achieved by the following formula: P I =(x I ,y I )= I T K K T R R T G G T B P B Among them, P B Represents the coordinates of a point in coordinate system B, P I Indicates P B The coordinate after conversion to coordinate system I, x I P I The coordinate on the x-axis in coordinate system I, y I P I The coordinate on the y-axis in coordinate system I; The conversion between the RGB image coordinate system I and the robotic arm base coordinate system R is achieved through the following formula: P R = R T K K T I P I Among them, P I represents the coordinates in coordinate system I, P R Indicates P I The coordinates after conversion to coordinate system R.

3. The method for fully automatic spinal ultrasound scanning based on multi-source vision according to claim 2, characterized in that: In step S2, a pixel set with color components R>G>B in the RGB image of the back of the human body is extracted, the largest connected area in the pixel set is calculated, and the largest connected area is used as the back area of ​​the human body. K T I The back area of ​​the human body is converted into a three-dimensional point cloud in the coordinate system K.

4. The method for fully automatic spinal ultrasound scanning based on multi-source vision according to claim 3, characterized in that: Use x min 、x max 、y min and y max Respectively represent the minimum and maximum coordinates of the back area of ​​the human body on the x-axis and y-axis in the coordinate system I, through Calculate the initial scanning point in coordinate system I Then, the initial scanning point in coordinate system I is converted to the initial scanning point in coordinate system R through the conversion relationship between coordinate system I and coordinate system R. Thus, the initial scanning point position is obtained, and finally J=x max -x min Calculate the total number of scan points.

5. The method for fully automatic spinal ultrasound scanning based on multi-source vision according to claim 4, characterized in that: The multi-task deep neural network Multi-task VGG11-FPN includes a common part, a view classification part and a vertebral key point detection part, wherein the common part includes a first feature extraction module, a first pooling layer, a second feature extraction module, a second pooling layer, a third feature extraction module and a third pooling layer connected in sequence, and the third pooling layer is also connected to the fourth feature extraction module of the view classification part and the sixth feature extraction module of the vertebral key point detection part respectively; the view classification part includes a fourth feature extraction module, a fourth pooling layer, a fifth feature extraction module, a fifth pooling layer, a sixth pooling layer, a fully connected layer and a Softmax layer connected in sequence; the vertebral key point detection part includes a sixth feature extraction module, a seventh pooling layer, a seventh feature extraction module, an eighth pooling layer, an eighth feature extraction module, a first upsampling layer, a first fuser, a second upsampling layer, a second fuser, a third upsampling layer, a third fuser, a ninth feature extraction module, and a fourth upsampling layer; the vertebral key point detection part further includes a first feature fusion layer, a second feature fusion layer, and a third feature fusion layer, wherein the input and output of the first feature fusion layer are respectively connected to the second pooling layer and the third fuser; the input and output of the second feature fusion layer are respectively connected to the third pooling layer and the second fuser; the input and output of the third feature fusion layer are respectively connected to the seventh pooling layer and the first fuser; at the same time, the eighth feature extraction module, the first and second fusers are also respectively connected to the ninth feature extraction module; The first and second feature extraction modules of the multi-task deep neural network Multi-task VGG11-FPN have the same structure, both of which are convolution layers with a convolution kernel size of 3*3, and both use the ReLU nonlinear activation function; the third, fourth, fifth, sixth, seventh and ninth feature extraction modules have the same structure, both include two convolution layers with a convolution kernel size of 3*3 connected in series, and both use the ReLU nonlinear activation function; the first, second, third, fourth, fifth, seventh and eighth pooling layers are all maximum pooling layers with a size of 2*2; the sixth pooling layer is an average pooling layer with a size of 7*7; the first, second, third and fourth upsampling modules are all bilinear interpolation upsampling layers; the first, second and third feature fusion layers and the eighth feature extraction module are all convolution layers with a size of 1*1; the first, second and third fusion devices respectively realize the fusion of the two input feature maps, that is, the pixel values ​​of the corresponding points of the two feature maps are added.

6. The method for fully automatic spinal ultrasound scanning based on multi-source vision according to claim 5, characterized in that: After the ultrasound image is input into the network, it will first pass through the shared part, then the view classification part and the vertebral keypoint detection part. The softmax layer outputs the predicted probabilities of the three view categories, and the view category with the highest probability is used as the prediction result. The fourth upsampling layer outputs three predicted images. The pixel value of each pixel in the predicted image is between 0 and 1. The pixel coordinates of the maximum pixel value in each of the three images are the coordinates of the three vertebral keypoints.

7. The method for fully automatic spinal ultrasound scanning based on multi-source vision according to claim 6, characterized in that: In step S4, after the robotic arm scans the j-th scanning point, the obtained ultrasonic image of the j-th scanning point is input into the multi-task deep neural network Multi-task VGG11-FPN to obtain the coordinates of the vertex of the vertebral center in the ultrasonic image of the j-th scanning point in coordinate system B. The robot arm position of the j+1th scanning point is calculated using a multi-source vision fusion algorithm. The specific process is as follows: Among them, P Ij for Coordinates in coordinate system I, x Ij P Ij The coordinate on the x-axis in coordinate system I, y Ij P Ij The coordinate on the y-axis in coordinate system I; P I(j+1) =(x I(j+1) ,and I(j+1) )=(x Ij -1,and Ij ) Among them, P I(j+1) is the coordinate of the j+1th scanning point in coordinate system I, x I(j+1) P I(j+1) The coordinate on the x-axis in coordinate system I, y I(j+1) P I(j+1) The coordinate on the y-axis in coordinate system I; in, is the position of the robotic arm at the j+1th scanning point in the coordinate system R.

8. The method for fully automatic spinal ultrasound scanning based on multi-source vision according to claim 7, characterized in that: In step S4, the back three-dimensional point cloud provided by step S2 is used to Select three points around to form a plane, and calculate the normal vector of the plane. Make the probe parallel to the normal vector and the end of the probe perpendicular to the scanning path, so as to obtain the posture of the robotic arm at the j+1th scanning point The robot arm starts scanning the j+1th scanning point until it completes scanning J scanning points.

Images