Ultrasound-guided robotic needle puncture system and method

Abstract

The application discloses an ultrasound-guided puncture robot puncture needle positioning system and method, and the positioning system comprises an ultrasound imaging module, an ultrasound image preprocessing module, an ultrasound image positioning module, a mechanical positioning module, a fusion positioning module and a coordinate system conversion module. The application collects the ultrasound image of the target area containing the puncture needle in operation in real time through the ultrasound imaging equipment, proposes a target segmentation network framework with a pyramid structure named RPN-P-KNet for real-time acquisition of accurate two-dimensional position information of the needle tip, and introduces a new loss function for strengthening the segmentation ability of the network to the edge part of the needle tip. RPN-P-KNet is also applicable to semantic segmentation tasks in other scenes, can effectively improve the segmentation accuracy, and especially improve the segmentation ability to the target edge. The application can effectively improve the positioning precision of the needle tip of the puncture needle, and provide high-precision feedback for a percutaneous puncture robot puncture needle position control system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of surgical robots, and relates to an ultrasound-guided puncture robot puncture needle positioning system and method, in particular to an ultrasound-guided puncture robot puncture needle positioning system and method based on fusion of ultrasound images and mechanical tracking. BACKGROUND

[0002] Percutaneous puncture, as an interventional procedure, is widely used in tissue biopsy, tumor ablation, nerve block, fracture repair, vascular intervention and other minimally invasive surgeries due to its small wound and low risk of infection, compared with traditional open surgery. The puncture position and angle of the puncture needle directly affect the success rate of percutaneous puncture surgery. Ultrasound imaging has the advantages of real-time, convenience, safety and low cost. Ultrasound-guided percutaneous puncture can allow surgeons to visually see the relative position of the puncture needle and the target region, so as to adjust the puncture path. However, due to the need for the surgeon to hold the probe to ensure that the puncture needle is in the ultrasound imaging plane while completing the high-precision puncture operation, the traditional ultrasound-guided percutaneous puncture puts high requirements on the surgeon's operation skills and clinical experience. Moreover, the accuracy of the puncture needle positioning during the operation seriously affects the success rate of the operation and the treatment effect. In recent years, with the development of artificial intelligence and robot technology, the application of surgical assistive robot systems can greatly improve the accuracy and stability of surgical operations. The position feedback of the ultrasound-guided percutaneous puncture robot needle comes from ultrasound image positioning, mechanical positioning and external sensor positioning, etc. Mwikirize et al. (Mwikirize C, Nosher J, Hacihaligolu L, “Convolution Neural Networks for Real-time Needle Detection and Localization in 2D Ultrasound,” International Journal of Computer Assisted Radiology and Surgery, vol. 13, no. 5, pp. 647-657, May 2018.) proposed a neural network framework using a set of fully convolutional and fast convolutional networks for automatic and accurate detection of needle heads in two-dimensional ultrasound data. Through ex vivo experiments, the method was applied to needle trajectory estimation and needle tip positioning. Gao et al. (Gao J, Liu P, Liu G, et al. “Robust Needle Localization and Enhancement Algorithm for Ultrasound by Deep Learning and Beam Steering Methods,” Journal of Computer Science and Technology, vol. 36, no. 2, pp. 334-346, Apr 2021.) proposed a new robust needle localization and enhancement algorithm based on deep learning and beam steering methods. The proposed algorithm not only greatly reduces the processing time, but also significantly improves the accuracy of needle positioning and enhances the visualization of the needle head, which can be used for real-time clinical intervention applications.The traditional puncture needle positioning method has certain advantages in positioning accuracy, but needs to optimize the detection strategy when encountering ultrasound images with large differences. The deep learning method has great advantages in robustness and real-time performance, but still needs to improve the accuracy and stability to meet the clinical requirements of the robot puncture. SUMMARY

[0003] In order to solve the problems of low positioning accuracy and poor robustness of the ultrasound-guided puncture robot, the present application provides an ultrasound-guided puncture robot needle positioning system and method, which can ensure the use of low-cost hardware and consider the accuracy and real-time performance of positioning.

[0004] The purpose of the present application is achieved by the following technical solutions:

[0005] An ultrasound-guided puncture robot needle positioning system comprises an ultrasound imaging module, an ultrasound image preprocessing module, an ultrasound image positioning module, a mechanical positioning module, a fusion positioning module and a coordinate system conversion module, wherein:

[0006] The ultrasound imaging module is used to realize real-time imaging of the surgical target area by the ultrasound-guided puncture robot during the operation, and to obtain real-time two-dimensional ultrasound images containing the puncture needle;

[0007] The ultrasound image preprocessing module is used to perform size cropping and image enhancement operations on the two-dimensional ultrasound images obtained by the ultrasound imaging module;

[0008] The ultrasound image positioning module is used to store the trained RPN-P-KNet needle tip segmentation network, and to locate the two-dimensional needle tip position of the puncture needle and the puncture angle of the puncture needle;

[0009] The RPN-P-KNet needle tip segmentation network is composed of a convolutional layer, a region generation network RPN structure, a pyramid pooling feature extraction structure and a KNet structure. First, the feature map extracted by the convolutional neural network is subjected to anchor frame generation, classification regression and non-maximum suppression operations of the RPN structure, and outputs the candidate region and corresponding coordinates of the puncture needle, while generating a corresponding feature map with smaller size. Second, the processed feature map is input into the pyramid pooling feature extraction structure for four downsampling and convolution operations to obtain feature maps of different sizes, and then the feature maps are upsampled and spliced with the feature maps output by the RPN to obtain a global feature map. Finally, the global feature map is used in the KNet structure, and the kernel and mask prediction are iteratively updated by the adaptive kernel update strategy to complete the final needle tip segmentation result prediction;

[0010] The mechanical positioning module is used to calculate the position and puncture angle of the puncture needle by using the encoder on the puncture robot puncture device;

[0011] The fusion positioning module is configured to fuse the positions and angles of the needle tip of the puncture needle output by the ultrasound image positioning module and the mechanical positioning module, and obtain a final two-dimensional positioning result of the puncture needle by using a Kalman filtering method.

[0012] The coordinate system conversion module is configured to convert the final two-dimensional positioning result of the puncture needle obtained by the fusion positioning module into a three-dimensional working space coordinate system of the robot by coordinate system conversion.

[0013] A method for positioning a puncture needle of an ultrasound-guided puncture robot by using the puncture needle positioning system, comprising the following steps:

[0014] In step S1, an ultrasound imaging module is used to collect an ultrasound image of a target region of a surgery containing a puncture needle in real time during a puncture process of the ultrasound-guided puncture robot, and the ultrasound image is preprocessed by an ultrasound image preprocessing module to obtain an ultrasound image with enhanced image.

[0015] In step S2, the ultrasound image preprocessed in step S1 is input into an ultrasound image positioning module to realize needle tip positioning and axial positioning of the puncture needle based on the ultrasound image, and the specific steps are as follows:

[0016] In step S21, the ultrasound image preprocessed in step S1 is input into a trained RPN-P-KNet needle tip segmentation network to output a candidate box in which the puncture needle is located and a region of the needle tip of the puncture needle, and the training process of the RPN-P-KNet needle tip segmentation network is as follows: data augmentation and data expansion are performed on an original data set; a pre-trained network model is used to accelerate the training process, and a self-adaptive optimization learning rate method is used to further avoid overfitting of the model; a loss function L K-B is introduced to strengthen the segmentation capability of the network for the edge part of the needle tip, and the calculation formula is as follows:

[0017] L K-B =λ1L cls +λ2L ce +λ3L Dice +λ4L B

[0018] wherein, L cls is a focal loss, L ce is a cross-entropy loss, L Dice is a Dice loss, L B is an edge loss function, and λ1, λ2, λ3 and λ4 are proportional coefficients corresponding to the four losses, respectively.

[0019] Step S22, taking the point with the maximum horizontal coordinate of the pixels in the segmentation region as the position of the needle tip in the probe coordinate system, taking the slope of the line connecting the minimum point of the horizontal and axial coordinates of the puncture needle candidate frame and the position of the needle tip as the puncture angle of the puncture needle, completing the two-dimensional positioning of the puncture needle based on the ultrasound image, the position of the needle tip in the probe coordinate system and the puncture angle a u The calculation formula is as follows:

[0020]

[0021]

[0022] Wherein, the point (x s , y s ) is the point with the maximum horizontal coordinate of the pixels in the segmentation region, and the point (x d , y d ) is the minimum point of the horizontal and axial coordinates of the puncture needle candidate frame.

[0023] Step S3, during the puncture process, the mechanical positioning module uses the encoder of the puncture needle driving device to output the position of the needle tip and the axial angle in real time;

[0024] Step S4, the inputs of the ultrasound image positioning module and the mechanical positioning module are fused into the fusion positioning module, and the image-mechanical fusion puncture needle positioning algorithm based on Kalman filtering is used for data fusion to obtain the final two-dimensional positioning result of the puncture needle, wherein the specific steps of the image-mechanical fusion puncture needle positioning algorithm based on Kalman filtering are as follows:

[0025] Step S41, the state vector is x[k]=[p v] T , wherein p is the position of the needle tip, and v is the puncture speed, the observation vector z[k]=[z u,p z m,p z m,v ] T , wherein z u,p is the measurement of the position of the needle tip in the ultrasound image, z m,p is the measurement of the position of the needle tip in the encoder, and z m,v is the speed measurement of the encoder, and the state equation and the observation equation of the system are:

[0026]

[0027] , wherein is the state transition matrix, and Δt is the time step, is the observation matrix, w[k] is the process noise, the covariance matrix of which is Q, v[k] is the observation noise, the covariance matrix of which is R, and k represents the k time, so the expressions of the prior state estimation and the error estimation are:

[0028]

[0029] wherein, and P[k|k-1] represent the prior estimates of the state and error at time k-1 respectively, and P[k|k] represent the posterior estimates of the state and error at time k respectively;

[0030] The Kalman gain K[k] for fusing the state estimate with the measurement is obtained by means of the following formula:

[0031] K[k] = P[k|k-1] · H T ·(H·P[k|k-1]·H T +R) -1

[0032] Step S42, the posterior estimates of the state and error are updated using the Kalman gain:

[0033]

[0034] Step S43, steps S41 and S42 are repeated, the prediction and the update of the measurement are performed at each time step, so as to obtain more accurate puncture needle position information;

[0035] Step S5, the two-dimensional puncture needle positioning information is transformed into a three-dimensional space by means of a coordinate system conversion module, so as to realize the puncture needle positioning of the ultrasound-guided puncture robot.

[0036] Compared with the prior art, the present application has the following advantages:

[0037] The present application acquires the ultrasound image of the intraoperative target area containing the puncture needle in real time through the ultrasound imaging device, proposes a target segmentation network framework with a pyramid structure named RPN-P-KNet for real-time acquisition of accurate two-dimensional position information of the needle tip, and introduces a new loss function for strengthening the segmentation ability of the network to the edge part of the needle tip. Compared with other mainstream segmentation networks, RPN-P-KNet has better segmentation ability for the needle tip of the puncture needle. More importantly, the RPN-P-KNet segmentation network is also applicable to semantic segmentation tasks in other scenarios, and can effectively improve the segmentation accuracy, especially the segmentation ability for the target edge. Secondly, in order to better estimate the position of the puncture needle, a puncture needle positioning method based on the fusion of ultrasound image-mechanical encoder based on Kalman filtering is proposed, the position of the needle tip is accurately estimated, the feedback accuracy of the control system is improved, and the needle tip positioning error in the ultrasound phantom is reduced to 0.15±0.10mm. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a structural diagram of an ultrasound-guided puncture robot puncture needle positioning system;

[0039] Figure 2 is a flow chart of an ultrasound-guided puncture robot puncture needle positioning method;

[0040] Figure 3 is an RPN-P-KNet needle tip segmentation network architecture diagram. DETAILED DESCRIPTION

[0041] The technical solutions of the present application will be further described below in conjunction with the drawings, but are not limited thereto, and any modification or equivalent replacement of the technical solutions of the present application without departing from the spirit and scope of the present application shall be covered in the protection scope of the present application.

[0042] The present application provides an ultrasound-guided puncture robot puncture needle positioning system, as shown in Figure 1 The system mainly comprises an ultrasound imaging module, an ultrasound image preprocessing module, an ultrasound image positioning module, a mechanical positioning module, a fusion positioning module and a coordinate system conversion module, wherein:

[0043] The ultrasound imaging module is used to realize real-time imaging of a surgical target area by an ultrasound-guided puncture robot during surgery, and to obtain real-time two-dimensional ultrasound images containing a puncture needle;

[0044] The ultrasound image preprocessing module is used to perform size cropping, image enhancement and the like on the two-dimensional ultrasound images obtained by the ultrasound imaging module;

[0045] The ultrasound image positioning module is used to store a trained needle tip positioning neural network model based on deep convolution, to locate the two-dimensional needle tip position of the puncture needle and the puncture angle of the puncture needle;

[0046] The mechanical positioning module is used to calculate the position and puncture angle of the puncture needle by using an encoder carried on a puncture robot puncture device;

[0047] The fusion positioning module is used to perform data fusion by using the puncture needle tip position and angle output by the ultrasound image positioning module and the mechanical positioning module, in combination with a Kalman filtering method, to obtain a final puncture needle two-dimensional positioning result;

[0048] The coordinate system conversion module is used to convert the final puncture needle two-dimensional positioning result obtained by the fusion positioning module into a robot three-dimensional workspace coordinate system through coordinate system conversion.

[0049] The present application also provides a method for ultrasound-guided puncture robot puncture needle positioning by using the above puncture needle positioning system, as shown in Figure 2As shown, the method comprises the following steps:

[0050] In the process of ultrasound-guided puncture robot puncture, the ultrasound imaging module is used to collect the ultrasound image of the intraoperative surgical target area containing the puncture needle in real time, and the ultrasound image is preprocessed by the ultrasound image preprocessing module to obtain the ultrasound image after image enhancement, which is more conducive to the positioning of the puncture needle.

[0051] In step S2, the preprocessed ultrasound image is input into the ultrasound image positioning module to realize the needle tip positioning and axial positioning of the puncture needle based on the ultrasound image.

[0052] This step proposes a target segmentation network framework with a pyramid structure named RPN-P-KNet for the needle tip positioning and axial positioning based on the ultrasound image, as shown in Figure 3 As shown, it mainly consists of convolutional layers, region generation network RPN structure, pyramid pooling feature extraction structure and KNet structure, which is used for real-time segmentation of the needle tip area, and outputs accurate needle tip coordinates and needle axial angle by combining the positioning algorithm. The proposed framework takes KNet as the core. Compared with UNet, PSPNet, DeepLabV3+ and other classic segmentation networks, KNet network introduces a dynamic semantic kernel method, which continuously segments semantic categories through a set of learnable kernels. Each kernel will generate a corresponding mask for a class of content, so each kernel can accurately predict image feature information. By continuously training the parameters of each kernel, the resolution capability of the kernel is enhanced, and the segmentation performance is significantly improved. More importantly, since the kernel update head of the KNet module does not need to rely on a specific model architecture to run, it can perform semantic segmentation tasks by integrating the kernel update head into any existing model that relies on semantic kernels. The kernel update of KNet network includes three key steps: group feature combination, adaptive kernel update and kernel interaction.

[0053] However, due to the large artifacts generated by the puncture needle during ultrasound imaging and the small size of the needle tip area, the kernel differentiation ability is not enough to distinguish the needle tip and the background area more accurately. In order to further improve the segmentation accuracy of the KNet network for the small area of the needle tip, we added the region generation network (RPN) structure and pyramid pooling feature extraction structure to the network. First, the feature map extracted by the convolutional neural network is processed by the anchor box generation, classification regression, and non-maximum suppression operations of the RPN structure, and the candidate region and corresponding coordinates of the puncture needle are output. At the same time, a smaller feature map corresponding to the size is generated. Using this feature map for segmentation can enhance the network's extraction of needle tip area features, while greatly reducing the network parameter amount and improving the network's real-time performance. Second, the processed feature map is input into the pyramid pooling structure, which performs four downsampling and convolution operations to obtain feature maps of different sizes. Then, it is upsampled and concatenated with the RPN output feature map to obtain global features. Finally, the global feature map is used in the KNet structure, and the kernel and mask prediction are iteratively updated through the adaptive kernel update strategy to complete the final needle tip segmentation result prediction.

[0054] During training, in order to improve the training effect and obtain a better network model, we first performed data augmentation and data expansion on the original dataset to enrich and expand the dataset. Second, we used a pre-trained network model to accelerate the training process. And using the method of adaptive optimization learning rate, further avoid the model from overfitting. We trained the initial network for 200 epochs, with a batchsize of 8, and used the Adam optimizer to adjust each layer. The loss function is an important indicator for evaluating the training process of the network. Since we pay more attention to the boundary of the needle tip area during needle tip segmentation, we designed a new loss function L K-B for training, whose calculation formula is as follows:

[0055] L K-B = λ1L cls + λ2L ce + λ3L Dice + λ4L B (1)

[0056] Where L cls is the focal loss, L ce is the cross-entropy loss, L Dice is the Dice loss, and L B is the edge loss function. λ1, λ2, λ3, and λ4 are the proportion coefficients of the four losses, respectively. The focal loss is used to handle the class imbalance problem, the cross-entropy loss is used to handle the pixel classification problem, the Dice loss is used to handle the segmentation problem, and the edge loss function is calculated as follows:

[0057]

[0058] where θ i (x i ) is the contribution of a single prediction pair edge loss function, θ ij (x i ,x j ) is the contribution of a pair of predictions pair edge loss function.

[0059] θ i (x i ) = -log P(x i ) (3)

[0060]

[0061] where P(x i ) is the probability of the network output x i corresponding class. p i and p j denote the position of x i and x j , I i and I j denote the color of x i and x j . μ(x i ,x j ) is a control function that only works when the classes of pixels x i and x j are different. ω1, ω2, σ α , σ β and σ γ are all fixed constants. So θ ij (x i ,x j ) is used to encourage better boundaries, the principle is that if the position is close and the color is close on the boundary, it will be punished, and at the same time it encourages the production of smooth segmentation effect.

[0062] The intraoperative real-time ultrasound image is transmitted into the proposed needle tip segmentation network and outputs the candidate box of the puncture needle and the region where the needle tip is located. We select the point with the maximum horizontal coordinate of the pixels in the segmentation region as the position of the needle tip. Moreover, we take the slope of the line connecting the minimum points of the horizontal and axial coordinates of the puncture needle candidate box and the position of the needle tip as the puncture angle of the puncture needle. The position of the needle tip in the probe coordinate system and the calculation formula of the puncture angle α u are as follows:

[0063]

[0064]

[0065] where point (x s ,y s ) is the point of maximum horizontal coordinate of pixels in the needle tip segmentation region. Point (x d ,y d ) is the minimum point of horizontal and axial coordinates of the puncture needle candidate box, s x and s y represent the horizontal and vertical resolution of the ultrasound image, respectively.

[0066] Step S3, at the same time, during the puncture process, the mechanical positioning module uses the encoder of the puncture needle driving device to output the position and axial angle of the puncture needle tip in real time.

[0067] Step S4, the inputs of the ultrasound image positioning module and the mechanical positioning module are fused into the fusion positioning module. Since the accuracy of the needle tip positioning method based on the ultrasound image is limited by the resolution of the ultrasound image, and the accuracy of the needle tip positioning method based on mechanical tracking is limited by the encoder noise and mechanical structure, there will be more or less errors in the positioning process. In order to further reduce the positioning error, an image-mechanical fusion puncture needle positioning algorithm based on Kalman filtering is proposed. Kalman filtering is widely used in control and measurement systems, which can make the system robust to noise and uncertainty. The basic idea of Kalman filtering is to estimate the state of the system by using the dynamic equation of the system and the sensor measurement data, including two main steps of prediction and update.

[0068] In this system, the state vector is x[k] = [p v] T , where p is the position of the needle tip, and v is the puncture speed. The observation vector z[k] = [z u,p z m,p z m,v ] T , where z u,p is the measurement of the needle tip position of the ultrasound image, z m,p is the measurement of the needle tip position of the encoder, and z m,v is the speed measurement of the encoder. Since we control the motor output to be constant speed during puncture, the system motion model can be regarded as a uniform motion model. Then the state equation and the observation equation of the system are:

[0069]

[0070] where is the state transition matrix, and Δt is the time step. is the observation matrix. w[k] is the process noise, and its covariance matrix is Q. v[k] is the observation noise, and its covariance matrix is R. k represents the k time. Then the expressions of the prior state estimation and the error estimation are:

[0071]

[0072] wherein, and P[k|k-1] represent the prior estimates of the state and error at time k-1, respectively. and P[k|k] represent the posterior estimates of the state and error at time k, respectively. Then, the Kalman gain K[k] for fusing the state estimate with the measurement can be obtained by means of formula (9):

[0073] K[k] = P[k|k-1] · HT·(H·P[k|k-1]·H T +R) -1 (9)

[0074] Further, the posterior estimates of the state and error are updated with the Kalman gain, as shown in formula (10):

[0075]

[0076] By repeating the above process, we can continuously perform the prediction and measurement update at each time step to obtain more accurate puncture needle position information.

[0077] Step S5, the fusion positioning module outputs the final predicted two-dimensional puncture needle positioning information, and then the coordinate system conversion module converts the two-dimensional positioning information into three-dimensional space to realize the final puncture needle positioning of the ultrasound-guided puncture robot.

[0078] Embodiment:

[0079] The embodiment provides an ultrasound-guided puncture robot puncture needle positioning method, taking scanning of a puncture ultrasound phantom tissue as an example, and the method specifically comprises the following steps:

[0080] Step S1, during the process of the ultrasound-guided puncture robot puncturing the ultrasound phantom tissue, the ultrasonic imaging module collects the ultrasonic image of the intraoperative surgical target area containing the puncture needle in real time, the ultrasonic image is preprocessed by the ultrasonic image preprocessing module, the ultrasonic image with irrelevant information removed is obtained, and the image is subjected to data enhancement.

[0081] Step S2, the preprocessed ultrasonic image is input into the ultrasonic image positioning module, first, the image is input into the trained RPN-P-KNet needle tip segmentation network to output the candidate box where the puncture needle is located and the region of the puncture needle tip. The point with the maximum horizontal coordinate of the pixels in the segmentation region is taken as the needle tip position. Moreover, the slope of the line connecting the minimum points of the horizontal coordinate and the axial coordinate of the puncture needle candidate box and the needle tip position is taken as the puncture angle of the puncture needle, and the two-dimensional positioning of the puncture needle based on the ultrasonic image is completed.

[0082] Step S3, during the puncture process, the position of the needle tip and the puncture angle of the puncture needle are calculated according to the encoder readings of the puncture needle driving device obtained by the mechanical positioning module in real time, and the two-dimensional positioning of the puncture needle based on mechanical encoder tracking is completed.

[0083] Step S4, since the accuracy of the needle tip positioning method based on the ultrasound image is limited by the resolution of the ultrasound image, and the accuracy of the needle tip positioning method based on mechanical tracking is limited by the encoder noise and the mechanical structure, both of them will have more or less errors during the positioning process. In order to further reduce the positioning error, the puncture needle positioning results obtained in steps S2 and S3 are fused by Kalman filtering, and through updating and prediction, a more accurate puncture needle is obtained.

[0084] Step S5, finally, the coordinate system conversion module determines the coordinate transformation formula according to the rigid connection relationship between different coordinate systems, and realizes the three-dimensional positioning of the puncture needle.

[0085] The accuracy of the puncture needle tip positioning method based on ultrasound image feedback, mechanical encoder feedback and the method proposed in this embodiment is tested respectively. The puncture experiment is carried out on the ultrasound phantom, in which the positioning error of the method based on ultrasound image is 0.26±0.12mm, the positioning error of the method based on mechanical encoder feedback is 0.19±0.09mm, and the positioning error of the method based on Kalman filtering fusion is 0.15±0.10mm. Therefore, the image-mechanical fusion puncture needle positioning algorithm based on Kalman filtering can effectively improve the positioning accuracy of the puncture needle tip, and provide high-precision feedback for the puncture needle position control system of the percutaneous puncture robot.

Claims

1. An ultrasound-guided puncture robot needle positioning system, characterized in that The positioning system includes an ultrasonic imaging module, an ultrasonic image preprocessing module, an ultrasonic image positioning module, a mechanical positioning module, a fusion positioning module, and a coordinate system conversion module, wherein: The ultrasound imaging module is used to enable the ultrasound-guided puncture robot to perform real-time imaging of the surgical target area during surgery, and obtain a real-time two-dimensional ultrasound image containing the puncture needle; The ultrasonic image preprocessing module is used to perform size cropping and image enhancement operations on the two-dimensional ultrasonic image acquired by the ultrasonic imaging module; The ultrasound image positioning module is used to store the trained RPN-P-KNet needle tip segmentation network to locate the two-dimensional needle tip position and puncture angle of the puncture needle; The RPN-P-KNet needle tip segmentation network consists of a convolutional layer, a region generation network (RPN) structure, a pyramid pooling feature extraction structure, and a KNet structure. First, the feature map extracted by the convolutional neural network undergoes anchor frame generation, classification regression, and non-maximum suppression operations in the RPN structure to output the candidate region and corresponding coordinates of the puncture needle, while also generating the corresponding feature map. Second, the processed feature map is input into the pyramid pooling feature extraction structure for four downsampling and convolution operations to obtain feature maps of different sizes. These maps are then upsampled and concatenated with the feature map output by the RPN to obtain a global feature map. Finally, the global feature map is used in the KNet structure, and the kernel and mask predictions are iteratively updated through an adaptive kernel update strategy to complete the final needle tip segmentation result prediction. The mechanical positioning module is used to calculate the position and puncture angle of the puncture needle using the encoder on the puncture device of the puncture robot; The fusion positioning module is used to use the position and angle of the puncture needle tip output by the ultrasound image positioning module and the mechanical positioning module, and combine the Kalman filter method to perform data fusion to obtain the final puncture needle two-dimensional positioning result; The coordinate system conversion module is used to convert the final two-dimensional positioning result of the puncture needle obtained by the fusion positioning module into the robot's three-dimensional workspace coordinate system through coordinate system transformation.

Images