Positioning system and method for full-automatic needle insertion

Abstract

The invention discloses a full-automatic needle inserting positioning injection system and method, which are used for obtaining a three-dimensional image of the interior of an eyeball and determining an injection position based on the three-dimensional image so as to implement a needle inserting operation. When the needle is inserted, the real-time three-dimensional image of the injection process is obtained, registration is carried out according to the reference three-dimensional image, and then path planning can be obtained through calculation, so that the injection instrument is driven to reach the determined injection position to carry out needle insertion injection. Real-time feedback is provided for navigation positioning of the injection instrument through real-time imaging, dynamic changes of the eyeball structure and the injection instrument are monitored in real time, accurate positioning of needle inserting injection is achieved, and the accuracy of positioning injection of the injection instrument on the injection position in the full-automatic needle inserting process is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intraocular injection, and in particular to a fully automatic needle insertion positioning system and method. Background Art

[0002] Fundus surgery refers to surgical operations on the fundus (retina and its underlying tissues), the main purpose of which is to treat or improve fundus diseases that may affect vision or cause blindness. Intraocular injection is a common type of fundus surgery. This technology can effectively treat a variety of fundus lesions by directly injecting drugs into the eye.

[0003] In intraocular injection treatment, precise positioning is one of the key points that need to be paid attention to. Specifically, intraocular injection needs to ensure that the drug is injected into the exact position to avoid weakening the drug effect or causing the injection to be ineffective. What's more, inaccurate acupuncture position may cause other injuries to the eye, so intraocular injection requires high accuracy in positioning the injection position. In this regard, fully automatic needle insertion technology refers to the use of automated instruments (such as surgical robots) to insert needles and inject drugs. Complementary to this is the instrument navigation technology that can improve the accuracy of injection position positioning, such as image navigation and optical tracking systems; compared with manual operation, fully automatic needle insertion technology and instrument navigation technology complement each other to improve the accuracy of intraocular injection position positioning. However, in the current application of instrument navigation technology, due to the limitations of the imaging system, it is impossible to obtain a three-dimensional image of the internal structure of the eye, which will affect the accuracy of the injection position positioning of the injection instrument during the fully automatic needle insertion process, thereby resulting in a decrease in the accuracy of the injection positioning injection by the injection instrument.

[0004] Therefore, during the fully automatic needle insertion process of intraocular injection treatment, it is necessary to improve the accuracy of the injection instrument in positioning the injection. Summary of the invention

[0005] The present invention provides a fully automatic needle insertion positioning injection system and method, which uses a three-dimensional ultrasonic imaging system to perform real-time three-dimensional imaging of the inside of the eyeball, provide real-time feedback for the positioning navigation of instrument injection, and can effectively improve the accuracy of positioning injection of the injection instrument.

[0006] In order to solve the above technical problems, the present invention provides a fully automatic needle insertion positioning injection system, including an imaging module, an injection point selection module, a three-dimensional image registration module, an instrument needle tip detection module, a path planning module and a motion module, specifically:

[0007] An imaging module for obtaining a reference three-dimensional image of the internal structure of the eyeball;

[0008] An injection point selection module is used to select injection points based on a reference three-dimensional image and determine a reference injection position;

[0009] A motion module drives the needle tip of the instrument to perform a needle insertion operation based on a reference injection position;

[0010] The imaging module is also used to collect real-time three-dimensional images in response to the needle insertion operation;

[0011] A three-dimensional image registration module is used to perform algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image;

[0012] An instrument needle tip detection module, used to obtain the position of the instrument needle tip based on the registered three-dimensional image;

[0013] A path planning module is used to perform path planning based on the reference injection position and the instrument needle tip position to obtain a target path;

[0014] The motion module is also used to guide the needle tip of the instrument to a reference injection position for injection based on a target path.

[0015] In the above scheme, the present invention provides a fully automatic needle insertion positioning injection system, which obtains a three-dimensional image of the inside of the eyeball and determines the injection position based on this reference to implement the needle insertion operation. When inserting the needle, by obtaining a real-time three-dimensional image of the injection process and aligning it with the reference three-dimensional image, the path planning can be calculated to drive the injection device to the determined injection position for needle insertion and injection. This system provides real-time feedback for the navigation and positioning of the injection device through real-time imaging, and monitors the dynamic changes of the eyeball structure and the injection device in real time to achieve precise positioning of the needle insertion and injection, thereby improving the accuracy of the injection device in positioning the injection position during the fully automatic needle insertion process.

[0016] Preferably, the specific imaging process of the imaging module is implemented through a three-dimensional ultrasonic imaging system.

[0017] In the above scheme, by adopting a three-dimensional ultrasonic imaging system, the limitations of the binocular camera system are overcome, and real-time three-dimensional imaging of the inside of the eyeball is performed, which can provide instant feedback for the positioning and navigation of the injection device; dynamic imaging can be performed during the injection process, and the dynamic changes of the eyeball structure and the injection device can be monitored in real time, which can effectively improve the accuracy of the injection device in locating the injection position during the fully automatic needle insertion process.

[0018] Preferably, the three-dimensional image registration module is used to perform algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image, specifically:

[0019] Acquire a source point set and a target point set based on a real-time three-dimensional image and a reference three-dimensional image;

[0020] Obtain the best transformation matrix based on the source point set and the target point set;

[0021] Acquire the centroid of the real-time three-dimensional image and the centroid of the reference three-dimensional image based on the source point set and the target point set;

[0022] Calculate based on the real-time 3D image centroid, the reference 3D image centroid and the best transformation matrix to obtain the best rotation matrix;

[0023] Calculate based on the real-time 3D image centroid, the reference 3D image centroid and the best rotation matrix to obtain a translation vector;

[0024] Based on the translation vector and the optimal transformation matrix, the transformation is applied to each point in the source point set to obtain the registered three-dimensional image.

[0025] In the above scheme, algorithm registration is performed based on the real-time three-dimensional image and the reference three-dimensional image. The algorithm used in the process of obtaining the registered image is the ICP algorithm. By performing registration through the ICP algorithm, the real-time three-dimensional image and the reference three-dimensional image can be accurately corresponded in the same coordinate system, which is convenient for subsequent recognition.

[0026] Furthermore, in the 3D image registration module, the best rotation matrix is ​​obtained by calculation based on the centroid of the real-time 3D image, the centroid of the reference 3D image and the best transformation matrix, which is specifically:

[0027] Based on the centroid of the real-time 3D image and the centroid of the reference 3D image, each point in the source point set and the target point set is removed from the centroid to obtain a removed centroid source point set and a removed centroid target point set;

[0028] Obtaining a covariance matrix based on a de-centroided source point set, a de-centroided target point set, and an optimal transformation matrix;

[0029] Get the optimal rotation matrix based on the optimal transformation matrix and covariance matrix.

[0030] Furthermore, in the 3D image registration module, based on the centroid of the real-time 3D image and the centroid of the reference 3D image, each point in the source point set and the target point set is removed from the centroid, and a removed-centroid source point set and a removed-centroid target point set are obtained, specifically:

[0031] Perform difference calculation on each point in the source point set and the centroid of the real-time three-dimensional image to obtain the centroid-removed source point set;

[0032] Each point in the target point set is subtracted from the centroid of the reference three-dimensional image to obtain the target point set without the centroid.

[0033] Furthermore, in the 3D image registration module, the optimal rotation matrix is ​​obtained based on the optimal transformation matrix and the covariance matrix, specifically:

[0034] Perform singular value decomposition based on the covariance matrix to obtain the corresponding orthogonal matrix;

[0035] The optimal rotation matrix is ​​obtained by performing calculations based on the corresponding orthogonal matrix and the optimal transformation matrix.

[0036] Preferably, in the instrument needle tip detection module, the position of the instrument needle tip is obtained based on the registered three-dimensional image, specifically:

[0037] Acquisition is performed based on the registered three-dimensional image to obtain feature images at different resolutions;

[0038] Based on the feature image, the preset convolution layer is used to detect the key point features corresponding to the needle tip of the instrument;

[0039] Perform feature fusion based on key point features to obtain a fused feature map;

[0040] Based on the fused feature map, the preset convolution layer is used for processing to obtain the needle tip position of the instrument.

[0041] In the above scheme, the method for detecting the needle tip of the injection device based on the registered three-dimensional image is based on the key point detection algorithm HRNet of deep learning. Through the trained neural network model, it can realize the automatic recognition and positioning of the needle tip of the injection device in the three-dimensional image. The overall algorithm structure of this registration method is a residual structure. Specifically, the registered three-dimensional image is acquired through a parallel multi-resolution subnet to obtain the feature image containing the injection device at different resolutions, and the key point features corresponding to the needle tip of the device are obtained after processing; the key point features obtained are fused to obtain the fused feature map, and finally the fused features are processed using the preset convolution layer to finally obtain the position of the needle tip of the device.

[0042] Preferably, the path planning module is used to perform path planning based on the reference injection position and the instrument needle tip position to obtain the target path, specifically:

[0043] Acquire the needle tip position coordinate point and the reference position coordinate point based on the instrument needle tip point position and the reference injection position;

[0044] Calculate based on the needle tip position coordinate point and the reference position coordinate point to obtain the spatial distance between the needle tip position coordinate point and the reference position coordinate point;

[0045] Calculate the total number of steps based on the spatial distance and the preset step length;

[0046] Generate path points based on the total number of steps and the preset step length and in combination with the needle tip position coordinate point and the reference position coordinate point;

[0047] The path planning for the needle insertion operation is performed based on the path points to generate the target path.

[0048] In the above scheme, a vision-based navigation scheme for injection devices is provided by calculating and obtaining a target path that can be recognized by a control system installed on the injection device, and the injection device includes but is not limited to a robot and an automatic injection device.

[0049] Furthermore, in the path planning module, path points are generated based on the total number of steps and step length in combination with the needle tip position coordinate point and the reference position coordinate point, specifically:

[0050] Based on the needle tip position coordinate point and the reference position coordinate point, calculations are performed step by step according to the total number of steps, the preset step length, and the spatial distance to obtain all the intermediate points from the needle tip position coordinate point to the reference position coordinate point;

[0051] The waypoints are generated step by step based on the obtained intermediate points.

[0052] In the above scheme, by calculating the position deviation between the instrument needle tip position and the reference injection position and solving the navigation path, it is possible to visually guide the injection instrument to adjust its position and posture, so that the injection instrument can accurately reach the selected injection position.

[0053] Accordingly, an embodiment of the present invention further provides a fully automatic needle insertion positioning injection method, comprising the following steps:

[0054] Obtain a baseline 3D image of the internal structure of the eye;

[0055] Select injection points based on the reference three-dimensional image and determine the reference injection position;

[0056] In response to the reference injection position, the needle tip of the instrument is driven to perform a needle insertion operation and acquire a real-time three-dimensional image;

[0057] Performing algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image;

[0058] Acquire the position of the instrument needle tip based on the registered three-dimensional image;

[0059] Perform path planning based on the reference injection position and the instrument needle tip position to obtain the target path;

[0060] Based on the target path, guide the instrument needle tip to the reference injection position for injection.

[0061] In the above scheme, the present invention provides a method for positioning injection of fully automatic needle insertion, which performs the needle insertion operation by obtaining a three-dimensional image of the inside of the eyeball and determining the injection position based on this reference. When inserting the needle, by obtaining a real-time three-dimensional image of the injection process and aligning it with the reference three-dimensional image, the path planning can be calculated to drive the injection device to the determined injection position for needle insertion and injection. This method provides real-time feedback for the navigation and positioning of the injection device through real-time imaging, and monitors the dynamic changes of the eyeball structure and the injection device in real time to achieve precise positioning of the needle insertion and injection, thereby improving the accuracy of the injection device in positioning the injection position during the fully automatic needle insertion process.

[0062] In summary, the embodiments of the present invention provide a fully automatic needle insertion positioning injection system and method, which have the following beneficial effects:

[0063] The present invention provides a fully automatic positioning injection system and method, including: before injection, using a three-dimensional ultrasonic imaging system to perform baseline imaging of the internal structure of the eyeball, and selecting the target injection position based on the imaging results; during the injection process, the injection process is also imaged in real time by a three-dimensional ultrasonic system, and the dynamic changes of the eyeball structure and the injection device are monitored in real time. Based on the results of the baseline imaging and real-time imaging, registration is performed to obtain a registration image, and then the path planning is calculated to guide the injection device to the target injection position for injection. In this regard, the present invention uses real-time imaging to design a vision-based navigation scheme, which can provide real-time feedback for the navigation and positioning of the injection device, realize fully automatic needle insertion of the injection device, and improve the accuracy of the injection device in positioning the injection position during the fully automatic needle insertion process.

[0064] In addition, the present invention can effectively avoid the risk of poor injection effect or damage to other structures of the eyeball due to deviation between the injection position and the target position by realizing a precise fully automatic needle insertion process of the injection device, thereby further improving the safety of the intraocular injection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a structural schematic diagram of an embodiment of a fully automatic needle insertion positioning injection system provided by the present invention;

[0066] Figure 2 It is a schematic flow chart of an embodiment of a fully automatic needle insertion positioning injection method provided by the present invention;

[0067] Figure 3 It is a flow chart of a three-dimensional image registration method based on the ICP algorithm provided by the present invention;

[0068] Figure 4It is a structural diagram of the HRNet network referred to by the needle tip detection method for detecting injection equipment provided by the present invention. DETAILED DESCRIPTION

[0069] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Example 1

[0071] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a structure of an embodiment of a fully automatic needle insertion positioning injection system provided by the present invention. Figure 1 As shown, the system includes an imaging module, an injection point selection module, a three-dimensional image registration module, an instrument needle tip detection module, a path planning module and a motion module, specifically:

[0072] An imaging module for obtaining a reference three-dimensional image of the internal structure of the eyeball;

[0073] An injection point selection module is used to select injection points based on a reference three-dimensional image and determine a reference injection position;

[0074] A motion module drives the needle tip of the instrument to perform a needle insertion operation based on a reference injection position;

[0075] The imaging module is also used to collect real-time three-dimensional images in response to the needle insertion operation;

[0076] A three-dimensional image registration module is used to perform algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image;

[0077] An instrument needle tip detection module, used to obtain the position of the instrument needle tip based on the registered three-dimensional image;

[0078] A path planning module is used to perform path planning based on the reference injection position and the instrument needle tip position to obtain a target path;

[0079] The motion module is also used to guide the needle tip of the instrument to a reference injection position for injection based on a target path.

[0080] In this embodiment, the present invention provides a fully automatic needle insertion positioning injection system, which obtains a three-dimensional image of the inside of the eyeball and determines the injection position based on this reference to implement the needle insertion operation. When inserting the needle, by obtaining a real-time three-dimensional image of the injection process and aligning it with the reference three-dimensional image, the path planning can be calculated to drive the injection device to the determined injection position for needle insertion and injection. This system provides real-time feedback for the navigation and positioning of the injection device through real-time imaging, and monitors the dynamic changes of the eyeball structure and the injection device in real time to achieve precise positioning of the needle insertion and injection, thereby improving the accuracy of the injection device in positioning the injection position during the fully automatic needle insertion process.

[0081] Preferably, the specific imaging process of the imaging module is implemented through a three-dimensional ultrasonic imaging system.

[0082] In this embodiment, the above-mentioned imaging process breaks through the limitations of the binocular camera system by adopting a three-dimensional ultrasonic imaging system, and performs real-time three-dimensional imaging of the inside of the eyeball, which can provide instant feedback for the positioning and navigation of the injection device; it can perform dynamic imaging during the injection process, and monitor the dynamic changes of the eyeball structure and the injection device in real time, which can effectively improve the accuracy of the injection device in positioning the injection position during the fully automatic needle insertion process.

[0083] Preferably, the three-dimensional image registration module is used to perform algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image, specifically:

[0084] Acquire a source point set and a target point set based on a real-time three-dimensional image and a reference three-dimensional image;

[0085] Obtain the best transformation matrix based on the source point set and the target point set;

[0086] Acquire the centroid of the real-time three-dimensional image and the centroid of the reference three-dimensional image based on the source point set and the target point set;

[0087] Calculate based on the real-time 3D image centroid, the reference 3D image centroid and the best transformation matrix to obtain the best rotation matrix;

[0088] Calculate based on the real-time 3D image centroid, the reference 3D image centroid and the best rotation matrix to obtain a translation vector;

[0089] Based on the translation vector and the optimal transformation matrix, the transformation is applied to each point in the source point set to obtain the registered three-dimensional image.

[0090] In this embodiment, algorithm registration is performed based on the real-time three-dimensional image and the reference three-dimensional image. The algorithm used in the process of obtaining the registered image is the ICP algorithm. By performing the registration with the ICP algorithm, the real-time three-dimensional image and the reference three-dimensional image can be accurately corresponded in the same coordinate system, which is convenient for subsequent recognition.

[0091] Furthermore, in the 3D image registration module, the best rotation matrix is ​​obtained by calculation based on the centroid of the real-time 3D image, the centroid of the reference 3D image and the best transformation matrix, which is specifically:

[0092] Based on the centroid of the real-time 3D image and the centroid of the reference 3D image, each point in the source point set and the target point set is removed from the centroid to obtain a removed centroid source point set and a removed centroid target point set;

[0093] Obtaining a covariance matrix based on a de-centroided source point set, a de-centroided target point set, and an optimal transformation matrix;

[0094] Get the optimal rotation matrix based on the optimal transformation matrix and covariance matrix.

[0095] Furthermore, in the 3D image registration module, based on the centroid of the real-time 3D image and the centroid of the reference 3D image, each point in the source point set and the target point set is removed from the centroid, and a removed-centroid source point set and a removed-centroid target point set are obtained, specifically:

[0096] Perform difference calculation on each point in the source point set and the centroid of the real-time three-dimensional image to obtain the centroid-removed source point set;

[0097] Each point in the target point set is subtracted from the centroid of the reference three-dimensional image to obtain the target point set without the centroid.

[0098] Furthermore, in the 3D image registration module, the optimal rotation matrix is ​​obtained based on the optimal transformation matrix and the covariance matrix, specifically:

[0099] Perform singular value decomposition based on the covariance matrix to obtain the corresponding orthogonal matrix;

[0100] The optimal rotation matrix is ​​obtained by performing calculations based on the corresponding orthogonal matrix and the optimal transformation matrix.

[0101] Preferably, in the instrument needle tip detection module, the position of the instrument needle tip is obtained based on the registered three-dimensional image, specifically:

[0102] Acquisition is performed based on the registered three-dimensional image to obtain feature images at different resolutions;

[0103] Based on the feature image, the preset convolution layer is used to detect the key point features corresponding to the needle tip of the instrument;

[0104] Perform feature fusion based on key point features to obtain a fused feature map;

[0105] Based on the fused feature map, the preset convolution layer is used for processing to obtain the needle tip position of the instrument.

[0106] In this embodiment, the method for detecting the needle tip of the injection device based on the registered three-dimensional image is based on the key point detection algorithm HRNet of deep learning. Through the trained neural network model, the automatic identification and positioning of the needle tip of the injection device in the three-dimensional image can be realized. The overall algorithm structure of the registration method is a residual structure. Specifically, the registered three-dimensional image is collected through a parallel multi-resolution subnet to obtain feature images containing injection devices at different resolutions, and the key point features corresponding to the needle tip of the device are obtained after processing; the key point features obtained are subjected to feature fusion to obtain a fused feature map, and finally the fused features are processed using a preset convolutional layer to finally obtain the position of the needle tip of the device.

[0107] Preferably, the path planning module is used to perform path planning based on the reference injection position and the instrument needle tip position to obtain the target path, specifically:

[0108] Acquire the needle tip position coordinate point and the reference position coordinate point based on the instrument needle tip point position and the reference injection position;

[0109] Calculate based on the needle tip position coordinate point and the reference position coordinate point to obtain the spatial distance between the needle tip position coordinate point and the reference position coordinate point;

[0110] Calculate the total number of steps based on the spatial distance and the preset step length;

[0111] Generate path points based on the total number of steps and the preset step length and in combination with the needle tip position coordinate point and the reference position coordinate point;

[0112] The path planning for the needle insertion operation is performed based on the path points to generate the target path.

[0113] In the above scheme, the target path is obtained by calculation and can be recognized by the control system installed on the injection device, which is a vision-based injection device navigation scheme. The injection device includes but is not limited to a robot and an automatic injection device.

[0114] Furthermore, in the path planning module, path points are generated based on the total number of steps and step length in combination with the needle tip position coordinate point and the reference position coordinate point, specifically:

[0115] Based on the needle tip position coordinate point and the reference position coordinate point, calculations are performed step by step according to the total number of steps, the preset step length, and the spatial distance to obtain all the intermediate points from the needle tip position coordinate point to the reference position coordinate point;

[0116] The waypoints are generated step by step based on the obtained intermediate points.

[0117] In this embodiment, by calculating the position deviation between the instrument needle tip position and the reference injection position and solving the navigation path, it is possible to visually guide the injection instrument to adjust its position and posture, so that the injection instrument can accurately reach the selected injection position.

[0118] In this embodiment, a method for implementing path planning based on the reference injection position and the instrument needle tip position to obtain the target path is:

[0119] (1) The needle tip point obtained by the instrument needle tip detection module is recorded as (x1, y1, z1), and the point selected by the injection point selection module is defined as the target point, which is represented by (x2, y2, z2).

[0120] (2) Calculate the spatial distance d from the needle tip to the target point:

[0121]

[0122] (3) Calculate the total number of steps N:

[0123]

[0124] Wherein, Δs represents the preset step size.

[0125] (4) Generate all intermediate points from the current needle tip to the target point:

[0126]

[0127] Among them, i ranges from 1 to N.

[0128] (5) Step by step, path points are generated based on the obtained intermediate points.

[0129] Example 2

[0130] See also Figure 2 , Figure 2 1 is a flow chart of an embodiment of a fully automatic needle insertion positioning injection method provided by the present invention. The method includes steps S1-S7, and the specific steps are as follows:

[0131] Step S1: Acquire a reference three-dimensional image of the internal structure of the eyeball.

[0132] Step S2: selecting injection points based on the reference three-dimensional image to determine a reference injection position.

[0133] Step S3: In response to the reference injection position, the needle tip of the instrument is driven to perform a needle insertion operation and acquire a real-time three-dimensional image.

[0134] In this embodiment, the three-dimensional imaging processes in step S1 and step S3 are both implemented through a three-dimensional ultrasonic imaging system, which can break through the limitations of the binocular camera system, perform real-time three-dimensional imaging of the inside of the eyeball, and provide instant feedback for the positioning and navigation of the injection device; it can perform dynamic imaging during the injection process, and monitor the dynamic changes of the eyeball structure and the injection device in real time, which can effectively improve the accuracy of the injection device in positioning the injection position during the fully automatic needle insertion process.

[0135] Step S4: performing algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image.

[0136] In this embodiment, the algorithm used in the process of algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image is the ICP algorithm. By performing the registration with the ICP algorithm, the real-time three-dimensional image and the reference three-dimensional image can be accurately corresponded in the same coordinate system, which is convenient for subsequent recognition.

[0137] Step S5: Acquire the needle tip position of the instrument based on the registered three-dimensional image.

[0138] In this embodiment, the method for detecting the needle tip of the injection device based on the registered three-dimensional image is based on the key point detection algorithm HRNet of deep learning. Through the trained neural network model, the automatic identification and positioning of the needle tip of the injection device in the three-dimensional image can be realized. The overall algorithm structure of the registration method is a residual structure. Specifically, the registered three-dimensional image is collected through a parallel multi-resolution subnet to obtain feature images containing injection devices at different resolutions, and the key point features corresponding to the needle tip of the device are obtained after processing; the key point features obtained are subjected to feature fusion to obtain a fused feature map, and finally the fused features are processed using a preset convolutional layer to finally obtain the position of the needle tip of the device.

[0139] Step S6: Perform path planning based on the reference injection position and the instrument needle tip position to obtain a target path.

[0140] In this embodiment, the target path can be identified by a navigation system of an injection device, and the injection device includes but is not limited to a robot and an automatic injection device.

[0141] Step S7: Based on the target path, guide the instrument needle tip to reach the reference injection position for injection.

[0142] Example 3

[0143] See also Figure 3 , Figure 3 4 is a flow chart of a three-dimensional image registration method based on the ICP algorithm provided by the present invention. The method includes steps S401-S406. The specific steps are as follows:

[0144] Step S401: acquiring a source point set and a target point set based on a real-time three-dimensional image and a reference three-dimensional image.

[0145] Step S402: Obtain an optimal transformation matrix based on the source point set and the target point set.

[0146] In this embodiment, the specific implementation process of steps S401-S402 is:

[0147] (1) Assume that the real-time 3D image is: P = {p i}, the reference 3D image is Q = {q i}.

[0148] (2) For each point p i ∈P find the closest point q i ∈Q, so that q i =argmin q∈Q ‖p i -q‖.

[0149] (3) Obtain the optimal transformation matrix T, so that T = argmin R,t ∑ i ‖Rp i +tq i ‖ 2 , where R is the rotation matrix and t is the translation vector.

[0150] Step S403: acquiring the centroid of the real-time three-dimensional image and the centroid of the reference three-dimensional image based on the source point set and the target point set.

[0151] In this embodiment, the specific process of calculating the centroid of the real-time 3D image and the centroid of the reference 3D image is as follows:

[0152]

[0153] Step S104: performing calculation based on the real-time three-dimensional image centroid, the reference three-dimensional image centroid and the best transformation matrix to obtain the best rotation matrix.

[0154] In this embodiment, the specific process of step S404 is:

[0155] (1) Remove the centroid of each point in the source point set and the target point set:

[0156]

[0157] Among them, p i ′ is the source point after removing the centroid, q i ′ is the target point after removing the centroid.

[0158] (2) Calculate the covariance matrix of the source and target points: H = ∑ i p i ′ q i ′T .

[0159] (3) Singular value decomposition (SVD) of the covariance matrix H: H = UΣV T .

[0160] Among them, U and V are orthogonal matrices, and Σ is a diagonal matrix.

[0161] (4) Calculate the optimal rotation matrix R: R = VU T .

[0162] Step S405: performing calculation based on the real-time three-dimensional image centroid, the reference three-dimensional image centroid and the optimal rotation matrix to obtain a translation vector.

[0163] In this embodiment, the specific formula for obtaining the translation vector is:

[0164] Calculate the translation vector t:

[0165] Step S406: Based on the translation vector and the optimal transformation matrix, apply transformation to each point in the source point set to obtain a registered three-dimensional image.

[0166] In this embodiment, the specific formula for applying the transformation to each point in the source point set to obtain the registered three-dimensional image is:

[0167] Apply the transformation to each point in the source point to obtain the registered image: T(p i )=Rp i +t.

[0168] Example 4

[0169] See also Figure 4 , Figure 4 It is a structural diagram of the HRNet network referred to by the needle tip detection method for detecting injection devices provided by the present invention. The HRNet network is mainly divided into three parts: a residual structure, a parallel multi-resolution subnet and feature fusion.

[0170] The residual structure has two modules, Bottleneck and BasicBlock. The Bottleneck module is located in stage 1 of the network, which consists of two 1x1 convolution kernels (conv1×1) and one 5x5 convolution kernel (conv5×5); the BasicBlock module is located in stage 2, stage 3, and stage 4, and consists of two 1x1 convolution kernels.

[0171] The parallel multi-resolution subnetwork is used to process feature maps of different resolutions simultaneously to capture rich contextual information and maintain high-resolution features. At the output of each stage, a low-resolution branch is generated, and the resolution of each branch is 1 / 2 of the resolution of the highest resolution branch. n , n represents the stage.

[0172] Feature fusion is located in stage 2, stage 3, and stage 4. Upsampling, downsampling, and weighted combination are performed before the output of each stage. The fused feature map is further processed through convolutional layers to generate the final feature representation.

[0173] In this embodiment, the present invention provides a fully automatic needle insertion positioning injection system and method, and the method for detecting the needle tip of the injection device based on the registered three-dimensional image is based on the key point detection algorithm HRNet of deep learning, and the trained neural network model can realize automatic recognition and positioning of the needle tip of the injection device in the three-dimensional image. The overall algorithm structure of the registration method is a residual structure, specifically, the registered three-dimensional image is collected through a parallel multi-resolution subnet, and the feature image containing the injection device at different resolutions is obtained, and the key point features corresponding to the needle tip of the device are obtained after processing; the key point features obtained are subjected to feature fusion to obtain a fused feature map, and finally the fused features are processed using a preset convolutional layer to finally obtain the position of the needle tip of the device.

[0174] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

[0175] Those skilled in the art can understand that all or part of the processes in the above embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

Claims

1. A fully automatic needle insertion positioning injection system, characterized in that: It includes imaging module, injection point selection module, 3D image registration module, instrument needle tip detection module, path planning module and motion module, specifically: The imaging module is used to obtain a reference three-dimensional image of the internal structure of the eyeball; The injection point selection module is used to select injection points based on the reference three-dimensional image and determine the reference injection position; The motion module drives the needle tip of the instrument to perform a needle insertion operation based on the reference injection position; The imaging module is further used to collect real-time three-dimensional images in response to the needle insertion operation; The three-dimensional image registration module is used to perform algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image; The instrument needle tip point detection module is used to obtain the instrument needle tip point position based on the registered three-dimensional image; The path planning module is used to perform path planning based on the reference injection position and the instrument needle tip point position to obtain a target path; The motion module is also used to guide the instrument needle tip to the reference injection position for injection based on the target path.

2. A fully automatic needle insertion positioning injection system as claimed in claim 1, characterized in that: The specific imaging process of the imaging module is realized through a three-dimensional ultrasonic imaging system.

3. A fully automatic needle insertion positioning injection system as claimed in claim 1, characterized in that: The three-dimensional image registration module is used to perform algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image, specifically: Acquire a source point set and a target point set based on the real-time three-dimensional image and the reference three-dimensional image; Acquire an optimal transformation matrix based on the source point set and the target point set; Acquire a real-time three-dimensional image centroid and a reference three-dimensional image centroid based on the source point set and the target point set; Perform calculation based on the real-time three-dimensional image centroid, the reference three-dimensional image centroid and the optimal transformation matrix to obtain an optimal rotation matrix; Calculate based on the real-time three-dimensional image centroid, the reference three-dimensional image centroid and the optimal rotation matrix to obtain a translation vector; Based on the translation vector and the optimal transformation matrix, a transformation is applied to each point in the source point set to obtain a registered three-dimensional image.

4. A fully automatic needle insertion positioning injection system as claimed in claim 3, characterized in that: In the three-dimensional image registration module, the calculation is performed based on the real-time three-dimensional image centroid, the reference three-dimensional image centroid and the optimal transformation matrix to obtain the optimal rotation matrix, specifically: Based on the real-time three-dimensional image centroid and the reference three-dimensional image centroid, each point in the source point set and the target point set is removed from the centroid to obtain a removed-centroid source point set and a removed-centroid target point set; Acquire a covariance matrix based on the de-centroided source point set, the de-centroided target point set, and the optimal transformation matrix; An optimal rotation matrix is ​​obtained based on the optimal transformation matrix and the covariance matrix.

5. A fully automatic needle insertion positioning injection system as claimed in claim 4, characterized in that: In the three-dimensional image registration module, based on the centroid of the real-time three-dimensional image and the centroid of the reference three-dimensional image, each point in the source point set and the target point set is removed from the centroid to obtain a removed centroid source point set and a removed centroid target point set, specifically: Performing difference calculation on each point in the source point set and the centroid of the real-time three-dimensional image to obtain the centroid-removed source point set; Subtract each point in the target point set from the centroid of the reference three-dimensional image to obtain the centroid-removed target point set.

6. A fully automatic needle insertion positioning injection system as claimed in claim 4, characterized in that: In the three-dimensional image registration module, the optimal rotation matrix is ​​obtained based on the optimal transformation matrix and the covariance matrix, specifically: Perform singular value decomposition based on the covariance matrix to obtain a corresponding orthogonal matrix; The optimal rotation matrix is ​​obtained by performing calculation based on the corresponding orthogonal matrix and the optimal transformation matrix.

7. A fully automatic needle insertion positioning injection system as claimed in claim 1, characterized in that: In the instrument needle tip point detection module, the position of the instrument needle tip point is obtained based on the registered three-dimensional image, specifically: Acquiring feature images at different resolutions based on the registered three-dimensional image; Based on the feature image, a preset convolution layer is used to detect key point features corresponding to the needle tip of the instrument; Perform feature fusion based on the key point features to obtain a fused feature map; The fused feature map is processed using a preset convolutional layer to obtain the needle tip point position of the instrument.

8. A fully automatic needle insertion positioning injection system as claimed in claim 1, characterized in that: The path planning module is used to perform path planning based on the reference injection position and the instrument needle tip position to obtain a target path, specifically: Acquire a needle tip position coordinate point and a reference position coordinate point based on the instrument needle tip point position and the reference injection position; Calculating based on the needle tip position coordinate point and the reference position coordinate point to obtain a spatial distance between the needle tip position coordinate point and the reference position coordinate point; Calculate the total number of steps based on the spatial distance and the preset step length; Generate a path point based on the total number of steps and the preset step length and in combination with the needle tip position coordinate point and the reference position coordinate point; The path planning for the needle insertion operation is performed based on the path points to generate a target path.

9. A fully automatic needle insertion positioning injection system as claimed in claim 8, characterized in that: In the path planning module, the path points are generated based on the total number of steps and the preset step length and in combination with the needle tip position coordinate point and the reference position coordinate point, specifically: Based on the needle tip position coordinate point and the reference position coordinate point, calculating step by step according to the total number of steps, the preset step length and the spatial distance, to obtain all intermediate points from the needle tip position coordinate point to the reference position coordinate point; The waypoints are generated step by step based on the intermediate points.

10. A fully automatic needle insertion positioning injection method, characterized in that The positioning method adopts the positioning injection system according to any one of claims 1 to 9, comprising the following steps: Obtain a baseline 3D image of the internal structure of the eye; Selecting injection points based on the reference three-dimensional image to determine a reference injection position; In response to the reference injection position, the needle tip of the instrument is driven to perform a needle insertion operation and acquire a real-time three-dimensional image; Performing algorithm registration based on the real-time three-dimensional image and the reference three-dimensional image to obtain a registered three-dimensional image; Acquire the needle tip position of the instrument based on the registered three-dimensional image; Perform path planning based on the reference injection position and the instrument needle tip position to obtain a target path; Based on the target path, the instrument needle tip is guided to reach the reference injection position for injection.

Images