Magnetic control intervention guide wire closed-loop control method based on ultrasonic feedback

Abstract

The invention relates to a magnetic control intervention guide wire closed-loop control method based on ultrasonic feedback, and the method comprises the steps: firstly constructing a guide wire physical model which is used for reflecting the bending deformation of a magnetic guide wire under the action of a magnetic field, and a magnetic drive model which is constructed according to the change of a magnetic field signal generated by a magnetic source in a space where the magnetic guide wire is located; and coupling the guide wire physical model of the magnetic guide wire under the control of a magnetic field signal of the magnetic driving model to obtain a guide wire state change model. Then, the tip of the magnetic guide wire is tracked in real time in an ultrasonic mode, so that a three-dimensional position signal is obtained, and the position of the tip of the magnetic guide wire is accurately recognized. And finally, according to the state change model, the three-dimensional position signal and the pushing model of the magnetic guide wire, the three-dimensional position signal, and the pushing model of the magnetic guide wire are output through a predictive control strategy to obtain an optimal control variable, so that the magnetic guide wire can be controlled to run along a reference path. The purposes of enabling the instrument to obtain accurate three-dimensional real-time positioning and automatic navigation and avoiding radiation damage can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of vascular interventional surgery robot control methods, and particularly to a closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback. Background Art

[0002] Cardiovascular and cerebrovascular diseases have become the main cause of the world's disease burden, with high morbidity and mortality rates. Compared with traditional open surgeries, the currently widely used vascular interventional surgeries have various advantages such as small incisions, high success rates, fast recovery, and no need for general anesthesia. They are widely used in the treatment of various cardiovascular and cerebrovascular diseases. Under the guidance of medical imaging equipment, using instruments such as guide wires, catheters, balloons, and stents, blood flow reconstruction is achieved by dilating stenotic arteries and improving local blood supply, which is an important means for treating various cardiovascular and cerebrovascular diseases.

[0003] However, existing vascular interventional surgeries pose challenges such as high radiation damage, long operation times, and high requirements for doctor experience. Therefore, domestic and foreign companies and institutions have started to develop master-slave vascular interventional surgery robot systems to address the challenges in existing vascular interventional surgeries. In recent years, some commercial vascular interventional surgery robots have been successively developed, and a few have obtained FDA and CE certifications and been launched in foreign countries, and have begun to be applied clinically to assist doctors in better completing interventional treatments. However, existing master-slave interventional surgery robots are still limited to operating passive pre-shaped guide wires / catheters and cannot achieve active steering and autonomous navigation of the instruments.

[0004] Furthermore, in multi-branched blood vessels in the body, achieving three-dimensional real-time positioning and automatic navigation of master-slave interventional guide wires / catheters remains a challenge. Currently, although existing technologies have made great progress in improving interventional surgeries and have carried out medical applications, they generally rely on fluoroscopic imaging as intraoperative guidance, and the X-rays generated pose potential radiation damage to the human body, and it is difficult for fluoroscopic imaging to achieve precise three-dimensional positioning of interventional instruments in the body. In addition, few technologies have achieved real-time positioning and tracking and autonomous navigation of interventional guide wires / catheters in simulated tissues or in the body, so the autonomy of robots still needs to be improved.

[0005] Therefore, aiming at the main drawbacks of the existing technology, the present invention independently designs a closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback, providing an automated and radiation-free interventional surgery solution, enabling doctors to achieve real-time tracking and automatic navigation of interventional guide wires in simulated tissues or in the body without relying on X-rays, thereby greatly improving the navigation efficiency of interventional guide wires, shortening the interventional operation time, keeping doctors away from radiation damage, reducing fatigue damage, and realizing radiation-free vascular interventional surgeries in the future. Summary of the Invention

[0006] To solve the defect that in the prior art, precise three-dimensional real-time positioning and automatic navigation of an instrument cannot be achieved through guidance in fluorescence imaging, and radiation damage will be caused, the present invention proposes a closed-loop control method for a magnetically controlled interventional wire based on ultrasonic feedback.

[0007] The technical solution adopted by the present invention is a closed-loop control method for a magnetically controlled interventional wire based on ultrasonic feedback, including:

[0008] According to the structure of the magnetic wire, a physical model of the magnetic wire changing with the magnetic field is constructed;

[0009] A magnetic drive model is constructed in space according to the type of magnetic source;

[0010] A state change model of the magnetic wire is established according to the magnetic drive model and the physical model of the wire;

[0011] The tip of the magnetic wire is tracked in real time by ultrasound to obtain the three-dimensional position signal of the tip of the magnetic wire;

[0012] According to the state change model, the three-dimensional position signal, and the pushing model of the magnetic wire, the magnetic wire is controlled to run along the reference path through a predictive control strategy.

[0013] Preferably, the physical model of the wire is a Cosserat rod model.

[0014] Preferably, the type of magnetic source is a permanent magnet.

[0015] Preferably, in the step of constructing a magnetic drive model in space according to the type of magnetic source, it includes:

[0016] Performing linearization derivation on the magnetic load;

[0017] Deriving the pose change Jacobian matrix of the permanent magnet.

[0018] Preferably, the predictive control strategy is a model predictive control method.

[0019] Preferably, in the step of controlling the magnetic wire to run along the reference path through a predictive control strategy according to the state change model, the three-dimensional position signal, and the pushing model of the magnetic wire, it includes:

[0020] Based on the pose change Jacobian matrix and the pushing model, a system model of the magnetic wire is constructed;

[0021] Constructing a path tracking control algorithm based on the model predictive control method, and obtaining the optimal spatial velocity of the permanent magnet and the optimal delivery method of the magnetic wire.

[0022] Preferably, after the step of constructing a path tracking control algorithm based on the model predictive control method and obtaining the optimal spatial velocity of the permanent magnet and the optimal delivery method of the magnetic wire, it further includes:

[0023] Convert the optimal space velocity and the optimal delivery method into control signals according to the type of control device, and use the control signals to control the movement of the control device.

[0024] Preferably, after the step of controlling the magnetic wire to run along the reference path through the predictive control strategy according to the state change model, the three-dimensional position signal, and the pushing model of the magnetic wire, it includes:

[0025] Discretize the reference path of the magnetic wire;

[0026] Always place the scanning center directly above each reference path point and make the scanning center close to the scanning plane, so as to perform path planning for the scanning center.

[0027] Preferably, in the step of obtaining the three-dimensional position signal of the tip of the magnetic wire by ultrasonically tracking the tip of the magnetic wire in real time, it includes:

[0028] Ultrasonically track the tip of the magnetic wire in real time to obtain ultrasonic images;

[0029] Collect and crop the ultrasonic images;

[0030] Segment the tip contour of the magnetic wire;

[0031] Calculate the position of the tip of the magnetic wire to obtain the three-dimensional position signal of the tip of the magnetic wire.

[0032] Preferably, in the step of segmenting the tip contour of the magnetic wire and calculating the position of the tip of the magnetic wire, it includes:

[0033] Call the pre-trained U-Net prediction model to segment the mask area of the tip of the magnetic wire;

[0034] Calculate the centroid pixel coordinates of the mask area of the tip of the magnetic wire in the segmented image plane;

[0035] Transform the pixel values of the tip of the magnetic wire in the image coordinate system to the world coordinate system to obtain the absolute coordinates in the world coordinate system;

[0036] Based on the current position and the previous position of the tip of the magnetic wire, calculate the two attitude angles of the tip of the magnetic wire;

[0037] According to the absolute coordinates and the two attitude angles, reconstruct the three-dimensional position signal of the tip of the magnetic wire.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present application discloses a closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback. First, a physical model of the guide wire is constructed to reflect the bending deformation of the magnetic guide wire under the action of a magnetic field, and a magnetic drive model is constructed according to the change of the magnetic field signal generated by the magnetic source in the space where the magnetic guide wire is located. After coupling the physical model of the magnetic guide wire under the control of the magnetic field signal of the magnetic drive model, a guide wire state change model is obtained. Then, the tip of the magnetic guide wire is tracked in real time by ultrasonic waves to obtain three-dimensional position signals, and the position of the tip of the magnetic guide wire is accurately identified. Finally, according to the state change model, the three-dimensional position signal, and the pushing model of the magnetic guide wire, the above three are output through a predictive control strategy to obtain the optimal control variables, so as to control the magnetic guide wire to run along the reference path. The magnetic guide wire is a master-slave type interventional guide wire, and a physical model of the guide wire and a magnetic drive model are constructed based on the magnetic guide wire. The magnetic guide wire can realize the functions of automatic steering and autonomous navigation under the action of the magnetic field of the magnetic source. The further established state change model can make the controllability of the magnetic guide wire higher and obtain more accurate operation accuracy. At the same time, the real-time ultrasonic tracking method can also achieve accurate three-dimensional positioning in the body to the greatest extent, avoiding the situation of poor fluorescence imaging effect. In addition, the ultrasonic tracking method is a non-radiative imaging method and will not cause potential radiation damage to the human body when obtaining three-dimensional position signals. And through the predictive control strategy, closed-loop control can be realized in real time according to the position information of the magnetic guide wire, so as to correct the magnetic guide wire in real time, and then greatly improve the delivery accuracy of the magnetic guide wire.

[0040] Compared with the prior art, a closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback disclosed in the present application can achieve the purpose of enabling the instrument to obtain accurate three-dimensional real-time positioning and automatic navigation and avoiding radiation damage. Brief Description of the Drawings

[0041] The present invention will be described in detail below in conjunction with embodiments and drawings, where:

[0042] Figure 1 It shows a schematic diagram of magnetic guide wire modeling in a closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback provided by an embodiment of the present invention;

[0043] Figure 2 It shows a block diagram of a closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback provided by an embodiment of the present invention;

[0044] Figure 3 It shows a diagram of a 3D real-time ultrasonic tracking method based on ultrasonic feedback in a closed-loop control method for a magnetically controlled interventional guide wire provided by an embodiment of the present invention;

[0045] Figure 4 Shows the flowchart of the path tracking control algorithm based on ultrasonic feedback in a closed-loop control method for a magnetically controlled interventional guidewire provided according to an embodiment of the present invention;

[0046] Figure 5 Shows the experimental result diagram of the path tracking control based on ultrasonic feedback in a closed-loop control method for a magnetically controlled interventional guidewire provided according to an embodiment of the present invention;

[0047] Figure 6 Shows the flowchart of a closed-loop control method for a magnetically controlled interventional guidewire provided according to an embodiment of the present invention. Specific embodiments

[0048] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar components or components with the same or similar functions from beginning to end. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0049] The present invention discloses a closed-loop control method for a magnetically controlled interventional guidewire based on ultrasonic feedback. Please refer to Figure 2 and Figure 6 , including:

[0050] Construct a physical model of the magnetic guidewire varying with the magnetic field according to the structure of the magnetic guidewire;

[0051] Construct a magnetic drive model in space according to the type of magnetic source;

[0052] Establish a state change model of the magnetic guidewire based on the magnetic drive model and the physical model of the guidewire;

[0053] Real-time track the tip of the magnetic guidewire through ultrasonic waves to obtain the three-dimensional position signal of the tip of the magnetic guidewire;

[0054] Control the magnetic guidewire to run along the reference path through a predictive control strategy according to the state change model, the three-dimensional position signal, and the pushing model of the magnetic guidewire.

[0055] First, a physical model of the magnetic wire for reflecting the bending deformation of the magnetic wire under the action of the magnetic field and a magnetic drive model constructed according to the change of the magnetic field signal generated by the magnetic source in the space where the magnetic wire is located are constructed. After coupling the physical model of the magnetic wire under the control of the magnetic field signal of the magnetic drive model, a state change model of the magnetic wire is obtained. Then, the tip of the magnetic wire is tracked in real time by ultrasonic waves to obtain three-dimensional position signals and accurately identify the position of the tip of the magnetic wire. Finally, according to the state change model, the three-dimensional position signal and the pushing model of the magnetic wire, the above three are output through a predictive control strategy to obtain the optimal control variables, so as to control the magnetic wire to run along the reference path. The magnetic wire is a master-slave intervention wire. Based on the magnetic wire, a physical model of the wire and a magnetic drive model are constructed. The magnetic wire can realize automatic steering and autonomous navigation functions under the action of the magnetic field of the magnetic source. The further established state change model can make the controllability of the magnetic wire higher and obtain more accurate running accuracy. At the same time, the ultrasonic real-time tracking method can also achieve accurate three-dimensional positioning in the body to the greatest extent, avoiding the poor fluorescence imaging effect. In addition, the ultrasonic tracking method is a radiation-free imaging method and will not cause potential radiation damage to the human body when obtaining three-dimensional position signals. And through the predictive control strategy, closed-loop control can be realized in real time according to the position information of the magnetic wire, so as to correct the magnetic wire in real time, and then greatly improve the delivery accuracy of the magnetic wire. Compared with the prior art, a closed-loop control method for a magnetically controlled intervention wire based on ultrasonic feedback disclosed in this application can achieve the purpose of obtaining accurate three-dimensional real-time positioning and automatic navigation of the instrument and avoiding radiation damage.

[0056] S10. In the step of constructing a physical model of the magnetic wire varying with the magnetic field according to the structure of the magnetic wire. Among them, the purpose of constructing the physical model of the wire is to realize the control of the deformation of the magnetic wire such as bending, torsion, and stretching, so as to prepare for the subsequent construction of the state change model of the magnetic wire.

[0057] S20. In the step of constructing a magnetic drive model in space according to the type of magnetic source. The type of magnetic source can be a permanent magnet source or an electromagnetic source. Different magnetic field types can be generated according to different types of magnetic sources. By constructing a magnetic drive model in space with this type of magnetic source, the purpose of constructing the magnetic drive model is to obtain the magnetic field signal at any point in space, including its magnitude and direction, so as to prepare for the subsequent construction of the state change model of the magnetic wire.

[0058] S30. In the step of establishing a state change model of the magnetic wire according to the magnetic drive model and the physical model of the wire. In the present invention, by coupling the magnetic drive model and the physical model of the wire, a state change model of the magnetic wire is established to predict its equilibrium configuration under the action of the permanent magnetic field.

[0059] S40, in the step of obtaining the three-dimensional position signal of the tip of the magnetic guide wire by ultrasonically and real-time tracking the tip of the magnetic guide wire. With the aid of ultrasonic imaging technology, the present invention uses ultrasonic imaging to accurately and real-time track the tip of the magnetic guide wire, and based on the position of the magnetic guide wire feedback by ultrasonic, can control the magnetic guide wire to accurately track the reference path and reach the target position, so that doctors can achieve real-time tracking and automatic navigation of the interventional guide wire in the phantom tissue or in vivo without relying on X-rays, stay away from radiation damage, and improve the automation level of the operation.

[0060] S50, in the step of controlling the magnetic guide wire to run along the reference path by a predictive control strategy according to the state change model, the three-dimensional position signal and the pushing model of the magnetic guide wire. Among them, the predictive control strategy control is a way for modern control theory to predict the optimal control variable. The predictive control strategy in this application is preferably model predictive control, and in other embodiments, it can also be predictive control strategies such as dynamic matrix control, internal model control or generalized predictive control.

[0061] In some embodiments, the wire physical model is a Cosserat rod model.

[0062] The Cosserat rod theory is very suitable for the numerical modeling of continuum robots in non-uniform magnetic fields. Therefore, the present invention establishes a state change model of the magnetic guide wire by coupling the magnetic drive model with the Cosserat rod model, for predicting its equilibrium configuration under the action of a permanent magnetic field. The modeling schematic diagram is as Figure 1 shown. The tip of the magnetic guide wire consists of a flexible section and a rigid section.

[0063] The tip of the magnetic guide wire can be modeled as a rod, with M magnets along its length direction. The internal state x of the rod is parameterized by the arc length s relative to the proximal position of the guide wire. Therefore, it can be described by the state vector x(s) as

[0064]

[0065] Assume that a rod consists of N segments, then the mapping relationship of the distal state of any segment relative to its proximal state can be represented by the following transformation

[0066] x i = T i (x i-1 , L i , w ext,i ) (2)

[0067] where L i , w ext,i are the length and external load acting on this segment. If i = 2, the distal state of the magnetic guide wire is given by the following formula

[0068] x2 = T2(T1(x0))(3)

[0069] Transformation of the rigid segment: The internal permanent magnet (IPM) at the tip of the magnetic wire is the rigid segment, and its state transformation is represented by rigid body kinematics as

[0070] p(σ) = p p + σR(σ)v *

[0071] O(σ) = O p (4)

[0072] where σ ∈ [0, L2], L2 is the total arc length of the rigid segment, used to replace the global arc length s. The subscript p represents the proximal end of this segment, and v * is the tangent direction of the center line of the rod in its coordinate system. The equilibrium equations for the internal forces and moments of the rigid cross-section are as follows

[0073]

[0074] where f ext is the total external force applied to the tip rod, and it can be assumed to be a point force at the center of this segment.

[0075] Transformation of the flexible segment: The state evolution of the flexible segment can be described by the differential form of the Cosserat rod equation. The two most crucial parts are the constitutive relation and the equilibrium equation, which can be collectively expressed as

[0076]

[0077] where σ ∈ [0, L1], L1 is the total arc length of the flexible segment, and u * is the intrinsic curvature of the rod in the reference frame.

[0078] K se is the shear / tensile stiffness matrix, and K bt is the bending / torsion stiffness matrix. Here, it is assumed that the tip rod of the magnetic wire can bend and twist, while ignoring shear and tensile strains, satisfying the Kirchhoff rod model (where v * = [0 0 1] T , and K se = 0). K bt is mainly determined by the equivalent elastic modulus of the tip of the magnetic wire.

[0079] Obviously, in other embodiments, the physical model of the wire can also be established according to other continuum robot models.

[0080] In some embodiments, the type of magnetic source is a permanent magnet.

[0081] Among them, the permanent magnet can form a passive magnetic field by itself, and its magnetic field structure is more stable. In addition, the volume and magnetic field strength of the permanent magnet are easy to be selected and controlled. Therefore, it is more suitable to construct a magnetic drive model based on the permanent magnet.

[0082] In some specific embodiments, in the step of constructing a magnetic drive model in space according to the type of magnetic source, it includes:

[0083] Performing a linearization derivation on the magnetic load;

[0084] Deriving the Jacobian matrix of the pose change of the permanent magnet.

[0085] In the method of describing the magnetic field of a permanent magnet, the dipole model is often selected because of its simple calculation. In a moving magnetic drive system, the external permanent magnet (EPM) can be approximately regarded as a point source, so it is simplified to a magnetic dipole model in a non-uniform magnetic field. The magnetic field B generated by the dipole source at the internal permanent magnet (IPM) is expressed as

[0086]

[0087] where μ0 = 4π×10 -7 T·m·A -1 is the magnetic permeability of vacuum, p is the distance vector from the center of the external magnet to the magnetic tip (p = P I - P E ), is the identity matrix, and m E is the magnetic dipole moment of the EPM.

[0088] Linearization derivation of the magnetic load:

[0089] The magnetic load generated by the action of the external magnetic field on the tip of the magnetic wire is a combined form of magnetic force and magnetic torque, that is, W m = [f m , τ m T . By separately extracting the unit vectors and of the magnetic moments of the EPM and IPM, the f m and τ m at the tip of the magnetic wire can be further expanded as

[0090]

[0091] where, and are the magnetic moments of the IPM and EPM respectively; the unit distance vectors and where It is called the identity matrix, and || || represents the Euclidean norm.

[0092] In addition, since it is difficult to solve the EPM pose that generates the desired magnetic force and torque through the above two non-linear equations, it is necessary to linearly represent the magnetic load. The linear expression of the change in magnetic load with respect to the change in EPM pose can be expressed as

[0093] δW m =J w δr m (10)

[0094] where δr m is equivalent to δr E , both representing the change in EPM pose. Mapping the change in magnetic load to the change in EPM pose, the specific derivation process is as follows.

[0095] The relationship between the change in EPM pose and the change in UR5 robot joints can be mapped through the geometric Jacobian matrix and is expressed as

[0096]

[0097] ω E The component of makes the EPM rotate around the axis of the magnetic moment and does not change and the magnetic field. Therefore, there is a fixed relationship Substituting it into the above formula, we can express the small change in EPM pose in the following differential form

[0098]

[0099] where S(a) converts the cross product operation into the form of a skew-symmetric matrix, and approximately maps the small change in EPM pose to the small change in the manipulator joints.

[0100] For the magnetic load W m Taking partial derivatives with respect to its three components respectively, we obtain the following expressions

[0101]

[0102] where p is the distance vector between the IPM and the EPM, which can be calculated by obtaining the end-effector pose of the UR5 robot, and can be calculated by predicting the pose of the magnetic wire tip with the help of a geometric model or a data-driven method.

[0103] Due to the existence of the expression p = P I -P E, the contribution of the IPM pose change to the magnetic load change can be separated from the EPM pose change and is represented by denoted as Substituting the above expression and Equation (12) into Equation (13), δW m can be further expanded as

[0104]

[0105] Considering the actual needs, it is assumed that only the EPM pose change δr E is considered for the contribution to the magnetic load change δW m . Define J w as

[0106]

[0107] where maps the change in the magnetic load to the change in the EPM pose. is the identity matrix, and is the all-zero matrix.

[0108] Derivation of the Jacobian matrix of the magnet pose change:

[0109] Understanding how the pose change of the external permanent magnet (EPM) affects the equilibrium state configuration of the magnetic wire tip is an important prerequisite for subsequent control of the magnetic wire. Therefore, it is necessary to derive the Jacobian matrix of the magnet pose change. Based on the transformation relationship of the above formula (2), the Jacobian matrix corresponding to the distal state of the i-th rod can be expressed as

[0110]

[0111] where the change in the EPM pose r m only affects the magnetic load on the rigid segment. Therefore

[0112]

[0113] where is equivalent to J in (10) w . For the rigid segment, the partial derivative of the state transformation with respect to the magnetic load can be obtained from Equations (4)-(5)

[0114]

[0115] where, if i = 2 (rigid segment), then p2(L2) - p2(0) can be obtained from the rigid segment transformation in Equation (4). For the rigid segment, x2 is the tip state. Therefore

[0116]

[0117] where segment 1 is flexible. Therefore Since the proximal part of the magnetic guide wire is usually stiffer than the tip flexible rod, it is assumed that the proximal state x0 of the guide wire is not affected by the magnetic field change generated by the EPM, that is Therefore, Equation (19) is further expressed as

[0118]

[0119] where is the Jacobian matrix of the EPM pose change, which maps the change in the state of the magnetic guide wire tip to the change in the EPM pose. This Jacobian matrix is the most important part of the subsequent derived magnetic guide wire control Jacobian matrix J c and will provide a general understanding of the quasi-static behavior of the magnetic-driven continuum robot in the permanent magnetic field.

[0120] In some embodiments, in the step of obtaining the three-dimensional position signal of the tip of the magnetic guide wire by real-time tracking of the tip of the magnetic guide wire by ultrasound, it includes:[[]]

[0121] Real-time tracking of the tip of the magnetic guide wire by ultrasound to obtain an ultrasound image;

[0122] Collecting and cropping the ultrasound image;

[0123] Segmenting the tip contour of the magnetic guide wire;

[0124] Calculating the tip position of the magnetic guide wire to obtain the three-dimensional position signal of the tip of the magnetic guide wire.

[0125] To achieve the purpose of accurately obtaining the three-dimensional position signal of the magnetic guide wire tip, it is necessary to obtain an ultrasound image by real-time tracking with ultrasound, and collect and crop the ultrasound image. The purpose of collection and cropping is to further screen the ultrasound images that meet the requirements and crop the ultrasound image at the tip contour to be obtained. After obtaining the ultrasound image at the tip contour, it is necessary to segment the tip contour to avoid interference from other factors, facilitate the calculation and processing in the subsequent steps, and calculate based on the segmented tip contour to obtain the tip position, so as to obtain the three-dimensional position signal of the magnetic guide wire tip.

[0126] In some specific embodiments, in the steps of segmenting the tip contour of the magnetic guide wire and calculating the tip position of the magnetic guide wire, it includes:[[]]

[0127] Calling a pre-trained U-Net prediction model to segment the mask region of the magnetic guide wire tip;

[0128] Calculating the centroid pixel coordinates of the mask region of the magnetic guide wire tip in the segmented image plane;

[0129] Transform the pixel values of the magnetic guide wire tip in the image coordinate system to the world coordinate system to obtain the absolute coordinates in the world coordinate system;

[0130] Based on the current position and the previous position of the magnetic guide wire tip, calculate the two attitude angles of the magnetic guide wire tip;

[0131] According to the absolute coordinates and the two attitude angles, reconstruct the three-dimensional position signal of the magnetic guide wire tip.

[0132] Due to the low signal-to-noise ratio (SNR) and feature similarity of ultrasound images, it is difficult for conventional image processing methods to robustly identify and track the guide wire tip in ultrasound images in real time. The U-Net convolutional network is a relatively advanced method in the field of biomedical image segmentation. Research shows that U-Net is more robust than the geometric method in tracking micro-robots with ultrasound, and the positioning error is smaller. Therefore, the present invention proposes a 3D real-time ultrasound tracking method based on the U-Net network, which can accurately track the magnetic guide wire tip in real time and reconstruct the 3D pose of the tip in the world coordinate system for subsequent closed-loop control.

[0133] The working process of the entire ultrasound tracking method is as Figure 3 (a) shown, which mainly includes three parts: ultrasound image acquisition and cropping, tip contour segmentation, and tip position calculation. The specific implementation steps are as follows: First, capture the original image (720×576) of the ultrasound imaging terminal through a video capture card and crop it into a valid image (256×256) for segmentation. Then, call the pre-trained U-Net prediction model to segment the mask area of the guide wire tip. Next is the tip position calculation part, which needs to complete: 1) Calculate the centroid pixel coordinates of the tip mask in the image plane; 2) Transform the pixel values of the tip in the image coordinate system to the world coordinate system where the UR5 robot base is located, so as to obtain the absolute position of the tip; 3) In order to remove some invalid values caused by ultrasound image noise, it is necessary to filter the tip position to obtain a valid position; 4) Based on the current tip position and the previously saved tip position, calculate the two attitude angles of the tip. Finally, the 3D pose of the guide wire tip is reconstructed and fed back to the controller in real time.

[0134] The training process of the U-Net model used in the above process is as Figure 3As shown in (b), it is divided into four steps. First, using the femoral artery gelatin phantom as the scanning object, ultrasonic images of the tip of the magnetic guide wire at different positions of all arterial branches are collected by an ultrasonic probe, and about 1400 effective pictures are obtained through processing. Secondly, the region of interest (ROI) of the tip in all pictures is manually labeled as a polygon label using the Labelme software, thus generating 1400 pairs of data sets. Then, the data sets are divided into a training set and a test set according to the ratio of 7:3, and then input into a built standard U-Net network for training. After a total of 200 epochs of training, the loss is lower than 0.06, and the best model is obtained. Finally, the trained model is used to predict the tip of the guide wire in a batch of new ultrasonic images, and the result shows that the segmentation accuracy of the model is higher than 95%.

[0135] Figure 3 (c) shows a schematic diagram of coordinate transformation in tip position calculation, aiming to transform the position coordinates of the tip of the guide wire into the world coordinate system where the base of the UR5 robot is located. The coordinate transformation relationship is as follows:

[0136]

[0137] where is the transformation matrix from coordinate system {t} to coordinate system {w}, and other transformation matrices can be defined in a similar way; r u is the pose of the probe in coordinate system {w}, which can be obtained through the sensors of the UR5 robot; w i is the width of the plane of the cropped ultrasonic image; γ is the scale factor, that is, the physical distance corresponding to the length of each pixel in the ultrasonic image coordinate system {I}. The position component in is the position vector Pt of the tip of the guide wire.

[0138] Since the origin of the probe coordinate system {u} is located at the midpoint of the upper edge of the US image plane, it can be deduced that is

[0139]

[0140] where and are the position vector and the orientation matrix rotated 90° around the x-axis respectively, and P uI can be expressed as

[0141]

[0142] The pixel coordinates of the tip of the guide wire in the image coordinate system {I} are denoted as (p x , p y ), and the transformation matrix can be expressed as

[0143]

[0144] wherein and are a position vector and an identity matrix respectively, and P It can be expressed as

[0145] P It =[p x γ p y γ 0] T (26)

[0146] where γ is a scale factor.

[0147] In some embodiments, the predictive control strategy is a model predictive control method.

[0148] The model predictive control (MPC) method can not only predict future values based on a model, but also correct the prediction results according to the actual output values of the feedback (error correction), which can prevent the model mismatch or environmental interference from affecting the prediction results; at the same time, it minimizes the output error through rolling optimization, so as to finally output the optimal control variables. Therefore, the MPC algorithm has stronger robustness and is more suitable for the precise path tracking control of the magnetic wire in the blood vessel with blood flow.

[0149] In some more specific embodiments, in the step of controlling the magnetic wire to run along the reference path according to the state change model, the three-dimensional position signal and the pushing model of the magnetic wire, it includes:

[0150] Constructing a magnetic wire system model based on the pose change Jacobian matrix and the pushing model;

[0151] Constructing a path tracking control algorithm based on the model predictive control method, and obtaining the optimal spatial velocity of the permanent magnet and the optimal delivery mode of the magnetic wire.

[0152] Magnetic wire system model based on the control Jacobian matrix:

[0153] The moving magnetic drive system and the wire pushing module form a coordinated system, where the control variable u is the system input, including the magnet pose r m and the propulsion displacement L, and the state x of the magnetic wire tip is the output of the coordinated system in the world coordinate system. u and x can be expressed as

[0154] u=[r m ,L] T (27)

[0155] x=[x t ,y t ,z t T (28)​

[0156] where is the magnet pose, which determines the magnetic moment W m .

[0157] The mapping relationship of the entire coordination system from the input change to the output change can be represented by the control Jacobian matrix J c as

[0158] δx = J c δu (29)

[0159] where δx is the state change vector of the magnetic wire tip, and δu is the control change vector of the coordination system is the control Jacobian matrix of the magnetic wire, which maps the change of the control variable to the change of the tip state configuration. If a control variable relative to the desired tip state needs to be obtained, the pseudo-inverse of the control Jacobian matrix can be used to solve the kinematic inverse solution of the above equation

[0160] Referring to the representation method of a similar Jacobian matrix in the relevant literature, J c can be expressed as

[0161]

[0162] where is the external magnet pose change Jacobian matrix, given by Equation (21) is the displacement change Jacobian matrix of the magnetic wire pushing module is the propulsion axis vector of the magnetic wire tip. v * is the direction tangent to the centerline of the tip rod in the reference coordinate system

[0163] Path tracking control algorithm based on model predictive control (MPC):

[0164] The Model Predictive Control (MPC) method can not only predict future values based on a model, but also correct the prediction results according to the actual output values of the feedback (error correction), which can prevent model mismatch or environmental interference from affecting the prediction results. At the same time, it minimizes the output error through rolling optimization, and finally outputs the optimal control variable. Therefore, the MPC algorithm has stronger robustness and is more suitable for the precise path tracking control of magnetic guide wires in blood vessels with flow velocity. Here, we propose a path tracking control algorithm based on MPC, specifically using the idea of generalized predictive control. According to the input reference tip path and the tip pose of the real-time ultrasound feedback, this algorithm can call the MPC controller for each desired path point to solve the optimal control increment, and then perform inverse kinematics solution and send the command to each motion module for execution, so as to control the magnetic guide wire tip to reach within the threshold range of the desired path point. Repeat the above process until the tip reaches the final target point. The flow chart of the proposed path tracking control algorithm is as Figure 4 shown.

[0165] In our problem, the system model is the control Jacobian matrix solved above, which can approximately linearly describe the relationship between the input control variable and the output tip state, so it can predict the output states of multiple future time steps through iteration. The predicted state can be expressed as

[0166]

[0167] where N is the prediction step length, is the control increment for each step. In each iteration, we need to update the initial predicted state x with the actual feedback state p,1 .

[0168] The control objective is to solve the optimal control input by optimizing the weighted sum of the output error and the control increment. Therefore, the objective function and constraints can be expressed as

[0169]

[0170] In the formula is the optimal control increment ( the first element of ), and are the output state error, state matrix and control matrix of the magnetic guide wire tip respectively, which can be expressed as

[0171] A more critical aspect of the MPC algorithm is feedback correction, that is, correcting the predicted output state according to the actual state of the ultrasound feedback. The correction process can be deduced as

[0172] x cor,k = x p,k + e p H(35)

[0173] where x cor,k is the corrected state. and are the prediction error and error correction vector.

[0174] After obtaining the optimal control increment at each time step, the corrected state increment must be moved to the next step to continue the iteration and achieve rolling optimization. Finally, after completing all iterations, the controller will output the optimal control increment , and provide the input for the subsequent inverse kinematics solution of each motion module.

[0175] In some more specific embodiments, after the steps of constructing a path tracking control algorithm based on the model predictive control method and obtaining the optimal spatial velocity of the permanent magnet and the optimal delivery mode of the magnetic wire, it further includes:

[0176] Convert the optimal spatial velocity and the optimal delivery mode into control signals according to the control device type, and use the control signals to control the movement of the control device.

[0177] Kinematics of the magnetic drive and push module:

[0178] After the above MPC controller calculates the optimal magnet spatial velocity and forward velocity, it is further converted into joint velocity and pulse number through kinematics, and then sent to the UR controller of the magnetic drive system and the motor controller of the push module to achieve the motion control of the magnet and the stepper motor. The external magnet in the mobile magnetic drive system is installed at the end of the UR5 robot, so the movement of the magnet is determined by the movement of the six joints of the robot. The mapping relationship between the joint velocity and the magnet spatial velocity can be described by the Jacobian matrix J ma of the UR5 robot. According to the optimal magnet spatial velocity calculated by the MPC controller the optimal joint velocity corresponding to the magnet can be obtained by solving the inverse kinematics expressed as

[0179]

[0180] where is the pseudo-inverse of J ma .

[0181] The forward velocity of the wire push module is determined by the rotation angle of the stepper motor, and the rotation angle of the stepper motor is determined by the pulse number received by the motor controller. Therefore, it is necessary to deduce the kinematic model of the push module. According to the optimal propulsion velocity calculated by the MPC controller The relationship between the propulsion speed and the number of pulses can be derived to obtain the optimal number of pulses for the motor. Expressed as

[0182]

[0183] In the formula, r w is the radius of the friction wheel, the variable n p is the actual number of pulses of the motor, and the constant N pr is the number of pulses corresponding to one revolution of the motor (N pr = 360 / δ), where δ is the step angle of the stepper motor.

[0184] In some embodiments, after the step of controlling the magnetic wire to run along the reference path by the predictive control strategy according to the state change model, the three-dimensional position signal, and the pushing model of the magnetic wire, it includes:

[0185] Discretize the reference path of the magnetic wire;

[0186] Always place the scanning center directly above each reference path point and keep the scanning center close to the scanning plane, so as to perform path planning for the scanning center.

[0187] Path planning and control strategy of the ultrasonic probe:

[0188] In order to achieve real-time tracking of the magnetic wire through ultrasonic imaging, the ultrasonic probe needs to closely follow the tip of the wire in real time and coordinate the movement trends and speeds of the two, so as to provide real-time feedback of the tip position to the controller using the above ultrasonic tracking algorithm. For this purpose, we need to plan the reference path of the probe. The idea is to first discretize the tip reference path, and then always place the center of the probe coordinate system directly above each reference path point, and the probe is close to the scanning plane. In addition, the increment of the rotation angle of the probe around the z-axis is consistent with the horizontal turning angle of each tip path point, and the rotation angles of the other two axes are consistent with the initial tracking state. The pose of the planned detection path is expressed as

[0189]

[0190] where [x t , y t and θ t are the x coordinate, y coordinate, and horizontal turning angle of the tip reference path point respectively, and z s is the z coordinate of the surface of the gelatin phantom tissue. From the above formula, the change of the probe reference pose can be further obtained, denoted as

[0191] The movement of the ultrasonic probe is actually determined by the movements of the six joints of the UR5 robot. The mapping relationship between the joint speed and the probe spatial speed (probe pose change) can be represented by the Jacobian matrix of the robot It is described as follows. According to the change of the reference pose of the probe, the inverse kinematics solution is solved to obtain the joint velocity corresponding to the probe It is expressed as

[0192]

[0193] wherein represents the damping pseudo-inverse of J u and the calculation method is the same as that of the same

[0194] It should be emphasized that the principle prototype of the present invention has been manufactured, and closed-loop control experiments have been carried out for three reference paths in the simulated blood vessel model. The results of the path tracking experiment are as Figure 5 shown. The experiment shows that this method can control the guide wire to accurately track the reference path, and the average tracking error of the three paths is 1.50±0.30mm. Therefore, the experiment verifies the feasibility of the provided control method

[0195] In the description of this specification, if terms such as "Embodiment 1", "this embodiment", "in one embodiment" appear, it means that the specific features, structures, materials or characteristics described in connection with this embodiment or example are included in at least one embodiment or example of the invention or the invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example; moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner

[0196] In the description of this specification, terms such as "connection", "installation", "fixation", "setting", "having" are all understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances

[0197] In the description of this specification, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0198] The above description of the embodiments is for the convenience of those of ordinary skill in the art to understand and apply the technology of this case. Those who are familiar with the technology in this field can obviously make various modifications to these examples easily and apply the general principles described herein to other embodiments without creative efforts. Therefore, this case is not limited to the above embodiments, and modifications in the following several situations should be within the protection scope of this case: ① A new technical solution implemented based on the technical solution of the present invention and combined with the existing common knowledge, and the technical effect produced by this new technical solution does not exceed the technical effect of the present invention; ② An equivalent replacement of some features of the technical solution of the present invention using well-known technologies, and the technical effect produced is the same as the technical effect of the present invention; ③ Expansion based on the technical solution of the present invention, and the substantial content of the expanded technical solution does not exceed the technical solution of the present invention; ④ Equivalent transformations made using the content of the specification and drawings of the present invention, directly or indirectly applied in other related technical fields.

Claims

1. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback, characterized in that, Including: Construct a physical model of the magnetic wire according to the structure of the magnetic wire, which describes how the magnetic wire changes with the magnetic field; Construct a magnetic drive model in space according to the type of magnetic source; Establish a state change model of the magnetic wire based on the magnetic drive model and the physical model of the wire; Real-time track the tip of the magnetic wire by ultrasound to obtain the three-dimensional position signal of the tip of the magnetic wire; Based on the state change model, the three-dimensional position signal, and the pushing model of the magnetic wire, control the magnetic wire to run along the reference path through a predictive control strategy.

2. The closed-loop control method of a magnetically controlled interventional guide wire based on ultrasonic feedback according to claim 1, characterized in that, The physical model of the wire is a Cosserat rod model.

3. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback according to claim 1, characterized in that, The type of magnetic source is a permanent magnet.

4. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback according to claim 3, characterized in that In the step of constructing a magnetic drive model in space according to the type of magnetic source, it includes: Perform linearization derivation on the magnetic load; Derive the pose change Jacobian matrix of the permanent magnet.

5. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback according to claim 1, characterized in that The predictive control strategy is a model predictive control method.

6. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback according to claim 4, characterized in that, In the step of controlling the magnetic wire to run along the reference path through a predictive control strategy based on the state change model, the three-dimensional position signal, and the pushing model of the magnetic wire, it includes: Construct a magnetic wire system model based on the pose change Jacobian matrix and the pushing model; Construct a path tracking control algorithm based on the model predictive control method, and obtain the optimal spatial velocity of the permanent magnet and the optimal delivery method of the magnetic wire.

7. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback according to claim 6, characterized in that, After the step of constructing a path tracking control algorithm based on the model predictive control method and obtaining the optimal spatial velocity of the permanent magnet and the optimal delivery method of the magnetic wire, it further includes: Convert the optimal spatial velocity and the optimal delivery method into control signals according to the type of control device, and use the control signals to control the movement of the control device.

8. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback according to any one of claims 1 to 7, characterized in that, After the step of controlling the magnetic wire to run along the reference path through a predictive control strategy based on the state change model, the three-dimensional position signal, and the pushing model of the magnetic wire, it includes: Discretize the reference path of the magnetic wire; Always place the scanning center directly above each reference path point and keep the scanning center close to the scanning plane, so as to perform path planning for the scanning center.

9. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback according to any one of claims 1 to 7, characterized in that, In the step of real-time tracking the tip of the magnetic wire by ultrasound to obtain the three-dimensional position signal of the tip of the magnetic wire, it includes: Real-time track the tip of the magnetic wire by ultrasound to obtain an ultrasound image; Collect and crop the ultrasound image; Segment the contour of the tip of the magnetic wire; Calculate the position of the tip of the magnetic wire to obtain the three-dimensional position signal of the tip of the magnetic wire.

10. A closed-loop control method for a magnetically controlled interventional guide wire based on ultrasonic feedback according to claim 9, characterized in that, In the steps of segmenting the contour of the tip of the magnetic wire and calculating the position of the tip of the magnetic wire, it includes: Call a pre-trained U-Net prediction model to segment the mask region of the tip of the magnetic wire; Calculate the centroid pixel coordinates of the mask region of the tip of the magnetic wire in the segmented image plane; Transform the pixel values of the tip of the magnetic wire in the image coordinate system to the world coordinate system to obtain the absolute coordinates in the world coordinate system. Calculate two attitude angles of the magnetic wire tip based on the current position and the previous position of the magnetic wire tip; Reconstruct the three-dimensional position signal of the magnetic wire tip according to the absolute coordinates and the two attitude angles.

Images