Medical Imaging Guided Magnetic Actuation and Navigation System for Clinical Applications

Abstract

The present invention relates to a device for tracking a magnetic element and a system comprising said device, the device comprising: a) a magnetic actuation unit comprising: i) an eye-in-hand sensing module; ii) a plurality of magnetic sources evenly arranged around the eye-in-hand sensing module, each of the plurality of magnetic sources being inclined at an inclination angle and separated from an adjacent magnetic source by an adjacent angle; and iii) an adjustment mechanism connected to each of the plurality of magnetic sources for adjusting the inclination angle and the adjacent angle to achieve a desired magnetic field; and a robotic platform for mounting the magnetic actuation unit and providing dexterous attitude control of the magnetic actuation unit.

Description

[Technical field]

[0001] The present invention relates to an integrated system combining a mobile magnetic actuation unit and an imaging tool for remote control of a magnetic robot through the body. The mobility of the robotic platform ensures sufficient workspace for magnetic actuation and target tracking for clinical applications. [Background technology]

[0002] Magnetic robotics shows great potential in minimally invasive medical applications. Due to the inherent advantages of magnetic fields such as low distortion when penetrating the human body, harmlessness to biological tissue, and efficient wireless force transmission, medical magnetic robots have been applied to various clinical tasks including, but not limited to, tissue sampling, drug delivery, and real-time diagnosis. Magnetic steering systems that perform multiple degree of freedom (DOF) magnetic control have been proposed to realize dexterous magnetic manipulation. Magnetic steering systems can be classified into two categories according to the magnetic source used: permanent magnet systems (PMS) and electromagnetic systems (EMS).

[0003] In the first category, robotic platforms are commonly utilized to control the pose of permanent magnet actuators to generate the desired magnetic field, making it a safe and effective way to actuate various magnetic robots. However, such systems have difficulty generating complex or rapidly changing modulated magnetic fields. Furthermore, the rapidly decreasing magnetic field limits the reasonable actuation depth and is difficult to turn off. The desired magnetic field is controlled by varying the current supplied to the EMS. Using several electromagnets placed around the workspace, dexterous magnetic manipulation can be achieved, ranging from micrometer robots to centimeter robots. The supplied current controls the magnetic field, but stronger magnetic fields lead to difficulties in temperature control and risk control. The workspace is often surrounded / semi-enclosed by electromagnets, limiting the technique to bench-top experiments. EMS can generate more controllable magnetic fields, but has a rather large footprint and requires supporting subsystems such as temperature monitoring, cooling units, as well as current control units. Magnetic field estimation is also more difficult due to the nonlinear behavior of the soft iron core. The proposed invention includes a mobile electromagnetic actuation unit that employs a robotic platform, which can provide a large working space for the electromagnets and avoid interference with the imaging tool. The dexterity of the system is determined by the number of electromagnets and the controllable degrees of freedom of the robotic platform. The flexibility of the magnetic actuation unit can be designed according to different magnetic field generation requirements.

[0004] To bridge the gap between benchtop testing and clinical applications, various imaging and tracking methods have been developed for in-vivo magnetic robot actuation and navigation. Visual servoing by cameras or stereoscopes is often used in laboratory environments, but is difficult to apply as a tracking tool in clinical applications. Alternative localization and imaging methods are needed for realistic scenarios in non-line-of-sight working environments. X-ray is one of the commonly used medical imaging modalities that can capture information inside the human body with sufficient accuracy and refresh rate. Ultrasound imaging is another effective and efficient imaging tool that can obtain a cross-sectional view of the imaging environment with micrometer-level accuracy. Magnetic localization is a tracking scheme that can extract the absolute position and orientation of the magnetic robot according to the magnetic field generated by the embedded magnet. In the proposed invention, one or more medical imaging tools can be integrated into the imaging and tracking unit for precise and accurate manipulation. Multiple imaging and tracking fusion methods are integrated into the host control unit for different imaging modalities.

[0005] The actuation and navigation methods of medical robots include hydraulic, tendon-driven, and magnetic-driven. Hydraulic actuation can be applied to medical robots to produce active deformation and movement. Through integration with tendons, medical catheter robots can realize functions such as steering, buckling, and stiffness adjustment. Due to the inherent advantages of magnetic fields, such as biological friendliness and low distortion when passing through low-permeability structures, magnetic actuation is widely applied to wireless control of untethered medical robots for in-vivo applications. In the present invention, the actuation magnetic field is generated by a moving magnetic source with a reconfiguration design. It can meet the requirements of various magnetic field generation methods by adjusting the relative position of each magnetic source or increasing the total number of magnetic sources.

[0006] Magnetic medical robots can be classified into tethered and untethered medical robots. Catheterization is one of the widely adopted methods applied in medical procedures such as cardiac ablation, tissue sampling, cell / drug delivery, and vascular disease treatment procedures. Tethered capsule robots are designed for GI tract examination and real-time diagnosis. Alternatively, untethered robots are another type of controlled objects that can better adapt to tortuous environments and perform medical tasks in hard-to-reach areas of the human body. Remote wireless control of untethered robots requires more robust control algorithms and precise imaging tracking methods. Furthermore, the advanced and complex movement principles of the robots impose higher levels of magnetic field generation requirements such that stronger and more complex magnetic fields are needed. Untethered robots are widely used in intravascular applications or GI tract treatments where standard medical tools are difficult to access non-invasively. Aimed at various medical applications, magnetic robots are designed with multiple functions and require different control field requirements. Magnetically driven robots are being investigated to achieve GI tract monitoring or microsurgery in the eye. Torque actuated robots, such as helical or rotating robots, will have more efficient locomotion and small scale. Furthermore, some functional designs of robots, such as drug delivery or tissue sampling, require specific magnetic field designs to trigger the mechanisms precisely. The proposed system will generate complex and controllable fields in a large workspace with optimized configurations. The combination of dexterous magnetic actuation with real-time pose and environment information can be applied to clinical applications such as long-distance drug delivery, clot clearance, whole GI tract monitoring, and therapeutic procedures in the distal endovascular system.

[0007] US Patent No. 7,311,107 B2 relates to a device and control system for operating and locating an in-vivo vehicle by detecting and manipulating the strength and direction of the electromagnetic field vector of said vehicle. It is mentioned that a soft magnetic core can be attached to the electromagnet.

[0008] Another patent, U.S. Pat. No. 6,311,082 B1, discloses a movable bed system for supporting a patient and a set of electromagnets mounted on a curved shell beneath the bed, which can provide a magnetic field or magnetic gradient through appropriate energization of a subset of the magnets.

[0009] U.S. Patent No. 6,148,823 relates to a system for controlling a magnetic element in a body using a gapped toroid magnet with a pair of mutually attractive opposing magnets separated by a gap. The magnets are preferably permanent magnets connected by a magnetic flux return path. It also discloses that the magnets can be mounted on a movable support and the system includes a portable support for the patient and an X-ray imaging device.

[0010] US Patent Application Publication No. 6,216,026 B1 relates to an invention having controllable magnetic moments suitable for navigating a catheter or flexible endoscope within a patient's body when incorporated into an MRI. The disclosed embodiment has three different axially positioned coils, thus providing various combinations of magnetic moments by varying the current input to the three coils.

[0011] US Patent No. 8,830,648 B2 discloses the magnetic actuation system Octomag, which is a magnetic system with two groups of electromagnetic coils, each group equipped with four electromagnets or permanent magnets, capable of controlling millimeter-scale robots to move with five degrees of freedom in a small workspace such as the cavity of the human eye.

[0012] A further device is disclosed in US Patent No. 8,041,411 B2, which discloses a magnetic control system having a circumferentially movable magnet attached to a curved arm and using X-ray imaging to navigate the magnetic element.

[0013] A device consisting of eight magnetic coils with magnetic cores mounted on two opposing spherical shells is introduced in US Patent No. 8,027,714 B2. The embodiment allows a servo motor to position the relative distance between the two above electromagnet sets and a radar to be installed for imaging to eliminate radiation such as X-rays. The embodiment can generate magnetic field strengths and gradients between B=0.04T-0.15T and 1.6T / m-3.0T / m, respectively, thus exerting sufficient strength and orientation effects to move a magnetically responsive catheter tip. Summary of the Invention

[0014] The present invention provides a device for tracking a magnetic element. In one embodiment, the device comprises: a) a magnetic actuation unit, the magnetic actuation unit comprising: i) an eye-in-hand sensing module; ii) a plurality of magnetic sources evenly arranged around the eye-in-hand sensing module, each of the plurality of magnetic sources being tilted at an inclination angle and separated from an adjacent magnetic source by an adjacent angle; and iii) an adjustment mechanism connected to each of the plurality of magnetic sources for adjusting the inclination angle and the adjacent angle to achieve a desired magnetic field; and b) a robotic platform for mounting the magnetic actuation unit and providing dexterous attitude control of the magnetic actuation unit.

[0015] The present invention also provides a system for tracking a magnetic element. In one embodiment, the system comprises: a) a magnetic element; b) one or more devices according to claim 1; c) an eye-to-hand imaging system (2); and d) a computer processor, the computer processor executing an algorithm for coordinating the one or more devices with the eye-to-hand imaging system (2), the algorithm comprising: i) receiving pose information of components in the system; ii) calculating a desired trajectory of the one or more devices according to claim 1 and the eye-to-hand imaging system (2) before operation based on a command from a user to steer the magnetic element; iii) analyzing in real time the position of an object in the field of view of the eye-to-hand imaging system during the operation; and iv) adjusting the desired trajectory based on the real-time position using a predictive control strategy to detect potential collisions. [Brief description of the drawings]

[0016] [Figure 1] 1 shows a schematic diagram of an integrated system for magnetic control of a magnetic element within a patient's body. [Diagram 2] A workflow diagram of the integrated system is shown. [Figure 3a] 1 illustrates one embodiment of an electromagnetic source design showing adjacent angles. [Figure 3b] 1 illustrates one embodiment of an electromagnetic source design showing the tilt angle. [Figure 3c] 1 illustrates an embodiment of the present invention showing a reconfigurable coil support structure design for individual coils. [Figure 3d] 1 shows one embodiment of the present invention in which the attitude of individual electromagnets can be adjusted by a single rotary actuator. [Figure 3e] An embodiment of the attitude adjustment device of the present invention is shown, which is composed of a linear actuator (28), a universal ball joint (29), and a revolute joint (30). [Figure 3f]1 illustrates an embodiment of the attitude adjustment design of the present invention that uses a single linear actuator to adjust the attitude of individual electromagnetic coils. [Figure 3g] Four embodiments of the core chip design of the present invention are presented. [Figure 3h] 1 shows two embodiments of the electromagnetic coil shape design of the present invention. [Figure 4a] FIG. 1 illustrates a cross-sectional view of one embodiment of an electromagnetic coil cooling design. [Figure 4b] 1 illustrates one embodiment of a cooling channel design. [Figure 4c] 4c-4f show four embodiments of the core cooling system of the present invention. [Figure 4d] 4c-4f show four embodiments of the core cooling system of the present invention. [Figure 4e] 4c-4f show four embodiments of the core cooling system of the present invention. [Figure 4f] Figures 4c to 4f show four embodiments of the core cooling system of the present invention. [Figure 5a] Figure 5a shows a serial robot arm designed to achieve motion control by using two revolute joints (36, 37) to control in-plane motion and two other revolute joints to control out-of-plane motion. [Figure 5b] A robotic platform designed with a leverage structure for sufficient load capacity to support a larger magnetic source is shown. [Figure 5c] 1 illustrates one embodiment of a robotic platform structure of the present invention. [Figure 5d] 1 shows an embodiment of the design of the present invention that uses a moving magnetic source to achieve multiple magnetic field control points. [Figure 6a] Figure 6a shows an embodiment of the design of the eye-in-hand tracking system, where the ultrasound probe is held by a micro-Stewart platform capable of controlling six degrees of freedom (DOF) of the ultrasound (US) imaging plane. [Figure 6b]1 shows an embodiment of an eye-in-hand tracking system design in which one rotational DOF is controlled by a rotational actuator. [Figure 7a] 7a-7c show an embodiment of a magnetic isolation shield design (50) and a magnetic field monitoring system design (51). [Figure 7b] 7a-7c show an embodiment of a magnetic isolation shield design (50) and a magnetic field monitoring system design (51). [Figure 7c] 7a-7c show an embodiment of a magnetic isolation shield design (50) and a magnetic field monitoring system design (51). [Figure 8a] Figure 8a shows an arrangement of the subsystems such that the magnetic actuation and imaging tools share the same human-sized workspace. The subsystems are mounted on rails (58, 54) that allow stable over-the-ground movement around the patient bed. [Figure 8b] 1 shows details of the design of a magnetic source according to the present invention, consisting of four coils and an eye-in-hand ultrasound imaging tool (59). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The present invention provides an integrated magnetic system for actuation and navigation of magnetic elements through the body, consisting of a moving magnetic source for generating a controllable magnetic field in a large working space, an imaging and tracking system for locating targets and acquiring internal information of the body, a central processing system for executing control algorithms and data analysis, a human-machine interface for translating operator's commands into control signals, and an assisted bed equipped with medical devices that can reflect the patient's condition. The magnetic source is controlled by a robotic platform that allows the movement of a heavy-duty magnetic source in a human-scale working space. The desired magnetic field is generated by the cooperation of the robotic platform and the supplied electric current. The motion control of the imaging and tracking system results in the same human-scale tracking space as the magnetic actuation system and consists of multiple imaging modalities and tracking tools.

[0018] Based on the prior art, one objective of the present invention is an integrated system combining magnetic field actuation and medical imaging tools for maneuvering and navigating in a large workspace of a medical magnetic robot for in vivo clinical applications. The desired magnetic field generation in a large workspace is achieved by cooperating a robotic platform and a magnetic source. To meet the requirements of clinical applications, an imaging and tracking subsystem is integrated to locate the controlled object and image the internal information of the body.

[0019] One feature of the present invention is that mobility ensures successful cooperation of magnetic actuation and imaging modalities. For different medical applications, the present invention can be adjusted to generate specific control fields accordingly without interfering with the quality of imaging and tracking. Furthermore, the magnetic source can be moved to the vicinity of the controlled object to generate the desired actuation magnetic field, so that current overshoot and saturation of the soft iron core can be avoided. The present invention has a human-scale working space and can also be focused on a specific area without using bulky magnetic sources.

[0020] A cluster of reconfigurable magnetic sources generates the magnetic field. In one embodiment, the magnetic sources are made of permanent magnets. The attitude control device generates the desired magnetic field that controls the orientation of the magnets. Based on the shape of the permanent magnets and the desired working magnetic field, the positioning device can be designed accordingly. The working magnetic field, such as a rotational field or a vibration field, can be realized by adjusting the movement pattern of each magnetic source. In one embodiment, the magnetic sources are electromagnetic coils. The desired magnetic field can be generated by the cooperation of the robotic platform and the supplied electric current. According to the imposition principle, the magnetic field at the desired position is the sum of the magnetic fields of each magnetic source. By changing the relative position of each electromagnet, a specific field can be achieved, including but not limited to a gradient-dominant field, a torque-dominant field, a gradient-free field, and a torque-free field.

[0021] The electromagnetic source assembly has a reconfigurable design to accommodate the relative configuration adjustment device. In one embodiment, the magnetic source is composed of several permanent magnets that can be rotated and translated to generate the desired magnetic field. In one embodiment, the magnetic source is composed of a cluster of electromagnetic coils. The supplied current, orientation, and relative position can be adjusted to generate the desired field. In one embodiment, the core of the magnetic coil is made of soft iron and has internal cooling channels. The tip shape is designed to meet various magnetic field generation requirements.

[0022] In one embodiment, a temperature control subsystem is invented to protect the electromagnetic source, including a cooling module, a temperature sensing module, and a circulation module. In one embodiment, the cooling module consists of external cooling, where the cooling device is on the surface of the coil. In one embodiment, an internal cooling design is implemented in the core of the electromagnetic core. The internal temperature can be better controlled by designing cooling channels inside the soft iron core. If the external cooling unit cannot provide sufficient temperature control, a design of the internal cooling structure designed by finite element analysis is developed. To overcome the nonlinearity introduced by the internal cooling coil, a numerical method is invented to model the generated magnetic field. To provide a better knowledge of the real-time state of the coil, a six-point temperature sensing deployment method is invented. In one embodiment, the coolant circulation speed and the cooling temperature are adjusted according to the feedback sensed temperature.

[0023] For the electromagnetic source, a method of spatial arrangement of electromagnetic coils is invented to generate the desired magnetic field without compromising accessibility and compatibility. In one embodiment, the coils are arranged symmetrically with the axes coinciding at one point so that the focal point can have the maximum magnetic field strength. In one embodiment, multiple focal points are designed to allow multiple local maximum fields. In one embodiment, the coils are arranged to generate a field of uniform strength. In one embodiment, the coils are arranged to generate a field of uniform gradient.

[0024] A current regulating device is invented to provide a large four-quadrant power supply to the electromagnetic coil. The current regulating subsystem consists of a bidirectional DC power supply, an H-bridge circuit supported by an H-bridge driver circuit, and a microcontroller to generate a high frequency control PWM signal.

[0025] A design method of magnetic actuation system configuration is invented to avoid collision with the patient bed and interference with the imaging and tracking system. Based on the method of the present invention, the workspace is scalable for different experimental subjects and can be coordinated with the doctor's examination.

[0026] In one embodiment, a bedside installation configuration is designed. It can be matched with the C-arm imaging module and allows sufficient access space for doctor's examination. In one embodiment, the magnetic actuation system is installed on a U-shaped sliding rail to better adjust to the operating room environment. In one embodiment, multiple sets of moving magnetic sources are designed to focus on multiple application points or optimize the magnetic field generation at one point.

[0027] A mobile magnetic source supported on a robotic platform is invented to provide multiple DOFs for magnetic field generation. Mobility can be realized in different configurations. In one embodiment, a serial robot arm with a smaller footprint but more stringent load requirements supports the magnetic source. In one embodiment, a 3DOF robotic structure made with a screw table is invented to improve load and precision. In one embodiment, a side-mounted parallel robotic structure is used to adjust the pose of the magnetic source. Mobility can be realized in different configurations.

[0028] A hybrid medical imaging system is invented to fuse different types of tracking and imaging modules. In one embodiment, an eye-to-hand imaging system integrated with a mobile C-arm is designed to realize tracking of controlled targets in a large workspace. In one embodiment, an eye-in-hand tracking system is integrated with a moving magnetic source. Such a design ensures accurate tracking performance in a large workspace without complex structural design or additional sensing assistance modules. In one embodiment, an eye-to-hand sensing system and an eye-in-hand sensing system are combined to provide information for intrabody magnetic robot navigation.

[0029] A magnetic field hybrid modeling method is invented to estimate the magnetic field in the workspace for various electromagnetic coil designs. Prior inventions present techniques to model the magnetic field with a dipole model, a modified dipole model, and an analytical eddy current-based model. The modeling method of the present invention combines FEM data using a learning-based mapping approach and calibrates the data to estimate the generated fields for various shape and core designs.

[0030] A magnetic field prediction method for a moving magnetic source is invented such that the duration of the transition between each position is taken into account. By implementing the method of the present invention, the magnetic field exerted on the controlled object is continuous. One challenge of the magnetic actuation system of the present invention is to accurately predict the magnetic field.

[0031] The integrated system of the present invention combines a moving magnetic source with a medical imaging tool to provide precise magnetic navigation and actuation for clinical applications. The magnetic system can move with the controlled target and navigate the magnetic elements through long distances without using high currents or superconducting materials. The imaging system cooperates with the magnetic source to provide real-time controlled target posture and internal body information. The multi-functional patient bed is designed to provide connectivity to medical devices. In one embodiment, sensing and tracking modules are integrated into the patient bed to provide localization of the controlled object. In one embodiment, the patient bed is integrated with an ECG and a ventilator.

[0032] The present invention provides an integrated navigation system combining mobile magnetic actuation and medical imaging tools to manipulate magnetic elements for in vivo applications. In one embodiment, the system includes: (a) a mobile magnetic source controlled by a robotic platform to realize large workspace magnetic field generation in a patient's body; (b) an imaging system for providing position information and intrabody information of a controlled element in a human-scale workspace, where an eye-in-hand sensing module and an eye-to-hand imaging system cooperate to provide accurate and precise localization for control and diagnostic purposes, where the eye-in-hand sensing module moves with the magnetic source in pace to actively track the controlled element, and the eye-to-hand imaging system is controlled by a mobile structure to provide imaging and localization information of the patient's body; and (c) a ground rail on the robotic platform to provide stable positioning of the magnetic source around the patient's bed. (d) an integrated patient bed, which is compatible with medical monitoring devices and provides the patient's status to a central processing module; (e) a central processing module consisting of a host computer and a DAQ microcontroller, which processes feedback data from the imaging system and gives instructions to the motion control unit and the current control unit; and (f) a human-machine interface, which converts operator commands into specific motor and current signals and sends instructions to corresponding subsystems, whereby detailed operating processes and states of controlled elements are visualized on a monitor.

[0033] In one embodiment, the motion control of the magnetic source of the present invention is achieved by a composite leverage structure that can provide multiple DOFs and heavy loads simultaneously, where the magnetic source is supported by a rotary joint that stands on a base joint so that the weight is transferred to the base part instead of being supported by the rotary joint, said rotary joint is the fulcrum, and the distal end of the leverage is actuated by a linear actuator that controls the in-plane rotational motion, and said linear actuator connecting the magnetic source controls the distance between the magnetic field and the patient.

[0034] In one embodiment, the imaging system of the present invention is a collaboration of motion control, an eye-in-hand imaging module, and an eye-to-hand (external) imaging module, and a collaborative motion control method is invented to avoid interference and collision with magnetic actuation methods, the eye-in-hand tracking devices include, but are not limited to, ultrasound imaging, photoacoustic imaging, magnetic sensor arrays, cameras, and infrared imaging, and the eye-to-hand imaging tools include, but are not limited to, x-ray, fluoroscopy, fluorescence imaging, and laser-induced fluorescence.

[0035] In one embodiment, a ground rail around the patient bed of the present invention is for supporting and controlling the attitude of the magnetic sources, and the arrangement of the rails can be two separate or U-shaped which can accommodate the two ends of the patient bed, and the number of magnetic sources can be more than two without affecting the imaging module and the patient.

[0036] In one embodiment, the electromagnetic source of the present invention is controlled by a current control unit of the present invention, which consists of three units: a bidirectional power supply unit, an h-bridge regulation unit, and a microcontroller unit.

[0037] In one embodiment, the magnetically controlled magnetic elements of the present invention include magnetic catheters, tethered magnetic robots, magnetic swarm particles, and magnetic endoscopes.

[0038] In one embodiment, the human-machine interface of the present invention is an integrated module that combines a joystick, pedals, a steering wheel, and 3D glasses, incorporating voice control, attitude control algorithms, and augmented reality technology.

[0039] In one embodiment, the imaging and tracking devices of the present invention collect multi-dimensional information including internal body information, absolute target pose, temperature information, magnetic field information, and cross-sectional views of the patient body into a central processing system, which fuses the data together to generate the desired magnetic fields and control the movement of the robotic structure.

[0040] The present invention also provides a method for controlling an integrated mobile magnetic actuation and imaging system that can achieve simultaneous localization and navigation of magnetic elements. In one embodiment, the method includes the steps of: (a) an imaging module providing information of the controlled elements, the location of the magnetic source, and the robotic platform structure to a central processing system; (b) data fusion, calculation of the desired magnetic field, translation of motion commands, and motion control of the robotic structure are processed on the central processing system, which sends control signals for each subsystem to the distributed processing units; (c) a magnetic source is moved to the vicinity of the magnetic elements to calculate a current that can generate the desired magnetic field, and a temperature control unit controls the cooling and circulation of coolant in the internal and external cooling channels. (d) the imaging tool motion control unit moves the external imaging device so that the magnetic element is moving within the field of view and the current position of the magnetic element is fed back to the central processing system; (e) the desired current and movement of the magnetic source is updated on the central processing unit; (f) the human-machine interface updates operating instructions from the operator and translates the instructions to the central processing system into specific magnetic field and movement requirements; and (g) patient information, status of the magnetic element, and internal body information are presented on a screen to assist operation.

[0041] In one embodiment, the temperature control system of the present invention consists of two units, an inner circulation channel is built into a soft iron core with temperature sensors installed at the inlet and outlet, and an outer circulation channel is wound onto the surface of an electromagnetic coil with temperature sensors built into the core surface, the coil surface, the inlet and outlet of the circulation channel.

[0042] In one embodiment, the internal cooling system of the present invention is designed to balance the overall volume and heat generation and channels are distributed through the core.

[0043] In one embodiment, a magnetic modeling method for a non-conventional electromagnetic coil having an internal structured soft iron core and a customized electromagnetic coil geometry, where the simulated magnetic field is calibrated with a manually measured magnetic field by a learning-based modeling method.

[0044] In one embodiment, the eye-in-hand imaging system of the present invention comprises one or more types of tracking tools, such as ultrasound imaging, a magnetic sensor array, and a camera, while the ultrasound probe is controlled by a positioning device such as a Stewart platform or a rotary joint.

[0045] The present invention further provides a hierarchical control method for coordinating motion control and magnetic field generation to avoid collision and interference between the imaging system and the actuation system, where the first level is safety distance control and collision control, and the second level is magnetic field generation control and imaging field of view control.

[0046] In one embodiment, the controlled magnetic elements include, but are not limited to, magnetic catheters, tethered magnetic microrobots, and magnetic capsule endoscopes.

[0047] The present invention further provides an electromagnetic shielding design that is made from multiple layers of materials with different magnetic permeabilities, while the electromagnetic geometry is designed to close the magnetic field inside the workspace.

[0048] The present invention also provides a magnetic field monitoring system design consisting of different sensors, including but not limited to magnetic sensors, temperature sensors, and humidity sensors, and the sensor positions are located within a relatively large distance from the working space to observe the status of the electromagnetic source in real time.

[0049] The present invention provides a device for tracking a magnetic element. In one embodiment, the device comprises: a) a magnetic actuation unit, the magnetic actuation unit comprising: i) an eye-in-hand sensing module; ii) a plurality of magnetic sources evenly arranged around the eye-in-hand sensing module, each of the plurality of magnetic sources being tilted at an inclination angle and separated from an adjacent magnetic source by an adjacent angle; and iii) an adjustment mechanism connected to each of the plurality of magnetic sources for adjusting the inclination angle and the adjacent angle to achieve a desired magnetic field; and b) a robotic platform for mounting the magnetic actuation unit and providing dexterous attitude control of the magnetic actuation unit.

[0050] In one embodiment, the eye-in-hand sensing module comprises one or more components selected from the group consisting of a camera, a 2D / 3D ultrasound probe, an X-ray generator, an electrostatic sensor, a magnetic sensor, a fluorescent sensor, an optical transmittance meter, a force sensor, a grating sensor, a photoacoustic probe, a laser speckle imager, an infrared thermography camera, a radio frequency probe, and a humidity sensor. In another embodiment, the ultrasound probe is controlled by a positioning device selected from the group consisting of a Stewart platform and a revolute joint.

[0051] In one embodiment, the adjustment mechanism includes a configuration selected from the group consisting of: a) two linear actuators (17) and two universal ball joints (19); b) a four-bar link formed by four revolute joints (26, 27); c) a linear actuator (28), a universal ball joint (29), and a revolute joint (30); and d) a linear or revolute actuator (31).

[0052] In one embodiment, the multiple magnetic sources include electromagnets or permanent magnets. In another embodiment, the electromagnet comprises a soft iron core, copper wire, and a temperature control module for controlling the temperature of the soft iron core, the temperature control module comprising: a) a cooling channel having an inlet and an outlet, the cooling channel looping around or inside the soft iron core; b) a cooled heat exchange fluid entering and exiting the cooling channel through the inlet and the outlet, respectively; and c) a temperature sensor located at the inlet, the outlet, and embedded in or on the soft iron core. In a further embodiment, the soft iron core has a stepped design. In yet another embodiment, the soft iron core comprises a cylindrical, conical, stepped, or rounded tip shape.

[0053] In one embodiment, a current control unit is used to control the current supplied to the electromagnet to achieve the desired magnetic field, the current control unit comprising a bidirectional power supply unit, an h-bridge adjustment unit, and a microcontroller unit.

[0054] In one embodiment, the magnetic element is selected from the group consisting of a tethered magnetic robot, a magnetic catheter, a magnetic guidewire, a magnetic needle, a magnetic sleeve, a magnetic soft robot, a magnetic continuum robot, a tethered magnetic robot, a magnetic helical swimmers, a magnetic roller, a magnetic gripper, a magnetic surgical tool, a magnetic endoscope, and a magnetic capsule endoscope.

[0055] In one embodiment, the device further includes a magnetic isolation shield made of layers of materials having different magnetic permeabilities.

[0056] In one embodiment, the robotic platform comprises an arrangement for dexterous attitude control of the magnetic actuation unit, the arrangement being selected from the group consisting of: a) two revolute joints (36, 37) for controlling in-plane motion and two revolute joints (38) for controlling out-of-plane motion; b) a first linear actuation mechanism (39) supported by a first revolute joint (40) and a second revolute joint (42), the second revolute joint being connected to a second linear actuation mechanism (41); and c) a rotary actuator (47) and a linear actuator (48).

[0057] In one embodiment, the robotic platform includes rails (43) for stable translational movement of the device.

[0058] The present invention also provides a system for tracking a magnetic element. In one embodiment, the system comprises: a) a magnetic element; b) one or more devices according to claim 1; c) an eye-to-hand imaging system (2); and d) a computer processor, the computer processor executing an algorithm for coordinating the one or more devices with the eye-to-hand imaging system (2), the algorithm comprising: i) receiving pose information of components in the system; ii) calculating a desired trajectory of the one or more devices according to claim 1 and the eye-to-hand imaging system (2) before operation based on a command from a user to steer the magnetic element; iii) analyzing in real time the position of an object in the field of view of the eye-to-hand imaging system during the operation; and iv) adjusting the desired trajectory based on the real-time position using a predictive control strategy to detect potential collisions.

[0059] In one embodiment, the system further comprises a patient bed having an integrated sensing and tracking module.

[0060] In one embodiment, the eye-to-hand imaging system is integrated with a mobile C-arm.

[0061] In one embodiment, the magnetic element is selected from the group consisting of a magnetic catheter, a tethered magnetic microrobot, and a magnetic capsule endoscope.

[0062] In one embodiment, the multiple magnetic sources comprise electromagnets and a current control unit for controlling current supplied to the electromagnets to achieve the desired magnetic field, the current control unit comprising a bidirectional power supply unit, an h-bridge regulation unit, and a microcontroller unit.

[0063] In one embodiment, the algorithm further includes a hierarchical control method for coordinating the motion and magnetic field generation of the system, the hierarchical control method including a first level for controlling safe distance and preventing collisions, and a second level for controlling magnetic field generation and imaging field of view.

[0064] In one embodiment, the system further comprises a module for a user to intervene and govern the control of the system beyond the algorithm.

[0065] FIG. 1 shows a schematic diagram of an embodiment of an integrated magnetic actuation and navigation system. The system comprises a mobile magnetic control device (1) with a human-scale workspace, an imaging device (2), a support bed (3) for a patient (4), a central processing system (5), a visualization system (6), a human-machine interface (7), and a power control unit (8). According to the present invention, a mobile electromagnetic source (1) is designed to actuate elements inside the body of the patient (4). The mobile mobility is achieved by a robotic platform consisting of (11, 12, 13, 14). (11) is a linear rail actuator, which moves the magnetic source in a human-scale range. (12) and (13) provide localized magnetic source adjustment. The magnetic actuator attached to (14) includes a cluster of magnetic sources. In one embodiment, a set of electromagnets is utilized to generate a controllable magnetic field, such as a uniform magnetic field or a gradient magnetic field. An external medical imaging subsystem movement mechanism (2) is designed to provide human-scale tracking and imaging capabilities. The imaging system has a base joint (21), a rotating waist joint (22), two rotating joints (23, 24) and a C-arm (25) at the end.

[0066] In the embodiment shown in FIG. 1, a central processing module (5) controls the motion and magnetic field generation of the magnetic actuation subsystem (1). In one embodiment, the central processing module consists of a host computer and a microcontroller. In one embodiment, a pose control device is used to control the permanent magnet to generate the desired magnetic field. In one embodiment, the supply current of the electromagnetic source is controlled to generate the desired magnetic field. The above-mentioned processing module collects the sensed data and generates the desired magnetic field. In this embodiment, a joystick (7) is used for convenient operation for the operator, and the human machine interface incorporates algorithms for magnetic field control and motion control. Other embodiments of the human machine interface, such as touch screen, keyboard, pose control, and voice control, are also applicable to the present invention. The system includes a power supply unit (8) that provides power to the robotic platform and the magnetic source. The imaging device (2) has an eye-to-hand configuration in which a mobile support structure allows multiple DOF control of the imaging area.

[0067] FIG. 2 shows a schematic diagram of the workflow of the integrated system. The whole system consists of five modules: 1) a mobile magnetic actuation subsystem that receives commands from a central processing module to generate a precise magnetic field in a large workspace; 2) an imaging and tracking subsystem that provides posture information and internal information inside the patient body; 3) a central processing module that implements control and tracking algorithms to adjust the system to avoid interference and collision; 4) a human-machine interface that is convenient to access and can reduce the workload of the doctor; and 5) a patient bed supported by medical devices that provide the patient's status. In the mobile magnetic actuation subsystem, a robotic platform controls the posture of the magnetic source and moves the actuation magnetic field to the human-scale workspace. In one embodiment, the magnetic source comprises a cluster of electromagnetic coils that generate a desired magnetic field according to the current supplied. The magnetic actuation subsystem translates the desired magnetic field into current and motion commands and transmits this information to different modules. A temperature control module is built into the actuation subsystem to avoid overheating and nonlinearity of the magnetic field generation. Furthermore, compatibility with the imaging and tracking subsystem is governed by a collision and interference control module. In the imaging and tracking subsystem, localization and imaging information from both eye-to-hand and eye-in-hand imaging subsystems are fused to achieve precise intrabody magnetic tracking and navigation of the controlled element. A motion control mechanism supports the eye-to-hand imaging and tracking system to achieve imaging of a large workspace. In one embodiment, a mobile C-arm is employed to position the external imaging system. In one embodiment, a 6DOF robotic arm is used to control the movement of the external imaging system.

[0068] FIG. 3 shows some possible magnetic source design embodiments according to the present invention. FIG. 3a shows one embodiment of the electromagnetic source design. The coils are evenly distributed around the eye-in-hand sensing module (11) for simple calculation. In one embodiment, ultrasound and camera are utilized to sense the controlled element. (9) is the adjacent angle between the axis of each coil, which determines the number of electromagnetic coils. (10) is the individual magnetic coils, which are composed of soft iron cores, copper wires, and a cooling system. (16) is the tilt angle between the axis of the individual magnetic coil and the central axis. These two angles are designed according to the requirements of the magnetic field. The adjacent angle determines the number of individual magnetic sources and the DOF of the magnetic field generation. The tilt angle shapes the magnetic field generation profile. In one embodiment of the present invention, the attitude of the individual magnetic sources can be configured by linear actuators or rotary actuators. FIG. 3c shows an embodiment of the present invention showing a reconfigurable coil support structure design for the individual coils. (17) is a linear actuator and (19) is a universal ball joint. (18) is a static structure that connects to the robotic platform, and (20) is an individual electromagnetic coil. Figure 3d shows an embodiment of the present invention where the attitude of the individual electromagnets can be adjusted by one rotary actuator. (26) and (27) are rotary joints that form a four-bar linkage. A rotary actuator installed in one of the joints can change the attitude of the magnetic source. Figure 3e shows an embodiment of the attitude adjustment device of the present invention, which is composed of a linear actuator (28), a universal ball joint (29), and a rotary joint (30). Figure 3f shows an embodiment of the attitude adjustment design of the present invention that uses a single linear actuator to adjust the attitude of the individual electromagnetic coils. Figure 3g shows four embodiments of the core tip design of the present invention. The cylindrical tip design allows for a flat magnetic field generation profile, while the conical shape can focus a larger magnetic field strength near the tip. The stepped tip design allows for a strong field in a relatively large area in front. The rounded tip design allows for a magnetic field profile similar to the dipole model, which is advantageous for magnetic field modeling. FIG. 3h shows two embodiments of the electromagnetic coil shape design of the present invention.(32) is a step-shaped coil design that allows a stronger magnetic field to be generated with the same amount of copper wire.

[0069] FIG. 4 shows an embodiment of the design of the temperature control system. FIG. 4(a) shows a cross-sectional view of one embodiment of the electromagnetic coil cooling design. (33) is a cooling channel made of a high thermal conductivity material such as copper. The cooled heat exchange fluid is circulated in (33) to bring the excess heat from the surface of the electromagnetic coil. FIG. 4(b) shows one embodiment of the cooling channel design. Temperature sensors are installed in the soft iron core, in the copper wire, at the outlet and inlet of the circulation channel. FIG. 4c-4f show four embodiments of the core cooling system of the present invention. The inner cooling channel of the soft iron core circulates the same heat exchange fluid as the outer cooling channel. Several sets of loops increase the cooling efficiency, and the inlet and outlet temperatures are monitored for circulation speed regulation. An embodiment of the design of the channel shape is shown in the figure. The cooling channel will occupy the volume of the soft iron core, thereby reducing the magnetic field generation capacity, so it needs to be designed with the cooling efficiency and magnetic field generation requirements in mind.

[0070] A magnetic field modeling method is invented for an electromagnetic source with an internally structured soft iron core. Previous studies that reported magnetic field modeling techniques include dipole field modeling, FEM modeling and learning-based modeling. The hybrid modeling method of the present invention is suitable for electromagnetic fields with various shapes and internal structures. There are three steps: first, a simulation of the magnetic field distribution of a CAD magnetic coil model; second, the magnetic distribution is manually measured at several points in the working space or region of interest; and third, the simulation data is calibrated with the measured data by a learning-based mapping modeling method. By increasing the number of measurements, the modeling accuracy can be improved.

[0071] FIG. 5 shows an embodiment of the robotic platform design of the present invention. The motion of the magnetic source is controlled by a robotic structure, which can provide dexterous attitude control of the magnetic source. In one embodiment, as shown in FIG. 5a, a serial robotic arm is designed to realize motion control by using two revolute joints (36, 37) to control the in-plane motion and the other two revolute joints to control the out-of-plane motion. Although the load capacity is limited, the serial robotic arm can provide more DOF. In one embodiment, as shown in FIG. 5b, the robotic platform is designed with a leverage structure for sufficient load capacity that can support a larger magnetic source. (40) is a passive revolute joint that supports a linear actuation mechanism (39). (41) is a linear actuator that can change the position of the revolute joint (42). The robotic structure can realize heavy load in-plane motion control by controlling the two actuators. FIG. 5c shows an embodiment of the robotic platform structure of the present invention. (43) is a rail that is placed around the patient bed, and this design allows stable translational motion of the robotic structure. The robotic structure is composed of a rotary actuator (47) and a linear actuator (48). (46) is one embodiment of the electromagnetic source of the present invention. (45) is an integrated eye-in-hand imaging tool. Figure 5d shows an embodiment of the design of the present invention that uses a moving magnetic source to achieve multiple magnetic field control points. The number of sets is determined by the magnetic field generation requirements and space limitations.

[0072] Figure 6 shows an embodiment of an eye-in-hand tracking system design. In the embodiment shown in Figure 6a, the ultrasound probe is held by a micro-Stewart platform that can control 6DOF of the US imaging plane. In one embodiment shown in Figure 6b, one rotational DOF is controlled by a rotational actuator. This has a simpler design and can provide 2D in-plane information of the controlled target. The eye-in-hand imaging system follows the movement of the controlled element and provides effective and efficient localization.

[0073] FIG. 7 shows an embodiment of a magnetic isolation shield design (50) and a magnetic field monitoring system design (51). The shield is made of multiple layers including high and low permeability materials to isolate high and low frequency magnetic fields. Its shape is designed to close the magnetic field flux and limit the influence of the magnetic field inside the working space (52). The monitoring module is made of multiple sensors including but not limited to magnetic, temperature, and humidity sensors. Based on the additional information of the electromagnetic source provided by the monitoring module, the host computer can estimate the desired operating magnetic field with higher accuracy.

[0074] The present invention will be better understood by reference to the details of the examples which follow, but those skilled in the art will readily recognize that the specific structures shown are for illustrative purposes only and are not intended to limit the invention to the precise form described herein, the invention being defined by the claims which follow.

[0075] Throughout this application, various references or publications are cited. The disclosures of these references or publications in their entireties are incorporated by reference into this application in order to more fully describe the state of the art to which this invention pertains. It should be noted that the transitional term "comprising," which is synonymous with "including," "containing," or "characterized by," is inclusive or open-ended and does not exclude additional, unrecited elements or methods or steps. EXAMPLES

[0076] Figure 8 shows an exemplary design according to the invention, including a four-part, mobile magnetic source with 3DOF motion control (56, 57), an external fluoroscope (55) supported by a C-arm that can adjust its position and rotation angle, an eye-in-hand ultrasound tracking probe (59) that provides precise localization of the controlled object, and a central processing system (53) that can coordinate the entire system to achieve in vivo magnetic actuation and tracking of the controlled object. Figure 8a shows the arrangement of the subsystems such that the magnetic actuation and imaging tools share the same human-scale workspace. The subsystems are placed on rails (58, 54) that allow stable over-ground movement around the patient bed. Figure 8b shows details of the magnetic source design according to the invention, consisting of four coils and an eye-in-hand ultrasound imaging tool (59).

Claims

1. 1. A device for tracking a magnetic element, comprising: a. a magnetic actuation unit, i. an eye-in-hand sensing module; ii. a plurality of magnetic sources evenly spaced about the eye-in-hand sensing module, each of the plurality of magnetic sources being tilted at an oblique angle and separated from an adjacent magnetic source by an adjacent angle; and iii. providing an adjustment mechanism connected to each of the plurality of magnetic sources for adjusting the tilt angle and the adjacent angle to achieve a desired magnetic field; A magnetic actuation unit; b. a robotic platform for mounting said magnetic actuation unit and for providing dexterous attitude control of said magnetic actuation unit; A device comprising:

2. 10. The device of claim 1, wherein the eye-in-hand sensing module comprises one or more components selected from the group consisting of a camera, a 2D / 3D ultrasound probe, an X-ray generator, an electrostatic sensor, a magnetic sensor, a fluorescence sensor, an optical transmittance meter, a force sensor, a grating sensor, an optoacoustic probe, a laser speckle imager, an infrared thermography camera, a radio frequency probe, and a humidity sensor.

3. The device of claim 2 , wherein the ultrasound probe is controlled by a positioning device selected from the group consisting of a Stewart platform and a rotational joint.

4. The adjustment mechanism is a. two linear actuators (17) and two universal ball joints (19); b. A four-bar link formed by four revolute joints (26, 27); c. A linear actuator (28), a universal ball joint (29), and a revolute joint (30); d. A linear actuator (31) or a rotary actuator; The device of claim 1 , comprising a configuration selected from the group consisting of:

5. The device of claim 1 , wherein the plurality of magnetic sources comprises electromagnets or permanent magnets.

6. The electromagnet comprises a soft iron core, a copper wire, and a temperature control module for controlling a temperature of the soft iron core, the temperature control module comprising: a. a cooling channel having an inlet and an outlet, the cooling channel looped around or within the soft iron core; b. a cooled heat exchange fluid entering and exiting the cooling channel through the inlet and the outlet, respectively; c. Temperature sensors located at the inlet, the outlet, and embedded in or on the soft iron core; The device of claim 5 , comprising:

7. The device of claim 6 , wherein the soft iron core has a stepped design.

8. The device of claim 6 , wherein the soft iron core comprises a cylindrical, conical, stepped, or rounded tip shape.

9. 6. The device of claim 5, wherein a current control unit is used to control a current supplied to the electromagnet to achieve the desired magnetic field, the current control unit comprising a bidirectional power supply unit, an h-bridge adjustment unit, and a microcontroller unit.

10. 10. The device of claim 1, wherein the magnetic element is selected from the group consisting of a tethered magnetic robot, a magnetic catheter, a magnetic guidewire, a magnetic needle, a magnetic sleeve, a magnetic soft robot, a magnetic continuum robot, a tethered magnetic robot, a magnetic helical swimmers, a magnetic roller, a magnetic gripper, a magnetic surgical tool, a magnetic endoscope, and a magnetic capsule endoscope.

11. The device of claim 1 further comprising a magnetic isolation shield made of layers of materials having different magnetic permeabilities.

12. the robotic platform comprising an arrangement for dexterous attitude control of the magnetic actuation unit, the arrangement comprising: a. two revolute joints (36, 37) for controlling in-plane motion and two revolute joints (38) for controlling out-of-plane motion; b. a first linear actuation mechanism (39) supported by a first rotary joint (40) and a second rotary joint (42), the second rotary joint being connected to a second linear actuation mechanism (41); c. A rotary actuator (47) and a linear actuator (48); The device of claim 1 , comprising:

13. The device of claim 1 , wherein the robotic platform comprises rails (43) for stable translational movement of the device.

14. a. a magnetic element; b. one or more devices according to claim 1; c. an eye-to-hand imaging system (2); d. A computer processor; A system comprising: The computer processor executes an algorithm for coordinating the one or more devices with the eye-to-hand imaging system (2), the algorithm comprising: i. receiving attitude information of components in the system; ii. calculating a desired trajectory of the one or more devices of claim 1 and the eye-to-hand imaging system (2) prior to operation based on a command from a user to steer the magnetic element; iii. analyzing in real time a position of an object within a field of view of the eye-to-hand imaging system during said movement; and iv. adjusting the desired trajectory based on the real-time position using a predictive control strategy to detect potential collisions. a computer processor, A system comprising:

15. 15. The system of claim 14, further comprising a patient bed having an integrated sensing and tracking module.

16. The system of claim 14 , wherein the eye-to-hand imaging system is integrated with a mobile C-arm.

17. The system of claim 14 , wherein the magnetic element is selected from the group consisting of a magnetic catheter, a tethered magnetic microrobot, and a magnetic capsule endoscope.

18. 15. The system of claim 14, wherein the multiple magnetic sources comprise electromagnets and a current control unit for controlling current supplied to the electromagnets to achieve the desired magnetic field, the current control unit comprising a bidirectional power supply unit, an h-bridge regulation unit, and a microcontroller unit.

19. 15. The system of claim 14, wherein the algorithm further includes a hierarchical control method for coordinating motion and magnetic field generation of the system, the hierarchical control method including a first level for controlling a safe distance and preventing collisions, and a second level for controlling magnetic field generation and imaging field of view.

20. The system of claim 14 , further comprising a module for a user to intervene and govern control of the system beyond the algorithm.

Images