Micro-nano robot magnetic field control system and method integrating touch sense and visual perception

Abstract

The invention provides a touch and visual perception integrated micro-nano robot magnetic field control system and method, and the system comprises a camera which collects the image of a micro-nano robot in a pipeline in real time; the magnetic field generating device generates a gradient magnetic field for driving the micro-nano robot to move through a permanent magnet; the tail end of the mechanical arm is connected with the permanent magnet and adjusts the three-dimensional pose to change the magnetic field distribution; the force feedback equipment is used for providing a touch signal for the micro-nano robot to be in contact with the pipeline boundary; and the computer is configured to identify the aggregation state of the micro-nano robot according to the camera image, increase the resistance of the force feedback equipment and reduce the rotation speed of the permanent magnet when the dispersion state is detected, and reduce the resistance and recover the conventional magnetic field control after the aggregation state is recovered. According to the system, accurate control and abnormal state self-adaptive response of the micro-nano robot in a complex experiment scene are realized through touch-vision collaborative perception and dynamic magnetic field regulation and control.

Description

Technical Field

[0001] The present invention relates to a magnetic field control system and method for micro-nano robots integrating tactile and visual perception. Background Art

[0002] Micro-nano robots, due to their small size characteristics, have shown significant advantages in fields such as targeted drug delivery and precision medicine. However, limited by their simple mechanical structure, it is difficult for them to penetrate narrow channels in the body when equipped with a driving device. Therefore, there is an urgent need to develop cable-free driving technology to achieve the expected functions. Among various driving methods for micro-robots, magnetic field driving, due to its excellent biological tissue penetration, no radiation hazard, high efficiency, and fast response characteristics, can achieve safe and efficient non-contact manipulation in complex and narrow environments, and has become a research hotspot in this field.

[0003] In terms of micro-robot control strategies, existing methods are mainly divided into two categories: fully automated control and manual operation. The fully automated system relies on preset parameters and planned paths to execute tasks. However, when faced with a dynamically changing in-vivo environment, its fixed algorithm is difficult to handle unexpected situations, which may lead to task failure. Although the manual operation mode can adjust strategies in real time by professional operators, this method has extremely high requirements for the operator's skills and there is a risk of human error. In addition, the continuity and stability of manual operation are limited and it is difficult to meet the needs of long-term medical intervention. Therefore, existing control schemes have obvious defects in terms of environmental adaptability and operation reliability. Summary of the Invention

[0004] The present invention proposes a magnetic field control system and method for micro-nano robots integrating tactile and visual perception, aiming to solve the problems that fully automated control is difficult to handle unexpected situations and manual operation has deficiencies in terms of continuity and stability when studying magnetically controlled micro-nano robots in controllable laboratory environments such as teaching and scientific research.

[0005] In a first aspect, a micro-nano robot magnetic field manipulation system integrating tactile and visual perception is proposed, comprising: a camera for real-time acquisition of images of a micro-nano robot in a pipeline; a magnetic field generating device for generating a gradient magnetic field through a permanent magnet to drive the micro-nano robot to move in the pipeline; a robotic arm, the end of which is connected to the permanent magnet, for driving the permanent magnet to move to adjust its three-dimensional posture, thereby changing the distribution direction and gradient of the gradient magnetic field to remotely control the motion trajectory of the micro-nano robot in the pipeline; a force feedback device for providing a force feedback signal of contact between the micro-nano robot and the pipeline boundary; a computer, respectively communicatively connected to the camera, the force feedback device, and the robotic arm, and configured to: identify the aggregation state of the micro-nano robot based on the image captured by the camera; when it is identified that the micro-nano robot is in a dispersed state, control the increase of the resistance of the force feedback device and control the decrease of the rotation speed of the permanent magnet; when it is identified that the micro-nano robot has returned to an aggregated state, control the decrease of the resistance of the force feedback device and control the restoration of the rotation speed of the permanent magnet.

[0006] In some examples, the magnetic field generating device includes a controller, a driver, and a stepper motor. The controller generates a PWM control signal. The driver adjusts the rotation angle of the stepper motor according to the PWM control signal to drive the permanent magnet to form a gradient magnetic field distribution.

[0007] In some examples, the camera includes: a binocular vision module, including two RGB lenses, for synchronously capturing color images; and a structured light ranging module, including an infrared laser dot matrix emitter and an infrared image sensor, which calculates the depth data of the micro-nano robot by actively projecting an infrared laser dot matrix and capturing its deformation.

[0008] In some examples, the camera also includes an infrared LED array for enhancing scene brightness in low-light environments and assisting in the simultaneous acquisition of color images and depth data.

[0009] In some examples, the computer processes the color image captured by the camera through a target detection algorithm to identify the position of the micro-nano robot in the pipeline, and calculates the three-dimensional spatial coordinates of the micro-nano robot in combination with the acquired depth data.

[0010] In some examples, the computer further tracks continuous frame images through color histogram and texture feature matching algorithm to generate the motion trajectory of the micro-nano robot, and constructs a three-dimensional point cloud model of the scene in the pipeline based on multiple frames of three-dimensional coordinate data to provide posture guidance for the robotic arm.

[0011] In some examples, when the micro-nano robot is detected to be dispersed, the force feedback device increases the operating resistance and triggers a visual alarm; when the micro-nano robot returns to a gathered state, the tactile damping and the alarm are simultaneously released.

[0012] In a second aspect, a control method for the micro-nano robot magnetic field manipulation system integrating tactile and visual perception is proposed, including: S1. The operator drives the permanent magnet of the magnetic field generating device carried by the robotic arm to move along a preset trajectory through a force feedback device, and the permanent magnet generates a gradient magnetic field to guide the movement of the micro-nano robot; S2. The camera captures the movement state of the micro-nano robot in the pipeline in real time, and the computer calculates the three-dimensional pose through the fusion of target detection and depth data; S3. When the micro-nano robot contacts the pipe wall or has an abnormal shape, the force feedback device generates a reverse force in real time to give a tactile prompt to the operator; at the same time, the camera triggers a visual alarm; S4. The operator adjusts the movement parameters of the robotic arm based on the tactile feedback and visual alarm, and the computer synchronously optimizes the path planning algorithm until the micro-nano robot returns to a stable state and reaches the target position.

[0013] By combining tactile and visual feedback, in the face of a complex or dynamic environment, the system can adapt to sudden situations according to the judgment of the operator, thereby improving the success rate of the targeted delivery task.

[0014] The present invention uses teleoperation technology for magnetic field manipulation, provides a new interaction method, can significantly improve the operation experience and efficiency of the operator, and enhances the intuitiveness and accuracy of the operation.

[0015] The present invention uses a force feedback device for tactile perception, improves the efficiency and safety of the detection and navigation of the micro-robot group transportation pipeline; through camera imaging, improves the visualization of the operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of a micro-nano robot magnetic field manipulation system integrating tactile and visual perception according to an embodiment of the present invention.

[0017] Figure 2 is a schematic diagram of a magnetic field generating device and a robotic arm according to an embodiment of the present invention.

[0018] Figure 3 is a flowchart of the visual information processing of the camera according to an embodiment of the present invention.

[0019] Figure 4 is a flowchart of controlling the micro-nano robot according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Figure 1Disclosed is a magnetic field manipulation system for micro-nano robots integrating tactile and visual perception. This system is applied to controllable laboratory environments such as teaching and scientific research. The organisms used are bionic models, and the pipelines use transparent silicone tubes to simulate blood vessel structures. The system includes a computer 1, a robotic arm 2, a force feedback device 3, a magnetic field generating device 4, and a camera 5, and realizes the precise manipulation of micro-robot swarms in organisms through multi-modal perception and collaborative control.

[0021] The system operation process is as follows: The computer 1 serves as the core controller, responsible for coordinating the collaborative work of the robotic arm 2, the force feedback device 3, the magnetic field generating device 4, and the camera 5, realizing data acquisition, processing, and information interaction between devices. The robotic arm 2 realizes six-degree-of-freedom motion through the teleoperation of the force feedback device 3. Its main functions include: carrying the magnetic field generating device 4 to perform spatial positioning outside the organism; changing the magnetic field distribution direction and gradient through three-dimensional pose adjustment, thereby remotely manipulating the motion trajectory of the micro-robot swarm in the organism; realizing the targeted delivery and path guidance of the micro-robot swarm. The force feedback device 3 provides three-axis (x / y / z) force feedback functions, supporting the operator to perform: up and down flipping, left and right swaying, and lateral movement operations; real-time perception and feedback of tactile information on the pipeline boundary. The magnetic field generating device 4 is driven by a stepper motor to rotate a permanent magnet, and controls the motion trajectory of the micro-robot swarm through a gradient magnetic field. The camera 5 adopts a high-resolution imaging system, captures the motion state of the micro-robot swarm in the pipeline in real time, extracts key parameters such as position and pose through image recognition algorithms, and provides visual feedback for the closed-loop control of the system.

[0022] The robotic arm 2 adopts a six-degree-of-freedom structural design, has an effective payload capacity of 16 kg, and a working radius of 1034 mm. Its joint system is configured as follows: The base rotation range is ±175°, the shoulder movement range is +85° / -265°, the elbow can reach ±160°, and all three axes of the wrist support ±175° rotation. This robotic arm has the characteristics of high-speed motion. The maximum rotation speeds of the base and the shoulder are ±120° / s, and the other joints can reach up to 180° / s, and achieve a repeat positioning accuracy of sub-millimeter level (±0.03 mm), meeting the requirements of high-precision operations.

[0023] As the core means of tactile perception, force feedback technology simulates the physical contact feeling in the real world through an electric drive device, and provides tactile feedback to the user by using the resistance or thrust generated by the motor. This technology precisely senses the user's operation actions (including three-dimensional space positioning and stylus posture), and the motor system generates a reverse force in real time, so as to achieve an immersive interaction with virtual objects. The key technical indicators of typical force feedback devices include parameters such as working space, position resolution, maximum output force, and joint stiffness. Its compact design usually has 6 degrees of freedom of position sensing input (covering up and down flipping, left and right swinging, and lateral movement) and 3 degrees of freedom of force feedback output. The current mainstream force feedback algorithms are based on physical modeling methods, using finite element analysis and spring-mass models to calculate the interaction forces between virtual objects, and transmitting them to the user through special devices. For example, the technical parameters of a certain model of force feedback device: the working space size is 160×120×70mm, it supports hand movements such as wrist rotation, the nominal position resolution reaches 0.055mm, the maximum output force of the orthogonal arm is 3.3N, and the stiffness of each axis is 1.26N / mm for the x-axis, 2.31N / mm for the y-axis, and 1.02N / mm for the z-axis, which can provide a high-precision and high-durability interaction experience.

[0024] As Figure 2 As shown in the figure, the magnetic field generating device 4 includes a power supply 8, a driver 9, a controller 10, a motor holder 11, a stepper motor 12, a permanent magnet holder 13, and a permanent magnet 14. Its working principle is as follows: The power supply 8 provides energy support for the driver 9; the controller 10 receives the motor control pulse parameters transmitted by the host computer software, generates a PWM wave control signal through protocol conversion, and transmits it to the driver 9 through the control bus; the driver 9 adjusts the output current of the power supply 8 according to the received PWM signal, and precisely controls the rotational movement of the stepper motor 12. The stepper motor 12 is mechanically fixed through the motor holder 11, and its rotational movement is transmitted to the permanent magnet holder 13 through the transmission mechanism, driving the permanent magnet 14 to rotate synchronously, so as to form a uniform gradient magnetic field distribution in space that can drive the micro-nano robot to move in the human body.

[0025] The visual perception system realizes the recognition and positioning of micro-nano robots by integrating computer vision and machine learning technologies. Figure 3It shows the visual information processing flow of the camera. The camera 5 adopts a binocular + structured light ranging scheme, integrating an infrared laser dot matrix emitter (active depth vision) and an infrared illumination LED (supporting night vision function), and synchronously obtains the color image and depth data of the pipeline environment through an RGB lens (such as IMX378) and a dual global shutter lens (such as OV9282, with an IR filter). After the collected original image data is preliminarily preprocessed by the camera hardware (such as median filtering for noise reduction and histogram equalization for contrast enhancement), it is transmitted to the computer through a high-speed interface for depth processing. As a front-end perception device, the structured light ranging system of the camera 5 includes a high-computing-power AI acceleration unit, supports OpenCV algorithm optimization, and meets the real-time visual perception requirements of the micro-nano robot in the pipeline.

[0026] On the computer side, the YOLOv5 object detection model analyzes the image based on the training data set, identifies the micro-nano robot in the pipeline, and outputs the bounding box coordinates. The system combines the Z-axis information provided by the depth sensor with the object detection result, and calculates the three-dimensional space coordinates (x, y, z) of the micro-nano robot through the principle of triangulation. In the continuous frame processing, the system realizes the robust tracking of the micro-nano robot through the color histogram + texture feature matching algorithm and the depth information time series analysis. The multi-sensor data fusion algorithm registers the RGB image and the depth map, generates point cloud data, and constructs a three-dimensional scene model to provide accurate pose guidance for the robotic arm.

[0027] The present invention is a magnetic field control system for micro-nano robots in the human-in-the-loop. This system deeply integrates the flexibility of manual operation and the high-precision execution ability of automatic control technology. Especially for challenges such as complex environment navigation, high-difficulty, and multi-task collaboration at the micron scale, it realizes efficient, intelligent, and robust micro-cluster control under human-machine collaboration. In traditional automated control systems, although high-speed and high-frequency mechanical operations can be achieved, there are still limitations in dealing with sudden environmental changes, abnormal event interventions, and tasks with significant unstructured and dynamic characteristics. The human-in-the-loop system not only introduces human subjective initiative and historical experience but also can respond to emergencies in real time, effectively handle uncertain factors that are difficult for automatic systems to handle, and achieve flexible and efficient task planning and execution.

[0028] Under the human-in-the-loop control framework designed by the present invention, the operator can perceive the motion state of the microcluster and the environmental feedback in real time through multimodality (such as touch, vision, and even hearing). The visual information is transmitted to the computer through the camera. The operator can analyze the progress of the task in real time based on the rich data stream and the interaction interface, and identify and respond to the abnormal state during the delivery process of the microcluster in combination with their own experience. Correspondingly, the computer undertakes complicated computing tasks such as basic path planning, magnetic field scheduling of the lower computer, and optimal decision-making reasoning to ensure that the microscale actions of the microcluster can be executed strictly according to the established strategy. During the operation process, if the operator encounters unexpected obstacles (such as geometric mutations in the lumen of the tube, foreign object blockage, hydrodynamic disturbance, etc.) or complex branched environments, when the morphology of the microcluster changes, the control priority is switched through manual quick decision-making to achieve instant adjustment of the microcluster attitude, speed control, and dynamic switching of aggregation and dispersion. This dynamic human-machine division of labor mode greatly enhances the system's ability to handle complex environments and rare tasks, enabling the microcluster to have higher adaptability and safety margins in scenarios such as precise delivery, defect repair, and positioning sampling. Therefore, the integrated innovation of the magnetic control microcluster manipulation system around the human-in-the-loop concept has become the research frontier and technical difficulty in promoting the practical application of micro-nano robots.

[0029] Figure 4It shows the process of human-machine cooperation in manipulating micro-nano robots. In a typical workflow, the operator uses a high-sensitivity force feedback device to drive the robotic arm to move along a preset trajectory. The permanent magnet loaded at the end of the robotic arm directly acts on the target area to achieve multi-level magnetic field regulation from the macro scale to the micro scale. When there is a trajectory deviation or the micro-cluster interacts with the environment (such as contacting the inner wall of the lumen), the tactile sensor of the system converts the relevant mechanical changes into electrical signals and feeds them back to the operation handle in real time. The operator can accurately perceive and analyze the environmental complexity through fingers or wrists. This force-sensing-action closed loop greatly improves the perception dimension and operation accuracy of the operator during remote control, overcoming the limitation of traditional reliance solely on visual information. Meanwhile, the camera tracking module continuously monitors the state of the micro-cluster, including but not limited to the position, shape, dispersion, and aggregation of the micro-cluster. The data it collects is synchronously transmitted to the computer. After motion analysis and behavior prediction, if abnormalities are found (such as loose aggregation state, heading drift, etc.), the system can automatically trigger auxiliary adjustment operation suggestions or correct the force feedback signal, thereby assisting the operator to accurately give subsequent action instructions. For example, when the micro-cluster state tends to disperse, the camera detects and identifies the misaligned state of the micro-cluster and transmits it to the system. After the system detects a change in the shape of the micro-cluster, it increases the resistance during the teleoperation process through the tactile device and gives an error warning reminder on the camera to tell the operator, so that the operator can continuously modify the moving speed and trajectory of the micro-cluster in a timely manner to maintain the state of the micro-cluster as much as possible; when the micro-robot cluster tends to aggregate, the tactile device restores the initial damping, and the camera also cancels the error reminder correspondingly. The operator can perform targeted delivery normally until the target point.

[0030] In summary, through multi-dimensional cooperation of human-machine-environment and the complementarity of human operation and automatic control, the entire system can not only achieve high-precision delivery of micro-clusters in scenarios such as complex microchannels, pores, and bio-like tissues, but also flexibly respond to changing task requirements. The actual measurement shows that tactile feedback significantly improves the path navigation accuracy and aggregation stability, reducing the risk of misoperation caused by visual blind spots, sudden disturbances, etc. This research provides strong support and theoretical basis for the in-depth application of micro swarm robots in the fields of precise drug delivery, minimally invasive surgery, environmental detection, and intelligent manufacturing.

Claims

1. A magnetic field manipulation system for micro-nano robots integrating tactile and visual perception, characterized in that include: A camera, used to collect images of the micro-nano robot in the pipeline in real time; a magnetic field generating device, which generates a gradient magnetic field through a permanent magnet to drive the micro-nano robot to move in the pipeline; a robotic arm, the end of which is connected to the permanent magnet, for driving the permanent magnet to move to adjust its three-dimensional posture, thereby changing the distribution direction and gradient of the gradient magnetic field, so as to remotely control the motion trajectory of the micro-nano robot in the pipeline; A force feedback device, configured to provide a force feedback signal indicating contact between the micro-nano robot and the pipe boundary; The computer is communicatively connected to the camera, the force feedback device, and the robotic arm, and is configured to: identify the aggregation state of the micro-nano robot based on the image captured by the camera; when it is identified that the micro-nano robot is in a dispersed state, control the increase of the resistance of the force feedback device and control the decrease of the rotation speed of the permanent magnet; when it is identified that the micro-nano robot has restored the aggregation state, control the decrease of the resistance of the force feedback device and control the restoration of the rotation speed of the permanent magnet.

2. The system according to claim 1, wherein The working space of the force feedback device is 160×120×70mm, the position resolution is not less than 0.055mm, the axial stiffness meets the following requirements: x-axis 1.26N / mm, y-axis 2.31N / mm, z-axis 1.02N / mm, and the maximum output force is 3.3N.

3. The system according to claim 1, wherein The robotic arm adopts a six-degree-of-freedom structure design.

4. The system according to claim 1, wherein The magnetic field generating device includes a controller, a driver and a stepper motor. The controller generates a PWM control signal. The driver adjusts the rotation angle of the stepper motor according to the PWM control signal to drive the permanent magnet to form a gradient magnetic field distribution.

5. The system according to claim 1, characterized in that, The camera includes: a binocular vision module, including two RGB lenses, for synchronously capturing color images; and a structured light ranging module, including an infrared laser dot matrix emitter and an infrared image sensor, which calculates the depth data of the micro-nano robot by actively projecting the infrared laser dot matrix and capturing its deformation.

6. The system according to claim 5, wherein The camera also includes an infrared LED array for enhancing scene brightness in low-light environments and assisting in the simultaneous acquisition of color images and depth data.

7. The system according to claim 5, characterized in that The computer processes the color image captured by the camera through a target detection algorithm to identify the position of the micro-nano robot in the pipeline, and calculates the three-dimensional spatial coordinates of the micro-nano robot in combination with the acquired depth data.

8. The system according to claim 7, characterized in that, The computer further tracks continuous frame images through a color histogram and texture feature matching algorithm to generate a motion trajectory of the micro-nano robot, and constructs a three-dimensional point cloud model of the scene in the pipeline based on multiple frames of three-dimensional coordinate data to provide posture guidance for the robotic arm.

9. The system according to claim 1, wherein When the micro-nano robots are detected to be dispersed, the force feedback device increases the operating resistance and triggers a visual alarm; when the micro-nano robots return to a gathered state, the tactile damping and the alarm are simultaneously released.

10. A control method for a magnetic field control system of a micro-nano robot integrating tactile and visual perception according to any one of claims 1-9, characterized in that, include: S1. The operator uses a force feedback device to drive the permanent magnet of the magnetic field generating device on the robotic arm to move along a preset trajectory. The permanent magnet generates a gradient magnetic field to guide the movement of the micro-nano robot. S2. The camera captures the motion of the micro-nano robot in the pipeline in real time, and the computer calculates the 3D position and posture through target detection and depth data fusion. S3. When the micro-nano robot comes into contact with the tube wall or its shape becomes abnormal, the force feedback device generates a reverse force in real time to tactilely prompt the operator; meanwhile, the camera triggers a visual alarm. S4. Based on the tactile feedback and visual alarm, the operator adjusts the motion parameters of the robotic arm, and the computer synchronously optimizes the path planning algorithm until the micro-nano robot resumes a stable state and reaches the target position.

Images