A remote control system with high precision in fundus blood vessel puncture process

Abstract

The application discloses a remote control system with high precision in an eye fundus blood vessel puncture process, which comprises a 5G communication module, a master hand control device comprising a master hand execution structure, an execution robot device comprising a puncture execution structure, and a motion control module comprising a SCARA mechanism and an RCM mechanism. Action information of the master hand execution structure is extracted; the action information is put into a communication protocol of the 5G communication module prepared in advance, is converted into frame data containing position information, a serial number and an error check sum, is transmitted to the execution robot device by using a 5G CPE device, the frame data is converted into action information of the execution robot device again by the execution robot device, the action information of the execution robot device is converted into data of rotating directions, rotating speeds and rotating distances of each motor, and the motor is controlled to work; the system is precise and low in time delay, drives a syringe needle to an accurate position, maintains an accurate angle and an accurate direction of a needle tip of the syringe needle, and executes a puncture action.

Description

Technical Field

[0001] This invention belongs to the field of remote control system technology for ophthalmic surgery, and in particular to a remote control system for fundus vascular puncture and injection; specifically, it is a remote control system with high precision in the fundus vascular puncture process. Background Technology

[0002] Occlusion of blood vessels in the retina is a serious ophthalmic emergency. Currently, the common clinical approach is to inject thrombolytic drugs into the affected blood vessels for targeted treatment and patency. This procedure is a microsurgical operation requiring the surgeon to perform precise manipulations under a microscope using ophthalmic instruments. However, due to the small size of the eyeball and the delicate and fragile structure of its tissues, the injection of medication for retinal vascular occlusion demands exceptional hand-eye coordination and a keen sense of precision to avoid surgical failure.

[0003] The most scarce resource for this type of surgery is highly skilled surgeons and the precise technique for puncturing blood vessels during drug injection. This is because the diameter of blood vessels in the retina is approximately 80 μm, while the average hand tremor of an ophthalmologist can reach 156 μm. Therefore, if the surgeon's skill level is insufficient, the risk of puncture failure is extremely high. Remote control is an effective way to ensure the rational utilization of high-quality medical resources.

[0004] One of the challenges of remote transmission is ensuring the consistency of the actions between the master control device and the execution robot device, avoiding delays, and ensuring that the coordination of each structure can meet the accuracy requirements of fundus vascular puncture.

[0005] To address the above-mentioned problems, this invention provides a remote control system with high precision for the fundus vascular puncture process. Summary of the Invention

[0006] To address the aforementioned issues, this application discloses a highly accurate remote control system for retinal vascular puncture. It solves the problem of ensuring consistency in the actions of the master control device and the executing robot during remote control, avoiding delays, and guaranteeing that the coordination of various structures meets the accuracy requirements of retinal vascular puncture.

[0007] This application utilizes tunneling protocol technology to achieve information encryption, access control, and network security protection between the master control device and the execution robot device. The data transmission terminal consists of two 5G CPE devices, located at the master control device and the execution robot device respectively. These devices can send control, feedback, and video stream information from both ends to a dedicated cloud network, and can also read data sent by the other party from the dedicated cloud network and transmit it to the other device simultaneously. This method ensures fast data transmission and low latency. Furthermore, the master control device performs independent control of the motion control module in the execution robot device, reducing data transmission complexity and further achieving low latency. Additionally, for the master execution structure's host, motion information is extracted, placed into a pre-defined communication protocol, and converted into frame data containing position information, sequence number, and error checksum, which is then sent to the execution robot device. This data is then converted into the rotation direction, speed, and distance of each axis motor of the robot, controlling the movement of the corresponding mechanisms. This method effectively achieves remote control and low-latency data transmission.

[0008] The specific technical solution is as follows: a remote control system with high precision in the process of fundus vascular puncture, which includes a 5G communication module, a master control device, and an execution robot device.

[0009] The 5G communication module includes a dedicated cloud network using tunneling protocol technology and a data transmission terminal consisting of two 5G CPE devices; the two 5G CPE devices are located at the main control device end and the execution robot end, respectively.

[0010] The master control device includes a master execution structure, a master host, and a display structure.

[0011] The robotic device includes an execution host, a puncture execution structure, a motion control module, and a video acquisition module.

[0012] The host computer collects and processes the action information of the host execution structure, and extracts the action information of the host execution structure.

[0013] The motion information of the main hand actuator is put into a pre-defined communication protocol and converted into frame data containing position information, sequence number and error checksum, which is then transmitted to the actuator robot device using a 5G CPE device.

[0014] The execution robot device then converts the frame data into motion information, which includes the position, direction, and speed of the injection needle. The execution host processes this motion information and converts it into data on the rotation direction, speed, and distance of each motor. Based on this data, the host controls each motor to operate, ultimately controlling the motion control module to move the injection needle to the correct position, maintain the correct angle and needle tip orientation, and control the puncture execution structure to perform the puncture action.

[0015] The accuracy of information can be easily verified by setting frame data with location information, sequence number and error checksum. In addition, 5G network ensures low latency to the greatest extent.

[0016] Furthermore, after receiving data, the control structure of the execution robot first checks whether the sequence number and error checksum correspond. If they do not correspond, it indicates that the data frame is incorrect. The execution robot will predict the erroneous data frame based on previously received data and replace the erroneous data with the predicted data. If the data is correct, the motion information is extracted from the received data frame and processing begins. This method ensures that the execution robot receives accurate data, guarantees the accuracy of its execution, and thus ensures the precision requirements of retinal vascular puncture.

[0017] Furthermore, the motion control module includes a SCARA mechanism and an RCM mechanism. The SCARA mechanism controls the injection needle's position determination, while the RCM mechanism controls the injection needle's angle determination and tip orientation. The SCARA mechanism includes four motors, the RCM mechanism includes three motors, and the puncture execution mechanism includes one motor. All motors are connected to an 8-axis motor control structure. Each motor's control structure has a control card. The received data on rotation direction, speed, and distance are input into the control card, which then controls the corresponding motor to move accordingly. This integrated motor control structure allows for better integrated data control, ensuring effective data transmission. The method of receiving individual action commands from each control card guarantees the effectiveness of the motion and ensures the precision of such high-precision retinal surgery.

[0018] Furthermore, the motion information of displacement, velocity, and angle collected by the main hand actuator is extracted, filtered, and debouncing is performed. The data commands are then converted into joint angles and end effector poses of the robot device. Finally, inverse kinematics algorithms are used to calculate the motor control signals, such as the rotation direction, rotation speed, and rotation distance of each axis motor in the RCM or SCARA mechanism. These motor control signals are then transmitted to the motor control structure, which controls the movement of each axis motor based on these signals. This method further ensures the coordination between the main hand actuator's movements and the robot device, avoiding interference from hand tremors on the high-precision positioning and angle of fundus injections.

[0019] Furthermore, when the master hand actuator performs repetitive movements in a single direction, each individual movement process controls only a distance change of no more than 1mm in the SCARA mechanism and a 1-degree angle adjustment in the RCM mechanism. Each individual movement process is further divided into 100 small changes, controlling the changes in the SCARA and RCM mechanisms accordingly. This setup ensures precise adjustments to the injection needle position and angle. Furthermore, the time synchronization module synchronizes the clocks of the master hand control device and the execution robot, enabling the execution robot to obtain the accurate time of the control signal through timestamps. The time synchronization module further ensures visual stamp synchronization, providing the control algorithm with accurate time information to reduce the impact of delays on control.

[0020] Furthermore, robot control algorithms in delayed manipulation environments use past control values ​​to predict current control values. Assuming the current time is t, the received control value is actually at time t−d (i.e., ...). The solution involves updating the predicted value at time t using the control value at time t−d. This approach avoids the significant fluctuations that latency can cause in high-latency network environments.

[0021] Technical effect

[0022] By extracting the motion information of the main hand actuator and placing it into a pre-defined communication protocol, the motion information is converted into frame data containing position information, sequence number, and error checksum. This frame data is then converted into motion information for the execution robot device. The motion information of the execution robot device is further converted into data on the rotation direction, speed, and distance of each motor. This allows the motion control module to drive the injection needle to the precise position, maintain the accurate angle and needle tip orientation, and control the puncture actuator to perform the puncture motion. This significantly improves transmission speed and operational synchronization, thus meeting the needs of such delicate fundus puncture surgery.

[0023] By setting up eight motors—the SCARA mechanism, RCM mechanism, and puncture actuator—and connecting all eight motors to an eight-axis motor control structure for centralized management, each action can be completed more effectively, ensuring the validity and accuracy of data transmission. This improves the precision of the vascular puncture process.

[0024] The foot-operated switching mechanism enables separate control of the SCARA and RCM mechanisms, which effectively improves the step-by-step nature of motion control, making operation simple and convenient, and making changes in micro-distance and micro-angle more precise and controllable.

[0025] By executing repetitive movements in a single direction through a master actuator, and by achieving small distance or small angle changes in a single movement process, the precision requirements of remote control position changes can be greatly guaranteed.

[0026] By setting up a time synchronization module, controlling the prediction model, and adding safety gains, the synchronization of the operation between the master control device and the execution robot device is greatly ensured, the stability of the surgical operation is guaranteed, and the requirements for precise fundus vascular puncture are met. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the data transmission from the master control device to the execution robot in this invention.

[0028] Figure 2 This is a schematic diagram of the overall process structure of the robot of the present invention;

[0029] Figure 3 This is a schematic diagram of the overall structure of the motion control module of the present invention;

[0030] Figure 4 This is a schematic diagram of the SCARA mechanism structure of the present invention;

[0031] Figure 5 This is a schematic diagram of the front side view of the RCM mechanism structure of the present invention;

[0032] Figure 6 This is a schematic diagram of the upper side view of the RCM mechanism structure of the present invention;

[0033] Figure 7 This is a schematic diagram of the puncture execution structure of the present invention, which includes a motor for rotating the injection needle.

[0034] Figure 8 This is a schematic diagram of the puncture execution structure of the present invention;

[0035] Figure 9 This is a schematic diagram of the operational process of the robot device of the present invention;

[0036] Figure 10This is a flowchart illustrating the signal transmission and corresponding time between the main control device and the execution robot in this invention.

[0037] Explanation of main figure symbols

[0038] 1. Main hand control device; 11. Main hand execution structure; 12. Foot pedal switching structure; 13. Video display structure;

[0039] 2. 5G communication module; 21. Cloud-based dedicated network; 22. Data transmission terminal; 221. CPE device;

[0040] 3. Execution Robot Device; 31. Puncture Execution Structure; 311. Puncture Motor; 312. Fiber Bragg Grating; 313. Grating Demodulator; 314. Injection Needle; 315. Microfluidic Pump; 316. Flexible Tube; 32. Motion Control Module; 321. SCARA Mechanism; 3211. Motor A; 3212. Motor B; 3213. Motor C; 3214. Motor D; 3215. Connecting Plate 1; 3216. Connecting Plate 2; 3217. Vertical Seat; 3218. Lifting... 3219. Lifting structure; 322. RCM mechanism; 3221. Tilting motor; 3222. Pitch motor; 32221. Belt pulley one; 32222. Belt pulley two; 32223. Annular belt; 3223. Motor for rotating injection needle; 3224. Setting plate; 32251. Parallelogram structure; 32252. Moving block; 32253. Pitch guide rail; 32254. Moving block; 32255. Lead screw; 33. Video acquisition module;

[0041] 4. Motor control structure. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] To keep the drawings concise, only the parts relevant to this application are shown schematically in each drawing, and they do not represent the actual structure of the product. In addition, to make the drawings concise and easy to understand, in some drawings, only one of the components with the same structure or function is shown schematically, or only one of them is labeled.

[0044] In this document, unless otherwise expressly specified and limited, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; unless otherwise specified or explained, the term "multiple" refers to two or more; the terms "connected," "fixed," etc., should be interpreted broadly. For example, "connected" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0045] refer to Figure 2 A remote control system for high precision in fundus vascular puncture, comprising a master control device 1, a 5G communication module, and an execution robot device 3.

[0046] The master control device 1 includes a master control unit, a master execution structure 11, a foot pedal switching structure 12, and a video display structure 13.

[0047] The 5G communication module includes a dedicated cloud network 21 and a data transmission terminal 22. The dedicated cloud network 21 uses VPN technology to achieve information encryption, access control, and network security protection between the master control device 1 and the execution robot device 3. The data transmission terminal 22 consists of two 5G CPE devices 221, located at the master control device 1 and the execution robot device 3 respectively. These devices can send control, feedback, and video stream information from both ends to the dedicated cloud network 21, and can also read data sent by the other end from the dedicated cloud network 21 and transmit it to the other end. Dual-end communication via the 5G CPE devices 221 utilizes the high bandwidth and low latency of 5G to achieve remote surgical operations. The 5G CPE devices 221 can achieve a maximum transmission rate of 1167Mbps and have four built-in signal amplifiers, greatly enhancing wireless coverage and signal strength while ensuring secure and stable network connections. Through dual-end communication using two 5G CPE devices 221 (5G signal conversion terminal devices), the TCP transmission protocol and transmission method can be preferentially selected to transmit the surgical images of the execution robot device 3 to the master control device in high definition and in real time.

[0048] The execution robot device 3 includes an execution host, a puncture execution structure 31, a motion control module 32, and a video acquisition module 33;

[0049] refer to Figure 3-6The motion control module 32 includes two mechanisms: a SCARA mechanism 321 and an RCM mechanism 322. The SCARA mechanism 321 is controlled by four servo motors and includes a Z-axis movement mechanism and a horizontal movement structure. The first end of the horizontal movement structure is combined with the vertical movement structure, and the second end of the horizontal movement rod structure is connected to the RCM mechanism 322. The SCARA mechanism 321 drives the RCM mechanism 322, which is equipped with an injection needle 314, to change its three-dimensional position, thereby determining the puncture position of the injection needle 314. After the position of the injection needle 314 is determined, the position of the tip of the injection needle 314 will not change until the puncture action occurs. The RCM mechanism 322 has three motors: a tumbling motor, a pitching motor 3222, and a needle rotation motor 3223. The tumbling motor and the pitching motor 3222 determine the puncture angle of the needle 314. The needle tip orientation of the needle 314 is determined by setting a rotating structure at the execution end of the RCM to achieve rotation along the longitudinal axis of the needle 314. The needle 314 is set at the RCM point position.

[0050] refer to Figure 4 The SCARA mechanism 321 includes a horizontal movement structure and a vertical movement structure. The horizontal movement structure includes three servo motors: an A motor 3211 rotating along the A-axis, a B motor 3212 rotating along the B-axis, and a C motor 3213 rotating along the C-axis. The A motor 3211 and the B motor 3212 are connected by a first connecting plate 3215, and the B motor 3212 and the C motor 3213 are connected by a second connecting plate 3216. The vertical movement structure includes a vertical seat 3217 with a lifting rail 3219, a lifting structure 3218 connected to the C motor 3213, and the lifting structure 3218 moving along the lifting rail 3219. A D motor 3214 rotating along the D-axis is also provided, which drives the lifting structure 3218 to move along the lifting rail 3219. The A motor 3211 is connected to the RCM mechanism 322 mounting plate 3224. This setup ensures horizontal movement using three servo motors and their connecting plates, and vertical movement using a vertical movement structure. When the main hand operates structure 11, only simple motion transmission is needed to adjust the horizontal and vertical positions. The four AD motors 3214, through internal encoders and feedback devices, allow for precise control of their position, speed, and acceleration. A braking device is also included to provide additional safety protection.

[0051] refer to Figure 5-7The RCM mechanism 322 includes a tumbling motor rotating along the E-axis mounted on a device plate, a parallelogram structure 32251, a parallelogram structure rotating with the tumbling motor, a pitch guide rail 32253, a moving block 32254 32252 extending from the parallelogram structure 32251 and moving within the pitch guide rail 32253, the moving block 32254 32252 being hinged to the parallelogram structure 32251, and the pitch guide rail... A lead screw 32255 is installed inside 32253, and a moving block 32254 32252 is installed on the lead screw 32255. A pulley 32221 is installed on the side of the lead screw 32255 near the tumbling motor, and a pitch motor 3222 is installed below the track on the side near the tumbling motor. The rotating shaft is the F-axis. A pulley 32222 extends from the pitch motor 3222, and a ring belt 32223 connects the pulley 32221 and the pulley 32222. The rotation of motor 3222 in different directions drives the lead screw 32255 to rotate in different directions. The rotation of the lead screw 32255 in different directions drives the moving blocks 32254 and 32252 to move in different directions, thereby forming a parallelogram structure 32251 with different pitch angles. A rotating structure is fixedly installed at the end of the parallelogram structure 32251. A motor 3223 for rotating the injection needle is installed inside the rotating structure. The axis of the motor 3223 for rotating the injection needle is the G-axis. A puncture execution structure 31 is installed inside the rotating structure. The needle tip of the puncture execution structure 31 is set at the RCM point. The E-axis and G-axis pass through the RCM point, and the E-axis is parallel to the pitch guide rail 32253. Through this arrangement, the pitch motor 3222 enables the parallelogram structure 32251 to pitch around the RCM point. The rotation of the E-axis and G-axis ensures that the rotation does not change the position of the RCM point, thereby achieving accurate adjustment of the puncture angle after finding the puncture position. The RCM mechanism 322 not only allows for adjustment of the pitch angle but also the tilt angle. Most importantly, the needle rotation motor 3223 enables adjustment of the needle tip orientation of the injection needle 314, ensuring optimal puncture angle and needle tip orientation for successful insertion. Each motor's adjustment is controlled by its speed and direction, making signal transmission simple and convenient. The tilt motor is a servo motor with a maximum speed of 6000 rpm, achieving high-precision positioning with high efficiency and low noise. The pitch motor 3222 and the needle rotation motor 3223 are brushed motors, featuring a lightweight and compact design while maintaining high power, an efficiency approaching 90%, low energy consumption, and quiet operation.

[0052] A motor control structure 4 is set up to connect to the motor control signal. The motor control structure 4 controls the rotation of each axis motor according to the motor signal, thereby controlling the changes in each position and angle.

[0053] refer to Figure 8 The puncture execution structure 31 is used to set the injection needle 314 and perform the puncture action. A puncture motor 311 is installed inside the puncture execution structure 31, which rotates to drive the injection needle 314 to perform the puncture action. The master execution structure 11 controls the puncture path and sends the puncture path to the puncture execution structure 31 in real time via a 5G network, controlling the rotation of the puncture motor 311 to execute the puncture path. More specifically, the puncture path signal is converted into the rotation speed and number of revolutions of the puncture motor 311, thereby enabling the master to control the puncture path. A Bragg grating is installed on the injection needle 314 for real-time force measurement. Three fiber optic gratings 312 are closely attached to the upper part of the injection needle 314 tip at 120° intervals. The fiber optic gratings 312 contain phase gratings, which form a narrow-band transmission filter inside the fiber core. When a beam of light passes through the grating, wavelengths that meet certain conditions will be reflected, while the remaining wavelengths will continue to be transmitted through the grating. During injection, the injection needle 314 deforms, causing the optical fiber to deform as well. This leads to a change in the grating period inside the fiber, resulting in a change in the output wavelength. The wavelength shift inside the fiber is calculated based on the output wavelength, thus determining the needle's deformation and the magnitude of the force. The direction of the force is obtained through data fusion from the three optical fibers, enabling precise force feedback. The micro-force measurement module includes this force detection section and an instrument that receives the optical fiber output signal and calculates the force. A high-precision, high-resolution grating demodulator 313 processes the Bragg grating measurement data. The grating demodulator 313 is used for demodulating signals from various types of grating sensors and acquiring sensor data. It has a selectable scanning frequency between 1 and 1000 Hz, ensuring the accuracy of the micro-force data. The received real-time force is transmitted back to the main actuator 11 via a 5G network. Force feedback allows the operator to understand the force situation during the puncture process, ensuring a successful puncture.

[0054] The lifting structure 3218 of the vertical moving structure of the SCARA mechanism 321 and the microfluidic pump 315 are both fixedly mounted on the base, and the base is then fixed to a platform that meets the height requirements (the platform is not shown). This arrangement can effectively fix the structure to the bedside. Placing the SCARA mechanism 321 and the microfluidic pump 315 on the same base facilitates the integration and operation of the structure, and there is no need to set an excessively long flexible tube 316, as long as the follow-up requirements are met.

[0055] The haptic device of the master hand actuator 11 uses a high-precision pen-shaped end effector, which provides high-precision position capture and maintains accurate gravity compensation in translation and positioning space; the haptic device renders high contact force at a rate of 4kHz and combines passive and actuating parts to improve haptic transparency. The master hand actuator 11 can optionally use an Omega.6 haptic feedback device. When the foot pedal switching structure 12 switches to control the SCARA mechanism 321, it controls the pen-shaped end effector to perform forward, backward, left, right, and up-down movements. However, because the range of motion of the retinal blood vessels is small, the complete stroke in a single direction only reaches a set distance for the SCARA mechanism 321. The position is continuously adjusted by repeated single-direction movements. For example, when the pen-shaped end effector performs a complete forward-backward movement, the SCARA mechanism 321 moves forward by 1 mm; when it moves backward-forward, the SCARA mechanism 321 moves backward by 1 mm. The movement distance of the main hand actuator 11 in one direction is positively correlated with the movement distance of the SCARA mechanism 321. Through repeated micro-adjustments, the SCARA mechanism 321 moves with the injection needle 314 to the appropriate position. Furthermore, the complete stroke in a single direction only achieves an angular change within the 3221° range of the RCM mechanism; each single stroke in a single direction is further divided into no fewer than 100 small movement variations, thus meeting the precision requirements of fundus vascular puncture. The operator can very easily and conveniently control the pen-shaped end effector to perform the aforementioned actions.

[0056] refer to Figure 1To ensure the accuracy of transmitted data and meet the precision requirements of vascular puncture surgery, the main hand unit processes the operator's motion data after receiving it. First, the motion information is extracted and then placed into the corresponding position in a pre-defined communication protocol. This protocol includes the sequence number of the transmitted data and a checksum for one frame of data. The main hand execution structure 11 sends the frame data containing the position information, sequence number, and checksum to the execution robot device 3. If various situations occur during transmission that lead to errors in the data frame or transmission failure, the execution robot device 3 will detect a mismatch in the sequence number or an error in the checksum. After receiving the data, the control structure of the execution robot device 3 first checks whether the sequence number and checksum correspond. If they do not correspond, it indicates a data frame error. The execution robot device 3 will predict the erroneous data frame based on previously received data and replace the erroneous data with the predicted data. If the data is correct, the motion information of the execution robot device 3 is extracted from the received data frame and processing begins. The received motion information from the robot includes position information, direction of motion, and speed. The host of the robot execution device 3 processes this information: it is filtered by an extended Kalman filter and a low-pass filter, and then calculated using a corresponding nonlinear motion model to ultimately determine the rotation direction, speed, and distance of each axis motor of the robot, controlling the movement of the corresponding mechanisms. The force data obtained by the robot execution device 3 is also processed using the same method as the main hand execution structure 11, and the resulting data frame is transmitted back to the main hand execution structure 11. After receiving the force data, the host first fuses it with the data obtained through a multimodal sensor fusion algorithm, and then transmits it to the host hand to feed the force back to the operator.

[0057] refer to Figure 9At any given time, the master actuator 11 can only control one of the RCM and SCARA mechanisms, and the switching is performed by the foot pedal switching structure 12: pressing the left button controls the SCARA mechanism 321; pressing the right button controls the RCM mechanism. The motors in both the RCM and SCARA mechanisms 321 are connected to an 8-axis motor control structure 4. The control data on the master actuator 11 is input into a control card, which then controls the corresponding motor to move according to the input data. When the operator presses the button on the main hand actuator 11, the main hand actuator 11 begins recording motion data. While the operator is holding down the button, the main hand actuator 11 moves, recording the hand displacement, movement speed, angle, and angular velocity in real time. This data enters the main hand actuator 11 control unit, undergoes simple processing, and is then sent to the execution robot device 3 via UDP protocol. The execution robot device 3 control unit further processes this data: first, it extracts the displacement, speed, and angle data collected by the main hand actuator 11; after filtering and debouncing, it converts the data commands into the robot's joint angles and end effector pose; finally, it uses inverse kinematics algorithms to calculate the motor control signals, such as the rotation direction, rotation speed, and rotation distance of each axis motor in the RCM or SCARA mechanism 321. These motor control signals are then transmitted to the motor control structure 4, which controls the movement of each axis motor based on these signals. When the operator releases the button on the main hand actuator 11, no information will be transmitted to the motor regardless of movement, and the motor will remain fixed at the last position reached. The operator's hand displacement and speed are converted into motor rotation information, thus enabling the main hand actuator 11 to control the 8-axis motion mechanism. During actual surgery, the operator first presses the left button on the foot pedal and uses the main hand actuator 11 to control the SCARA mechanism 321. At this time, no motor in the RCM mechanism 322 will move. The operator controls the injection needle 314 to move above the injection target. Once the injection needle 314 has reached the target position, the operator can press the right button on the foot pedal and use the main hand actuator 11 to control the RCM mechanism 322. At this time, none of the motors on the SCARA mechanism 321 will move, and the position of the needle will not change regardless of the movement of the RCM mechanism 322. At this point, the operator can control the RCM mechanism 322 to change the direction and angle of needle insertion. Once all positions, directions, and angles are determined, the operator locks each axis motor of the RCM mechanism 322 through software, leaving only the motor controlling the movement of the injection needle 314, and controls the motor to move the needle up and down to complete the injection task.Of course, the specific adjustment process also needs to consider the need for the injection needle 314 to enter the surgical position. This requires repeated switching of the foot pedal switching structure 12 and multiple adjustments to the control of the RCM or SCARA mechanism 321 to ensure that the angle and position are appropriate.

[0058] To ensure effective stopping of the motors, Hall effect sensors are used as limit switches to restrict movement when the motors are not in operation. Emergency braking of each axis motor is achieved by relays, with relay action and return times both less than 0.02s, thus improving robot safety.

[0059] Delay in remote control is unavoidable. To address the delay issue, the following strategies are used to achieve precise control even with delay: the control data from the master control terminal is compressed and encoded into data frames, and then quickly transmitted to the execution robot device 3 via the UDP protocol; the force data from the execution robot device 3 is processed in the same way as the control data from the master control terminal and then quickly transmitted to the master control terminal via the TCP protocol.

[0060] The entire transmission process is based on 5G technology. To achieve low latency, high reliability, and determinism in remote communication, this patent creates an independent virtual network for remote data transmission and divides the transmitted data into different standardized service levels, dynamically allocating resources; it reduces transmission latency through short frame structures, unrestricted scheduling, and pre-scheduling technologies; and it ensures the reliability of data transmission by creating redundant links.

[0061] To avoid delays, a time synchronization module is set up. The time synchronization module is responsible for synchronizing the clocks of the master control device 1 and the execution robot device 3, so that the execution robot device 3 can obtain the accurate time of the control signal through the timestamp and keep up with real-time updates on network latency and clock offset.

[0062] refer to Figure 10 The specific algorithm for the time synchronization module is as follows: To ensure time synchronization between the main hand control device 1 and the execution robot device 3, two problems must be solved: 1) Clock offset between the main hand control device 1 and the execution robot device 3. Each computing unit has its own internal clock. Upon power-on, the computer obtains a world time through the network. However, due to technical reasons, the world time obtained by each computer will have a certain deviation, which we denote as... Assume the network latency is a fixed value over a certain period of time, denoted as . Network latency and time deviation can be calculated by exchanging synchronization timestamps between the master control device 1 and the execution robot device 3.

[0063]

[0064]

[0065] Set up a second-order filter with a sampling rate of 0.1Hz and a cutoff frequency of 0.001Hz. The specific filtering formula is as follows:

[0066]

[0067] After filtering

[0068]

[0069] Achieve timestamp synchronization. A common method to ensure timestamp synchronization is used, achieving time synchronization through the equality of timestamps to avoid delays; ensures synchronized and precise control between the main hand execution structure 11 and the execution robot device 3, and guarantees the precision requirements of the fundus blood vessels.

[0070] In high-latency network environments during implementation, although the control signal can obtain an accurate timestamp through the time synchronization module, this control signal is already a control quantity from a certain time ago, and this delay may fluctuate significantly. Therefore, a more past control quantity is needed to predict the current control quantity. Assuming the current time is t, the received control value is actually from time t−d (i.e.,...). The problem needs to be solved by updating the predicted value at time t using the control value at time t−d.

[0071] Let the state of the most recent update be...

[0072] The state prediction equation before calibration at the current time is:

[0073]

[0074] The covariance prediction equation is:

[0075]

[0076] Map the measured delay value to the current time:

[0077]

[0078] The covariance across d steps is:

[0079]

[0080] Gain coefficient:

[0081]

[0082] Covariance update:

[0083]

[0084] Status Update:

[0085]

[0086] At this time This is the estimated value of the control signal at the current moment. This method...

[0087] In high-latency network environments, excessive delay in control inputs can lead to decreased robot motion stability. Therefore, a safety gain is added, which gradually decreases as the control signal delay increases.

[0088]

[0089] in To ensure a safe delay threshold, in practice we set the threshold to 500ms.

[0090] Besides avoiding operational delays, the procedure is also crucial because the injection distance into the retinal blood vessels is short, and force feedback occurs instantaneously. If force feedback is not timely and effective, it will significantly delay force feedback during the puncture process, thus hindering the effective completion of the puncture. Specifically, force prediction algorithms for environments with indeterminate delays address the issue of force feedback extension, requiring the force feedback from the robotic needle tip to the operator, improving the operator's feel and control. Due to the remote environment, the algorithm also needs to address latency issues.

[0091] The specific processing method is as follows: sensor force preprocessing, 300Hz sampling filtering:

[0092]

[0093] Based on a 10Hz cutoff frequency, the coefficients are set as follows:

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] The frequency is reduced to 30Hz, and then every 3 data points are sent to the main device via the network at a frequency of 10Hz.

[0100] Time synchronization module: Same as before.

[0101] Force prediction algorithm for indeterminate delay environments:

[0102] The main handpiece can receive sensor force data in groups of three at a time. , , , The time is 33.3ms. Let the current time be... Because the robot and the main device have already synchronized their times, therefore and They are all operating on the same clock. This means that data from these three sensors needs to be used to predict the force data at the current moment in order to mitigate the effects of network latency fluctuations.

[0103] First, calculate the first derivative of the force.

[0104]

[0105] Find the second derivative of the force.

[0106]

[0107] set up Using second-order Taylor expansion to extrapolate the force at this moment

[0108]

[0109] This method has good accuracy when the delay is small, but a sudden increase in delay can cause the second-order terms to be severely amplified. Dynamic second-order term weights are set to avoid the inconsistency between the force of the main hand and the force of the puncture execution structure 31 caused by a sudden increase in delay.

[0110] The specific solution and corresponding formula are as follows:

[0111]

[0112] Obtain sensor force The main hand feedback force needs to be designed next.

[0113] Distinguishing between effective contact force and interference force: Setting a force threshold , when At that time, it is determined that there is no contact, and the feedback force is applied. To avoid minor noises interfering with perception.

[0114] Increase the magnification factor to enhance force perception; the magnification ratio is [value missing].

[0115]

[0116] Adding damping term to improve feel The damping coefficient is... The speed of the main hand movement.

[0117]

[0118] By amplifying force perception, the force received by the injection needle 314 of the puncture actuator 31 is increased, making it easier for the operator to perceive, thereby ensuring that the main hand actuator 11 receives more sensitive force.

[0119] The injection needle 314 is connected to the syringe on the microfluidic pump 315 via a flexible tube 316. Micro-injection is performed via the microfluidic pump 315. Once the injection needle 314 punctures and enters the blood vessel, as long as the patient's position remains unchanged, the stopped motor and other mechanisms ensure the precise and stable position of the injection needle 314. Therefore, manual injection is no longer necessary. By controlling the speed of the microfluidic pump 315, the safe injection of thrombolytic drugs and other related medications into the retinal vessels can be guaranteed. This pathway is established; simply changing the type of drug at the pump end eliminates the difficulty of re-locating the blood vessel and allows for highly precise injection of multiple drugs. The microfluidic pump 315 can achieve a minimum injection volume of 1 μL and an injection flow rate from 0.05 nL / min to 520 μL / min; this flow control meets the micro-injection requirements of the retina.

[0120] The video acquisition module 33 is a USB video acquisition module. A video acquisition card is mounted on the ophthalmic microscope, which acquires the video stream information obtained by the ophthalmic microscope in real time during surgery. After processing, the data is sent to the cloud by the control module. Data transmission via USB improves data transmission efficiency and further avoids latency issues. The camera of the video acquisition module 33 has a maximum resolution of 2160*3840, built-in dual microphones, and connects to the video acquisition card via USB, enabling high-definition image acquisition and high-speed transmission.

[0121] The video acquisition module 33 collects and transmits back in real-time video stream information regarding the position and angle of the injection needle 314. Based on this video stream information, the master hand execution structure 11 remotely controls the SCARA mechanism 321 to determine the puncture position; controls the RCM mechanism 322 to determine the puncture angle; and controls the puncture execution structure 31 to perform the puncture action. The foot pedal switching structure 12 allows the master hand to switch between control of the SCARA mechanism 321 and the RCM mechanism 322. Real-time video stream transmission ensures hand-eye coordination.

[0122] The usage process of the equipment is as follows: First, use 5G equipment and 5G network to connect the main hand control device 1 and the execution robot device 3, and then assemble the main hand control device 1 and the execution robot device 3. The patient lies on the operating table, waiting for other surgical procedures to be completed. When the retinal vascular puncture and injection step is required, the execution robot device 3 is set up next to the operating table. The remote doctor holds the pen-shaped end effector and observes the video stream information sent back by the hemostasis robot device on the video display structure 13. The pen-shaped end effector is operated according to the real-time video stream information. First, the left button of the foot switch structure 12 is pressed, which causes the pen-shaped end effector to control the SCARA mechanism 321 to move. It is determined whether the angle of the puncture execution structure 31 needs to be adjusted. When the angle needs to be adjusted, the right button of the foot switch structure 12 is pressed. The angle of the puncture execution structure 31 is adjusted, and then the left button is switched again to adjust the position. This process is repeated until the accurate position and angle of the puncture execution structure 31 are found. Then, the pen-shaped end effector controls the puncture motor 311 of the puncture execution structure 31 to work. During the process, the real-time force feedback obtained by the Bragg grating is sent to the pen-shaped end effector. When the force feedback of successful puncture is felt, it proves that the injection needle 314 has entered the retinal blood vessel. Then release the pen-shaped end actuator. Once stopped, all motors stop, and the injection needle 314 stops at the injection position where it enters the blood vessel. Then, the microfluidic pump 315 slowly injects the drug into the blood vessel in the fundus.

[0123] The equipment demonstrated high operational stability and accuracy: the latency for two-way data transmission within Shanghai was less than 15ms; the video latency was less than 180ms; and the system enabled remote control between Beijing and Xi'an, Beijing and Guangzhou, and Beijing and Haikou. Under remote control within Shanghai, the robot was able to accurately puncture three points within 150 micrometers of the simulated artificial eyeball and accurately inject liquid. The fundus bulge of the simulated artificial eyeball could be observed under a microscope, and the fundus bulge indicated successful puncture and drug injection.

[0124] The above description is merely a specific embodiment of this application. Under the guidance of the above teachings, those skilled in the art can make other improvements or modifications based on the above embodiments. Those skilled in the art should understand that the above specific description is only to better explain the purpose of this application, and the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A remote control system for highly precise fundus vascular puncture procedures, characterized in that, include The master control device includes a master execution structure, a master host, and a display structure; The execution robot device includes an execution host, a puncture execution structure, a motion control module, and a video acquisition module; the puncture execution structure and the motion control module both include motors for controlling motion. The 5G communication module includes a dedicated cloud network and a data transmission terminal consisting of two 5G CPE devices; the two 5G CPE devices are located at the main hand control device end and the execution robot end, respectively. The time synchronization module is responsible for synchronizing the clocks of the master control device and the executing robot device. This allows the executing robot device to obtain the accurate time of the control signal through the timestamp, ensuring that the timestamp of the executing robot device is the same as that of the master control device. The time synchronization module uses past control values ​​to predict the current control value. Assuming the current time is t, the received control value is actually at time td (i.e., ...). To address the issue of updating the predicted value at time t using the control value at time td, the time synchronization module adds a safety gain, which gradually decreases as the control signal is delayed. The force prediction algorithm for uncertain latency environments uses a time synchronization module to complete time synchronization, and then uses the force prediction algorithm for uncertain latency environments to predict the force data at the current moment in order to resist the impact of network latency fluctuations. The main hand unit collects and extracts the action information of the main hand execution structure and processes it to extract the action information of the main hand execution structure; The motion information of the main hand actuator is put into a pre-defined communication protocol and converted into frame data containing position information, sequence number and error checksum, which is then transmitted to the execution robot device using the 5G CPE device. The execution robot device then converts the frame data into motion information of the execution robot device, which includes the position and angle information of the injection needle, the direction of movement, and the speed of movement. The execution host processes the motion information of the execution robot device and converts it into data on the rotation direction, rotation speed, and rotation distance of each motor of the execution robot device. Based on the data, the host controls the motors of the execution robot device to work, and finally controls the motion control module to drive the injection needle to the accurate position, maintain the accurate angle and the accurate orientation of the injection needle tip, and control the puncture execution structure to perform the puncture action. The motion control module includes a SCARA mechanism and an RCM mechanism. The SCARA mechanism is controlled by four servo motors and includes a Z-axis movement mechanism and a horizontal movement structure. The first end of the horizontal movement structure is combined with the vertical movement structure, and the second end of the horizontal movement rod structure is connected to the RCM mechanism. The SCARA mechanism controls the injection needle to perform the position determination action. The RCM mechanism has three motors: a roll motor, a pitch motor, and an injection needle rotation motor. The roll motor and pitch motor determine the injection needle puncture angle, and the injection needle rotation motor determines the direction of the injection needle tip. When the main actuator performs repeated movements in a single direction, each single movement process controls the SCARA mechanism to change a distance of no more than 1 mm and controls the RCM mechanism to adjust an angle of 1 degree. The single movement process is further divided into 100 small changes, controlling the changes of the SCARA mechanism and the RCM mechanism.

2. The system according to claim 1, characterized in that, After receiving data, the control structure of the execution robot device first checks whether the sequence number and error checksum correspond. If they do not correspond, it means that the data frame is wrong. The execution robot device will predict the erroneous data frame based on the previously received data and replace the erroneous data with the predicted data. If the data is correct, the motion information is extracted from the received data frame and processing begins.

3. The system according to claim 1, characterized in that, The SCARA mechanism includes 4 motors, the RCM mechanism includes 3 motors, and the puncture actuator includes 1 motor; all motors are connected to an 8-axis motor control structure; the motor control structure is equipped with a control card for each motor, and the received data on rotation direction, rotation speed and rotation distance are input into the control card, and the control card controls the corresponding motor to rotate according to the input data.

4. The system according to claim 3, characterized in that, The motion information of displacement, velocity, and angle of the main hand actuator is extracted, filtered and debouncing, and the data commands are converted into the joint angles and end-effector poses of the actuator robot device. The motor control signals of the rotation direction, rotation speed, and rotation distance of each axis motor in the RCM or SCARA mechanism are calculated by inverse kinematics algorithm; the motor control signals are then transmitted to the motor control structure.

5. The system according to claim 3, characterized in that, The master control device includes a foot pedal switching structure. The foot pedal controls the master control device to control the RCM and SCARA mechanisms at different times. The control of the foot pedal switching structure is determined by the video signal of the position and angle of the injection needle of the master control device.

6. The system according to claim 3, characterized in that, Each motor is equipped with a Hall sensor, which is used as a limit switch, and an emergency stop relay is also provided; the relay action and return time are both less than 0.02s.

Images