Remote control system with high accuracy in fundus blood vessel puncture process

Abstract

The invention discloses a remote control system with high accuracy in the fundus blood vessel puncture process. The remote control system comprises a 5G communication module, and a main hand control device comprises a main hand execution structure; the execution robot device comprises a puncture execution structure and a motion control module, and the motion control module comprises an SCARA mechanism and an RCM mechanism. Extracting action information of the master manipulator execution structure; the action information is put into a communication protocol, which is formulated in advance, of a 5G communication module, is converted into frame data containing position information, a serial number and an error checksum, and is transmitted to the execution robot device by utilizing the 5G CPE equipment; the execution robot device converts the frame data into action information of the execution robot device, and the action information of the execution robot device is converted into data of the rotation direction, the rotation speed and the rotation distance of each motor to control the motors to work; the system accurately drives the injection needle to reach the accurate position in a low-delay mode, the accurate angle and the accurate orientation of the needle point of the injection needle are maintained, and the puncture action is executed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote control systems for ophthalmic surgery, in particular to a remote control system for fundus vascular puncture injection. BACKGROUND

[0002] Fundus vascular obstruction is a serious ophthalmic emergency. The current clinical method is to inject thrombolytic drugs at the lesion blood vessels to relieve the symptoms. This operation is a kind of ophthalmic microsurgery, which requires the operating surgeon to hold ophthalmic instruments to complete the corresponding fine operation under a microscope. However, due to the small size of the eyeball, the delicate and fragile structure of the eyeball tissue, and the requirement of high hand-eye coordination and fine operation perception of the surgeon to avoid surgical failure.

[0003] The most scarce for such surgery is the high-tech level of doctors and the vascular puncture action during drug injection. The diameter of the blood vessels on the fundus retina is about 80 μm, while the average amplitude of hand tremor of a general ophthalmologist reaches 156 μm. Therefore, if the doctor is not highly skilled, the risk of puncture failure is extremely high. Remote control is an effective way to realize the rational use of high-quality medical resources.

[0004] One of the problems of remote transmission is how to ensure the action consistency of the master control device and the execution robot device, avoid time delay, and ensure that the cooperation of each structure can meet the precision requirements of fundus vascular puncture.

[0005] The application provides a remote control system with high precision in fundus vascular puncture process. SUMMARY

[0006] The application discloses a remote control system with high precision in fundus vascular puncture process. The problem solved is the action consistency of the master control device and the execution robot device in the remote control process, avoiding time delay and ensuring that the cooperation of each structure can meet the precision requirements of fundus vascular puncture.

[0007] The application realizes information encryption, access control and network security protection between the master control device and the execution robot device through tunnel protocol technology; the data transmission terminal is composed of two 5G CPE devices, and the two devices are respectively located at the master control device end and the execution robot device end, can send the control, feedback and video stream information sent by both ends to the cloud special network, and can also read the data sent by the other party from the cloud special network and transmit it to the other party device, so that the data transmission is fast and low in delay. In addition, the master control device controls the motion control module in the execution robot device separately, so that the complexity of data transmission is reduced, and low delay can be further realized. In addition, the host of the master execution structure extracts the action information, puts the action information into the communication protocol prepared in advance, converts the frame data containing position information, sequence number and error checksum into the rotation direction, rotation speed and rotation distance of each axis motor of the robot, and sends it to the execution robot device, and then converts it into the corresponding motor. The rotation direction, rotation speed and rotation distance of the robot shaft motor control the corresponding mechanism movement of the robot; in this way, remote control and low delay data transmission are effectively realized.

[0008] The specific technical scheme is: a remote control system with high precision in fundus blood vessel puncture process, comprising a 5G communication module, a master control device and an execution robot device.

[0009] The 5G communication module comprises a cloud special network using tunnel protocol technology and a data transmission terminal composed of two 5G CPE devices; the two 5G CPE devices are respectively located at the master control device end and the execution robot device end.

[0010] The master control device comprises a master execution structure, a master host and a display structure.

[0011] The execution robot device comprises an execution host, a puncture execution structure, a motion control module and a video acquisition module.

[0012] The master host collects and processes the action information of the master execution structure, and extracts the action information of the master execution structure.

[0013] The action information of the master execution structure is put into the communication protocol prepared in advance, and is converted into frame data containing position information, sequence number and error checksum, and is transmitted to the execution robot device by using the 5G CPE device;

[0014] The execution robot device converts the frame data into execution robot device action information, which includes injection needle position information and movement direction and speed; the host processor processes the execution robot device action information to convert it into data of the rotation direction, rotation speed and rotation distance of each motor of the execution robot device, and then controls each motor according to the data of the rotation direction, rotation speed and rotation distance of each motor to finally control the movement control module to drive the injection needle to the accurate position, maintain the accurate angle and accurate orientation of the injection needle tip, and control the puncture execution structure to perform the puncture action.

[0015] The frame data of the position information, the serial number and the error check sum can be used to check the accuracy of the information, and the 5G network can ensure low latency.

[0016] Further, after receiving the data, the control structure first detects whether the serial number and the error check sum correspond, and if they do not correspond, it means that the data frame is incorrect, and the execution robot device will predict the incorrect data frame according to the previously received data and replace the incorrect data with the predicted data. If the data is correct, the action information is extracted from the received data frame and processed. In this way, the execution robot device can receive accurate data and ensure the accuracy of the execution robot device, thereby ensuring the accuracy of the fundus blood vessel puncture.

[0017] Further, the movement control module includes a SCARA mechanism and an RCM mechanism, wherein the SCARA mechanism controls the injection needle to perform position determination action, and the RCM mechanism controls the injection needle to perform angle determination and the injection needle tip orientation; the SCARA mechanism includes four motors, the RCM mechanism includes three motors, and the puncture execution mechanism includes one motor; all the motors are connected to an 8-axis motor control structure; the motor control structure is provided with a control card for each motor, and the received rotation direction, rotation speed and rotation distance data are input into the control card, and the control card controls the corresponding motor to move according to the input data. The integrated motor control structure can better realize integrated control of data, ensure the effectiveness of data transmission, and the way that each control card receives each action instruction can ensure the effectiveness of movement and the accuracy of such high-precision fundus surgery operation.

[0018] Furthermore, the displacement, velocity, and angle motion information collected by the master-hand actuator is extracted and, after filtering and de-jittering, converted into the joint angles and end-point pose of the actuator robot. Finally, an inverse kinematics algorithm is used to calculate the motor control signals, such as the rotation direction, speed, and distance of each axis motor in the RCM or SCARA mechanism. These motor control signals are then transmitted to the motor control structure, which controls the movement of each axis motor based on these signals. This method further ensures the coordination between the master-hand actuator and the actuator robot, preventing hand shaking from interfering with the high-precision fundus injection position and angle.

[0019] Furthermore, when the master-hand actuator performs repetitive single-directional motion, each individual movement only controls the SCARA mechanism to change distance by no more than 1mm and the RCM mechanism to adjust angle by 1 degree. This single movement is then broken down into 100 smaller changes, controlling the SCARA and RCM mechanisms. This setup ensures fine-tuning of the needle's position and angle. Furthermore, the time synchronization module synchronizes the clocks of the master-hand control device and the actuator robot, enabling the actuator robot to obtain the exact time of control signals through timestamps. The time synchronization module further ensures visual stamp synchronization, providing the control algorithm with an accurate time basis to mitigate the impact of control delays.

[0020] Furthermore, the robot control algorithm in the delayed manipulation environment uses the past control value to predict the current control value. Assuming that the current time is t, the control value received is actually at time t−d (i.e. ), we need to solve the problem of using the control value at time t-d to update the predicted value at time t. This approach can avoid the large fluctuations that may be caused by delays in high-latency network environments.

[0021] Technical Effects

[0022] By extracting the motion information of the main hand execution structure and placing the motion information into a pre-established communication protocol, it is converted into frame data containing position information, serial number and error checksum, and the frame data is converted into motion information of the execution robot device; after converting the motion information of the execution robot device into data on the rotation direction, rotation speed and rotation distance of each motor, the motion control module is controlled to drive the injection needle to the exact position, maintain the accurate angle and the accurate direction of the injection needle tip, and control the way the puncture execution structure performs the puncture movement, which can greatly improve the transmission speed and the synchronization of the operation, thereby meeting the needs of such delicate fundus puncture surgery.

[0023] Through the setting of the SCARA mechanism, the RCM mechanism and the 8 motor of the puncture execution mechanism, and connecting the 8 motor to the 8 shaft motor control structure for centralized management, each action can be better completed, the effectiveness and accuracy of data transmission are ensured, and the fineness of the blood vessel puncture process is improved through the mode.

[0024] Through the foot pedal switching structure, the SCARA mechanism and the RCM mechanism can be controlled separately, so that the step-by-step nature of action control is effectively improved, the operation becomes simple and convenient, and the micro-distance and micro-angle change becomes more accurate and controllable.

[0025] Through the main hand execution structure, a single direction repeated motion is executed, and a small distance or a small angle change is realized in a single motion process, so that the fine requirement of the position change of remote control is greatly ensured.

[0026] Through the setting of the time synchronization module, the control prediction model and the addition of the safety gain greatly ensure the synchronization of the main hand control device and the execution robot device operation, the stability of the surgical operation, and meet the fine requirement of the fundus blood vessel puncture. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The running flowchart of the data transmission from the main hand control device end of the application to the execution robot end;

[0028] Figure 2 The overall flow structure diagram of the robot of the application;

[0029] Figure 3 The overall structure diagram of the motion control module of the application;

[0030] Figure 4 The structure diagram of the SCARA mechanism of the application;

[0031] Figure 5 The structure diagram of the front side of the RCM mechanism structure of the application;

[0032] Figure 6 The structure diagram of the upper side of the RCM mechanism structure of the application;

[0033] Figure 7 The structure diagram of the puncture execution structure with the motor for rotating the injection needle of the application;

[0034] Figure 8 The structure diagram of the puncture execution structure of the application;

[0035] Figure 9 The running flow structure diagram of the execution robot device of the application;

[0036] Figure 10Signal transmission and corresponding time flow chart between master hand control device end and execution robot end of the application;

[0037] Main figure mark explanation

[0038] 1, master hand control device; 11, master hand execution structure; 12, foot pedal switching structure; 13, video display structure;

[0039] 2, 5G communication module; 21, cloud special network; 22, data transmission terminal; 221, CPE device;

[0040] 3, execution robot device; 31, puncture execution structure; 311, puncture motor; 312, fiber grating; 313, grating demodulator; 314, injection needle; 315, micro flow pump; 316, soft tube; 32, motion control module; 321, SCARA mechanism; 3211, A motor; 3212, B motor; 3213, C motor; 3214, D motor; 3215, connecting plate one; 3216, connecting plate two; 3217, vertical seat; 3218, lifting structure; 3219, lifting track; 322, RCM mechanism; 3221, overturning motor; 3222, pitching motor; 32221, pulley one; 32222, pulley two; 32223, annular belt; 3223, injection needle rotation motor; 3224, setting plate; 32251, parallelogram structure; 32252, moving block; 32253, pitching guide track; 32254, moving block; 32255, lead screw; 33, video acquisition module;

[0041] 4, motor control structure. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0043] In order to make the figure simple, only the parts related to the present application are shown in the figures, which do not represent the actual structure of the product. In addition, in order to make the figure simple and easy to understand, in some figures, only one of the parts with the same structure or function is shown, or only one of them is marked.

[0044] In this article, unless otherwise expressly specified and limited, the terms "first", "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance; unless otherwise specified or explained, the term "multiple" refers to two or more; the terms "connection", "fixation" and the like should be broadly understood, for example, "connection" can be fixed connection, or detachable connection, or integrally connected, or electrically connected; it can be directly connected, or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0045] Reference Figure 2 A remote control system with high precision in fundus blood vessel puncture process, comprising a master control device 1, a 5G communication module and an execution robot device 3.

[0046] The master control device 1 comprises a control master host, a master hand execution structure 11, a foot pedal switching structure 12 and a video display structure 13.

[0047] The 5G communication module comprises a cloud special network 21 and a data transmission terminal 22; the cloud special network 21 uses VPN technology to realize information encryption, access control and network security protection between the master control device 1 and the execution robot device 3; the data transmission terminal 22 is composed of two 5G CPE devices 221, which are respectively located at the master control device 1 end and the execution robot device 3 end, and can send the control, feedback and video stream information sent by both ends to the cloud special network 21, and also can read the data sent by the other party from the cloud special network 21 and transmit it to the other party device. Through the 5G CPE device 221 for double-end communication, the high bandwidth and low delay characteristics of 5G are used to realize remote operation. The 5G CPE device 221 can reach a transmission rate of 1167Mbps, and has four built-in signal amplifiers, which greatly enhances the wireless coverage range and signal strength, while ensuring the safety and stability of network connection. Through 2 5G CPE devices 221 (5G signal conversion terminal equipment) for double-end communication, the execution robot device 3 operation picture can be preferentially transmitted to the master control end in high definition and real time by selecting TCP transmission protocol and transmission mode.

[0048] The execution robot device 3 comprises an execution host, a puncture execution structure 31, a motion control module 32 and a video acquisition module 33;

[0049] Reference Figures 3-6; The motion control module 32 includes two mechanisms of SCARA mechanism 321 and RCM mechanism 322, the SCARA mechanism 321 is controlled by using four servo motors, which includes a Z-axis moving mechanism and a horizontal moving structure, the first end of the horizontal moving structure is combined with a vertical moving structure, the second end of the horizontal moving rod structure is connected with the RCM mechanism 322, the RCM mechanism 322 provided with the injection needle 314 is driven by the SCARA mechanism 321 to change the three-dimensional position, the determination of the puncture position of the injection needle 314 is performed, and the position of the injection needle 314 does not change after the determination of the position of the tip of the injection needle 314 and before the puncture action occurs. The RCM mechanism 322 has three motors, which are respectively a rolling motor, a pitching motor 3222 and an injection needle rotating motor 3223; the rolling motor and the pitching motor 3222 perform the determination of the puncture angle of the injection needle 314, and the determination of the direction of the tip of the injection needle 314 is completed by the rotating structure provided at the execution end of the RCM to rotate along the longitudinal axis of the injection needle 314; the injection needle 314 is arranged at the RCM point position.

[0050] Reference Figure 4 ; The SCARA mechanism 321 includes a horizontal moving structure and a vertical moving structure, the horizontal moving structure includes three servo motors, an A motor 3211 rotating along an A axis, a B motor 3212 rotating along a B axis and a C motor 3213 rotating along a C axis, the A motor 3211 and the B motor 3212 are connected by a connecting plate one 3215, and the B motor 3212 and the C motor 3213 are connected by a connecting plate two 3216; the vertical moving structure includes a vertical seat 3217 with a lifting track 3219 and a lifting structure 3218 connected with the C motor 3213, the lifting structure 3218 moves along the lifting track 3219, and a D motor 3214 rotating along a D axis is further arranged, the D motor 3214 drives the lifting structure 3218 to move along the lifting track 3219; the A motor 3211 is connected with a setting plate 3224 of the RCM mechanism 322. Through the above arrangement, the horizontal movement can be realized by using the three servo motors and the connecting plates therebetween, and the vertical movement can be realized by cooperating with the vertical moving structure. When the master hand execution structure 11 is operated, only simple action transmission is needed to adjust the horizontal and vertical positions. The four servo motors of the A-D motor 3214 can accurately control the position, speed and acceleration of the motor through the internal encoder and feedback device, and a brake device is further configured to provide additional safety protection.

[0051] Reference Figures 5-7RCM mechanism 322 includes a rolling motor arranged on the device plate to rotate along the E axis, a parallelogram structure 32251 arranged on the rolling motor to rotate with the rolling motor, a pitch guide rail 32253, the parallelogram structure 32251 extending a moving block 3225432252 moving in the pitch guide rail 32253, the moving block 3225432252 being hinged to the parallelogram structure 32251, a lead screw 32255 arranged in the pitch guide rail 32253, the moving block 3225432252 being arranged on the lead screw 32255, a belt pulley one 32221 being arranged on the side of the lead screw 32255 close to the rolling motor, a pitch motor 3222 being arranged below the rail on the side close to the rolling motor, the rotation axis being the F axis, a belt pulley two 32222 extending from the pitch motor 3222, the belt pulley one 32221 and the belt pulley two 32222 being connected by a ring belt 32223, the pitch motor 3222 rotating in different directions to drive the lead screw 32255 to rotate in different directions, the lead screw 32255 rotating in different directions to drive the moving block 3225432252 to move in different directions, thereby forming the parallelogram structure 32251 with different pitch angles, the end of the parallelogram structure 32251 being fixedly arranged with a rotating structure, a needle rotating motor 3223 being arranged in the rotating structure, the axis of the needle rotating motor 3223 being the G axis, a puncture execution structure 31 being arranged in the rotating structure, the needle tip of the puncture execution structure 31 being arranged on the RCM point, the E axis and the G axis passing through the RCM point, and the E axis being parallel to the pitch guide rail 32253, the pitch motor 3222 being arranged to pitch the parallelogram structure 32251 around the RCM point, the rotation of the E axis and the G axis not changing the position of the RCM point, thereby accurately adjusting the puncture angle after finding the puncture position. The arrangement of the RCM mechanism 322 can not only adjust the pitch angle, but also adjust the turning angle, and most importantly, the needle rotating motor 3223 can adjust the direction of the needle tip of the injection needle 314, so that the injection needle 314 can puncture at the optimal puncture angle, ensure that the injection needle 314 punctures at the optimal puncture angle and the direction of the needle tip, maximize the success rate of puncture, and the adjustment of each motor is controlled by the speed and direction of the motor, the whole signal transmission is simple and convenient. The rolling motor is a servo motor with a maximum speed of 6000 rpm, which can realize high-precision positioning and has the characteristics of high efficiency and low noise; the pitch motor 3222 and the needle rotating motor 3223 are brush motors, which are designed to be light and compact, have high power, a work efficiency close to 90%, low energy consumption, and are quiet.

[0052] The motor control structure 4 connected to the motor control signal is arranged to control the rotation of the motor of each axis, thereby controlling the change of position and angle.

[0053] Reference Figure 8 The puncture execution structure 31 is arranged to set the injection needle 314 and perform the puncture action. The puncture motor 311 is arranged in the puncture execution structure 31 and rotates to drive the injection needle 314 to perform the puncture action. The master hand execution structure 11 controls the puncture path, which is sent to the puncture execution structure 31 in real time through the 5G network, and controls the rotation of the puncture motor 311 to perform the puncture path. More specifically, the puncture path signal is converted into the rotation speed and rotation number information of the puncture motor 311, and the master hand controls the puncture path through this conversion mode. The Bragg grating is arranged on the injection needle 314 to measure the force in real time. The three optical fiber gratings 312 are tightly arranged above the head of the injection needle 314 at an interval of 120° therebetween. The optical fiber grating 312 has a phase grating inside, which forms a narrow-band transmission filter in the fiber core. When a beam of light passes through the grating, the partial wavelength that meets the condition will be reflected, and the remaining wavelength will continue to transmit through the grating. When the injection work is performed, the injection needle 314 deforms, driving the optical fiber to deform, thereby causing the change of the grating period inside the optical fiber, driving the change of the output wavelength. The wavelength shift inside the optical fiber is calculated according to the output wavelength, so as to obtain the deformation amount of the needle head, and then the force is calculated. The direction of the force is obtained through the data fusion of the three optical fibers, so as to realize the accurate force feedback. The micro-force measurement module refers to the instrument including the force detection part and receiving the output signal of the optical fiber and calculating the size of the force. The grating demodulator 313 with high precision and high resolution is used to process the data measured by the Bragg grating. The grating demodulator 313 is used for demodulation and sensing data acquisition of various types of grating sensors, and the optional scanning frequency is between 1-1000 Hz, which can ensure the accuracy of the micro-force data. The real-time force received is sent back to the master hand execution structure 11 through the 5G network, and the operator can understand the force condition in the puncture process through the force feedback, so as to ensure the success of the puncture.

[0054] The lifting structure 3218 of the vertical movement structure of the SCARA mechanism 321 and the micro-flow pump 315 are fixedly arranged on the base, and the base is fixed to a high platform that meets the height requirement. The high platform is not shown. Through this arrangement, the structure can be effectively fixed beside the bed. The SCARA mechanism 321 and the micro-flow pump 315 are arranged on the same base, which can facilitate the integration and operation of the structure, and there is no need to arrange a soft tube 316 that is too long. Only the follow-up requirement needs to be met.

[0055] The haptic device of master execution structure 11 uses a high-precision pen-shaped end effector, which provides high-precision position capture and can maintain precise gravity compensation in the translational and positioning space; the haptic device renders high contact force at a rate of 4KHz and combines passive components and actuated components together to improve haptic transparency. The master execution structure 11 can select the Omega.6 touch force feedback device. When the foot pedal switching structure 12 is switched to control the SCARA mechanism 321 to operate, the control of the pen-shaped end effector to perform forward and backward movement, left and right movement and up and down movement, the SCARA mechanism 321 occurs forward, left, right and up and down movement, but because the activity range of the fundus blood vessels is small, the control of the single direction complete stroke is only the movement of a set distance of the SCARA mechanism 321, and the position is adjusted by repeated single direction movement, such as the pen-shaped end effector occurs a complete self forward and backward movement, the SCARA mechanism 321 occurs a forward movement of 1mm; a complete self backward and forward movement controls the SCARA mechanism 321 to occur a backward movement of 1mm, the movement distance of a single direction of the master execution structure 11 is positively correlated with the movement distance of the SCARA mechanism 321; through repeated multiple micro-adjustments, the SCARA mechanism 321 moves to the appropriate position with the injection needle 314. In addition, the single direction complete stroke is only the angle change of 321° of the RCM mechanism 322; each single direction single stroke is divided into no less than 100 small movement changes, which meets the fine requirements of fundus blood vessel puncture through the above-mentioned way. The operator can very simply and conveniently control the pen-shaped end effector to occur the above-mentioned actions.

[0056] Reference Figure 1In order to ensure the correctness of the transmission data to meet the requirements of vascular puncture operation refinement. After obtaining the action data of the operator, the master host of the master hand processes the action data: first, the action information is extracted, and then the action information is placed in the corresponding position of the communication protocol prepared in advance. In this protocol, there is a sequence number of transmission data and a frame data error checksum. The master hand execution structure 11 sends the frame data containing position information, sequence number and error checksum to the execution robot device 3. If various situations occurring during transmission cause the data frame to be incorrect or transmission to fail, the execution robot device 3 will detect that the sequence number does not correspond or the error checksum is incorrect. After receiving the data, the control structure of the execution robot device 3 first detects whether the sequence number and the error checksum correspond. If they do not correspond, it means that the data frame is incorrect. The execution robot device 3 will predict the incorrect data frame according to the data received before and replace the incorrect data with the predicted data. If the data is correct, the action information of the execution robot device 3 will be extracted from the received data frame, and processing will begin. The action information of the execution robot received includes position information and motion direction, motion speed. The host of the execution robot device 3 will process these information: these information will be filtered through the extended Kalman filter and the low-pass filter, and calculated through the corresponding nonlinear motion model, and finally become the rotation direction, rotation speed and rotation distance of each axis motor of the robot, control the corresponding mechanism of the robot to move. The force data obtained by the execution robot device 3 is also processed in the same way as the master hand execution structure 11. The final generated data frame is transmitted back to the master hand execution structure 11. After receiving the force data, the master host of the master hand first fuses the data obtained through the multi-modal sensor fusion algorithm, and then transmits it to the master hand, and feeds back the force to the operator.

[0057] Reference Figure 9;At the same time, the master hand execution structure 11 can only control one mechanism in the RCM and SCARA, and is switched by the foot switch structure 12: the left button is pressed down, the master hand execution structure 11 controls the SCARA mechanism 321 to work; the right button is pressed down, the master hand execution structure 11 controls the RCM structure to work. The motors in the RCM and SCARA mechanism 321 are connected to an 8-axis motor control structure 4, and the control data input control card on the master hand execution structure 11, the control card controls the corresponding motor to move according to the input data. When the operator presses the master hand execution structure 11 button, the master hand execution structure 11 starts to record the motion data; the master hand execution structure 11 moves while the operator presses the master hand execution structure 11 button, and the master hand execution structure 11 records the real-time hand displacement, motion speed and master hand execution structure 11 angle and angular velocity, which are input into the master host of the master hand execution structure 11, and after simple processing, are sent to the execution robot device 3 end through the UDP protocol, and the execution robot device 3 end host will further process these data: first, the displacement, speed, angle and other data collected by the master hand execution structure 11 are extracted, and after filtering and de-bouncing, the data instructions are converted into the joint angle and end pose of the robot, and finally the motor control signals such as the rotation direction, rotation speed and rotation distance of each axis motor in the RCM or SCARA mechanism 321 are calculated through the inverse kinematics algorithm. The motor control signal will be transmitted to the motor control structure 4, and the motor control structure 4 controls the movement of each axis motor according to the signal. When the operator releases the master hand execution structure 11 button, no matter how to move, there will be no information transmitted to the motor, and the motor will be fixed at the position reached by the last movement. The displacement and speed of the operator's hand are converted into motor rotation information, so as to realize the control of the master hand execution structure 11 on the 8-axis motion mechanism. In the actual operation process, the operator first steps on the left button of the foot pedal, and controls the SCARA mechanism 321 using the master hand execution structure 11, at this time any motor in the RCM mechanism 322 will not move. The operator controls the needle 314 head to move above the injection target, when the needle 314 head has reached the target position, the operator can step on the right button of the foot pedal, and control the RCM mechanism 322 using the master hand execution structure 11, at this time the axis motor of the SCARA mechanism 321 will not move, and no matter how the RCM mechanism 322 moves, the position of the needle will not change. At this time the operator can control the RCM mechanism 322 to change the direction and angle of the needle penetration, when all positions, directions and angles are determined, the operator locks the axis motor of the RCM mechanism 322 through the software, only leaving the motor controlling the movement of the needle 314 head, and controls the motor to move the needle up and down, completing the injection task.Of course, the specific adjustment process also needs to consider the need for the injection needle 314 to enter the surgical position, so multiple repeated switching of the foot pedal switching structure 12 and multiple adjustments of the control of the RCM or SCARA mechanism 321 are required to ensure that the angles and positions are appropriate.

[0058] In order to ensure the effective stop of the motor, a Hall sensor is used as a limit switch to limit the movement of the motor when it is not working. The brake emergency stop of each axis motor is realized by a relay, and the relay action and return time are both less than 0.02s, which improves the safety of the robot.

[0059] The delay in remote control cannot be avoided, and the following strategies are used to achieve precise control in the presence of delay: the control data of the master control end is compressed and encoded to form a data frame, and then transmitted quickly to the execution robot device 3 through the UDP protocol; the force data of the execution robot device 3 is processed in the same way as the control data of the master control end, and then transmitted quickly to the master control end through the TCP protocol.

[0060] The entire transmission process is based on 5G technology. In order to realize low delay, high reliability and determinacy of remote communication, this patent creates an independent virtual network for remote data transmission, and divides the transmission data into different standardized service levels and dynamically allocates resources; reduces transmission delay through short frame structure, exemption from right scheduling, pre-scheduling and other technologies; ensures the reliability of data transmission by creating redundant links.

[0061] In order to avoid delay, a time synchronization module is set, which is responsible for synchronizing the master control device 1 and the execution robot device 3 clock, so that the execution robot device 3 can obtain the accurate time of the control signal through the timestamp, and keep real-time updating of network delay and clock offset.

[0062] Reference Figure 10 The specific algorithm of the time synchronization module is that in order to synchronize the master control device 1 and the execution robot device 3, two problems must be solved. 1) The clock offset of the master control device 1 and the execution robot device 3. Each computing unit has its own internal clock. When starting, the computer will get a world time through the network, but due to technical reasons, each computer will have a certain deviation in getting the world time, which we call . Assuming that the network delay is a fixed value within a certain time, denoted as . By exchanging synchronization timestamps between the master control device 1 and the execution robot device 3, the network delay and time deviation can be calculated.

[0063]

[0064]

[0065] Set a 2nd order filter, sampling rate 0.1Hz, cutoff frequency: 0.001Hz, the specific filtering formula is as follows:

[0066]

[0067] After filtering

[0068]

[0069] Achieve timestamp synchronization. A public method for ensuring timestamp synchronization achieves time synchronization through the equality of timestamps to avoid delays; ensure the master hand execution structure 11 and the execution robot device 3 are synchronously and accurately controlled to ensure the accuracy requirements of the fundus blood vessels.

[0070] In a high-latency network environment, the control signal can obtain an accurate timestamp thanks to the time synchronization module, but the control signal is already a control value from a certain time ago, and this delay may fluctuate greatly. Therefore, it is necessary to use control values ​​from the past to predict the current control value. Assuming that the current time is t, the control value received is actually at time t-d (i.e. ), it is necessary to solve the problem of updating the predicted value at time t with the control value at time t−d.

[0071] Assume the last updated status is

[0072] Then the state prediction equation before calibration at the current moment is:

[0073]

[0074] The covariance prediction equation is:

[0075]

[0076] Map the measured value of the delay to the current moment:

[0077]

[0078] The covariance across d steps is:

[0079]

[0080] Gain factor:

[0081]

[0082] Covariance update:

[0083]

[0084] Status Update:

[0085]

[0086] The current time That is the estimated value of the control signal at the current time. In this way

[0087] In a high-delay network environment, if the control quantity delay is too large, it may cause the robot motion stability to decline, so a safety gain is added, which will gradually decay with the delay of the control signal,

[0088]

[0089] Where The safety delay threshold is 500ms in practice.

[0090] The operation process avoids delay, because the distance of fundus vascular injection is small, and force feedback also occurs in an instant. If force feedback cannot be timely and effective, it will greatly delay the force feedback in the puncture process, and thus cannot effectively complete the puncture action. The force prediction algorithm for the uncertain delay environment solves the extension problem of force feedback, which needs to give the force feedback of the robot needle tip to the main operator, improve the operator's sense of touch and control ability. Since it is a remote environment, the algorithm also needs to deal with the problem of delay.

[0091] The specific processing method is: sensor force preprocessing, 300Hz sampling filtering:

[0092]

[0093] According to the 10Hz cutoff frequency, the coefficient is set to:

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] Down to 30Hz, then every 3 data as a group, according to the 10Hz frequency network to the master hand

[0100] Time synchronization module: same as before

[0101] Force prediction algorithm in uncertain delay environment:

[0102] The master hand can receive sensor force data in groups of three data each time is 33.3ms, let the current time be , because the robot end and master end have completed time synchronization, so and are the same clock. That is, the force data at the current time is predicted using the three sensor data to resist the impact of network delay fluctuations.

[0103] First, the first derivative of the force is found

[0104]

[0105] Then the second derivative of the force is found

[0106]

[0107] Let , the force at this moment is extrapolated using the second-order Taylor expansion

[0108]

[0109] When the delay is small, this method has good accuracy, and when the delay suddenly increases, the second-order term will be greatly amplified. Setting a dynamic second-order term weight can avoid the problem of inconsistency between the master force and the force of the puncture execution structure 31 when the delay suddenly increases.

[0110] The specific solution and the corresponding formula are:

[0111]

[0112] Obtain sensor force After that, the master feedback force

[0113] Distinguish between effective contact force and disturbance force: set a force threshold , when , it is determined that there is no contact, and the feedback force , to avoid small noise interference perception.

[0114] Increase the amplification term to expand the force perception, and the amplification ratio is

[0115]

[0116] Increase the damping term to improve the hand feeling, is the damping coefficient, is the master movement speed.

[0117] ​​​

[0118] By expanding the force perception to ensure that the force received by the puncture execution structure 31 injection needle 314 becomes larger, so that the operator is more easily perceived, and then ensure that the master hand execution structure 11 receives more sensitive force.

[0119] The injection needle 314 is connected to the syringe on the micro flow pump 315 through the soft tube 316, and micro injection is performed through the micro flow pump 315. When the injection needle 314 penetrates into the blood vessel, as long as the patient's position does not change, the position of the injection needle 314 can be ensured to be accurately stable by using the stopped motor and each mechanism, so that the injection does not need to rely on artificial methods, and the speed of the micro flow pump 315 is controlled to ensure the safe completion of the injection of thrombolytic drugs and other related drugs in the fundus blood vessels. The path has been established, as long as the drug type is replaced at the injection pump end, the difficulty of finding the blood vessel again is saved, and the injection of multiple drugs can be very accurately completed. The micro flow pump 315 can realize the lower limit of 1pl injection amount and the injection flow rate of 0.05nl / min to 520ul / min; through such flow control, the micro injection demand of the fundus can be met.

[0120] The video acquisition module 33 is a USB video acquisition module 33, and a video acquisition card is arranged on the ophthalmic microscope. The video stream information obtained by the ophthalmic microscope during the operation is collected in real time, processed, and then sent to the cloud by the control module; data transmission through USB can improve the data transmission efficiency and further avoid the occurrence of delay problems. The highest resolution of the camera of the video acquisition module 33 can reach 2160*3840, and a built-in double microphone is used. The USB is connected to the video acquisition card, and high-definition image acquisition and high-speed transmission can be realized.

[0121] The video acquisition module 33 collects and sends back the video stream information of the position and angle of the injection needle 314 in real time, and combines the sent back video stream information to remotely control the SCARA mechanism 321 of the master hand execution structure 11 to execute the determination of the puncture position; control the RCM mechanism 322 to execute the determination of the puncture angle; control the puncture execution structure 31 to execute the puncture action; wherein the switching of the master hand control of the SCARA mechanism 321 and the RCM mechanism 322 is realized through the foot pedal switching structure 12. Through the real-time video stream information sent back, the consistency of the hand-eye coordination can be ensured.

[0122] The use flow of the device is as follows: first, the master control device 1 and the execution robot device 3 are connected by using the 5G device and the 5G network, and the master control device 1 and the execution robot device 3 are assembled. The patient lies on the operating bed and waits for other surgical operations to be completed. When the fundus vascular puncture injection step needs to be performed, the execution robot device 3 is set beside the operating bed; the remote doctor holds the pen-shaped end effector, and the eyes observe the video stream information sent back by the hemostasis robot device end on the video display structure 13, and operate the pen-shaped end effector according to the real-time video stream information; first, press the left key of the foot pedal switching structure 12, so that the pen-shaped end effector instrument control SCARA mechanism 321 moves, and whether the angle of the puncture execution structure 31 needs to be adjusted is judged. When the angle needs to be adjusted, press the right key of the foot pedal switching structure 12; the angle of the puncture execution structure 31 is adjusted, and after adjustment, the left key is switched to adjust the position, and this is repeated until the accurate position and angle of the puncture execution structure 31 are found; then the pen-shaped end execution structure controls the puncture motor 311 of the puncture execution structure 31 to work, and in the process, the real-time force feedback obtained by the Bragg grating is fed back to the pen-shaped end execution structure. When the force feedback of successful puncture is felt, it is proved that the injection needle 314 enters the fundus blood vessel. Then release the pen-shaped end execution structure, and once all the motors stop, the injection needle 314 stops at the injection position of entering the blood vessel, and then the micro-flow pump 315 is used to slowly inject the drug into the fundus blood vessel.

[0123] The use effect of the device is as follows: the above-mentioned device shows high operation stability and precision: the two-end data transmission delay in Shanghai is less than 15ms; the video delay is less than 180ms; and the system realizes remote control between Beijing-Xi'an, Beijing-Guangzhou, Beijing-Haikou; under the remote control in Shanghai, the robot can accurately pierce to the point within 150 microns of the three sizes of the simulated artificial eyeball, and can accurately inject into the liquid, and can observe the fundus bulge of the simulated artificial eyeball through the microscope, and can mark the puncture and drug injection success through the fundus bulge.

[0124] The above is only a specific embodiment of the present application, and those skilled in the art can make other improvements or modifications on the basis of the above-mentioned embodiments under the above-mentioned teaching of the present application. Those skilled in the art should understand that the above-mentioned specific description is only to better explain the purpose of the present application, and the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A remote control system with high accuracy for fundus vascular puncture, characterized in that: include The master hand control device includes a master hand execution structure, a master hand host and a display structure; The execution robot device includes an execution host, a puncture execution structure, a motion control module, and a video acquisition module; the puncture execution structure and the motion control module both include motors for controlling motion; 5G communication module, including a dedicated cloud network and a data transmission terminal consisting of two 5G CPE devices; the two 5G CPE devices are located on the main hand control device and the execution robot device respectively; The master hand host collects and extracts the motion information of the master hand execution structure for processing, and extracts the motion information of the master hand execution structure; The action information of the master hand execution structure is placed into a pre-established communication protocol, converted into frame data containing position information, sequence number and error checksum, and transmitted to the execution robot device using the 5G CPE device; The execution robot device then converts the frame data into the motion information of the execution robot device, and the motion information of the execution robot device includes the position and angle information, movement direction, and movement speed of the injection needle; the execution host processes the motion information of the execution robot device and converts it into data on the rotation direction, rotation speed, and rotation distance of each motor of the execution robot device, and then controls the operation of the motor of the execution robot device according to the data, and finally controls the motion control module to drive the injection needle to the accurate position, maintain the accurate angle and the accurate direction of the injection needle tip, and control the puncture execution structure to perform the puncture action.

2. The system according to claim 1, wherein: After the execution robot device receives the data, the control structure first checks whether the serial number and the error checksum correspond. If they do not correspond, it means that the data frame is wrong. The execution robot device will predict the erroneous data of this frame based on the previously received data and replace the erroneous data with the predicted data. If the data is correct, the action information will be extracted from the received data frame and processing will begin.

3. The system according to claim 1, wherein: The motion control module includes a SCARA mechanism and an RCM mechanism, wherein the SCARA mechanism controls the injection needle to perform position determination, and the RCM mechanism controls the injection needle to perform angle determination and injection needle tip direction; the SCARA mechanism includes 4 motors, the RCM mechanism includes 3 motors, and the puncture actuator includes 1 motor; all motors are connected to an 8-axis motor control structure; the motor control structure is provided with a control card for each motor, and the received data on rotation direction, rotation speed and rotation distance are input into the control card, and the control card controls the corresponding motor to rotate according to the input data.

4. The system according to claim 3, characterized in that The displacement, speed and angle motion information of the main hand execution structure are extracted, and after filtering and debouncing, the data instructions are converted into the joint angles and end postures of the execution robot device. The motor control signals of the rotation direction, rotation speed and rotation distance of each axis motor in the RCM or SCARA mechanism are calculated through the inverse kinematics algorithm; the motor control signals are transmitted to the motor control structure.

5. The system according to claim 3, wherein: The master hand control device includes a foot-operated switching structure, which controls the RCM and SCARA mechanisms of the master hand control device at different times. The video signal of the position and angle of the injection needle of the master hand control device determines whether to perform the control switching of the foot-operated switching structure.

6. The system according to claim 3, wherein: When the main hand execution structure performs repeated movements in a single direction, each single movement process only controls the SCARA mechanism to make a distance change of no more than 1mm and controls the RCM mechanism to make an angle adjustment of 1 degree. The single movement process is further divided into 100 small changes to control the changes of the SCARA mechanism and the RCM mechanism.

7. The system according to claim 6, characterized in that Each motor is equipped with a Hall effect sensor that serves as a limit switch, and an emergency stop relay; the relay action and return time are both less than 0.02s.

8. The system according to claim 1, wherein: A time synchronization module is set up, which is responsible for synchronizing the clocks of the master hand control device and the execution robot device, so that the execution robot device can obtain the accurate time of the control signal through the timestamp, so that the timestamp of the execution robot device is the same as the timestamp of the master hand control device.

9. The system according to claim 8, characterized in that Use the past control value to predict the current control value. Assume that the current time is t, and the control value received is actually at time t-d (i.e. ), it is necessary to solve the problem of updating the predicted value at time t with the control value at time t−d.

10. The system according to claim 9, characterized in that A safety gain is added, and the safety gain gradually decays as the control signal is delayed.

Images