An intranasal neurosurgery robot system based on force feedback guidance and its control method

Abstract

The present invention relates to the field of optical communications technology, and more specifically, to a force feedback-guided intranasal neurosurgery robot system and its control method. Using force feedback adaptive control, the endoscope is able to fit the nasal wall to the greatest possible extent, maximizing the operating space available for other surgical instruments. The proposed robot control method can automatically track lesions (or medical instruments) and adjust the RCM center point, thereby automatically adjusting the endoscope's field of view, completely freeing the surgeon from the burden of holding the endoscope.

Description

Technical Field

[0001] The present invention relates to the field of optical communication technology, and more particularly to an intranasal neurosurgery robot system based on force feedback guidance and a control method thereof. Background Art

[0002] There are no existing surgical robots suitable for intranasal neurosurgery scenarios. Existing endoscope-holding robots can only perform simple endoscope-holding operations. In fact, the doctor still needs to manually control the angle of the endoscope with a remote control. Ultimately, it does not completely solve the problems encountered by doctors during surgery and cannot truly free doctors from endoscope-holding operations. In addition, the control method of existing surgical robots is simple remote control, lacks the assistance of algorithms, and cannot automatically adjust according to the condition of the lesion; on the other hand, existing surgical robots rarely add flexible controllers based on force feedback in the scenario of endoscope-holding. At the same time, the use of endoscopes during surgery will have certain spatial limitations, and the movement of the endoscope at the wound site will be constrained. Existing endoscope-holding robots do not take this into account and do not use remote center motion technology. Summary of the Invention

[0003] In order to overcome the above-mentioned defects in the prior art, the present invention provides a force feedback-guided intranasal neurosurgery robot system and its control method, which realizes the robot's adaptive control of endoscope movement and automatically adjusts the position of the endoscope according to the condition of the intranasal lesion.

[0004] To solve the above technical problems, the present invention adopts a technical solution: an intranasal neurosurgery robot system based on force feedback guidance, comprising:

[0005] Robot: used to receive instructions from the industrial computer and complete corresponding motion functions;

[0006] Industrial computer: used to connect with the robot and communicate with each other, as well as exchange data with other industrial computers;

[0007] The robot's terminal actuator is equipped with a force sensor: used to sense the force from the end of the robot;

[0008] Remote control: used to receive the doctor's control instructions for the robot, connect and communicate with the industrial computer, and control the robot through the industrial computer;

[0009] Video capture card: used to collect endoscopic video images for algorithm processing, to monitor the situation inside the nose in real time during surgery, and to connect and communicate with the industrial computer;

[0010] Monocular camera: used to transmit the video of the surgical scene back to the industrial computer in real time, and connect and communicate with the industrial computer.

[0011] The present invention also provides a control method for an intranasal neurosurgery robot based on force feedback guidance, comprising the following steps:

[0012] S1. The surgery begins. The robot confirms the patient's nasal cavity position based on the image captured by the monocular camera.

[0013] S2. The robot is controlled by an industrial computer to plan the path and enter the nasal cavity with the endoscope.

[0014] S3. The robot holds the endoscope close to the nasal cavity wall;

[0015] S4. The force feedback adaptive control system senses the force applied to the endoscope, and the robot maintains the set force against the inner wall of the nasal cavity.

[0016] S5. Perform remote center movement based on the contact position between the endoscope and the inner wall of the nasal cavity, and transmit the image of the nasal cavity captured by the endoscope back to the industrial computer;

[0017] S6. Automatically adjust the remote center motion (RCM) point position based on the image inside the nasal cavity and automatically track the lesion or instrument;

[0018] S7. The robot maintains its posture and takes endoscopic photos, waiting for the doctor to complete the operation.

[0019] The present invention uses robotic automatic control and remote motion center technology, enabling doctors to complete neurosurgery without the need to hold an endoscope. The present invention also uses image recognition algorithms to capture lesions in real time through endoscopic images, automatically adjust the position of the endoscope according to the intranasal situation, communicate with the robot and adjust the robot's posture, and return real-time intranasal video data to facilitate doctors to perform surgery. In addition, the present invention also includes a force feedback adaptive control system, which allows the endoscope to lean against the inner wall of the patient's nose on one side with a certain force, thereby making room for other surgical instruments, allowing doctors to complete intranasal neurosurgery without expending extra energy.

[0020] Currently, endonasal neurosurgery relies on an experienced physician using a nasal endoscope to monitor the nasal condition. However, few systems exist that eliminate the need for manual endoscope adjustment, provide real-time assistance through visual algorithms, automatically track instrument movement, or even completely replace the physician's endoscope control. This invention not only frees the physician from this manual endoscope control, but also provides visual assistance and prompts during color Doppler ultrasound examinations, automatically planning a path to the desired location.

[0021] Furthermore, the industrial computer controls the robot to move, including the following steps:

[0022] S21. Establish communication between the industrial computer and the robot. Use the robot's built-in external interface to establish TCP and UDP communication between the industrial computer and the robot.

[0023] S22. Receiving commands in the ROS space: The industrial computer establishes a robot control node in the ROS space for controlling the robot. This node subscribes to topics containing control commands from other nodes. Upon receiving the subscribed message, the robot control node enters a callback function, which calls the robot's built-in function to achieve the purpose of controlling the robot.

[0024] S23. Robot return information: After completing a certain movement, the robot returns some key information, including the current coordinates and posture, and the external forces acting on each robot sensor.

[0025] Through this control method, the robot can accurately execute various commands from the industrial computer. By completing different commands, the robot can generally achieve three different movement modes: free dragging, automatic movement, and remote control movement. Switching between these three movement modes is controlled by different buttons on the remote control. When the doctor needs to switch to a different usage mode, he can press the corresponding button, and the robot will switch to the three movement modes respectively.

[0026] Furthermore, in step S2, the robot is controlled to move outside the nasal cavity by free dragging and remote control movement, and then the endoscope is moved into the nasal cavity by the robot's automatic movement.

[0027] Furthermore, the free dragging includes the following steps: when transmitting the function to start the movement to the robot through TCP, the Cartesian coordinate system impedance control is selected as a parameter, and the robot enters the impedance control mode. In this mode, the robot relies on impedance to control the movement of the position in the Cartesian coordinate system; when the impedance coefficient is not 0, the robot is dragged, but the robot will return to its original position by itself; when the impedance coefficient is set to 0, the robot can be freely dragged to any position and will not return by itself.

[0028] Furthermore, the automatic movement is a movement mode based on Cartesian coordinate position control under the control of an adaptive algorithm, and the free movement includes the following steps:

[0029] The robot control node uses built-in functions to make the robot enter the single-position motion mode in the Cartesian coordinate system. At the same time, in another thread, the robot control node starts to subscribe to the "control command" topic in the ROS space and reads and publishes the robot end coordinates in real time.

[0030] The motion planning node receives the robot end coordinate message, calculates the direction and speed the robot end should move at the next moment, and publishes the robot end coordinate to the "control instructions" subscribed by the robot control node;

[0031] After receiving the message, the robot enters the callback function and moves in the determined direction according to the message content.

[0032] Furthermore, remote control movement is a movement method controlled by a robot remote controller or keyboard, which is generally used for large-scale movement at the beginning or end of surgery. The remote control movement includes the following steps:

[0033] The robot control node uses built-in functions to put the robot into single-position motion mode in the Cartesian coordinate system. At the same time, in another thread, the robot control node begins to subscribe to the "control command" topic in the ROS space and reads and publishes the robot's end coordinates and the external forces acting on the robot in real time.

[0034] The keyboard reading node or remote control node reads the remote control operation instructions in real time and publishes the instructions to the "control instructions" topic;

[0035] After receiving the message, the robot enters the callback function and moves in the determined direction according to the message content.

[0036] Furthermore, the force feedback adaptive control is implemented between the industrial computer, the robot, and the force sensor. The control process of the force feedback adaptive control system includes:

[0037] First, the industrial computer establishes a connection with the force sensor through a serial port reading and writing program. The industrial computer reads the force readings on the force sensor and processes the force information in real time. The force information is then added to the compliance control loop to obtain new position control information, which is then converted into corresponding robot control instructions.

[0038] Furthermore, the industrial computer communicates with the robot and uses the robot's automatic control mode. The industrial computer sends robot control instructions converted from position control information to the robot; after the robot completes the instructions, it calculates the position deviation and then adds it to the flexible control loop to correct the position information.

[0039] Furthermore, the remote center movement specifically includes the following steps:

[0040] The endoscope enters the nasal cavity and is placed against the inner wall of the nasal cavity using force feedback adaptive control to obtain images of the interior of the nasal cavity. Image recognition technology is used to preliminarily determine and record the location of the lesion or medical device.

[0041] According to the position of the contact point between the endoscope and the inner wall of the nose, this point is selected as the center point of RCM, and the posture information of the endoscope after movement is obtained according to the RCM algorithm;

[0042] Based on the obtained endoscope posture information and lesion or medical device image information, the robot's desired position is calculated through an adaptive algorithm. The trajectory and control instructions for the robot's end are generated based on the position information, and the corresponding instructions are then sent from the industrial computer to the robot.

[0043] After the robot moves to a new position, it obtains the internal image information and force feedback information of the nasal cavity again, adjusts the selected position of the RCM center point, and combines the RCM algorithm to perform the next RCM operation and continue to control the robot to move the endoscope.

[0044] Compared with the existing technology, the beneficial effects are: the present invention provides an intranasal neurosurgery robot system based on force feedback guidance and its control method, which uses force feedback adaptive control to enable the endoscope to fit the inner wall of the nose to the greatest extent, thereby maximizing the doctor's operating space for other instruments during surgery; the robot control method proposed in the present invention can automatically track lesions (or medical devices) and adjust the RCM center point, thereby automatically adjusting the endoscope field of view, completely freeing the doctor from the work of holding the endoscope. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the relationship between the present invention and the system.

[0046] Figure 2 It is a schematic flow chart of the method of the present invention.

[0047] Figure 3 Schematic diagram of remote center movement provided by the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The present invention is described in one of the embodiments below in combination with the specific implementation methods. Among them, the drawings are only for illustrative purposes and represent only schematic diagrams rather than physical drawings, and cannot be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0049] In the description of the present invention, it should be understood that if the terms "upper," "lower," "left," "right," etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, they are only for the purpose of facilitating the description of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, the terms describing the positional relationship in the accompanying drawings are only for illustrative purposes and should not be construed as limiting this patent. For those skilled in the art, the specific meanings of the above terms can be understood based on the specific circumstances. In addition, if there are descriptions of "first," "second," etc. in the embodiments of the present invention, the descriptions of "first," "second," etc. are only for descriptive purposes and should not be construed as indicating or implying their relative importance or implicitly indicating the number of the technical features indicated. Therefore, the definition of "first" or "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel solutions. For example, "A and / or B" includes solution A, solution B, or solutions that meet both A and B.

[0050] Example 1:

[0051] like Figure 1 As shown, an intranasal neurosurgery robot system based on force feedback guidance includes:

[0052] Robot: used to receive instructions from the industrial computer and complete corresponding motion functions;

[0053] Industrial computer: used to connect with the robot and communicate with each other, as well as exchange data with other industrial computers;

[0054] The robot's terminal actuator is equipped with a force sensor: used to sense the force from the end of the robot;

[0055] Remote control: used to receive the doctor's control instructions for the robot, connect and communicate with the industrial computer, and control the robot through the industrial computer;

[0056] Video capture card: used to collect endoscopic video images for algorithm processing, to monitor the situation inside the nose in real time during surgery, and to connect and communicate with the industrial computer;

[0057] Monocular camera: used to transmit the video of the surgical scene back to the industrial computer in real time, and connect and communicate with the industrial computer;

[0058] The display is used to display the captured videos and images for the doctor to view at any time.

[0059] Example 2

[0060] like Figure 2As shown, this embodiment provides a control method for an intranasal neurosurgery robot based on force feedback guidance, comprising the following steps:

[0061] S1. The surgery begins. The robot confirms the patient's nasal cavity position based on the image captured by the monocular camera.

[0062] S2. The robot is controlled by an industrial computer to plan the path and enter the nasal cavity with the endoscope.

[0063] S3. The robot holds the endoscope close to the nasal cavity wall;

[0064] S4. The force feedback adaptive control system senses the force applied to the endoscope, and the robot maintains the set force against the inner wall of the nasal cavity.

[0065] S5. Perform remote center movement based on the contact position between the endoscope and the inner wall of the nasal cavity, and transmit the image of the nasal cavity captured by the endoscope back to the industrial computer;

[0066] S6. Automatically adjust the remote center motion (RCM) point position based on the image inside the nasal cavity and automatically track the lesion or instrument;

[0067] S7. The robot maintains its posture and takes endoscopic photos, waiting for the doctor to complete the operation.

[0068] The present invention uses robotic automatic control and remote motion center technology, enabling doctors to complete neurosurgery without the need to hold an endoscope. The present invention also uses image recognition algorithms to capture lesions in real time through endoscopic images, automatically adjust the position of the endoscope according to the intranasal situation, communicate with the robot and adjust the robot's posture, and return real-time intranasal video data to facilitate doctors to perform surgery. In addition, the present invention also includes a force feedback adaptive control system, which allows the endoscope to lean against the inner wall of the patient's nose on one side with a certain force, thereby making room for other surgical instruments, allowing doctors to complete intranasal neurosurgery without expending extra energy.

[0069] Currently, endonasal neurosurgery relies on an experienced physician using a nasal endoscope to monitor the nasal condition. However, few systems exist that eliminate the need for manual endoscope adjustment, provide real-time assistance through visual algorithms, automatically track instrument movement, or even completely replace the physician's endoscope control. This invention not only frees the physician from this manual endoscope control, but also provides visual assistance and prompts during color Doppler ultrasound examinations, automatically planning a path to the desired location.

[0070] Specifically, the industrial computer controls the robot to move, including the following steps:

[0071] S21. Establish communication between the industrial computer and the robot. Use the robot's built-in external interface to establish TCP and UDP communication between the industrial computer and the robot.

[0072] S22. Receiving commands in the ROS space: The industrial computer establishes a robot control node in the ROS space for controlling the robot. This node subscribes to topics containing control commands from other nodes. Upon receiving the subscribed message, the robot control node enters a callback function, which calls the robot's built-in function to achieve the purpose of controlling the robot.

[0073] S23. Robot return information: After completing a certain movement, the robot returns some key information, including the current coordinates and posture, and the external forces acting on each robot sensor.

[0074] Through this control method, the robot can accurately execute various commands from the industrial computer. By completing different commands, the robot can generally achieve three different movement modes: free dragging, automatic movement, and remote control movement. Switching between these three movement modes is controlled by different buttons on the remote control. When the doctor needs to switch to a different usage mode, he can press the corresponding button, and the robot will switch to the three movement modes respectively.

[0075] In step S2, the robot is controlled to move outside the nasal cavity by free dragging and remote control, and then the endoscope is moved into the nasal cavity by the robot's automatic movement.

[0076] Force feedback adaptive control works between the industrial computer, the robot, and the force sensor. The control process of the force feedback adaptive control system includes:

[0077] First, the industrial computer establishes a connection with the force sensor through a serial port reading and writing program. The industrial computer reads the force readings on the force sensor and processes the force information in real time. The force information is then added to the compliance control loop to obtain new position control information, which is then converted into corresponding robot control instructions.

[0078] Furthermore, the industrial computer communicates with the robot and uses the robot's automatic control mode. The industrial computer sends robot control instructions converted from position control information to the robot; after the robot completes the instructions, it calculates the position deviation and then adds it to the flexible control loop to correct the position information.

[0079] The telecenter movement specifically includes the following steps:

[0080] The endoscope enters the nasal cavity and is placed against the inner wall of the nasal cavity using force feedback adaptive control to obtain images of the interior of the nasal cavity. Image recognition technology is used to preliminarily determine and record the location of the lesion or medical device.

[0081] According to the position of the contact point between the endoscope and the inner wall of the nose, this point is selected as the center point of RCM, and the posture information of the endoscope after movement is obtained according to the RCM algorithm;

[0082] Based on the obtained endoscope posture information and lesion or medical device image information, the robot's desired position is calculated through an adaptive algorithm. The trajectory and control instructions for the robot's end are generated based on the position information, and the corresponding instructions are then sent from the industrial computer to the robot.

[0083] After the robot moves to a new position, it obtains the internal image information and force feedback information of the nasal cavity again, adjusts the selected position of the RCM center point, and combines the RCM algorithm to perform the next RCM operation and continue to control the robot to move the endoscope.

[0084] Example 3

[0085] This embodiment is the same as Embodiment 1 in other steps, except that, in this embodiment, the free dragging includes the following steps: when transmitting the function for starting the movement to the robot via TCP, the Cartesian coordinate system impedance control is selected as a parameter, and the robot enters the impedance control mode. In this mode, the robot relies on the impedance to control the movement of the position in the Cartesian coordinate system; when the impedance coefficient is not 0, the robot is dragged, but the robot will return to its original position by itself; when the impedance coefficient is set to 0, the robot can be freely dragged to any position and will not return by itself.

[0086] Automatic movement is a movement mode based on Cartesian coordinate system position control under the control of an adaptive algorithm. The free movement includes the following steps:

[0087] 1. The robot control node uses built-in functions to make the robot enter the single-position motion mode in the Cartesian coordinate system. At the same time, in another thread, the robot control node starts to subscribe to the "control command" topic in the ROS space and reads and publishes the robot's end coordinates in real time.

[0088] 2. The motion planning node receives the robot end coordinate message, calculates the direction and speed the robot end should move at the next moment, and publishes the robot end coordinate to the "control instructions" subscribed by the robot control node;

[0089] 3. After receiving the message, the robot enters the callback function and moves in the determined direction according to the message content.

[0090] Remote control movement is a movement method controlled by a robot remote control or keyboard. It is generally used for large-scale movement at the beginning or end of surgery. The remote control movement includes the following steps:

[0091] 1. The robot control node uses built-in functions to make the robot enter the single-position motion mode in the Cartesian coordinate system. At the same time, in another thread, the robot control node starts to subscribe to the "control command" topic in the ROS space and reads and publishes the robot's end coordinates and the external forces acting on the robot in real time.

[0092] 2. The keyboard reading node or remote control node reads the remote control operation instructions in real time and publishes the instructions to the "control instructions" topic;

[0093] 3. After receiving the message, the robot enters the callback function and moves in the determined direction according to the message content.

[0094] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the claims of the present invention.

Claims

1. A force feedback guided intranasal neurosurgery robot system, characterized in that: include: Robot: used to receive instructions from the industrial computer and complete corresponding motion functions; Industrial computer: used to connect with the robot and communicate with each other, as well as exchange data with other industrial computers; The robot's terminal actuator is equipped with a force sensor: used to sense the force from the end of the robot; Remote control: used to receive the doctor's control instructions for the robot, connect and communicate with the industrial computer, and control the robot through the industrial computer; Video capture card: used to collect endoscopic video images for algorithm processing, to monitor the situation inside the nose in real time during surgery, and to connect and communicate with the industrial computer; Monocular camera: used to transmit the video of the surgical scene back to the industrial computer in real time, and connect and communicate with the industrial computer; The robot's force feedback adaptive control is achieved through the following steps: the industrial computer establishes a connection with the force sensor through a serial port reading and writing program. The industrial computer reads the force reading on the force sensor and can also process the force information in real time; the force information is then added to the compliance control loop to obtain new position control information, which is converted into corresponding robot control instructions; the industrial computer communicates with the robot, uses the robot's automatic control mode, and sends the robot control instructions converted from the position control information to the robot; after the robot completes the instruction, the position deviation is calculated and then added to the compliance control loop to correct the position information; The robot receives instructions from the industrial computer and completes the corresponding motion functions, including: The robot holds the endoscope and enters the nasal cavity. Based on force feedback, it adaptively controls the endoscope to lean against the inner wall of the nasal cavity, obtains images of the interior of the nasal cavity, and uses image recognition technology to preliminarily determine the location of the lesion or medical device and record it. According to the position of the contact point between the endoscope and the inner wall of the nose, this point is selected as the center point of RCM, and the posture information of the endoscope after movement is obtained according to the RCM algorithm; Based on the obtained endoscope posture information and lesion or medical device image information, the robot's desired position is calculated through an adaptive algorithm. The trajectory and control instructions for the robot's end are generated based on the position information, and the corresponding instructions are then sent from the industrial computer to the robot. After the robot moves to a new position, it obtains the internal image information and force feedback information of the nasal cavity again, adjusts the selected position of the RCM center point, and combines the RCM algorithm to perform the next RCM operation and continue to control the robot to move the endoscope.

2. The intranasal neurosurgery robot system based on force feedback guidance according to claim 1, characterized in that: The industrial computer controls the robot to move, including the following steps: S21. Establish communication between the industrial computer and the robot. Use the robot's built-in external interface to establish TCP and UDP communication between the industrial computer and the robot. S22. Receiving commands in the ROS space: The industrial computer establishes a robot control node in the ROS space for controlling the robot. This node subscribes to topics containing control commands from other nodes. Upon receiving the subscribed message, the robot control node enters a callback function, which calls the robot's built-in function to achieve the purpose of controlling the robot. S23. Robot return information: After completing a certain movement, the robot returns some key information, including the current coordinates and posture, and the external forces acting on each robot sensor.

3. The force feedback guided intranasal neurosurgery robot system according to claim 2, characterized in that: The ways in which the industrial computer controls the robot's movement include free dragging, automatic movement, and remote control movement.

4. The force feedback-guided intranasal neurosurgery robot system according to claim 3, characterized in that: By freely dragging and remotely controlling the movement, the robot is controlled to move outside the nasal cavity, and then the endoscope is moved into the nasal cavity by the robot's automatic movement.

5. The force feedback-guided intranasal neurosurgery robot system according to claim 4, characterized in that: The free dragging includes the following steps: when transmitting the function to start the movement to the robot through TCP, the Cartesian coordinate system impedance control is selected as the parameter, and the robot enters the impedance control mode. In this mode, the robot relies on the impedance to control the movement of the position in the Cartesian coordinate system; when the impedance coefficient is not 0, the robot is dragged, but the robot will return to its original position on its own; when the impedance coefficient is set to 0, the robot can be freely dragged to any position and will not return on its own.

6. The intranasal neurosurgery robot system based on force feedback guidance according to claim 4, characterized in that: The automatic movement comprises the following steps: The robot control node uses built-in functions to put the robot into single-position motion mode in the Cartesian coordinate system. At the same time, in another thread, the robot control node begins to subscribe to the "control command" topic in the ROS space and reads and publishes the robot's end coordinates in real time. The motion planning node receives the robot end coordinate message, calculates the direction and speed the robot end should move at the next moment, and publishes the robot end coordinate to the "control instructions" subscribed by the robot control node; After receiving the message, the robot enters the callback function and moves in the determined direction according to the message content.

7. The intranasal neurosurgery robot system based on force feedback guidance according to claim 4, characterized in that: The remote control movement comprises the following steps: The robot control node uses built-in functions to put the robot into single-position motion mode in the Cartesian coordinate system. At the same time, in another thread, the robot control node begins to subscribe to the "control command" topic in the ROS space and reads and publishes the robot's end coordinates and the external forces acting on the robot in real time. The keyboard reading node or remote control node reads the remote control operation instructions in real time and publishes the instructions to the "control instructions" topic; After receiving the message, the robot enters the callback function and moves in the determined direction according to the message content.

Images