Bronchial intervention surgery robot control system and method based on eye tracking

Abstract

The application provides a bronchial intervention surgery robot control system and method based on eye movement tracking. A collection module collects gaze point coordinate original data of an operator; a calibration mapping module receives the gaze point coordinate original data, and establishes a mapping relationship between the gaze point coordinate and a display terminal; an image display module displays the corresponding gaze point coordinate on an airway image according to the mapping relationship; a signal processing module receives and processes the gaze point coordinate original data, and obtains a gaze point sequence; an intention determination module updates the gaze point coordinate according to the gaze point sequence; a region determination module divides an operable region from the airway image, and generates a control instruction in the operable region by using the updated gaze point coordinate; and an execution control module controls the robot according to the control instruction generated by the region determination module. The application can reduce the operation burden of the operator, improve the control response speed, enhance the rapid positioning ability of the operable region, and improve the operation efficiency.

Description

Technical Field

[0001] This application relates to the field of medical devices, specifically to a control system and method for a bronchial interventional surgical robot based on eye tracking. Background Technology

[0002] With the rapid advancement of industrialization and the leapfrog development of science and technology, human society is flourishing in terms of material and information resources. However, this has also brought increasingly severe environmental and health challenges. Air pollution caused by industrial waste gas and vehicle exhaust, as well as the mental stress and unhealthy lifestyle habits brought about by a fast-paced life, have contributed to the persistently high incidence and mortality rates of lung cancer. As a result, lung cancer places a heavy burden on people's lives, necessitating strengthened prevention and control measures.

[0003] Currently, the main treatment options for respiratory diseases include medication, lifestyle modifications, and surgery. For pulmonary nodules, new technologies such as electromagnetic navigation bronchoscopy (ENB) and robot-assisted bronchoscopy (RAB) have demonstrated high safety and effectiveness in diagnosing subsolid nodules. Bronchial interventional surgery offers advantages such as small surgical incisions, less bleeding, rapid postoperative recovery, and a low complication rate. During bronchial interventional surgery, the patient is first assessed and anesthetized preoperatively to establish a safe airway. After anesthesia takes effect, the physician inserts a bronchoscope through the mouth or nose to observe the trachea, bronchi, and more distal branches. Then, combining the bronchoscopic images with preoperative imaging, the physician confirms the lesion location, understands the patient's condition, and uses lasers, high-frequency electrosurgical units, balloon catheters, etc., introduced through the bronchoscope's working channel for mechanical dilation, resection and ablation, or stent implantation; if necessary, biopsy, suctioning, or lavage are performed simultaneously. Finally, the treatment devices and bronchoscope were removed in sequence, and the patient's vital signs and blood oxygenation were continuously monitored to complete postoperative management.

[0004] During bronchial interventional surgery, due to the numerous bronchial branches and highly complex anatomical structures, the advancement of the endoscope, branch selection, and lesion identification rely heavily on the surgeon's extensive experience. This places higher demands on the surgeon's proficiency, making it difficult for novice surgeons to achieve the same level of accuracy as expert physicians. Furthermore, factors such as lighting and the presence of human tissue introduce significant uncertainty into the identification of bronchial branches during surgery, making it impossible to accurately identify and precisely locate anatomical structures.

[0005] Surgical robot technology is constantly evolving with technological advancements, bringing benefits to doctors and patients. Currently, most bronchial interventional surgical robot systems developed domestically and internationally adopt a master-slave teleoperation control architecture. A master-slave bronchial interventional surgical robot system generally includes a master control console, a slave robot, and an electronic bronchoscope imaging system. During the procedure, the surgeon can observe the lesion location outside the operating room based on real-time transmitted bronchoscopic images. Then, by controlling the master console, the surgeon can remotely control the slave robot located inside the operating room to perform operations such as guidewire / catheter delivery, stent deployment, and biopsy sampling. Thus, the master-slave surgical robot can overcome surgical difficulties caused by differences in operator skill and has an anti-shake effect. Current research has developed several bronchial interventional robots, but they generally suffer from low levels of intelligence and difficulty in capturing the operator's intentions.

[0006] A literature search of existing technologies revealed that Chinese patent CN121313315A proposes a rapid and replaceable bronchoscope interventional surgical robot that can deliver a disposable bronchoscope directly to the lesion. However, its shortcomings include high operational complexity, heavy cognitive burden, and an unnatural human-computer interaction method.

[0007] A literature search of existing technologies revealed that Chinese patent CN114886571A proposes a control method and system for an interventional surgical robot. This system can stop the movement of the interventional surgical robot in a timely manner when an abnormal situation occurs, so as to prevent dangerous behavior from occurring. However, its drawback is that it is difficult to respond to the doctor's visual attention area in a timely manner, which limits the efficiency of operation.

[0008] A literature search of existing technologies revealed that Chinese patent CN118542635A proposes an endoscope control system based on eye movement recognition, which realizes direct eye movement control of the endoscope function. By continuously monitoring and analyzing the operator's eye movements, the position, angle and focal length of the endoscope are adjusted to meet the immediate needs during surgery, improve the accuracy and response speed of surgical operations, and significantly enhance the convenience of operation and the overall efficiency of surgery. Its shortcoming is that it is difficult to achieve precise control of bronchial interventional surgical robots.

[0009] Currently, teleoperation control requires doctors to continuously coordinate hand and eye movements, increasing operational complexity and easily leading to distraction, thus placing higher demands on surgical safety. Furthermore, in complex airway environments, teleoperation struggles to respond promptly to the doctor's visual focus area, limiting the intuitiveness and efficiency of the procedure. Therefore, introducing a more natural and efficient human-computer interaction method to improve the safety of bronchial interventional surgical robots is extremely important. There is an urgent need to develop an interventional surgical robot control system and method that can achieve rapid response and precise control. Summary of the Invention

[0010] In view of the deficiencies in the prior art, the purpose of this application is to provide a control system and method for a bronchial interventional surgery robot based on eye tracking.

[0011] According to one aspect of this application, a control system for a bronchial interventional surgical robot based on eye tracking is provided, comprising: The data acquisition module collects the raw data of the operator's gaze coordinates; The calibration mapping module receives the raw gaze point coordinate data collected by the acquisition module and establishes a mapping relationship between the gaze point coordinates and the display terminal. The image display module displays the coordinates of the corresponding fixation point on the airway image of the patient's bronchus according to the mapping relationship established by the calibration mapping module; The signal processing module receives and processes the raw gaze point coordinate data acquired by the acquisition module to obtain a gaze point sequence; The intent determination module updates the gaze point coordinates based on the gaze point sequence obtained by the signal processing module. The region determination module divides the operable region from the airway image and generates control commands within the operable region using the gaze point coordinates updated by the intent determination module. The execution control module controls the robot according to the control instructions generated by the area determination module.

[0012] Optionally, the acquisition module is integrated into the eye tracker and includes an image acquisition submodule and a gaze estimation submodule, wherein: The image acquisition submodule acquires image information of the operator's eyes; The gaze estimation submodule receives and processes the eye image information acquired by the image acquisition submodule to generate raw data of gaze point coordinates.

[0013] Optionally, the calibration mapping module includes: Spatial calibration is performed based on the calibration marks placed at the boundary of the display terminal; Based on spatial calibration information, the eye tracker's gaze is calibrated and the display area of ​​the display terminal is defined. Based on the original data of the gaze point coordinates, a mapping relationship between the gaze point coordinates and the display area of ​​the display terminal is established.

[0014] Optionally, the image display module uses a display terminal to display airway images of the patient's bronchi.

[0015] Optionally, the signal processing module is integrated into the eye tracker and includes: The original gaze point coordinate data is processed by sliding window midpoint filtering to obtain intermediate gaze point coordinate data after suppressing transient noise. The intermediate data of fixation point coordinates after sliding window midpoint filtering are smoothed by exponential moving average filtering, thereby obtaining a continuous fixation point sequence.

[0016] Optionally, the intent determination module inputs the gaze sequence into a pre-trained network model to identify the operator's eye movement state, which includes one of fixation, saccade, and smoothing. Specifically, when the intent determination module first identifies the eye movement state as a gaze state at a certain moment, it extracts a gaze point subsequence that matches that moment from the gaze point sequence, and calculates and generates an initial anchor point from the gaze point subsequence; thereafter, when the eye movement state is identified as a gaze state, it extracts the gaze point subsequence corresponding to that moment from the gaze point sequence to update the anchor point, and keeps the coordinates of the anchor point unchanged in the saccade state or smooth state; Control the current gaze point coordinates to converge to the current anchor point coordinates, and update the gaze point coordinates.

[0017] Optionally, the region determination module includes: The airway image is semantically segmented to divide the airway image into an operable region and a prohibited region. The operable region is the image area in which surgical operations are allowed to be performed during the operation, and the prohibited region is the image area in which surgical operations are not allowed to be performed during the operation. When the updated gaze point coordinates are located within the operable area, a control command is generated and issued; when the updated gaze point coordinates are located within the prohibited area, the output of the control command is paused.

[0018] Optionally, the control command is obtained based on the relative position of the updated gaze point coordinates and the target lesion in the airway image.

[0019] Optionally, the execution control module sends the received control commands to the robot, causing the bronchoscope structure in the robot to adjust its posture and feed depth, and approach the target lesion along its preset planned path.

[0020] According to another aspect of this application, a method for controlling a bronchial interventional surgical robot based on eye tracking is provided, comprising: Establish a mapping relationship between the gaze point coordinates and the display terminal; The airway image of the patient's bronchus is displayed on the display terminal in real time. Based on the mapping relationship between the gaze point coordinates and the display terminal, the corresponding gaze point coordinates are displayed on the airway image. The operator's gaze point coordinates are collected in real time, and the raw gaze point coordinates are processed to obtain a gaze point sequence. Update the gaze point coordinates based on the gaze point sequence; The operable area is delineated from the airway image, and control commands are generated within the operable area using the updated gaze coordinates. The robot is controlled based on control commands.

[0021] This application provides a bronchial interventional surgical robot control system based on eye tracking. The system utilizes a data acquisition module to collect the operator's gaze coordinates in real time and maps these coordinates onto an airway image on a display terminal. This enables real-time tracking of the operator's gaze coordinates on the airway image and generates control commands for the robot based on these coordinates. The control system monitors and processes the operator's gaze coordinate data to generate corresponding control commands. By focusing on visual attention, the system reduces the operator's workload, improves control response speed, and enhances the ability to quickly locate the operable area, thereby improving the safety and efficiency of the surgical procedure.

[0022] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0023] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the control system of the bronchial interventional surgery robot in one embodiment of this application; Figure 2 This is an overall scene diagram of a bronchial interventional surgical robot according to one embodiment of this application; Figure 3 This is a schematic diagram of a joystick handle in one embodiment of this application; Figure 4 This is a flowchart of a bronchial interventional surgery robot control method in one embodiment of this application; Figure 5 This is a flowchart illustrating the workflow of a bronchial interventional surgical robot in one embodiment of this application.

[0024] In the diagram: 1. Eye tracker; 2. Display terminal; 3. Robot; 31. Second support; 32. First support; 33. Bronchoscope structure; 331. Bronchoscope; 34. Base; 35. Power plug; 36. USB interface; 37. Photoelectric switch; 38. Drive device; 381. Second drive device; 3811. Deflection servo; 3812. Paddle; 382. First drive device; 3821. Lower gear pair; 3822. Upper gear pair; 3823. Motor; 383. Third drive device; 3831. Linear motion motor; 4. Handle; 41. Left joystick; 42. Right joystick. Detailed Implementation

[0025] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0026] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0027] Surgical robot technology is constantly evolving with technological advancements. Currently, most bronchial interventional surgical robot systems developed domestically and internationally adopt a master-slave teleoperation control architecture. Many existing bronchial interventional robots generally suffer from low levels of intelligence and difficulty in capturing the operator's intentions. Teleoperation control requires continuous hand-eye coordination from the surgeon, increasing operational complexity and easily leading to distraction, thus placing higher demands on surgical safety. Furthermore, in complex airway environments, teleoperation struggles to respond promptly to the surgeon's visual focus area, limiting the intuitiveness and efficiency of the procedure. Based on these issues, this application provides an eye-tracking-based bronchial interventional surgical robot control system to address these problems.

[0028] Reference Figure 1As shown in the figure, this application provides a bronchial interventional surgery robot control system 100 based on eye tracking, including a data acquisition module 110, a calibration and mapping module 120, an image display module 130, a signal processing module 140, an intent determination module 150, a region determination module 160, and an execution control module 170. The data acquisition module 110 acquires raw data of the operator's gaze coordinates; the calibration and mapping module 120 receives the raw data of the gaze coordinates acquired by the data acquisition module 110 and establishes a mapping relationship between the gaze coordinates and the display terminal; the image display module 130, according to the calibration and mapping module... The mapping relationship established by the blocks displays the corresponding gaze point coordinates on the airway image of the patient's bronchus; the signal processing module 140 receives and processes the raw gaze point coordinate data acquired by the acquisition module 110 to obtain a gaze point sequence; the intent determination module 150 updates the gaze point coordinates according to the gaze point sequence obtained by the signal processing module 140; the region determination module 160 divides the operable region from the airway image and generates control commands in the operable region using the gaze point coordinates updated by the intent determination module 150; the execution control module 170 controls the robot according to the control commands generated by the region determination module 160.

[0029] The embodiments described above utilize a data acquisition module to collect the operator's gaze coordinates in real time and map these coordinates onto the airway image on the display terminal. This enables real-time tracking of the operator's gaze coordinates on the airway image and generates control commands for the robot based on these coordinates. The control system of this application monitors and processes the operator's gaze coordinate data and generates corresponding control commands. By focusing on visual attention, the operator's workload is reduced, control response speed is improved, and the ability to quickly locate the operable area is enhanced, thereby improving the safety and efficiency of surgical procedures.

[0030] In some specific embodiments of this application, the acquisition module is integrated into the eye tracker, including an image acquisition submodule and a gaze estimation submodule; wherein, the image acquisition submodule acquires image information of the operator's eyes, and the gaze estimation submodule receives and processes the eye image information acquired by the image acquisition submodule to generate raw data of gaze point coordinates.

[0031] In the above embodiments of this application, an acquisition module can be used to acquire the operator's eye image information. The eye image information, as an eye movement signal, can express the operator's gaze point position in real time. Subsequently, the gaze point position can be used to generate precise control commands for the robot.

[0032] In some specific embodiments of this application, the calibration mapping module may further include: Spatial calibration is performed based on the calibration marks placed on the boundary of the display terminal 2. Based on the spatial calibration information, the gaze of the eye tracker 1 is calibrated and the display area of ​​the display terminal 2 is defined. Based on the original data of the gaze point coordinates, a mapping relationship between the gaze point coordinates and the display area of ​​the display terminal 2 is established.

[0033] The embodiments described above establish a mapping relationship between the gaze point coordinates and the display area of ​​the display terminal. This can be used to display the gaze point coordinates on the airway image of the display terminal, achieving matching between the eye movement coordinate system and the airway image pixel coordinate system. This allows the gaze point to accurately fall on the fine structure of the airway, resulting in high positioning accuracy.

[0034] In some specific embodiments of this application, the image display module uses a display terminal to display airway images of the patient's bronchi.

[0035] In some specific embodiments of this application, the signal processing module is integrated into the eye tracker and may further include: The original fixation point coordinate data is processed by sliding window midpoint filtering to obtain intermediate fixation point coordinate data after suppressing transient noise. The intermediate fixation point coordinate data after sliding window midpoint filtering is then smoothed by exponential moving average filtering to obtain a continuous fixation point sequence.

[0036] The embodiments described above employ a sliding window midpoint filtering method to suppress instantaneous noise in the original gaze point coordinate data; and an exponential moving average filtering method to resolve minor physiological jitter and subtle sampling fluctuations remaining after sliding window midpoint filtering, ultimately generating a stable and coherent gaze point sequence.

[0037] In some specific embodiments of this application, the intent determination module inputs the gaze point sequence into a pre-trained network model to identify the operator's eye movement state, which includes one of fixation, saccade, and smoothing. Specifically, when the intent determination module first identifies the eye movement state as a gaze state at a certain moment, it extracts the gaze point subsequence that matches that moment from the gaze point sequence and calculates and generates the initial anchor point from the gaze point subsequence; thereafter, when the eye movement state is identified as a gaze state, it extracts the gaze point subsequence corresponding to that moment from the gaze point sequence to update the anchor point, and keeps the coordinates of the anchor point unchanged in the saccade state or smooth state. Control the current gaze point coordinates to converge to the current anchor point coordinates, and update the gaze point coordinates.

[0038] For example, the anchor point is generated by averaging the x-coordinates and y-coordinates of all gaze points in the gaze point subsequence. The average x-coordinate is used as the x-coordinate of the anchor point, and the average y-coordinate is used as the y-coordinate of the anchor point.

[0039] In the above embodiments of this application, the anchor point is updated when the eye movement is fixation, and no anchor point is updated during saccade or smoothing movements. This can effectively avoid the interference noise and pseudo-fixation points generated during rapid eye saccades or smoothing, avoid anchor point offset, jitter and false updates caused by unstable viewpoints, improve the accuracy and temporal stability of anchor point positioning, and make the anchor point update logic more in line with the real movement law of human eyes.

[0040] In some specific embodiments of this application, the region determination module may further include: Semantic segmentation of airway images is performed to divide the airway images into operable regions and prohibited regions. Operable regions are image areas where surgical operations are allowed during the operation, while prohibited regions are image areas where surgical operations are not allowed during the operation. When the updated gaze point coordinates are within the operable area, control commands are generated and issued; when the updated gaze point coordinates are within the prohibited area, the output of control commands is paused.

[0041] In the above embodiments of this application, the region determination module uses whether the gaze point falls within the operable area as the basis for determining the start and stop of the control command. The rule boundaries are clear and the logic is rigorous and consistent. Control commands are only allowed to be output when the gaze point is within the effective operable range, which can effectively avoid false triggering and false operation, and greatly improve the accuracy and stability of gaze interaction.

[0042] In some specific embodiments of this application, the control commands are obtained based on the relative positional relationship between the updated gaze point coordinates and the target lesion in the airway image.

[0043] In the above embodiments of this application, the gaze point is used as the control source. Control commands can be generated in real time based on the coordinates of the gaze point on the airway image. The control command generation process is simple and has strong real-time performance. The robot can be controlled through eye movement information, which can effectively reduce the operator's workload.

[0044] In some specific embodiments of this application, the execution control module sends the received control commands to the robot, so that the bronchoscope structure in the robot adjusts its posture and feed depth, and approaches the target lesion along its preset planned path.

[0045] The above embodiments of this application generate control commands with strong purpose, which can accurately drive the bronchoscope to approach the target lesion along the planned path.

[0046] For example, the signal processing module is used to filter and determine the stability of the gaze point coordinate data. Only when the gaze point coordinates meet the preset stability conditions will the corresponding gaze point sequence be sent as a valid signal to the subsequent intent determination module, so as to avoid false triggering caused by instantaneous gaze fluctuations, thereby improving the reliability and stability of system control.

[0047] Reference Figure 2 As shown in some specific embodiments of this application, the eye tracker 1 is disposed on one side of the display terminal 2 to track the operator's / doctor's gaze in real time and completes calibration through a preset calibration process, thereby continuously outputting the coordinates of the gaze point corresponding to the display area of ​​the display terminal 2. While displaying the bronchoscopic image (i.e., the bronchial airway image), the display terminal 2 can also overlay and display the gaze point position or target area prompt information, thereby characterizing the operator's area of ​​focus and generating control commands accordingly to drive the robot 3 to perform corresponding movements.

[0048] Furthermore, the collaborative design of the eye tracker 1 and the display terminal 2 enables real-time acquisition and visual feedback of the doctor's gaze points, facilitating intuitive control and status confirmation by the operator. The eye tracker 1 collects the doctor's eye movement information and converts it into control commands, which are then sent to the robot 3. By continuously tracking and stabilizing the gaze signals, the eye tracker 1 accurately triggers the control commands, ensuring the stability and reliability of the control process, thereby improving the accuracy and safety of surgical procedures. The eye tracker 1 and robot 3 employ an eye-tracking signal-based control mode, supporting rapid calibration and stable interaction, achieving non-contact input, which helps improve human-computer interaction efficiency and reduce operational burden.

[0049] Furthermore, the acquisition module 110, calibration mapping module 120, image display module 130, signal processing module 140, intent determination module 150, region determination module 160, and execution control module 170 constitute the eye-tracking control mode of this application.

[0050] In some specific embodiments of this application, the bronchial interventional surgery robot control system is further provided with a manual auxiliary control mode, an eye-tracking control mode and a manual auxiliary control mode, which control the robot independently.

[0051] Reference Figure 3 As shown, the manual auxiliary control mode uses handle 4, which is wirelessly connected to robot 3. Handle 4 includes a left joystick 41 and a right joystick 42. The left joystick 41 can be used to control the forward and backward movement of the bronchoscope in robot 3 by operating it up and down, and the left joystick 41 can be used to control the overall rotation of the bronchoscope by operating it left and right. The right joystick 42 can be used to control the deflection of the tip of the bronchoscope by operating it left and right.

[0052] Reference Figure 4 As shown, based on the same inventive concept, another embodiment of this application provides a control method for a bronchial interventional surgical robot based on eye tracking, implemented based on the bronchial interventional surgical robot control system in the above embodiment, including: S1. Establish the mapping relationship between the gaze point coordinates and the display terminal 2; S2. Display the airway image of the patient's bronchus on the display terminal 2 in real time, and display the corresponding gaze point coordinates on the airway image based on the mapping relationship between the gaze point coordinates and the display terminal 2. S3. Real-time acquisition of the operator's gaze point coordinates; processing of the gaze point coordinates to obtain a gaze point sequence; S4. Determine the operator's eye movement state based on the fixation point sequence to update the fixation point coordinates; S5. Delineate the operable area from the airway image and generate control commands based on the operable area and the updated gaze point coordinates on the airway image. S6. The robot is controlled by an execution control module based on control commands.

[0053] In the embodiments of the eye-tracking-based bronchial interventional surgery robot control method described above in this application, the detailed operation of other components or mechanisms can be found in the description of the corresponding components or mechanisms in the eye-tracking-based bronchial interventional surgery robot control system, and will not be repeated here.

[0054] This application employs an eye tracker, intent determination module, region determination module, and execution control module to achieve real-time acquisition of the operator's gaze point and generation of control commands. This enables the operator to perform bronchoscopy in a non-contact manner, reducing the operator's workload and improving surgical efficiency. Simultaneously, by determining the stability of the eye movement signals, it effectively reduces operational errors and improves the reliability and safety of system control. The eye movement signals can express the doctor's gaze point position in real time, and are used as control inputs mapped to the robot's motion commands. This application can reduce the operator's workload, improve control response speed, and enhance the ability to quickly locate critical areas, thereby improving the safety and efficiency of surgical procedures, and has significant clinical value and practicality.

[0055] The following examples and comparative examples further illustrate this application to better understand the above-mentioned technical solutions. It should be understood that the following are merely examples and are not intended to limit this application. Example 1 provides a method for controlling a bronchial interventional surgical robot based on eye-tracking, aiming to achieve precise puncture and biopsy of lung lesions by combining intraoperative eye-tracking control techniques, such as... Figure 5 As shown, the specific steps are as follows: (1) System layout and initialization calibration The robot control system is deployed, and the bronchoscope structural components are installed. At the same time, spatial calibration is performed using calibration marks (preferably ArUco codes) set on the boundary of the display terminal to complete the eye tracker's gaze calibration, and the display area of ​​the display terminal is determined as the effective control area to establish the mapping relationship between the gaze point coordinates and the display area.

[0056] (2) Intraoperative image acquisition and display Real-time images of the airway are obtained through a bronchoscope and displayed on a display terminal, while the gaze point position or prompt information is superimposed.

[0057] (3) Eye-tracking data acquisition and preprocessing The system collects the operator's gaze coordinates in real time and filters the gaze coordinates, including: suppressing instantaneous noise through sliding window midpoint filtering and smoothing the data through exponential moving average filtering, thereby obtaining a stable and continuous gaze sequence.

[0058] (4) Gaze state recognition and intent determination The processed gaze point sequence is input into a pre-trained model (preferably an LSTM model) to classify eye movement states, yielding one of the following results: fixation, saccade, or smoothing. Anchor points are updated only when a fixation state is identified, and the gaze points are gradually converged towards the anchor points to form a stable control input.

[0059] (5) Determination of operating area Real-time semantic segmentation (preferably using the U-Net model) is performed on the intraoperative images, dividing them into operable regions (i.e., the area containing the hole) and prohibited regions (i.e., the background area). Control commands are only allowed to be generated when the gaze point is within the operable region; otherwise, control output is paused.

[0060] (6) Robot execution control The control command that meets the conditions is sent to robot 3, which drives the bronchoscope to perform corresponding operations, including forward, backward, rotation and front deflection, so as to approach the target lesion along the planned path, so as to facilitate tissue puncture, biopsy sampling and other operations.

[0061] (7) Instrument retraction and system reset After sampling is completed, control the bronchoscope to retract along the path and reset the system or replace instrument components.

[0062] The robot 3 in the above application example 1 can be implemented using a structure known in the art. For example, one can refer to a rapid replaceable bronchoscopic interventional surgical robot disclosed in Chinese Patent Publication No. CN121313315A. This bronchoscopic interventional surgical robot includes a base, a bronchoscope structure, a drive device, a first support part, and a second support part. The drive device is fixed on the base, one end of the bronchoscope structure is connected to the drive device, the first support part is connected to the bronchoscope structure, one end of the second support part is connected to the base, and the other end is connected to the first support part. The bronchoscope structure includes a bronchoscope, which is inserted into the first support part.

[0063] In the robot execution control process of step (6) of Example 1: In eye-tracking control mode, control commands that meet the conditions are sent to the drive device of robot 3. The drive device drives the bronchoscope to perform corresponding operations, including forward, backward, rotation and front deflection, so as to approach the target lesion along the planned path, which facilitates operations such as tissue puncture and biopsy sampling. In manual auxiliary control mode, the control signal of handle 4 is sent to the drive device of robot 3. The drive device drives the bronchoscope to perform corresponding operations, including forward, backward, rotation and front deflection, so as to approach the target lesion along the planned path.

[0064] In summary, through the above-described implementation methods, the robot control system of this application can achieve a complete closed-loop operation from system calibration, intraoperative perception, eye-tracking intention recognition to robot execution control. The robot drives the bronchoscope to perform forward, backward, rotation, and tip deflection operations according to control commands that meet the conditions, achieving precise navigation and positioning of the target lesion and completing tissue puncture and biopsy sampling. Compared to traditional teleoperation methods, this application introduces an eye-tracking-based non-contact human-computer interaction mechanism, allowing doctors to complete control input simply through visual attention, thereby significantly reducing operational complexity and learning costs. It also boasts advantages such as fast response speed, intuitive control, and high safety, making it suitable for the auxiliary diagnosis and precise intervention of complex lung lesions.

[0065] In the description of the embodiments of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0066] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0067] In the description of the embodiments in this application, "multiple" means two or more, unless otherwise explicitly specified. In this application, unless otherwise explicitly specified and limited, the terms "installed," "connected," "linked," "fixed," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0068] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0069] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0070] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A control system for a bronchial interventional surgical robot based on eye tracking, characterized in that, include: The data acquisition module collects the raw data of the operator's gaze coordinates; The calibration mapping module receives the raw gaze point coordinate data collected by the acquisition module and establishes a mapping relationship between the gaze point coordinates and the display terminal. The image display module displays the coordinates of the corresponding fixation point on the airway image of the patient's bronchus according to the mapping relationship established by the calibration mapping module; The signal processing module receives and processes the raw gaze point coordinate data acquired by the acquisition module to obtain a gaze point sequence; The intent determination module updates the gaze point coordinates based on the gaze point sequence obtained by the signal processing module. The region determination module divides the operable region from the airway image and generates control commands within the operable region using the gaze point coordinates updated by the intent determination module. The execution control module controls the robot according to the control instructions generated by the area determination module.

2. The control system for a bronchial interventional surgery robot based on eye tracking according to claim 1, characterized in that, The acquisition module is integrated into the eye tracker and includes an image acquisition submodule and a gaze estimation submodule, wherein: The image acquisition submodule acquires image information of the operator's eyes; The gaze estimation submodule receives and processes the eye image information acquired by the image acquisition submodule to generate raw data of gaze point coordinates.

3. The control system for a bronchial interventional surgery robot based on eye tracking according to claim 2, characterized in that, The calibration mapping module includes: Spatial calibration is performed based on the calibration marks placed at the boundary of the display terminal; The eye tracker's gaze is calibrated based on spatial calibration information, and the display area of ​​the display terminal is defined. A mapping relationship between the gaze point coordinates and the display area of ​​the display terminal is established based on the original data of the gaze point coordinates.

4. The control system for a bronchial interventional surgery robot based on eye tracking according to claim 1, characterized in that, The image display module uses a display terminal to display airway images of the patient's bronchi.

5. The control system for a bronchial interventional surgery robot based on eye tracking according to claim 1, characterized in that, The signal processing module is integrated into the eye tracker and includes: The original gaze point coordinate data is processed by sliding window midpoint filtering to obtain intermediate gaze point coordinate data after suppressing transient noise. The intermediate data of fixation point coordinates after sliding window midpoint filtering is smoothed by exponential moving average filtering, thereby obtaining a continuous fixation point sequence.

6. The control system for a bronchial interventional surgery robot based on eye tracking according to claim 1, characterized in that, The intent determination module inputs the gaze sequence into a pre-trained network model to identify the operator's eye movement state, which includes one of fixation, saccade, and smoothing. Specifically, when the intent determination module first identifies the eye movement state as a gaze state at a certain moment, it extracts a gaze point subsequence that matches that moment from the gaze point sequence, and calculates and generates an initial anchor point from the gaze point subsequence; thereafter, when the eye movement state is identified as a gaze state, it extracts the gaze point subsequence corresponding to that moment from the gaze point sequence to update the anchor point, and keeps the coordinates of the anchor point unchanged in the saccade state or smooth state; Control the current gaze point coordinates to converge to the current anchor point coordinates, and update the gaze point coordinates.

7. The control system for a bronchial interventional surgery robot based on eye tracking according to claim 1, characterized in that, The region determination module includes: The airway image is semantically segmented to divide the airway image into an operable region and a prohibited region. The operable region is the image area in which surgical operations are allowed to be performed during the operation, and the prohibited region is the image area in which surgical operations are not allowed to be performed during the operation. When the updated gaze point coordinates are located within the operable area, a control command is generated and issued; when the updated gaze point coordinates are located within the prohibited area, the output of the control command is paused.

8. The control system for a bronchial interventional surgery robot based on eye tracking according to claim 1, characterized in that, The control commands are derived based on the updated gaze point coordinates and the relative position of the target lesion in the airway image.

9. A bronchial interventional surgical robot control system based on eye tracking according to claim 1, characterized in that, The execution control module sends the received control commands to the robot, causing the bronchoscope structure in the robot to adjust its posture and feed depth, and approach the target lesion along its preset planned path.

10. A control method for a bronchial interventional surgical robot based on eye tracking, implemented based on the bronchial interventional surgical robot control system based on eye tracking as described in any one of claims 1-9, characterized in that, include: Establish a mapping relationship between the gaze point coordinates and the display terminal; The airway image of the patient's bronchus is displayed on the display terminal in real time. Based on the mapping relationship between the gaze point coordinates and the display terminal, the corresponding gaze point coordinates are displayed on the airway image. The operator's gaze point coordinates are collected in real time, and the raw gaze point coordinates are processed to obtain a gaze point sequence. Update the gaze point coordinates based on the gaze point sequence; The operable area is delineated from the airway image, and control commands are generated within the operable area using the updated gaze coordinates. The robot is controlled based on control commands.

Images