A mirror supporting robot control method and system based on a graphical user interface

Abstract

The application discloses a kind of based on graphical user interface mirror supporting robot control method and system, it is related to machine vision technical field;Method includes receiving endoscopic image, and the center position of endoscopic image is determined as the initial center position of surgical instrument;Endoscopic image is preprocessed to obtain target endoscopic image, then identification result is obtained by identification;According to identification result, the center position of focus is determined;In the process that surgical instrument moves from initial center position to the center position of focus, endoscopic instrument image is frame by frame collected and integrated to obtain endoscopic instrument image set;According to endoscopic instrument image set, endoscope is frame by frame fine-tuned to obtain rotation angle set;Rotation angle set, initial center position and the center position of focus, generate tracking instrument image set;Tracking instrument image set is transmitted to graphical user interface in real time.The application effectively improves image stability by automatic fine-tuning tracking of endoscope field of view.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, specifically relating to a control method and system for a mirror-adjusting robot based on a graphical user interface. Background Technology

[0002] With the rapid development of minimally invasive surgical techniques, laparoscopic surgery, thoracoscopic surgery, and robot-assisted surgery have become widely used in clinical practice. In minimally invasive surgery, surgical procedures are performed by capturing real-time images inside the patient's body using an endoscope, which are then transmitted via video cable to a monitor in the operating room. Currently, robotic endoscope control replaces manual endoscope manipulation, securing the endoscope in a preset position and adjusting its posture according to the surgeon's instructions. This improves the stability and durability of the field of vision. Combined with RCM (Remote Motion Center) constraints, torque sensing, and PID precise control, millimeter-level or even sub-millimeter-level positioning accuracy can be achieved. Simultaneously, it avoids accidental movement or excessive displacement that can occur with manual operation, significantly reducing the risk of accidental damage to the patient's incision tissue and internal organs. Current robotic endoscope control is based on a graphical user interface (GUI) touchscreen control. A graphical control interface is presented on a touchscreen display in the operating room, allowing the surgeon to control the endoscope's movement by tapping directional buttons, sliders, or virtual joysticks on the screen.

[0003] Furthermore, existing endoscope-guided robots require doctors to frequently manually adjust the endoscope via the GUI to observe the field of view. This makes it difficult for doctors to make fine adjustments to the field of view during surgical procedures. The endoscope-guided robot automatically tracks the field of view, which means that doctors cannot focus solely on the surgical procedure but need to operate and control the endoscope-guided robot to adjust the field of view, increasing surgical time and thus increasing surgical risks. Summary of the Invention

[0004] The purpose of this invention is to solve the problem that the endoscopic robot requires doctors to frequently manually adjust the GUI to observe the target field of view, making it difficult for the endoscopic robot to automatically track the field of view when doctors need to make fine adjustments and movements during surgical operations. Therefore, this invention proposes an endoscopic robot control method and system based on a graphical user interface.

[0005] In a first aspect of this invention, a control method for a mirror-adjusting robot based on a graphical user interface is first proposed, the method comprising: Receive endoscopic images captured by the endoscope and determine the center position of the endoscopic image as the initial center position of the surgical instruments; The endoscopic image is preprocessed to obtain the target endoscopic image; The target endoscopic image is identified to obtain the identification result; the identification result includes the location and size of the lesion; the center position of the lesion is determined based on the location and size of the lesion; During the process of the surgical instruments moving from the initial central position to the central position of the lesion, endoscopic instrument images are acquired frame by frame and integrated to obtain an endoscopic instrument image set; The rotation angle set is obtained by fine-tuning the endoscope frame by frame based on the image set of endoscopic instruments; The set of rotation angles, the initial center position, and the center position of the lesion are used to generate a tracking device image set; The tracking device image set is transmitted to the graphical user interface in real time.

[0006] Optionally, preprocessing the endoscopic image to obtain the target endoscopic image includes: The endoscopic images were processed by 3×3 convolution, 3×3 convolution, and 3×3 convolution respectively to obtain the first convolution endoscopic feature map, the second convolution endoscopic feature map, and the third convolution endoscopic feature map respectively. The first convolutional endoscopic feature map, the second convolutional endoscopic feature map, and the third convolutional endoscopic feature map are dilated in parallel with dilation rates of 1, 2, and 3 to obtain the first dilated feature map, the second dilated feature map, and the third dilated feature map; The first, second, and third dilated feature maps are concatenated to obtain the initial endoscopic feature map; The target endoscopic image is obtained by performing a 1×1 convolution on the initial endoscopic feature map.

[0007] This preprocessing operation can simultaneously capture multi-scale features from micro to macro. The small expansion rate branch accurately identifies fine targets such as sutures and microvessels, while the large receptive field branch perceives the overall layout of organs, effectively solving the problem of huge differences in target scale during surgery.

[0008] Optionally, the identification of the target endoscopic image to obtain the identification result includes: The target endoscopic image is continuously downsampled to obtain a first downsampled endoscopic image, a second downsampled endoscopic image, and a third downsampled endoscopic image; Multi-scale feature fusion was performed on the third downsampled endoscopic image to obtain the first lesion feature map; After upsampling the first lesion feature map, it is aggregated with the second downsampled endoscopic image to obtain the second lesion feature map. After upsampling the second lesion feature map, it is aggregated with the third downsampled endoscopic image to obtain the third lesion feature map. The first lesion feature map, the second lesion feature map, and the third lesion feature map are respectively matched to obtain the corresponding target lesion features; The identification results are obtained by querying and comparing the characteristics of the target lesion with the database.

[0009] This feature aggregation mechanism overcomes the limitation of deep features lacking spatial detail, enabling the final feature maps at all levels to possess both strong discriminative power and precise localization capabilities. It effectively addresses challenges such as blurred edges and low contrast with normal tissue in small, early lesions, significantly reducing the false negative rate. Simultaneously, through multi-level feature matching and database comparison, the system can quickly eliminate false positives, significantly improving the accuracy and recall of lesion detection, laying a reliable foundation for subsequent precise tracking.

[0010] Optionally, the specific process of lesion feature aggregation includes: Obtain the first input lesion features and the second input lesion features; Global and local feature extractions are performed on the first and second input lesion features respectively to obtain global and local lesion features. The global lesion features and local lesion features are spliced ​​together to obtain the fused lesion features; Channel attention operations and spatial attention operations are performed on the fused lesion features to obtain channel features and spatial features, respectively; The input lesion features are obtained by adding the channel features and spatial features element by element.

[0011] This aggregation strategy overcomes the limitations of single feature extraction, achieving deep complementarity between macroscopic semantics and microscopic texture. The parallel introduction of a dual attention mechanism enables the network to adaptively select the most discriminative feature channels while accurately pinpointing the spatial location of lesions, significantly enhancing the robustness of features to changes in illumination. The final output fused features exhibit extremely high feature matching stability, significantly improving the anti-interference capability during lesion identification and tracking.

[0012] Optionally, determining the central location of the lesion based on its location and size includes: Extract the top-left pixel coordinates of the bounding box in the lesion location, and extract the width and height of the bounding box in the lesion size; The top-left pixel coordinates, width, and height of the bounding box are determined as the initial center position of the lesion; If the identification result contains multiple discontinuous lesion areas, the area with the largest area is selected as the main lesion area, and the center position of the bounding box of the main lesion area is taken as the center position of the lesion.

[0013] It simplifies decision-making complexity in multi-lesion scenarios, ensuring that the endoscopic field of view is always anchored to the most important and threatening suspected lesion, avoiding frequent shifts or tracking failures due to interference from multiple small areas. The geometric center calculation method requires no iterative optimization, has a very small computational load, and perfectly meets the requirements of millisecond-level real-time response during surgery, providing a stable coordinate reference for subsequent instrument tip offset vector calculation.

[0014] Optionally, the step of fine-tuning the endoscope frame by frame to obtain the rotation angle set based on the endoscopic instrument image set includes: Extract the current frame image and historical frame images from the endoscopic instrument image set; Perform key feature point detection on the current frame image and the historical frame image respectively to obtain the coordinates of the end of the device in the current frame and the coordinates of the end of the device in the historical frame; The sequence of discrete points of device end motion is obtained by integrating the device end coordinates of the current frame and the device end coordinates of the historical frames. The motion weight coefficient value of the device end is calculated based on the discrete point sequence of the device end motion. If the motion weight coefficient value of the instrument tip is less than or equal to the preset motion weight coefficient value, then the coordinates of the instrument tip in the current frame are extracted and determined as the first offset direction vector along with the center position of the lesion; the instrument tip offset vector of the next frame is predicted based on the sequence of discrete points of the instrument tip motion and the first offset direction vector; the rotation angle of the endoscope in the current frame is generated by integrating the first offset direction vector, the instrument tip offset vector of the next frame and the motion weight coefficient value of the instrument tip. If the motion weight coefficient value at the end of the device is greater than the preset motion weight coefficient value, a braking signal is issued and braking is performed by the back end. The rotation angles of all frames are obtained by analogy and denoted as the rotation angle set.

[0015] When the instrument moves smoothly, the advanced prediction mechanism eliminates the inherent delay of traditional feedback control, enabling the endoscope to predict the instrument's trajectory and achieve lag-free, smooth tracking. In the event of sudden and severe shaking of the instrument, the braking mode is switched immediately. This flexible switching strategy ensures both extreme smoothness under normal operation and absolute safety under critical conditions.

[0016] Optionally, the step of calculating the device end-effector motion weight coefficient value based on the discrete point sequence of the device end-effector motion includes: The velocity sequence and acceleration sequence are obtained by performing first-order and second-order difference calculations on the discrete point sequence of the device's end effector. The motion state parameters of the instrument's end effector are calculated by integrating the velocity sequence and acceleration sequence; the motion state parameters of the instrument's end effector include the variance of the instrument's end effector velocity and the range of the instrument's end effector acceleration. The motion weighting coefficient of the instrument's end effector is calculated by weighted summation of the variance of the instrument's end effector velocity and the range of the instrument's end effector acceleration.

[0017] The velocity variance reflects the sustained stability of the doctor's handpiece operation, while the acceleration range sensitively detects minute hand tremors or sudden impacts. Through weighted fusion, a single coefficient can comprehensively assess the risk level of the current movement. This provides objective, precise, and quantitative mathematical criteria for subsequent fine-tuning and braking decisions, making the setting of trigger thresholds for the control strategy scientific, adjustable, and highly reliable.

[0018] Optionally, the braking process at the back end includes: setting the rotation angle of the endoscope in the current frame to zero, and limiting the maximum angular velocity in subsequent frames to not exceed the angular velocity threshold.

[0019] This dual mechanism of emergency stop and gradual resuscitation decisively cuts off the erroneous momentum that could lead to instrument impact on tissues, while preventing secondary loss of control due to system oscillation or inertial overshoot after the emergency stop by limiting angular velocity. This approach effectively absorbs the sudden impact energy, ensuring the absolute physical safety of the patient's incision and internal organs, while providing valuable buffer time for doctors or back-end systems to intervene and assess the situation. This significantly improves the fault tolerance and robustness of the robotic system under extreme conditions.

[0020] In a second aspect of this invention, a control system for a mirror-adjusting robot based on a graphical user interface is provided, comprising: Data acquisition module: Receives endoscopic images captured by the endoscope and determines the center position of the endoscopic image as the initial center position of the surgical instruments; Preprocessing module: preprocesses the endoscopic image to obtain the target endoscopic image; Lesion identification module: identifies the target endoscopic image to obtain identification results; the identification results include the location and size of the lesion; and determines the center position of the lesion based on the location and size of the lesion. Endoscopic instrument image acquisition module: During the process of the surgical instruments moving from the initial center position to the center position of the lesion, endoscopic instrument images are acquired frame by frame and integrated to obtain an endoscopic instrument image set; Endoscope fine-tuning module: Based on the image set of endoscopic instruments, the endoscope is fine-tuned frame by frame to obtain the rotation angle set; Tracking device module: Generates a tracking device image set from the set of rotation angles, the initial center position, and the center position of the lesion; Visualization interface module: Transmits the image set of the tracking device to the graphical user interface in real time.

[0021] The beneficial effects of this invention are as follows: First, it eliminates the constraints of manual endoscope adjustment, allowing doctors to focus entirely on the surgical procedure and avoiding disruptions to the surgical rhythm and increased cognitive burden caused by frequently switching attention to GUI control. Second, by finely adjusting the rotation angle frame by frame, the endoscopic field of view can smoothly and continuously lock onto the moving instrument tip, effectively eliminating the delay and overshoot phenomena present in manual adjustment, ensuring that the center of the field of view is always aligned with the surgical operation area, and significantly improving image stability and observation continuity. Third, the entire tracking process is completed autonomously by the system, with a response speed far exceeding that of manual operation. Especially in high-frequency instrument operation scenarios, it can adapt to changes in the field of view in real time, thereby effectively shortening the total operation time, reducing the risk of tissue damage caused by improper field of view adjustment, and comprehensively improving the efficiency and safety of minimally invasive surgery. Attached Figure Description

[0022] The invention will now be further described with reference to the accompanying drawings.

[0023] Figure 1 A flowchart illustrating a control method for a mirror-adjusting robot based on a graphical user interface, provided in an embodiment of the present invention; Figure 2 A framework diagram of a mirror-adjusting robot control method based on a graphical user interface provided in an embodiment of the present invention; Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] It should be noted that all formula calculations in the scheme are purely numerical calculations.

[0026] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] This invention provides a control method for a mirror-adjusting robot based on a graphical user interface. See also... Figure 1 The method includes the following steps: S101: Receives endoscopic images captured by the endoscope through the endoscope by the endoscope-guided robot, and determines the center position of the endoscopic image as the initial center position of the surgical instruments; S102: Preprocess the endoscopic image to obtain the target endoscopic image; S103: Identify the target endoscopic image to obtain identification results; the identification results include the location and size of the lesion; determine the center location of the lesion based on its location and size; S104: During the process of the surgical instruments moving from the initial central position to the central position of the lesion, endoscopic instrument images are acquired frame by frame and integrated to obtain an endoscopic instrument image set; S105: Fine-tune the endoscope frame by frame based on the endoscopic instrument image set to obtain the rotation angle set; S106: Generate a tracking device image set from the rotation angle set, the initial center position, and the center position of the lesion; S107: Transmits the tracking device image set to the graphical user interface in real time.

[0028] In one embodiment, preprocessing an endoscopic image to obtain a target endoscopic image includes: The endoscopic images were processed by 3×3 convolution, 3×3 convolution, and 3×3 convolution respectively to obtain the first convolution endoscopic feature map, the second convolution endoscopic feature map, and the third convolution endoscopic feature map respectively. The first convolutional endoscopic feature map, the second convolutional endoscopic feature map, and the third convolutional endoscopic feature map are dilated in parallel with dilation rates of 1, 2, and 3 to obtain the first dilated feature map, the second dilated feature map, and the third dilated feature map; The first, second, and third dilated feature maps are concatenated to obtain the initial endoscopic feature map; The target endoscopic image is obtained by performing a 1×1 convolution on the initial endoscopic feature map.

[0029] In one implementation, the preprocessing operation can simultaneously capture multi-scale features from micro to macro. The small dilation rate branch accurately identifies fine targets such as sutures and microvessels, while the large receptive field branch perceives the overall organ layout, effectively solving the problem of huge differences in target scale during surgery. To address the challenge of scale differences in endoscopic images, where fine structures such as sutures and microvessels coexist with macroscopic information such as the overall organ layout, parallel dilated convolutions with dilation rates of 1, 2, and 3 are used in three branches to capture features from different receptive fields. The small dilation rate branch preserves high-resolution details and can accurately identify millimeter-level sutures and microvessels, while the large dilation rate branch has global perception capabilities and can understand organ boundaries and spatial relationships. This effectively solves the problem of huge spans in target scale from sub-millimeter to several centimeters during surgery, making subsequent feature extraction no longer affected by scale and significantly improving the accuracy and robustness of lesion identification in complex tissue backgrounds.

[0030] In one embodiment, the identification result obtained by identifying the target endoscopic image includes: The target endoscopic image is continuously downsampled to obtain a first downsampled endoscopic image, a second downsampled endoscopic image, and a third downsampled endoscopic image; Multi-scale feature fusion was performed on the third downsampled endoscopic image to obtain the first lesion feature map; After upsampling the first lesion feature map, it is aggregated with the second downsampled endoscopic image to obtain the second lesion feature map. After upsampling the second lesion feature map, it is aggregated with the third downsampled endoscopic image to obtain the third lesion feature map. The first lesion feature map, the second lesion feature map, and the third lesion feature map are respectively matched to obtain the corresponding target lesion features; The identification results are obtained by querying and comparing the characteristics of the target lesion with the database.

[0031] Specifically, multi-scale feature fusion is an operation implemented based on the SPPF module in the YOLOv8 detection model.

[0032] In one implementation, addressing the clinical challenge of early lesions often having blurred edges and extremely low grayscale contrast with normal tissue, a multi-level feature pyramid is constructed through continuous downsampling. Then, deep semantic information and shallow spatial details are aggregated level by level from top to bottom, enabling each level of feature map to possess both strong discriminative power and precise localization capabilities. This effectively solves the problem of inaccurate localization of small lesions due to low spatial resolution in deep networks, significantly reducing the missed detection rate of early-stage micro-tumors. Simultaneously, through a dual verification mechanism of multi-scale feature matching and database comparison, interference from falsely detected normal tissue folds, reflective spots, etc., can be quickly eliminated, significantly improving the accuracy and recall rate of lesion detection. This provides a reliable target basis for subsequent precise endoscopic tracking, directly impacting the timely detection and treatment of lesions during surgery.

[0033] In one embodiment, the specific process of lesion feature aggregation includes: Obtain the first input lesion features and the second input lesion features; Global and local feature extractions are performed on the first and second input lesion features respectively to obtain global and local lesion features. The global lesion features and local lesion features are spliced ​​together to obtain the fused lesion features; Channel features and spatial features are obtained by performing channel attention operations and spatial attention operations on the features of the fused lesions, respectively. The input lesion features are obtained by adding the channel features and spatial features element by element.

[0034] In one implementation, to address the practical challenge of drastic changes in image appearance caused by variable intraoperative lighting conditions and frequent endoscopic angle switching, a strategy of parallel extraction and fusion of global and local features is designed, and a dual mechanism of channel attention and spatial attention is introduced to enhance key information. Global features capture the morphological semantics of lesions, while local features preserve edge texture details. The fused features, which complement each other, have a strong anti-interference ability against changes in lighting, tissue occlusion, and viewpoint rotation. Channel attention allows the network to automatically focus on the feature dimensions that best distinguish lesions from normal tissue, while spatial attention accurately locks the pixel region where the lesion is located. The collaboration of the two makes the output aggregated features highly stable in subsequent matching and comparison, and can maintain reliable recognition performance even under adverse conditions such as tissue bleeding and instrument shadow interference, significantly improving the clinical applicability of the system.

[0035] In one embodiment, determining the central location of the lesion based on its location and size includes: Extract the top-left pixel coordinates of the bounding box in the lesion location, and extract the width and height of the bounding box in the lesion size; The top-left pixel coordinates, width, and height of the bounding box are used to determine the initial center position of the lesion; If the identification results contain multiple discontinuous lesion areas, the area with the largest area is selected as the main lesion area, and the center position of the bounding box of the main lesion area is taken as the center position of the lesion.

[0036] In one implementation, to address the real-world scenario where multiple discontinuous regions may exist within the endoscopic field of view, the coordinates, width, and height of the upper left corner of the bounding box are extracted, and the geometric center is calculated. If multiple regions exist, the one with the largest area is selected as the primary target. This strategy simplifies the decision-making complexity under multiple coexisting lesions, ensuring that the endoscope-guided robot always anchors the center of the field of view to the primary lesion with the largest area and the highest clinical threat, avoiding frequent swaying or tracking failure of the field of view due to interference from multiple small regions. At the same time, the geometric center calculation does not require iterative optimization, and the processing time per frame is in the microsecond range, meeting the millisecond-level real-time response requirements during surgery, and providing a stable, low-latency coordinate reference for the subsequent calculation of the instrument end offset vector.

[0037] In one implementation, the calculation steps for the initial center position of the lesion are as follows: the horizontal coordinate of the center is Cx = x + w / 2, and the vertical coordinate of the center is Cy = y + h / 2; (x, y) is the pixel coordinate of the upper left corner of the bounding box in the lesion position, (Cx, Cy) is the initial center position of the lesion, and the width and height of the bounding box in the size of the lesion are denoted as w and h, respectively. In one embodiment, fine-tuning the endoscope frame by frame based on the endoscopic instrument image set to obtain the rotation angle set includes: Extract the current frame image and historical frame images from the endoscopic instrument image set; Perform key feature point detection on the current frame image and the historical frame image respectively to obtain the coordinates of the end of the device in the current frame and the coordinates of the end of the device in the historical frame. The sequence of discrete points of device end motion is obtained by integrating the device end coordinates of the current frame and the device end coordinates of the historical frames. The motion weight coefficient of the device end is calculated based on the discrete point sequence of the device end motion. If the motion weight coefficient value of the instrument tip is less than or equal to the preset motion weight coefficient value, then the coordinates of the instrument tip in the current frame are extracted and determined as the first offset direction vector along with the center position of the lesion; the instrument tip offset vector of the next frame is predicted based on the sequence of discrete points of the instrument tip motion and the first offset direction vector; the rotation angle of the endoscope in the current frame is generated by integrating the first offset direction vector, the instrument tip offset vector of the next frame and the motion weight coefficient value of the instrument tip. If the motion weight coefficient value at the end of the device is greater than the preset motion weight coefficient value, a braking signal is issued and braking is performed by the back end. The rotation angles of all frames are obtained by analogy and denoted as the rotation angle set.

[0038] In one implementation, firstly, the direction vector of the instrument tip pointing to the lesion center in the current frame is obtained, denoted as V_dir; the offset vector of the instrument tip in the next frame, predicted based on the historical motion sequence, is denoted as V_pred; and the calculated motion weight coefficient of the instrument tip is denoted as λ (the value ranges from 0 to 1, with smaller values ​​indicating smoother motion). Then, the dynamic fusion coefficient is calculated as α = α_base × (1-λ), where α_base is a preset base prediction weight (e.g., 0.7). When the motion is smooth, α is larger, causing the prediction component to dominate to achieve advanced tracking; when jitter intensifies, α automatically decreases to weaken the prediction influence. Then, the two directions are... The quantities are weighted and fused to obtain V_fusion=(1-α)×V_dir+α×V_pred; then the fused vector is mapped to the current frame rotation angle θ_t=θ_last+β×arctan(|V_fusion| / f)×sign(V_fusion), where θ_last is the angle of the previous frame to ensure continuity, β is the gain coefficient, f is the camera focal length, and the arctangent function realizes smooth control of large offset fast rotation and small offset fine adjustment; finally, if λ exceeds the preset safety threshold (such as 0.9), braking is triggered, the above calculation is skipped, the rotation angle is set to zero and the subsequent angular velocity is limited to ensure safety.

[0039] In one implementation, during the movement of the instrument from its initial position toward the lesion, the stability of the instrument's movement is assessed in real time, and a decision is made based on this assessment to adopt predictive tracking or emergency braking. When the instrument's end-effector motion weight coefficient value is less than or equal to a preset motion weight coefficient value, the position of the instrument's end-effector in the next frame is predicted in advance by combining the current offset direction and historical motion sequence, and the endoscope rotation angle is generated to achieve predictive, lag-free tracking. When the instrument's end-effector motion weight coefficient value is greater than the preset motion weight coefficient value, the end-effector braking is triggered immediately. This intelligent switching mechanism eliminates the inherent delay of traditional feedback control during normal operation by the doctor, and the endoscope field of view and the instrument's end-effector are always synchronized. The doctor does not need to be distracted by adjusting the GUI interface and can focus on delicate operations such as suturing and dissection throughout the procedure. In the event of sudden shaking, rapid impact, or loss of control of the instrument, it instantly switches to a safety braking mode, effectively preventing mechanical damage to organs caused by accidental endoscope movement, ensuring both the smoothness of routine surgery and physical safety in critical situations.

[0040] Specifically, the preset motion weight coefficient value is obtained based on the staff's historical experience.

[0041] In one embodiment, the calculation of the device end-effector motion weight coefficient value based on the discrete point sequence of device end-effector motion includes: The velocity sequence and acceleration sequence are obtained by performing first-order and second-order difference calculations on the discrete point sequence of the device's end effector. The motion state parameters of the instrument's end effector are obtained by integrating the velocity sequence and acceleration sequence; the motion state parameters of the instrument's end effector include the variance of the instrument's end effector velocity and the range of the instrument's end effector acceleration. The motion weighting coefficient of the instrument's end effector is calculated by weighted summation of the variance of the instrument's end effector velocity and the range of the instrument's end effector acceleration.

[0042] In one implementation, the discrete point sequence of the device's end effector motion is subjected to first-order and second-order differences to extract velocity and acceleration sequences, respectively. Then, the variance of velocity and the range of acceleration are calculated, and the two are weighted and summed to obtain a quantified motion weight coefficient. The velocity variance statistically characterizes the continuous smoothness of the doctor's operation of the handle; the smaller the value, the smoother and more uniform the operation. The acceleration range is extremely sensitive to instantaneous impacts and high-frequency hand tremors, and can promptly capture violent changes in speed caused by muscle fatigue or sudden collisions. The single coefficient formed by the weighted fusion of the two comprehensively assesses the risk level of the current motion from both the dimensions of smoothness and suddenness.

[0043] In one embodiment, the backend braking process includes: setting the rotation angle of the endoscope in the current frame to zero and limiting the maximum angular velocity in subsequent frames to not exceed the angular velocity threshold.

[0044] In one embodiment, the implementation process of S106 is as follows: based on the calculated rotation angle of each frame of the endoscope, combined with the initial center position and the lesion center position, the original acquired endoscopic instrument image is subjected to frame-by-frame geometric transformation (rotation + translation) to generate a set of tracking image sequences in which the center of the field of view is always aligned with the end of the surgical instrument, and the lesion is located in a reasonable position within the field of view. Finally, the image is output to the graphical user interface (GUI) for real-time display.

[0045] This invention provides a control system for a mirror-adjusting robot based on a graphical user interface. See also... Figure 2 The system includes the following modules: Data acquisition module (201): Receives endoscopic images captured by the endoscope through the endoscope by the endoscope-guided robot, and determines the center position of the endoscopic image as the initial center position of the surgical instruments; Preprocessing module (202): preprocesses the endoscopic image to obtain the target endoscopic image; Lesion identification module (203): Identifies the target endoscopic image to obtain identification results; the identification results include the location and size of the lesion; the center position of the lesion is determined based on the location and size of the lesion; Endoscopic instrument image acquisition module (204): During the process of the surgical instrument moving from the initial central position to the central position of the lesion, endoscopic instrument images are acquired frame by frame and integrated to obtain an endoscopic instrument image set; Endoscope fine-tuning module (205): Fine-tunes the endoscope frame by frame based on the endoscopic instrument image set to obtain the rotation angle set; Tracking device module (206): Generates a tracking device image set from the rotation angle set, the initial center position, and the center position of the lesion; Visualization interface module (207): Transmits the tracking device image set to the graphical user interface in real time.

[0046] The foregoing has described one embodiment of the present invention in detail, but this content is merely a preferred embodiment and should not be considered as limiting the scope of the present invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the scope of the claims of this invention.

Claims

1. A control method for a mirror-adjusting robot based on a graphical user interface, characterized in that, The method includes: Receive endoscopic images captured by the endoscope and determine the center position of the endoscopic image as the initial center position of the surgical instruments; The endoscopic image is preprocessed to obtain the target endoscopic image; The target endoscopic image is identified to obtain the identification result; the identification result includes the location and size of the lesion; the center position of the lesion is determined based on the location and size of the lesion; During the process of the surgical instruments moving from the initial central position to the central position of the lesion, endoscopic instrument images are acquired frame by frame and integrated to obtain an endoscopic instrument image set; The rotation angle set is obtained by fine-tuning the endoscope frame by frame based on the image set of endoscopic instruments; The set of rotation angles, the initial center position, and the center position of the lesion are used to generate a tracking device image set; The tracking device image set is transmitted to the graphical user interface in real time.

2. The method for controlling a mirror-adjusting robot based on a graphical user interface according to claim 1, characterized in that, Preprocessing the endoscopic image to obtain the target endoscopic image includes: The endoscopic images were processed by 3×3 convolution, 3×3 convolution, and 3×3 convolution respectively to obtain the first convolution endoscopic feature map, the second convolution endoscopic feature map, and the third convolution endoscopic feature map respectively. The first convolutional endoscopic feature map, the second convolutional endoscopic feature map, and the third convolutional endoscopic feature map are dilated in parallel with dilation rates of 1, 2, and 3 to obtain the first dilated feature map, the second dilated feature map, and the third dilated feature map; The first, second, and third dilated feature maps are concatenated to obtain the initial endoscopic feature map; The target endoscopic image is obtained by performing a 1×1 convolution on the initial endoscopic feature map.

3. The method for controlling a mirror-adjusting robot based on a graphical user interface according to claim 1, characterized in that, The identification results obtained by identifying the target endoscopic image include: The target endoscopic image is continuously downsampled to obtain a first downsampled endoscopic image, a second downsampled endoscopic image, and a third downsampled endoscopic image; Multi-scale feature fusion was performed on the third downsampled endoscopic image to obtain the first lesion feature map; After upsampling the first lesion feature map, it is aggregated with the second downsampled endoscopic image to obtain the second lesion feature map. After upsampling the second lesion feature map, it is aggregated with the third downsampled endoscopic image to obtain the third lesion feature map. The first lesion feature map, the second lesion feature map, and the third lesion feature map are respectively matched to obtain the corresponding target lesion features; The identification results are obtained by querying and comparing the characteristics of the target lesion with the database.

4. The method for controlling a mirror-adjusting robot based on a graphical user interface according to claim 3, characterized in that, The specific process of lesion feature aggregation includes: Obtain the first input lesion features and the second input lesion features; Global and local feature extractions are performed on the first and second input lesion features respectively to obtain global and local lesion features. The global lesion features and local lesion features are spliced ​​together to obtain the fused lesion features; Channel attention operations and spatial attention operations are performed on the fused lesion features to obtain channel features and spatial features, respectively; The input lesion features are obtained by adding the channel features and spatial features element by element.

5. The method for controlling a mirror-adjusting robot based on a graphical user interface according to claim 1, characterized in that, Determining the center location of the lesion based on its location and size includes: Extract the top-left pixel coordinates of the bounding box in the lesion location, and extract the width and height of the bounding box in the lesion size; The top-left pixel coordinates, width, and height of the bounding box are determined as the initial center position of the lesion; If the identification result contains multiple discontinuous lesion areas, the area with the largest area is selected as the main lesion area, and the center position of the bounding box of the main lesion area is taken as the center position of the lesion.

6. The method for controlling a mirror-adjusting robot based on a graphical user interface according to claim 1, characterized in that, The step of fine-tuning the endoscope frame by frame based on the endoscopic instrument image set to obtain the rotation angle set includes: Extract the current frame image and historical frame images from the endoscopic instrument image set; Perform key feature point detection on the current frame image and the historical frame image respectively to obtain the coordinates of the end of the device in the current frame and the coordinates of the end of the device in the historical frame; The sequence of discrete points of device end motion is obtained by integrating the device end coordinates of the current frame and the device end coordinates of the historical frames. The motion weight coefficient value of the device end is calculated based on the discrete point sequence of the device end motion. If the motion weight coefficient value of the instrument tip is less than or equal to the preset motion weight coefficient value, then the coordinates of the instrument tip in the current frame are extracted and determined as the first offset direction vector along with the center position of the lesion; the instrument tip offset vector of the next frame is predicted based on the sequence of discrete points of the instrument tip motion and the first offset direction vector; the rotation angle of the endoscope in the current frame is generated by integrating the first offset direction vector, the instrument tip offset vector of the next frame and the motion weight coefficient value of the instrument tip. If the motion weight coefficient value at the end of the device is greater than the preset motion weight coefficient value, a braking signal is issued and braking is performed by the back end. The rotation angles of all frames are obtained by analogy and denoted as the rotation angle set.

7. The method for controlling a mirror-adjusting robot based on a graphical user interface according to claim 6, characterized in that, The calculation of the device end-effector motion weight coefficient value based on the discrete point sequence of the device end-effector motion includes: The velocity sequence and acceleration sequence are obtained by performing first-order and second-order difference calculations on the discrete point sequence of the device's end effector. The motion state parameters of the instrument's end effector are calculated by integrating the velocity sequence and acceleration sequence; the motion state parameters of the instrument's end effector include the variance of the instrument's end effector velocity and the range of the instrument's end effector acceleration. The motion weighting coefficient of the instrument's end effector is calculated by weighted summation of the variance of the instrument's end effector velocity and the range of the instrument's end effector acceleration.

8. The method for controlling a mirror-adjusting robot based on a graphical user interface according to claim 6, characterized in that, The braking process at the back end includes: setting the rotation angle of the endoscope in the current frame to zero, and limiting the maximum angular velocity in subsequent frames to not exceed the angular velocity threshold.

9. A control system for a mirror-adjusting robot based on a graphical user interface, used to implement the control method for a mirror-adjusting robot based on a graphical user interface as described in any one of claims 1-8, characterized in that, The system includes: Data acquisition module: Receives endoscopic images captured by the endoscope and determines the center position of the endoscopic image as the initial center position of the surgical instruments; Preprocessing module: preprocesses the endoscopic image to obtain the target endoscopic image; Lesion identification module: identifies the target endoscopic image to obtain identification results; the identification results include the location and size of the lesion; and determines the center position of the lesion based on the location and size of the lesion. Endoscopic instrument image acquisition module: During the process of the surgical instruments moving from the initial center position to the center position of the lesion, endoscopic instrument images are acquired frame by frame and integrated to obtain an endoscopic instrument image set; Endoscope fine-tuning module: Based on the image set of endoscopic instruments, the endoscope is fine-tuned frame by frame to obtain the rotation angle set; Tracking device module: Generates a tracking device image set from the set of rotation angles, the initial center position, and the center position of the lesion; Visualization interface module: Transmits the image set of the tracking device to the graphical user interface in real time.

Images