Surgical robot assisted control method, apparatus, storage medium and program product

Abstract

The application provides a surgical robot auxiliary control method, device, storage medium and program product, and relates to the technical field of surgical robots. The method comprises the following steps: in response to a trigger measurement, obtaining starting position information of a joint of a surgical robot; during the measurement, controlling a surgical instrument end operating point to move according to a movement control instruction for the surgical instrument end operating point, and obtaining real-time position information of the joint; according to the starting position information and the real-time position information, obtaining a measurement result of the displacement of the surgical instrument end operating point based on a robot kinematics model; and outputting the measurement result, which is used for auxiliary control of the surgical robot. The application can provide intuitive and accurate auxiliary information for robot surgery.

Description

Technical Field

[0001] This application relates to the field of surgical robot technology, and in particular to a surgical robot-assisted control method, device, storage medium, and program product. Background Technology

[0002] With the widespread adoption of surgical robot technology, minimally invasive surgery can now achieve a high degree of precision. In these surgeries, surgeons control the robotic arms and surgical instruments that enter the patient's body via a master controller. This master-slave control mode significantly improves the flexibility and stability of the procedure.

[0003] In minimally invasive surgery, accurate assessment of lesion size and precise control of instrument depth are crucial. During the procedure, surgeons need to understand the size of the lesion, the length of blood vessels, and the depth of the instrument tip relative to the tissue to ensure accurate resection without damaging deeper tissues. However, while endoscopic images are clear, they often fail to directly reflect the actual physical dimensions and spatial depth. In the two-dimensional or pseudo-three-dimensional images displayed on the screen, it is difficult for surgeons to directly determine the actual distance between two anatomical points or accurately perceive the depth of the surgical instrument. This lack of spatial awareness forces surgeons to rely on experience-based estimations during delicate procedures.

[0004] Therefore, there is an urgent need for a surgical robot-assisted control solution that can provide doctors with intuitive dimensions. Summary of the Invention

[0005] This application provides surgical robot assisted control methods, devices, storage media, and program products to achieve the effect of providing intuitive and accurate auxiliary information for robotic surgery.

[0006] In a first aspect, embodiments of this application provide a surgical robot-assisted control method, including:

[0007] The system responds to trigger measurements to obtain the initial position information of the joints of the surgical robot.

[0008] During the measurement process, the movement of the surgical instrument end-effector is controlled according to the movement control command for the end-effector, and the real-time position information of the joint is obtained.

[0009] Based on the initial position information and real-time position information, and using the robot kinematics model, the displacement of the surgical instrument end effector is measured.

[0010] The measurement results are output and used to assist in the control of the surgical robot.

[0011] In one possible implementation, the output measurement results include:

[0012] Based on augmented reality technology, the measurement results are fused and displayed in real-time endoscopic images.

[0013] In one possible implementation, the measurement results include the depth displacement component of the surgical instrument tip operating point along the optical axis of the endoscope. Based on augmented reality technology, the measurement results are fused and displayed in a real-time endoscopic image, including:

[0014] If the depth displacement component is less than the depth determination threshold, the measurement type of the measurement result is determined to be surface measurement; if the depth displacement component is greater than or equal to the depth determination threshold, the measurement type of the measurement result is determined to be depth measurement.

[0015] Based on the output format corresponding to the measurement type, the measurement results are fused and displayed in real-time endoscopic images using augmented reality technology.

[0016] In one possible implementation, based on the output format corresponding to the measurement type, the measurement results are fused and displayed in the real-time endoscopic image using augmented reality technology, including:

[0017] Based on the robot-endoscope hand-eye transformation matrix, the three-dimensional physical coordinates corresponding to the measurement results are converted into two-dimensional pixel coordinates corresponding to the endoscope image;

[0018] Based on the measurement results and two-dimensional pixel coordinates, measurement guidance information is generated, which includes the measurement trajectory and measurement data;

[0019] Based on augmented reality technology, an image rendering engine is used to overlay measurement guidance information onto real-time endoscopic images according to the output format, so as to fuse and display the measurement results in the real-time endoscopic images.

[0020] In one possible implementation, the measurement results include the depth displacement component of the surgical instrument tip operating point along the optical axis of the endoscope, and the surgical robot-assisted control method further includes:

[0021] If the detected depth displacement component exceeds the safe depth displacement threshold, a depth risk warning is triggered. The depth risk warning is used to alert the surgical robot operator that there is an operational risk.

[0022] In one possible implementation, the initial position information is the initial joint angle vector of the surgical robot, the real-time position information is the real-time joint angle vector of the surgical robot, and the robot kinematic model includes a positive kinematic model.

[0023] Based on the initial position information and real-time position information, and using the robot's kinematic model, the measurement results of the displacement of the surgical instrument's end effector point are obtained, including:

[0024] Based on the initial joint angle vector and the real-time joint angle vector, and using the forward kinematics model, the three-dimensional physical coordinates of the surgical instrument end-effector manipulation point in the world coordinate system are calculated.

[0025] The displacement of the surgical instrument end-effector's operating point is measured based on the three-dimensional physical coordinates.

[0026] In one possible implementation, the measurement is triggered in the following manner:

[0027] When the surgical robot is in active remote control mode, monitor the status signal of the end effector of the master hand;

[0028] If the status signal indicates that the measurement has started, determine whether to trigger the measurement.

[0029] Secondly, embodiments of this application provide a surgical robot-assisted control device, comprising:

[0030] The acquisition module is used to respond to triggered measurements and acquire the initial position information of the joints of the surgical robot;

[0031] The control module is used to control the movement of the surgical instrument end effector point according to the movement control command for the end effector point during the measurement process, and to obtain the real-time position information of the joint.

[0032] The processing module is used to obtain the measurement results of the displacement of the surgical instrument end effector point based on the robot kinematic model, according to the initial position information and real-time position information.

[0033] The output module is used to output measurement results, which are then used to assist in the control of the surgical robot.

[0034] Thirdly, embodiments of this application provide a surgical robot-assisted control device, including: a memory and a processor;

[0035] The memory stores the instructions that the computer executes;

[0036] The processor executes computer execution instructions stored in memory, causing the processor to perform the methods described in the various possible implementations of the first aspect above.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the methods described in the various possible implementations of the first aspect above.

[0038] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed, implements the methods described in the various possible implementations of the first aspect above.

[0039] The surgical robot assisted control method, device, storage medium, and program product provided in this application acquire the initial position information of the joints of the surgical robot through response-triggered measurement. During the measurement process, the movement of the surgical instrument's end effector is controlled according to the movement control command for the end effector, and the real-time position information of the joints is acquired. Based on the initial position information and the real-time position information, and based on the robot's kinematic model, the measurement result of the displacement of the surgical instrument's end effector is obtained. The measurement result is output and used to assist in the control of the surgical robot. The surgical robot assisted control method provided in this application can enable real-time surgical area measurement during robotic surgery, providing an intuitive dimensional reference for the minimally invasive surgical environment. In addition, based on the positional changes of the surgical robot's joints and the robot's kinematic model during the measurement process, the measurement result of the displacement of the surgical instrument's end effector is obtained, with a small measurement error, providing intuitive and accurate auxiliary information for robotic surgery. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1 Flowchart of the surgical robot-assisted control method provided in the embodiments of this application Figure 1 ;

[0042] Figure 2 A schematic diagram illustrating the principle of deep security monitoring provided in this application embodiment;

[0043] Figure 3 This is a schematic diagram of a surgical robot-assisted control scenario provided in an embodiment of this application;

[0044] Figure 4 Flowchart of the surgical robot-assisted control method provided in the embodiments of this application Figure 2 ;

[0045] Figure 5 Flowchart of the surgical robot-assisted control method provided in the embodiments of this application Figure 3 ;

[0046] Figure 6 This is a schematic diagram of the structure of the surgical robot-assisted control device provided in the embodiments of this application;

[0047] Figure 7 This is a schematic diagram of the structure of the surgical robot-assisted control device provided in the embodiments of this application.

[0048] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] Currently, although existing surgical robot systems are equipped with high-precision sensors capable of acquiring real-time motion data from the robotic arm's end effector, several issues remain regarding data application. First, related technologies primarily use motion data for motion control or path correction, failing to convert this data into visualized dimensional information. This prevents surgeons from effectively utilizing the robot's positioning advantages for measurement assistance. Second, these technologies focus on instrument planar position control, lacking analysis of depth-direction motion. They cannot distinguish whether instruments are moving on the surface or advancing deeper, and cannot issue warnings when operations are performed too deeply. Finally, existing interaction modes lack convenient measurement functions, preventing surgeons from quickly obtaining the necessary spatial dimensional data without interrupting the surgical procedure. Therefore, there is an urgent need for a surgical robot-assisted control method that can fully utilize robot kinematic data to provide surgeons with intuitive dimensional measurements and depth risk warnings.

[0051] To address the aforementioned technical issues, this application proposes a measurement method for surgical robot-assisted control. This method utilizes the main controller of the surgical robot to trigger measurements, calculates the kinematic data of the surgical robot joints, and measures and displays the actual measurement results between two points in real time. Simultaneously, it identifies the displacement changes of instruments in the depth direction and automatically issues a warning when the operating depth exceeds a safe threshold. Through these technical means, doctors are provided with real-time and accurate spatial judgment, reducing surgical risks.

[0052] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0053] Figure 1 Flowchart of the surgical robot-assisted control method provided in the embodiments of this application Figure 1,like Figure 1 As shown, the method includes:

[0054] S101, Response trigger measurement, obtain the starting position information of the joints of the surgical robot.

[0055] In one implementation, measurements can be triggered by gestures from the main operator. For example, during surgery, the surgeon initiates the measurement by "pinching" the joints, acquiring the clamping signal from the main operator, determining the starting point of the measurement in real time, and recording the state data of each joint of the robot to determine the initial position information. This method eliminates the need for additional operating equipment, allowing the surgeon to focus on the surgical area, simplifying the procedure and improving efficiency.

[0056] S102. During the measurement process, the movement of the surgical instrument end-effector is controlled according to the movement control command for the end-effector of the surgical instrument, and the real-time position information of the joint is obtained.

[0057] Specifically, during the measurement process, the surgeon controls the displacement of the surgical instrument's end effector point via the main operator, and these displacements are achieved through changes in the joint positions of the surgical robot. High-precision position feedback from the servo motors of each joint of the robot is read; this data source is stable and reliable, unaffected by intraoperative smoke, blood, or lighting conditions.

[0058] S103. Based on the initial position information and real-time position information, and using the robot kinematics model, obtain the measurement results of the displacement of the end effector point of the surgical instrument.

[0059] For example, the robot's kinematic model is a forward kinematic model. The initial position information and actual coordinate information of the robot joints are input into the forward kinematic model, and the precise three-dimensional physical coordinates of the surgical instrument's end effector point in the world coordinate system are continuously calculated. This coordinate directly reflects the actual spatial position of the surgical instrument's end effector within the patient's body. Its accuracy depends on the positioning accuracy of the surgical robot's robotic arm itself, typically reaching the millimeter level.

[0060] Based on the three-dimensional physical coordinates, the three-dimensional Euclidean distance between the initial position and the real-time position is calculated to obtain the measurement result of the displacement of the operating point at the end of the surgical instrument.

[0061] S104. Output the measurement results, which are used to assist in the control of the surgical robot.

[0062] For example, the measurement results can be output to an output device such as a display screen. In one embodiment, the measurement results are fused and output to a real-time endoscopic image to provide the surgical robot operator with intuitive surgical robot-assisted control information.

[0063] The surgical robot-assisted control method provided in this application embodiment can perform real-time intraoperative measurements in response to measurement triggers. Based on the positional changes of the surgical robot's joints and the robot's kinematic model during the measurement process, the measurement results of the displacement of the surgical instrument's end-effector operation point are obtained. The measurement error is small, providing intuitive and accurate auxiliary information for robotic surgery.

[0064] In one possible implementation, the output measurement results include:

[0065] Based on augmented reality technology, the measurement results are fused and displayed in real-time endoscopic images.

[0066] Augmented Reality (AR) technology is an interactive technology that uses visual perception and spatial positioning technology to accurately overlay and integrate computer-generated virtual information, including 3D models, text annotations, data images, navigation guides, etc., into real-world scenes and present them in real time. It does not replace the real environment, but rather supplements key information on the real scene, achieving a combination of virtual and real, real-time synchronization and spatial alignment, allowing users to obtain intuitive and efficient augmented information feedback while observing reality.

[0067] Specifically, based on augmented reality technology, the measurement results are precisely registered, spatially aligned, and rendered in real time with the real two-dimensional images acquired by the endoscope during the operation, and finally superimposed and presented in the same visual interface.

[0068] The surgical robot-assisted control method provided in this application embodiment is based on augmented reality technology. It integrates measurement results into real-time endoscopic images, providing surgical robot operators with enhanced visualization effects that combine realistic anatomical views with quantitative navigation information, effectively improving the accuracy and safety of surgical operations.

[0069] In one possible implementation, the measurement results include the depth displacement component of the surgical instrument tip operating point along the optical axis of the endoscope. Based on augmented reality technology, the measurement results are fused and displayed in a real-time endoscopic image, including:

[0070] If the depth displacement component is less than the depth determination threshold, the measurement type of the measurement result is determined to be surface measurement; if the depth displacement component is greater than or equal to the depth determination threshold, the measurement type of the measurement result is determined to be depth measurement.

[0071] Based on the output format corresponding to the measurement type, the measurement results are fused and displayed in real-time endoscopic images using augmented reality technology.

[0072] To compensate for the lack of depth perception caused by 2D screen imaging, the displacement vector of the surgical instrument tip is decomposed, with particular attention paid to the displacement component Δz along the optical axis of the endoscope (usually defined as the Z-axis). A preset depth determination threshold λ is used for analysis: If If so, the current measurement type is determined to be a surface measurement, mainly involving movement on the tissue surface; if If the measurement type is determined to be depth measurement, it indicates the presence of obvious longitudinal puncture or deep penetration. This represents the displacement of the operating point at the end of the surgical instrument. This refers to the depth displacement component of the surgical instrument's end-effector along the optical axis of the endoscope.

[0073] Depending on the measurement type, the measurement results can be output in different formats to further guide the surgical robot operator in the direction of movement of the end effector point. For example, the measurement results include the trajectory line of the end effector point and measurement data. If the measurement type is surface measurement, the trajectory line is displayed in a standard color (such as green or white) and its length is shown. If the measurement type is depth measurement, the trajectory line is automatically switched to a warning color (such as red), and a text prompt "Depth: X mm" can be added next to the trajectory length, effectively preventing misjudgments of distance by the surgeon due to fluoroscopic projection.

[0074] In one implementation, the depth determination threshold λ can be set differently in different surgeries. The embodiments of this application provide the following methods for obtaining the depth determination threshold λ: determined according to preoperative examination, or manually input through the human-computer interaction interface of the surgical robot; before the measurement function is triggered, the movement range of the surgical instrument end point is measured first, and the depth determination threshold λ is determined according to the measurement results; or the depth determination threshold λ is determined based on statistics according to historical cases.

[0075] The surgical robot assisted control method provided in this application measures the depth displacement component of the surgical instrument end-effector along the optical axis of the endoscope, determines the measurement type of the measurement result based on the depth displacement component, and outputs different output formats for different types of measurement results, which can provide more intuitive and clear auxiliary information for the surgical robot operator.

[0076] In one possible implementation, based on the output format corresponding to the measurement type, the measurement results are fused and displayed in the real-time endoscopic image using augmented reality technology, including:

[0077] Based on the robot-endoscope hand-eye transformation matrix, the three-dimensional physical coordinates corresponding to the measurement results are converted into two-dimensional pixel coordinates corresponding to the endoscope image. Based on the measurement results and two-dimensional pixel coordinates, measurement guidance information is generated, which includes the measurement trajectory and measurement data. Based on augmented reality technology, an image rendering engine is used to overlay and draw the measurement guidance information on the real-time endoscope image according to the output format, so as to fuse and display the measurement results in the real-time endoscope image.

[0078] Specifically, the robot-endoscope hand-eye transformation matrix, which has been pre-calibrated before surgery, is used. The calculated three-dimensional physical coordinates of the surgical instrument end-effector manipulation point Convert to two-dimensional pixel coordinates corresponding to endoscopic images Its transformation formula is:

[0079]

[0080] in, Two-dimensional pixel coordinates, For robot-endoscope hand-eye transformation matrix, Let K be the three-dimensional physical coordinates of the operating point at the end of the surgical instrument, and K be the intrinsic parameter matrix of the endoscope camera.

[0081] Then, dynamic visualization rendering is performed. The graphics rendering engine overlays measurement guidance information onto the real-time endoscopic video screen based on the mapped pixel coordinates. The measurement guidance information includes the measurement trajectory and measurement data. The image rendering engine dynamically connects the starting point and the current surgical instrument end point on the screen with virtual dotted lines according to the output format. For example, if the measurement type is surface measurement, the trajectory line is displayed in a normal color (such as green or white) and the trajectory line length is shown; if the measurement type is "depth measurement", the trajectory line is automatically switched to a warning color (such as red), and a text prompt "Depth: X mm" can be added next to the trajectory length, thereby effectively preventing doctors from misjudging distances due to fluoroscopic projection.

[0082] The surgical robot-assisted control method provided in this application establishes a hand-eye relationship between the robot and the endoscope, enabling precise mapping of measurement results from three-dimensional physical space to two-dimensional image space of the endoscope. Based on this, guidance information containing measurement trajectory and data is generated. Then, with the help of augmented reality technology and image rendering engine, the guidance information is superimposed and drawn on the endoscope screen in real time, ultimately allowing the measurement results to be seamlessly integrated and synchronously presented with the real surgical field of view. This provides intuitive and accurate visual navigation for surgical operations and improves operational efficiency and surgical safety.

[0083] In one possible implementation, the measurement results include the depth displacement component of the surgical instrument tip operating point along the optical axis of the endoscope, and the surgical robot-assisted control method further includes:

[0084] If the detected depth displacement component exceeds the safe depth displacement threshold, a depth risk warning is triggered. The depth risk warning is used to alert the surgical robot operator that there is an operational risk.

[0085] Specifically, the system monitors the depth-direction movement of the surgical instrument's end effector point in real time. In two-dimensional images, it's difficult to clearly distinguish whether the instrument is moving along the surface or penetrating deeper layers. By analyzing the depth feed of the instrument's end effector point, the system ensures that it doesn't exceed the safe depth range. Once the system detects a depth exceeding the safe depth threshold, it will promptly warn the physician, helping them avoid misjudging the depth.

[0086] Figure 2 This is a schematic diagram illustrating the depth safety monitoring principle provided in this application embodiment. First, the joint angles of each joint of the surgical robot are acquired via position sensors. Simultaneously, depth limits, i.e., safe depth displacement thresholds, are obtained through a user-defined interface and stored in a safety threshold register. Then, the real-time tip depth is obtained through an operational calculation module and compared with a reference value in the safety threshold register. If the detected tip depth exceeds the reference value, a risk warning can be issued via a display, buzzer, or other means, while simultaneously using a motor drive to perform safety control of the surgical robot.

[0087] The surgical robot-assisted control method provided in this application monitors the depth displacement component during the operation of the surgical robot in real time and compares it with a preset safe depth displacement threshold. Once the depth displacement exceeds the safe range, a depth risk warning is immediately triggered, which can promptly alert the operator to potential operational risks and effectively avoid safety hazards such as tissue damage and instrument penetration caused by excessively deep operations, thereby improving the stability and safety of the surgical process.

[0088] In one possible implementation, the initial position information is the initial joint angle vector of the surgical robot, the real-time position information is the real-time joint angle vector of the surgical robot, and the robot kinematic model includes a positive kinematic model.

[0089] Based on the initial position information and real-time position information, and using the robot's kinematic model, the measurement results of the displacement of the surgical instrument's end effector point are obtained, including:

[0090] Based on the initial joint angle vector and the real-time joint angle vector, and using the forward kinematics model, the three-dimensional physical coordinates of the surgical instrument end-effector manipulation point in the world coordinate system are calculated.

[0091] The displacement of the surgical instrument end-effector's operating point is measured based on the three-dimensional physical coordinates.

[0092] The measurement results in this embodiment do not rely on image recognition, but rather directly read the high-precision position feedback from the servo motors of each joint of the robot. This data source is stable and reliable, and is not affected by intraoperative smoke, blood, or lighting conditions.

[0093] Specifically, the robot's forward kinematics model is invoked, and the joint angle vectors acquired in real time are used... As input, the precise three-dimensional physical coordinates of the surgical instrument end effector point in the world coordinate system are continuously calculated. This coordinate system directly reflects the actual spatial position of the surgical instrument's end effector within the patient's body, and its accuracy depends on the robotic arm's own positioning precision. It typically achieves millimeter-level accuracy.

[0094] While the surgical robot is in measurement mode and moving the end effector point of the surgical instrument, the coordinates of the current end effector point of the surgical instrument are continuously acquired. And calculate its coordinates relative to the starting anchor point. The three-dimensional Euclidean distance between them. Specifically, the three-dimensional Euclidean distance is calculated according to the following formula:

[0095] .

[0096] in, It is a three-dimensional Euclidean distance. The coordinates of the current surgical instrument tip are shown. These are the coordinates of the starting anchor point. The three-dimensional Euclidean distance is the actual physical dimension between the two points.

[0097] Furthermore, it can also be based on the coordinates of the current surgical instrument end effector. Analyze the trajectory and depth information of the surgical instrument end effector.

[0098] Figure 3 This is a schematic diagram of a surgical robot-assisted control scenario provided in an embodiment of this application. Figure 3 As shown in the diagram, measurements are taken starting from point A. The position information of the joints of the surgical robot corresponding to point A is the initial position information. The surgical robot operator actively remotely moves the end effector of the surgical instrument to point B. The position information of the joints of the surgical robot corresponding to point B is the real-time position information. Based on the initial position information and the real-time position information, and using the robot's kinematic model, the displacement L of the end effector is obtained. Furthermore, the feed depth of the end effector can be monitored. When the feed depth exceeds the safe depth limit, for example, when the end effector is at point C, a risk warning is triggered.

[0099] The surgical robot assisted control method provided in this application uses the initial joint angle vector as a reference and the real-time joint angle vector as a dynamic input. It combines the forward kinematics model to accurately calculate the three-dimensional physical coordinates of the surgical instrument end-effector operation point in the world coordinate system, thereby obtaining the displacement measurement results. This enables real-time, accurate, and quantitative monitoring of the instrument end-effector position and motion displacement, providing reliable position data support for the precision control, risk warning, and navigation guidance of surgical operations, and improving the stability and safety of surgical robot operation.

[0100] In one possible implementation, the measurement is triggered in the following manner:

[0101] When the surgical robot is in active remote control mode, monitor the status signal of the end effector of the master hand;

[0102] If the status signal indicates that the measurement has started, determine whether to trigger the measurement.

[0103] Specifically, when the surgical robot is in master-slave teleoperation mode, the operator controls the slave surgical instruments via the master controller. The system continuously monitors the status signals of the master end effector. For example, when the operator manipulates the instrument end effector to contact target point A within the body and performs a "pinching" motion, the system detects this change in gripping state and determines the current moment as the measurement start time t0. The system immediately locks and records the encoder values ​​of each joint of the robot at time t0, using them as the spatial reference data for the measurement. This triggering method does not require the surgeon to take their eyes off the surgical area, nor does it require additional foot pedals or touchscreen operation.

[0104] Optionally, measurements can also be triggered via voice control, button control, or other methods.

[0105] The surgical robot assisted control method provided in this application embodiment uses a mechanism that triggers the measurement process through the gestures of the main operator, eliminating the need for additional operating equipment. This allows the surgical robot operator to focus on the surgical area, simplifying the operation process and improving efficiency.

[0106] In one implementation, endoscopic video images are first acquired in real time, and combined with the robot's kinematic model and sensor data, the three-dimensional position of the surgical instrument's end effector is calculated. Based on this data, the system displays the calculated spatial dimensions, such as incision length and tumor size, on the surgical display interface in real time. This allows the surgical robot operator to directly view key dimensional data without relying on experience-based estimations, thus improving surgical accuracy.

[0107] The system can also monitor the depth movement of instruments in real time. In two-dimensional images, it's difficult to clearly distinguish whether an instrument is moving along a surface or penetrating deeper layers. By analyzing the instrument's depth feed, the system ensures it doesn't exceed the safe depth range. If the system detects a depth exceeding the limit, it will promptly warn the surgical robot operator, helping them avoid misjudging the depth.

[0108] Furthermore, when the surgical robot operator needs to quickly measure the distance between two points, the system triggers the measurement function with a simple gesture. The operator can directly obtain the required dimensional information without switching equipment or interrupting the surgical procedure. This interactive method makes measurement more convenient and improves surgical efficiency.

[0109] This embodiment, through real-time calculation and feedback, enables the system to flexibly respond to dynamic changes during surgery, avoiding the shortcomings of existing technologies that rely on preset paths or external coordinates. This method provides intuitive and precise surgical support, improving the operator's precision and surgical safety.

[0110] Figure 4 Flowchart of the surgical robot-assisted control method provided in the embodiments of this application Figure 2 In one implementation, such as Figure 4 As shown, the overall process can be divided into a data acquisition stage, a core calculation stage, and an interactive feedback stage. In the data acquisition stage, the user triggers the measurement and selects the starting point A, obtaining the coordinates P1 of point A. Then, based on the motion data of the surgical robot, the coordinates P2 of the surgical instrument's end effector point, i.e., point B, are calculated in real time. In the core calculation stage, the Euclidean distance between P1 and P2 is calculated to obtain the measurement result, and the validity of the result is verified. Finally, in the interactive feedback stage, based on the measurement results, the measurement line and the measurement data L between P1 and P2 are overlaid and displayed in the endoscopic image view.

[0111] Figure 5 Flowchart of the surgical robot-assisted control method provided in the embodiments of this application Figure 3 In one implementation, such as Figure 5As shown, the system acquires the input signal from the main controller, triggers measurement based on the input signal, and then calculates the pose of the robotic arm's end effector (the surgical instrument's end point) based on the kinematic phase, obtaining the measurement results. Based on the measurement results, the current working mode can be determined as either surface measurement or depth monitoring. When the depth displacement component is small, it is determined to be surface measurement; when the depth displacement component is large, it is determined to be depth monitoring. In surface measurement mode, the 3D coordinates of the starting point P1 and the real-time moving point P2 are recorded, the Euclidean distance D is calculated, and the result is overlaid on the display interface using augmented reality technology. In depth monitoring mode, the depth direction component Z is extracted. When Z exceeds a preset safety threshold, a depth risk warning is triggered, and the warning information is displayed on the surgical display terminal interface.

[0112] Regarding spatial dimension feedback, the surgical robot-assisted control method provided in this application converts the robot's high-precision positioning data into visualized dimensional information in real time, directly displaying key data such as incision length and tumor size on the surgical display interface. The surgical robot operator can obtain accurate real-time dimensional data without relying on experience-based estimations, thereby improving decision support and accuracy during surgery. In terms of depth direction recognition, this method can accurately distinguish between surface manipulation and depth feeding of instruments, and monitor in real time whether instruments have entered dangerous areas. Through depth direction analysis, the surgical robot operator can obtain timely depth perception feedback, thereby avoiding operational risks caused by depth misjudgment. Regarding interactive measurement functions, this method designs a gesture-triggered instant measurement mechanism, allowing the surgical robot operator to obtain the distance between two points at any time during surgery. This interactive method eliminates the need for the surgical robot operator to switch equipment or interrupt the surgical procedure, improving operational flexibility and efficiency.

[0113] Figure 6 This is a schematic diagram of the surgical robot-assisted control device provided in the embodiments of this application, as shown below. Figure 6 As shown, the surgical robot-assisted control device 60 provided in this embodiment includes:

[0114] The acquisition module 601 is used to acquire the starting position information of the joints of the surgical robot in response to the trigger measurement.

[0115] The control module 602 is used to control the movement of the surgical instrument end-effector point according to the movement control command for the end-effector point during the measurement process, and to acquire the real-time position information of the joint.

[0116] The processing module 603 is used to obtain the measurement results of the displacement of the end effector point of the surgical instrument based on the robot kinematic model, according to the initial position information and the real-time position information.

[0117] Output module 604 is used to output measurement results, which are used to assist in the control of the surgical robot.

[0118] In one possible implementation, the output module 604 is specifically used for:

[0119] Based on augmented reality technology, the measurement results are fused and displayed in real-time endoscopic images.

[0120] In one possible implementation, the measurement results include the depth displacement component of the surgical instrument tip operating point along the optical axis of the endoscope, and the output module 604 is specifically used for:

[0121] If the depth displacement component is less than the depth determination threshold, the measurement type of the measurement result is determined to be surface measurement; if the depth displacement component is greater than or equal to the depth determination threshold, the measurement type of the measurement result is determined to be depth measurement.

[0122] Based on the output format corresponding to the measurement type, the measurement results are fused and displayed in real-time endoscopic images using augmented reality technology.

[0123] In one possible implementation, the output module 604 is specifically used for:

[0124] Based on the robot-endoscope hand-eye transformation matrix, the three-dimensional physical coordinates corresponding to the measurement results are converted into two-dimensional pixel coordinates corresponding to the endoscope image;

[0125] Based on the measurement results and two-dimensional pixel coordinates, measurement guidance information is generated, which includes the measurement trajectory and measurement data;

[0126] Based on augmented reality technology, an image rendering engine is used to overlay measurement guidance information onto real-time endoscopic images according to the output format, so as to fuse the measurement results in the real-time endoscopic images for display.

[0127] In one possible implementation, the measurement results include the depth displacement component of the surgical instrument tip operating point along the optical axis of the endoscope, and the output module 604 is further used for:

[0128] If the detected depth displacement component exceeds the safe depth displacement threshold, a depth risk warning is triggered. The depth risk warning is used to alert the surgical robot operator that there is an operational risk.

[0129] In one possible implementation, the initial position information is the initial joint angle vector of the surgical robot, the real-time position information is the real-time joint angle vector of the surgical robot, and the robot kinematic model includes a positive kinematic model.

[0130] In one possible implementation, the processing module 603 is specifically used for:

[0131] Based on the initial joint angle vector and the real-time joint angle vector, and using the forward kinematics model, the three-dimensional physical coordinates of the surgical instrument end-effector manipulation point in the world coordinate system are calculated.

[0132] The displacement of the surgical instrument end-effector's operating point is measured based on the three-dimensional physical coordinates.

[0133] In one possible implementation, the measurement is triggered in the following manner:

[0134] When the surgical robot is in active remote control mode, monitor the status signal of the end effector of the master hand;

[0135] If the status signal indicates that the measurement has started, determine whether to trigger the measurement.

[0136] The surgical robot-assisted control device provided in this embodiment can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0137] Figure 7 This is a schematic diagram of the structure of the surgical robot-assisted control device provided in an embodiment of this application. Figure 7 As shown, the surgical robot-assisted control device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the surgical robot-assisted control device 70 further includes a communication interface 703. The processor 701, memory 702, and communication interface 703 are connected via a communication bus 704.

[0138] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0139] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0140] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0141] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0142] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0143] This application also provides a computer program product, including a computer program that, when executed, implements the above-described method.

[0144] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the above-described method.

[0145] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0146] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0147] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0152] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A surgical robot-assisted control method, characterized in that, include: The system responds to a trigger measurement to obtain the initial position information of the joints of the surgical robot. During the measurement process, the movement of the surgical instrument end-effector is controlled according to the movement control command for the end-effector of the surgical instrument, and the real-time position information of the joint is obtained. Based on the initial position information and the real-time position information, and using the robot kinematics model, the measurement result of the displacement of the end effector point of the surgical instrument is obtained; The measurement results are output, and these results are used to assist in controlling the surgical robot.

2. The surgical robot-assisted control method according to claim 1, characterized in that, The measurement results are output, including: Based on augmented reality technology, the measurement results are fused and displayed in real-time endoscopic images.

3. The surgical robot-assisted control method according to claim 2, characterized in that, The measurement results include the depth displacement component of the surgical instrument tip operating point along the optical axis of the endoscope. The process of fusing and displaying the measurement results in a real-time endoscopic image using augmented reality technology includes: If the depth displacement component is less than the depth determination threshold, the measurement type of the measurement result is determined to be surface measurement; if the depth displacement component is greater than or equal to the depth determination threshold, the measurement type of the measurement result is determined to be depth measurement. Based on the output format corresponding to the measurement type, the measurement results are fused and displayed in the real-time endoscopic image using augmented reality technology.

4. The surgical robot-assisted control method according to claim 3, characterized in that, The step of fusing and displaying the measurement results in a real-time endoscopic image based on augmented reality technology, according to the output format corresponding to the measurement type, includes: Based on the robot-endoscope hand-eye transformation matrix, the three-dimensional physical coordinates corresponding to the measurement results are converted into two-dimensional pixel coordinates corresponding to the endoscope image; Based on the measurement results and the two-dimensional pixel coordinates, measurement guidance information is generated, which includes the measurement trajectory and measurement data; Based on augmented reality technology, an image rendering engine is used to overlay and draw the measurement guidance information on the real-time endoscopic image according to the output format, so as to fuse and display the measurement results in the real-time endoscopic image.

5. The surgical robot-assisted control method according to any one of claims 1 to 4, characterized in that, The measurement results include the depth displacement component of the surgical instrument tip operating point along the optical axis of the endoscope, and the surgical robot-assisted control method further includes: If the depth displacement component is detected to be greater than the safe depth displacement threshold, a depth risk warning is triggered. The depth risk warning is used to alert the surgical robot operator that there is an operational risk.

6. The surgical robot-assisted control method according to any one of claims 1 to 4, characterized in that, The initial position information is the initial joint angle vector of the surgical robot, the real-time position information is the real-time joint angle vector of the surgical robot, and the robot kinematic model includes a positive kinematic model. The step of obtaining the measurement result of the displacement of the surgical instrument end effector point based on the initial position information and the real-time position information and the robot kinematic model includes: Based on the initial joint angle vector and the real-time joint angle vector, the three-dimensional physical coordinates of the surgical instrument end-effector operation point in the world coordinate system are calculated using a forward kinematics model. The displacement of the surgical instrument end-effector's operating point is measured based on the three-dimensional physical coordinates.

7. The surgical robot-assisted control method according to any one of claims 1 to 4, characterized in that, The measurement is triggered in the following manner: When the surgical robot is in active remote control mode, the status signal of the end effector of the master hand is monitored. If the status signal indicates that a measurement has started, the measurement is triggered.

8. A surgical robot-assisted control device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program, which, when executed, implements the method according to any one of claims 1 to 7.

Images