Remote surgery robot master-slave operation delay measurement method and device thereof

Abstract

The application provides a master-slave operation delay measurement method and device for a remote surgery robot, and relates to the technical field of medical instruments. The method comprises the following steps: calculating the theoretical two-dimensional coordinates of the instrument tips on each slave hand operating arm in a display picture according to the mechanical structure of the slave hand operating arm and an endoscope imaging model; obtaining the display picture at the master hand end, identifying and calculating the actual two-dimensional coordinates of the instrument tips in the display picture; obtaining the theoretical two-dimensional coordinates and the actual two-dimensional coordinates of the instrument tips during the movement of the master hand end, and constructing the displacement curves of the two coordinates changing with time; determining the time difference corresponding to the theoretical displacement curve and the actual displacement curve when the displacement difference of the two is within a preset tolerance range as the master-slave operation delay. The application can realize the accurate measurement of the comprehensive delay of the master-slave communication delay and the picture transmission delay in the remote surgery scene, so as to improve the real-time response and control accuracy of the master-slave operation.

Description

Technical Field

[0002] This invention relates to the technical field of medical devices, and in particular to a method and apparatus for measuring the master-slave operation delay of a remote surgical robot. Background Technology

[0004] The remote surgical robot system employs a master-slave remote control mode. The surgeon uses the master hand to control instruments on the slave hand to perform precise operations, while simultaneously monitoring the surgical progress in real-time via a display on the master hand. In this master-slave remote control mode, the system's real-time performance is crucial for ensuring surgical safety.

[0005] Master-slave operation latency is a crucial indicator of system real-time performance. This latency primarily includes the communication time from the doctor's operation of the master hand to the slave hand receiving joint commands, and the transmission time from the endoscope image to the display screen on the master hand. The combined effect of these two latency components directly impacts the doctor's hand-eye coordination and real-time feedback during surgical procedures.

[0006] In existing technologies, delay measurement methods for surgical robots mainly obtain delay data from the master and slave ends by acquiring motion data from both ends. These methods are primarily used for delay measurement in local surgical robots and are difficult to apply directly to remote surgical robots. These methods are typically tested under conditions where the master and slave ends are physically close, and their measurement targets are mainly the processing time of signal processing and command response in the local control system. They cannot cover the additional transmission time caused by network communication and endoscopic image transmission in remote surgical scenarios, making it difficult to accurately measure the master-slave operation delay that includes the complete transmission link. Summary of the Invention

[0008] The purpose of this application is to provide a method and apparatus for measuring the master-slave operation delay of a remote surgical robot, which can achieve accurate measurement of the combined master-slave communication delay and image transmission delay in remote surgical scenarios, so as to improve the real-time response and control accuracy of master-slave operations.

[0009] In a first aspect, embodiments of this application provide a method for measuring the master-slave operation delay of a remote surgical robot, including: Based on the mechanical structure of the hand-operated arm and the endoscope imaging model, calculate the theoretical two-dimensional coordinates of the instrument tip on each of the hand-operated arms in the display screen; Acquire the display screen of the master handpiece, identify and calculate the actual two-dimensional coordinates of each instrument endpiece in the display screen; The theoretical two-dimensional coordinates and the actual two-dimensional coordinates of each instrument end during the movement of the main hand end are obtained, and displacement curves of the two as a function of time are constructed respectively. When the displacement difference between the two is within a preset tolerance range, the time difference between the theoretical displacement curve and the actual displacement curve is used as the master-slave operation delay.

[0010] Furthermore, the step of calculating the theoretical two-dimensional coordinates of the instrument ends on each slave arm in the display screen based on the mechanical structure of the slave arm and the endoscopic imaging model includes: Based on the mechanical structure of the hand-operated arm, a kinematic model from the base to the end of the device is established; Based on the kinematic model, calculate the local coordinates of the instrument tip on each of the manipulator arms in the endoscope tip coordinate system; Based on the endoscopic imaging model, obtain the camera intrinsic parameters of the endoscope; Based on the camera intrinsic parameters of the endoscope, the local coordinates are converted into theoretical two-dimensional coordinates of each instrument tip in the display screen.

[0011] Furthermore, establishing a kinematic model from the base to the end effector based on the mechanical structure of the hand-operated arm includes: Based on the mechanical structure of the slave arm, a base coordinate system is established at the base, and a local coordinate system is established at at least some joints of the slave arm, thereby determining the transformation relationship between the coordinate systems.

[0012] Furthermore, the calculation of the local coordinates of each instrument tip on the operating arm in the endoscopic endoscope coordinate system based on the kinematic model includes: Obtain the current position of each joint on the hand-operated arm; Based on the transformation relationship, calculate the global coordinates of the instrument end of each slave manipulator arm in the base coordinate system; Based on the transformation relationship between the local coordinate systems, the transformation matrix from the end of the instrument on the hand operating arm to the end of the endoscope is obtained; Based on the transformation matrix, calculate the local coordinates of the instrument tip on each slave manipulator arm in the endoscope end coordinate system.

[0013] Furthermore, the acquisition of camera intrinsic parameters of the endoscope includes: Images of the checkerboard calibration board in multiple different orientations were acquired using an endoscope; The Zhang calibration method was used to process the acquired checkerboard calibration board images to obtain the camera intrinsic parameters of the endoscope.

[0014] Furthermore, the identification and calculation of the actual two-dimensional coordinates of each instrument end effector in the display screen includes: The edge features of each instrument's end in the displayed screen are extracted using image recognition or AI recognition. The actual pixel coordinates of each instrument end in the displayed screen are calculated based on the extracted edge features.

[0015] Furthermore, obtaining the theoretical two-dimensional coordinates and the actual two-dimensional coordinates of each instrument end during the movement of the main hand includes: The operator moves the master hand a preset distance, ensuring that the ends of each instrument remain within the display screen during the movement; During the movement, the theoretical two-dimensional coordinates and the actual two-dimensional coordinates of each instrument end are continuously recorded.

[0016] Furthermore, determining the time difference between the theoretical displacement curve and the actual displacement curve when the displacement difference between the two is within a preset tolerance range includes: On the theoretical displacement curve and the actual displacement curve, find the time point corresponding to when the displacement value is nearly consistent within a preset tolerance range; Calculate the time difference between the aforementioned time points as the master-slave operation delay.

[0017] Furthermore, it also includes: Repeat the steps of moving the main hand multiple times, and record the time difference for each operation; Calculate the statistical value of the time difference multiple times, and use it as the final master-slave operation delay.

[0018] Secondly, embodiments of this application provide an apparatus for measuring the master-slave operation delay of a remote surgical robot. The theoretical coordinate calculation module is used to calculate the theoretical two-dimensional coordinates of the instrument end on each of the slave manipulator arms in the display screen based on the mechanical structure of the slave manipulator arm and the endoscope imaging model. The actual coordinate recognition module is used to acquire the display screen of the main handpiece, identify and calculate the actual two-dimensional coordinates of each instrument endpiece in the display screen; The displacement curve construction module is used to obtain the theoretical two-dimensional coordinates and the actual two-dimensional coordinates of each instrument end during the movement of the main hand end, and to construct displacement curves of the two as a function of time. The delay determination module is used to determine the time difference between the theoretical displacement curve and the actual displacement curve when the displacement difference between the two is within a preset tolerance range, and uses this as the master-slave operation delay.

[0019] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the aforementioned remote surgical robot master-slave operation delay measurement method.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which are read and executed by a processor to perform the aforementioned remote surgical robot master-slave operation delay measurement method.

[0021] The remote surgical robot master-slave operation delay measurement method and apparatus provided in this application have at least the following beneficial effects: This application calculates the theoretical two-dimensional coordinates of the instrument's end effector based on the mechanical structure of the slave manipulator and the endoscopic imaging model. These theoretical coordinates reflect the expected position of the instrument based on a kinematic model. Simultaneously, it obtains the actual two-dimensional coordinates of the instrument's end effector by identifying the display screen on the master end. These actual coordinates represent the actual position observed from the image acquired at the master end to the display screen on the master end. By synchronously recording the theoretical and actual coordinates over time, displacement curves of both are constructed. The theoretical motion trajectory of the instrument's end effector is compared with the actual observed trajectory on the time axis. This enables accurate measurement of the comprehensive delay, including master-slave command transmission and image transmission, under remote operation conditions. This solves the problem that existing technologies can only measure local master-slave control delay and cannot cover remote transmission links.

[0022] Based on this, by comparing displacement curves, the superposition effect of master-slave command transmission and image transmission can be comprehensively reflected. The measured delay truly reflects the actual time difference between the doctor's operation instructions and visual feedback during remote surgery. This delay data can serve as a compensation basis for the master-slave control system, for example, for predictive control, motion smoothing filtering, or dynamically adjusting the master-slave mapping ratio, thereby improving the real-time responsiveness, operational accuracy, and safety of remote surgery.

[0023] In addition, the time delay measurement method of this application does not rely on additional external measurement equipment (such as high-speed cameras or dedicated signal generators), and can be completed entirely based on the kinematic data and image data of the surgical robot system itself. It has good integration and repeatability, and is easy to implement online in actual clinical scenarios. Attached Figure Description

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the main control console in the remote surgical robot provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the slave operating platform in the remote surgical robot provided in the embodiments of this application; Figure 3 A flowchart illustrating the remote surgical robot master-slave operation delay measurement method provided in this application embodiment; Figure 4 This is a flowchart illustrating the process of calculating the theoretical two-dimensional coordinates of the end effector on each slave manipulator arm in the display screen. Figure 5 This is a comparison chart of theoretical displacement curves and actual displacement curves plotted on the same time axis. Figure 6 This is a structural block diagram of the remote surgical robot master-slave operation delay measurement method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention.

[0027] icon: 10 - Theoretical coordinate calculation module; 20 - Actual coordinate identification module; 30 - Displacement curve construction module; 40 - Delay determination module; 100 - Main control console; 110 - Main operator; 120 - Observation bay; 200 - Slave operating platform; 210 - Base; 220 - Slave operating arm; 130 - Processor; 131 - Memory; 132 - Bus; 133 - Communication interface. Detailed Implementation

[0029] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] This embodiment provides a latency measurement method for remote surgical robot scenarios. The surgical robot system includes at least a master control console 100 (i.e., the master end) and a slave operating platform 200 (i.e., the slave end). It should be noted that the latency measurement method provided in this embodiment can be used to measure the end-to-end latency between the master and slave ends, from the issuance of a control command from the master end to the execution of the command by the slave end, and the feedback of the image acquired by the slave end back to the master end.

[0031] like Figure 1 As shown, the master control console 100 serves as the operating end of the teleoperated surgical robot, and it mainly includes the master operator 110 and the observation chamber 120.

[0032] The master hand 110 is used to receive hand movement information from the operator (e.g., a doctor). Specifically, the operator generates kinematic data such as displacement and rotation by manipulating the master hand 110. This data is collected in real time and converted into motion control signals. These motion control signals are transmitted (remotely transmitted) to the slave hand via a network to drive the actuators (e.g., surgical instruments) of the slave hand operating platform 200 to perform corresponding actions, thereby realizing teleoperation.

[0033] The observation chamber 120 is a display component that allows the operator to obtain visual feedback on the target area. The observation chamber 120 is equipped with an eyepiece, which integrates a display device, which can be one or more two-dimensional displays. The operator observes the displayed image through the eyepiece to obtain an immersive viewing experience. In one implementation, the display screen is configured to provide the operator with a stereoscopic image (i.e., a three-dimensional image) to assist the operator in accurate depth perception. The content presented in this stereoscopic image is the scene image upon which the operator relies during their operation.

[0034] like Figure 2 As shown, the slave operating platform 200 serves as the execution end of the teleoperated surgical robot, and mainly includes a base 210 and slave operating arms 220 mounted on the base 210. To achieve multi-instrument collaborative operation, the number of slave operating arms 220 can be multiple, such as two, three, or four. The specific number can be configured according to actual needs and complexity, and this embodiment does not limit this.

[0035] In this embodiment, surgical instruments are mounted on the distal end of a portion of the slave arm 220 (i.e., the end furthest from the base 210) for performing surgical operations. These instruments include, for example, needle forceps, electrocautery hooks, scissors, or staplers, used for clamping, cutting, suturing, electrocautery, and other surgical procedures in actual use. One distal end of the slave arm 220 is used to mount an endoscope for acquiring scene images of the target area. Optionally, this endoscope can be a binocular endoscope (i.e., a three-dimensional endoscope) to acquire stereoscopic images with depth information. The endoscope transmits the acquired image data in real time to the observation chamber 120 of the main control console 100 for display, providing visual feedback to the operator.

[0036] Specifically, each slave manipulator 220 is equipped with multiple joints, including active and passive joints. Passive joints are used to achieve a wide range of arm positioning and adjustment, and are driven and controlled by a drive mechanism. Active joints respond to motion control commands sent by the master manipulator 110 of the master console 100, driving surgical instruments or endoscopes to perform precise movements during actual applications. Each joint is equipped with an encoder, which is used to collect joint motion data in real time. The motion data includes at least the joint's angle, angular velocity, or position information.

[0037] The slave operating platform 200 also includes a remote center of motion (RCM). This RCM is located between the distal end of the slave operating arm 220 and the tip of the surgical instrument. Specifically, along the kinematic chain direction of the slave operating arm 220, the RCM is set at the pivot position where the puncture cannula contacts the body wall of the patient. The tip of the surgical instrument always passes through this RCM and extends into the body cavity during movement. Through the aforementioned arrangement, the tip of the surgical instrument can achieve flexible, multi-degree-of-freedom movement within the body cavity without enlarging the puncture site.

[0038] Based on the aforementioned structure of the master control console 100 and slave operation platform 200, this embodiment provides a method for measuring the master-slave operation delay of a remote surgical robot. This method, by integrating kinematic models and image recognition technology, achieves accurate measurement of the end-to-end delay, including command transmission, execution response, and image feedback. Figure 3 As shown, the method includes the following steps: S110, based on the mechanical structure of the slave manipulator 220 and the endoscope imaging model, calculate the theoretical two-dimensional coordinates of the instrument end on each slave manipulator 220 in the display screen.

[0039] S120: Acquire the display screen of the master handpiece, identify and calculate the actual two-dimensional coordinates of each instrument end in the display screen.

[0040] S130: Obtain the theoretical and actual two-dimensional coordinates of each instrument end during the movement of the main hand end, and construct the displacement curves of the two as a function of time.

[0041] S140, when the displacement difference between the two is within the preset tolerance range, the time difference between the theoretical displacement curve and the actual displacement curve is determined as the master-slave operation delay.

[0042] This embodiment calculates the theoretical two-dimensional coordinates of the instrument's end effector based on the mechanical structure of the slave manipulator 220 and the endoscopic imaging model. These theoretical coordinates reflect the expected position of the instrument based on the kinematic model. Simultaneously, the actual two-dimensional coordinates of the instrument's end effector are obtained by identifying the display screen on the master end. These actual coordinates represent the actual position observed from the acquisition of the image at the master end to the display screen on the master end. By synchronously recording the theoretical and actual coordinates over time, displacement curves of both are constructed. The theoretical motion trajectory of the instrument's end effector is compared with the actual observed trajectory on the time axis. This enables accurate measurement of the comprehensive delay, including master-slave command transmission, instrument execution response, image acquisition and transmission, and screen display, under remote operation conditions. This solves the problem that existing technologies can only measure local master-slave control delay and cannot cover remote transmission links.

[0043] This embodiment, through the comparison of displacement curves, comprehensively reflects the superposition effect of master-slave command transmission and image transmission. The measured delay truly reflects the actual time difference between the doctor's operational commands and visual feedback during remote surgery. This delay data can serve as a compensation basis for the master-slave control system, for example, for predictive control, motion smoothing filtering, or dynamically adjusting the master-slave mapping ratio, thereby improving the real-time responsiveness, operational accuracy, and safety of remote surgery.

[0044] In addition, the time delay measurement method in this embodiment does not rely on additional external measurement equipment (such as high-speed cameras or dedicated signal generators). It can be completed entirely based on the kinematic data and image data of the surgical robot system itself, which has good integration and repeatability, and is easy to implement online in actual clinical scenarios.

[0045] The following will provide a detailed explanation of steps S110 to S140 above.

[0046] Regarding step S110 above, namely, "calculating the theoretical two-dimensional coordinates of the surgical instrument ends on each slave manipulator 220 in the display screen based on the mechanical structure of the slave manipulator 220 and the endoscopic imaging model," this embodiment provides the following specific implementation method. For example... Figure 4 As shown, step S110 specifically includes the following four aspects.

[0047] S111, Based on the mechanical structure of the hand-operated arm 220, establish a kinematic model from the base 210 to the end of the device.

[0048] This step describes the relative pose relationships between the components of the manipulator 220 by establishing a multi-level coordinate system. It further includes the following sub-steps: Based on the mechanical structure of the slave manipulator 220, a base coordinate system is established at the base 210, and a local coordinate system is established at at least some joints of the slave manipulator 220, thereby determining the transformation relationship between the coordinate systems.

[0049] First, a base coordinate system is established at the base 210 of the slave manipulator 220. This base coordinate system serves as a global reference coordinate system, used to describe the absolute position of each part of the slave manipulator 220 in space. The origin and the direction of each coordinate axis of the base coordinate system can be set according to the actual installation layout. For example, the origin of the base coordinate system can be set at the center of the base 210, with the Z-axis pointing vertically upward and the X and Y axes horizontally distributed.

[0050] Secondly, local coordinate systems are established at at least some joints of the slave arm 220. For example, a local coordinate system is fixed at the drive axis or rotation axis of each joint according to the Denavit Hartenberg (DH) parameter method or a modified DH parameter method. The relative motion relationships between the links of the slave arm 220 can be accurately described by the homogeneous transformation matrix between adjacent local coordinate systems.

[0051] It should be noted that the establishment of the local coordinate system includes active joints and passive joints. Among them, the active joints move in real time in response to the control commands of the master manipulator 110 at the master end during operation, and their joint parameters change dynamically over time; the passive joints are used to realize the large-range positioning and pose adjustment of the slave manipulator 220. After being adjusted to the correct position before use, they remain locked. In the kinematic model, they can be treated as fixed transformations, that is, their corresponding homogeneous transformation matrix is ​​a constant matrix.

[0052] After establishing the coordinate system, it is necessary to determine the transformation relationships between the coordinate systems. Specifically, the homogeneous transformation matrices between adjacent coordinate systems are obtained through measurement or calibration, including: a fixed transformation matrix from the base coordinate system to the first joint local coordinate system, and transformation matrices between adjacent joint local coordinate systems that vary with joint parameters. These transformation relationships together constitute a complete kinematic link from the base 210 to the end of the surgical instrument, which is used for subsequent coordinate calculations.

[0053] S112, based on the kinematic model, calculate the local coordinates of the instrument ends on each slave manipulator arm 220 in the endoscope end-effector coordinate system. Specifically, S112 includes the following sub-steps: (1) Obtain the current position of each joint on the hand-operated arm 220.

[0054] Specifically, encoders configured at each joint collect the current position information of the joints in real time. These encoders include, but are not limited to, photoelectric encoders, magnetic encoders, or absolute encoders. Depending on the joint type, corresponding position data is collected: for rotary joints, the rotation angle is collected; for translational joints, the displacement is collected. For active joints, the position information changes in real time with the control commands from the master control unit. The sampling frequency is set according to the system control cycle, for example, it can be set to be the same as the master-slave control frequency to ensure the timeliness and synchronization of the position information. For passive joints, their position information remains unchanged after adjustment before use and can be treated as a constant in the kinematic model. The acquired current position of each joint is used as the input variable for forward kinematics calculation.

[0055] (2) Based on the transformation relationship, calculate the global coordinates of the instrument end on each slave manipulator 220 in the base coordinate system.

[0056] Using the transformation relationships between the coordinate systems established in step S111, and combining them with the current positions of each joint obtained above, the pose of the surgical instrument end effector in the base coordinate system is calculated through forward kinematic chain multiplication. Specifically, starting from the base coordinate system, the transformation proceeds sequentially through the local coordinate systems of each joint and the local coordinate system of the fixed point, finally reaching the coordinate system of the surgical instrument end effector. The transformation matrices between the coordinate systems are multiplied sequentially according to the chain order to obtain the homogeneous transformation matrix of the surgical instrument end effector in the base coordinate system.

[0057] (3) Based on the transformation relationship between local coordinate systems, the transformation matrix from the end of the instrument on the hand operating arm 220 to the end of the endoscope is obtained.

[0058] To transform the spatial position of the instrument tip to the endoscopic imaging coordinate system, it is necessary to establish the relative pose relationship between the instrument tip and the endoscope tip.

[0059] First, an endoscope end-tip coordinate system is established. This coordinate system is fixed to the center point of the endoscope's surface, and its coordinate axes are related to the imaging direction of the endoscope. Since the endoscope is mounted at the distal end of the slave manipulator 220, and the slave manipulator 220 also conforms to the aforementioned kinematic model, the same forward kinematics calculation method as in sub-step (2) is used to obtain the homogeneous transformation matrix of the endoscope end in the base coordinate system. Then, the homogeneous transformation matrix of the instrument end in the base coordinate system obtained in sub-step (2) is multiplied by the inverse matrix of the homogeneous transformation matrix of the endoscope end in the base coordinate system to obtain the transformation matrix from the instrument end coordinate system to the endoscope end coordinate system. This matrix fully describes the spatial pose relationship of the surgical instrument end relative to the center point of the endoscope surface, including the relative positional offset and pose difference between the two.

[0060] (4) Calculate the local coordinates of the instrument end on each slave manipulator 220 in the endoscope end coordinate system according to the transformation matrix.

[0061] By extracting the position vector from the transformation matrix obtained in sub-step (3), the local coordinates of the surgical instrument tip in the endoscope tip coordinate system can be obtained. This local coordinate system contains components in three directions: the horizontal and vertical components determine the two-dimensional projection position of the instrument tip on the display screen, and the component along the optical axis reflects the depth information of the instrument tip relative to the endoscope, used to determine the scaling ratio of the fluoroscopic projection. Through the above calculations, the position description of the instrument tip is transformed into a local coordinate system with the endoscope as the reference, providing input data for subsequent three-dimensional to two-dimensional projection transformation using the endoscope imaging model.

[0062] S113, based on the endoscopic imaging model, obtains the camera intrinsic parameters of the endoscope.

[0063] This step is based on a pre-established endoscopic imaging model (such as a pinhole imaging model), and obtains the camera intrinsic parameters in the model through calibration. Specifically, it includes the following: The checkerboard calibration plate is placed within the endoscopic field of view, and images of the calibration plate in multiple different poses are acquired through the endoscope. Specifically, the pose of the calibration plate is adjusted so that it appears in the endoscopic field of view at different translational positions and rotation angles. To obtain high calibration accuracy, it is preferable to acquire at least 15 images of the calibration plate in at least 15 different poses, and the pose of the checkerboard calibration plate in each pose should cover different areas of the endoscopic field of view as much as possible, including the central area, the edge area, and different degrees of tilt angle, so as to fully reflect the imaging characteristics of the endoscopic lens.

[0064] Furthermore, Zhang's calibration method was used to process the acquired checkerboard calibration board images. This calibration method extracts the sub-pixel coordinates of the checkerboard corner points in each image and, combined with predefined world coordinates of the checkerboard corner points (based on the actual physical size of the calibration board), establishes a correspondence between the pixel coordinates of the corner points and the world coordinates. Using corresponding point pairs from multiple images, a perspective projection equation was constructed based on the endoscope imaging model, and the camera intrinsic parameters of the endoscope were calculated by solving this equation. These camera intrinsic parameters include focal length (including horizontal and vertical focal lengths), principal point coordinates (i.e., the coordinates of the intersection of the optical axis and the image plane), and distortion coefficients. Among them, the distortion coefficients include radial distortion coefficients and tangential distortion coefficients. Radial distortion describes the deformation of the image along the radial direction, and tangential distortion describes the deformation caused by the non-parallelism between the lens and the imaging plane.

[0065] S114, based on the camera intrinsic parameters of the endoscope, converts local coordinates into theoretical two-dimensional coordinates of each instrument tip in the display screen.

[0066] Specifically, for each instrument tip, based on the principle of perspective projection and combined with the focal length in the camera's intrinsic parameters, the three-dimensional coordinates are mapped onto the image plane. Then, radial and tangential distortion corrections are performed using distortion coefficients to obtain the theoretical two-dimensional pixel coordinates of the instrument tip in the displayed image. This process is repeated to obtain the theoretical two-dimensional coordinates of all instrument tips.

[0067] In step S120, the actual two-dimensional coordinates of each instrument end in the display screen are identified and calculated. Specifically, this includes: extracting the edge features of each instrument end in the display screen through image recognition or AI recognition; and calculating the actual pixel coordinates of each instrument end in the display screen based on the extracted edge features.

[0068] This step uses image recognition technology to extract the actual position information of the surgical instrument tips from the display screen on the doctor's console. Specifically, firstly, image data of the main display screen is acquired in real time using an image acquisition card; then, an image recognition algorithm is used to detect and segment the surgical instrument tips in the image, extracting the edge features of each instrument tip. The image recognition algorithm includes a semantic segmentation network based on deep learning or a traditional image processing algorithm; finally, based on the extracted edge features, the actual pixel coordinates of each instrument tip in the display screen are calculated through contour fitting or feature point localization.

[0069] In step S130, the theoretical and actual two-dimensional coordinates of each instrument end during the movement of the master hand are obtained, including: moving the master hand a preset distance and keeping each instrument end within the display screen during the movement; and continuously recording the theoretical and actual two-dimensional coordinates of each instrument end during the movement.

[0070] Specifically, the operator controls the master hand to move a preset distance, ensuring that the ends of each surgical instrument remain within the display screen during the movement to prevent data interruption caused by the instrument ends moving out of the field of view. During the movement, the theoretical and actual two-dimensional coordinates of each surgical instrument end are continuously recorded using a unified time reference. The theoretical coordinates are the calculation results from step S114, and the actual coordinates are the identification results from step S120. The unified time reference achieves clock synchronization between the master and slave hands through a time synchronization protocol, ensuring accurate alignment of theoretical and actual coordinates on the same time axis. After data acquisition, the recorded theoretical coordinate sequence is plotted as a theoretical displacement curve in chronological order, and the actual coordinate sequence is plotted as an actual displacement curve in chronological order, facilitating subsequent calculation of the time difference between the theoretical and actual trajectories.

[0071] In step S140, determining the time difference between the theoretical displacement curve and the actual displacement curve when the displacement difference between the two is within a preset tolerance range includes: finding the time point on the theoretical displacement curve and the actual displacement curve when the displacement value is close to the same within the preset tolerance range; and calculating the time difference between the time points as the master-slave operation delay.

[0072] Specifically, the theoretical displacement curve and the actual displacement curve are compared on the same time axis. The time point on the two curves when the displacement values ​​are nearly identical within a preset tolerance range is found. This preset tolerance range can be set according to the measurement accuracy requirements, for example, it can be set to less than 2 pixels or less than 5% of the diameter of the instrument tip. Figure 5As shown, when the difference between the actual displacement value and the theoretical displacement value at a certain point on the theoretical displacement curve is within the preset tolerance range, the theoretical position and the actual position of that point are considered to be basically consistent. The theoretical time point t1 and the actual time point t2 corresponding to that point are recorded, and the time difference between the two time points is calculated as follows: The time difference This serves as a delay for the master-slave operation at that moment.

[0073] Furthermore, to improve the accuracy and reliability of delay measurement, this embodiment also provides a preferred implementation method: repeatedly performing the master hand movement steps multiple times and recording the time difference corresponding to each operation. Specifically, under the same preset motion trajectory, the master hand is manipulated to move multiple times, and the time difference corresponding to each movement is recorded according to the methods in steps S130 and S140. After completing multiple measurements, the statistical value of the recorded multiple time differences is calculated, and the statistical value is used as the final master-slave operation delay. The statistical value includes, but is not limited to, the average, median, or the corrected average after outlier removal. By performing multiple measurements and using the statistical value as the final result, random errors and accidental interference that may exist in a single measurement can be effectively eliminated, improving the accuracy and stability of delay measurement, and making the master-slave control compensation of the remote surgical robot more reliable.

[0074] Secondly, embodiments of this application provide an apparatus for measuring the master-slave operation delay of a remote surgical robot, such as... Figure 6 As shown, the device includes: a theoretical coordinate calculation module, an actual coordinate identification module, a displacement curve construction module, and a delay determination module. Specifically: The theoretical coordinate calculation module 10 is used to calculate the theoretical two-dimensional coordinates of the instrument end on each slave manipulator 220 in the display screen based on the mechanical structure of the slave manipulator 220 and the endoscope imaging model. The actual coordinate recognition module 20 is used to acquire the display screen of the main handpiece, identify and calculate the actual two-dimensional coordinates of each instrument end in the display screen; The displacement curve construction module 30 is used to obtain the theoretical two-dimensional coordinates and actual two-dimensional coordinates of the ends of each instrument during the movement of the main hand end, and to construct displacement curves of the two as a function of time. The delay determination module 40 is used to determine the time difference between the theoretical displacement curve and the actual displacement curve when the displacement difference between the two is within a preset tolerance range, and uses this as the master-slave operation delay.

[0075] The remote surgical robot master-slave operation delay measurement device provided in this application embodiment calculates the theoretical two-dimensional coordinates of the instrument end effector based on the mechanical structure of the slave manipulator 220 and the endoscopic imaging model using the theoretical coordinate calculation module 10. These theoretical coordinates reflect the expected position of the instrument based on the kinematic model. Simultaneously, the actual coordinate recognition module acquires the actual two-dimensional coordinates of the instrument end effector in the master end display screen. These actual coordinates represent the actual position observed from the acquisition of the hand-end image to the master end display screen. The displacement curve construction module 30 records the theoretical and actual coordinates synchronously over time, constructing displacement curves for both over time. The theoretical motion trajectory of the instrument end effector is compared with the actual observed trajectory on the time axis. The delay determination module 40 calculates the time offset between the two curves when the displacement difference is within a preset tolerance range, thus achieving accurate measurement of the master-slave operation delay.

[0076] This device effectively measures the overall latency, including master-slave command transmission, instrument execution response, image acquisition and transmission, and screen display. It overcomes the limitations of existing technologies, which can only measure local master-slave control latency and cannot cover remote transmission and visual feedback links. By comparing displacement curves, the master-slave command transmission and image transmission are clearly reflected. The measured latency accurately reflects the actual time difference between the surgeon's commands and visual feedback during remote surgery, providing a basis for improving the real-time response and control precision of master-slave operations. Furthermore, this device does not rely on additional external measuring equipment; it can complete latency measurements entirely based on the surgical robot system's own kinematic and image data, facilitating its implementation in real-world clinical scenarios.

[0077] Thirdly, embodiments of this application provide an electronic device, such as... Figure 7 As shown, the electronic device includes a memory 131 and a processor 130. The memory 131 stores a computer program, and the processor 130 runs the computer program to make the electronic device perform the above-described method.

[0078] Furthermore, combined Figure 7 The electronic device shown also includes a bus 132 and a communication interface 133, with the processor 130, the communication interface 133 and the memory 131 connected via the bus 132.

[0079] The memory 131 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 133 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 132 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0080] Processor 130 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 130 or by instructions in software form. Processor 130 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 131, and processor 130 reads the information in memory 131 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0081] Fourthly, embodiments of this application provide a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0083] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0084] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0086] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for measuring the master-slave operation delay of a remote surgical robot, characterized in that, include: Based on the mechanical structure of the hand-operated arm and the endoscope imaging model, calculate the theoretical two-dimensional coordinates of the instrument tip on each of the hand-operated arms in the display screen; Acquire the display screen of the master handpiece, identify and calculate the actual two-dimensional coordinates of each instrument endpiece in the display screen; The theoretical two-dimensional coordinates and the actual two-dimensional coordinates of each instrument end during the movement of the main hand end are obtained, and displacement curves of the two as a function of time are constructed respectively. When the displacement difference between the two is within a preset tolerance range, the time difference between the theoretical displacement curve and the actual displacement curve is used as the master-slave operation delay.

2. The method for measuring the master-slave operation delay of a remote surgical robot according to claim 1, characterized in that, The calculation of the theoretical two-dimensional coordinates of the instrument ends on each slave arm in the display screen, based on the mechanical structure of the slave arm and the endoscopic imaging model, includes: Based on the mechanical structure of the hand-operated arm, a kinematic model from the base to the end of the device is established; Based on the kinematic model, calculate the local coordinates of the instrument tip on each of the manipulator arms in the endoscope tip coordinate system; Based on the endoscopic imaging model, obtain the camera intrinsic parameters of the endoscope; Based on the camera intrinsic parameters of the endoscope, the local coordinates are converted into theoretical two-dimensional coordinates of each instrument tip in the display screen.

3. The method for measuring the master-slave operation delay of a remote surgical robot according to claim 2, characterized in that, The step of establishing a kinematic model from the base to the end effector based on the mechanical structure of the hand-operated arm includes: Based on the mechanical structure of the slave arm, a base coordinate system is established at the base, and a local coordinate system is established at at least some joints of the slave arm, thereby determining the transformation relationship between the coordinate systems.

4. The method for measuring the master-slave operation delay of a remote surgical robot according to claim 3, characterized in that, The calculation of the local coordinates of each instrument tip on the manipulator arm in the endoscopic endoscope coordinate system based on the kinematic model includes: Obtain the current position of each joint on the hand-operated arm; Based on the transformation relationship, calculate the global coordinates of the instrument end of each slave manipulator arm in the base coordinate system; Based on the transformation relationship between the local coordinate systems, the transformation matrix from the end of the instrument on the hand operating arm to the end of the endoscope is obtained; Based on the transformation matrix, calculate the local coordinates of the instrument tip on each slave manipulator arm in the endoscope end coordinate system.

5. The method for measuring the master-slave operation delay of a remote surgical robot according to claim 2, characterized in that, The acquisition of camera intrinsic parameters for the endoscope includes: Images of the checkerboard calibration board in multiple different orientations were acquired using an endoscope; The Zhang calibration method was used to process the acquired checkerboard calibration board images to obtain the camera intrinsic parameters of the endoscope.

6. The method for measuring the master-slave operation delay of a remote surgical robot according to claim 1, characterized in that, The process of identifying and calculating the actual two-dimensional coordinates of each instrument end in the display screen includes: The edge features of each instrument's end in the displayed screen are extracted using image recognition or AI recognition. The actual pixel coordinates of each instrument end in the displayed screen are calculated based on the extracted edge features.

7. The method for measuring the master-slave operation delay of a remote surgical robot according to claim 1, characterized in that, The acquisition of the theoretical two-dimensional coordinates and the actual two-dimensional coordinates of each instrument end during the movement of the main hand includes: The operator moves the master hand a preset distance, ensuring that the ends of each instrument remain within the display screen during the movement; During the movement, the theoretical two-dimensional coordinates and the actual two-dimensional coordinates of each instrument end are continuously recorded.

8. The method for measuring the master-slave operation delay of a remote surgical robot according to claim 1, characterized in that, The determination of the time difference between the theoretical displacement curve and the actual displacement curve when the displacement difference between the two is within a preset tolerance range includes: On the theoretical displacement curve and the actual displacement curve, find the time point corresponding to when the displacement value is nearly consistent within a preset tolerance range; Calculate the time difference between the aforementioned time points as the master-slave operation delay.

9. The method for measuring the master-slave operation delay of a remote surgical robot according to claim 8, characterized in that, Also includes: Repeat the steps of moving the main hand multiple times, and record the time difference for each operation; Calculate the statistical value of the time difference multiple times, and use it as the final master-slave operation delay.

10. An apparatus for measuring the master-slave operation delay of a remote surgical robot based on any one of claims 1-9, characterized in that, include: The theoretical coordinate calculation module is used to calculate the theoretical two-dimensional coordinates of the instrument end on each of the slave manipulator arms in the display screen based on the mechanical structure of the slave manipulator arm and the endoscope imaging model. The actual coordinate recognition module is used to acquire the display screen of the main handpiece, identify and calculate the actual two-dimensional coordinates of each instrument endpiece in the display screen; The displacement curve construction module is used to obtain the theoretical two-dimensional coordinates and the actual two-dimensional coordinates of each instrument end during the movement of the main hand end, and to construct displacement curves of the two as a function of time. The delay determination module is used to determine the time difference between the theoretical displacement curve and the actual displacement curve when the displacement difference between the two is within a preset tolerance range, and uses this as the master-slave operation delay.

11. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the remote surgical robot master-slave operation delay measurement method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium stores computer program instructions, which are read and executed by a processor to perform the remote surgical robot master-slave operation delay measurement method according to any one of claims 1-9.

Images