Force-driven and vision-driven hybrid control method and system for laparoscope field of view adjustment

Abstract

The present invention provides a hybrid control method, system, storage medium and electronic device for adjusting the field of view of a laparoscope driven by force and vision, and relates to the field of laparoscope field of view adjustment. The present invention integrates the interactive force / torque captured by the integrated tactile handle, the virtual force / torque generated based on visual information, and the preset viscous force / torque into a mirror-holding dynamics model with wound point constraints, generates the movement of the laparoscope support robot, and adjusts the laparoscope field of view through hybrid control. This method eliminates the need to frequently switch control modes during the adjustment of the laparoscope field of view, and can simultaneously achieve visual tracking and tactile interactive guidance, taking into account the use of visual drive for continuous field of view adjustment and the use of force drive for large-scale field of view adjustment. In addition, the gain of the interactive force / torque can be adjusted by the grip force on the integrated tactile handle, so that the interaction force generated by the doctor's accidental touch will not move the laparoscope support robot, thereby ensuring the safety of the doctor's operation.

Description

Technical Field

[0001] The present invention relates to the field of laparoscope field of view adjustment, and in particular to a force-driven and vision-driven laparoscope field of view adjustment hybrid control method, system, storage medium and electronic equipment. Background Art

[0002] Laparoscopic support robots can improve the stability and accuracy of visualization in minimally invasive surgery.

[0003] For example, patent CN107440748A discloses an intelligent automatic tracking laparoscope system for the surgical field, including a fully automatic pan-tilt head, a control handle, a camera and lighting unit, a video processor and a display; one end of the fully automatic pan-tilt head is fixed on the control handle, and the camera and lighting unit is fixed on the other end of the fully automatic pan-tilt head. The camera and lighting unit includes a fixed frame and a light source and a camera arranged in the fixed frame. The video processor is connected to the display, the fully automatic pan-tilt head and the camera.

[0004] For example, patent CN113143461A provides a minimally invasive endoscope-holding robot system for human-machine collaboration. This system obtains the endoscope view and robot pose; obtains the status of each robot joint and solves the endoscope camera pose based on the robot's forward kinematics principle; uses the YOLOv3 algorithm to detect the position of the surgical instrument tip in the endoscope view; obtains the distance from the surgical instrument tip to the center of the view based on the position of each surgical instrument tip in the view; obtains the visual tracking vector based on the surgical instrument tip position, the distance from the surgical instrument tip to the center of the view, and the camera parameters; obtains the endoscope insertion distance constraint vector; and obtains the speed of each joint of the collaborative robot arm based on the visual tracking vector and the endoscope insertion distance constraint vector.

[0005] It can be seen that when the existing laparoscope support robot relies solely on visual servoing, it cannot move efficiently and quickly in large-scale field of view adjustment tasks such as searching for human tissue and suture needles outside the field of view, and has certain limitations. Summary of the Invention

[0006] (1) Technical problems solved

[0007] In response to the shortcomings of the existing technology, the present invention provides a force-driven and vision-driven hybrid control method, system, storage medium and electronic device for laparoscope field of view adjustment, which solves the technical problem of being unable to move quickly in large-scale field of view adjustment tasks when relying solely on visual servoing.

[0008] (2) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] A hybrid control method for adjusting the field of view of a laparoscope using force-driven and vision-driven techniques is provided. The method is based on a laparoscope support robot comprising an integrated tactile handle, an end effector, and a robotic arm. The method comprises:

[0011] Construct a dynamic model of the laparoscopic support robot based on wound point constraints;

[0012] Constructing interactive force / torque through the integrated tactile handle;

[0013] Construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view;

[0014] Obtaining input of a dynamic model of the laparoscope-supported robot according to the interaction force / torque, the virtual force / torque, and the preset viscous force / torque to generate an angular acceleration of the end effector;

[0015] The angular acceleration is converted into the joint angular velocity of the robotic arm, and the field of view of the laparoscope is adjusted by hybrid control.

[0016] Preferably, constructing a four-degree-of-freedom dynamic model of a laparoscope-supported robot includes:

[0017] The laparoscope motion based on the incision point constraint is simplified to continuously passing through a fulcrum. The four-degree-of-freedom rigid body motion refers to the fulcrum Three different rotations are performed along the longitudinal axis z of the cavity mirror. R′ Perform a translation; it represents the angular velocity of the cavity mirror relative to the current fulcrum coordinate system {R′} and line speed The vector is:

[0018]

[0019] in, represents the angular velocity and linear velocity of the end effector, x R′ ,y R′ and z R′ Respectively represent the coordinate axes of the x, y, and z directions of the coordinate system {R′};

[0020] According to the Newton-Euler equation, the dynamic equation of the laparoscope support robot in four degrees of freedom is calculated as follows:

[0021]

[0022] in, represents the sum of the moments acting on the cavity mirror, ∑F z Indicates the action on the longitudinal axis z of the cavity mirror R′ The sum of the forces above, represents the angular acceleration of the end effector;

[0023]

[0024] in, m represents the inertia matrix and mass of the cavity mirror respectively; the cavity mirror is regarded as a long cylinder with a base radius of e and a height of L, then Indicates the length of the extracorporeal part of the laparoscope; Representation and vector The associated skew-symmetric matrix.

[0025] Preferably, the integrated tactile handle includes a force / torque sensor and a flexible sensor array, and the process of constructing the interactive force / torque includes:

[0026] The force / torque acting on the force / torque sensor Expressed as:

[0027]

[0028] in, Respectively represent the readings of the force / torque sensor; x H′ ,y H′ and z H′ Represent the three coordinate axes of the current handle coordinate system {H′} relative to the current fulcrum coordinate system {R′};

[0029] The interaction force / torque The calculation process is as follows:

[0030]

[0031] in, represent the moment and force of the interaction force / torque, respectively;

[0032] δ is the interaction force / torque gain, defined as:

[0033]

[0034] Among them, Se j represents the measurement value of the jth sensing element on the flexible sensor array; k H Represents a constant; n c represents the number of sensing elements subjected to contact force, k min represents the minimum number of sensing elements required to apply actuation to control the cavity mirror; J represents the number of sensing elements on the flexible sensor array.

[0035] Preferably, the virtual force / torque The calculation process is as follows:

[0036]

[0037] in, The moment and force of the virtual force / torque are represented respectively; x C′ ,y C′ and z C′ Represents the three coordinate axes of the current laparoscope coordinate system {C′} relative to the current fulcrum coordinate system {R′}; n s Indicates the number of surgical instruments in the image; tracks the position of surgical instruments in the current laparoscope coordinate system {C′} Visual tracking ability Expressed as:

[0038] F i =F(d i )·V i (10)

[0039] Among them, d i ,F(d i ) and V i The calculation of is as follows:

[0040]

[0041] Among them, V i Indicates that the laparoscope needs to move to track The unit direction of The position of the surgical instrument in the laparoscope coordinate system scaled by unit depth is expressed as:

[0042]

[0043] Among them, f x , f y , u0 and v0 are camera internal parameters; express The distance to the center of the image; the function F(d i )∈[0,F max ] proves continuity and differentiability within its domain, F max Indicates tracking The maximum force that can be achieved.

[0044] Preferably, the viscous force / torque The calculation process is as follows:

[0045]

[0046] in, are the moment and force of viscous force / torque respectively; k f and k t are viscous resistance parameters.

[0047] Preferably, obtaining the input of the dynamic model of the laparoscope supporting robot includes:

[0048] According to the rigid transformation of the coordinate system on the cavity mirror, it can be deduced that:

[0049]

[0050] Then the input of the dynamic model of the laparoscope supporting robot [∑M ∑F z ] T Expressed as:

[0051]

[0052] Preferably, converting the angular acceleration into the joint angular velocity of the robotic arm comprises:

[0053] Since the cavity mirror and the actuator are rigid bodies, the angular acceleration of the end effector and the angular acceleration of the cavity mirror are both And converted into quaternion velocity

[0054]

[0055] Where Δt is the differential of time;

[0056] For the initial fulcrum coordinate system {R}, the new attitude of the cavity mirror is expressed by quaternion The calculation is as follows:

[0057]

[0058] in, Quaternion form representing cavity mirror rotation express The rotation part; Qm(·) represents quaternion multiplication;

[0059] Translation of the cavity mirror relative to the coordinate system {R} The calculation is as follows:

[0060]

[0061] in, express The translation part of

[0062] The new position of the laparoscope Calculated relative to the coordinate system {R} as:

[0063]

[0064] in, Represents a function that converts a quaternion into a rotation matrix;

[0065] After the coordinate system transformation, the end effector coordinate system {E′ N The transformation of {E′} and {E′} relative to the coordinate system {B} is calculated as:

[0066]

[0067] in, represents the fixed coordinate system transformation recorded by the laparoscope supporting robot in the initial state. Since the laparoscope is a rigid body, the transformation relationship between the coordinate systems {E} and {R} remains unchanged.

[0068] Through real-time Jacobian transformation, the joint angular velocity of the robotic arm Calculated as:

[0069]

[0070] in, It represents the real-time Jacobian transformation relationship of the laparoscope supporting robot.

[0071] A force-driven and vision-driven hybrid control system for adjusting the field of view of a laparoscope is based on a laparoscope support robot. The laparoscope support robot includes an integrated tactile handle, an end effector, and a robotic arm. The system includes:

[0072] Model building module, used to build a dynamic model of the laparoscope support robot based on wound point constraints;

[0073] A tactile force sensing module, configured to construct an interactive force / torque through the integrated tactile handle;

[0074] The instrument tracking module is used to construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view;

[0075] a velocity generation module, configured to obtain input of a dynamic model of the laparoscope-supported robot based on the interaction force / torque, the virtual force / torque, and the preset viscous force / torque, so as to generate an angular acceleration of the end effector;

[0076] The velocity conversion module is used to convert the angular acceleration into the joint angular velocity of the robotic arm, so as to adjust the field of view of the laparoscope by hybrid control.

[0077] A storage medium stores a computer program for hybrid control of laparoscope field of view adjustment driven by force and vision, wherein the computer program enables a computer to execute the hybrid control method of laparoscope field of view adjustment as described above.

[0078] An electronic device, comprising:

[0079] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the laparoscope field of view adjustment hybrid control method as described above.

[0080] (3) Beneficial effects

[0081] The present invention provides a hybrid control method, system, storage medium, and electronic device for adjusting the field of view of a laparoscope using force-driven and vision-driven methods. Compared with existing technologies, it has the following advantages:

[0082] This method integrates interactive forces / torques captured by a tactile handle, virtual forces / torques generated based on visual information, and preset viscous forces / torques into a scope-holding dynamics model with wound-point constraints. This generates the motion of the laparoscope support robot, using hybrid control to adjust the laparoscope's field of view. This method eliminates the need for frequent switching of control modes during laparoscope field of view adjustment, enabling simultaneous visual tracking and tactile interactive guidance, while balancing continuous field of view adjustment using vision-driven control with large-scale field of view adjustment using force-driven control. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0084] Figure 1 A technical schematic diagram of a hybrid control method for adjusting the field of view of a laparoscope using force-driven and vision-driven methods provided by an embodiment of the present invention;

[0085] Figure 2 A block diagram of a hybrid control method for adjusting the field of view of a laparoscope using force-driven and vision-driven methods provided by an embodiment of the present invention;

[0086] FIG3( a ) is a schematic diagram of a laparoscope support robot under wound point constraint provided by an embodiment of the present invention;

[0087] FIG3( b ) is a schematic diagram of a coordinate system definition of a laparoscope support robot provided in an embodiment of the present invention;

[0088] Figure 4 A schematic diagram of an experimental platform provided by an embodiment of the present invention;

[0089] FIG5( a ) is a snapshot of an experiment provided by an embodiment of the present invention;

[0090] FIG5( b ) is a force / torque applied to a laparoscope supporting robot according to an embodiment of the present invention;

[0091] FIG5( c ) is a schematic diagram of a cavity mirror trajectory provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0092] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0093] The embodiments of the present application provide a hybrid control method, system, storage medium and electronic device for laparoscope field of view adjustment driven by force and vision, thereby solving the technical problem of being unable to move efficiently in large-scale field of view adjustment tasks when relying solely on visual servoing, while taking into account both the use of vision drive for continuous field of view adjustment and the use of force drive for large-scale field of view adjustment.

[0094] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0095] The solution provided by the embodiment of the present invention is a laparoscope support robot system that adopts visual drive to perform continuous field of view adjustment and adopts force drive to perform large-scale field of view adjustment.

[0096] Specifically, a laparoscopic support robot system with an integrated tactile handle was designed, which can measure the human-machine interaction force during the tracking of the surgical area. Furthermore, a hybrid control method of force-driven and vision-driven laparoscopic field of view adjustment was proposed. Figure 1 As shown in the figure, this method integrates interactive forces / torques captured by the integrated tactile handle, virtual forces / torques generated based on visual information, and preset viscous forces / torques into a laparoscopic dynamics model with wound point constraints. This generates robot motion and uses hybrid control to adjust the laparoscopic field of view. Furthermore, the grip force on the integrated tactile handle can be used to adjust the gain of the interactive forces / torques, ensuring that the interaction forces generated by the doctor's accidental touch will not cause the laparoscopic support robot to move, ensuring the safety of the doctor's operation.

[0097] Therefore, this method eliminates the need to frequently switch control modes during laparoscope field of view adjustment and can simultaneously achieve visual tracking and tactile interactive guidance.

[0098] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0099] Example 1:

[0100] like Figure 2 As shown, an embodiment of the present invention provides a force-driven and vision-driven hybrid control method for adjusting the field of view of a laparoscope, based on a laparoscope support robot, wherein the laparoscope support robot includes an integrated tactile handle, an end effector, and a robotic arm; the method includes:

[0101] S1. Construct a dynamic model of the laparoscopic support robot based on wound point constraints;

[0102] S2. constructing an interactive force / torque through the integrated tactile handle;

[0103] S3, constructing a virtual force / torque based on the position of the surgical instrument in the laparoscope field of view;

[0104] S4. Obtaining input of the dynamic model of the laparoscope supporting robot according to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque to generate an angular acceleration of the end effector;

[0105] S5. Convert the angular acceleration into the joint angular velocity of the robotic arm, and adjust the field of view of the laparoscope by hybrid control.

[0106] The embodiments of the present invention eliminate the need for frequent switching of control modes during laparoscope field of view adjustment, can simultaneously achieve visual tracking and tactile interactive guidance, and take into account both the use of visual drive for continuous field of view adjustment and the use of force drive for large-scale field of view adjustment.

[0107] It is necessary to first add the definition of the coordinate system of the laparoscope support robot:

[0108] The wound-site-constrained laparoscope support robot is shown in Figure 3(a). It includes an integrated tactile handle, an end effector, and a robotic arm (not shown). A six-dimensional force / torque sensor is attached to the robot's end effector to measure the force and torque applied by the surgeon. The laparoscope's motion is driven by the robot while adhering to the wound-site constraint.

[0109] Figure 3(b) gives the specific definition of the coordinate system: the coordinate system {B} is used as the base coordinate system of the robot system. When the robot is in the initial posture, the coordinate system of the laparoscope support robot system is represented by {*}; when the robot moves, the current coordinate system of the laparoscope support robot system is represented by {*′}; when the robot is in a new posture, the coordinate system of the laparoscope support robot system is represented by {*′ N}.

[0110] The fulcrum coordinate system {R} is defined to be located at the wound point when the robot is in the initial position. The origin of the fulcrum coordinate system {R} is As the incision point. {R′} is the current fulcrum coordinate system, its origin Located at the wound site. The robot’s end effector coordinate system is denoted as {E} and {E′}, with its origin located on the axis of the laparoscope. The homogeneous transformation matrix between the end effector and the base coordinate system is is the joint angle reading of the robot, where λ represents the degree of freedom of the robot. The tactile handle coordinate system is denoted as {H} and {H′}, and the laparoscope coordinate system is denoted as {C} and {C′}.

[0111] On this basis, the following steps will be introduced in detail:

[0112] In step S1, a dynamic model of a laparoscope support robot based on wound point constraints is constructed.

[0113] The laparoscope is inserted into the patient's body through a natural opening or a small incision, and the movement of the external robotic arm must be constrained by the incision point.

[0114] Specifically, this step simplifies the laparoscope motion based on the wound point constraint to continuously passing through a fulcrum. The four-degree-of-freedom rigid body motion refers to the fulcrum Three different rotations are performed along the longitudinal axis z of the cavity mirror. R′ Perform a translation; it represents the angular velocity of the cavity mirror relative to the current fulcrum coordinate system {R′} and line speed The vector is:

[0115]

[0116] in, represents the angular velocity and linear velocity of the end effector, x R′ ,y R′ and z R′ Respectively represent the coordinate axes of the x, y, and z directions of the coordinate system {R′};

[0117] According to the Newton-Euler equation, the dynamic equation of the laparoscope support robot in four degrees of freedom is calculated as follows:

[0118]

[0119] in, represents the sum of the moments acting on the cavity mirror, ∑F z Indicates the action on the longitudinal axis z of the cavity mirror R′ The sum of the forces above, represents the angular acceleration of the end effector;

[0120]

[0121] in, m represents the inertia matrix and mass of the cavity mirror respectively; the cavity mirror is regarded as a long cylinder with a base radius of e and a height of L, then Indicates the length of the extracorporeal part of the laparoscope; Representation and vector The associated skew-symmetric matrix.

[0122] After building the dynamic model of the laparoscope support robot based on the wound point constraint, it is necessary to pay attention to the following Figure 1 As shown, the embodiment of the present invention proposes to use interactive force / torque, virtual force / torque and viscous force / torque to hybrid control and adjust the field of view of the laparoscope. For details, please refer to the following introduction:

[0123] In step S2, an interactive force / torque is constructed through the integrated tactile handle.

[0124] The laparoscopic support robot provided by the present invention differs from traditional laparoscopic support robots. During the autonomous tracking of surgical instruments, the surgeon cannot intervene in the robot's movements, resulting in the robot not being able to perfectly coordinate with the surgeon's surgical procedures. As shown in Figure 3(a), the present invention provides a laparoscopic support robot system with an integrated tactile handle. The integrated tactile handle includes a force / torque sensor and a flexible sensor array.

[0125] Accordingly, the interaction force / torque is determined by the force / torque applied by the doctor on the force / torque sensor and the grip force from the flexible sensor array. Specifically, the interaction force / torque includes a parameter adaptive adjustment mechanism to prevent any unintentional touch or contact from affecting the force handle.

[0126] The force / torque acting on the force / torque sensor Expressed as:

[0127]

[0128] in, Respectively represent the readings of the force / torque sensor; x H′ ,y H′ and z H′ Represent the three coordinate axes of the current handle coordinate system {H′} relative to the current fulcrum coordinate system {R′};

[0129] The interaction force / torque The calculation process is as follows:

[0130]

[0131] in, represent the moment and force of the interaction force / torque, respectively;

[0132] δ is the interaction force / torque gain, defined as:

[0133]

[0134] Among them, Se j represents the measurement value of the jth sensing element on the flexible sensor array; k H Represents a constant; n c represents the number of sensing elements subjected to contact force, k min represents the minimum number of sensing elements required to apply actuation to control the cavity mirror; J represents the number of sensing elements on the flexible sensor array.

[0135] In step S3, a virtual force / torque is constructed according to the position of the surgical instrument in the laparoscope field of view.

[0136] Virtual force / torque Used to prevent the instrument from moving out of the field of view. The calculation is as follows:

[0137]

[0138] in, The moment and force of the virtual force / torque are represented respectively; x C′ ,y C′ and z C′ Represents the three coordinate axes of the current laparoscope coordinate system {C′} relative to the current fulcrum coordinate system {R′}; n s Indicates the number of surgical instruments in the image; tracks the position of surgical instruments in the current laparoscope coordinate system {C′} Visual tracking ability Expressed as:

[0139] F i =F(d i )·V i (10)

[0140] Among them, d i ,F(d i ) and V i The calculation of is as follows:

[0141]

[0142]

[0143] Among them, V i Indicates that the laparoscope needs to move to track The unit direction of The position of the surgical instrument in the laparoscope coordinate system scaled by unit depth is expressed as:

[0144]

[0145] Among them, f x , f y , u0 and v0 are camera internal parameters; express The distance to the center of the image; the function F(d i )∈[0,F max ] proves continuity and differentiability within its domain, F max Indicates tracking The maximum force that can be achieved.

[0146] Furthermore, in order to ensure the stability of the system, the embodiment of the present invention also proposes a viscous force / torque It combines viscous drag to constrain the four-degree-of-freedom motion of the cavity mirror. The calculation process is as follows:

[0147]

[0148] in, are the moment and force of viscous force / torque respectively; k f and k t are viscous resistance parameters.

[0149] In step S4, the input of the dynamic model of the laparoscope-supported robot is obtained according to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque to generate the angular acceleration of the end effector.

[0150] Since the coordinate system on the cavity mirror is a rigid transformation and does not involve mutual rotation, it can be deduced that:

[0151]

[0152] Then the input of the dynamic model of the laparoscope supporting robot [∑M ∑F z ] T Expressed as:

[0153]

[0154] Finally, according to formula (3), the angular acceleration of the end effector can be obtained

[0155] In step S5, the angular acceleration is converted into the joint angular velocity of the robot arm, and the field of view of the laparoscope is adjusted by hybrid control.

[0156] Since the cavity mirror and the actuator are rigid bodies, the angular acceleration of the end effector and the angular acceleration of the cavity mirror are both

[0157] In order to avoid the universal joint deadlock problem during the robot speed conversion process, the angular acceleration of the cavity mirror is Convert to quaternion velocity

[0158]

[0159] Where Δt is the differential of time;

[0160] For the initial fulcrum coordinate system {R}, the new attitude of the cavity mirror is expressed by quaternion The calculation is as follows:

[0161]

[0162] in, Quaternion form representing cavity mirror rotation express The rotation part; Qm(·) represents quaternion multiplication;

[0163] Translation of the cavity mirror relative to the coordinate system {R} The calculation is as follows:

[0164]

[0165] in, express The translation part of

[0166] The new position of the laparoscope Calculated relative to the coordinate system {R} as:

[0167]

[0168] in, Represents a function that converts a quaternion into a rotation matrix;

[0169] After the coordinate system transformation, the end effector coordinate system {E′ N The transformation of {E′} and {E′} relative to the coordinate system {B} is calculated as:

[0170]

[0171] in, represents the fixed coordinate system transformation recorded by the laparoscope supporting robot in the initial state. Since the laparoscope is a rigid body, the transformation relationship between the coordinate systems {E} and {R} remains unchanged.

[0172] Through real-time Jacobian transformation, the joint angular velocity of the robotic arm Calculated as:

[0173]

[0174] in, It represents the real-time Jacobian transformation relationship of the laparoscope supporting robot.

[0175] Thus, the embodiment of the present invention has completed the hybrid control of the laparoscope field of view adjustment driven by force and vision.

[0176] In order to verify the effectiveness of the method proposed in the embodiment of the present invention, we use Figure 4 The experimental platform shown is used for experimental testing:

[0177] The wound point is set on the laparoscope axis, and the initial end effector posture is adjusted. Perform fixed coordinate system transformation Determine the coordinate system posture of the wound point The model is integrated every 1 ms using the Euler method to generate reference joint positions for the robot. This Franka robot control mode has high tracking accuracy. The laparoscope system is mounted on the robot's end effector, which communicates with the computer via a USB port. The camera frame rate is set to 25 fps. The camera calibration method proposed by Zhang et al. is used to calibrate the laparoscope's internal parameters. In addition, the YOLO v7 algorithm is used to achieve high-performance, low-latency surgical instrument detection.

[0178] In minimally invasive surgery, due to the limited field of view, surgeons need to make extensive laparoscopic movements to locate suture needles outside their current field of view. In this experiment, the operator's goal was to locate a suture needle placed on a simulated organ. The operator first controlled the movement of the surgical instrument in a localized area. Then, the operator manipulated a tactile handle to facilitate large-scale adjustments in the field of view to locate the suture needle.

[0179] The experimental results are shown in Figure 5. The experimental snapshot is shown in Figure 5(a). As shown in Figure 5(b), the interaction force / torque and virtual forces / torques Figure 5(c) shows the trajectory of the laparoscope axis, where the light-colored bold lines correspond to the initial pose, the dark-colored bold lines correspond to the final pose, and the black dots represent the pivot points. The results show that throughout the experiment, the laparoscope's motion consistently adheres to the RCM constraints (i.e., the wound point constraints described above).

[0180] In the initial stage, the operator manipulates the surgical instruments on the artificial organ to generate virtual forces / torques to track the movement of the instrument (from 0 to t3). At t4, the surgical instrument stops moving and remains in the center area of ​​the screen. The operator then interacts with the force / torque Applied to the tactile handle, the laparoscope is controlled to find the position of the suture needle. At time t5, the interaction force / torque Exceeding virtual force / torque This causes the laparoscope to deviate from the current viewing angle. After t6, the instrument is no longer visible in the image. The operator guides the laparoscope to precisely locate the suture needle through a noticeable change in the field of view. At t7, the suture needle appears at the edge of the screen. At this point, the operator releases the tactile handle, generating a virtual force / torque This facilitates quick repositioning of surgical instruments back to the center of the screen. 10 ), with the help of laparoscopic tracking, the surgeon successfully removed the suture needle. As the internal force / torque of the system, it is not shown in the figure.

[0181] Therefore, it is clear that this method can not only perform continuous field of view adjustment under the guidance of surgical instruments, but also can be well adapted to large-scale field of view adjustment tasks such as finding suture needles outside the field of view.

[0182] Example 2:

[0183] An embodiment of the present invention provides a force-driven and vision-driven hybrid control system for adjusting the field of view of a laparoscope. The system is based on a laparoscope support robot. The laparoscope support robot includes an integrated tactile handle, an end effector, and a robotic arm. The system includes:

[0184] Model building module, used to build a dynamic model of the laparoscope support robot based on wound point constraints;

[0185] A tactile force sensing module, configured to construct an interactive force / torque through the integrated tactile handle;

[0186] The instrument tracking module is used to construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view;

[0187] a velocity generation module, configured to obtain input of a dynamic model of the laparoscope-supported robot based on the interaction force / torque, the virtual force / torque, and the preset viscous force / torque, so as to generate an angular acceleration of the end effector;

[0188] The velocity conversion module is used to convert the angular acceleration into the joint angular velocity of the robotic arm, so as to adjust the field of view of the laparoscope by hybrid control.

[0189] Example 3:

[0190] An embodiment of the present invention provides a storage medium storing a computer program for force-driven and vision-driven hybrid control of laparoscope field of view adjustment, wherein the computer program enables a computer to execute the laparoscope field of view hybrid control method as described above.

[0191] Example 4:

[0192] An embodiment of the present invention provides an electronic device, including:

[0193] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the laparoscope field of view adjustment hybrid control method as described above.

[0194] It can be understood that the force-driven and vision-driven laparoscope field of view adjustment hybrid control system, storage medium and electronic device provided in the embodiments of the present invention correspond to the force-driven and vision-driven laparoscope field of view adjustment hybrid control method provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the laparoscope field of view adjustment hybrid control method, and will not be repeated here.

[0195] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0196] 1. The embodiments of the present invention eliminate the need to frequently switch control modes during laparoscope field of view adjustment, and can simultaneously achieve visual tracking and tactile interactive guidance, taking into account both the use of visual drive for continuous field of view adjustment and the use of force drive for large-scale field of view adjustment.

[0197] 2. The embodiment of the present invention adjusts the gain of the interactive force / torque by integrating the grip force on the tactile handle, so that the interactive force generated by the doctor's accidental touch will not cause the robot to move, ensuring the safety of the doctor's operation.

[0198] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0199] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A hybrid control system for adjusting the field of view of a cavity mirror driven by force and vision, characterized in that: Based on a laparoscope support robot, the laparoscope support robot includes an integrated tactile handle, an end effector and a robotic arm; the system includes: Model building module, used to build a dynamic model of the laparoscope support robot based on wound point constraints; A tactile force sensing module, configured to construct an interactive force / torque through the integrated tactile handle; The instrument tracking module is used to construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view; a velocity generation module, configured to obtain input of a dynamic model of the laparoscope-supported robot based on the interaction force / torque, the virtual force / torque, and the preset viscous force / torque, so as to generate an angular acceleration of the end effector; A velocity conversion module, configured to convert the angular acceleration into the joint angular velocity of the robotic arm, so as to adjust the field of view of the laparoscope by hybrid control; Construct a four-degree-of-freedom dynamic model of the laparoscope support robot, including: The laparoscope motion based on the incision point constraint is simplified to continuously passing through a fulcrum. The four-degree-of-freedom rigid body motion refers to the fulcrum Three different rotations are performed along the longitudinal axis z of the cavity mirror. R′ Perform a translation; it represents the angular velocity of the cavity mirror relative to the current fulcrum coordinate system {R′} and line speed The vector is: in, represents the angular velocity and linear velocity of the end effector, x R′ ,y R′ and z R′ Respectively represent the coordinate axes of the x, y, and z directions of the coordinate system {R′}; According to the Newton-Euler equation, the dynamic equation of the laparoscope support robot in four degrees of freedom is calculated as follows: in, represents the sum of the moments acting on the cavity mirror, ∑F z Indicates the action on the longitudinal axis z of the cavity mirror R′ The sum of the forces above, represents the angular acceleration of the end effector; in, m represents the inertia matrix and mass of the cavity mirror respectively; the cavity mirror is regarded as a long cylinder with a base radius of e and a height of L, then Indicates the length of the extracorporeal part of the laparoscope; Representation and vector Related skew-symmetric matrices; The integrated tactile handle includes a force / torque sensor and a flexible sensor array, and the process of constructing the interactive force / torque includes: The force / torque acting on the force / torque sensor Expressed as: in, Respectively represent the readings of the force / torque sensor; x H′ ,y H′ and z H′ Represent the three coordinate axes of the current handle coordinate system {H′} relative to the current fulcrum coordinate system {R′}; The interaction force / torque The calculation process is as follows: in, represent the moment and force of the interaction force / torque, respectively; δ is the interaction force / torque gain, defined as: Among them, Se j represents the measurement value of the jth sensing element on the flexible sensor array; k H Represents a constant; n c represents the number of sensing elements subjected to contact force, k min represents the minimum number of sensing elements required to apply actuation to control the cavity mirror; J represents the number of sensing elements on the flexible sensor array.

2. The hybrid control system for adjusting the field of view of a laparoscope according to claim 1, characterized in that: The virtual force / torque The calculation process is as follows: in, The moment and force of the virtual force / torque are represented respectively; x C′ ,y C′ and z C′ Represents the three coordinate axes of the current laparoscope coordinate system {C′} relative to the current fulcrum coordinate system {R′}; n s Indicates the number of surgical instruments in the image; tracks the position of surgical instruments in the current laparoscope coordinate system {C′} Visual tracking ability Expressed as: F i =F(d i )·V i (10) Among them, d i ,F(d i ) and V i The calculation of is as follows: Among them, V i Indicates that the laparoscope needs to move to track The unit direction of The position of the surgical instrument in the laparoscope coordinate system scaled by unit depth is expressed as: Among them, f x , f y , u0 and v0 are camera internal parameters; express The distance to the center of the image; the function F(d i )∈[0,F max ] proves continuity and differentiability within its domain, F max Indicates tracking The maximum force that can be achieved.

3. The hybrid control system for adjusting the field of view of a laparoscope according to claim 2, characterized in that: The viscous force / torque The calculation process is as follows: in, are the moment and force of viscous force / torque respectively; k f and k t are viscous resistance parameters.

4. The hybrid control system for adjusting the field of view of a laparoscope according to claim 3, characterized in that: Obtaining input for the dynamics model of the laparoscope support robot includes: According to the rigid transformation of the coordinate system on the cavity mirror, it can be deduced that: Then the input of the dynamic model of the laparoscope supporting robot [∑M ∑F z ] T Expressed as:

5. The hybrid control system for adjusting the field of view of a laparoscope according to claim 1, characterized in that: Converting the angular acceleration into the joint angular velocity of the robotic arm includes: Since the cavity mirror and the actuator are rigid bodies, the angular acceleration of the end effector and the angular acceleration of the cavity mirror are both And converted into quaternion velocity Where Δt is the differential of time; For the initial fulcrum coordinate system {R}, the new attitude of the cavity mirror is expressed by quaternion The calculation is as follows: in, Quaternion form representing cavity mirror rotation express The rotation part; Qm(·) represents quaternion multiplication; Translation of the cavity mirror relative to the coordinate system {R} The calculation is as follows: in, express The translation part of The new position of the laparoscope Calculated relative to the coordinate system {R} as: in, Represents a function that converts a quaternion into a rotation matrix; After the coordinate system transformation, the end effector coordinate system {E′ N The transformation of {E′} and {E′} relative to the coordinate system {B} is calculated as: in, represents the fixed coordinate system transformation recorded by the laparoscope supporting robot in the initial state. Since the laparoscope is a rigid body, the transformation relationship between the coordinate systems {E} and {R} remains unchanged. Through real-time Jacobian transformation, the joint angular velocity of the robotic arm Calculated as: in, It represents the real-time Jacobian transformation relationship of the laparoscope supporting robot.

6. A storage medium, characterized in that The invention relates to a method for hybrid control of laparoscope field of view adjustment based on force-driven and vision-driven laparoscope field of view adjustment. The method is based on a laparoscope support robot, which includes an integrated tactile handle, an end effector, and a robotic arm. The method includes: Construct a dynamic model of the laparoscopic support robot based on wound point constraints; Constructing interactive force / torque through the integrated tactile handle; Construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view; Obtaining input of a dynamic model of the laparoscope-supported robot according to the interaction force / torque, the virtual force / torque, and the preset viscous force / torque to generate an angular acceleration of the end effector; Converting the angular acceleration into the joint angular velocity of the robotic arm to adjust the field of view of the laparoscope by hybrid control; Construct a four-degree-of-freedom dynamic model of the laparoscope support robot, including: The laparoscope motion based on the incision point constraint is simplified to continuously passing through a fulcrum. The four-degree-of-freedom rigid body motion refers to the fulcrum Three different rotations are performed along the longitudinal axis z of the cavity mirror. R′ Perform a translation; it represents the angular velocity of the cavity mirror relative to the current fulcrum coordinate system {R′} and line speed The vector is: in, represents the angular velocity and linear velocity of the end effector, x R′ ,y R′ and z R′ Respectively represent the coordinate axes of the x, y, and z directions of the coordinate system {R′}; According to the Newton-Euler equation, the dynamic equation of the laparoscope support robot in four degrees of freedom is calculated as follows: in, represents the sum of the moments acting on the cavity mirror, ∑F z Indicates the action on the longitudinal axis z of the cavity mirror R′ The sum of the forces above, represents the angular acceleration of the end effector; in, m represents the inertia matrix and mass of the cavity mirror respectively; the cavity mirror is regarded as a long cylinder with a base radius of e and a height of L, then Indicates the length of the extracorporeal part of the laparoscope; Representation and vector Related skew-symmetric matrices; The integrated tactile handle includes a force / torque sensor and a flexible sensor array, and the process of constructing the interactive force / torque includes: The force / torque acting on the force / torque sensor Expressed as: in, Respectively represent the readings of the force / torque sensor; x H′ ,y H′ and z H′ Represent the three coordinate axes of the current handle coordinate system {H′} relative to the current fulcrum coordinate system {R′}; The interaction force / torque The calculation process is as follows: in, represent the moment and force of the interaction force / torque, respectively; δ is the interaction force / torque gain, defined as: Among them, Se j represents the measurement value of the jth sensing element on the flexible sensor array; k H Represents a constant; n c represents the number of sensing elements subjected to contact force, k min represents the minimum number of sensing elements required to apply actuation to control the cavity mirror; J represents the number of sensing elements on the flexible sensor array.

7. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a method for executing a hybrid control method for adjusting the field of view of a laparoscope, wherein the hybrid control method for adjusting the field of view of a laparoscope is based on a laparoscope support robot, wherein the laparoscope support robot includes an integrated tactile handle, an end effector, and a robotic arm; the method includes: Construct a dynamic model of the laparoscopic support robot based on wound point constraints; Constructing interactive force / torque through the integrated tactile handle; Construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view; Obtaining input of a dynamic model of the laparoscope-supported robot according to the interaction force / torque, the virtual force / torque, and the preset viscous force / torque to generate an angular acceleration of the end effector; Converting the angular acceleration into the joint angular velocity of the robotic arm to adjust the field of view of the laparoscope by hybrid control; Construct a four-degree-of-freedom dynamic model of the laparoscope support robot, including: The laparoscope motion based on the incision point constraint is simplified to continuously passing through a fulcrum. The four-degree-of-freedom rigid body motion refers to the fulcrum Three different rotations are performed along the longitudinal axis z of the cavity mirror. R′ Perform a translation; it represents the angular velocity of the cavity mirror relative to the current fulcrum coordinate system {R′} and line speed The vector is: in, represents the angular velocity and linear velocity of the end effector, x R′ ,y R′ and z R′ Respectively represent the coordinate axes of the x, y, and z directions of the coordinate system {R′}; According to the Newton-Euler equation, the dynamic equation of the laparoscope support robot in four degrees of freedom is calculated as follows: in, represents the sum of the moments acting on the cavity mirror, ∑F z Indicates the action on the longitudinal axis z of the cavity mirror R′ The sum of the forces above, represents the angular acceleration of the end effector; in, m represents the inertia matrix and mass of the cavity mirror respectively; the cavity mirror is regarded as a long cylinder with a base radius of e and a height of L, then Indicates the length of the extracorporeal part of the laparoscope; Representation and vector Related skew-symmetric matrices; The integrated tactile handle includes a force / torque sensor and a flexible sensor array, and the process of constructing the interactive force / torque includes: The force / torque acting on the force / torque sensor Expressed as: in, Respectively represent the readings of the force / torque sensor; x H′ ,y H′ and z H′ Represent the three coordinate axes of the current handle coordinate system {H′} relative to the current fulcrum coordinate system {R′}; The interaction force / torque The calculation process is as follows: in, represent the moment and force of the interaction force / torque, respectively; δ is the interaction force / torque gain, defined as: Among them, Se j represents the measurement value of the jth sensing element on the flexible sensor array; k H Represents a constant; n c represents the number of sensing elements subjected to contact force, k min represents the minimum number of sensing elements required to apply actuation to control the cavity mirror; J represents the number of sensing elements on the flexible sensor array.

Images