Force-driven and vision-driven endoscope field-of-view adjustment hybrid control method and system

Abstract

The invention provides a force-driven and vision-driven endoscope view field adjustment hybrid control method and system, a storage medium and electronic equipment, and relates to the field of endoscope view field adjustment. Interaction force / moment captured by an integrated tactile handle, virtual force / moment generated based on visual information and preset viscous force / moment are integrated into a endoscope holding dynamic model with wound point constraint at the same time, motion of an endoscope holding robot is generated, and the endoscope view field is controlled and adjusted in a mixed mode. According to the method, the requirement for frequently switching control modes in the endoscope view field adjusting process is eliminated, visual tracking and touch interaction guiding can be achieved at the same time, and continuous view field adjustment through visual driving and large-scale view field adjustment through force driving are both considered. Besides, gains of interaction force / torque can be adjusted through grip strength on the integrated touch handle, so that the endoscope supporting robot cannot move due to the interaction force generated when a doctor touches the endoscope supporting robot by mistake, and the operation safety of the doctor is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of laparoscope field of view adjustment, and in particular to a force-driven and vision-driven laparoscope field of view adjustment hybrid control method, system, storage medium and electronic equipment. Background Art

[0002] Laparoscopic support robots can improve the stability and accuracy of visualization in minimally invasive surgery.

[0003] For example, patent CN107440748A discloses an intelligent automatic tracking laparoscope system for a surgical field, including a fully automatic pan-tilt head, a control handle, a camera and lighting unit, a video processor and a display; one end of the fully automatic pan-tilt head is fixed on the control handle, and the camera and lighting unit is fixed on the other end of the fully automatic pan-tilt head. The camera and lighting unit includes a fixed frame and a light source and a camera arranged in the fixed frame. The video processor is connected to the display, the fully automatic pan-tilt head and the camera.

[0004] For another example, patent CN113143461A provides a human-machine collaborative minimally invasive endoscope holding robot system. By obtaining the endoscope view and robot posture; obtaining the status of each joint of the robot, and solving the endoscope camera posture according to the robot forward kinematics principle; using the YOLOv3 algorithm to detect the position of the surgical instrument tip in the endoscope view; based on the position of each surgical instrument tip in the view, obtaining the distance from the surgical instrument tip to the center point of the view; obtaining the visual tracking vector based on the surgical instrument tip position, the distance from the surgical instrument tip to the center point of the view and the camera parameters; obtaining the endoscope insertion distance constraint vector; obtaining the speed of each joint of the collaborative robot arm based on the visual tracking vector and the endoscope insertion distance constraint vector.

[0005] It can be seen that when the existing laparoscope support robot only relies on visual servoing, it cannot move efficiently and quickly in large-range field of view adjustment tasks such as searching for human tissues and suture needles outside the field of view, and has certain limitations. Summary of the invention

[0006] 1. Technical issues to be solved

[0007] In view of the shortcomings of the prior art, the present invention provides a force-driven and vision-driven laparoscope field of view adjustment hybrid control method, system, storage medium and electronic device, which solves the technical problem of being unable to move quickly in a large-range field of view adjustment task when relying solely on visual servoing.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] A force-driven and vision-driven laparoscope field of view adjustment hybrid control method, characterized in that it is based on a laparoscope support robot, the laparoscope support robot includes an integrated tactile handle, an end effector and a mechanical arm; the method includes:

[0011] Construct a dynamic model of the laparoscope support robot based on wound point constraints;

[0012] By means of the integrated tactile handle, an interactive force / torque is established;

[0013] Construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view;

[0014] According to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque, an input of the dynamic model of the laparoscope supporting robot is obtained to generate an angular acceleration of the end effector;

[0015] The angular acceleration is converted into the joint angular velocity of the robot arm, and the field of view of the laparoscope is adjusted by hybrid control.

[0016] Preferably, constructing a four-degree-of-freedom laparoscope support robot dynamics model includes:

[0017] The laparoscope motion based on the incision point constraint is simplified to continuous passing through a fulcrum. The four-degree-of-freedom rigid body motion refers to the fulcrum Three different rotations are performed along the longitudinal axis z of the cavity mirror. R′ Perform a translation; then it represents the angular velocity of the cavity mirror relative to the current fulcrum coordinate system {R′} and line speed The vector is:

[0018]

[0019] in, represents the angular velocity and linear velocity of the end effector, x R′ ,y R′ and z R′ Respectively represent the coordinate axes of the x, y, and z directions of the coordinate system {R′};

[0020] According to the Newton-Euler equation, the dynamic equation of the laparoscope support robot in four degrees of freedom is calculated as follows:

[0021]

[0022] in, represents the sum of the moments acting on the cavity mirror, ∑F z Indicates the action on the longitudinal axis z of the laparoscope R′ The sum of the forces above, represents the angular acceleration of the end effector;

[0023]

[0024] in, m represents the inertia matrix and mass of the cavity mirror respectively; the cavity mirror is regarded as a long cylinder with a base radius of e and a height of L, then Indicates the length of the extracorporeal part of the laparoscope; Representation and vector The associated skew-symmetric matrix.

[0025] Preferably, the integrated tactile handle includes a force / torque sensor and a flexible sensor array, and the process of constructing the interactive force / torque includes:

[0026] The force / torque acting on the force / torque sensor It is expressed as:

[0027]

[0028] in, Respectively represent the readings of the force / torque sensor; x H′ ,y H′ and z H′ Respectively represent the three coordinate axes of the current handle coordinate system {H′} relative to the current fulcrum coordinate system {R′};

[0029] The interaction force / torque The calculation process is as follows:

[0030]

[0031] in, represent the moment and force of the interaction force / torque, respectively;

[0032] δ is the interaction force / torque gain, defined as:

[0033]

[0034] Among them, Se j represents the measurement value of the jth sensing element on the flexible sensor array; k H Represents a constant; n c represents the number of sensing elements subjected to contact force, k min represents the minimum number of sensor elements required to apply actuation to control the laparoscope; J represents the number of sensor elements on the flexible sensor array.

[0035] Preferably, the virtual force / torque The calculation process is as follows:

[0036]

[0037] in, represent the moment and force of the virtual force / torque respectively; x C′ ,y C′ and z C′ Represents the three coordinate axes of the current laparoscope coordinate system {C′} relative to the current fulcrum coordinate system {R′}; n s Indicates the number of surgical instruments in the image; tracks the position of surgical instruments in the current laparoscope coordinate system {C′} Visual tracking ability It is expressed as:

[0038] F i =F(d i )·V i (10)

[0039] Among them, d i , F(d i ) and V i The calculation of is as follows:

[0040]

[0041] Among them, V i Indicates that the laparoscope needs to move to track The unit direction of It represents the position of the surgical instrument in the laparoscope coordinate system scaled by unit depth, expressed as:

[0042]

[0043] Among them, f x , f y , u0 and v0 are the camera internal parameters; express The distance to the center of the image; function F(d i )∈[0,F max ] proves continuity and differentiability within its domain, F max Indicates tracking The maximum force that can be achieved.

[0044] Preferably, the viscous force / torque The calculation process is as follows:

[0045]

[0046] in, are the moment and force of the viscous force / torque, respectively; k f and k t are viscous resistance parameters.

[0047] Preferably, obtaining the input of the dynamics model of the laparoscope supporting robot includes:

[0048] According to the rigid transformation of the coordinate system on the laparoscope, it can be deduced that:

[0049]

[0050] Then the input of the dynamic model of the laparoscope supporting robot [∑M∑F z ] T It is expressed as:

[0051]

[0052] Preferably, converting the angular acceleration into the joint angular velocity of the robotic arm comprises:

[0053] Since the laparoscope and the actuator are rigid bodies, the angular acceleration of the end effector and the angular acceleration of the laparoscope are both And converted into quaternion velocity

[0054]

[0055] Where Δt is the differential of time;

[0056] For the initial fulcrum coordinate system {R}, the new attitude of the cavity mirror expressed by quaternion The calculation is as follows:

[0057]

[0058] in, Quaternion form representing the rotation of the cavity mirror express The rotation part of ; Qm(·) represents quaternion multiplication;

[0059] Translation of the cavity mirror relative to the coordinate system {R} The calculation is as follows:

[0060]

[0061] in, express The translation part of

[0062] The new position of the laparoscope Calculated relative to the coordinate system {R} as:

[0063]

[0064] in, Represents a function that converts a quaternion into a rotation matrix;

[0065] After the coordinate system transformation, the end effector coordinate system {E′ N The transformation of {E′} and {E′} relative to the coordinate system {B} is calculated as:

[0066]

[0067] in, represents the fixed coordinate system transformation recorded by the laparoscope supporting robot in the initial state. Since the laparoscope is a rigid body, the transformation relationship between the coordinate systems {E} and {R} remains unchanged.

[0068] Through real-time Jacobian transformation, the joint angular velocity of the robot arm Calculated as:

[0069]

[0070] in, It represents the real-time Jacobian transformation relationship of the laparoscope supporting robot.

[0071] A force-driven and vision-driven laparoscope field of view adjustment hybrid control system is based on a laparoscope support robot, wherein the laparoscope support robot includes an integrated tactile handle, an end effector and a mechanical arm; the system includes:

[0072] Model building module, used to build the dynamic model of the laparoscope support robot based on wound point constraints;

[0073] A tactile force sensing module, used to construct an interactive force / torque through the integrated tactile handle;

[0074] The instrument tracking module is used to construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view;

[0075] A velocity generation module, used for obtaining the input of the dynamic model of the laparoscope supporting robot according to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque, so as to generate the angular acceleration of the end effector;

[0076] The speed conversion module is used to convert the angular acceleration into the joint angular velocity of the robot arm, so as to adjust the field of view of the laparoscope by hybrid control.

[0077] A storage medium stores a computer program for force-driven and vision-driven laparoscope field of view adjustment hybrid control, wherein the computer program enables a computer to execute the laparoscope field of view adjustment hybrid control method as described above.

[0078] An electronic device, comprising:

[0079] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the laparoscope field of view adjustment hybrid control method as described above.

[0080] (III) Beneficial effects

[0081] The present invention provides a force-driven and vision-driven laparoscope field of view adjustment hybrid control method, system, storage medium and electronic device. Compared with the prior art, it has the following beneficial effects:

[0082] The present invention integrates the interactive force / torque captured by the integrated tactile handle, the virtual force / torque generated based on visual information, and the preset viscous force / torque into the mirror holding dynamics model with wound point constraints, generates the movement of the laparoscope support robot, and adjusts the laparoscope field of view through hybrid control. This method eliminates the need to frequently switch control modes during the adjustment of the laparoscope field of view, can simultaneously achieve visual tracking and tactile interactive guidance, and takes into account the use of visual drive for continuous field of view adjustment and force drive for large-scale field of view adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 A technical schematic diagram of a hybrid control method for adjusting the field of view of a laparoscope driven by force and vision provided in an embodiment of the present invention;

[0085] Figure 2 A block diagram of a force-driven and vision-driven laparoscope field of view adjustment hybrid control method provided in an embodiment of the present invention;

[0086] FIG3( a ) is a schematic diagram of a laparoscope support robot under wound point constraint provided by an embodiment of the present invention;

[0087] FIG3( b ) is a schematic diagram of a coordinate system definition of a laparoscope supporting robot provided in an embodiment of the present invention;

[0088] Figure 4 A schematic diagram of an experimental platform provided by an embodiment of the present invention;

[0089] FIG5( a ) is a snapshot of an experiment provided by an embodiment of the present invention;

[0090] FIG5( b ) is a force / torque applied to a laparoscope supporting robot according to an embodiment of the present invention;

[0091] FIG5( c ) is a schematic diagram of a laparoscope trajectory provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] The embodiments of the present application provide a hybrid control method, system, storage medium and electronic device for laparoscope field of view adjustment driven by force and vision, thereby solving the technical problem of being unable to move efficiently in large-range field of view adjustment tasks when relying solely on visual servoing, while taking into account both the use of vision drive for continuous field of view adjustment and the use of force drive for large-scale field of view adjustment.

[0094] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0095] The solution provided by the embodiment of the present invention is a laparoscope supporting robot system which adopts visual drive to perform continuous field of view adjustment and force drive to perform large-scale field of view adjustment.

[0096] Specifically, a laparoscopic support robot system with an integrated tactile handle was designed, which can measure the human-machine interaction force during the surgical area tracking process. Furthermore, a force-driven and vision-driven hybrid control method for laparoscopic field of view adjustment was proposed. Figure 1 As shown in the figure, this method integrates the interactive force / torque captured by the integrated tactile handle, the virtual force / torque generated based on visual information, and the preset viscous force / torque into the laparoscope dynamics model with wound point constraints, generates the robot's motion, and adjusts the laparoscope field of view through hybrid control. In addition, the gain of the interactive force / torque can be adjusted by the grip force on the integrated tactile handle, so that the interactive force generated by the doctor's accidental touch will not cause the laparoscope support robot to move, ensuring the safety of the doctor's operation.

[0097] Therefore, this method eliminates the need to frequently switch control modes during laparoscope field of view adjustment and can achieve visual tracking and tactile interactive guidance at the same time.

[0098] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0099] Embodiment 1:

[0100] like Figure 2 As shown, an embodiment of the present invention provides a force-driven and vision-driven laparoscope field of view adjustment hybrid control method, based on a laparoscope support robot, the laparoscope support robot includes an integrated tactile handle, an end effector and a robotic arm; the method includes:

[0101] S1. Construct the dynamic model of the laparoscope support robot based on wound point constraints;

[0102] S2. constructing interactive force / torque through the integrated tactile handle;

[0103] S3, constructing a virtual force / torque according to the position of the surgical instrument in the laparoscope field of view;

[0104] S4, obtaining the input of the dynamic model of the laparoscope supporting robot according to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque to generate the angular acceleration of the end effector;

[0105] S5. Convert the angular acceleration into the joint angular velocity of the robot arm, and adjust the laparoscope field of view by hybrid control.

[0106] The embodiments of the present invention eliminate the need for frequent switching of control modes during laparoscope field of view adjustment, can simultaneously realize visual tracking and tactile interactive guidance, and take into account both the use of visual drive for continuous field of view adjustment and the use of force drive for large-scale field of view adjustment.

[0107] It is necessary to first add the definition of the coordinate system of the laparoscope support robot:

[0108] The laparoscope support robot under wound point constraint is shown in Figure 3(a), which includes an integrated tactile handle, an end effector, and a robotic arm (not shown in the drawing), wherein a six-dimensional force / torque sensor is fixed on the end effector of the robot to measure the force and torque applied by the doctor. The movement of the laparoscope is driven by the robot while complying with the wound point constraint.

[0109] Figure 3(b) gives the specific definition of the coordinate system: the coordinate system {B} is used as the base coordinate system of the robot system. When the robot is in the initial posture, the coordinate system of the laparoscope support robot system is represented by {*}; when the robot moves, the current coordinate system of the laparoscope support robot system is represented by {*′}; when the robot is in a new posture, the coordinate system of the laparoscope support robot system is represented by {*′ N}.

[0110] Define the fulcrum coordinate system {R} to be located at the wound point when the robot is in the initial position. The origin of the fulcrum coordinate system {R} As the incision point. {R′} is the current fulcrum coordinate system, its origin Located at the wound site. The robot’s end effector coordinate system is denoted as {E} and {E′}, with its origin located on the axis of the laparoscope. The homogeneous transformation matrix between the end effector and the base coordinate system is is the joint angle reading of the robot, where λ represents the degree of freedom of the robot. The tactile handle coordinate system is denoted by {H} and {H′}, and the laparoscope coordinate system is denoted by {C} and {C′}.

[0111] On this basis, the following will introduce the various steps of the above scheme in detail:

[0112] In step S1, a dynamic model of a laparoscope support robot based on wound point constraints is constructed.

[0113] The laparoscope is inserted into the patient's body through a natural opening or a small incision, and the movement of the external robotic arm must be constrained by the incision point.

[0114] Specifically, this step simplifies the laparoscope motion based on the incision point constraint to continuously passing through a fulcrum. The four-degree-of-freedom rigid body motion refers to the fulcrum Three different rotations are performed along the longitudinal axis z of the cavity mirror. R′ Perform a translation; then it represents the angular velocity of the cavity mirror relative to the current fulcrum coordinate system {R′} and line speed The vector is:

[0115]

[0116] in, represents the angular velocity and linear velocity of the end effector, x R′ ,y R′ and z R′ Respectively represent the coordinate axes of the x, y, and z directions of the coordinate system {R′};

[0117] According to the Newton-Euler equation, the dynamic equation of the laparoscope support robot in four degrees of freedom is calculated as follows:

[0118]

[0119] in, represents the sum of the moments acting on the cavity mirror, ∑F z Indicates the action on the longitudinal axis z of the laparoscope R′ The sum of the forces above, represents the angular acceleration of the end effector;

[0120]

[0121] in, m represents the inertia matrix and mass of the cavity mirror respectively; the cavity mirror is regarded as a long cylinder with a base radius of e and a height of L, then Indicates the length of the extracorporeal part of the laparoscope; Representation and vector The associated skew-symmetric matrix.

[0122] After constructing the dynamic model of the laparoscope support robot based on wound point constraints, it is necessary to pay attention to the following Figure 1 As shown, the embodiment of the present invention proposes to use three forces / torques, namely, interactive force / torque, virtual force / torque and viscous force / torque, to hybridly control and adjust the laparoscope field of view. For details, see the following introduction:

[0123] In step S2, an interactive force / torque is established through the integrated tactile handle.

[0124] The laparoscopic support robot provided in the embodiment of the present invention is different from the traditional laparoscopic support robot. In the process of the traditional laparoscopic support robot autonomously tracking the surgical instrument, the doctor cannot intervene in the movement of the robot, resulting in the robot being unable to perfectly cooperate with the doctor's surgical operation. As shown in Figure 3(a), the embodiment of the present invention designs a laparoscopic support robot system with an integrated tactile handle, and the integrated tactile handle includes a force / torque sensor and a flexible sensor array.

[0125] Accordingly, the interaction force / torque is determined by the force / torque operated by the doctor on the force / torque sensor and the grip force from the flexible sensor array. In particular, the interaction force / torque includes a parameter adaptive adjustment mechanism to prevent any unintentional touch or contact from affecting the touch handle.

[0126] The force / torque acting on the force / torque sensor It is expressed as:

[0127]

[0128] in, Respectively represent the readings of the force / torque sensor; x H′ ,y H′ and z H′ Respectively represent the three coordinate axes of the current handle coordinate system {H′} relative to the current fulcrum coordinate system {R′};

[0129] The interaction force / torque The calculation process is as follows:

[0130]

[0131] in, represent the moment and force of the interaction force / torque, respectively;

[0132] δ is the interaction force / torque gain, defined as:

[0133]

[0134] Among them, Se j represents the measurement value of the jth sensing element on the flexible sensor array; k H Represents a constant; n c represents the number of sensing elements subjected to contact force, k min represents the minimum number of sensor elements required to apply actuation to control the laparoscope; J represents the number of sensor elements on the flexible sensor array.

[0135] In step S3, a virtual force / torque is constructed according to the position of the surgical instrument in the laparoscope field of view.

[0136] Virtual force / torque Used to prevent the instrument from moving out of the field of view.

[0137] The calculation is as follows:

[0138]

[0139] in, represent the moment and force of the virtual force / torque respectively; x C′ ,y C′ and z C′ Represents the three coordinate axes of the current laparoscope coordinate system {C′} relative to the current fulcrum coordinate system {R′}; n s Indicates the number of surgical instruments in the image; tracks the position of surgical instruments in the current laparoscope coordinate system {C′} Visual tracking ability It is expressed as:

[0140] F i =F(d i )·V i (10)

[0141] Among them, d i , F(d i ) and V i The calculation of is as follows:

[0142]

[0143] Among them, V i Indicates that the laparoscope needs to move to track The unit direction of It represents the position of the surgical instrument in the laparoscope coordinate system scaled by unit depth, expressed as:

[0144]

[0145] Among them, f x , f y , u0 and v0 are the camera internal parameters; express The distance to the center of the image; function F(d i )∈[0,F max ] proves continuity and differentiability within its domain, F max Indicates tracking The maximum force that can be achieved.

[0146] Furthermore, in order to ensure the stability of the system, the embodiment of the present invention also proposes a viscous force / torque It combines viscous drag to constrain the four-degree-of-freedom motion of the cavity mirror. The viscous force / torque The calculation process is as follows:

[0147]

[0148] in, are the moment and force of the viscous force / torque, respectively; k f and k t are viscous resistance parameters.

[0149] In step S4, the input of the dynamic model of the laparoscope supporting robot is obtained according to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque to generate the angular acceleration of the end effector.

[0150] Since the coordinate system on the cavity mirror is a rigid transformation and does not involve mutual rotation, it can be deduced that:

[0151]

[0152] Then the input of the dynamic model of the laparoscope supporting robot [∑M∑F z ] T It is expressed as:

[0153]

[0154] Finally, according to formula (3), the angular acceleration of the end effector can be obtained:

[0155] In step S5, the angular acceleration is converted into the joint angular velocity of the robot arm, and the field of view of the laparoscope is adjusted by hybrid control.

[0156] Since the laparoscope and the actuator are rigid bodies, the angular acceleration of the end effector and the angular acceleration of the laparoscope are both

[0157] To avoid the universal joint deadlock problem during the robot speed conversion process, the angular acceleration of the cavity mirror is Convert to quaternion velocity

[0158]

[0159] Where Δt is the differential of time;

[0160] For the initial fulcrum coordinate system {R}, the new attitude of the cavity mirror expressed by quaternion The calculation is as follows:

[0161]

[0162] in, Quaternion form representing the rotation of the cavity mirror express The rotation part of ; Qm(·) represents quaternion multiplication;

[0163] Translation of the cavity mirror relative to the coordinate system {R} The calculation is as follows:

[0164]

[0165] in, express The translation part of

[0166] The new position of the laparoscope Calculated relative to the coordinate system {R} as:

[0167]

[0168] in, Represents a function that converts a quaternion into a rotation matrix;

[0169] After the coordinate system transformation, the end effector coordinate system {E′ N The transformation of {E′} and {E′} relative to the coordinate system {B} is calculated as:

[0170]

[0171] in, represents the fixed coordinate system transformation recorded by the laparoscope supporting robot in the initial state. Since the laparoscope is a rigid body, the transformation relationship between the coordinate systems {E} and {R} remains unchanged.

[0172] Through real-time Jacobian transformation, the joint angular velocity of the robot arm Calculated as:

[0173]

[0174] in, It represents the real-time Jacobian transformation relationship of the laparoscope supporting robot.

[0175] So far, the embodiment of the present invention has completed the hybrid control of the laparoscope field of view adjustment driven by force and vision.

[0176] In order to verify the effectiveness of the method proposed in the embodiment of the present invention, the Figure 4 The experimental platform shown is used for experimental testing:

[0177] The wound point is set on the laparoscope axis, and the initial end effector posture is adjusted. Perform fixed coordinate system transformation Determine the coordinate system posture of the wound point The initial posture of the laparoscope is obtained by integrating the model every 1 ms using the Euler method to generate reference joint positions provided to the robot. This Franka robot operation control mode has high tracking accuracy. The laparoscope system is mounted on the robot's end effector, which communicates with the computer through a USB port, and the camera frame rate is set to 25fps. The camera calibration method proposed by Zhang et al. is used to calibrate the laparoscope internal parameters. In addition, the YOLO v7 algorithm is used to achieve high-performance and low-latency surgical instrument detection.

[0178] In minimally invasive surgery, due to limited field of view, doctors need to perform large-scale laparoscopic movements to locate suture needles outside their current field of view. In this experiment, the operator's goal is to find a suture needle placed on a simulated organ. The operator first controls the movement of the surgical instrument in a local area. Subsequently, the operator manipulates the tactile handle to facilitate large-scale adjustments in the field of view to locate the suture needle.

[0179] The experimental results are shown in Figure 5. The experimental snapshot is shown in Figure 5(a). As shown in Figure 5(b), the interaction force / torque and virtual forces / torques Figure 5(c) shows the trajectory of the laparoscope axis, where the light bold lines correspond to the initial pose, the dark bold lines correspond to the final pose, and the black dots represent the pivots. The results show that the motion of the laparoscope always follows the constraints of the RCM (i.e., the incision point constraints mentioned above) throughout the experiment.

[0180] In the initial stage, the operator manipulates the surgical instruments on the artificial organ to generate virtual forces / torques to track the movement of the instrument (from 0 to t3). At t4, the surgical instrument stops moving and remains in the center area of ​​the screen. Applied to the tactile handle, the laparoscope is controlled to find the position of the suture needle. At t5, the interaction force / torque Exceeding virtual force / torque This causes the laparoscope to deviate from the current viewing angle. After t6, the instrument is no longer visible in the image. The operator guides the laparoscope to precisely locate the position of the suture needle through a noticeable change in the field of view. At t7, the suture needle appears at the edge of the screen. At this point, the operator releases the haptic handle, generating a virtual force / torque This allows for quick repositioning of surgical instruments back to the center of the screen. 10 ), with the help of laparoscopic tracking, the surgeon

[0181] The suture needle was successfully removed. Note that the viscous force / torque As it is the internal force / torque of the system, it is not shown in the figure.

[0182] Therefore, it is clear that this method can not only perform continuous field of view adjustment under the guidance of surgical instruments, but also can be well adapted to large-scale field of view adjustment tasks such as finding suture needles outside the field of view.

[0183] Embodiment 2:

[0184] The embodiment of the present invention provides a force-driven and vision-driven laparoscope field of view adjustment hybrid control system, based on a laparoscope support robot, the laparoscope support robot includes an integrated tactile handle, an end effector and a mechanical arm; the system includes:

[0185] Model building module, used to build the dynamic model of the laparoscope support robot based on wound point constraints;

[0186] A tactile force sensing module, used to construct an interactive force / torque through the integrated tactile handle;

[0187] The instrument tracking module is used to construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view;

[0188] A velocity generation module, used for obtaining the input of the dynamic model of the laparoscope supporting robot according to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque, so as to generate the angular acceleration of the end effector;

[0189] The speed conversion module is used to convert the angular acceleration into the joint angular velocity of the robot arm, so as to adjust the field of view of the laparoscope by hybrid control.

[0190] Embodiment 3:

[0191] An embodiment of the present invention provides a storage medium storing a computer program for force-driven and vision-driven laparoscope field of view adjustment hybrid control, wherein the computer program enables a computer to execute the laparoscope field of view adjustment hybrid control method as described above.

[0192] Embodiment 4:

[0193] An embodiment of the present invention provides an electronic device, including:

[0194] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the laparoscope field of view adjustment hybrid control method as described above.

[0195] It can be understood that the force-driven and vision-driven laparoscope field of view adjustment hybrid control system, storage medium and electronic device provided in the embodiments of the present invention correspond to the force-driven and vision-driven laparoscope field of view adjustment hybrid control method provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the laparoscope field of view adjustment hybrid control method, which will not be repeated here.

[0196] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0197] 1. The embodiments of the present invention eliminate the need for frequent switching of control modes during laparoscope field of view adjustment, and can simultaneously achieve visual tracking and tactile interactive guidance, taking into account both the use of visual drive for continuous field of view adjustment and the use of force drive for large-scale field of view adjustment.

[0198] 2. The embodiment of the present invention adjusts the gain of the interactive force / torque by integrating the grip force on the tactile handle, so that the interactive force generated by the doctor's accidental touch will not cause the robot to move, thereby ensuring the safety of the doctor's operation.

[0199] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0200] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid control method for adjusting the field of view of a laparoscope driven by force and vision, characterized in that: Based on a laparoscope support robot, the laparoscope support robot includes an integrated tactile handle, an end effector and a mechanical arm; the method includes: Construct a dynamic model of the laparoscope support robot based on wound point constraints; By means of the integrated tactile handle, an interactive force / torque is established; Construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view; According to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque, an input of the dynamic model of the laparoscope supporting robot is obtained to generate an angular acceleration of the end effector; The angular acceleration is converted into the joint angular velocity of the robot arm, and the field of view of the laparoscope is adjusted by hybrid control.

2. The hybrid control method for adjusting the field of view of a laparoscope according to claim 1, characterized in that: Construct a four-degree-of-freedom laparoscope support robot dynamics model, including: The laparoscope motion based on the incision point constraint is simplified to continuous passing through a fulcrum. The four-degree-of-freedom rigid body motion refers to the fulcrum Three different rotations are performed along the longitudinal axis z of the cavity mirror. R , and performs a translation; then it represents the angular velocity of the cavity mirror relative to the current fulcrum coordinate system {R′} and line speed The vector is: in, represents the angular velocity and linear velocity of the end effector, x R′ ,y R′ and z R′ Respectively represent the coordinate axes of the x, y, and z directions of the coordinate system {R′}; According to the Newton-Euler equation, the dynamic equation of the laparoscope support robot in four degrees of freedom is calculated as follows: in, represents the sum of the moments acting on the cavity mirror, ∑F z Indicates the effect on the longitudinal axis z of the laparoscope R′ The sum of the forces above, represents the angular acceleration of the end effector; in, m represents the inertia matrix and mass of the cavity mirror respectively; the cavity mirror is regarded as a long cylinder with a base radius of e and a height of L, then Indicates the length of the extracorporeal part of the laparoscope; Representation and vector The associated skew-symmetric matrix.

3. The hybrid control method for adjusting the field of view of a laparoscope according to claim 2, characterized in that: The integrated tactile handle includes a force / torque sensor and a flexible sensor array, and the process of constructing the interactive force / torque includes: The force / torque acting on the force / torque sensor It is expressed as: in, Respectively represent the readings of the force / torque sensor; x H′ ,y H′ and z H′ Respectively represent the three coordinate axes of the current handle coordinate system {H′} relative to the current fulcrum coordinate system {R′}; The interaction force / torque The calculation process is as follows: in, represent the moment and force of the interaction force / torque, respectively; δ is the interaction force / torque gain, defined as: Among them, Se j represents the measurement value of the jth sensing element on the flexible sensor array; k H Represents a constant; n c represents the number of sensing elements subjected to contact force, k min represents the minimum number of sensor elements required to apply actuation to control the laparoscope; J represents the number of sensor elements on the flexible sensor array.

4. The hybrid control method for adjusting the field of view of a laparoscope according to claim 3, characterized in that: The virtual force / torque The calculation process is as follows: in, represent the moment and force of the virtual force / torque respectively; x C′ ,y C′ and z C′ Represents the three coordinate axes of the current laparoscope coordinate system {C′} relative to the current fulcrum coordinate system {R′}; n s Indicates the number of surgical instruments in the image; tracks the position of surgical instruments in the current laparoscope coordinate system {C′} Visual tracking ability It is expressed as: F i =F(d i )·V i (10) Among them, d i , F(d i ) and V i The calculation of is as follows: Among them, V i Indicates that the laparoscope needs to move to track The unit direction of It represents the position of the surgical instrument in the laparoscope coordinate system scaled by unit depth, expressed as: Among them, f x , f y , u0 and v0 are the camera internal parameters; express The distance to the center of the image; function F(d i )∈[0,F max ] proves continuity and differentiability within its domain, F max Indicates tracking The maximum force that can be achieved.

5. The hybrid control method for adjusting the field of view of a laparoscope according to claim 4, characterized in that: The viscous force / torque The calculation process is as follows: in, are the moment and force of the viscous force / torque, respectively; k f and k t are viscous resistance parameters.

6. The hybrid control method for adjusting the field of view of a laparoscope according to claim 5, characterized in that: Obtaining the input of the dynamics model of the laparoscope supporting robot includes: According to the rigid transformation of the coordinate system on the laparoscope, it can be deduced that: Then the input of the dynamic model of the laparoscope supporting robot [∑M ∑F z ] T It is expressed as:

7. The hybrid control method for adjusting the field of view of a laparoscope according to claim 2, characterized in that: The angular acceleration is converted into the joint angular velocity of the robot arm, including: Since the laparoscope and the actuator are rigid bodies, the angular acceleration of the end effector and the angular acceleration of the laparoscope are both And converted into quaternion velocity Among them, △t is the differential of time; For the initial fulcrum coordinate system {R}, the new attitude of the cavity mirror expressed by quaternion The calculation is as follows: in, Quaternion form representing the rotation of the cavity mirror express The rotation part of ; Qm(·) represents quaternion multiplication; Translation of the cavity mirror relative to the coordinate system {R} The calculation is as follows: in, express The translation part of The new position of the laparoscope Calculated relative to the coordinate system {R} as: in, Represents a function that converts a quaternion into a rotation matrix; After the coordinate system transformation, the end effector coordinate system {E′ N The transformation of {E′} and {E′} relative to the coordinate system {B} is calculated as: in, represents the fixed coordinate system transformation recorded by the laparoscope supporting robot in the initial state. Since the laparoscope is a rigid body, the transformation relationship between the coordinate systems {E} and {R} remains unchanged. Through real-time Jacobian transformation, the joint angular velocity of the robot arm Calculated as: in, It represents the real-time Jacobian transformation relationship of the laparoscope supporting robot.

8. A force-driven and vision-driven laparoscope field of view adjustment hybrid control system, characterized in that: Based on a laparoscope support robot, the laparoscope support robot includes an integrated tactile handle, an end effector and a mechanical arm; the system includes: Model building module, used to build the dynamic model of the laparoscope support robot based on wound point constraints; A tactile force sensing module, used to construct an interactive force / torque through the integrated tactile handle; The instrument tracking module is used to construct virtual forces / torques based on the position of surgical instruments in the laparoscope field of view; A velocity generation module, used for obtaining the input of the dynamic model of the laparoscope supporting robot according to the interactive force / torque, the virtual force / torque, and the preset viscous force / torque, so as to generate the angular acceleration of the end effector; The speed conversion module is used to convert the angular acceleration into the joint angular velocity of the robot arm, so as to adjust the field of view of the laparoscope by hybrid control.

9. A storage medium, characterized in that: The device stores a computer program for force-driven and vision-driven laparoscope field of view adjustment hybrid control, wherein the computer program enables the computer to execute the laparoscope field of view adjustment hybrid control method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the laparoscope field of view adjustment hybrid control method as described in any one of claims 1 to 7.

Images