Lens holding arm control method based on interactive surgical instrument segmentation

Abstract

The invention discloses a lens holding arm control method based on interactive surgical instrument segmentation, and belongs to the technical field of surgical robot control. Comprising the following steps: carrying out interactive segmentation on the tip of a surgical instrument in a first-frame image of a video returned by an endoscope at the tail end of an endoscope holding arm; automatically acquiring a tip mask of a surgical instrument in a video subsequent frame image through a tracking network model, and calculating a center coordinate of the tip mask; calculating the center coordinate of the ROI region of the image and the error between the center coordinate and the expected position coordinate by combining the weight of the user-defined surgical instrument; a joint space angle control quantity is obtained through a multi-task whole-body controller with priorities to carry out motion control on a lens holding arm, and errors of high-level tasks and low-level tasks are converged to be zero along with time. According to the method, any type and number of surgical instruments can be quickly and stably tracked without an additional model training process. The endoscope holding arm automatically adjusts the pose of the endoscope according to the central position of the tip of the surgical instrument so as to optimize the surgical field, and the man-machine cooperation surgical operation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robot control, and in particular to a scope holding arm control method based on interactive surgical instrument segmentation. Background Art

[0002] In recent years, laparoscopic minimally invasive surgery has become increasingly popular among patients due to its many advantages, such as less trauma and faster postoperative recovery. In traditional laparoscopic minimally invasive surgery, the assistant holding the scope needs to support the laparoscope and adjust the position of the laparoscope under the instruction of the surgeon to ensure that the surgical instruments always remain within the field of view. However, the lack of cooperation between the surgeon and the assistant holding the scope will cause instrument collisions and conflicts, which may endanger patient safety; the assistant holding the scope will experience muscle fatigue and hand tremors due to maintaining a fixed posture for a long time, which will reduce the stability of the picture and affect the progress of the operation. Compared with traditional laparoscopic minimally invasive surgery, robot-assisted laparoscopic surgery has the characteristics of high precision and high stability, and can effectively replace the assistant holding the scope to complete the work of holding the scope, improve surgical efficiency and liberate medical resources.

[0003] The current advanced mirror-holding robot control systems are mainly divided into master-slave systems and visual servo systems. In the master-slave system, the doctor guides the robot to perform mirror-holding operations through remote operation, voice control, head device control, pedal buttons, etc., and the robot itself has no intelligent behavior. This type of system still requires the doctor to divide his attention to control the mirror-holding robot, which has the disadvantage of delayed operation response and also increases the duration of the operation. The visual servo system tracks the surgical instruments to enable the mirror-holding robot to autonomously adjust the mirror-holding posture so that the instruments are always in the field of view. Compared with the master-slave system, the visual servo system can improve the degree of autonomy of the surgical process and is an intelligent solution that is more in line with the human-machine collaborative surgical operation scenario.

[0004] Currently, advanced instrument tracking solutions usually use color marking methods to track surgical instruments, but this type of method will fail in surgical scenarios such as bleeding and occlusion. There are also markerless tracking solutions based on target detection and segmentation, but this type of method has certain limitations, such as the availability of privacy surgical data and the need to repeatedly train models for new instruments. Summary of the invention

[0005] In view of the above problems, the present invention proposes a method for controlling an endoscope holding arm based on interactive surgical instrument segmentation. The positioning and tracking of any surgical instrument can be achieved without retraining the model; at the same time, in order to achieve a better control effect of the endoscope holding arm, a multi-level quadratic programming solver is used to solve the whole-body control task with priority to realize the motion control of the endoscope holding arm. The endoscope holding arm autonomously adjusts the posture of the endoscope according to the center position of the tip of the surgical instrument to optimize the surgical field of view and improve the efficiency of human-machine collaborative surgery. To achieve the above purpose, the technical solution of the present invention has the following steps:

[0006] A method for controlling a surgical instrument holding arm based on interactive surgical instrument segmentation, comprising:

[0007] The first frame of the image captured in real time by the endoscope at the end of the endoscope holding arm is obtained, and the tip of the surgical instrument to be tracked is interactively segmented to obtain the mask of the tip of each surgical instrument;

[0008] Inputting each surgical tip mask in the first frame image and subsequent frame images into the tracking network model, and tracking the position of each surgical instrument tip mask in the subsequent frame images;

[0009] Calculate the center coordinates of the mask of each surgical instrument tip, and determine the center coordinates of the ROI area in combination with the weight of each surgical instrument; adjust the field of view by adjusting the weight of each surgical instrument;

[0010] The spatial angle control amount of the joint of the endoscope holding arm is obtained by a multi-task whole body controller with priority to control the movement of the endoscope holding arm; the multi-task whole body controller with priority includes a high-priority distal center constraint task and a low-priority image visual servoing task; the distal motion center task is to minimize the error between the distal motion center point on the endoscope and the surgical incision point, and calculate the high-level joint speed control amount; the image visual servoing task is to minimize the error of the ROI area center coordinate relative to the expected position coordinate, and calculate the low-level joint speed control amount;

[0011] The high-level and low-level joint speed control quantities are integrated to obtain the mirror-holding arm joint speed control quantity, and the joint space angle control quantity is integrated to realize the motion control of the mirror-holding arm.

[0012] Furthermore, the interactive segmentation of the surgical instrument tip that needs to be tracked is performed to obtain the mask of each surgical instrument tip, specifically: discrete points and / or rectangular frames representing the surgical instrument tip area are extracted in the first frame image, and the discrete point and / or rectangular frame information is passed through the prompt encoder of the lightweight segmentation network model to obtain prompt embedding; the first frame image is input into the image encoder of the lightweight segmentation model to obtain image features; the prompt embedding and the image features are input into the mask decoder of the lightweight segmentation model to obtain the surgical instrument tip mask.

[0013] Furthermore, the discrete points and / or rectangular frames are marked by an interactive graphical interface, which displays an image captured by the endoscope and is provided with a marking tool.

[0014] Furthermore, the lightweight segmentation network model adopts a fine-tuned MobieSAM segmentation network model, and the fine-tuning method of the MobieSAM segmentation network model is: freezing the prompt encoder of MobieSAM, and only fine-tuning the image encoder and the mask decoder; each surgical instrument segmentation image sample used for fine-tuning is marked with a surgical instrument tip mask, and a number of discrete points are randomly sampled in the mask part as prompt information and input into the prompt encoder to obtain prompt embedding, and the prompt embedding and the image features generated by the image encoder are input into the mask decoder to predict the surgical instrument tip mask, and the binary cross entropy loss and Dice similarity coefficient loss of the predicted mask and the true mask are calculated, and the combined loss function is adopted to back-propagate and update the image encoder and mask decoder parameters.

[0015] Furthermore, the tracking of the mask position of each surgical instrument tip in the subsequent frame image is specifically as follows:

[0016] The first frame image and the first frame surgical instrument tip mask obtained by interactive segmentation are combined and stored in the network memory through the value encoder of the XMem tracking network model; the subsequent frame images are used as query values ​​in turn through the query encoder of the XMem tracking network model to extract image features; the surgical instrument tip masks of the subsequent frame images are generated in turn through the decoder of the XMem tracking network model.

[0017] Furthermore, the coordinates of the center of the mask of each surgical instrument tip are calculated, and the coordinates of the center of the ROI region are determined in combination with the weight of each surgical instrument, specifically:

[0018] Convert each surgical instrument tip mask into a binary image;

[0019] Perform corrosion and expansion operations on each binary image successively;

[0020] Calculate the image moments of each mask after erosion and dilation operations;

[0021] Calculate the center coordinates of the mask of each surgical instrument tip according to the image moment of each mask;

[0022] The weighted average coordinates of the mask centers of all surgical instrument tips are calculated according to the weights of each surgical instrument as the coordinates of the center of the ROI area.

[0023] Furthermore, in the distal motion center task, the distal motion center point is the point on the endoscope axis that is closest to the surgical incision point, and the coordinates of the distal center point are obtained by calculating the kinematic model of the endoscope holding arm and the physical parameters of the endoscope.

[0024] Furthermore, the calculation of the high-level joint speed control amount is specifically as follows:

[0025] The distal motion center constraint task is modeled as a constrained optimization problem with joint velocity constraints, expressed as:

[0026]

[0027] Among them, J rcm K is the Jacobian matrix of the error between the distal motion center and the surgical incision point with respect to the joint variables, rcm is a positive definite matrix, e rcm is the error between the distal motion center and the surgical incision point, w is the relaxation factor, and w + 、w - is the maximum and minimum value of the relaxation factor; q is the joint angle, q + ,q - are the maximum and minimum values ​​of the joint angles, is the joint velocity, t is the time, ||.|| 2 To find the square of the modulus;

[0028] The high-level joint speed control quantity is obtained by solving the quadratic programming problem, denoted as

[0029] Furthermore, the calculation of the low-level joint speed control amount is specifically as follows:

[0030] The image visual servoing task is modeled as a constrained optimization problem with joint velocity constraints, expressed as:

[0031]

[0032]

[0033]

[0034] Among them, J img is the image Jacobian matrix used to describe the relationship between the image feature point velocity and the camera motion velocity, J cam is the Jacobian matrix used to describe the relationship between camera motion speed and joint speed, For J rcm The null space projection of n is the identity matrix, is the matrix J rcm The pseudo-inverse of K v is a positive definite matrix, s des =(x des ,y des ) is the expected position coordinate, that is, the center point coordinate of the frame image; s ROI =(x ROI ,y ROI ) is the center coordinate of the image ROI area;

[0035] The low-level joint speed control quantity is obtained by solving the quadratic programming problem, denoted as

[0036] Furthermore, the combined high- and low-level joint speed control amounts are used to obtain the mirror-holding arm joint speed control amount, specifically:

[0037] Control amount based on high-level joint velocity and the underlying joint velocity control amount Calculate the final joint velocity control amount

[0038]

[0039] Will The current joint position is superimposed after time integration to obtain the next joint position output.

[0040] Compared with the existing mirror holding arm visual servo control method, the beneficial effects of the present invention are:

[0041] (1) The present invention adopts the fine-tuned MobieSAM segmentation network model to achieve high-quality segmentation of any surgical instrument, effectively improving the generalization performance of the method.

[0042] (2) The present invention adopts a method that combines a lightweight segmentation network model with a tracking network model to achieve real-time tracking of surgical instruments in the video stream, effectively ensuring the real-time requirements of the control system.

[0043] (3) The present invention uses a multi-level quadratic programming solver to solve the whole-body control task with priority, realizes the control of multiple tasks and the constraint of the joint speed of the endoscope holding arm, and effectively improves the safety of the endoscope holding arm movement. The present invention can quickly and stably track any type and number of surgical instruments. The endoscope holding arm can autonomously adjust the posture of the endoscope according to the center position of the tip of the surgical instrument to optimize the surgical field of view and improve the efficiency of human-machine collaborative surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0045] Figure 2 An example of an interactive surgical instrument segmentation interface of the present invention;

[0046] Figure 3 A schematic diagram of the MobieSAM fine-tuning method of the present invention;

[0047] Figure 4 This is a surgical instrument mask tracking effect diagram of the present invention;

[0048] Figure 5It is a schematic diagram of the distal motion center task of the present invention. DETAILED DESCRIPTION

[0049] In order to more clearly illustrate the technical solution of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0050] like Figure 1 As shown, the present invention proposes a scope arm control method based on interactive surgical instrument segmentation, which mainly includes: an interactive graphical interface, a segmentation and tracking network model, and a multi-task whole-body controller with priority.

[0051] The interactive graphical interface is used to obtain prompt information for surgical instruments that need to be tracked. The prompt information is input into the segmentation and tracking network model to obtain the surgical instrument tip mask in the video frame. The center coordinates of the surgical instrument tip mask are calculated based on the surgical instrument tip mask. The center coordinates of the ROI area are determined in combination with the custom surgical instrument weights. The error between the distal motion center point on the endoscope and the surgical incision point, and the error between the ROI area center coordinates and the expected position are input into the multi-task whole body controller with priority to obtain the joint motion control amount of the surgical instrument. By adjusting the surgical instrument weight, the endoscope holding arm autonomously adjusts the endoscope posture according to the center position of the surgical instrument tip to optimize the surgical field of view.

[0052] like Figure 2 As shown, in the interactive graphical interface, the surgical instrument tip to be tracked in the first frame image is selected for segmentation in the form of a rectangular box and / or discrete points, such as the electrocoagulation hook shown in the figure, with both the rectangular box and the discrete points marked. The interactive interface inputs the rectangular box and discrete point information into the prompt encoder in the fine-tuned MobieSAM segmentation network to obtain the prompt embedding, and inputs the image into the image encoder of MobieSAM to obtain the image features; the prompt embedding and the image features are jointly input into the mask decoder of MobieSAM to generate the surgical instrument tip mask.

[0053] like Figure 3 As shown in the figure, the MobieSAM segmentation network model is a lightweight segmentation network model that has been fully fine-tuned on a public high-quality surgical instrument segmentation dataset. The figure shows the fine-tuning method: freeze the prompt encoder of MobieSAM, and only fine-tune the image encoder and mask decoder; each surgical instrument segmentation image sample used for fine-tuning is marked with a surgical instrument tip mask, and 5 discrete points are randomly sampled in the mask part as prompt information and input into the prompt encoder to obtain the prompt embedding, and the prompt embedding and the image features generated by the image encoder are input into the mask decoder to predict the surgical instrument tip mask, and the binary cross entropy loss and Dice similarity coefficient loss of the predicted mask and the true mask are calculated. The combined loss function formula is:

[0054]

[0055] in, is the binary cross entropy loss function between the predicted mask and the true mask, is the Dice similarity coefficient loss function between the predicted mask and the true mask.

[0056] The present invention adopts XMem as a tracking network model. The combination of the first frame image and the first frame surgical instrument tip mask obtained by interactive segmentation is stored in the network memory through the value encoder of the XMem tracking network model; the subsequent images are sequentially used as query values ​​to extract image features through the query encoder of the XMem tracking network model; and the surgical instrument tip mask of the subsequent frame image is sequentially generated through the decoder of the XMem tracking network model. The XMem model is an open source model known to those skilled in the art and will not be described in detail here.

[0057] The coordinates of the center of the surgical instrument tip mask are calculated specifically as follows: the surgical instrument tip mask is converted into a binary image; the binary image is corroded and expanded successively, the image moment of the surgical instrument tip mask after the corrosion and expansion operations is calculated, and the mask center is calculated according to the image moment.

[0058] The specific method of determining the center of the ROI region is to calculate the weighted average coordinates of the mask centers of the tips of each surgical instrument according to the weight of the surgical instrument. The formula is:

[0059]

[0060] in, is the center coordinate of the i-th surgical instrument tip mask, and is the image zero-order moment and first-order moment of the i-th surgical instrument tip mask, x ROI ,y ROI is the center coordinate of ROI area, w i is the weight of the ith surgical instrument, and n is the number of surgical instrument tips to be tracked.

[0061] like Figure 4 As shown, taking single surgical instrument tracking as an example, the figure shows the tracking effect of the surgical instrument tip mask in the video frame, and the red point in the figure represents the center coordinate of the ROI area.

[0062] The multi-task whole body controller with priority for the endoscope arm includes a high-priority distal center constraint task and a low-priority image visual servoing task. The distal motion center task is to minimize the error between the distal motion center point on the endoscope and the surgical incision point, and calculate the high-level joint speed control amount; the image visual servoing task is to minimize the error between the center coordinates of the image ROI area and the expected position coordinates, and calculate the low-level joint speed control amount.

[0063] like Figure 5 As shown in the figure, the distal motion center constraint task takes minimizing the error between the distal center point and the surgical incision point as the control target, which is represented by e in the figure. rcm Indicates; the surgical incision point is indicated by P in the figure trocar The distal end movement center point is the point on the axis of the endoscope closest to the surgical incision point, which is indicated by P in the figure. rcm The coordinates of the distal center point are obtained by calculating the kinematic model of the endoscope arm and the physical parameters of the endoscope. The specific calculation formula is:

[0064]

[0065]

[0066] p r =p trocar -p pre

[0067]

[0068] where p pre is the position coordinate of the end of the mirror holding arm, which is obtained by inputting the joint value into the forward kinematics formula f of the mirror holding arm; p r From the surgical incision point trocar Point to p pre The direction vector of post is the position coordinate of the endoscope tip, according to p pre and the calculated physical parameters of the endoscope; The error between the distal center point and the surgical incision point is the vertical distance from the surgical incision point to the endoscope axis, which is represented by p in the figure. e The calculation formula is: e =p trocar -p rcm Therefore, rcm =||p e || can be further expressed as p e The Euclidean space norm of .

[0069] In a specific implementation of the present invention, the calculation of the high-level joint speed control amount is specifically: calculating the error e between the distal motion center point and the surgical incision point rcm Jacobian matrix J of joint variables rcm , the distal motion center constraint task is modeled as a constrained optimization problem with joint speed constraints, which can be expressed as:

[0070]

[0071]

[0072]

[0073] Among them, J rcm K is the Jacobian matrix of the error between the distal motion center and the surgical incision point with respect to the joint variables, rcm is a positive definite matrix, e rcm is the error between the distal motion center and the surgical incision point, w is the relaxation factor, and w + 、w - is the maximum and minimum value of the relaxation factor; q is the joint angle, q + ,q - are the maximum and minimum values ​​of the joint angles, is the joint velocity, t is the time, ||.|| 2 To find the square of the modulus;

[0074] Further convert it into a quadratic programming problem to solve and obtain the high-level joint speed control value

[0075] The calculation of the low-level joint velocity control amount is specifically as follows: calculating the image Jacobian matrix J describing the relationship between the image feature point velocity and the camera motion velocity img ; Calculate the Jacobian matrix J describing the relationship between camera motion speed and joint speed cam ; The image visual servoing problem is modeled as a constrained optimization problem with joint speed constraints, which can be expressed as:

[0076]

[0077]

[0078]

[0079] Among them, J img is the image Jacobian matrix used to describe the relationship between the image feature point velocity and the camera motion velocity, J cam is the Jacobian matrix used to describe the relationship between camera motion speed and joint speed, For J rcm The null space projection of n is the identity matrix, is the matrix J rcm The pseudo-inverse of K v is a positive definite matrix, s des =(x des ,y des ) is the expected position coordinate, that is, the center point coordinate of the frame image; s ROI =(x ROI ,y ROI) is the center coordinate of the image ROI area;

[0080] It is further transformed into a quadratic programming problem to solve and obtain the low-level joint speed control value

[0081] The high-level and low-level joint speed control quantities of the integrated controller are used to obtain the joint speed control quantity of the mirror-holding arm. Specifically, the high-level joint speed control quantity is used to obtain the joint speed control quantity of the mirror-holding arm. and the underlying joint velocity control amount The final joint speed control amount is calculated according to the following formula:

[0082]

[0083] Will The current joint position is superimposed after time integration to obtain the next joint position output.

[0084] The present invention uses a multi-task whole-body controller with priority to obtain the joint space angle control value to control the motion of the endoscope holding arm. The errors of high-level and low-level tasks will converge to zero over time. The present invention can quickly and stably track any type and number of surgical instruments without the need for additional model training. The endoscope holding arm autonomously adjusts the posture of the endoscope according to the center position of the surgical instrument tip to optimize the surgical field of view and improve the efficiency of human-machine collaborative surgery.

[0085] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A method for controlling a surgical instrument arm based on interactive surgical instrument segmentation, characterized in that: include: The first frame of the image captured in real time by the endoscope at the end of the endoscope holding arm is obtained, and the tip of the surgical instrument to be tracked is interactively segmented to obtain the mask of the tip of each surgical instrument; Inputting each surgical tip mask in the first frame image and subsequent frame images into the tracking network model, and tracking the position of each surgical instrument tip mask in the subsequent frame images; Calculate the center coordinates of the mask of each surgical instrument tip, and determine the center coordinates of the ROI area in combination with the weight of each surgical instrument; adjust the field of view by adjusting the weight of each surgical instrument; The spatial angle control amount of the joint of the endoscope holding arm is obtained by a multi-task whole body controller with priority to control the movement of the endoscope holding arm; the multi-task whole body controller with priority includes a high-priority distal center constraint task and a low-priority image visual servoing task; the distal motion center task is to minimize the error between the distal motion center point on the endoscope and the surgical incision point, and calculate the high-level joint speed control amount; the image visual servoing task is to minimize the error of the ROI area center coordinate relative to the expected position coordinate, and calculate the low-level joint speed control amount; The high-level and low-level joint speed control quantities are integrated to obtain the mirror-holding arm joint speed control quantity, and the joint space angle control quantity is integrated to realize the motion control of the mirror-holding arm.

2. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 1, characterized in that: The interactive segmentation of the surgical instrument tip that needs to be tracked is performed to obtain the mask of each surgical instrument tip, specifically: discrete points and / or rectangular frames representing the surgical instrument tip area are extracted in the first frame image, and the discrete point and / or rectangular frame information is obtained by the prompt encoder of the lightweight segmentation network model to obtain the prompt embedding; the first frame image is input into the image encoder of the lightweight segmentation model to obtain the image feature; the prompt embedding and the image feature are input into the mask decoder of the lightweight segmentation model to obtain the surgical instrument tip mask.

3. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 2, characterized in that: The discrete points and / or rectangular frames are marked through an interactive graphical interface, which displays the image captured by the endoscope and is provided with a marking tool.

4. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 2, characterized in that: The lightweight segmentation network model adopts a fine-tuned MobieSAM segmentation network model. The fine-tuning method of the MobieSAM segmentation network model is as follows: freezing the prompt encoder of MobieSAM and only fine-tuning the image encoder and the mask decoder; each surgical instrument segmentation image sample used for fine-tuning is marked with a surgical instrument tip mask, a number of discrete points are randomly sampled in the mask part as prompt information and input into the prompt encoder to obtain prompt embedding, the prompt embedding and the image features generated by the image encoder are input into the mask decoder to predict the surgical instrument tip mask, the binary cross entropy loss and the Dice similarity coefficient loss of the predicted mask and the true mask are calculated, and the combined loss function is adopted to back-propagate and update the image encoder and mask decoder parameters.

5. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 1, characterized in that: The tracking of the mask position of each surgical instrument tip in the subsequent frame image is specifically as follows: The first frame image and the first frame surgical instrument tip mask obtained by interactive segmentation are combined and stored in the network memory through the value encoder of the XMem tracking network model; the subsequent frame images are used as query values ​​in turn through the query encoder of the XMem tracking network model to extract image features; the surgical instrument tip masks of the subsequent frame images are generated in turn through the decoder of the XMem tracking network model.

6. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 1, characterized in that: The coordinates of the center of the mask of each surgical instrument tip are calculated, and the coordinates of the center of the ROI area are determined in combination with the weight of each surgical instrument, specifically: Convert each surgical instrument tip mask into a binary image; Perform corrosion and expansion operations on each binary image successively; Calculate the image moments of each mask after erosion and dilation operations; Calculate the center coordinates of the mask of each surgical instrument tip according to the image moment of each mask; According to the weight of each surgical instrument, the weighted average coordinates of the mask centers of all surgical instrument tips are calculated as the coordinates of the center of the ROI area. The formula is: in, is the center coordinate of the i-th surgical instrument tip mask, and is the image zero-order moment and first-order moment of the i-th surgical instrument tip mask, x ROI ,y ROI is the center coordinate of ROI area, w i is the weight of the ith surgical instrument, and n is the number of surgical instrument tips to be tracked.

7. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 1, characterized in that: In the distal motion center task, the distal motion center point is the point on the endoscope axis that is closest to the surgical incision point.

8. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 1, characterized in that: The calculation of the high-level joint speed control amount is specifically as follows: The distal motion center constraint task is modeled as a constrained optimization problem with joint velocity constraints, expressed as: Among them, J rcm K is the Jacobian matrix of the error between the distal motion center and the surgical incision point with respect to the joint variables, rcm is a positive definite matrix, e rcm is the error between the distal motion center and the surgical incision point, w is the relaxation factor, and w + 、w - is the maximum and minimum value of the relaxation factor; q is the joint angle, q + ,q - are the maximum and minimum values ​​of the joint angles, is the joint velocity, t is the time, ||.|| 2 To find the square of the modulus; The high-level joint speed control quantity is obtained by solving the quadratic programming problem, denoted as 9. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 8, characterized in that: The calculation of the low-level joint speed control amount is specifically as follows: The image visual servoing task is modeled as a constrained optimization problem with joint velocity constraints, expressed as: Among them, J img is the image Jacobian matrix used to describe the relationship between the image feature point velocity and the camera motion velocity, J cam is the Jacobian matrix used to describe the relationship between camera motion speed and joint speed, For J rcm The null space projection of n is the identity matrix, is the matrix J rcm The pseudo-inverse of K v is a positive definite matrix, s des =(x des ,y des ) is the expected position coordinate, that is, the center point coordinate of the frame image; s ROI =(x ROI ,y ROI ) is the center coordinate of the image ROI area; The low-level joint speed control quantity is obtained by solving the quadratic programming problem, denoted as 10. The method for controlling a surgical instrument arm based on interactive surgical instrument segmentation according to claim 1, characterized in that: The combined high- and low-level joint speed control amounts are used to obtain the mirror-holding arm joint speed control amount, specifically: Control amount based on high-level joint velocity and the underlying joint velocity control amount Calculate the final joint velocity control amount Will The current joint position is superimposed after time integration to obtain the next joint position output.

Images