A mirror-holding arm control method based on interactive surgical instrument segmentation

Abstract

The application discloses a kind of based on interactive surgical instrument segmentation mirror holding arm control method, belong to surgical robot control technical field.It includes: the tip of surgical instrument in the first frame image of endoscope video returned by mirror holding arm end is interactively segmented;Subsequent frame image surgical instrument tip mask is automatically obtained by tracking network model and the center coordinates thereof are calculated;Combining the error of ROI region center coordinates and expected position coordinates calculated by custom surgical instrument weight;The joint space angle control quantity is obtained by the multi-task whole body controller with priority to carry out the motion control of mirror holding arm, and the error of high and low level tasks will converge to zero with time.The application can quickly and stably track any kind and quantity of surgical instrument without additional model training process.Mirror holding arm adjusts endoscope pose according to the center position of surgical instrument tip to optimize surgical field of view, and improves the efficiency of human-machine cooperative surgical operation.

Description

Technical Field

[0001] This invention relates to the field of surgical robot control technology, and in particular to a method for controlling a surgical instrument holding arm based on interactive surgical instrument segmentation. Background Technology

[0002] In recent years, laparoscopic minimally invasive surgery has become increasingly popular among patients due to its numerous advantages, such as less trauma and faster postoperative recovery. In traditional laparoscopic minimally invasive surgery, the assistant holding the endoscope needs to support it and adjust its position under the surgeon's instructions to ensure the surgical instruments remain within the field of vision. However, poor coordination between the surgeon and the assistant can lead to instrument collisions, potentially endangering patient safety. Furthermore, the assistant may experience muscle fatigue and hand tremors from maintaining a fixed posture for extended periods, reducing image stability and affecting the surgical process. Compared to traditional laparoscopic minimally invasive surgery, robot-assisted laparoscopic surgery offers high precision and stability, effectively replacing the assistant in holding the endoscope, improving surgical efficiency, and freeing up medical resources.

[0003] Current advanced endoscope-holding robot control systems are mainly divided into master-slave systems and visual servo systems. In master-slave systems, surgeons guide the robot to hold the endoscope through remote operation, voice control, head-mounted device control, and pedal buttons; the robot itself does not exhibit intelligent behavior. This type of system still requires the surgeon to take a break to control the robot, resulting in delayed response times and increasing surgical time. Visual servo systems, on the other hand, track surgical instruments, enabling the robot to autonomously adjust its position to keep the instruments within the field of vision. Compared to master-slave systems, visual servo systems improve the autonomy of the surgical process and are a more suitable intelligent solution for human-machine collaborative surgical scenarios.

[0004] Current advanced instrument tracking solutions typically use color-coding to track surgical instruments, but this method fails in surgical scenarios such as bleeding or occlusion. Alternatively, there are label-free tracking solutions based on object detection and segmentation, but these methods have certain limitations, such as the availability of privacy-sensitive surgical data and the need to repeatedly train the model for new instruments. Summary of the Invention

[0005] To address the above problems, this invention proposes a method for controlling the endoscope arm based on interactive surgical instrument segmentation. It achieves the localization and tracking of any surgical instrument without requiring model retraining. Furthermore, to achieve better endoscope arm control, a multi-level quadratic programming solver is used to solve the prioritized whole-body control task, thus realizing the motion control of the endoscope arm. The endoscope arm autonomously adjusts the endoscope pose based on the center position of the surgical instrument tip to optimize the surgical field of view and improve the efficiency of human-machine collaborative surgical operations. To achieve the above objectives, the technical solution of this invention includes the following steps:

[0006] A method for controlling a surgical instrument holding arm based on interactive surgical instrument segmentation includes:

[0007] The first frame image captured in real time by the endoscope at the end of the endoscope arm is obtained, and the tips of the surgical instruments that need to be tracked are interactively segmented to obtain the mask of each surgical instrument tip.

[0008] The surgical instrument tip masks in the first frame and subsequent frame images are input into the tracking network model to track the position of each surgical instrument tip mask in the subsequent frame images.

[0009] Calculate the center coordinates of the mask for each surgical instrument tip, and determine the center coordinates of the ROI region by combining the weights of each surgical instrument; adjust the field of view by adjusting the weights of each surgical instrument.

[0010] The movement of the endoscope arm is controlled by obtaining the joint spatial angle control quantities of the endoscope arm through a priority-based multi-task whole-body controller. The priority-based multi-task whole-body controller includes a high-priority distal center constraint task and a low-priority image visual servoing task. The distal motion center task is to minimize the error between the distal motion center point on the endoscope and the surgical incision point, and calculate the high-level joint velocity control quantities. The image visual servoing task is to minimize the error between the center coordinates of the ROI region and the desired position coordinates, and calculate the low-level joint velocity control quantities.

[0011] By integrating the joint velocity control quantities at high and low levels, the joint velocity control quantity of the mirror-holding arm is obtained, and the joint spatial angle control quantity is obtained by integration, thus realizing the motion control of the mirror-holding arm.

[0012] Furthermore, the interactive segmentation of the surgical instrument tips to be tracked to obtain the masks for each surgical instrument tip specifically involves: extracting discrete points and / or bounding boxes representing the surgical instrument tip region in the first frame image; obtaining cue embeddings by passing the cue encoder of the lightweight segmentation network model through the discrete point and / or bounding box information; inputting the first frame image into the image encoder of the lightweight segmentation model to obtain image features; and inputting the cue embeddings and image features into the mask decoder of the lightweight segmentation model to obtain the surgical instrument tip mask.

[0013] Furthermore, the discrete points and / or rectangles are marked via an interactive graphical interface that displays images captured by the endoscope and includes marking tools.

[0014] Furthermore, the lightweight segmentation network model adopts a fine-tuned MobieSAM segmentation network model. The fine-tuning method of the MobieSAM segmentation network model is as follows: freeze the MobieSAM cue encoder, and only fine-tune the image encoder and mask decoder; each surgical instrument segmentation image sample used for fine-tuning is marked with a surgical instrument tip mask, and several discrete points are randomly sampled in the mask part as cue information input to the cue encoder to obtain the cue embedding. The cue embedding and the image features generated by the image encoder are input into the mask decoder to predict the surgical instrument tip mask. The binary cross-entropy loss and Dice similarity coefficient loss of the predicted mask and the real mask are calculated. The combined loss function is used to backpropagate and update the parameters of the image encoder and mask decoder.

[0015] Furthermore, the tracking of the mask positions of the surgical instrument tips in subsequent frame images specifically involves:

[0016] The first frame image and the surgical instrument tip mask obtained from interactive segmentation are combined and stored in the network memory through the value encoder of the XMem tracking network model; subsequent frame images are used as query values ​​to extract image features through the query encoder of the XMem tracking network model; and the surgical instrument tip mask of subsequent frame images is generated sequentially through the decoder of the XMem tracking network model.

[0017] Furthermore, the calculation of the center coordinates of the mask of each surgical instrument tip, and the determination of the center coordinates of the ROI region in combination with the weight of each surgical instrument, specifically involves:

[0018] Convert the masks at the tips of each surgical instrument into binary images;

[0019] Erosion and dilation operations are performed sequentially on each binary image;

[0020] Calculate the image moments of each mask after erosion and dilation operations;

[0021] Calculate the center coordinates of the mask for each surgical instrument tip based on the image moments of each mask;

[0022] The weighted average coordinates of the center of the mask of the tips of all surgical instruments are calculated based on the weight of each surgical instrument and used as the center coordinates of the ROI region.

[0023] Furthermore, in the remote motion center task, the remote motion center point is the point closest to the surgical incision point along the endoscopic axis, and the coordinates of the remote center point are calculated using the kinematic model of the endoscope arm and the physical parameters of the endoscope.

[0024] Furthermore, the calculation of the high-level joint speed control quantity specifically involves:

[0025] The task of constraining the remote center of motion is modeled as a constrained optimization problem with joint velocity limitations, expressed as:

[0026]

[0027] Among them, J rcm K is the Jacobian matrix of the error between the distal motion center point and the surgical incision point with respect to joint variables. rcm e is a positive definite matrix rcm The error between the distal center of motion and the surgical incision point is represented by w, where w is the relaxation factor. + w - q represents the maximum and minimum values ​​of the relaxation factor; q is the joint angle. + q - These are the maximum and minimum values ​​of the joint angle, respectively. Let t be the joint velocity and t be the time. 2 To find the square of the modulus;

[0028] The high-level joint velocity control variable is obtained by solving a quadratic programming problem, denoted as...

[0029] Furthermore, the calculation of the low-level joint velocity control quantity specifically involves:

[0030] The image visual servoing task is modeled as a constrained optimization problem with joint velocity limitations, expressed as:

[0031]

[0032]

[0033]

[0034] Among them, J img J is the image Jacobian matrix used to describe the relationship between the velocity of image feature points and the camera motion velocity. cam Let Jacobian be the matrix used to describe the relationship between camera motion velocity and joint velocity. For J rcm null projection, I n It is the identity matrix. For matrix J rcm The pseudo-inverse, K v s is a positive definite matrix. des =(x des ,y des ) represents the desired position coordinates, i.e., the coordinates of the center point of the frame image; s ROI =(x ROI ,y ROI () represents the center coordinates of the ROI region in the image;

[0035] The low-level joint velocity control variables are obtained by solving a quadratic programming problem, denoted as...

[0036] Furthermore, the method of obtaining the joint speed control amount of the mirror-holding arm by integrating the high and low level joint speed control amounts is specifically as follows:

[0037] Based on high-level joint speed control amount and the lowest level joint speed control quantity Calculate the final joint velocity control value

[0038]

[0039] Will The current joint position is then superimposed after time integration to obtain the next joint position output.

[0040] Compared to existing visual servo control methods for lens-holding arms, the advantages of this invention are:

[0041] (1) The present invention uses a finely tuned MobieSAM segmentation network model to achieve high-quality segmentation of any surgical instrument, which effectively improves the generalization performance of the method.

[0042] (2) The present invention adopts a method combining a lightweight segmentation network model and a tracking network model to realize real-time tracking of surgical instruments in video stream, effectively ensuring the real-time requirements of the control system.

[0043] (3) This invention employs a multi-level quadratic programming solver to solve the priority-based whole-body control task, achieving control of multiple tasks and constraint on the joint velocity of the endoscope-holding arm, effectively improving the safety of the endoscope-holding arm movement. This invention can quickly and stably track any type and number of surgical instruments, and the endoscope-holding arm autonomously adjusts the endoscope posture according to the center position of the tip of the surgical instrument to optimize the surgical field of view and improve the efficiency of human-machine collaborative surgical operations. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0045] Figure 2 This is an example of the interactive surgical instrument segmentation interface of the present invention;

[0046] Figure 3 This is a schematic diagram of the MobieSAM fine-tuning method of the present invention;

[0047] Figure 4 This is a diagram illustrating the effect of the surgical instrument mask tracking according to the present invention.

[0048] Figure 5This is a schematic diagram of the remote motion center task of the present invention. Detailed Implementation

[0049] To more clearly illustrate the technical solution of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0050] like Figure 1 As shown, this invention proposes a control method for the endoscope arm based on interactive surgical instrument segmentation, which mainly includes: an interactive graphical interface, a segmentation and tracking network model, and a priority-based multi-task whole-body controller.

[0051] The interactive graphical interface is used to obtain prompts for surgical instruments that need to be tracked. These prompts are input into a segmentation and tracking network model to obtain a mask of the surgical instrument tip in the video frame. The center coordinates of the surgical instrument tip mask are calculated based on this mask, and the center coordinates of the Region of Interest (ROI) are determined by combining this with custom surgical instrument weights. The errors between the distal motion center point on the endoscope and the surgical incision point, and between the ROI center coordinates and the desired position, are input into a priority-based multi-task whole-body controller to obtain the joint motion control parameters for the surgical instruments. By adjusting the surgical instrument weights, the endoscope arm autonomously adjusts the endoscope pose based on the center position of the surgical instrument tip to optimize the surgical field of view.

[0052] like Figure 2 As shown, in the interactive graphical interface, the surgical instrument tip to be tracked in the first frame image is selected for segmentation using bounding boxes and / or discrete points, such as the electrocoagulation hook shown in the figure. Both bounding boxes and discrete points are marked. The interactive interface inputs the bounding box and discrete point information into the cue encoder of the fine-tuned MobieSAM segmentation network to obtain the cue embedding. The image is input into the image encoder of MobieSAM to obtain image features. The cue embedding and image features are then input into the mask decoder of MobieSAM to generate a surgical instrument tip mask.

[0053] like Figure 3 As shown, the MobieSAM segmentation network model is a lightweight segmentation network model that has been fully fine-tuned on a publicly available high-quality surgical instrument segmentation dataset. The figure illustrates the fine-tuning method: the MobieSAM cue encoder is frozen, and only the image encoder and mask decoder are fine-tuned; each surgical instrument segmentation image sample used for fine-tuning is labeled with a surgical instrument tip mask. Five discrete points are randomly sampled from the mask portion as cue information and input into the cue encoder to obtain the cue embedding. The cue embedding and the image features generated by the image encoder are input into the mask decoder to predict the surgical instrument tip mask. The binary cross-entropy loss and Dice similarity coefficient loss of the predicted mask and the ground mask are calculated. The combined loss function formula is:

[0054]

[0055] in, To predict the binary cross-entropy loss function between the mask and the real mask, The loss function is the Dice similarity coefficient between the predicted mask and the real mask.

[0056] This invention employs XMem as the tracking network model. The first frame image and the surgical instrument tip mask obtained from interactive segmentation are combined and stored in the network memory via the value encoder of the XMem tracking network model. Subsequent images are used sequentially as query values ​​to extract image features via the query encoder of the XMem tracking network model. The surgical instrument tip masks for subsequent frames are then generated sequentially via the decoder of the XMem tracking network model. The XMem model is an open-source model known to those skilled in the art and will not be described in detail here.

[0057] The specific steps for calculating the center coordinates of the surgical instrument tip mask are as follows: convert the surgical instrument tip mask into a binary image; perform erosion and dilation operations on the binary image sequentially; calculate the image moments of the surgical instrument tip mask after the erosion and dilation operations; and calculate the mask center based on the image moments.

[0058] The specific method for determining the center of the ROI region is as follows: Calculate the weighted average coordinates of the mask centers of the tips of each surgical instrument based on their respective weights. The formula is:

[0059]

[0060] in, Let the coordinates be the center coordinates of the i-th surgical instrument tip mask. and Let x be the zeroth and first moments of the image of the i-th surgical instrument tip mask. ROI ,y ROI Let w be the center coordinate of the ROI region. i Let be the weight of the i-th surgical instrument, and n be the number of surgical instrument tips being tracked.

[0061] like Figure 4 As shown, taking single surgical instrument tracking as an example, the figure shows the tracking effect of the surgical instrument tip mask in the video frame. The red dots in the figure represent the center coordinates of the ROI region.

[0062] The prioritized multi-task whole-body controller for the endoscope arm includes a high-priority distal center constraint task and a low-priority image visual servoing task. The distal center of motion task is to minimize the error between the distal center of motion point on the endoscope and the surgical incision point, and to calculate high-level joint velocity control quantities; the image visual servoing task is to minimize the error between the center coordinates of the image ROI region and the desired position coordinates, and to calculate low-level joint velocity control quantities.

[0063] like Figure 5 As shown, the distal motion center constraint task aims to minimize the error between the distal center point and the surgical incision point. In the figure, e represents this error. rcm Indicates that the surgical incision point is marked with P in the diagram. trocar The distal motion center point is the point closest to the surgical incision point along the endoscopic axis, denoted as P in the diagram. rcm The coordinates of the distal center point are obtained through calculation using the kinematic model of the endoscope arm and the physical parameters of the endoscope. The specific calculation formula is as follows:

[0064]

[0065]

[0066] p r =p trocar -p pre

[0067]

[0068] Where p pre These are the coordinates of the end effector position of the lens-holding arm, calculated by inputting the joint values ​​into the forward kinematics formula f of the lens-holding arm; p r From the surgical incision point p trocar Point to p pre The direction vector; p post These are the position coordinates of the endoscope tip, based on p pre And the results obtained from the calculation of the physical parameters of the endoscope; This represents a unit vector along the axial direction of the endoscope. The error between the distal center point and the surgical incision point is the perpendicular distance from the surgical incision point to the endoscope axis, denoted as p in the figure. e It is represented by the formula: p e =p trocar -p rcm Therefore, e rcm =||p e ||This can be further represented as p e The Euclidean space norm.

[0069] In one specific embodiment of the present invention, the calculation of the high-level joint velocity control quantity specifically involves: calculating the error e between the distal motion center point and the surgical incision point. rcm Regarding the Jacobian matrix J of the joint variables rcm Modeling the remote motion center constraint task as a constrained optimization problem with joint velocity limitations can be expressed as:

[0070]

[0071]

[0072]

[0073] Among them, J rcm K is the Jacobian matrix of the error between the distal motion center point and the surgical incision point with respect to joint variables. rcm e is a positive definite matrix rcm The error between the distal center of motion and the surgical incision point is represented by w, where w is the relaxation factor. + w - q represents the maximum and minimum values ​​of the relaxation factor; q is the joint angle. + q - These are the maximum and minimum values ​​of the joint angle, respectively. Let t be the joint velocity and t be the time. 2 To find the square of the modulus;

[0074] This problem is further transformed into a quadratic programming problem to obtain the high-level joint velocity control variables.

[0075] The calculation of the low-level joint velocity control quantity specifically involves: calculating the image Jacobian matrix J, which describes the relationship between the velocity of image feature points and the camera motion velocity. img Calculate the Jacobian matrix J that describes the relationship between camera motion velocity and joint velocity. cam The image visual servoing problem can be modeled as a constrained optimization problem with joint velocity limitations, which can be expressed as:

[0076]

[0077]

[0078]

[0079] Among them, J img J is the image Jacobian matrix used to describe the relationship between the velocity of image feature points and the camera motion velocity. cam Let Jacobian be the matrix used to describe the relationship between camera motion velocity and joint velocity. For J rcm null projection, I n It is the identity matrix. For matrix J rcm The pseudo-inverse, K v s is a positive definite matrix. des =(x des ,y des ) represents the desired position coordinates, i.e., the coordinates of the center point of the frame image; s ROI =(x ROI ,y ROI() represents the center coordinates of the ROI region in the image;

[0080] This problem is further transformed into a quadratic programming problem to obtain the low-level joint velocity control variables.

[0081] The integrated controller obtains the joint speed control of the mirror-holding arm by measuring the high and low level joint speed control values ​​specifically as follows: based on the high level joint speed control values... and the lowest level joint speed control quantity The final joint velocity control value is calculated using the following formula:

[0082]

[0083] Will The current joint position is then superimposed after time integration to obtain the next joint position output.

[0084] This invention utilizes a priority-based multi-task whole-body controller to obtain joint spatial angle control quantities for motion control of the endoscope-holding arm. Errors between high- and low-level tasks converge to zero over time. This invention can quickly and stably track any type and number of surgical instruments without requiring additional model training. The endoscope-holding arm autonomously adjusts the endoscope's pose based on the center position of the surgical instrument tip to optimize the surgical field of view and improve the efficiency of human-machine collaborative surgical procedures.

[0085] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A control system for a surgical instrument holding arm based on interactive surgical instrument segmentation, characterized in that, include: An interactive graphical interface module is used to acquire the first frame image captured in real time by the endoscope at the end of the endoscope arm, and to interactively segment the tips of the surgical instruments that need to be tracked to obtain the mask of each surgical instrument tip. The tracking and adjustment module is used to input the surgical instrument tip masks in the first frame image and subsequent frame images into the tracking network model to track the position of each surgical instrument tip mask in the subsequent frame images; and to calculate the center coordinates of each surgical instrument tip mask and determine the center coordinates of the ROI region by combining the weights of each surgical instrument; and to adjust the field of view by adjusting the weights of each surgical instrument. A multi-task whole-body control module is used to obtain the joint spatial angle control quantities of the endoscope-holding arm through a priority-based multi-task whole-body controller to control the movement of the endoscope-holding arm. The priority-based multi-task whole-body controller includes a high-priority distal center constraint task and a low-priority image visual servoing task. The distal motion center task is to minimize the error between the distal motion center point on the endoscope and the surgical incision point, and calculate the high-level joint velocity control quantities. The image visual servoing task is to minimize the error between the center coordinates of the ROI region and the desired position coordinates, and calculate the low-level joint velocity control quantities. Finally, the module integrates the high- and low-level joint velocity control quantities to obtain the joint velocity control quantities of the endoscope-holding arm, integrates them to obtain the joint spatial angle control quantities, and realizes the motion control of the endoscope-holding arm.

2. The endoscope-holding arm control system based on interactive surgical instrument segmentation according to claim 1, characterized in that, The interactive segmentation of the surgical instrument tips to be tracked to obtain the masks for each surgical instrument tip specifically involves: extracting discrete points and / or bounding boxes representing the surgical instrument tip region in the first frame image; obtaining cue embeddings by passing the cue encoder of the lightweight segmentation network model through the discrete point and / or bounding box information; inputting the first frame image into the image encoder of the lightweight segmentation model to obtain image features; and inputting the cue embeddings and image features into the mask decoder of the lightweight segmentation model to obtain the surgical instrument tip mask.

3. The endoscope-holding arm control system based on interactive surgical instrument segmentation according to claim 2, characterized in that, The discrete points and / or rectangles are marked via an interactive graphical interface that displays images captured by the endoscope and includes marking tools.

4. The endoscope-holding arm control system based on interactive surgical instrument segmentation according to claim 2, characterized in that, The lightweight segmentation network model employs a fine-tuned MobieSAM segmentation network model. The fine-tuning method for the MobieSAM segmentation network model is as follows: the MobieSAM cue encoder is frozen, and only the image encoder and mask decoder are fine-tuned. Each surgical instrument segmentation image sample used for fine-tuning is marked with a surgical instrument tip mask. Several discrete points are randomly sampled from the mask portion as cue information and input into the cue encoder to obtain the cue embedding. The cue embedding and the image features generated by the image encoder are input into the mask decoder to predict the surgical instrument tip mask. The binary cross-entropy loss and Dice similarity coefficient loss of the predicted mask and the real mask are calculated. A combined loss function is used to backpropagate and update the parameters of the image encoder and mask decoder.

5. The endoscope-holding arm control system based on interactive surgical instrument segmentation according to claim 1, characterized in that, The tracking of the mask positions of the surgical instrument tips in subsequent frame images specifically involves: The first frame image and the surgical instrument tip mask obtained from interactive segmentation are combined and stored in the network memory through the value encoder of the XMem tracking network model; subsequent frame images are used as query values ​​to extract image features through the query encoder of the XMem tracking network model; and the surgical instrument tip mask of subsequent frame images is generated sequentially through the decoder of the XMem tracking network model.

6. The endoscope-holding arm control system based on interactive surgical instrument segmentation according to claim 1, characterized in that, The calculation of the center coordinates of the mask for the tip of each surgical instrument, and the determination of the center coordinates of the ROI region based on the weight of each surgical instrument, is specifically as follows: Convert the masks at the tips of each surgical instrument into binary images; Erosion and dilation operations are performed sequentially on each binary image; Calculate the image moments of each mask after erosion and dilation operations; Calculate the center coordinates of the mask for each surgical instrument tip based on the image moments of each mask; The weighted average coordinates of the centers of the masked tips of all surgical instruments are calculated based on their respective weights and used as the center coordinates of the ROI region. The formula is as follows: ； in, For the first The center coordinates of the surgical instrument tip mask , and For the first The zero-order and first-order moments of the image of the surgical instrument tip mask. The coordinates of the center of the ROI region are... For the first There are several surgical instrument weights, where n is the number of surgical instrument tips being tracked.

7. The endoscope-holding arm control system based on interactive surgical instrument segmentation according to claim 1, characterized in that, In the aforementioned distal motion center task, the distal motion center point is the point closest to the surgical incision point along the endoscopic axis.

8. A control system for a surgical instrument segmentation-based endoscope arm according to claim 1, characterized in that, The calculation of the high-level joint speed control quantity is specifically as follows: The task of constraining the distal center of motion is modeled as a constrained optimization problem with joint velocity limitations, expressed as: ； in, This is the Jacobian matrix of the error between the distal motion center point and the surgical incision point with respect to joint variables. It is a positive definite matrix. This is the error between the distal center of motion and the surgical incision point. As a relaxation factor, These are the maximum and minimum values ​​of the relaxation factor; For joint angle, These are the maximum and minimum values ​​of the joint angle, respectively. For joint velocity, For time, To find the square of the modulus; The high-level joint velocity control variable is obtained by solving a quadratic programming problem, denoted as... .

9. A control system for a surgical instrument segmentation-based endoscope arm according to claim 8, characterized in that, The calculation of the low-level joint velocity control quantity is specifically as follows: The image visual servoing task is modeled as a constrained optimization problem with joint velocity limitations, expressed as: ； in, This is the image Jacobian matrix used to describe the relationship between the velocity of image feature points and the camera motion velocity. Let Jacobian be the matrix used to describe the relationship between camera motion velocity and joint velocity. for Zero-space projection, It is the identity matrix. For matrix The false rebellion, It is a positive definite matrix. These are the desired position coordinates, i.e., the coordinates of the center point of the frame image; The coordinates of the center of the ROI region in the image; The low-level joint velocity control variables are obtained by solving a quadratic programming problem, denoted as... .

10. A control system for a surgical instrument segmentation-based endoscope arm according to claim 1, characterized in that, The combined high and low level joint speed control values ​​are used to obtain the lens-holding arm joint speed control values, specifically as follows: Based on high-level joint speed control amount and the lowest level joint speed control quantity Calculate the final joint velocity control quantity : ； Will The current joint position is then superimposed after time integration to obtain the next joint position output.

Images